[bug #68167] UB in indexing

Please use the bug tracker to post updates to a bug report.  The mailing list is intended as a read-only notification stream.  Info posted to this mailing list address won't appear in the tracker database where it is most useful.

URL:
  <https://savannah.gnu.org/bugs/?68167>

                 Summary: UB in indexing
                   Group: GNU Octave
               Submitter: dasergatskov
               Submitted: Sun 22 Mar 2026 07:55:52 PM UTC
                Category: Interpreter
                Severity: 3 - Normal
                Priority: 5 - Normal
              Item Group: Unexpected Error or Warning
                  Status: None
             Assigned to: None
         Originator Name:
        Originator Email:
             Open/Closed: Open
         Discussion Lock: Unlocked
                 Release: dev
        Operating System: Any
           Fixed Release: None
         Planned Release: None


    _______________________________________________________

Follow-up Comments:


-------------------------------------------------------
Date: Sun 22 Mar 2026 07:55:52 PM UTC By: Dmitri A. Sergatskov <dasergatskov>
In Octave compiled with UBSAN I see:

octave:1> a = magic(3)
a =

   8   1   6
   3   5   7
   4   9   2

octave:2> a(1,1,1,Inf,1)
../liboctave/array/idx-vector.cc:229:53: runtime error: inf is outside the
range of representable values of type 'long'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
../liboctave/array/idx-vector.cc:229:53 
../liboctave/array/idx-vector.cc:218:25: runtime error: signed integer
overflow: -9223372036854775808 - 1 cannot be represented in type
'octave_idx_type' (aka 'long')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
../liboctave/array/idx-vector.cc:218:25 
../liboctave/util/lo-array-errwarn.cc:222:40: runtime error: signed integer
overflow: 9223372036854775807 + 1 cannot be represented in type
'octave_idx_type' (aka 'long')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
../liboctave/util/lo-array-errwarn.cc:222:40 
error: a(_,_,_,-9223372036854775808,_): subscripts must be either integers 1
to (2^63)-1 or logicals


that also make clang buildbots to fail BIST (in index.tst) see e.g.:
https://buildbot.octave.org/#/builders/24/builds/2009/steps/7/logs/stdio

The diff:

diff -r 7e3eee8f5517 liboctave/array/idx-vector.cc
--- a/liboctave/array/idx-vector.cc     Sun Mar 22 17:53:22 2026 +0100
+++ b/liboctave/array/idx-vector.cc     Sun Mar 22 15:54:22 2026 -0400
@@ -39,6 +39,11 @@
 #include "oct-error.h"
 #include "oct-locbuf.h"
 
+static constexpr octave_idx_type IDX_MAX
+   = std::numeric_limits<octave_idx_type>::max () - 1;
+
+static constexpr double IDX_MAX_DBL = double (IDX_MAX);
+
 OCTAVE_BEGIN_NAMESPACE(octave)
 
 OCTAVE_NORETURN static void err_invalid_range ()
@@ -226,11 +231,13 @@
 inline octave_idx_type
 convert_index (double x, octave_idx_type& ext)
 {
-  octave_idx_type i = static_cast<octave_idx_type> (x);
+  if (! std::isfinite (x)
+      || x <= 0
+      || x >= IDX_MAX_DBL
+      || x != std::trunc (x))
+  err_invalid_index (x - 1.0);
 
-  if (static_cast<double> (i) != x)
-    err_invalid_index (x-1);
-
+  octave_idx_type i = static_cast<octave_idx_type> (x);
   return convert_index (i, ext);
 }
 
diff -r 7e3eee8f5517 liboctave/util/lo-array-errwarn.cc
--- a/liboctave/util/lo-array-errwarn.cc        Sun Mar 22 17:53:22 2026
+0100
+++ b/liboctave/util/lo-array-errwarn.cc        Sun Mar 22 15:54:22 2026
-0400
@@ -35,6 +35,9 @@
 #include "lo-array-errwarn.h"
 #include "oct-error.h"
 
+static constexpr octave_idx_type IDX_MAX
+  = std::numeric_limits<octave_idx_type>::max () - 1;
+
 OCTAVE_BEGIN_NAMESPACE(octave)
 
 // Text constants used to shorten code below.
@@ -219,7 +222,13 @@
 err_invalid_index (octave_idx_type n, octave_idx_type nd,
                    octave_idx_type dim, const std::string& var)
 {
-  err_invalid_index (std::to_string (n + 1), nd, dim, var);
+  if (n == IDX_MAX + 1)
+    err_invalid_index
+      (std::to_string
+         (static_cast<std::make_unsigned_t<octave_idx_type>> (n) + 1),
+       nd, dim, var);
+  else
+    err_invalid_index (std::to_string (n + 1), nd, dim, var);
 }
 
 void


fixes that for me.

Dmitri.
-- 








    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?68167>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/

-----BEGIN PGP SIGNATURE-----

iHUEABYIAB0WIQQk97aszIMMAvLLwm6qLAuaBUf3TgUCacBJSwAKCRCqLAuaBUf3
TiLiAQD1rdPlvynI4cGbsXJTdPCdPfPoiF6P0eibY699FmeRjwD+ItxI5u8DeMb6
YWcO5wJFGJQo0fAD/C3Sy6u43s/01w0=
=YDtB
-----END PGP SIGNATURE-----