Operations when using "SET DECFLOAT BIND BIGINT,n" with result of 11+ digits, fail with "Decimal float invalid operation" [CORE6181] #6426
Comments
Modified by: @mrotteveeldescription: Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more. The resulting error is: Example: ===================== SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database; ===================== SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database; ===================== ===================== It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18-19 digits. Expected behaviour is that with BIGINT,3 will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer). As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers). Build 1614 is just after the INT128 changes. => Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more. The resulting error is: Example: ===================== SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database; ===================== SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database; ===================== ===================== It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18-19 digits. Expected behaviour is that with BIGINT,3 will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer). As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers). Build 1614 is just after the INT128 changes. |
Modified by: @mrotteveeldescription: Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more. The resulting error is: Example: ===================== SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database; ===================== SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database; ===================== ===================== It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18-19 digits. Expected behaviour is that with BIGINT,3 will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer). As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers). Build 1614 is just after the INT128 changes. => Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more. The resulting error is: Example: ===================== SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database; ===================== SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database; ===================== ===================== It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18 (full range) to 19 (partial range) digits. Expected behaviour is that with BIGINT,3 will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer). As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers). Build 1614 is just after the INT128 changes. |
Modified by: @mrotteveeldescription: Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more. The resulting error is: Example: ===================== SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database; ===================== SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database; ===================== ===================== It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18 (full range) to 19 (partial range) digits. Expected behaviour is that with BIGINT,3 will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer). As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers). Build 1614 is just after the INT128 changes. => Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more. The resulting error is: Example: ===================== SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database; ===================== SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database; ===================== ===================== It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18 (full range) to 19 (partial range) digits. Expected behaviour is that with BIGINT,3 will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer). As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers) and higher. Build 1614 is just after the INT128 changes. It is also reproducible in build 1641 (latest snapshot). |
Modified by: @mrotteveeldescription: Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more. The resulting error is: Example: ===================== SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database; ===================== SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database; ===================== ===================== It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18 (full range) to 19 (partial range) digits. Expected behaviour is that with BIGINT,3 will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer). As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers) and higher. Build 1614 is just after the INT128 changes. It is also reproducible in build 1641 (latest snapshot). => Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more. The resulting error is: Example: ===================== SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database; ===================== SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database; ===================== Expected result is: 12345678.901 SQL> select cast('12345678.90' as DECFLOAT(34)) from rdb$database; ===================== Expected result is: 12345678.900 It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18 (full range) to 19 (partial range) digits. Expected behaviour is that with BIGINT,3 will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer). As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers) and higher. Build 1614 is just after the INT128 changes. It is also reproducible in build 1641 (latest snapshot). |
Modified by: @AlexPeshkoffassignee: Alexander Peshkov [ alexpeshkoff ] |
Modified by: Sean Leyne (seanleyne)summary: Decfloat bind BIGINT,n with resulting value with 11 digits or more fail with Decimal float invalid operation. An indeterminant error occurred during an operation. => Operations using Decfloat bind BIGINT,n with result of 11+ digits, fail with "Decimal float invalid operation" |
Modified by: Sean Leyne (seanleyne)summary: Operations using Decfloat bind BIGINT,n with result of 11+ digits, fail with "Decimal float invalid operation" => Operations when using "SET DECFLOAT BIND BIGINT,n" with result of 11+ digits, fail with "Decimal float invalid operation" |
Modified by: @AlexPeshkoffstatus: Open [ 1 ] => Resolved [ 5 ] resolution: Fixed [ 1 ] Fix Version: 4.0 Beta 2 [ 10888 ] |
Modified by: @pavel-zotovstatus: Resolved [ 5 ] => Resolved [ 5 ] QA Status: No test => Done successfully |
Modified by: @pavel-zotovstatus: Resolved [ 5 ] => Closed [ 6 ] |
Submitted by: @mrotteveel
Conversions from a DECFLOAT using SET DECFLOAT BIND BIGINT,n fails when the resulting NUMERIC(18,n) value has 11 digits or more.
The resulting error is:
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.
Example:
SQL> set decfloat bind bigint,3;
SQL> select cast('1234567.890' as DECFLOAT(34)) from rdb$database;
=====================
1234567.890
SQL> select cast('1234567.8901' as DECFLOAT(34)) from rdb$database;
=====================
1234567.890
SQL> select cast('12345678.901' as DECFLOAT(34)) from rdb$database;
=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.
Expected result is: 12345678.901
SQL> select cast('12345678.90' as DECFLOAT(34)) from rdb$database;
=====================
Statement failed, SQLSTATE = 22000
Decimal float invalid operation. An indeterminant error occurred during an operation.
Expected result is: 12345678.900
It looks like an artificial limit of 10 digits is enforced, while a BIGINT can support 18 (full range) to 19 (partial range) digits.
Expected behaviour is that with BIGINT,3
select cast('9223372036854775.807' as DECFLOAT(34)) from rdb$database
will yield exactly 9223372036854775.807 (9223372036854775807 is the maximum value of a 64 bit integer).
As a datapoint, this worked fine in build 1575, but fails in 1614 (I don't have build in between those numbers) and higher. Build 1614 is just after the INT128 changes. It is also reproducible in build 1641 (latest snapshot).
Commits: 1827c56
The text was updated successfully, but these errors were encountered: