Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversion from numeric literals to DECFLOAT adds precision that is not originally present [CORE5697] #5963

Closed
firebird-automations opened this issue Jan 6, 2018 · 8 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: @mrotteveel

Depends on CORE6060

When inserting into a table with a DECFLOAT(16) column,

Using

insert into decfloat16 (column1) values (1E300);
insert into decfloat16 (column1) values ('1E300');

and then select cast(column1 as varchar(30)) from decfloat16 will result in:

1.000000000000000E+300
1E+300

In other words, the conversion from literal adds 15 digits of precision that weren't actually present in the literal. The value resulting from string conversion is what I expect to get from the literal.

Commits: 5d70d4a

@firebird-automations
Copy link
Collaborator Author

Modified by: @mrotteveel

description: When inserting into a table with a DECFLOAT(16) column,

Using

insert into decfloat16 (column1) values (1E300);
insert into decfloat16 (column1) values ('1E300');

and then select cast(column1 as varchar(30)) from decfloat16 will result in:

1.000000000000000E+300
1E+300

In other words, the conversion from literal adds 15 digits of precision that weren't actually present in the literal.

=>

When inserting into a table with a DECFLOAT(16) column,

Using

insert into decfloat16 (column1) values (1E300);
insert into decfloat16 (column1) values ('1E300');

and then select cast(column1 as varchar(30)) from decfloat16 will result in:

1.000000000000000E+300
1E+300

In other words, the conversion from literal adds 15 digits of precision that weren't actually present in the literal. The value resulting from string conversion is what I expect to get from the literal.

@firebird-automations
Copy link
Collaborator Author

Modified by: @mrotteveel

Version: 4.0 Alpha 1 [ 10731 ]

Component: Engine [ 10000 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

assignee: Alexander Peshkov [ alexpeshkoff ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

Link: This issue depends on CORE6060 [ CORE6060 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @mrotteveel

This works OK now.

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 4.0 Beta 2 [ 10888 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Resolved [ 5 ]

QA Status: No test => Done successfully

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Closed [ 6 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants