Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

round internal function (UDF) sometimes return wrong result for positive [CORE6098] #6347

Closed
firebird-automations opened this issue Jul 11, 2019 · 11 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: Alisa (alisa)

select
662.01 * 100.00 / 120.00,
round(662.01 * 100 / 120, 2),
rf2_round(662.01 * 100 / 120, 2)
from rdb$database

--- result is:
551,675; 551,67; 551,68

round - internal udf,
rf2_round - rfunc udf round

internal udf round take wrong result, dialect 1

@firebird-automations
Copy link
Collaborator Author

Commented by: Sean Leyne (seanleyne)

Have you tested this issue using ISQL?

It is possible that the rounding issue is with the tool/data components which you are using, and therefore not an Firebird/engine issue.

Additionally, are you aware that the rfunc UDF is not a project supported library?

@firebird-automations
Copy link
Collaborator Author

Commented by: Sean Leyne (seanleyne)

Internal/engine native functions are not UDFs

@firebird-automations
Copy link
Collaborator Author

Modified by: Sean Leyne (seanleyne)

Component: Engine [ 10000 ]

Component: UDF [ 10002 ] =>

@firebird-automations
Copy link
Collaborator Author

Commented by: Attila Molnár (e_pluribus_unum)

551.675 in 32 bit float = 5.5167498779296875E2
551.675 in 64 bit float = 5.51674999999999954525264911354

https://www.binaryconvert.com/result_float.html?decimal=053053049046054055053
https://www.binaryconvert.com/result_double.html?decimal=053053049046054055053

This is a Dialect1 issue, not round.

@firebird-automations
Copy link
Collaborator Author

Commented by: Alisa (alisa)

same udf from other dll, rfunc, work fine and return correct result.
in same select from same server rfunc return correct value.

Server version: WI-V6.3.9.27123 Firebird 2.5 HQbird

its bug 100%

client library is 32 bit

request is running from ibexpert
FB server and rfunc is 64 bit

@firebird-automations
Copy link
Collaborator Author

Modified by: Alisa (alisa)

summary: round UDF function sometimes return wrong result for positive => round internal function (UDF) sometimes return wrong result for positive

@firebird-automations
Copy link
Collaborator Author

Commented by: @mrotteveel

This is not a bug. The division operation in the - deprecated - dialect 1 yields a DOUBLE PRECISION value, and floating point values are imprecise to begin with. As Attila indicated, the value you are using is actually 551.674999999999954525264911354, so rounding to 551.67 is actually correct (more correct than rfuncs). However, given floating point values are imprecise, the actual rounding can depend on the exact operations and the order of operations used for rounding, and possibly the rounding mode used (eg up, down, half up, half down, half even, ceiling, floor, etc), so the behavior of rfunc round is also correct (just slightly less so, but it just happens to match your expectations).

The problem here is that you assume 662.01 * 100.00 / 120.00 is exactly 551,675 which is just not the case in dialect 1. If that is the behavior you want, then you should use dialect 3.

@firebird-automations
Copy link
Collaborator Author

Commented by: Sean Leyne (seanleyne)

As has been outlined, the dialect 1 functionality is "as designed".

@firebird-automations
Copy link
Collaborator Author

Modified by: Sean Leyne (seanleyne)

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Won't Fix [ 2 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pcisar

status: Resolved [ 5 ] => Closed [ 6 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Alisa (alisa)

pity, many people use dialect 1 and function need to return correct result.
if you cant wrote function that return correct result replace you code with rfunc.
who need function which work incorrectly. you may said everything,
as designed an so on, but same request to oracle return correct result.
-----------------
select
662.01 * 100.00 / 120.00,
round(662.01 * 100 / 120, 2)
from dual
-----------------

return
551.675 551.68

they are dont cnown about dialect in firebird, simply them return correct result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant