Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BIN_SHL and BIN_SHR: Incorrect results when shift >= 64 [CORE5230] #5509

Open
firebird-automations opened this issue May 8, 2016 · 0 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: @paulvink

If the second argument to BIT_SHL/BIT_SHR is 64 or greater, it is taken modulo 64 before the shift operation is performed. This leads to unexpected and incorrect results.

Example with positive numbers:

select bin_shr(120, 1) from rdb$database -> 60
select bin_shr(120, 6) from rdb$database -> 1
select bin_shr(120, 7) from rdb$database -> 0
..
select bin_shr(120, 63) from rdb$database -> 0
select bin_shr(120, 64) from rdb$database -> 120 (wrong)
select bin_shr(120, 65) from rdb$database -> 60 (wrong)

Example with negative numbers:

select bin_shr(-120, 1) from rdb$database -> -60
select bin_shr(-120, 2) from rdb$database -> -30
select bin_shr(-120, 7) from rdb$database -> -1
select bin_shr(-120, 9) from rdb$database -> -1
..
select bin_shr(-120, 63) from rdb$database -> -1
select bin_shr(-120, 64) from rdb$database -> -120 (wrong)
select bin_shr(-120, 65) from rdb$database -> -60 (wrong)

Instead of MOD'ing the shift argument by 64, it should be maximized at 64 (because all the original bits have been shifted out by then) - at least as long as the result type isn't bigger than int64.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant