Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems when a table grows beyond 65535 pointer pages [CORE4384] #4706

Closed
firebird-automations opened this issue Apr 4, 2014 · 9 comments
Closed

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: @ibprovider

Attachments:
miscelan-logs.zip
inserts-batch-and-sql.zip

The USHORT pp_sequence number wraps and it may lead to unexpected issues like an infinite loop inside extend_relation() or locate_space(). This issue manifests itself when a table (including garbage) gets bigger than 256GB (for 4KB page) ... 4TB (for 16KB page) and remains even after a garbage collection pass.

Commits: 1655ba4 bd3ce4e FirebirdSQL/fbt-repository@ec2c79f FirebirdSQL/fbt-repository@3d5e270

====== Test Details ======

There is no .fbt file for this ticket, but I've done such test and result was: table CAN be of such size that more than 64K PP will be related to it.

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

reporter: Dmitry Yemanov [ dimitr ] => Kovalenko Dmitry [ _dima_k_ ]

assignee: Dmitry Yemanov [ dimitr ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

Fix Version: 3.0 Beta 2 [ 10586 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 2.5.4 [ 10585 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @pavel-zotov

http://svn.code.sf.net/p/firebird/code/firebird/trunk/ChangeLog:
> 2014-12-29 17:40 dimitr <...>
> This commit is expected to fix CORE4384: Problems when a table grows beyond 65535 pointer pages. Testing would be appreciated.

I've done somewhat like test about ability to create and fill table with more than 65K pointer pages.
Short conclusion: we CAN create such table, at least on database with page_size = 4K.

Environment: OS = Windows XP, FB = WI-T6.3.0.31543, SuperServer; database = page_4k, fw_OFF

firebird.conf:

DefaultDbCachePages = 64K
LockHashSlots = 22111
TempCacheLimit = 2147483647

RemoteServicePort = 3333

MaxUnflushedWrites = -1
MaxUnflushedWriteTime = -1

ExternalFileAccess = Restrict C:\1INSTALL\FBTEST
MaxUserTraceLogSize = 99999

FileSystemCacheThreshold = 65536K
AuthServer = Legacy_Auth,Srp
AuthClient = Legacy_Auth,Srp,Win_Sspi
WireCrypt = Disabled
UserManager = Legacy_UserManager
BugcheckAbort = 1

Two tables are in this database:

-- main:
recreate table t(id bigint, s01 varchar(8), s02 varchar(8));

-- log with info for later analyzing of inserts speed:
recreate table ins_log(
dts_beg timestamp, dts_end timestamp,
elapsed_ms int computed by(datediff(millisecond from dts_beg to dts_end)),
gen_after_ins bigint, pp_after_ins int
);
commit;

recreate sequence g;
commit;

Test was run during several days, final database size = ~305Gb.

Result:

Database "E:\DATA\HUGE_CORE_4384.FDB"
Database header page information:
Flags 0
Generation 14032
System Change Number 0
Page size 4096
ODS version 12.0
Oldest transaction 1149
Oldest active 11496
Oldest snapshot 11496
Next transaction 11497
Sequence number 0
Next attachment ID 5741
Implementation HW=Intel/i386 little-endian OS=Windows CC=MSVC
Shadow count 0
Page buffers 0
Next header page 0
Database dialect 3
Creation date Jan 18, 2015 19:02:14
Attributes

Variable header data:
\*END\*

Database file sequence:
File E:\DATA\HUGE_CORE_4384.FDB is the only file

Analyzing database pages ...
T (128)
Primary pointer page: 211, Index root page: 212
Pointer pages: 127855, data page slots: 103306800
Data pages: 103306800, average fill: 67%
Primary pages: 103306800, secondary pages: 0, swept pages: 0
Empty pages: 5, full pages: 103306794
Fill distribution:
0 - 19% = 5
20 - 39% = 0
40 - 59% = 1
60 - 79% = 103306794
80 - 99% = 0

I decided to check trivial COUNT operation on this table and create INDEX on it.

Results are in attach "miscelan-logs.zip".

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

Attachment: miscelan-logs.zip [ 12670 ]

Attachment: inserts-batch-and-sql.zip [ 12671 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

Attachment: fill_huge_table_log_until_count_3e9_fw-ON.zip [ 12674 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

Attachment: fill_huge_table_log_until_count_3e9_fw-ON.zip [ 12674 ] =>

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Resolved [ 5 ]

QA Status: Done successfully

Test Details: There is no .fbt file for this ticket, but I've done such test and result was: table CAN be of such size that more than 64K PP will be related to it.

@firebird-automations
Copy link
Collaborator Author

Modified by: @pcisar

status: Resolved [ 5 ] => Closed [ 6 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment