Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[security3.fdb] wrong page type page 22 is of wrong type (expected pointer, found index root) [CORE5415] #5688

Closed
firebird-automations opened this issue Dec 8, 2016 · 25 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: @ibprovider

Attachments:
security3__copy.zip
firebird.log

FB SuperServer x64 3.0.2.32641

Under load tests I get the some unexpected problem with connection to database

Test configuration: 8 test threads.

Message at test side:
[THR:002956] [07.12.2016 01:22:37] [test] connect to database ...
[THR:002956] [07.12.2016 01:22:37] ERROR: [blob.002.cmd.disable_defer_data.ansi.TBL_CS__TIS620.COL_BLOB.param_adLongVarWChar.bind__wstr.large1.len_4053.chars_TIS620.ins_UTF8.sel_NONE] Data Source initialization
1. [Unknown DBMS]: Error occurred during login, please check server firebird.log for details
2. [LCPI.IBProvider.3]: Ошибка подключения к базе данных.
Неопознанная ошибка

Message in firebird.log:
HOME4 Wed Dec 7 01:22:37 2016
Authentication error
database file appears corrupt (D:\PROGRAM_FILES\FIREBIRD_3_0_X64\SECURITY3.FDB)
wrong page type
page 22 is of wrong type (expected pointer, found index root)

The previous tests and the next tests (in this and other threads) work without problem.

In other word, this problem not affect to work of other tests.

Commits: 9d361b9 43e0777 bad0c72 8b8544e f72eee4 fcc44c1 2cda032

====== Test Details ======

Letter from hvlad, 26.03.2017 00:30.

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

Does it happen when running some specific test or randomly in different tests?
What does gfix say about SECURITY3.FDB after failed test?

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

>Does it happen when running some specific test or randomly in different tests?
Randomly

HOME4 Thu Dec 8 22:10:18 2016
Authentication error
database file appears corrupt (D:\PROGRAM_FILES\FIREBIRD_3_0_X64\SECURITY3.FDB)
wrong page type
page 16 is of wrong type (expected pointer, found index root)

Other test - charsets.rw.cmd.ansi.TBL_CS__WIN1251.VARCHAR__8.param_adBSTR.ins_cn_WIN1251.ins_user_NONE.sel_cn_WIN1251.sel_user_NONE.small.len_8.chars_WIN1251.bind__str_byref

I think, the problem linked with connection operation. Not with scenario of test.

>What does gfix say about SECURITY3.FDB after failed test?
Immediatelly after "failed test" - don't know :)

1. I stop all tests
2. Rename and copy security3.fdb to security3_copy.fdb
3. Check security3_copy.fdb - no errors

HOME4 Fri Dec 9 20:15:24 2016
Database: D:\DATABASE\FB_03_0_0\SECURITY3__COPY.FDB
Validation started

HOME4 Fri Dec 9 20:15:24 2016
Database: D:\DATABASE\FB_03_0_0\SECURITY3__COPY.FDB
Validation finished: 0 errors, 0 warnings, 0 fixed

--------------
I will attach archive with my security3.fdb. May be it may help.

@firebird-automations
Copy link
Collaborator Author

Modified by: @ibprovider

Attachment: security3__copy.zip [ 13041 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

If gfix does not find errors DB copy hardly helps. We all know about damaged databases but suppose nobody heard before about database that got after it magically repaired.
I.e. it's not DB damage - damaged was something like a copy of a page in the cache.

In this message:
> page 22 is of wrong type (expected pointer, found index root)
I see that page number may be different. Are expected/found types (pointer/index root) always same?

@firebird-automations
Copy link
Collaborator Author

Modified by: @ibprovider

Attachment: firebird.log [ 13044 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

>Are expected/found types (pointer/index root) always same?

Seems so. I attached the "firebird.log" file

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

The problem still exists - Firebird 3.0.2.32692 x64 SuperServer (Win10)

HOME4 Sat Mar 11 04:49:05 2017
Authentication error
database file appears corrupt (D:\PROGRAM_FILES\FIREBIRD_3_0_X64\SECURITY3.FDB)
wrong page type
page 22 is of wrong type (expected pointer, found index root)

HOME4 Sat Mar 11 04:49:05 2017
Authentication error
database file appears corrupt (D:\PROGRAM_FILES\FIREBIRD_3_0_X64\SECURITY3.FDB)
wrong page type
page 22 is of wrong type (expected pointer, found index root)

HOME4 Sat Mar 11 04:49:05 2017
Authentication error
database file appears corrupt (D:\PROGRAM_FILES\FIREBIRD_3_0_X64\SECURITY3.FDB)
wrong page type
page 22 is of wrong type (expected pointer, found index root)

-------------
In this case I ran four process with infinite connect/disconnect. Three process were stopped by error.

-------------- [script source code]
option explicit

dim cn

set cn=createobject("ADODB.Connection")

dim n

n=0

while(true)
n=n+1

cn.Provider="LCPI.IBProvider.3"

cn.Properties("location").value="localhost:d:\database\ram\ibp_test_fb30_d3_2.gdb"
cn.Properties("user id").value="GAMER"
cn.Properties("password").value="vermut"
cn.Properties("dbclient_type").value="fb.direct"
cn.Properties("OLE DB Services").value=0

wscript.echo n

call cn.Open()

call cn.Close()
wend

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Could you produce stack trace (or full memory dump) at the moment when exception raised ?
It happens in src\jrd\cch.cpp at page_validation_error() routine.
IIRC, you use own builds of Firebird, thus it shouldn't be a problem for you.
Or i could provide you with custom build.

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

>It happens in src\jrd\cch.cpp at page_validation_error() routine.

Ok. I set the breakpoint in page_validation_error function.

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Thanks.

Also, what kind of auth do you use (srp or legacy) ?

I tried to reproduce it with few isql's running in parallel script with 1000 {connect; commit;} statements - no luck

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

>Also, what kind of auth do you use (srp or legacy) ?

SRP

>I tried to reproduce it with few isql's running in parallel script with 1000 {connect; commit;} statements - no luck

I've got the same :)

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

Bingo!

Dump file coming soon.

> engine12.dll!page_validation_error(Jrd::thread_db * tdbb, Jrd::win * window, short type)Строка 4283 C++
engine12.dll!CCH_fetch(Jrd::thread_db * tdbb, Jrd::win * window, int lock_type, char page_type, int wait, const bool read_shadow)Строка 651 C++
[Внедренный фрейм] engine12.dll!CCH_FETCH(Jrd::thread_db *)Строка 83 C++
engine12.dll!get_pointer_page(Jrd::thread_db * tdbb, Jrd::jrd_rel * relation, Jrd::RelationPages * relPages, Jrd::win * window, unsigned long sequence, unsigned short lock)Строка 3328 C++
engine12.dll!DPM_data_pages(Jrd::thread_db * tdbb, Jrd::jrd_rel * relation)Строка 729 C++
engine12.dll!DPM_cardinality(Jrd::thread_db * tdbb, Jrd::jrd_rel * relation, const Jrd::Format * format)Строка 374 C++
engine12.dll!get_cardinality(Jrd::thread_db * tdbb, Jrd::jrd_rel * relation, const Jrd::Format * format)Строка 3061 C++
engine12.dll!OPT_compile_relation(Jrd::thread_db * tdbb, Jrd::jrd_rel * relation, Jrd::CompilerScratch * csb, unsigned long stream, bool needIndices)Строка 925 C++
engine12.dll!Jrd::RelationSourceNode::compile(Jrd::thread_db * tdbb, Jrd::OptimizerBlk * opt, bool __formal)Строка 845 C++
engine12.dll!OPT_compile(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb, Jrd::RseNode * rse, Firebird::Stack<Jrd::BoolExprNode *,16> * parent_stack)Строка 642 C++
engine12.dll!CMP_post_rse(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb, Jrd::RseNode * rse)Строка 559 C++
engine12.dll!Jrd::ForNode::pass2(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb)Строка 4664 C++
[Внедренный фрейм] engine12.dll!Jrd::StmtNode::doPass2(Jrd::thread_db *)Строка 1456 C++
engine12.dll!Jrd::CompoundStmtNode::pass2(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb)Строка 841 C++
[Внедренный фрейм] engine12.dll!Jrd::StmtNode::doPass2(Jrd::thread_db *)Строка 1456 C++
engine12.dll!Jrd::ReceiveNode::pass2(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb)Строка 6381 C++
[Внедренный фрейм] engine12.dll!Jrd::StmtNode::doPass2(Jrd::thread_db *)Строка 1456 C++
engine12.dll!Jrd::CompoundStmtNode::pass2(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb)Строка 841 C++
[Внедренный фрейм] engine12.dll!Jrd::StmtNode::doPass2(Jrd::thread_db *)Строка 1456 C++
engine12.dll!Jrd::JrdStatement::makeStatement(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb, bool internalFlag)Строка 235 C++
engine12.dll!Jrd::JrdStatement::makeRequest(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb, bool internalFlag)Строка 291 C++
engine12.dll!CMP_compile2(Jrd::thread_db * tdbb, const unsigned char * blr, unsigned long blr_length, bool internal_flag, unsigned long dbginfo_length, const unsigned char * dbginfo)Строка 176 C++
[Внедренный фрейм] engine12.dll!Jrd::AutoCacheRequest::compile(Jrd::thread_db *)Строка 96 C++
engine12.dll!compute_access(Jrd::thread_db * tdbb, const Jrd::SecurityClass * s_class, const Jrd::jrd_rel * view, long obj_type, const Firebird::MetaName & obj_name)Строка 2819 C++
engine12.dll!SCL_get_class(Jrd::thread_db * tdbb, const char * par_string)Строка 2212 C++
engine12.dll!SCL_init(Jrd::thread_db * tdbb, bool create, const Jrd::UserId & tempId)Строка 2506 C++
engine12.dll!Jrd::JProvider::internalAttach(Firebird::CheckStatusWrapper * user_status, const char * const filename, unsigned int dpb_length, const unsigned char * dpb, const Jrd::UserId * existingId)Строка 1675 C++
engine12.dll!Jrd::JProvider::attachDatabase(Firebird::CheckStatusWrapper * user_status, const char * filename, unsigned int dpb_length, const unsigned char * dpb)Строка 1373 C++
engine12.dll!Firebird::IProviderBaseImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::IPluginBaseImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IProvider\ > > > > > >::cloopattachDatabaseDispatcher(Firebird::IProvider * self, Firebird::IStatus * status, const char * fileName, unsigned int dpbLength, const unsigned char * dpb)Строка 9055 C++
fbclient.dll!Why::Dispatcher::attachOrCreateDatabase(Firebird::CheckStatusWrapper * status, bool createFlag, const char * filename, unsigned int dpbLength, const unsigned char * dpb)Строка 5714 C++
fbclient.dll!Why::Dispatcher::attachDatabase(Firebird::CheckStatusWrapper * status, const char * filename, unsigned int dpbLength, const unsigned char * dpb)Строка 5634 C++
fbclient.dll!Firebird::IProviderBaseImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::IPluginBaseImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IProvider\ > > > > > >::cloopattachDatabaseDispatcher(Firebird::IProvider * self, Firebird::IStatus * status, const char * fileName, unsigned int dpbLength, const unsigned char * dpb)Строка 9055 C++
firebird.exe!Auth::SrpServer::authenticate(Firebird::CheckStatusWrapper * status, Firebird::IServerBlock * sb, Firebird::IWriter * writerInterface)Строка 138 C++
firebird.exe!Firebird::IServerBaseImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::IAuthImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IPluginBaseImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IServer\ > > > > > > > >::cloopauthenticateDispatcher(Firebird::IServer * self, Firebird::IStatus * status, Firebird::IServerBlock * sBlock, Firebird::IWriter * writerInterface)Строка 9813 C++
firebird.exe!`anonymous namespace'::ServerAuth::authenticate(packet * send, ServerAuthBase::AuthenticateFlags flags)Строка 434 C++
firebird.exe!accept_connection(rem_port * port, p_cnct * connect, packet * send)Строка 1785 C++
firebird.exe!process_packet(rem_port * port, packet * sendL, packet * receive, rem_port * * result)Строка 4191 C++
firebird.exe!loopThread(void * __formal)Строка 5834 C++

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

Dump file + sources + binaries + PDB and others:

http://dropmefiles.com/G1Drh (330 MB)

I hope, dump file with 20GB size is not problem for you.

-----
Common description of state:

- 2 database
- 4 tests for connect/disconnect
- 2 other tests

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

Additional info

The problem was catched after 4x275000 connect/disconnect operations (<11 hours)

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

Please notify me, when I can continue execution of tests.

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Thanks for the dump. It is very interesting:
- page 22 is pointer page
- engine fetches pointer page 22
- in-memory page buffer of page 22 contains correct bytes of pointer page 22 except of the first byte - where page type is stored
there must be 4 (pointer page) but in page buffer memory there is 6 (index root page)

So far i have no explanation for this and continue to explore the things.

PS You can continue to run your tests, thanks again

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

Description of previous error in log-file

HOME4 Sun Mar 12 14:17:42 2017
Authentication error
database file appears corrupt (D:\PROGRAM_FILES\FIREBIRD_3_0_X64\SECURITY3.FDB)
wrong page type
page 22 is of wrong type (expected pointer, found index root)

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

Second 4 dumps (for same binaries and sources)

http://dropmefiles.com/qhYqi - 1 dump (270MB)
http://dropmefiles.com/lCJuc - 3 dumps (707MB)

firebird_2.dmp - failed (first case)
firebird_2b.dmp - failed (second case)
firebird_2c-ok.dmp - ok
firebird_2d-failed.dmp - failed (third case)

[ok]

I set the breakpoint to cch.cpp-648

/*648*/ if (bdb->bdb_buffer->pag_type != page_type && page_type != pag_undefined)
/*649*/ page_validation_error(tdbb, window, page_type);
/*650*/
/*651*/ return window->win_buffer;

and page_validation_error was not called

@firebird-automations
Copy link
Collaborator Author

Modified by: @hvlad

assignee: Vlad Khorsun [ hvlad ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

Version: 2.5.7 [ 10770 ]

Version: 3.0.1 [ 10730 ]

Version: 2.5.6 [ 10721 ]

Version: 3.0.0 [ 10740 ]

Version: 4.0 Initial [ 10621 ]

Version: 2.5.5 [ 10670 ]

Version: 2.5.4 [ 10585 ]

Version: 2.5.3 Update 1 [ 10650 ]

Version: 2.1.7 [ 10651 ]

Version: 2.5.3 [ 10461 ]

Version: 2.5.2 Update 1 [ 10521 ]

Version: 2.5.2 [ 10450 ]

Version: 2.5.1 [ 10333 ]

Version: 2.5.0 [ 10221 ]

Fix Version: 4.0 Alpha 1 [ 10731 ]

Fix Version: 3.0.3 [ 10810 ]

Fix Version: 2.5.8 [ 10809 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Big thanks to the Dmitry Kovalenko for the case, the many hours of testing and for the patience.

@firebird-automations
Copy link
Collaborator Author

Modified by: @hvlad

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Resolved [ 5 ]

QA Status: No test => Deferred

Test Details: waitig for reply, letter 25.03.2017 1637

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Resolved [ 5 ]

QA Status: Deferred => Cannot be tested

Test Details: waitig for reply, letter 25.03.2017 1637 => Letter from hvlad, 26.03.2017 00:30.

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Closed [ 6 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment