Issue Details (XML | Word | Printable)

Key: CORE-5415
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Vlad Khorsun
Reporter: Kovalenko Dmitry
Votes: 0
Watchers: 4
Operations

If you were logged in you would be able to see more operations.
Firebird Core

[security3.fdb] wrong page type page 22 is of wrong type (expected pointer, found index root)

Created: 08/Dec/16 10:53 AM   Updated: 25/Mar/17 09:49 PM
Component/s: None
Affects Version/s: 2.5.0, 2.5.1, 2.5.2, 2.5.2 Update 1, 2.5.3, 2.1.7, 2.5.3 Update 1, 2.5.4, 2.5.5, 4.0 Initial, 3.0.0, 2.5.6, 3.0.1, 2.5.7, 3.0.2
Fix Version/s: 4.0 Alpha 1, 2.5.8, 3.0.3

File Attachments: 1. Text File firebird.log (16 kB)
2. Zip Archive security3__copy.zip (82 kB)

Environment: FB SS x64 3.0.2.32641, Win10, IBProvider Test System

QA Status: Cannot be tested
Test Details: Letter from hvlad, 26.03.2017 00:30.


 Description  « Hide
FB SuperServer x64 3.0.2.32641

Under load tests I get the some unexpected problem with connection to database

Test configuration: 8 test threads.

Message at test side:
[THR:002956] [07.12.2016 01:22:37] [test] connect to database ...
[THR:002956] [07.12.2016 01:22:37] ERROR: [blob.002.cmd.disable_defer_data.ansi.TBL_CS__TIS620.COL_BLOB.param_adLongVarWChar.bind__wstr.large1.len_4053.chars_TIS620.ins_UTF8.sel_NONE] Data Source initialization
1. [Unknown DBMS]: Error occurred during login, please check server firebird.log for details
2. [LCPI.IBProvider.3]: Ошибка подключения к базе данных.
Неопознанная ошибка

Message in firebird.log:
HOME4 Wed Dec 7 01:22:37 2016
Authentication error
database file appears corrupt (D:\PROGRAM_FILES\FIREBIRD_3_0_X64\SECURITY3.FDB)
wrong page type
page 22 is of wrong type (expected pointer, found index root)

The previous tests and the next tests (in this and other threads) work without problem.

In other word, this problem not affect to work of other tests.

 All   Comments   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Alexander Peshkov added a comment - 09/Dec/16 12:59 PM
Does it happen when running some specific test or randomly in different tests?
What does gfix say about SECURITY3.FDB after failed test?

Kovalenko Dmitry added a comment - 09/Dec/16 05:30 PM
>Does it happen when running some specific test or randomly in different tests?
Randomly

HOME4 Thu Dec 8 22:10:18 2016
Authentication error
database file appears corrupt (D:\PROGRAM_FILES\FIREBIRD_3_0_X64\SECURITY3.FDB)
wrong page type
page 16 is of wrong type (expected pointer, found index root)

Other test - charsets.rw.cmd.ansi.TBL_CS__WIN1251.VARCHAR__8.param_adBSTR.ins_cn_WIN1251.ins_user_NONE.sel_cn_WIN1251.sel_user_NONE.small.len_8.chars_WIN1251.bind__str_byref

I think, the problem linked with connection operation. Not with scenario of test.

>What does gfix say about SECURITY3.FDB after failed test?
Immediatelly after "failed test" - don't know :)

1. I stop all tests
2. Rename and copy security3.fdb to security3_copy.fdb
3. Check security3_copy.fdb - no errors

HOME4 Fri Dec 9 20:15:24 2016
Database: D:\DATABASE\FB_03_0_0\SECURITY3__COPY.FDB
Validation started

HOME4 Fri Dec 9 20:15:24 2016
Database: D:\DATABASE\FB_03_0_0\SECURITY3__COPY.FDB
Validation finished: 0 errors, 0 warnings, 0 fixed

--------------
I will attach archive with my security3.fdb. May be it may help.

Alexander Peshkov added a comment - 09/Dec/16 06:03 PM
If gfix does not find errors DB copy hardly helps. We all know about damaged databases but suppose nobody heard before about database that got after it magically repaired.
I.e. it's not DB damage - damaged was something like a copy of a page in the cache.

In this message:
> page 22 is of wrong type (expected pointer, found index root)
I see that page number may be different. Are expected/found types (pointer/index root) always same?

Kovalenko Dmitry added a comment - 09/Dec/16 06:24 PM
>Are expected/found types (pointer/index root) always same?

Seems so. I attached the "firebird.log" file

Kovalenko Dmitry added a comment - 11/Mar/17 05:33 AM
The problem still exists - Firebird 3.0.2.32692 x64 SuperServer (Win10)

HOME4 Sat Mar 11 04:49:05 2017
Authentication error
database file appears corrupt (D:\PROGRAM_FILES\FIREBIRD_3_0_X64\SECURITY3.FDB)
wrong page type
page 22 is of wrong type (expected pointer, found index root)


HOME4 Sat Mar 11 04:49:05 2017
Authentication error
database file appears corrupt (D:\PROGRAM_FILES\FIREBIRD_3_0_X64\SECURITY3.FDB)
wrong page type
page 22 is of wrong type (expected pointer, found index root)


HOME4 Sat Mar 11 04:49:05 2017
Authentication error
database file appears corrupt (D:\PROGRAM_FILES\FIREBIRD_3_0_X64\SECURITY3.FDB)
wrong page type
page 22 is of wrong type (expected pointer, found index root)

-------------
In this case I ran four process with infinite connect/disconnect. Three process were stopped by error.

-------------- [script source code]
option explicit

dim cn

set cn=createobject("ADODB.Connection")

dim n

n=0

while(true)
 n=n+1
 
 cn.Provider="LCPI.IBProvider.3"

 cn.Properties("location").value="localhost:d:\database\ram\ibp_test_fb30_d3_2.gdb"
 cn.Properties("user id").value="GAMER"
 cn.Properties("password").value="vermut"
 cn.Properties("dbclient_type").value="fb.direct"
 cn.Properties("OLE DB Services").value=0

 wscript.echo n

 call cn.Open()

 call cn.Close()
wend

Vlad Khorsun added a comment - 11/Mar/17 10:38 AM
Could you produce stack trace (or full memory dump) at the moment when exception raised ?
It happens in src\jrd\cch.cpp at page_validation_error() routine.
IIRC, you use own builds of Firebird, thus it shouldn't be a problem for you.
Or i could provide you with custom build.

Kovalenko Dmitry added a comment - 11/Mar/17 11:51 AM
>It happens in src\jrd\cch.cpp at page_validation_error() routine.

Ok. I set the breakpoint in page_validation_error function.

Vlad Khorsun added a comment - 11/Mar/17 01:31 PM
Thanks.

Also, what kind of auth do you use (srp or legacy) ?

I tried to reproduce it with few isql's running in parallel script with 1000 {connect; commit;} statements - no luck

Kovalenko Dmitry added a comment - 11/Mar/17 05:40 PM
>Also, what kind of auth do you use (srp or legacy) ?

SRP

>I tried to reproduce it with few isql's running in parallel script with 1000 {connect; commit;} statements - no luck

I've got the same :)

Kovalenko Dmitry added a comment - 12/Mar/17 07:16 AM
Bingo!

Dump file coming soon.

> engine12.dll!page_validation_error(Jrd::thread_db * tdbb, Jrd::win * window, short type)Строка 4283 C++
  engine12.dll!CCH_fetch(Jrd::thread_db * tdbb, Jrd::win * window, int lock_type, char page_type, int wait, const bool read_shadow)Строка 651 C++
  [Внедренный фрейм] engine12.dll!CCH_FETCH(Jrd::thread_db *)Строка 83 C++
  engine12.dll!get_pointer_page(Jrd::thread_db * tdbb, Jrd::jrd_rel * relation, Jrd::RelationPages * relPages, Jrd::win * window, unsigned long sequence, unsigned short lock)Строка 3328 C++
  engine12.dll!DPM_data_pages(Jrd::thread_db * tdbb, Jrd::jrd_rel * relation)Строка 729 C++
  engine12.dll!DPM_cardinality(Jrd::thread_db * tdbb, Jrd::jrd_rel * relation, const Jrd::Format * format)Строка 374 C++
  engine12.dll!get_cardinality(Jrd::thread_db * tdbb, Jrd::jrd_rel * relation, const Jrd::Format * format)Строка 3061 C++
  engine12.dll!OPT_compile_relation(Jrd::thread_db * tdbb, Jrd::jrd_rel * relation, Jrd::CompilerScratch * csb, unsigned long stream, bool needIndices)Строка 925 C++
  engine12.dll!Jrd::RelationSourceNode::compile(Jrd::thread_db * tdbb, Jrd::OptimizerBlk * opt, bool __formal)Строка 845 C++
  engine12.dll!OPT_compile(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb, Jrd::RseNode * rse, Firebird::Stack<Jrd::BoolExprNode *,16> * parent_stack)Строка 642 C++
  engine12.dll!CMP_post_rse(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb, Jrd::RseNode * rse)Строка 559 C++
  engine12.dll!Jrd::ForNode::pass2(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb)Строка 4664 C++
  [Внедренный фрейм] engine12.dll!Jrd::StmtNode::doPass2(Jrd::thread_db *)Строка 1456 C++
  engine12.dll!Jrd::CompoundStmtNode::pass2(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb)Строка 841 C++
  [Внедренный фрейм] engine12.dll!Jrd::StmtNode::doPass2(Jrd::thread_db *)Строка 1456 C++
  engine12.dll!Jrd::ReceiveNode::pass2(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb)Строка 6381 C++
  [Внедренный фрейм] engine12.dll!Jrd::StmtNode::doPass2(Jrd::thread_db *)Строка 1456 C++
  engine12.dll!Jrd::CompoundStmtNode::pass2(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb)Строка 841 C++
  [Внедренный фрейм] engine12.dll!Jrd::StmtNode::doPass2(Jrd::thread_db *)Строка 1456 C++
  engine12.dll!Jrd::JrdStatement::makeStatement(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb, bool internalFlag)Строка 235 C++
  engine12.dll!Jrd::JrdStatement::makeRequest(Jrd::thread_db * tdbb, Jrd::CompilerScratch * csb, bool internalFlag)Строка 291 C++
  engine12.dll!CMP_compile2(Jrd::thread_db * tdbb, const unsigned char * blr, unsigned long blr_length, bool internal_flag, unsigned long dbginfo_length, const unsigned char * dbginfo)Строка 176 C++
  [Внедренный фрейм] engine12.dll!Jrd::AutoCacheRequest::compile(Jrd::thread_db *)Строка 96 C++
  engine12.dll!compute_access(Jrd::thread_db * tdbb, const Jrd::SecurityClass * s_class, const Jrd::jrd_rel * view, long obj_type, const Firebird::MetaName & obj_name)Строка 2819 C++
  engine12.dll!SCL_get_class(Jrd::thread_db * tdbb, const char * par_string)Строка 2212 C++
  engine12.dll!SCL_init(Jrd::thread_db * tdbb, bool create, const Jrd::UserId & tempId)Строка 2506 C++
  engine12.dll!Jrd::JProvider::internalAttach(Firebird::CheckStatusWrapper * user_status, const char * const filename, unsigned int dpb_length, const unsigned char * dpb, const Jrd::UserId * existingId)Строка 1675 C++
  engine12.dll!Jrd::JProvider::attachDatabase(Firebird::CheckStatusWrapper * user_status, const char * filename, unsigned int dpb_length, const unsigned char * dpb)Строка 1373 C++
  engine12.dll!Firebird::IProviderBaseImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::IPluginBaseImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IProvider> > > > > > >::cloopattachDatabaseDispatcher(Firebird::IProvider * self, Firebird::IStatus * status, const char * fileName, unsigned int dpbLength, const unsigned char * dpb)Строка 9055 C++
  fbclient.dll!Why::Dispatcher::attachOrCreateDatabase(Firebird::CheckStatusWrapper * status, bool createFlag, const char * filename, unsigned int dpbLength, const unsigned char * dpb)Строка 5714 C++
  fbclient.dll!Why::Dispatcher::attachDatabase(Firebird::CheckStatusWrapper * status, const char * filename, unsigned int dpbLength, const unsigned char * dpb)Строка 5634 C++
  fbclient.dll!Firebird::IProviderBaseImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::IPluginBaseImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IProvider> > > > > > >::cloopattachDatabaseDispatcher(Firebird::IProvider * self, Firebird::IStatus * status, const char * fileName, unsigned int dpbLength, const unsigned char * dpb)Строка 9055 C++
  firebird.exe!Auth::SrpServer::authenticate(Firebird::CheckStatusWrapper * status, Firebird::IServerBlock * sb, Firebird::IWriter * writerInterface)Строка 138 C++
  firebird.exe!Firebird::IServerBaseImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::IAuthImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IPluginBaseImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IServer> > > > > > > > >::cloopauthenticateDispatcher(Firebird::IServer * self, Firebird::IStatus * status, Firebird::IServerBlock * sBlock, Firebird::IWriter * writerInterface)Строка 9813 C++
  firebird.exe!`anonymous namespace'::ServerAuth::authenticate(packet * send, ServerAuthBase::AuthenticateFlags flags)Строка 434 C++
  firebird.exe!accept_connection(rem_port * port, p_cnct * connect, packet * send)Строка 1785 C++
  firebird.exe!process_packet(rem_port * port, packet * sendL, packet * receive, rem_port * * result)Строка 4191 C++
  firebird.exe!loopThread(void * __formal)Строка 5834 C++

Kovalenko Dmitry added a comment - 12/Mar/17 07:41 AM
Dump file + sources + binaries + PDB and others:

http://dropmefiles.com/G1Drh (330 MB)

I hope, dump file with 20GB size is not problem for you.

-----
Common description of state:

- 2 database
- 4 tests for connect/disconnect
- 2 other tests

Kovalenko Dmitry added a comment - 12/Mar/17 07:46 AM
Additional info

The problem was catched after 4x275000 connect/disconnect operations (<11 hours)

Kovalenko Dmitry added a comment - 12/Mar/17 10:47 AM - edited
Please notify me, when I can continue execution of tests.

Vlad Khorsun added a comment - 12/Mar/17 11:09 AM
Thanks for the dump. It is very interesting:
- page 22 is pointer page
- engine fetches pointer page 22
- in-memory page buffer of page 22 contains correct bytes of pointer page 22 except of the first byte - where page type is stored
  there must be 4 (pointer page) but in page buffer memory there is 6 (index root page)

So far i have no explanation for this and continue to explore the things.

PS You can continue to run your tests, thanks again

Kovalenko Dmitry added a comment - 12/Mar/17 11:30 AM
Description of previous error in log-file

HOME4 Sun Mar 12 14:17:42 2017
Authentication error
database file appears corrupt (D:\PROGRAM_FILES\FIREBIRD_3_0_X64\SECURITY3.FDB)
wrong page type
page 22 is of wrong type (expected pointer, found index root)

Kovalenko Dmitry added a comment - 12/Mar/17 01:31 PM - edited
Second 4 dumps (for same binaries and sources)

http://dropmefiles.com/qhYqi - 1 dump (270MB)
http://dropmefiles.com/lCJuc - 3 dumps (707MB)

firebird_2.dmp - failed (first case)
firebird_2b.dmp - failed (second case)
firebird_2c-ok.dmp - ok
firebird_2d-failed.dmp - failed (third case)

[ok]

I set the breakpoint to cch.cpp-648

/*648*/ if (bdb->bdb_buffer->pag_type != page_type && page_type != pag_undefined)
/*649*/ page_validation_error(tdbb, window, page_type);
/*650*/
/*651*/ return window->win_buffer;

and page_validation_error was not called

Vlad Khorsun added a comment - 24/Mar/17 06:09 PM
Big thanks to the Dmitry Kovalenko for the case, the many hours of testing and for the patience.