Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLONE -fbserver periodically crashes under load [CORE1297] #1718

Closed
firebird-automations opened this issue Jun 3, 2007 · 9 comments
Closed

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: Thomas Stamm (thomas)

Duplicates CORE544

Votes: 1

I'm encountering the same problem, an for me it is a SHOW STOPPER, it happens in an production environment, after using nearly the same configuration for years.

Nearly same Environment except I already tried newest version 1.5.4.4910

I already installed the debug Version and now have and dump file of this exception, if it is of interest I can post it (~150MB).

00 062ff980 0048647b 00007fb0 0bc8252c 014fadb0 fbserver!cache_transactions+0x8e [C:\fb15build\R1_5_4\firebird2\src\jrd\tpc.cpp @ 421]
01 062ff994 00440d99 062ffaac 014fadb3 062ffaac fbserver!TPC_initialize_tpc+0x5b [C:\fb15build\R1_5_4\firebird2\src\jrd\tpc.cpp @ 140]
02 062ffa3c 0042282f 062ffaac 00000005 05c40340 fbserver!TRA_start+0x319 [C:\fb15build\R1_5_4\firebird2\src\jrd\tra.cpp @ 1608]
03 062ffa58 77e67159 062ffbb0 00000001 0001d4c0 fbserver!jrd8_start_multiple+0x14f [C:\fb15build\R1_5_4\firebird2\src\jrd\jrd.cpp @ 3758]
WARNING: Stack unwind information not available. Following frames may be wrong.
04 062ffd70 00404c5d 003d3b34 003d3b2c 00000148 kernel32!ResetEvent+0x83
05 062ffdb8 00404440 003d3b2c 0813bed8 0813beb0 fbserver!Firebird::BePlusTree<Firebird::BlockInfo,Firebird::BlockInfo,Firebird::MemoryPool::InternalAllocator,Firebird::DefaultKeyValueFirebird::BlockInfo\,Firebird::BlockInfo,100,100>::add+0x15d [C:\fb15build\R1_5_4\firebird2\src\common\classes\tree.h @ 488]
06 062ffdf0 0040486a 003d3998 0854e580 0261dfcc fbserver!Firebird::MemoryPool::addFreeBlock+0x40 [C:\fb15build\R1_5_4\firebird2\src\common\classes\alloc.cpp @ 520]
07 062ffe7c 0041b9cc 00000007 0041b9e5 0040acea fbserver!Firebird::MemoryPool::deallocate+0x15a [C:\fb15build\R1_5_4\firebird2\src\common\classes\alloc.cpp @ 625]
08 062ffe84 0041b9e5 0040acea 062fff30 0051716d fbserver!ThreadPriorityScheduler::InternalGet+0xc [C:\fb15build\R1_5_4\firebird2\src\jrd\os\win32\thd_priority.cpp @ 86]
09 062ffe88 0040acea 062fff30 0051716d 0000001d fbserver!ThreadPriorityScheduler::Set+0x5 [C:\fb15build\R1_5_4\firebird2\src\jrd\os\win32\thd_priority.cpp @ 95]
0a 062ffe90 0051716d 0000001d 0854e93c 0854e580 fbserver!put_specific+0xa [C:\fb15build\R1_5_4\firebird2\src\jrd\thd.cpp @ 1816]
0b 062fff50 0051914e 0592c0b4 0854e580 0854e800 fbserver!process_packet+0x22d [C:\fb15build\R1_5_4\firebird2\src\remote\server.cpp @ 3146]
0c 062fff84 77bcb530 00000000 00000000 00000000 fbserver!thread+0xce [C:\fb15build\R1_5_4\firebird2\src\remote\server.cpp @ 4863]
0d 062fffb8 77e66063 00ff2008 00000000 00000000 msvcrt!endthreadex+0xa3
0e 062fffec 00000000 77bcb4bc 00ff2008 00000000 kernel32!GetModuleFileNameA+0xeb

------------

0048687e 8b742434 mov esi,dword ptr [esp+34h]
00486882 33d2 xor edx,edx
00486884 8bc6 mov eax,esi
00486886 f7f1 div eax,ecx
00486888 2bf2 sub esi,edx
0048688a 3bf5 cmp esi,ebp
0048688c 7718 ja fbserver!cache_transactions+0x96 (004868a6)
0048688e 56 push esi
0048688f 57 push edi
00486890 e83b000000 call fbserver!allocate_tpc (004868d0)
00486895 8b4c2434 mov ecx,dword ptr [esp+34h]
00486899 83c408 add esp,8
0048689c 03f1 add esi,ecx
0048689e 8903 mov dword ptr [ebx],eax ds:0023:0bc8252c=????????
004868a0 3bf5 cmp esi,ebp
004868a2 8bd8 mov ebx,eax
004868a4 76e8 jbe fbserver!cache_transactions+0x7e (0048688e)
004868a6 8b442434 mov eax,dword ptr [esp+34h]
004868aa 55 push ebp
004868ab 50 push eax
004868ac 6a00 push 0
004868ae 57 push edi
004868af e81c93fbff call fbserver!TRA_get_inventory (0043fbd0)
004868b4 8b442420 mov eax,dword ptr [esp+20h]
004868b8 83c410 add esp,10h
004868bb 5f pop edi

@firebird-automations
Copy link
Collaborator Author

Modified by: Thomas Stamm (thomas)

priority: Major [ 3 ] => Blocker [ 1 ]

Version: 1.5.4 [ 10100 ]

Version: 1.5.3 [ 10028 ]

Version: 1.5.2 [ 10027 ]

description: SFID: 1467936#⁠
Submitted By: venditere

Firebird will crash peridically and enter the
following message in firebird.log
\firebird\bin\fbserver.exe: terminated abnormally
(4294967295)
The guardian will then restart the service. It does
not crash on a regular interval. Sometimes it will
have only been running for 20 minutes and others it
may have lasted several hours. When the server
crashes it produces a .lck file for which I not yet
found a use.

Environment:
Firebird Superserver 1.5.2.4731
Windows Server 2003 Service Pack 1
Dual Xeon Processor. Processor Affinity set on
fbserver.exe to processor 1.
HyperThreading is disabled in BIOS.

Application information:
Jaybird jdbc driver used.
Hibernate / Spring primarily use database.
Apache Tomcat container.

The load on the server is fairly constant as this is
a controlled test environment. There will be about 50
clients from 2 servers at all times. There will be
many more writes than reads in this test scenario. The
size of the database is not large about 200 MB. In
order to provide more information I installed the .pdb
files for firebird and attached WinDbg in order to
obtain the following information:

*******************************************************
************************

CThread::setThreadIndex FAILED hr = 0x80004002
CThread::setThreadIndex FAILED hr = 0x80004002
CThread::setThreadIndex FAILED hr = 0x80004002

FAULTING_IP:
fbserver!cache_transactions+8a
[C:\fb15build\latest\firebird2\src\jrd\tpc.cpp @ 420]
0048634a 8903 mov [ebx],eax

EXCEPTION_RECORD: ffffffff -- (.exr ffffffffffffffff)
ExceptionAddress: 0048634a (fbserver!
cache_transactions+0x0000008a)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
Parameter[1]: 0f3d586c
Attempt to write to address 0f3d586c

FAULTING_THREAD: 00000988

DEFAULT_BUCKET_ID: APPLICATION_FAULT

PROCESS_NAME: fbserver.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction
at "0x%08lx" referenced memory at "0x%08lx". The
memory could not be "%s".

WRITE_ADDRESS: 0f3d586c

BUGCHECK_STR: ACCESS_VIOLATION

LAST_CONTROL_TRANSFER: from 00485f2b to 0048634a

STACK_TEXT:
04ddf980 00485f2b 00007fb0 0f3d586c 0014f2e1 fbserver!
cache_transactions+0x8a [C:\fb15build\latest\firebird2
\src\jrd\tpc.cpp @ 420]
04ddf994 00440849 04ddfaac 0014f2e2 04ddfaac fbserver!
TPC_initialize_tpc+0x5b [C:\fb15build\latest\firebird2
\src\jrd\tpc.cpp @ 140]
04ddfa3c 004224ff 04ddfaac 00000005 0a2f9f70 fbserver!
TRA_start+0x319 [C:\fb15build\latest\firebird2
\src\jrd\tra.cpp @ 1599]
04ddfa74 0044546b 0f674984 000001f4 0f67486c fbserver!
jrd8_start_multiple+0x14f
[C:\fb15build\latest\firebird2\src\jrd\jrd.cpp @ 3758]
04ddfbf4 00404c3d 0f3cd5a4 0f3cd59c 000000e8 fbserver!
VIO_verb_cleanup+0x74b [C:\fb15build\latest\firebird2
\src\jrd\vio.cpp @ 2926]
00000000 00000000 00000000 00000000 00000000 fbserver!
Firebird::BePlusTree<Firebird::BlockInfo,Firebird::Bloc
kInfo,Firebird::MemoryPool::InternalAllocator,Firebird:
:DefaultKeyValueFirebird::BlockInfo\,Firebird::BlockIn
fo,100,100>::add+0x15d [C:\fb15build\latest\firebird2
\src\common\classes\tree.h @ 488]

FOLLOWUP_IP:
fbserver!cache_transactions+8a
[C:\fb15build\latest\firebird2\src\jrd\tpc.cpp @ 420]
0048634a 8903 mov [ebx],eax

SYMBOL_STACK_INDEX: 0

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: fbserver!cache_transactions+8a

MODULE_NAME: fbserver

IMAGE_NAME: fbserver.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 41bd4f03

STACK_COMMAND: ~21s ; kb

FAILURE_BUCKET_ID: ACCESS_VIOLATION_fbserver!
cache_transactions+8a

BUCKET_ID: ACCESS_VIOLATION_fbserver!
cache_transactions+8a

Followup: MachineOwner
---------

=>

I'm encountering the same problem, an for me it is a SHOW STOPPER, it happens in an production environment, after using nearly the same configuration for years.

Nearly same Environment except I already tried newest version 1.5.4.4910

I already installed the debug Version and now have and dump file of this exception, if it is of interest I can post it (~150MB).

00 062ff980 0048647b 00007fb0 0bc8252c 014fadb0 fbserver!cache_transactions+0x8e [C:\fb15build\R1_5_4\firebird2\src\jrd\tpc.cpp @ 421]
01 062ff994 00440d99 062ffaac 014fadb3 062ffaac fbserver!TPC_initialize_tpc+0x5b [C:\fb15build\R1_5_4\firebird2\src\jrd\tpc.cpp @ 140]
02 062ffa3c 0042282f 062ffaac 00000005 05c40340 fbserver!TRA_start+0x319 [C:\fb15build\R1_5_4\firebird2\src\jrd\tra.cpp @ 1608]
03 062ffa58 77e67159 062ffbb0 00000001 0001d4c0 fbserver!jrd8_start_multiple+0x14f [C:\fb15build\R1_5_4\firebird2\src\jrd\jrd.cpp @ 3758]
WARNING: Stack unwind information not available. Following frames may be wrong.
04 062ffd70 00404c5d 003d3b34 003d3b2c 00000148 kernel32!ResetEvent+0x83
05 062ffdb8 00404440 003d3b2c 0813bed8 0813beb0 fbserver!Firebird::BePlusTree<Firebird::BlockInfo,Firebird::BlockInfo,Firebird::MemoryPool::InternalAllocator,Firebird::DefaultKeyValueFirebird::BlockInfo\,Firebird::BlockInfo,100,100>::add+0x15d [C:\fb15build\R1_5_4\firebird2\src\common\classes\tree.h @ 488]
06 062ffdf0 0040486a 003d3998 0854e580 0261dfcc fbserver!Firebird::MemoryPool::addFreeBlock+0x40 [C:\fb15build\R1_5_4\firebird2\src\common\classes\alloc.cpp @ 520]
07 062ffe7c 0041b9cc 00000007 0041b9e5 0040acea fbserver!Firebird::MemoryPool::deallocate+0x15a [C:\fb15build\R1_5_4\firebird2\src\common\classes\alloc.cpp @ 625]
08 062ffe84 0041b9e5 0040acea 062fff30 0051716d fbserver!ThreadPriorityScheduler::InternalGet+0xc [C:\fb15build\R1_5_4\firebird2\src\jrd\os\win32\thd_priority.cpp @ 86]
09 062ffe88 0040acea 062fff30 0051716d 0000001d fbserver!ThreadPriorityScheduler::Set+0x5 [C:\fb15build\R1_5_4\firebird2\src\jrd\os\win32\thd_priority.cpp @ 95]
0a 062ffe90 0051716d 0000001d 0854e93c 0854e580 fbserver!put_specific+0xa [C:\fb15build\R1_5_4\firebird2\src\jrd\thd.cpp @ 1816]
0b 062fff50 0051914e 0592c0b4 0854e580 0854e800 fbserver!process_packet+0x22d [C:\fb15build\R1_5_4\firebird2\src\remote\server.cpp @ 3146]
0c 062fff84 77bcb530 00000000 00000000 00000000 fbserver!thread+0xce [C:\fb15build\R1_5_4\firebird2\src\remote\server.cpp @ 4863]
0d 062fffb8 77e66063 00ff2008 00000000 00000000 msvcrt!endthreadex+0xa3
0e 062fffec 00000000 77bcb4bc 00ff2008 00000000 kernel32!GetModuleFileNameA+0xeb

------------

0048687e 8b742434 mov esi,dword ptr [esp+34h]
00486882 33d2 xor edx,edx
00486884 8bc6 mov eax,esi
00486886 f7f1 div eax,ecx
00486888 2bf2 sub esi,edx
0048688a 3bf5 cmp esi,ebp
0048688c 7718 ja fbserver!cache_transactions+0x96 (004868a6)
0048688e 56 push esi
0048688f 57 push edi
00486890 e83b000000 call fbserver!allocate_tpc (004868d0)
00486895 8b4c2434 mov ecx,dword ptr [esp+34h]
00486899 83c408 add esp,8
0048689c 03f1 add esi,ecx
0048689e 8903 mov dword ptr [ebx],eax ds:0023:0bc8252c=????????
004868a0 3bf5 cmp esi,ebp
004868a2 8bd8 mov ebx,eax
004868a4 76e8 jbe fbserver!cache_transactions+0x7e (0048688e)
004868a6 8b442434 mov eax,dword ptr [esp+34h]
004868aa 55 push ebp
004868ab 50 push eax
004868ac 6a00 push 0
004868ae 57 push edi
004868af e81c93fbff call fbserver!TRA_get_inventory (0043fbd0)
004868b4 8b442420 mov eax,dword ptr [esp+20h]
004868b8 83c410 add esp,10h
004868bb 5f pop edi

environment: Firebird Superserver 1.5.4.4910
Windows Server 2003 R2 Service Pack 1
Dual Xeon Processor. Processor Affinity set on
fbserver.exe to processor 2.
HyperThreading is disabled in BIOS.

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Does you have reproducible test case ?
Or may be you can tell us more about how it happens ?
Does you have the same traces for all mentioned FB versions ?

@firebird-automations
Copy link
Collaborator Author

Commented by: Thomas Stamm (thomas)

I'm sorry but I don't have any clue how to force this, I hoped somebody could tell me what is wrong and how avoid the crash (until it is fixed).
The crash happens on an very infrequent interval, sometimes anything is fine for 4,5,6,7 days. Than it crashes after 2 days.
I have experienced this problem on an new customer installation with FB 1.5.2 (after using nearly the same constellation for years/months in other installations). After the first crash I update the system to 1.5.4 (debug version) and installed 'userdump' to catch the crash. Up to know I have two dumps (of two different installations using both 1.5.4).

Doing slight inquiries into the code, it seems to have something do with the number of pending transactions, that are cashed.
Only conspicuous for me it the value of 'oldest'=0x7fb0=32688 which is in all of my and in case of 'venditere' (the first who reported that issue) the same value.
I'm haven't understood the code so far, to tell if this 'oldest' is only the value that depends on pagesize and how many transactions could be stored in one page or if it is really an absolute counter of open transactions.
But assuming that 32688 means I need a new page ' *tip_cache_ptr = allocate_tpc (tdbb, base);' does deliver an invalid page (=0) (for what reason ever).

I'm sorry that I have not more information, but If I can do anything to collect more details please let me know.

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

Duplicates CORE544.

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Duplicate [ 3 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

Link: This issue duplicates CORE544 [ CORE544 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pcisar

status: Resolved [ 5 ] => Closed [ 6 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pcisar

Workflow: jira [ 12270 ] => Firebird [ 15598 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

QA Status: No test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant