Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Server hangs under load test [CORE5425] #5697

Closed
firebird-automations opened this issue Dec 18, 2016 · 13 comments
Closed

Server hangs under load test [CORE5425] #5697

firebird-automations opened this issue Dec 18, 2016 · 13 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: @ibprovider

Attachments:
FB_states.7z
5425.patch

Firebird server hangs under load tests

I launched 3 test processes with 4 test threads and own (3) databases.

Connection through fbclient.dll and INET

Firebird SRC, DUMP, EXE/DLL, PDB - http://dropmefiles.com/C61nz

Commits: c0f3524 9fd55f7

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

Text files with list of server threads and stacks

@firebird-automations
Copy link
Collaborator Author

Modified by: @ibprovider

Attachment: FB_states.7z [ 13049 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

Deadlock:

thread 10 (3a34)
frame 0d Jrd::JProvider::internalAttach
holds dbb_init_fini mutex
frame 05 Mutex::enter
wants plugins->mutex

thread 13 (1fd0)
frame 13 PluginManager::releasePlugin
holds plugins->mutex
frame 09 JRD_shutdown_database
wants dbb_init_fini mutex

Interesting threads stacks:

thread 10 (3a34)

#⁠ Call Site
00 ntdll!ZwWaitForAlertByThreadId+0x14
01 ntdll!LdrGetDllPath+0x8cf
02 ntdll!LdrGetDllPath+0x7ca
03 ntdll!LdrGetDllPath+0x60c
04 ntdll!RtlEnterCriticalSection+0x10b
05 fbclient!Firebird::Mutex::enter+0x6
06 fbclient!Firebird::MutexLockGuard::{ctor}+0x6
07 fbclient!Firebird::PluginManager::getPlugins+0x85
08 fbclient!Firebird::IPluginManagerBaseImpl<Firebird::PluginManager,Firebird::CheckStatusWrapper,Firebird::IVersionedImpl<Firebird::PluginManager,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IPluginManager\ > >::cloopgetPluginsDispatcher+0x76
09 engine12!Firebird::IPluginManager::getPlugins+0x2e
0a engine12!Firebird::GetPluginsFirebird::IKeyHolderPlugin\::GetPluginsFirebird::IKeyHolderPlugin\+0x134
0b engine12!Jrd::CryptoManager::KeyHolderPlugins::attach+0x66
0c engine12!Jrd::CryptoManager::attach+0x30
0d engine12!Jrd::JProvider::internalAttach+0xd86
0e engine12!Jrd::JProvider::attachDatabase+0x1c
0f engine12!Firebird::IProviderBaseImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::IPluginBaseImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IProvider\ > > > > > >::cloopattachDatabaseDispatcher+0x76
10 fbclient!Firebird::IProvider::attachDatabase+0x2e
11 fbclient!Why::Dispatcher::attachOrCreateDatabase+0x56f
12 fbclient!Why::Dispatcher::attachDatabase+0x1e
13 fbclient!Firebird::IProviderBaseImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::IPluginBaseImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IProvider\ > > > > > >::cloopattachDatabaseDispatcher+0x76
14 firebird!Auth::SrpServer::authenticate+0x26c
15 firebird!Firebird::IServerBaseImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::IAuthImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IPluginBaseImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IServer\ > > > > > > > >::cloopauthenticateDispatcher+0x6c
16 firebird!Firebird::IServer::authenticate+0x2d
17 firebird!`anonymous namespace'::ServerAuth::authenticate+0x2b2
18 firebird!accept_connection+0x39a
19 firebird!process_packet+0xc6
1a firebird!loopThread+0x1a8
1b firebird!`anonymous-namespace'::ThreadArgs::run+0x5
1c firebird!threadStart+0x54
1d ucrtbase!crt_at_quick_exit+0x7d
1e kernel32!BaseThreadInitThunk+0x22
1f ntdll!RtlUserThreadStart+0x34

thread 13 (1fd0)

#⁠ Call Site
00 ntdll!ZwWaitForAlertByThreadId+0x14
01 ntdll!LdrGetDllPath+0x8cf
02 ntdll!LdrGetDllPath+0x7ca
03 ntdll!LdrGetDllPath+0x60c
04 ntdll!RtlEnterCriticalSection+0x10b
05 engine12!Firebird::Mutex::enter+0xa
06 engine12!Jrd::Database::ExistenceRefMutex::enter+0xa
07 engine12!`anonymous-namespace'::RefMutexUnlock::enter+0xa
08 engine12!JRD_shutdown_database+0x11f
09 engine12!purge_attachment+0x424
0a engine12!Jrd::JAttachment::freeEngineData+0x1de
0b engine12!Firebird::IAttachmentBaseImpl<Jrd::JAttachment,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Jrd::JAttachment,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JAttachment,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IAttachment\ > > > >::cloopdetachDispatcher+0x56
0c fbclient!Firebird::IAttachment::detach+0x26
0d fbclient!Why::YAttachment::detach+0xb1
0e fbclient!Firebird::IAttachmentBaseImpl<Why::YAttachment,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Why::YAttachment,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Why::YAttachment,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IAttachment\ > > > >::cloopdetachDispatcher+0x56
0f srp!Firebird::IAttachment::detach+0x2a
10 srp!Auth::SrpManagement::release+0xd2
11 srp!Firebird::IManagementBaseImpl<Auth::SrpManagement,Firebird::CheckStatusWrapper,Firebird::IPluginBaseImpl<Auth::SrpManagement,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Auth::SrpManagement,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Auth::SrpManagement,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IManagement\ > > > > > >::cloopreleaseDispatcher+0x22
12 fbclient!Firebird::IReferenceCounted::release+0xa
13 fbclient!Firebird::PluginManager::releasePlugin+0x45
14 fbclient!Firebird::IPluginManagerBaseImpl<Firebird::PluginManager,Firebird::CheckStatusWrapper,Firebird::IVersionedImpl<Firebird::PluginManager,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::IPluginManager\ > >::cloopreleasePluginDispatcher+0x23
15 engine12!Firebird::IPluginManager::releasePlugin+0xe
16 engine12!Jrd::UserManagement::~UserManagement+0x1cf
17 engine12!Jrd::UserManagement::`scalar deleting destructor'+0x14
18 engine12!Jrd::jrd_tra::~jrd_tra+0xff
19 engine12!Jrd::jrd_tra::destroy+0x9f
1a engine12!TRA_commit+0x3b0
1b engine12!Jrd::JTransaction::commit+0x153
1c engine12!Firebird::ITransactionBaseImpl<Jrd::JTransaction,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Jrd::JTransaction,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JTransaction,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::ITransaction\ > > > >::cloopcommitDispatcher+0x56
1d fbclient!Firebird::ITransaction::commit+0x23
1e fbclient!Why::YTransaction::commit+0x5f
1f fbclient!Firebird::ITransactionBaseImpl<Why::YTransaction,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Why::YTransaction,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Why::YTransaction,Firebird::CheckStatusWrapper,Firebird::InheritFirebird::ITransaction\ > > > >::cloopcommitDispatcher+0x56
20 firebird!Firebird::ITransaction::commit+0x27
21 firebird!rem_port::end_transaction+0x1d0
22 firebird!process_packet+0x304
23 firebird!loopThread+0x1a8
24 firebird!`anonymous-namespace'::ThreadArgs::run+0x5
25 firebird!threadStart+0x54
26 ucrtbase!crt_at_quick_exit+0x7d
27 kernel32!BaseThreadInitThunk+0x22

@firebird-automations
Copy link
Collaborator Author

Modified by: Sean Leyne (seanleyne)

summary: [FB3] Server hang => Server hangs under load test

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

assignee: Alexander Peshkov [ alexpeshkoff ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

Dmitry, can you try with this patch? (looks like I was too pessimistic with original place for lock holder )

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

Attachment: 5425.patch [ 13051 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

Ok. Я запустил тесты.

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

At current moment (2 from 4 test sets are completed) all is ok.

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

All test sets completed.

It looks like the patch helped.

Server Name :Firebird (SuperServer, x64, Win)
Server Version :3.0.2.32646
Client Name :Firebird (x64)
Client Version :3.0.2.32646

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 3.0.2 [ 10785 ]

Fix Version: 4.0 Alpha 1 [ 10731 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Resolved [ 5 ]

QA Status: No test => Cannot be tested

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Closed [ 6 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants