Issue Details (XML | Word | Printable)

Key: CORE-5425
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Alexander Peshkov
Reporter: Kovalenko Dmitry
Votes: 0
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
Firebird Core

Server hangs under load test

Created: 18/Dec/16 10:21 AM   Updated: 25/Nov/17 08:24 PM
Component/s: Engine
Affects Version/s: 3.0.2
Fix Version/s: 3.0.2, 4.0 Alpha 1

File Attachments: 1. Text File 5425.patch (0.7 kB)
2. File FB_states.7z (3 kB)

Environment: FB 3.0.2.32644 SS x64, Win10 Pro Rus, 6950X/128GB, IBProvider Test System.

QA Status: Cannot be tested


 Description  « Hide
Firebird server hangs under load tests

I launched 3 test processes with 4 test threads and own (3) databases.

Connection through fbclient.dll and INET

Firebird SRC, DUMP, EXE/DLL, PDB - http://dropmefiles.com/C61nz

 All   Comments   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Kovalenko Dmitry added a comment - 18/Dec/16 10:22 AM
Text files with list of server threads and stacks

Vlad Khorsun added a comment - 18/Dec/16 11:31 AM
Deadlock:

thread 10 (3a34)
  frame 0d Jrd::JProvider::internalAttach
    holds dbb_init_fini mutex
  frame 05 Mutex::enter
    wants plugins->mutex

thread 13 (1fd0)
  frame 13 PluginManager::releasePlugin
    holds plugins->mutex
  frame 09 JRD_shutdown_database
    wants dbb_init_fini mutex


Interesting threads stacks:

thread 10 (3a34)

 # Call Site
00 ntdll!ZwWaitForAlertByThreadId+0x14
01 ntdll!LdrGetDllPath+0x8cf
02 ntdll!LdrGetDllPath+0x7ca
03 ntdll!LdrGetDllPath+0x60c
04 ntdll!RtlEnterCriticalSection+0x10b
05 fbclient!Firebird::Mutex::enter+0x6
06 fbclient!Firebird::MutexLockGuard::{ctor}+0x6
07 fbclient!Firebird::PluginManager::getPlugins+0x85
08 fbclient!Firebird::IPluginManagerBaseImpl<Firebird::PluginManager,Firebird::CheckStatusWrapper,Firebird::IVersionedImpl<Firebird::PluginManager,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IPluginManager> > >::cloopgetPluginsDispatcher+0x76
09 engine12!Firebird::IPluginManager::getPlugins+0x2e
0a engine12!Firebird::GetPlugins<Firebird::IKeyHolderPlugin>::GetPlugins<Firebird::IKeyHolderPlugin>+0x134
0b engine12!Jrd::CryptoManager::KeyHolderPlugins::attach+0x66
0c engine12!Jrd::CryptoManager::attach+0x30
0d engine12!Jrd::JProvider::internalAttach+0xd86
0e engine12!Jrd::JProvider::attachDatabase+0x1c
0f engine12!Firebird::IProviderBaseImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::IPluginBaseImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JProvider,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IProvider> > > > > > >::cloopattachDatabaseDispatcher+0x76
10 fbclient!Firebird::IProvider::attachDatabase+0x2e
11 fbclient!Why::Dispatcher::attachOrCreateDatabase+0x56f
12 fbclient!Why::Dispatcher::attachDatabase+0x1e
13 fbclient!Firebird::IProviderBaseImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::IPluginBaseImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Why::Dispatcher,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IProvider> > > > > > >::cloopattachDatabaseDispatcher+0x76
14 firebird!Auth::SrpServer::authenticate+0x26c
15 firebird!Firebird::IServerBaseImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::IAuthImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IPluginBaseImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Auth::SrpServer,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IServer> > > > > > > > >::cloopauthenticateDispatcher+0x6c
16 firebird!Firebird::IServer::authenticate+0x2d
17 firebird!`anonymous namespace'::ServerAuth::authenticate+0x2b2
18 firebird!accept_connection+0x39a
19 firebird!process_packet+0xc6
1a firebird!loopThread+0x1a8
1b firebird!`anonymous-namespace'::ThreadArgs::run+0x5
1c firebird!threadStart+0x54
1d ucrtbase!crt_at_quick_exit+0x7d
1e kernel32!BaseThreadInitThunk+0x22
1f ntdll!RtlUserThreadStart+0x34


thread 13 (1fd0)

 # Call Site
00 ntdll!ZwWaitForAlertByThreadId+0x14
01 ntdll!LdrGetDllPath+0x8cf
02 ntdll!LdrGetDllPath+0x7ca
03 ntdll!LdrGetDllPath+0x60c
04 ntdll!RtlEnterCriticalSection+0x10b
05 engine12!Firebird::Mutex::enter+0xa
06 engine12!Jrd::Database::ExistenceRefMutex::enter+0xa
07 engine12!`anonymous-namespace'::RefMutexUnlock::enter+0xa
08 engine12!JRD_shutdown_database+0x11f
09 engine12!purge_attachment+0x424
0a engine12!Jrd::JAttachment::freeEngineData+0x1de
0b engine12!Firebird::IAttachmentBaseImpl<Jrd::JAttachment,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Jrd::JAttachment,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JAttachment,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IAttachment> > > > >::cloopdetachDispatcher+0x56
0c fbclient!Firebird::IAttachment::detach+0x26
0d fbclient!Why::YAttachment::detach+0xb1
0e fbclient!Firebird::IAttachmentBaseImpl<Why::YAttachment,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Why::YAttachment,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Why::YAttachment,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IAttachment> > > > >::cloopdetachDispatcher+0x56
0f srp!Firebird::IAttachment::detach+0x2a
10 srp!Auth::SrpManagement::release+0xd2
11 srp!Firebird::IManagementBaseImpl<Auth::SrpManagement,Firebird::CheckStatusWrapper,Firebird::IPluginBaseImpl<Auth::SrpManagement,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IReferenceCountedImpl<Auth::SrpManagement,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Auth::SrpManagement,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IManagement> > > > > > >::cloopreleaseDispatcher+0x22
12 fbclient!Firebird::IReferenceCounted::release+0xa
13 fbclient!Firebird::PluginManager::releasePlugin+0x45
14 fbclient!Firebird::IPluginManagerBaseImpl<Firebird::PluginManager,Firebird::CheckStatusWrapper,Firebird::IVersionedImpl<Firebird::PluginManager,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IPluginManager> > >::cloopreleasePluginDispatcher+0x23
15 engine12!Firebird::IPluginManager::releasePlugin+0xe
16 engine12!Jrd::UserManagement::~UserManagement+0x1cf
17 engine12!Jrd::UserManagement::`scalar deleting destructor'+0x14
18 engine12!Jrd::jrd_tra::~jrd_tra+0xff
19 engine12!Jrd::jrd_tra::destroy+0x9f
1a engine12!TRA_commit+0x3b0
1b engine12!Jrd::JTransaction::commit+0x153
1c engine12!Firebird::ITransactionBaseImpl<Jrd::JTransaction,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Jrd::JTransaction,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Jrd::JTransaction,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::ITransaction> > > > >::cloopcommitDispatcher+0x56
1d fbclient!Firebird::ITransaction::commit+0x23
1e fbclient!Why::YTransaction::commit+0x5f
1f fbclient!Firebird::ITransactionBaseImpl<Why::YTransaction,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<Why::YTransaction,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<Why::YTransaction,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::ITransaction> > > > >::cloopcommitDispatcher+0x56
20 firebird!Firebird::ITransaction::commit+0x27
21 firebird!rem_port::end_transaction+0x1d0
22 firebird!process_packet+0x304
23 firebird!loopThread+0x1a8
24 firebird!`anonymous-namespace'::ThreadArgs::run+0x5
25 firebird!threadStart+0x54
26 ucrtbase!crt_at_quick_exit+0x7d
27 kernel32!BaseThreadInitThunk+0x22

Alexander Peshkov added a comment - 26/Dec/16 03:41 PM
Dmitry, can you try with this patch? (looks like I was too pessimistic with original place for lock holder )

Kovalenko Dmitry added a comment - 26/Dec/16 07:04 PM
Ok. Я запустил тесты.

Kovalenko Dmitry added a comment - 27/Dec/16 11:59 AM
At current moment (2 from 4 test sets are completed) all is ok.

Kovalenko Dmitry added a comment - 28/Dec/16 05:42 AM
All test sets completed.

It looks like the patch helped.

Server Name :Firebird (SuperServer, x64, Win)
Server Version :3.0.2.32646
Client Name :Firebird (x64)
Client Version :3.0.2.32646