Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Message "Operating system call munmap failed. Error code 12" can appear in firebird.log under heavy load (2.5.x, CS, SC) [CORE5643] #5909

Closed
firebird-automations opened this issue Oct 19, 2017 · 6 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: @pavel-zotov

Alex asked me to open special ticket for this issue.

Host has several databases which are served by single FB instance. Total number of attachments in work time is ~550.
After some time of normal work following message "Operating system call munmap failed. Error code 12" - did appear in firebird.log four times, and after that fb_smp_server was crashed.
I've sent stack-trace and additional info by e-mail, 18.10.2017 23:48.

Commits: d9f53fa f5a37fd a65276b

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

assignee: Alexander Peshkov [ alexpeshkoff ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

summary: Message "Operating system call munmap failed. Error code 12" can apper in firebird.log under heavy load (2.5.x, CS, SC) => Message "Operating system call munmap failed. Error code 12" can appear in firebird.log under heavy load (2.5.x, CS, SC)

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

Certainly one can't change Linux behavior regarding splitting VMA (Virtual Memory Area) structures in munmap() call.
What we get with this fix - memory block which can't be unmapped due to ENOMEM error is saved in linked list in MemoryPool and reused as soon as possible. I.e. we avoid throwing exception (which often happens in dtors and therefore highly unliked by c++11) and do not leak memory on this error. Remaining problem is unmapping such memory when firebird dynamic library (i.e. plugin) is unloaded - cleanup code does it's best to unmap linked blocks but if it fails (VMAs allocated by our library are mixed with allocated by others) memory leak appears unavoidable.

Solution was tested artificially reserving most of VMAs using code from https://stackoverflow.com/questions/43743555/munmap-failure-with-enomem-with-private-anonymous-mapping. After applying this fix SC server 2.5 stopped to fail randomly during massive deallocation of memory (was caused stopping TPC-C) and demonstrated stable behavior without memory leaks.

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 2.5.8 [ 10809 ]

Fix Version: 3.0.3 [ 10810 ]

Fix Version: 4.0 Beta 1 [ 10750 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Resolved [ 5 ]

QA Status: No test => Cannot be tested

Test Specifics: [Platform (Windows/Linux) specific]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Closed [ 6 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment