Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High load in touch thread [CORE5401] #5674

Closed
firebird-automations opened this issue Nov 18, 2016 · 6 comments
Closed

High load in touch thread [CORE5401] #5674

firebird-automations opened this issue Nov 18, 2016 · 6 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: David Hollings (dholling)

David Hollings posted on Firebird Support "Firebird Recurring Hourly Load".

The gist of the post is that he noticed that every hour all Firebird processes would wake up for a second or two causing high load, context switches etc - This using Firebird 2.5.x on Centos 6.x/7.x.

I've look into this a little bit more and notice that the issue seems to lie in the trace touch thread - specifically TouchThreadFunc in jrd/trace/TraceConfigStorage.cpp.
(Please note that I'm not a c++ developer, so there's a good chance that I'm missing something)

Each thread calls sem_timedwait [while (!semaphore->tryEnter(delay))] with the next touch time in seconds. The thread then blocks till seconds after epoch + current milliseconds.

When it returns it compares to the touch time:
if (!m_base->touch_time || m_base->touch_time < now)

Since touch_time is an integer value, current seconds will be equal to it for up to a full second. This results in a delay of zero being calculated and sem_trywait being called repeatedly for up to a second per thread until current seconds > touch_time.

To resolve shouldn't the comparison check for equality:
if (!m_base->touch_time || m_base->touch_time <= now)

I'm also curious as to why each thread is required to wake at the exact same time - would it not reduce contention if each thread blocked for a full hour?
Finally, for my own curiosity, what is the reason for touching the trace file hourly?

Commits: 59cf92b

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

reporter: Alexander Peshkov [ alexpeshkoff ] => David Hollings [ dholling ]

assignee: Alexander Peshkov [ alexpeshkoff ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 2.5.7 [ 10770 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Sean Leyne (seanleyne)

Edited to provide original reporter name and for readability.

@firebird-automations
Copy link
Collaborator Author

Modified by: Sean Leyne (seanleyne)

description: A few days ago I posted on Firebird Support "Firebird Recurring Hourly
Load".
The gist of the post is that we noticed that every hour all Firebird
processes would wake up for a second or two causing high load, context
switches etc - This using Firebird 2.5.x on Centos 6.x/7.x.

I've look into this a little bit more and notice that the issue seems to lie
in the trace touch thread - specifically TouchThreadFunc in
jrd/trace/TraceConfigStorage.cpp.
(Please note that I'm not a c++ developer, so there's a good chance that I'm
missing something)

Each thread calls sem_timedwait [while (!semaphore->tryEnter(delay))] with
the next touch time in seconds. The thread then blocks till seconds after
epoch + current milliseconds.
When it returns it compares to the touch time:

if (!m_base->touch_time || m_base->touch_time < now)

Since touch_time is an integer value, current seconds will be equal to it
for up to a full second. This results in a delay of zero being calculated
and sem_trywait being called repeatedly for up to a second per thread until
current seconds > touch_time.

To resolve shouldn't the comparison check for equality:

if (!m_base->touch_time || m_base->touch_time <= now)

I'm also curious as to why each thread is required to wake at the exact same
time - would it not reduce contention if each thread blocked for a full
hour?
Finally, for my own curiosity, what is the reason for touching the trace
file hourly?

=>

David Hollings posted on Firebird Support "Firebird Recurring Hourly Load".

The gist of the post is that he noticed that every hour all Firebird processes would wake up for a second or two causing high load, context switches etc - This using Firebird 2.5.x on Centos 6.x/7.x.

I've look into this a little bit more and notice that the issue seems to lie in the trace touch thread - specifically TouchThreadFunc in jrd/trace/TraceConfigStorage.cpp.
(Please note that I'm not a c++ developer, so there's a good chance that I'm missing something)

Each thread calls sem_timedwait [while (!semaphore->tryEnter(delay))] with the next touch time in seconds. The thread then blocks till seconds after epoch + current milliseconds.

When it returns it compares to the touch time:
if (!m_base->touch_time || m_base->touch_time < now)

Since touch_time is an integer value, current seconds will be equal to it for up to a full second. This results in a delay of zero being calculated and sem_trywait being called repeatedly for up to a second per thread until current seconds > touch_time.

To resolve shouldn't the comparison check for equality:
if (!m_base->touch_time || m_base->touch_time <= now)

I'm also curious as to why each thread is required to wake at the exact same time - would it not reduce contention if each thread blocked for a full hour?
Finally, for my own curiosity, what is the reason for touching the trace file hourly?

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Resolved [ 5 ]

QA Status: No test => Cannot be tested

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

status: Resolved [ 5 ] => Closed [ 6 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants