Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can not connect to any .fdb after stopped heavy OLTP-test (750 connections) [CORE3811] #4154

Open
firebird-automations opened this issue Apr 9, 2012 · 3 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: @pavel-zotov

Attachments:
gdb_fb_smp_server.zip

hi all

ISQL Version: WI-V2.5.1.26351 Firebird 2.5
Server version:
Firebird/linux AMD64 (access method), version "LI-V2.5.2.26448 Firebird 2.5"
Firebird/linux AMD64 (remote server), version "LI-V2.5.2.26448 Firebird 2.5/tcp (fbozzy)/P12"
Firebird/x86/Windows NT (remote interface), version "WI-V2.5.1.26351 Firebird 2.5/tcp (tlprg)/P12"
on disk structure version 11.2

Yesterday I run test with 750 connections with small and medium loading of database in each of them.
Today I've stopped this test (first I created some `special` stop-file which serves as signal to each connection script to be canceled - this way kills ~300 session; but after I kill all rest isql`s windows via pskill.exe).

Now the count of threads in fb_smp_server are 4:
bash-3.2$ ps -FLC fb_smp_server
UID PID PPID LWP C NLWP SZ RSS PSR STIME TTY TIME CMD
firebird 5982 5981 5982 0 4 20882 5108 2 13:50 ? 00:00:00 /opt/firebird/bin/fb_smp_server
firebird 5982 5981 5984 0 4 20882 5108 3 13:50 ? 00:00:00 /opt/firebird/bin/fb_smp_server
firebird 5982 5981 5985 0 4 20882 5108 7 13:50 ? 00:00:00 /opt/firebird/bin/fb_smp_server
firebird 5982 5981 5986 0 4 20882 5108 3 13:50 ? 00:00:00 /opt/firebird/bin/fb_smp_server

Unfortunatelly I can not establish new connections to any of databases that are placed on this machine.
The `top` utility for fb_smp_server shows that it is in idle state:

$ top -p $(pgrep fb_smp_server)

top - 15:06:57 up 1 day, 14:37, 7 users, load average: 0.02, 0.01, 0.00
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32933152k total, 20914936k used, 12018216k free, 224876k buffers
Swap: 33792716k total, 196k used, 33792520k free, 20130404k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5982 firebird 18 0 83528 5108 3624 S 0.0 0.0 0:00.41 fb_smp_server

The `iostat` utility also show that there is no any disk activity:

bash-3.2$ iostat -d -m -t 5
Linux 2.6.18-308.el5 (fbozzy) 04/09/2012

Time: 03:08:09 PM
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 195.40 0.66 2.44 91477 339559
sda1 0.00 0.00 0.00 1 0
sda2 187.78 0.01 1.94 1956 270086
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 7.61 0.64 0.50 89518 69472

Time: 03:08:14 PM
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.40 0.00 0.01 0 0
sda1 0.00 0.00 0.00 0 0
sda2 0.40 0.00 0.01 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 0.00 0.00 0.00 0 0

Time: 03:08:19 PM
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.00 0.00 0.00 0 0
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 0.00 0.00 0.00 0 0

Time: 03:08:24 PM
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 1.00 0.00 0.01 0 0
sda1 0.00 0.00 0.00 0 0
sda2 1.00 0.00 0.01 0 0
sda3 0.00 0.00 0.00 0 0
sda4 0.00 0.00 0.00 0 0
sda5 0.00 0.00 0.00 0 0

But any connect to any database (via local or TCP) hangs.
What should be done in such cases ?

@firebird-automations
Copy link
Collaborator Author

Commented by: @pavel-zotov

two files in attach:

1) gdb -q -x ... /opt/firebird/bin/fb_smp_server
1) gdb -q -x ... /opt/firebird/bin/.debug/fb_smp_server.debug

@firebird-automations
Copy link
Collaborator Author

Modified by: @pavel-zotov

Attachment: gdb_fb_smp_server.zip [ 12129 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @pavel-zotov

PS. I see the following output of netstat on this server (may be this will help):

netstat -anep | grep 3050
tcp 0 0 0.0.0.0:3050 0.0.0.0:* LISTEN 54323 382680 5982/fb_smp_server
tcp 157 0 127.0.0.1:3050 127.0.0.1:38899 CLOSE_WAIT 0 0 -
tcp 157 0 127.0.0.1:3050 127.0.0.1:38898 CLOSE_WAIT 0 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant