Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Firebird 2.1.3 64bit superserver - hanging TCP conenctions [CORE3002] #3384

Open
firebird-automations opened this issue May 12, 2010 · 14 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: Karel Rys (vandrovnik)

At the server side, there are left unclosed attachments.

These conenctions are closed at OS level (netstat -a -n does not list them).
Firebird still lists them in MON$ATTACHMENTS.

In Firebird.conf, I have set up:
ConnectionTimeout = 300
DummyPacketInterval = 120

Unfortunatelly, this does not help and connections are open, until Firebird is restarted.

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

SuperServer or Classic?

@firebird-automations
Copy link
Collaborator Author

Commented by: Karel Rys (vandrovnik)

Superserver - sorry, I put this info in subject only.

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

How long are those connections left open? DummyPacketInterval = 120 means two minutes between pings. Also, have you tried to reduce the default TCP keep-alive interval on Windows? Did it help?

@firebird-automations
Copy link
Collaborator Author

Commented by: Karel Rys (vandrovnik)

On windows, these connections are not active in the system. Netstat -a -n does not show them at all.

Firebird was restarted 50 hours ago.
Oldest connections I have seen are 43 hours old (yes, almost two days!).
Customer did not allow me to change default TCP keep-alive interval at their server, but as far as I know, this is about 2 hours long, so hanging conenctions are much older...

It seems that Firebird does not try to ping these conenctions, or it does try, but does not remove them, even thought connections are already dead.

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

Does the connection count returned via the server properties (server tray icon or Services / Server Properties in IBExpert) match the one returned with select count(*) from mon$attachments?

@firebird-automations
Copy link
Collaborator Author

Commented by: Karel Rys (vandrovnik)

Yes, at this moment, both show 5.

Server Version Info
---------------------------------------------------------------------------
Server Version: WI-V2.1.3.18185 Firebird 2.1
Server Implementation: Firebird/x86-64/Windows NT
Service Version: 2

Configuration Info
---------------------------------------------------------------------------
Base File: C:\Program Files\Firebird\Firebird_2_1\
Lock File: C:\Program Files\Firebird\Firebird_2_1\
Message File: C:\Program Files\Firebird\Firebird_2_1\
Security Database: C:\Program Files\Firebird\Firebird_2_1\security2.fdb

Database Info
---------------------------------------------------------------------------
Number of connections: 5
Number of databases: 1
Databases: D:\FB\CZECHPLANT.FDB

@firebird-automations
Copy link
Collaborator Author

Commented by: Karel Rys (vandrovnik)

netstat -a -n

TCP 192.168.1.2:3050 192.168.1.2:33824 NAVAZANO

(NAVAZANO = connected or something like that)

There are no other connection to TCP 192.168.1.2:3050

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

Does MON$ATTACHMENTS list all of them with value "TCPv4" in the MON$REMOTE_PROTOCOL field?

@firebird-automations
Copy link
Collaborator Author

Commented by: Karel Rys (vandrovnik)

Yes, it does (there are more connections to the server at this moment, than before, when I wrote, that there where 5 connections):

MON$ATTACHMENT_ID MON$SERVER_PID MON$STATE MON$ATTACHMENT_NAME MON$USER MON$ROLE MON$REMOTE_PROTOCOL MON$REMOTE_ADDRESS MON$REMOTE_PID MON$CHARACTER_SET_ID MON$TIMESTAMP MON$GARBAGE_COLLECTION MON$REMOTE_PROCESS MON$STAT_ID
2570 19864 1 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.3 828 51 12.5.2010 21:08:55 1 Q:\plantis\Bin\Plantis.exe 2
2568 19864 1 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.119 1012 51 12.5.2010 20:17:53 1 Q:\plantis\Bin\Plantis.exe 1153
2566 19864 1 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.2 21696 51 12.5.2010 18:50:10 1 C:\Program Files (x86)\HK-Software\IBExpert\ibexpert.exe 1184
2485 19864 1 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.121 2580 51 11.5.2010 9:32:52 1 Q:\plantis\Bin\Plantis.exe 1202
2483 19864 1 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.3 1860 51 11.5.2010 8:28:21 1 Q:\plantis\Bin\Plantis.exe 1228
2465 19864 0 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.3 2792 51 10.5.2010 16:36:09 1 Q:\plantis\Bin\Plantis.exe 1292
2446 19864 1 D:\FB\CZECHPLANT.FDB SYSDBA NONE TCPv4 192.168.1.120 3436 51 10.5.2010 14:39:48 1 Q:\plantis\Bin\Plantis.exe 1315

@firebird-automations
Copy link
Collaborator Author

Commented by: Karel Rys (vandrovnik)

About Firebirds ability to detect dead connections: I think Firebird sends a ping packet and waits for the answer. May be when there is already error during sending this ping packet, this situation is not handled correctly and server never destroys such a connection?
Most of the problems happen at 192.168.1.3, customer told me now, that this IP address belongs to a terminal server connection. Anyhow, when the TCP connection does not exist at operating system level, it should be destroyed by Firebird too...

@firebird-automations
Copy link
Collaborator Author

Commented by: @hvlad

> Firebird sends a ping packet and waits for the answer.

No, it not waits for the answer on dummy packet

> May be when there is already error during sending this ping packet, this situation is not handled correctly and server never destroys such a connection?

All socket errors, returned by the OS :
a) logged into firebird.log
b) force Firebird to close such socket immediately

> Most of the problems happen at 192.168.1.3, customer told me now, that this IP address belongs to a terminal server connection.

Do you have any network software on your server, such as firewall, antivirus, etc ?

> Anyhow, when the TCP connection does not exist at operating system level, it should be destroyed by Firebird too.

Of course. But if OS didn't notified Firebird about broken connection - how it could know about it ?

BTW, how client application owned of "hung" connections was closed - correctly or not ?
And what fbclient version they used ?

@firebird-automations
Copy link
Collaborator Author

Commented by: Karel Rys (vandrovnik)

Customer has ESET Mail Security 4 For Microsoft Exchange Server installed on the server, and they probably have windows firewall enabled.

Unfortunatelly, I do not know, how they close the application - I can imagine that sometimes their connection is just broken because of network problems...

FbClient.dll renamed to gds32.dll:
Version 2.1.3.18185 from 16.11.2009.11:56

I think Firebird should recognize a problem, when it tries to send ping packet over a non-existing connection (non-existing at the operating system level). Is it possible to enable debugging messages? I would like to see, what is going on with these connections - logging of each ping packet and a response would help a lot...

In firebird.log, there are messages like this:

SERVER (Server) Mon May 10 10:31:05 2010
INET/inet_error: read errno = 10054

SERVER (Server) Mon May 10 10:48:17 2010
INET/inet_error: read errno = 10054

@firebird-automations
Copy link
Collaborator Author

Modified by: Karel Rys (vandrovnik)

security: Developers [ 10012 ] =>

@firebird-automations
Copy link
Collaborator Author

Commented by: Evelyne Girard (evelyne)

I had the same kind of problem with Firebird 2.1 (17910) 32-bits SuperServer. The server considered some old connections as active even after the client application was closed. It kept those attachments actives forever (several weeks). It didn't matter if I restarted the client computer, the server still considered the attachment active. I have upgraded the server to 2.5 (changed nothing else except the fbclient.dll used by clients of course) and the problem is now fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant