Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Firebird 2.5 Classic Server uses 47% more memory than 2.1.5 when caching metadata [CORE3560] #3915

Open
firebird-automations opened this issue Jul 20, 2011 · 36 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: Jesus Angel Garcia Zarco (cointec)

Jira_subtask_outward CORE3633

Attachments:
Database.rar
UDFCointec.dll
TaskManager.png
ResourceMonitor.png
TaskManager.png
metadata.rar
MemoryUsageWindows.docx
Multiconexion.rar
gestlab_metadata.rar
Gestlab_medatata_wo_udf.rar

When the user connects to my application, executes one stored procedure to register the user login. This stored procedure update one table that force the load of most of the metadata in memory.

The difference is that while Firebird 2.1.5 uses 57100 Kb, Fb 2.5.1 uses 84100 Kb, that is a 47% more memory than 2.1. once executed the procedure and starting to work with the application, the memory consumption increases in parallel. That implies that for 200 users the less memory needed for 2.5 is near 16 Gb, while for 2.1 is near 11 Gb that is a big difference.

That implies that classic server will use per connection so much memory. I don't know if is a bug or not, but the difference is so much. If classic server is for complex enviroments with high load, then

Steps to reproduce the issue: Connect to the database and execute procedure usuario_entrada( 0, '', 'STATION01', '' )

Commits: 1639e12 d0184a5

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

The database file in fb 2.1

@firebird-automations
Copy link
Collaborator Author

Modified by: Jesus Angel Garcia Zarco (cointec)

Attachment: Database.rar [ 11975 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

The archive is password protected.

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

I have sent you the password to tour email.

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

It will be also good idea to provide source code for your UDF (UDFCointec.dll)

@firebird-automations
Copy link
Collaborator Author

Modified by: Jesus Angel Garcia Zarco (cointec)

Attachment: UDFCointec.dll [ 11977 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

Source code very simple.

library UDFCointec;

uses
SysUtils,
Classes,
UDFMath in 'UDFMath.pas';

exports
UDF_power,
UDF_sqrt;

begin
IsMultiThread := True;
end.

unit UDFMath;

interface

function UDF_power( var Base, Exponent: Double ): double; cdecl; export;
function UDF_sqrt( var Value: double ): double; cdecl; export;

implementation

uses Math;

function UDF_power(var Base, Exponent: Double): double; cdecl;
begin
try
Result := power( Base, Exponent );
except
Result := 0;
end;
end;

function UDF_sqrt( var Value: double ): double; cdecl;
begin
try
Result := sqrt( Value );
except
Result := 0;
end;
end;

initialization
IsMultiThread := True;

end.

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

assignee: Alexander Peshkov [ alexpeshkoff ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

I did not avoid additional memory usage in par.cpp & cmp.cpp, but due to better reusability of memory regions overall memory consumption decreased much.

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Fix Version: 2.5.1 [ 10333 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

I have been tested the new version, and the memory have been decreased from 84100 kb, to 73808 Kb, what means that fb 2.5.1 (currently testing also 2.5.2), uses 29% more than 2.1.4.

Is this a normal usage?

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

Jesus, I've checked it with linux, which has 2 different kinds of consumed memory - virtual address space and allocated pages. First one really increased much due to more threads used by a task (each thread's stack do use a lot of virtual address space), but that's not too big trouble. What about really allocated pages on linux consumption of this kind of memory happened to be on your test even a bit (1-2%) smaller on 2.5.1 rather than on 2.1.4. Therefore first question is wheather windows has some similiar RAM accounting, and if yes - what kind of memory do you talk about? Telling true I do not believe in too much different memory usahge patterns depending upon OS.

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

Hello Alex, in my test what i have done is connect To the database from ibexpert and execute one procedure that my client application runs after each connection. After that i go To Windows task manager and see the memory that each fb_inet_server consume and with that information i compute the difference.

I do not know If this procedure is valid, but this operations Are for do the calculation of the memory i need in one server for 200 concurrent users.

Thanks

@firebird-automations
Copy link
Collaborator Author

Commented by: Sean Leyne (seanleyne)

Windows Task Manager has several memory counters, which ones are you looking at?

Are you comparing values before, during or after transaction commit?

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

Please try isql instead ibexpert.

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

Hello Alex, i have used isql, ibexpert, my application, all with the same results.

Sean, to view the memory used to the process, i open task manager, see fb_inet_server process and in the processes tab see the column Memory( Private working set)->70528. I also have used processexplorer and in the process tree i see Private bytes 75320K and in Working Set 73856K. In the resource monitor i see in thrre different columns the three above values.

As i have writed, In firebird 2.1 the same database uses around 50000 K.

I attach the images

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

ScreenShots

@firebird-automations
Copy link
Collaborator Author

Modified by: Jesus Angel Garcia Zarco (cointec)

Attachment: TaskManager.png [ 12061 ]

Attachment: ResourceMonitor.png [ 12062 ]

Attachment: TaskManager.png [ 12063 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

After rechecking I have to agree that I also see that 2.5.1 is using a bit (11%) more RAM than 2.1. This appears to be more or less normal - it has a number of addiitonal features compared with 2.1, that may consume memory. I just do not understand why such a difference (11% against 29%).

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

My calcs is that 2.5.1 requires more than 37% more RAM, because i use 54000K as the base.

29% is if you calc that 2.1.4 requires less RAM than 2.5.1, but from my POV the calc must be done from 2.1.4.

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

Metadata for database creation

@firebird-automations
Copy link
Collaborator Author

Modified by: Jesus Angel Garcia Zarco (cointec)

Attachment: metadata.rar [ 12113 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

Document with steps

@firebird-automations
Copy link
Collaborator Author

Modified by: Jesus Angel Garcia Zarco (cointec)

Attachment: MemoryUsageWindows.docx [ 12114 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

Application for open several connections and open the tables that makes Firebird uses a lot of memory for load metadata.

@firebird-automations
Copy link
Collaborator Author

Modified by: Jesus Angel Garcia Zarco (cointec)

Attachment: Multiconexion.rar [ 12115 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Sean Leyne (seanleyne)

Re-opened based on comments and additional details received.

@firebird-automations
Copy link
Collaborator Author

Modified by: Sean Leyne (seanleyne)

status: Resolved [ 5 ] => Reopened [ 4 ]

resolution: Fixed [ 1 ] =>

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

To reproduce the problem, just write
update Pet_prueba set IDPETICION = IDPETICION where IDPETICION = 1
and prepare the statement.

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

backup with metadata only

@firebird-automations
Copy link
Collaborator Author

Modified by: Jesus Angel Garcia Zarco (cointec)

Attachment: gestlab_metadata.rar [ 12116 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

This is the backup metadata without the need of my UDFCointec.dll. It only uses ib_udf. This way can be restored in any OS, i think.

@firebird-automations
Copy link
Collaborator Author

Modified by: Jesus Angel Garcia Zarco (cointec)

Attachment: Gestlab_medatata_wo_udf.rar [ 12117 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Jesus Angel Garcia Zarco (cointec)

Is there any option to solve tris issue? Is a confirmed bug?

Another issue i'm having is that virtual size of classic processes in Windows 2008 64 is around of 18 gb for 100 connections while is using 10.4 gb in superclassic.

Do you need more info from me?

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

> Is there any option to solve tris issue?
> Is a confirmed bug?
When I checked this last time, on linux I did not get serious difference in RAM usage between 2.1 and 2.5 (about 7%, as far as I remember). IMHO not critical, but certainly something might change.

>Another issue i'm having is that virtual size of classic processes in Windows 2008 64 is around of 18 gb for 100 connections while is using 10.4 gb in superclassic.
This is normal. The main reason to have superclassic is saving resources (compared with classic), including memory.

>Do you need more info from me?
No, but I need to finish with last plugin type, planned for FB3.

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

Fix Version: 2.5.1 [ 10333 ] =>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants