Issue Details (XML | Word | Printable)

Key: CORE-3560
Type: Bug Bug
Status: Reopened Reopened
Priority: Major Major
Assignee: Alexander Peshkov
Reporter: Jesus Angel Garcia Zarco
Votes: 0
Watchers: 6
Operations

If you were logged in you would be able to see more operations.
Firebird Core

Firebird 2.5 Classic Server uses 47% more memory than 2.1.5 when caching metadata

Created: 20/Jul/11 11:12 AM   Updated: 17/May/12 10:44 AM
Component/s: Engine
Affects Version/s: 2.5.0
Fix Version/s: 2.5.1

Time Tracking:
Issue & Sub-Tasks
Issue Only
Not Specified

File Attachments: 1. File Database.rar (8.92 MB)
2. File Gestlab_medatata_wo_udf.rar (1.99 MB)
3. File gestlab_metadata.rar (2.10 MB)
4. Microsoft Word MemoryUsageWindows.docx (790 kB)
5. File metadata.rar (821 kB)
6. File Multiconexion.rar (594 kB)
7. File UDFCointec.dll (320 kB)

Image Attachments:

1. ResourceMonitor.png
(107 kB)

2. TaskManager.png
(111 kB)

3. TaskManager.png
(111 kB)
Environment: Windows family

Planning Status: Unspecified

Sub-Tasks  All   Open   

 Description  « Hide
When the user connects to my application, executes one stored procedure to register the user login. This stored procedure update one table that force the load of most of the metadata in memory.

The difference is that while Firebird 2.1.5 uses 57100 Kb, Fb 2.5.1 uses 84100 Kb, that is a 47% more memory than 2.1. once executed the procedure and starting to work with the application, the memory consumption increases in parallel. That implies that for 200 users the less memory needed for 2.5 is near 16 Gb, while for 2.1 is near 11 Gb that is a big difference.

That implies that classic server will use per connection so much memory. I don't know if is a bug or not, but the difference is so much. If classic server is for complex enviroments with high load, then

Steps to reproduce the issue: Connect to the database and execute procedure usuario_entrada( 0, '', 'STATION01', '' )

 All   Comments   Work Log   Change History   Version Control   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Jesus Angel Garcia Zarco added a comment - 20/Jul/11 11:20 AM
The database file in fb 2.1

Dmitry Yemanov added a comment - 20/Jul/11 11:23 AM
The archive is password protected.

Jesus Angel Garcia Zarco added a comment - 20/Jul/11 11:58 AM
I have sent you the password to tour email.

Alexander Peshkov added a comment - 21/Jul/11 12:01 PM
It will be also good idea to provide source code for your UDF (UDFCointec.dll)

Jesus Angel Garcia Zarco added a comment - 21/Jul/11 12:14 PM - edited
Source code very simple.

library UDFCointec;

uses
  SysUtils,
  Classes,
  UDFMath in 'UDFMath.pas';

exports
   UDF_power,
   UDF_sqrt;

begin
   IsMultiThread := True;
end.

unit UDFMath;

interface

function UDF_power( var Base, Exponent: Double ): double; cdecl; export;
function UDF_sqrt( var Value: double ): double; cdecl; export;

implementation

uses Math;

function UDF_power(var Base, Exponent: Double): double; cdecl;
begin
   try
      Result := power( Base, Exponent );
   except
      Result := 0;
   end;
end;

function UDF_sqrt( var Value: double ): double; cdecl;
begin
   try
      Result := sqrt( Value );
   except
      Result := 0;
   end;
end;

initialization
   IsMultiThread := True;

end.

Alexander Peshkov added a comment - 04/Aug/11 01:17 PM
I did not avoid additional memory usage in par.cpp & cmp.cpp, but due to better reusability of memory regions overall memory consumption decreased much.

Jesus Angel Garcia Zarco added a comment - 27/Oct/11 01:18 PM
I have been tested the new version, and the memory have been decreased from 84100 kb, to 73808 Kb, what means that fb 2.5.1 (currently testing also 2.5.2), uses 29% more than 2.1.4.

Is this a normal usage?

Alexander Peshkov added a comment - 28/Nov/11 08:51 AM
Jesus, I've checked it with linux, which has 2 different kinds of consumed memory - virtual address space and allocated pages. First one really increased much due to more threads used by a task (each thread's stack do use a lot of virtual address space), but that's not too big trouble. What about really allocated pages on linux consumption of this kind of memory happened to be on your test even a bit (1-2%) smaller on 2.5.1 rather than on 2.1.4. Therefore first question is wheather windows has some similiar RAM accounting, and if yes - what kind of memory do you talk about? Telling true I do not believe in too much different memory usahge patterns depending upon OS.

Jesus Angel Garcia Zarco added a comment - 28/Nov/11 09:52 PM
Hello Alex, in my test what i have done is connect To the database from ibexpert and execute one procedure that my client application runs after each connection. After that i go To Windows task manager and see the memory that each fb_inet_server consume and with that information i compute the difference.

I do not know If this procedure is valid, but this operations Are for do the calculation of the memory i need in one server for 200 concurrent users.

Thanks

Sean Leyne added a comment - 28/Nov/11 10:41 PM
Windows Task Manager has several memory counters, which ones are you looking at?

Are you comparing values before, during or after transaction commit?

Alexander Peshkov added a comment - 29/Nov/11 07:14 AM
Please try isql instead ibexpert.

Jesus Angel Garcia Zarco added a comment - 29/Nov/11 08:43 AM
Hello Alex, i have used isql, ibexpert, my application, all with the same results.

Sean, to view the memory used to the process, i open task manager, see fb_inet_server process and in the processes tab see the column Memory( Private working set)->70528. I also have used processexplorer and in the process tree i see Private bytes 75320K and in Working Set 73856K. In the resource monitor i see in thrre different columns the three above values.

As i have writed, In firebird 2.1 the same database uses around 50000 K.

I attach the images


Jesus Angel Garcia Zarco added a comment - 29/Nov/11 08:44 AM
ScreenShots

Alexander Peshkov added a comment - 29/Nov/11 11:23 AM
After rechecking I have to agree that I also see that 2.5.1 is using a bit (11%) more RAM than 2.1. This appears to be more or less normal - it has a number of addiitonal features compared with 2.1, that may consume memory. I just do not understand why such a difference (11% against 29%).

Jesus Angel Garcia Zarco added a comment - 29/Nov/11 11:48 AM
My calcs is that 2.5.1 requires more than 37% more RAM, because i use 54000K as the base.

29% is if you calc that 2.1.4 requires less RAM than 2.5.1, but from my POV the calc must be done from 2.1.4.

Jesus Angel Garcia Zarco added a comment - 12/Mar/12 08:39 AM
Metadata for database creation

Jesus Angel Garcia Zarco added a comment - 12/Mar/12 08:40 AM
Document with steps

Jesus Angel Garcia Zarco added a comment - 12/Mar/12 08:43 AM
Application for open several connections and open the tables that makes Firebird uses a lot of memory for load metadata.

Sean Leyne added a comment - 12/Mar/12 03:14 PM
Re-opened based on comments and additional details received.

Alexander Peshkov added a comment - 13/Mar/12 08:11 AM
To reproduce the problem, just write
update Pet_prueba set IDPETICION = IDPETICION where IDPETICION = 1
and prepare the statement.

Jesus Angel Garcia Zarco added a comment - 13/Mar/12 06:35 PM
backup with metadata only

Jesus Angel Garcia Zarco added a comment - 14/Mar/12 05:55 AM
This is the backup metadata without the need of my UDFCointec.dll. It only uses ib_udf. This way can be restored in any OS, i think.

Jesus Angel Garcia Zarco added a comment - 15/May/12 01:52 PM
Is there any option to solve tris issue? Is a confirmed bug?

Another issue i'm having is that virtual size of classic processes in Windows 2008 64 is around of 18 gb for 100 connections while is using 10.4 gb in superclassic.

Do you need more info from me?

Alexander Peshkov added a comment - 17/May/12 10:44 AM
> Is there any option to solve tris issue?
> Is a confirmed bug?
When I checked this last time, on linux I did not get serious difference in RAM usage between 2.1 and 2.5 (about 7%, as far as I remember). IMHO not critical, but certainly something might change.

>Another issue i'm having is that virtual size of classic processes in Windows 2008 64 is around of 18 gb for 100 connections while is using 10.4 gb in superclassic.
This is normal. The main reason to have superclassic is saving resources (compared with classic), including memory.

>Do you need more info from me?
No, but I need to finish with last plugin type, planned for FB3.