Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Massive blob field usage end up in exception in fbclient.dll / fbclient.so with error "Invalid Blob ID" [CORE5608] #5874

Open
firebird-automations opened this issue Sep 8, 2017 · 4 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: Sascha Michel (datiscum1)

Attachments:
Exception_fbclient_dll.png

Application is multithreaded but use 1 connection for each thread !

Massiv writing BlobFields via StoredProcedure ( sql insert not tested ) will end up in a exception in the fbclient.
example code for reproducing the problem.

if ( AMsg->Size < 8190 )
IBCSP_PutMailTemp->ParamByName("MessageShort")->AsString = AMsg->ToString();
else
IBCSP_PutMailTemp->ParamByName("Message")->LoadFromStream( AMsg , ftBlob);

IBCSP_PutMailTemp->ExecProc();

When AMsg->Size lower 8190 bytes the SP use the VarChar Parameter.
So i can write without any problems 3200 rows in 152 seconds.
This work without any Problem more then 20 times with 3200 rows.

When i only use the Blob version without the if.
IBCSP_PutMailTemp->ParamByName("Message")->LoadFromStream( AMsg , ftBlob);

I got an exception after 80-150 rows in the fbclient.dll.
The error message is "Invalid BLOB ID" and the debugger always catches it at the same position.

Here the sample database:

SET SQL DIALECT 3;
CREATE DOMAIN "d_BLOB_Text_Gross" AS
BLOB SUB_TYPE 1 SEGMENT SIZE 4096 CHARACTER SET UTF8;

CREATE DOMAIN "d_Akt_DatumZeit" AS
TIMESTAMP
DEFAULT current_timestamp;

CREATE DOMAIN "d_strMax" AS
VARCHAR(8191) CHARACTER SET UTF8
DEFAULT ''
COLLATE UTF8;

CREATE GENERATOR "GEN_E-MailNewIn_ID";
CREATE TABLE "E-MailNewIn" (
"ENI_MaildID" BIGINT NOT NULL,
"ENI_Message" "d_BLOB_Text_Gross",
"ENI_Groesse" INTEGER,
"ENI_Eingegangen" "d_Akt_DatumZeit"
);
ALTER TABLE "E-MailNewIn" ADD CONSTRAINT "PK_E-MailNewIn" PRIMARY KEY ("ENI_MaildID");

SET TERM ^ ;
/* Trigger: "E-MailNewIn_BI" */
CREATE OR ALTER TRIGGER "E-MailNewIn_BI" FOR "E-MailNewIn"
ACTIVE BEFORE INSERT POSITION 0
as
begin
if (new."ENI_MaildID" is null) then
new."ENI_MaildID" = gen_id("GEN_E-MailNewIn_ID",1);
end
^

create or alter procedure "NewMailIn" (
"Message" "d_BLOB_Text_Gross",
"MessageGroesse" integer,
"MessageShort" "d_strMax" = null)
returns (
"RetCode" integer)
as
begin
if ( :"MessageShort" is not null ) then
insert into "E-MailNewIn" ( "ENI_Message", "ENI_Groesse" ) values ( :"MessageShort" , :"MessageGroesse" );
else
insert into "E-MailNewIn" ( "ENI_Message", "ENI_Groesse" ) values ( :"Message" , :"MessageGroesse" );
suspend;
end
^
SET TERM ; ^

@firebird-automations
Copy link
Collaborator Author

Modified by: Sascha Michel (datiscum1)

Attachment: Exception_fbclient_dll.png [ 13178 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: Sascha Michel (datiscum1)

Version: 3.0.2 [ 10785 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: Sascha Michel (datiscum1)

description: Massiv writing BlobFields via StoredProcedure ( sql insert not tested ) will end up in a exception in the fbclient.
example code for reproducing the problem.

if ( AMsg->Size < 8190 )
IBCSP_PutMailTemp->ParamByName("MessageShort")->AsString = AMsg->ToString();
else
IBCSP_PutMailTemp->ParamByName("Message")->LoadFromStream( AMsg , ftBlob);

IBCSP_PutMailTemp->ExecProc();

When AMsg->Size lower 8190 bytes the SP use the VarChar Parameter.
So i can write without any problems 3200 rows in 152 seconds.
This work without any Problem more then 20 times with 3200 rows.

When i only use the Blob version without the if.
IBCSP_PutMailTemp->ParamByName("Message")->LoadFromStream( AMsg , ftBlob);

I got an exception after 80-150 rows in the fbclient.dll.
The error message is "Invalid BLOB ID" and the debugger always catches it at the same position.

Here the sample database:

SET SQL DIALECT 3;
CREATE DOMAIN "d_BLOB_Text_Gross" AS
BLOB SUB_TYPE 1 SEGMENT SIZE 4096 CHARACTER SET UTF8;

CREATE DOMAIN "d_Akt_DatumZeit" AS
TIMESTAMP
DEFAULT current_timestamp;

CREATE DOMAIN "d_strMax" AS
VARCHAR(8191) CHARACTER SET UTF8
DEFAULT ''
COLLATE UTF8;

CREATE GENERATOR "GEN_E-MailNewIn_ID";
CREATE TABLE "E-MailNewIn" (
"ENI_MaildID" BIGINT NOT NULL,
"ENI_Message" "d_BLOB_Text_Gross",
"ENI_Groesse" INTEGER,
"ENI_Eingegangen" "d_Akt_DatumZeit"
);
ALTER TABLE "E-MailNewIn" ADD CONSTRAINT "PK_E-MailNewIn" PRIMARY KEY ("ENI_MaildID");

SET TERM ^ ;
/* Trigger: "E-MailNewIn_BI" */
CREATE OR ALTER TRIGGER "E-MailNewIn_BI" FOR "E-MailNewIn"
ACTIVE BEFORE INSERT POSITION 0
as
begin
if (new."ENI_MaildID" is null) then
new."ENI_MaildID" = gen_id("GEN_E-MailNewIn_ID",1);
end
^

create or alter procedure "NewMailIn" (
"Message" "d_BLOB_Text_Gross",
"MessageGroesse" integer,
"MessageShort" "d_strMax" = null)
returns (
"RetCode" integer)
as
begin
if ( :"MessageShort" is not null ) then
insert into "E-MailNewIn" ( "ENI_Message", "ENI_Groesse" ) values ( :"MessageShort" , :"MessageGroesse" );
else
insert into "E-MailNewIn" ( "ENI_Message", "ENI_Groesse" ) values ( :"Message" , :"MessageGroesse" );
suspend;
end
^
SET TERM ; ^

=>

Application is multithreaded but use 1 connection for each thread !

Massiv writing BlobFields via StoredProcedure ( sql insert not tested ) will end up in a exception in the fbclient.
example code for reproducing the problem.

if ( AMsg->Size < 8190 )
IBCSP_PutMailTemp->ParamByName("MessageShort")->AsString = AMsg->ToString();
else
IBCSP_PutMailTemp->ParamByName("Message")->LoadFromStream( AMsg , ftBlob);

IBCSP_PutMailTemp->ExecProc();

When AMsg->Size lower 8190 bytes the SP use the VarChar Parameter.
So i can write without any problems 3200 rows in 152 seconds.
This work without any Problem more then 20 times with 3200 rows.

When i only use the Blob version without the if.
IBCSP_PutMailTemp->ParamByName("Message")->LoadFromStream( AMsg , ftBlob);

I got an exception after 80-150 rows in the fbclient.dll.
The error message is "Invalid BLOB ID" and the debugger always catches it at the same position.

Here the sample database:

SET SQL DIALECT 3;
CREATE DOMAIN "d_BLOB_Text_Gross" AS
BLOB SUB_TYPE 1 SEGMENT SIZE 4096 CHARACTER SET UTF8;

CREATE DOMAIN "d_Akt_DatumZeit" AS
TIMESTAMP
DEFAULT current_timestamp;

CREATE DOMAIN "d_strMax" AS
VARCHAR(8191) CHARACTER SET UTF8
DEFAULT ''
COLLATE UTF8;

CREATE GENERATOR "GEN_E-MailNewIn_ID";
CREATE TABLE "E-MailNewIn" (
"ENI_MaildID" BIGINT NOT NULL,
"ENI_Message" "d_BLOB_Text_Gross",
"ENI_Groesse" INTEGER,
"ENI_Eingegangen" "d_Akt_DatumZeit"
);
ALTER TABLE "E-MailNewIn" ADD CONSTRAINT "PK_E-MailNewIn" PRIMARY KEY ("ENI_MaildID");

SET TERM ^ ;
/* Trigger: "E-MailNewIn_BI" */
CREATE OR ALTER TRIGGER "E-MailNewIn_BI" FOR "E-MailNewIn"
ACTIVE BEFORE INSERT POSITION 0
as
begin
if (new."ENI_MaildID" is null) then
new."ENI_MaildID" = gen_id("GEN_E-MailNewIn_ID",1);
end
^

create or alter procedure "NewMailIn" (
"Message" "d_BLOB_Text_Gross",
"MessageGroesse" integer,
"MessageShort" "d_strMax" = null)
returns (
"RetCode" integer)
as
begin
if ( :"MessageShort" is not null ) then
insert into "E-MailNewIn" ( "ENI_Message", "ENI_Groesse" ) values ( :"MessageShort" , :"MessageGroesse" );
else
insert into "E-MailNewIn" ( "ENI_Message", "ENI_Groesse" ) values ( :"Message" , :"MessageGroesse" );
suspend;
end
^
SET TERM ; ^

@firebird-automations
Copy link
Collaborator Author

Commented by: Sascha Michel (datiscum1)

Test with Insert SQL.
With Insert SQL directly into the table everything works.
TField Blobs are used here. The Firebird API calls should also differ.
It even works with just one connection.
Unfortunately, I have also found no reasonable documentation on Firebird Multithreading.

IBCQ_PutMailTemp.Insert();
MsgBlob := IBCQ_PutMailTemp.CreateBlobStream( IBCQ_PutMailTemp.FieldByName('ENI_Message') , bmWrite );
MsgBlob.CopyFrom( AMsg , 0 );
IBCQ_PutMailTemp.FieldByName('ENI_Groesse').AsInteger := AMsg.Size ;
IBCQ_PutMailTemp.Post();

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant