Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't read a NONE-Array with WIN1251 chars in UNICODE_FSS connection charset [CORE2500] #2912

Closed
firebird-automations opened this issue Jun 8, 2009 · 9 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: @ibprovider

Attachments:
test_txt_array__utf8.wsf

Hello

1. Metadata: Text array with NONE charset
2. Insert to this array a WIN1251 text
3. Try select this array in UNICODE_FSS connection charset

And get the error

Test case
1. Download IBProvider (TRIAL or FREE)
2. Build project from TestCode\ActiveX\IBP\oledb_test\builds\vc8\ibp_oledb_test_vc8.sln
3. Create in ANY database the next structures

CREATE GENERATOR GEN_ID_TBL_CS__NONE;

CREATE TABLE TBL_CS__NONE ( TEST_ID T_INTEGER NOT NULL, COL_BLOB BLOB SUB_TYPE TEXT CHARACTER SET NONE, CHAR__1 CHAR(1) CHARACTER SET NONE, VARCHAR__1 VARCHAR(1) CHARACTER SET NONE, CHAR__8 CHAR(8) CHARACTER SET NONE, VARCHAR__8 VARCHAR(8) CHARACTER SET NONE, CHAR__32 CHAR(32) CHARACTER SET NONE, VARCHAR__32 VARCHAR(32) CHARACTER SET NONE, CHAR_ARRAY__8 CHAR(8) [0:2] CHARACTER SET NONE, VARCHAR_ARRAY__8 VARCHAR(8) [0:2] CHARACTER SET NONE, CHAR_ARRAY__32 CHAR(32) [0:2] CHARACTER SET NONE, VARCHAR_ARRAY__32 VARCHAR(32) [0:2] CHARACTER SET NONE, CONSTRAINT PK_TBL_CHAR__NONE PRIMARY KEY (TEST_ID) );

SET TERM !! ;
CREATE TRIGGER BI_TBL_CS__NONE_TEST_ID FOR TBL_CS__NONE BEFORE INSERT AS BEGIN IF(NEW.TEST_ID IS NULL)THEN NEW.TEST_ID=GEN_ID(GEN_ID_TBL_CS__NONE,1); END!!
SET TERM ; !!

4. Execute test case "array.002.unicode.write_array_str.TBL_CS__NONE.VARCHAR_ARRAY__8.ctype__WIN1251.chars__WIN1251"

---------
[BUG FIX]

file: cvt.cpp
function: transliterate

if \(\(charset1 \!= charset2\) &&
	\(charset1 \!= ttype\_none\) && // <\-\-\-\-  append this condition
	\(charset2 \!= ttype\_none\) &&
	\(charset1 \!= ttype\_binary\) &&
	\(charset2 \!= ttype\_binary\) &&
	\(charset1 \!= ttype\_dynamic\) && \(charset2 \!= ttype\_dynamic\)\)
@firebird-automations
Copy link
Collaborator Author

Modified by: @ibprovider

description: Hello

1. Metadata: Text array with NONE charset
2. Insert to this array a WIN1251 text
3. Try select this array in UNICODE_FSS connection charset

And get the error

Test case
1. Download IBProvider (TRIAL or FREE)
2. Build project from TestCode\ActiveX\IBP\oledb_test\builds\vc8\ibp_oledb_test_vc8.sln
3. Create in ANY database a next structures

CREATE GENERATOR GEN_ID_TBL_CS__NONE;

CREATE TABLE TBL_CS__NONE ( TEST_ID T_INTEGER NOT NULL, COL_BLOB BLOB SUB_TYPE TEXT CHARACTER SET NONE, CHAR__1 CHAR(1) CHARACTER SET NONE, VARCHAR__1 VARCHAR(1) CHARACTER SET NONE, CHAR__8 CHAR(8) CHARACTER SET NONE, VARCHAR__8 VARCHAR(8) CHARACTER SET NONE, CHAR__32 CHAR(32) CHARACTER SET NONE, VARCHAR__32 VARCHAR(32) CHARACTER SET NONE, CHAR_ARRAY__8 CHAR(8) [0:2] CHARACTER SET NONE, VARCHAR_ARRAY__8 VARCHAR(8) [0:2] CHARACTER SET NONE, CHAR_ARRAY__32 CHAR(32) [0:2] CHARACTER SET NONE, VARCHAR_ARRAY__32 VARCHAR(32) [0:2] CHARACTER SET NONE, CONSTRAINT PK_TBL_CHAR__NONE PRIMARY KEY (TEST_ID) );

SET TERM !! ;
CREATE TRIGGER BI_TBL_CS__NONE_TEST_ID FOR TBL_CS__NONE BEFORE INSERT AS BEGIN IF(NEW.TEST_ID IS NULL)THEN NEW.TEST_ID=GEN_ID(GEN_ID_TBL_CS__NONE,1); END!!
SET TERM ; !!

4. Execute test case "array.002.unicode.write_array_str.TBL_CS__NONE.VARCHAR_ARRAY__8.ctype__WIN1251.chars__WIN1251"

---------
[BUG FIX]

file: cvt.cpp
function: transliterate

if \(\(charset1 \!= charset2\) &&
	\(charset1 \!= ttype\_none\) && // <\-\-\-\-  append this condition
	\(charset2 \!= ttype\_none\) &&
	\(charset1 \!= ttype\_binary\) &&
	\(charset2 \!= ttype\_binary\) &&
	\(charset1 \!= ttype\_dynamic\) && \(charset2 \!= ttype\_dynamic\)\)

=>

Hello

1. Metadata: Text array with NONE charset
2. Insert to this array a WIN1251 text
3. Try select this array in UNICODE_FSS connection charset

And get the error

Test case
1. Download IBProvider (TRIAL or FREE)
2. Build project from TestCode\ActiveX\IBP\oledb_test\builds\vc8\ibp_oledb_test_vc8.sln
3. Create in ANY database the next structures

CREATE GENERATOR GEN_ID_TBL_CS__NONE;

CREATE TABLE TBL_CS__NONE ( TEST_ID T_INTEGER NOT NULL, COL_BLOB BLOB SUB_TYPE TEXT CHARACTER SET NONE, CHAR__1 CHAR(1) CHARACTER SET NONE, VARCHAR__1 VARCHAR(1) CHARACTER SET NONE, CHAR__8 CHAR(8) CHARACTER SET NONE, VARCHAR__8 VARCHAR(8) CHARACTER SET NONE, CHAR__32 CHAR(32) CHARACTER SET NONE, VARCHAR__32 VARCHAR(32) CHARACTER SET NONE, CHAR_ARRAY__8 CHAR(8) [0:2] CHARACTER SET NONE, VARCHAR_ARRAY__8 VARCHAR(8) [0:2] CHARACTER SET NONE, CHAR_ARRAY__32 CHAR(32) [0:2] CHARACTER SET NONE, VARCHAR_ARRAY__32 VARCHAR(32) [0:2] CHARACTER SET NONE, CONSTRAINT PK_TBL_CHAR__NONE PRIMARY KEY (TEST_ID) );

SET TERM !! ;
CREATE TRIGGER BI_TBL_CS__NONE_TEST_ID FOR TBL_CS__NONE BEFORE INSERT AS BEGIN IF(NEW.TEST_ID IS NULL)THEN NEW.TEST_ID=GEN_ID(GEN_ID_TBL_CS__NONE,1); END!!
SET TERM ; !!

4. Execute test case "array.002.unicode.write_array_str.TBL_CS__NONE.VARCHAR_ARRAY__8.ctype__WIN1251.chars__WIN1251"

---------
[BUG FIX]

file: cvt.cpp
function: transliterate

if \(\(charset1 \!= charset2\) &&
	\(charset1 \!= ttype\_none\) && // <\-\-\-\-  append this condition
	\(charset2 \!= ttype\_none\) &&
	\(charset1 \!= ttype\_binary\) &&
	\(charset2 \!= ttype\_binary\) &&
	\(charset1 \!= ttype\_dynamic\) && \(charset2 \!= ttype\_dynamic\)\)

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

assignee: Adriano dos Santos Fernandes [ asfernandes ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

Hello, again

This is separate small test case on VBScript (UTF-8 File)

0. Download and install last IBProvider (TRIAL)
1. Create metadata in ANY FB-database
2. Change c_db_location constant
3. Run cmd.exe
4. Execute test_txt_array__utf8.wsf

Test:
- connect to database with NONE charset
- create array with win1251 text
- connect to database with WIN1251 charset
- read array. OK
- connect to database with UNICODE_FSS charset
- read array. FAIL

For use debugger for this script, execute "test_txt_array__utf8.wsf //X"

@firebird-automations
Copy link
Collaborator Author

Modified by: @ibprovider

Attachment: test_txt_array__utf8.wsf [ 11456 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @asfernandes

The proposed fix is a band-aid which would certainly have others consequences.

The real problem is that array slices works with blr_text, blr_varying and blr_cstring, and not with the blr_*2 versions. So it ignores the charset and use the connection charset.

In your case, the target, which should be using NONE charset is marked with UNICODE_FSS charset. It then fails verifying malformed characters in the UNICODE_FSS descriptor. This is not the mistake, it's intentional. It just happens that something fail early.

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

>The proposed fix is a band-aid which would certainly have others consequences.

Yes, I know this.

I think, this condition is wrongly from begin. Because, at equal situation, I use other rules (one for all datatypes: Text BLOB, Text Array, Text Column)

With something like next:

[return data to user]

Step1:
Transformation of SourceCharsetID and TargetCharsetID to StableCharsetID:

For Source (ColumnCharset):
NONE->NONE
OCTETS->OCTETS
DYNAMIC->CONNECTION_CHARSET
Other->Other

For Target (ConnectionCharset):
if(TARGET==NONE || SOURCE==NONE || SOURCE==OCTETS)
return SOURCE_CHARSET

OCTETS-> Generate Error // Can't use OCTETS for Connection Charset

Other->Other

Step2:
Decision (Need translation or not)

if(StableSourceCharsetID!=StableTargetCharsetID)
{
return true; //Need translate between codepage
}

return false; //Equal codepages

---
"user" do not know about internal specific of array implementation. He uses CHAR/VARCHAR as for simple columns

And wait similar behavior for text transformation

---
PS. Adriano, I ask you fix this problem, because ARRAYS - my favorite data types :)

@firebird-automations
Copy link
Collaborator Author

Commented by: @ibprovider

Hi

I think, you can close this ticked - I found the decision :)

need:
1. use the "isc_put_slice", "isc_get_slice"
2. pass text data through "isc_blr_text2" and "isc_blr_varying2" datatypes

This is works for all FB/IB versions

@firebird-automations
Copy link
Collaborator Author

Modified by: @asfernandes

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Won't Fix [ 2 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @pcisar

status: Resolved [ 5 ] => Closed [ 6 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants