Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BLOB text column is not transliteration with UTF8 connection. [ODBC44] #42

Closed
firebird-automations opened this issue Aug 11, 2008 · 18 comments

Comments

@firebird-automations
Copy link

Submitted by: Minoru Yoshida (timeful2)

Votes: 1

I'm testing for OdbcFb_v2_0_145_Win32.

The VARCHAR/CHAR columns are working fine with UTF8 connection.
But BLOB text column is not transliteration.
I tried change to the CP943C connection. This BLOB's transliteration is fine.

I think that UTF8 connectioin result is odbc driver side bug since build 144.

My environments:

- WindowsXP sp2
- Firebird V2.1.1
- Excel 2003 japanese edition.

/* Create database */
CREATE DATABASE 'Localhost:TestDB'
USER 'sysdba'
PASSWORD 'password'
PAGE_SIZE 16384
DEFAULT CHARACTER SET UTF8;
SET SQL DIALECT 3;

/* Create table */
CREATE TABLE Table1 (
K INTEGER NOT NULL, B BLOB SUB_TYPE TEXT SEGMENT SIZE 80
, CONSTRAINT PK PRIMARY KEY (K));

-- My test code
Sub Fbodbctest()
Dim con As ADODB.Connection
Dim recordSet As ADODB.recordSet
Dim outPutSheet As Worksheet
Dim row As Long
Dim col As Long

Set con = New ADODB\.Connection
Set recordSet = New ADODB\.recordSet
Set outPutSheet = ThisWorkbook\.Sheets\("Sheet1"\) 'Output sheet
outPutSheet\.Cells\.Clear

'
con.Open "DRIVER=Firebird/InterBase(r) driver; SERVER=localhost; DATABASE=TestDB;" & _
"Extended Properties=""UID=sysdba;PWD=password;CHARSET=UTF8;"""
' "Extended Properties=""UID=sysdba;PWD=password;CHARSET=CP943C;"""
recordSet.Open "select * from Table1", con, adOpenForwardOnly, adLockReadOnly, -1

'Field names to row
For col = 0 To recordSet\.Fields\.Count \- 1
    outPutSheet\.Cells\(1, col \+ 1\)\.Value = recordSet\.Fields\(col\)\.Name
Next col

'Field values to row
row = 2
While (Not recordSet.EOF) And row < 60000
For col = 0 To recordSet.Fields.Count - 1
outPutSheet.Cells(row, col + 1).Value = recordSet.Fields(col).Value
Next col
recordSet.MoveNext
row = row + 1
Wend

recordSet\.Close
con\.Close

End Sub

--
Regards,
Minoru

@firebird-automations
Copy link
Author

Commented by: @alexpotapchenko

I think it is not a bug of ODBC driver.
If UTF8 connection than the client receives BLOB content in UTF8 encoding and the client must decode it. This is true with VARCHAR/CHAR columns too. If CP943C connection than the client receives content in CP943C encoding (server converts fields content to CP943C).
I don't understand why VARCHAR/CHAR columns are working fine (are displaying fine) with UTF8 connection in your case. May be these columns contains only english characters?

@firebird-automations
Copy link
Author

Commented by: Minoru Yoshida (timeful2)

Thanks for your comment.

>May be these columns contains only english characters?

No. Testing charsets are japanese value in VARCHAR/CHAR of UTF8 .
I think that transliteration is dependent on ADO. AFAIK, ADO connection is translated to UCS-2(UTF-16?) from UTF-8 .

I try another case,

- FB V2.0 default charset : SJIS_0208
- Client connection : UTF8

The BLOB column's transliteration is fine.
I think that ODBC driver side problem....

@firebird-automations
Copy link
Author

Commented by: Tao Lin (taol)

I am using Delphi 2009 with ADOConnection.

My configure: Delphi 2009, Win XP sp2, Fb 2.1, build 144. My database's default character is UTF8.

When it load the data from the database by FbODBC, varchar can return Chinese correct, but Blob field is not. I find out is that if I make the TADOQuery memo field from TMemoField to TWideMemoField, then the Chinese character dispaly correctly.

Because I don't have build 143, I don't whether the bug is introudce from 144, maybe someone can send me a odbcfb32.dll build 143 for testing.

@firebird-automations
Copy link
Author

Commented by: Minoru Yoshida (timeful2)

I retry it with RC1(Build 148 ) .

My environment:

- FB V2.0.4
- Client connection:UTF8

Case 1. Defaut charset :UTF8

- VARCHAR is fine
- BLOB text is NOT fine.

Case 2. Default charset:SJIS_0208

- VARCHAR is fine
- BLOB text is fine.
(Of cause, characters range is SJIS_0208 only)

I think that result must be the same at client connection is UTF8.

Why is a result different?

@firebird-automations
Copy link
Author

Commented by: @alexpotapchenko

I have understood with this problem.
This is a bug of Firebird 2.0. If ODS < 11.1 then BLOB charset <> attachment charset.
You can try Firebird 2.1.1 (backup/restore of the database is required) and you will see an identical behaviour in both cases.
ADO does not translate BLOB fields always. You must use an attachment charset = a data charset.

@firebird-automations
Copy link
Author

Modified by: @alexpotapchenko

assignee: Vladimir Tsvigun [ praktik ] => Alexander Potapchenko [ lightfore ]

@firebird-automations
Copy link
Author

Commented by: @alexpotapchenko

Do you confirm it? ODBC driver is only an intermediate layer between a server and a client application (ADO in this case). The driver transfers data to a client application in the form which a server returns their.

@firebird-automations
Copy link
Author

Modified by: @alexpotapchenko

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Won't Fix [ 2 ]

Fix Version: 2.0 RC1 [ 10310 ]

@firebird-automations
Copy link
Author

Commented by: Minoru Yoshida (timeful2)

I am usually using IBObject on FB2.0.4. The UTF8 BLOB column handling is no ploblem(This mean is *no* transliteration at any charset connection).
Then, i try IBProvidor with UTF8 ADODB.connection . That works fine now.

IBProvider Free Edition 32 bit
http://www.ibprovider.com/eng/download.html

>This is a bug of Firebird 2.0.

invalid BLOB ID on blob with different charset
Affects Version/s:2.1.1
CORE2086

Is your saying this?

My case is same(Defaut =Connection) charset.

>- FB V2.0.4
>- Client connection:UTF8

>Case 1. Defaut charset :UTF8

>- VARCHAR is fine
>- BLOB text is NOT fine.

@firebird-automations
Copy link
Author

Commented by: @alexpotapchenko

invalid BLOB ID on blob with different charset
Affects Version/s:2.1.1
CORE2086
- No
This - CORE1277
In Firebird 2.0 the server does not translate BLOB fields into a client charset i.e. BLOB charset on the client side = Default database charset.
It explains your Case 1 and Case 2.

@firebird-automations
Copy link
Author

Commented by: Minoru Yoshida (timeful2)

>It explains your Case 1 and Case 2.

Thanks for your explanation. i have understood.

>You can try Firebird 2.1.1 (backup/restore of the database is required) and you will see an identical behaviour in both cases.

I retry it on V2.1.1( and V2.5 Alpha).

My environment:

- FB V2.1.1
- Client connection:UTF8

Case 1. Defaut charset :UTF8

- VARCHAR is fine
- BLOB text is NOT fine. Result is the same built 145. it seems converted from ansi(CP932),

Case 2. Default charset:SJIS_0208

- Result recordset is empty .( and V2.5 result is the same, too)

My another application works fine in both cases(Using IBO).

I have not clear understanding...

@firebird-automations
Copy link
Author

Commented by: @alexpotapchenko

Result recordset is empty? What is It meaning?

@firebird-automations
Copy link
Author

Commented by: Minoru Yoshida (timeful2)

Sorry, i had mistake. This case is another case.

>My environment:

>- FB V2.1.1

Case 1. Defaut charset :UTF8 Client connection:UTF8

>- VARCHAR is fine
>- BLOB text is NOT fine. Result is the same built 145. it seems converted from ansi(CP932),

Case 2. Default charset:SJIS_0208 Client connection:UTF8

>- VARCHAR is fine
>- BLOB text is fine.
>(Of cause, characters range is SJIS_0208 only)

>Result recordset is empty? What is It meaning?

This mean is recordSet.EOF as bellow case 3.

My test code:
> While (Not recordSet.EOF) And row < 60000

Case 3. Defaut charset :UTF8 Client connection:SJIS_0208

- Result recordset is empty .( and V2.5 result is the same, too)

As far as i know , This case issue an arithmetic error from server. because my data have the over SJIS_0208 range charset.

Is this specification of ODBC driver?

@firebird-automations
Copy link
Author

Commented by: @alexpotapchenko

I have absolutely got confused in your non-reproducible cases at me.
1) You must use Firebird 2.1
2) Database ODS must be > = 11.1 (if database was created in Firebird 1.5 or 2.0 then backup/restore is required)
Please use for testing FrameRobin or IBExpert too.
If the result in these applications is differed from result in ODBC then inform me how I can reproduce it.
And also I think your troubles are not related with ODBC driver.

@firebird-automations
Copy link
Author

Commented by: Minoru Yoshida (timeful2)

>1) You must use Firebird 2.1

>- FB V2.1.1

>2) Database ODS must be > = 11.1 (if database was created in Firebird 1.5 or 2.0 then backup/restore is required)

Yes.

>Case 3. Defaut charset :UTF8 Client connection:SJIS_0208
>- Result recordset is empty .( and V2.5 result is the same, too)
Excel version:11.0 build 5612.

I try it with IBProvidor. I get -214217837(80049e53) automation error.
But ,This case will be not use case for me. It was reported only.

My ploblem is this case only.
>Case 1. Defaut charset :UTF8 Client connection:UTF8
>- VARCHAR is fine
>- BLOB text is NOT fine. Result is the same built 145. it seems converted from ansi(CP932),

Thanks for your kindly anser.

@firebird-automations
Copy link
Author

Commented by: @alexpotapchenko

I have told that ADO does not translate BLOB fields anyway.
Case 2. Default charset:SJIS_0208 Client connection:UTF8
- VARCHAR is fine
- BLOB text is fine.
Can you reproduce it with fb2.1, odc 11.1 and ODBC driver?
If yes, then please give me example of your database.

@firebird-automations
Copy link
Author

Commented by: Minoru Yoshida (timeful2)

>Can you reproduce it with fb2.1, odc 11.1 and ODBC driver?

I retry it now on FB2.1.

>Case 2. Default charset:SJIS_0208 Client connection:UTF8
- BLOB text is NOT fine. (Same the default charset is UTF8 )

Your saying is correct, i was mistaken about the IBProvidor's result.

@firebird-automations
Copy link
Author

Modified by: @alexpotapchenko

status: Resolved [ 5 ] => Closed [ 6 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants