Issue Details (XML | Word | Printable)

Key: ODBC-44
Type: Bug Bug
Status: Closed Closed
Resolution: Won't Fix
Priority: Major Major
Assignee: Alexander Potapchenko
Reporter: Minoru Yoshida
Votes: 1
Watchers: 1
Operations

If you were logged in you would be able to see more operations.
ODBC Driver

BLOB text column is not transliteration with UTF8 connection.

Created: 10/Aug/08 10:29 PM   Updated: 10/Dec/08 01:16 PM
Component/s: None
Affects Version/s: 2.0 Beta
Fix Version/s: 2.0 RC1

Time Tracking:
Not Specified

Environment: FB V2.1.1


 Description  « Hide
I'm testing for OdbcFb_v2_0_145_Win32.

The VARCHAR/CHAR columns are working fine with UTF8 connection.
But BLOB text column is not transliteration.
I tried change to the CP943C connection. This BLOB's transliteration is fine.

I think that UTF8 connectioin result is odbc driver side bug since build 144.

My environments:

- WindowsXP sp2
- Firebird V2.1.1
- Excel 2003 japanese edition.

/* Create database */
CREATE DATABASE 'Localhost:TestDB'
USER 'sysdba'
PASSWORD 'password'
PAGE_SIZE 16384
DEFAULT CHARACTER SET UTF8;
SET SQL DIALECT 3;

/* Create table */
CREATE TABLE Table1 (
K INTEGER NOT NULL, B BLOB SUB_TYPE TEXT SEGMENT SIZE 80
, CONSTRAINT PK PRIMARY KEY (K));

-- My test code
Sub Fbodbctest()
    Dim con As ADODB.Connection
    Dim recordSet As ADODB.recordSet
    Dim outPutSheet As Worksheet
    Dim row As Long
    Dim col As Long
    
    Set con = New ADODB.Connection
    Set recordSet = New ADODB.recordSet
    Set outPutSheet = ThisWorkbook.Sheets("Sheet1") 'Output sheet
    outPutSheet.Cells.Clear
'
    con.Open "DRIVER=Firebird/InterBase(r) driver; SERVER=localhost; DATABASE=TestDB;" & _
        "Extended Properties=""UID=sysdba;PWD=password;CHARSET=UTF8;"""
' "Extended Properties=""UID=sysdba;PWD=password;CHARSET=CP943C;"""
  recordSet.Open "select * from Table1", con, adOpenForwardOnly, adLockReadOnly, -1
   
    'Field names to row
    For col = 0 To recordSet.Fields.Count - 1
        outPutSheet.Cells(1, col + 1).Value = recordSet.Fields(col).Name
    Next col

   'Field values to row
   row = 2
    While (Not recordSet.EOF) And row < 60000
        For col = 0 To recordSet.Fields.Count - 1
            outPutSheet.Cells(row, col + 1).Value = recordSet.Fields(col).Value
        Next col
        recordSet.MoveNext
        row = row + 1
    Wend
    
    recordSet.Close
    con.Close
End Sub

--
Regards,
Minoru


 All   Comments   Work Log   Change History   Version Control   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Alexander Potapchenko added a comment - 26/Aug/08 07:24 AM
I think it is not a bug of ODBC driver.
If UTF8 connection than the client receives BLOB content in UTF8 encoding and the client must decode it. This is true with VARCHAR/CHAR columns too. If CP943C connection than the client receives content in CP943C encoding (server converts fields content to CP943C).
I don't understand why VARCHAR/CHAR columns are working fine (are displaying fine) with UTF8 connection in your case. May be these columns contains only english characters?

Minoru Yoshida added a comment - 01/Sep/08 11:34 PM
Thanks for your comment.

 >May be these columns contains only english characters?

No. Testing charsets are japanese value in VARCHAR/CHAR of UTF8 .
I think that transliteration is dependent on ADO. AFAIK, ADO connection is translated to UCS-2(UTF-16?) from UTF-8 .

I try another case,

- FB V2.0 default charset : SJIS_0208
- Client connection : UTF8

The BLOB column's transliteration is fine.
I think that ODBC driver side problem....


Tao Lin added a comment - 18/Nov/08 09:36 PM
I am using Delphi 2009 with ADOConnection.

My configure: Delphi 2009, Win XP sp2, Fb 2.1, build 144. My database's default character is UTF8.

When it load the data from the database by FbODBC, varchar can return Chinese correct, but Blob field is not. I find out is that if I make the TADOQuery memo field from TMemoField to TWideMemoField, then the Chinese character dispaly correctly.

Because I don't have build 143, I don't whether the bug is introudce from 144, maybe someone can send me a odbcfb32.dll build 143 for testing.

Minoru Yoshida added a comment - 20/Nov/08 08:53 PM - edited
I retry it with RC1(Build 148 ) .

My environment:

- FB V2.0.4
- Client connection:UTF8

Case 1. Defaut charset :UTF8

- VARCHAR is fine
- BLOB text is NOT fine.

Case 2. Default charset:SJIS_0208

- VARCHAR is fine
- BLOB text is fine.
(Of cause, characters range is SJIS_0208 only)

I think that result must be the same at client connection is UTF8.

Why is a result different?

Alexander Potapchenko added a comment - 21/Nov/08 08:23 AM
I have understood with this problem.
This is a bug of Firebird 2.0. If ODS < 11.1 then BLOB charset <> attachment charset.
You can try Firebird 2.1.1 (backup/restore of the database is required) and you will see an identical behaviour in both cases.
ADO does not translate BLOB fields always. You must use an attachment charset = a data charset.

Alexander Potapchenko added a comment - 26/Nov/08 05:53 AM
Do you confirm it? ODBC driver is only an intermediate layer between a server and a client application (ADO in this case). The driver transfers data to a client application in the form which a server returns their.

Minoru Yoshida added a comment - 28/Nov/08 09:44 PM - edited
 I am usually using IBObject on FB2.0.4. The UTF8 BLOB column handling is no ploblem(This mean is *no* transliteration at any charset connection).
Then, i try IBProvidor with UTF8 ADODB.connection . That works fine now.

IBProvider Free Edition 32 bit
http://www.ibprovider.com/eng/download.html

>This is a bug of Firebird 2.0.

invalid BLOB ID on blob with different charset
Affects Version/s:2.1.1
http://tracker.firebirdsql.org/browse/CORE-2086

Is your saying this?

My case is same(Defaut =Connection) charset.

>- FB V2.0.4
>- Client connection:UTF8

>Case 1. Defaut charset :UTF8

>- VARCHAR is fine
>- BLOB text is NOT fine.

Alexander Potapchenko added a comment - 29/Nov/08 05:24 AM
invalid BLOB ID on blob with different charset
Affects Version/s:2.1.1
http://tracker.firebirdsql.org/browse/CORE-2086
- No
This - http://tracker.firebirdsql.org/browse/CORE-1277
In Firebird 2.0 the server does not translate BLOB fields into a client charset i.e. BLOB charset on the client side = Default database charset.
It explains your Case 1 and Case 2.

Minoru Yoshida added a comment - 02/Dec/08 09:08 PM - edited
>It explains your Case 1 and Case 2.

Thanks for your explanation. i have understood.

>You can try Firebird 2.1.1 (backup/restore of the database is required) and you will see an identical behaviour in both cases.

I retry it on V2.1.1( and V2.5 Alpha).

My environment:

- FB V2.1.1
- Client connection:UTF8

Case 1. Defaut charset :UTF8

- VARCHAR is fine
- BLOB text is NOT fine. Result is the same built 145. it seems converted from ansi(CP932),

Case 2. Default charset:SJIS_0208

- Result recordset is empty .( and V2.5 result is the same, too)

My another application works fine in both cases(Using IBO).

I have not clear understanding...



Alexander Potapchenko added a comment - 04/Dec/08 04:43 AM
Result recordset is empty? What is It meaning?


Minoru Yoshida added a comment - 04/Dec/08 08:12 PM
Sorry, i had mistake. This case is another case.

>My environment:

>- FB V2.1.1

Case 1. Defaut charset :UTF8 Client connection:UTF8

>- VARCHAR is fine
>- BLOB text is NOT fine. Result is the same built 145. it seems converted from ansi(CP932),

Case 2. Default charset:SJIS_0208 Client connection:UTF8

>- VARCHAR is fine
>- BLOB text is fine.
>(Of cause, characters range is SJIS_0208 only)


>Result recordset is empty? What is It meaning?

This mean is recordSet.EOF as bellow case 3.

My test code:
> While (Not recordSet.EOF) And row < 60000

Case 3. Defaut charset :UTF8 Client connection:SJIS_0208

- Result recordset is empty .( and V2.5 result is the same, too)

As far as i know , This case issue an arithmetic error from server. because my data have the over SJIS_0208 range charset.

Is this specification of ODBC driver?




Alexander Potapchenko added a comment - 05/Dec/08 09:28 AM - edited
I have absolutely got confused in your non-reproducible cases at me.
1) You must use Firebird 2.1
2) Database ODS must be > = 11.1 (if database was created in Firebird 1.5 or 2.0 then backup/restore is required)
Please use for testing FrameRobin or IBExpert too.
If the result in these applications is differed from result in ODBC then inform me how I can reproduce it.
And also I think your troubles are not related with ODBC driver.

Minoru Yoshida added a comment - 05/Dec/08 09:12 PM
>1) You must use Firebird 2.1

>- FB V2.1.1

>2) Database ODS must be > = 11.1 (if database was created in Firebird 1.5 or 2.0 then backup/restore is required)

Yes.

>Case 3. Defaut charset :UTF8 Client connection:SJIS_0208
>- Result recordset is empty .( and V2.5 result is the same, too)
Excel version:11.0 build 5612.

I try it with IBProvidor. I get -214217837(80049e53) automation error.
But ,This case will be not use case for me. It was reported only.

My ploblem is this case only.
>Case 1. Defaut charset :UTF8 Client connection:UTF8
>- VARCHAR is fine
>- BLOB text is NOT fine. Result is the same built 145. it seems converted from ansi(CP932),

Thanks for your kindly anser.


 

Alexander Potapchenko added a comment - 06/Dec/08 06:14 AM
I have told that ADO does not translate BLOB fields anyway.
Case 2. Default charset:SJIS_0208 Client connection:UTF8
- VARCHAR is fine
- BLOB text is fine.
Can you reproduce it with fb2.1, odc 11.1 and ODBC driver?
If yes, then please give me example of your database.

Minoru Yoshida added a comment - 07/Dec/08 09:23 PM
>Can you reproduce it with fb2.1, odc 11.1 and ODBC driver?

I retry it now on FB2.1.

>Case 2. Default charset:SJIS_0208 Client connection:UTF8
- BLOB text is NOT fine. (Same the default charset is UTF8 )

Your saying is correct, i was mistaken about the IBProvidor's result.