[#CORE-3765] VARCHAR ARRAY of UTF8 Charset fails

Firebird Core

VARCHAR ARRAY of UTF8 Charset fails

Created: 17/Feb/12 11:08 PM Updated: 14/Jan/16 09:06 AM

Component/s:

Engine

Affects Version/s:

2.5.1

Fix Version/s:

None

Environment:

Win 7 64 bit and 32 bit

Description

« Hide

When I connect via UTF8 Charset and open a query and fetch a record and get the array data I get an error:
(From the IBO SQL trace monitor)

/*---
GET SLICE
  ARRAY ID (129, 501)
  ARRAY DTYPE 37
  ARRAY SCALE 0
  ARRAY LENGTH 15
  ARRAY DIMENSIONS 1
  ARRAY RELATION 'JOB'
  ARRAY FIELD 'LANGUAGE_REQ'
  SLICE DATA LENGTH 310
  SLICE DATA:
  1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
  2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
  3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
  4: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
  5: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
  6: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
  7: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
  8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
  9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
10: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
11: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
12: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
13: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
14: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
15: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
17: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
18: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~
19: 0 0 0 0 ~~~~

ERRCODE = 335544321
----*/
/*---
INTERPRET BUFFER =

ERRCODE = 60
----*/
/*---
INTERPRET BUFFER = arithmetic exception, numeric overflow, or string truncation

ERRCODE = 23
----*/
/*---
INTERPRET BUFFER = string right truncation

ERRCODE = -1
----*/

This is against the Employee.gdb database that comes standard with earlier versions of InterBase. If I change to Charset ASCII then I have no problem getting the data. Somehow something inside the engine is tripping up when it comes to handling UTF8 (MBCS) in array columns.

Description

When I connect via UTF8 Charset and open a query and fetch a record and get the array data I get an error: (From the IBO SQL trace monitor) /*--- GET SLICE ARRAY ID (129, 501) ARRAY DTYPE 37 ARRAY SCALE 0 ARRAY LENGTH 15 ARRAY DIMENSIONS 1 ARRAY RELATION 'JOB' ARRAY FIELD 'LANGUAGE_REQ' SLICE DATA LENGTH 310 SLICE DATA: 1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 4: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 5: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 6: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 7: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 10: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 11: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 12: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 13: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 14: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 15: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 17: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 18: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ~~~~~~~~~~~~~~~~~ 19: 0 0 0 0 ~~~~ ERRCODE = 335544321 ----*/ /*--- INTERPRET BUFFER = ERRCODE = 60 ----*/ /*--- INTERPRET BUFFER = arithmetic exception, numeric overflow, or string truncation ERRCODE = 23 ----*/ /*--- INTERPRET BUFFER = string right truncation ERRCODE = -1 ----*/ This is against the Employee.gdb database that comes standard with earlier versions of InterBase. If I change to Charset ASCII then I have no problem getting the data. Somehow something inside the engine is tripping up when it comes to handling UTF8 (MBCS) in array columns.

Show »

All

Comments

Change History

Subversion Commits

Sort Order:

[ Permalink | « Hide ]

Kovalenko Dmitry added a comment - 18/Feb/12 04:25 AM

Really critical?

Very strange :)

Hint: forget about official API for work with array and use only isc_put_slice/isc_get_slice.

Good luck.

[ Permalink | « Hide ]

Jason Wharton added a comment - 18/Feb/12 12:33 PM

Ok, I'm going to try and be polite here.

YES IT IS CRITICAL!

I am using isc_put_slice() and isc_get_slice() and I'm not looking for luck, I'm looking for results.

It is currently not possible, as best as I can tell, to make use of array VARCHAR data with UTF8 Charset. This means all of my customers who are trying to migrate apps to Unicode are stuck with no path forward other than to totally rework their apps to not use array columns anymore. This is not acceptable to them as this is a feature/capability that should work.

[ Permalink | « Hide ]

Adriano dos Santos Fernandes added a comment - 18/Feb/12 01:27 PM

My personal opinion is (serious) that we should remove arrays from Firebird sooner rather than later.

[ Permalink | « Hide ]

Alexander Peshkov added a comment - 19/Feb/12 08:00 AM

Not good idea to remove a feature that people are actively using.
For example, I've used to consult our sponsors from Luxemburg not long ago how to access them from UDF.

[ Permalink | « Hide ]

Antonio Fortuny added a comment - 25/Apr/12 06:50 AM

Hi all.

Alex is right. I'm the guy who asked him some help to deal with arrays and UDFs To be clear, I'm Antonio from Sita Software of Luxemburg. Thanks to Alexander's answers to my questions I get rid of that problem: It works. I'll be now on the way to deal with UTF8, and arrays will appear again for the varchar. And that's probably why Jason comes in because we make an intensive use of IBOs objects as well.
I'm aware that arrays should be avoided as much as possible from any DB but it's history (10 years) from our point of view and it must work.

Antonio.

[ Permalink | « Hide ]

Kovalenko Dmitry added a comment - 13/Jan/16 07:03 PM

> "YES IT IS CRITICAL! "

Hm.

Very strange.

IBProvider reads (now and in 2012 year also) this array-field in connection with UTF8-charset without any problem.

I offer to close this ticket :)

[ Permalink | « Hide ]

Jason Wharton added a comment - 13/Jan/16 07:51 PM - edited

I protest.

Note: Yes, I am irritated by how I have been treated so dismissively in this issue.

[ Permalink | « Hide ]

Kovalenko Dmitry added a comment - 14/Jan/16 05:25 AM - edited

Why "protest" ?

[IBProvider 3.27]

SQL: select LANGUAGE_REQ from JOB where JOB_CODE='SRep' and JOB_GRADE=4 and JOB_COUNTRY='France'

connection charset: UTF8 (irrelevant, because LANGUAGE_REQ has NONE charset)

[Connection through fbclient.dll]

Data from isc_get_slice (75 bytes):
0x04995F64 45 6e 67 6c 69 73 68 0a 20 20 20 20 20 20 20 46 72 65 6e 63 68 0a 20 20 20 English. French.
0x04995F7D 20 20 20 20 20 53 70 61 6e 69 73 68 0a 20 20 20 20 20 20 20 0a 20 20 20 20 Spanish. .
0x04995F96 20 20 20 20 20 20 20 20 20 20 0a 20 20 20 20 20 20 20 20 20 20 20 20 20 20 .

[Direct connection to Firebird through INET-protocol (TCP/IP)]

Data from op_get_slice (75 bytes):
0x02F37E6C 45 6e 67 6c 69 73 68 0a 20 20 20 20 20 20 20 46 72 65 6e 63 68 0a 20 20 20 English. French.
0x02F37E85 20 20 20 20 20 53 70 61 6e 69 73 68 0a 20 20 20 20 20 20 20 0a 20 20 20 20 Spanish. .
0x02F37E9E 20 20 20 20 20 20 20 20 20 20 0a 20 20 20 20 20 20 20 20 20 20 20 20 20 20 .

NOTE: My currect code obtains the VARCHAR-array as CHAR-array (for avoid the CORE-1588).

My previous versions, which received the VARCHAR-array as VARCHAR-array (by fact, server returns the CSTRING-array, see CORE-1588), also worked without problems.

----
So I propose to close the ticket.

[ Permalink | « Hide ]

Jason Wharton added a comment - 14/Jan/16 07:04 AM

Simple answer: I protest because the problem has not been fixed.
I do not accept the work around to treat VARCHAR as CHAR.

[ Permalink | « Hide ]

Kovalenko Dmitry added a comment - 14/Jan/16 07:23 AM

Read array's blob directly and you get the VARCHAR-array without any problem.

It is not difficult.

[ Permalink | « Hide ]

Jason Wharton added a comment - 14/Jan/16 07:45 AM

Of course it isn't difficult. I'm not here groveling for tech support.
I am using the API call that does that and encountering a problem.
Have you repeated the problem or do I need to spoon feed you?

[ Permalink | « Hide ]

Kovalenko Dmitry added a comment - 14/Jan/16 09:06 AM

Look to ~~CORE-2500~~

Use "isc_blr_text2" and "isc_blr_varying2" (with explicit definition of text charset) instead "isc_blr_text" and "isc_blr_varying".

Good luck.