Commented by: PizzaProgram Ltd. (szakilaci)

Screenshot:
http://pcpincer.hu/downloads/corruptdb/corrupted_trigger.jpg

Commented by: @mrotteveel

This looks more like a support question than a bug report. I'd suggest you post to the firebird-support mailinglist.

You say that you -fix_metadata on every restore. You really should not do that! It is the (probable) cause of the problem you describe. You should use it only once, and only when you know that your metadata is actually in the wrong character set. Once applied, the metadata is in utf-8 and if you reapply -fix_metadata , the bytes are read as win1250 (and not as utf8) and then written out as utf-8, so essentially you are continually reinterpreting (and therefor corrupting) the characters.

fix_metadata cannot "detect" that it is in the wrong character set: you are explicitly instructing Firebird to use the wrong character set, and in the end the only difference between character sets is in the way bytes (or combinations of bytes) are interpreted as characters.

Commented by: PizzaProgram Ltd. (szakilaci)

Yes, I think this is a bug. Using a simple backup-restore procedure by using the official gbak.exe program should NOT corrupt the database !
(It should SAVE the data (backup) from any harm, not destroying it.)

... >> IF << it's really THAT causing the problem.

1.) it is NOT possible to know for us, end users:
- if the .FBK file beeing restored is already a converted one, or an old backup that needs to be converted DURING restore.
So using -fix_metadata switch is a MUST, otherwise restore may fail.
(And doing it "twice" by first only "trying it" without switched is not an option for us.)

2.) IMHO the gbak.exe itself should recognize that,
possibly by setting an [already_converted_metadata=Win1250] flag inside the .FBK file during backup/restore procedure.
- if it's "already converted" >> should ignore the -fix_metadata switches
- if can not find the switch >> should restore procedure first analyze the backup file,
(I don't know how it should do that. Maybe by checking if it contains any of the characters mentioned by the -Fix_ ... charset parameter)
>> and set the flag after that, so this analyze procedure runs only once.

(I've thought database files already containing such a variable to recognize the current charset since FB changed to metadata to Unicode ... are you telling me they don't? Than HOW do they know, how to interpret the data stored inside themselves?)

Commented by: PizzaProgram Ltd. (szakilaci)

THANK YOU !!!

Commented by: @asfernandes

First thing: you created the bug, when not understood things and started to put invalid characters in database. Firebird 2.1 just allowed it.

Second, you didn't read the documentation and are using things incorrectly.

Commented by: PizzaProgram Ltd. (szakilaci)

Yes, I have red ALL the documentation about fix_metadata. Also reported a bug once before about it, and you guys fixed it nicely from 2.5.2 > to 2.5.3.

from gbak help:
-FIX_FSS_D(ATA) fix malformed UNICODE_FSS data
-FIX_FSS_M(ETADATA) fix malformed UNICODE_FSS metadata

... no word mentioning something about:
"WARNING! Use only once per database otherwise corrupting it..."
____________________

from Relase notes 2.5.3:
Page 7
GBAK restore switches FIX_FSS_DATA and FIX_FSS_METADATA to restore legacy databases with
UNICODE_FSS data and/or metadata correctly without resorting to scripts and manual techniques

... again: no word mentioning something about:
"WARNING! Use only once per database otherwise corrupting it..."

Page 40
... no warning

Page 100
... no warning

Page 109 >> even RECOMMENDS it (because I've used the old /misc/upgrade/metadata scripts sometimes
Effects of Unicode Metadata
If you have not previously updated text objects within the metadata of your databases to be in character set
UTF8, restoring a database until V.2.5 will fail with "malformed string" errors. To resolve this it is necessary
to pay attention to the files in the /misc/upgrade/metadata directory of your installation and to use the
new -fix_fss_data and -fix_fss_metadata switches in the gbak command line.
_____________

So blaming me for this alone ... is not very nice.

Commented by: PizzaProgram Ltd. (szakilaci)

But I don't want to argue, because Firebird is GREAT !!! I love it, use it since version 0.9, donated for it, (and will again) etc. Many thanks for doing this great work.

I just would like to see this bug fixed. Because I had no sleep since than. That's all.

THANK YOU !

Commented by: PizzaProgram Ltd. (szakilaci)

Can Anyone verify (by downloading the example DB) that the error occurred because the -fix_metadata switches?

Commented by: Sean Leyne (seanleyne)

@mark and Adriano,

While I agree that the issue is caused by needlessly using the "fix_fss_xxx" switches over-and-over, so there is no technical/engine bug.

I believe that Laszlo comment about the documentation not advising on the "one time use" is an issue, so this case should be changed to a documentation case, no?

@laszlo,

IMO, the subject/description of the case should read "Multiple use of GBAK/Restore "-fix_fss_xxx" switches can lead to corrupted metadata", OK?

Commented by: PizzaProgram Ltd. (szakilaci)

@sean Leyne
Thanks for responding. Yes, it would be a quick / half solution. But I'm still considered:
- It is impossible to know, if an .FBK file (a user is trying to restore) is already a converted one, or not.

So the only "way" to test it, to TEST it twice:
- fist time without the -fix_fss... switch.
Users trying to do that do not understand ANYTHING from a restore procedure-error messages, and usually the whole procedure runs hidden anyway... And the first failure destroys the current DB and leaves a corrupted one. So it makes impossible to restart the Application again, to run a second restore try. (So it leads to a 22-trap.)

Ignoring the -fss_... switches inside gbak.exe while restoring a converted backup file would be the most clear and user-friendly solution.

Personally, as a programmer myself I do not really understand, how did it happen, that the backup/restore procedure does not log this conversion event, to prevent of misusing such a destructive option twice.
If I'm upgrading a database or INI file structure, i do ALWAYS put a version field / increment number / etc ... to read / LOG / handle for a "next time".
Why would it be "so hard" to fix this in the 2.5.4 release?

Commented by: @asfernandes

Data in one character set may be valid in another character set and mean something completely different.

You should know your data or your access (read/write) pattern of your applications, operating systems, etc.

There is no such a thing as detect an invalid (in the meaning of it don't being what you want) data.

If it's really invalid (in bytes, not the glyphs you expect), a restore without fix options will tell you what it knows, i.e., that it's invalid.

Commented by: PizzaProgram Ltd. (szakilaci)

I assumed that too, that auto-recognizing an invalid charset would not be so easy.
That's why I've suggested a "simple" quick-fix:
* Putting a FLAG/Boolean/string...(whatever mark) inside the DB's metadata, that would be saved into the backup files telling:
-- "I'm already a once-converted database"

So basically it's ca. 5 lines of code:

While running a restore:

1. Try to READ out boolean from backup file's metadata: FIX_FSS_ONCE_ALREADY_APPLIED

1.1: IF (it's already Exists) >>>>>>
[ignore Fix_fss_... switches] (and maybe show a non-blocking warning about the event)

1.2: IF (it does not Exists) AND (Fix_fss_ switch presents) >>>>>>
[APPLY conversion] and WRITE that Boolean into the DB metadata

Commented by: Sean Leyne (seanleyne)

Laszlo,

It seems that you are alone in thinking that:
- the issue is a serious bug (Mark and Adriano might agree that it is a "nice to have feature")
- migrating databases from v2.1 to v2.5 is a common occurrence that requires safety
- it is the engine's responsibility to track whether "FIX_FSS" has already been used on a database

So, I don't see this case proceeding as you would like.

Accordingly, unless there are objections raised by other community members by the end of April 2015, this case will be closed as "Won't Fix".

Commented by: @hvlad

Adriano,

we already discussed this issue in the past and there was no good solution found.
The problem is that backup file have no flag that database had correct characters in
unicode fields. The database itself have such flag - we consider all database in ODS 11.2
as "good" database. If above is correct - what if we add source database ODS number
into backup file and use it as a flag to ignore FIX_FSS_XXX switches by restore ?

Commented by: PizzaProgram Ltd. (szakilaci)

Dear Vlad,
this would be a PERFECT solution :) That's all I ask in name of all "end-users".
Thanks !

PS: as soon as it's done, would be nice to mention that in the documentation too ;)

Commented by: @hvlad

Found i already offered to backup source database ODS number : see CORE3336, but that time discussion was left nowhere...

Commented by: Sean Leyne (seanleyne)

Vlad pointer to CORE3336 makes it clear that this issue is a duplicate, so this item should be closed and any further discussion/comments be added to the older/first reported issue.

Commented by: @asfernandes

Let's recap... v2.1 has released many years ago. In it, we developed a upgrade script, because conversion of character set in DDL operations was fixed. But it still allowed users to do bad things. It also had some problems with blobs.

Then v2.5 has released, also many years ago. Since it didn't allowed bad data anymore, and we didn't want people migrating from 2.0 to 2.5 directly to need to migrate to 2.1 to fix data, we developed the gbak options.

So many and many people did the right thing, which were supposed to be done when migrating from 2.0 to 2.1. Two users had problems.

Do you still think an effort should be put on this?

It seems that the only correct approach is to treat v2.5 databases as having good data. But users who known what the GBAK options are already know that.

Commented by: PizzaProgram Ltd. (szakilaci)

@adriano
Yes, it should be... because:
2.0 + 2.1 backups do not disappear! And they need to be restored frequently. What makes us to use the -Fix_fss... switches always. (Otherwise restore fails.)

Commented by: PizzaProgram Ltd. (szakilaci)

@sean
The previous topic is closed too, but unresolved. So if you close this, the other one should be REOPENED !

Commented by: Sean Leyne (seanleyne)

Support for all pre-v2.5 Firebird versions was discontinued in January 2015, and since this issue only relates to the migration/restore of pre-2.5 databases, all of the related topics will remain closed.