Issue Details (XML | Word | Printable)

Key: CORE-4462
Type: Improvement Improvement
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Alexander Peshkov
Reporter: Alexander Peshkov
Votes: 2
Watchers: 5
Operations

If you were logged in you would be able to see more operations.
Firebird Core

Make it possible to restore compressed .nbk files without explicitly decompressing them

Created: 17/Jun/14 10:19 AM   Updated: 02/Jan/20 08:40 PM
Component/s: NBACKUP
Affects Version/s: 3.0 Beta 1
Fix Version/s: 3.0.5, 4.0 Beta 2

File Attachments: 1. Text File nbackup-decompress-level1.patch (0.8 kB)


QA Status: Done successfully
Test Specifics: Platform (Windows/Linux) specific

Sub-Tasks  All   Open   

 Description  « Hide
Ability to compress nbackup output on the fly helps to save both backup time and avoid need in disk space for intermediate uncompressed files:
nbackup -b 0 employee stdout | bzip2 >e.b0.bz2
nbackup -b 1 employee stdout | bzip2 >e.b1.bz2

Unforutnately that trick does not work when restore is needed cause a set of uncompressed files is required for utility:
nbackup -r e.fdb e.b0 e.b1

It would be great to make nbackup decompress files on the fly one by one, without wasting resources for intermediate files.

 All   Comments   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Alexander Peshkov made changes - 17/Jun/14 10:21 AM
Field Original Value New Value
Assignee Alexander Peshkov [ alexpeshkoff ]
Alexander Peshkov added a comment - 17/Jun/14 11:26 AM
Added switch -DEcompress with parameter containing command line decompressing .nbk file to stdout. Symbol @ in that command line is replaced with file name to be decompressed. If that symbol is not met in command line file name (with space before it) is appended to the end of command line. Certainly if command line contains >1 word it should be quoted according to your shell rules.

Samples of how can be restored mentioned in description backup:
nbackup -de bzcat -r e.fdb e.b0.bz2 e.b1.bz2
nbackup -de 'bzip2 -d -c' -r e.fdb e.b0.bz2 e.b1.bz2
nbackup -de 'bzip2 -d -c @' -r e.fdb e.b0.bz2 e.b1.bz2
All this commands give same result - database is successfully restored.

Sean Leyne added a comment - 17/Jun/14 03:30 PM
Wouldn't it make sense to also add a new switch to have the 'backup' step generate a compressed file directly, without the need to pipe the output?

Alexander Peshkov added a comment - 18/Jun/14 06:24 AM
I will check is it possible. If there is opened database at the time of creating backup, this causes some problems when forking.
Anyway first of all I want windows version from Vlad to clone it for backup case.

Damyan Ivanov added a comment - 14/Jan/18 04:02 PM
Hi,

While piped decompression seems to be implemented in 3.0.x series (on POSIX systems), it has a bug which leads to an error when more than one file is given to -restore (e.g. multi-level restore is performed):

$ nbackup -user sysdba -decompress 'gzip -dc' -restore test-restored.fdb test.nbk.0.gz test.nbk.1.gz
IO error seeking file: test.nbk.1.gz
-Illegal seek

The reason is that for any incremental file after the level 0 file, nbackup tries to skip a page-size block off the start of the file using seek() (nbackup.cpp line 1430), but because of the -decompress involved, the file handle is a pipe leading to the error in seek().

Here's a patch which emulates the seek with a series of read()s so that the file pointer ends up one page size from the start of the file.

Damyan Ivanov made changes - 14/Jan/18 04:02 PM
Attachment nbackup-decompress-level1.patch [ 13201 ]
Vlad Khorsun added a comment - 07/Dec/19 06:08 PM
Windows implementation is committed (see CORE-4463).

It was checked using 7z and zstd:

nbackup -DE "7z -so e"
nbackup -DE "zstd -dc --no-progress"

Or, with more verbose output of decompressor

nbackup -DE "7z -so -bsp2 e"
nbackup -DE "zstd -dcv"

Dmitry Yemanov added a comment - 31/Dec/19 10:05 AM
Should this ticket be closed now? AFAIU, v3.0.5 contains this improvement for both Posix and WIndows.

Sean Leyne added a comment - 31/Dec/19 06:41 PM
Dmitry,

I would make 1 suggestion:

define default switches (to decompress the files) for the 7z and zstd methods so that the only the "-de + algorythm" need be specified ie. "nbackup -r -de 7z {db filename} {nbackup filename1} ..." or "nbackup -r -de zstd {db filename} {nbackup filename1} ...", extra decompression options/details should only be require for special cases.

Alexander Peshkov added a comment - 01/Jan/20 09:51 AM
Sean, in current state that's bad suggestion. Why this 2 particular utilities? Others may prefer other tools.
I think if we decide to do it then the set of utilities should be configurable. Or better leave it as is - it was initially designed to support any tool and any format, and adding exception for some of them will definitely raise a support question - why "-de rar" does not work?

Alexander Peshkov added a comment - 01/Jan/20 09:52 AM
@Dmitry - yes, we can close it now.

Dmitry Yemanov made changes - 01/Jan/20 10:17 AM
Status Open [ 1 ] Resolved [ 5 ]
Fix Version/s 3.0.5 [ 10885 ]
Fix Version/s 4.0 Beta 2 [ 10888 ]
Resolution Fixed [ 1 ]
Pavel Zotov made changes - 02/Jan/20 08:40 PM
Status Resolved [ 5 ] Resolved [ 5 ]
Test Specifics [Platform (Windows/Linux) specific]
QA Status Done successfully
Pavel Zotov made changes - 02/Jan/20 08:40 PM
Status Resolved [ 5 ] Closed [ 6 ]