Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a new param in GBAK to backup / restore tables/indexes in parallel (using all CPUs in CpuAffinity) [CORE1365] #1783

Closed
firebird-automations opened this issue Jul 18, 2007 · 7 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: Eduardo Jedliczka (edujed)

Is duplicated by CORE3958
Relate to CORE2992

Votes: 1

I have a 8-core machine, and when I do a Gbak of a mid-size database (more than 15 GB) the restore time is near 2 hours using only one CPU, because the restoration process is linear.
If is possible, backuping / restoring data from 1 table per CPU concurrently (in other words, in parallel) this time will be strongly lower.
if is possible, rebuild 1 index per CPU can optimise this restore operation.

@firebird-automations
Copy link
Collaborator Author

Commented by: Sean Leyne (seanleyne)

Given that a restore is very disk intensive, I believe the bottleneck for the restore is not the CPU/processor but the disk subsystem.

Therefore, I don't see any benefit in trying to perform the restore from multiple threads.

Finally, you should investigate the new NBackup utility (new in v2.0) it performs backups/restores at the physical database level and runs at hardware speeds!

@firebird-automations
Copy link
Collaborator Author

Commented by: Eduardo Jedliczka (edujed)

in this case, the bottleneck is not disk... It is a RAID 0+1 with SAS disks 15k RPM with 2 GB of cache. But one CPU uses 100% in all time of restore process.

Of coarse, I know nbackup, but in case of corruption (hardware fail, energy loss) nbackup do not recovery the database.

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

There are some problems with your suggestion.
First, in pre-vulcan/fb3 architecture you must use classic server (I suppose you already do on 8-CPU machine).
Next, gbak's backup format is strictly linear (it and contains data in DPB-like format - certainly, this is just analogy, but more or less correct), i.e. you can't say about the size of a table not reading all of it's data record by record.
And last seen at once problem - restoring database gbak creates it in single-user shutdown mode, i.e. no one (including sysdba) can attach before gbak completion. This solves a lot of problems happening due to attempts to attach to incomplete database. Such protection will have to be removed for multiway restore.

Anyway I like the suggestion - restore is really CPU intensive thing (for example, indices creation is also performed at the end of restore). Not sure all of your 8 CPU's will work efficiently, but certainly it may be done faster then now.

@firebird-automations
Copy link
Collaborator Author

Commented by: Eduardo Jedliczka (edujed)

Thanks for the answer.

Yes, you're wright, I user FireBird Classic Server.
Second, I understood the problem when a user connect in a database before the restore completion, I lost much time with it in FB 1.X, but it can be changed modifyind the way of restoration (only sub-processes - one for each index and/or CPU - of restoration can connect in a "in-restore" database)
Third, really, the greathest problem is the way of gbak works... DPB-like format.... Humm... initially I beleave gbak maybe work with pages (like FB) but it's nonsense.
Finally, I beleave, the most simple way of to that, is "sequential database restoration" with "parallel index creation".

@firebird-automations
Copy link
Collaborator Author

Modified by: @pcisar

Workflow: jira [ 12611 ] => Firebird [ 14213 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

Link: This issue is duplicated by CORE3958 [ CORE3958 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

Link: This issue relate to CORE2992 [ CORE2992 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants