Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Built-in logical replication [CORE2021] #2458

Closed
firebird-automations opened this issue Jul 31, 2008 · 27 comments
Closed

Built-in logical replication [CORE2021] #2458

firebird-automations opened this issue Jul 31, 2008 · 27 comments

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: Daniele Teti (d.teti)

Is duplicated by CORE2149
Is related to CORE4378
Is related to CORE743

Votes: 53

All big project need some replication stuff. All enteprise project needs replication. When software architect need to choose what database for his next project, look for replication and clustering/balancing.

There are some open/closed source vay to do this in FB, but there isn't NONE built-in.

Many customer's sysadmin when I propose Firebird, ask "there is a built-in replication service?"... when I say "no, but there a remany 3rdPart tools to do this..." they choose MySQL (IMHO a bad DB, but with an eye on enterpise).

Replication is THE many issue that I've found in firebird (and many our customer move from fb to mysql for this reason).

This is a big problem for today's project, do not ignore them.

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

priority: Critical [ 2 ] => Major [ 3 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

security: Developers [ 10012 ] =>

@firebird-automations
Copy link
Collaborator Author

Modified by: @pcisar

Link: This issue is duplicated by CORE2149 [ CORE2149 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: Sean Leyne (seanleyne)

Link: This issue is related to CORE4378 [ CORE4378 ]

@firebird-automations
Copy link
Collaborator Author

Modified by: Sean Leyne (seanleyne)

Link: This issue is related to CORE743 [ CORE743 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Sean Leyne (seanleyne)

A feature like replication requires a management/scheduler function to trigger replication to run, and to manage/track replication state information.

That type of functionality is what case CORE743 is about.

As seen by third-party tools like IB-Replicator, the need for engine level changes is not required (they are already doing it without any changes).

So, this case should be defined as a request for a new module, not engine level change.

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

Sean, I disagree. External replicators rely on the additional stuff that should not be necessary (triggers, log table, etc). The engine itself should provide hooks to be used by replication plugins. And some plugin can be built-in as a default one.

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

summary: Built-in replication like MySQL => Built-in replication

@firebird-automations
Copy link
Collaborator Author

Commented by: TIP Group, S.A. (Olivier Mascia) (tipgroup)

I like the view of Dmitry on this item and hooks to ease replication plugins are very interesting because no more alteration of any kind to the database schema would be needed. It allows for many usages, even things only remotely looking alike replication (business logging for instance).

Though, in a radically different direction, if the engine could be extended to participate in larger scheme features like OS snapshot techniques, proper physical replication could be outsourced completely of the database hands.

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

assignee: Dmitry Yemanov [ dimitr ]

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

Fix Version: 4.0 Alpha 1 [ 10731 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: Siva Ramanathan (s2ramana)

There are a few aspects to replication - data and schema changes. The best replication solutions that I have seen i.e. Oracle's Golden Gate etc. replicate both schema and data changes. What is the scope of this task? This would allow for the replication DB to also serve as a hot-stand-by.

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

Both data and schema changes will be replicated.

@firebird-automations
Copy link
Collaborator Author

Commented by: @kattunga

Will this feature support Master - Slave and Master - Master replication like MySql?

Example of Master - Master with MySql:

https://www.digitalocean.com/community/tutorials/how-to-set-up-mysql-master-master-replication

Supporting this configuration could also add some limited clustering capabilities to Firebird.

Regards

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

The current implementation is master-slave(s). Multi-master support may be added later though.

@firebird-automations
Copy link
Collaborator Author

Commented by: @kattunga

Hi Dmitry, I'm sorry to hear that, I'm really urged to scale my application to be HA over wan and I'm stuck with Firebird.

What is it planed?

May the slaves be readonly acceded during replication?

There will be a way to promote a slave to master and master to slave and automatically revert replication?

This things are really important and can give Firebird enough HA capabilities.

Regards

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

Master-slave is basically enough for HA purposes, multi-master is more about balancing the [write] load. And yes, slaves may be accessed read-only. Master<->slave promotion is manual, using gfix. It can be automated using scripts though.

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

Fix Version: 4.0 Beta 1 [ 10750 ]

Fix Version: 4.0 Alpha 1 [ 10731 ] =>

@firebird-automations
Copy link
Collaborator Author

Commented by: @kattunga

Hi Dmitry,

Will this feature support multiple readonly slaves for one master?

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

Yes.

@firebird-automations
Copy link
Collaborator Author

Commented by: @kattunga

Can the replication be enabled at server level, for all databases in a directory instead of by database?

This is critical for us because we are an ERP SAAS provider, right now we are hosting 40 databases in one FB Server and we spec to host 100 more in during next year.

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

This is not possible currently. You need to confure things manually anyway - different active/archive journal directories, different replica paths, etc - I don't see how they could be fully automated.

@firebird-automations
Copy link
Collaborator Author

Commented by: @kattunga

> This is not possible currently. You need to confure things manually.

ok, we will try to automatically setup new databases using a script.

> I don't see how they could be fully automated.

active/archive journal could be the same for all databases with automatically created sub-folders for each database, using a guid that uniquely identifies each database, this guid should be created inside the database when database is created first time, when database is replicated, this guid should be copied so master database and replicas will share this guid.

in firebird.conf you can have options:

ReplicationEnabled =
ReplicationDataBasePath =
ReplicationJournalPath =
ReplicationMaster = <host ip>

Replica path could be automatically determined by ReplicationMaster using the database guid.

This let us in future versions to improve replication creating replicasets like in MongoDb:

https://docs.mongodb.com/manual/replication/#redundancy-and-data-availability

with an option in firebord.conf

ReplicationNodes = <host-1 ip>, <host-2 ip>, <host-3 ip>

we can get real High Availability with Automatic Failover and read for multiple nodes at same time.

@firebird-automations
Copy link
Collaborator Author

Commented by: Philip Williams (unordained)

Dmitry, I can't tell you how excited I am that you guys are working on this. Is there an existing source of public documentation on what this feature will support? (my google-foo has failed me) Below is what I'd be curious about, and would be happy to read in docs:

I've dealt recently with SQLServer's Database Mirroring (DBM) and AvailabilityGroup (AG) replication approaches, and it sounds like you're somewhere in between:
* unlike DBM and like AG, you support read-only replicas
* unlike DBM and like AG, you support multiple replicas
* like DBM and unlike AG, you support only per-database replication (no availability groups of multiple databases -- which makes sense given Firebird is stricter about preventing you from doing cross-database queries, so the need should be lower)
* unlike either, no automatic failover mechanism yet, I'm guessing you're going to depend on an external mechanism (WSFC, pacemaker) to control scripts and a VIP on your behalf?
* assumed: more like AG, you would expect clients to connect to a VIP, and retry if the connection drops, rather than configuring every client to know about every possible replica (like DBM)?
* unlike AG, no applicationintent=readonly routing (and potentially load-balancing) to help a client find a readonly replica (for reports, etc.)
* unclear: synchronous replication only, or asynchronous as an option? (per-replica or per-database setting?)
* how does the "seeding" process work, since you don't have the equivalent of SQLServer's FULL backup mode to prevent garbage-collection and keep a .delta file (equivalent) while waiting for the replica to be seeded and start consuming changes since the backup was made, but you'd need an OAT marker to tell you that such a seed is in the process of being prepared ... ? can a new replica be seeded and joined without outage to the current master?
* do clients connecting to a read-only replica get transaction#⁠'s assigned, and how do you avoid conflicts with the real tx#⁠'s being assigned by the primary? (Maybe I should look at how read-only database mode works today, as you'd have the same problem there -- a database stored on a DVD might contain references to many transactions from its previous life, but the read-only connection would always be "after" them all and not really need a true tx#⁠)
* unclear: would clients connected to a read-only replica get "booted off" if the slave was promoted to master?

@firebird-automations
Copy link
Collaborator Author

Commented by: limingfeng (lmf1967)

Very useful to us! Thank you for your hardwork!

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

summary: Built-in replication => Built-in logical replication

@firebird-automations
Copy link
Collaborator Author

Modified by: @dyemanov

status: Open [ 1 ] => Resolved [ 5 ]

resolution: Fixed [ 1 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants