Issue Details (XML | Word | Printable)

Key: CORE-2021
Type: New Feature New Feature
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Dmitry Yemanov
Reporter: Daniele Teti
Votes: 53
Watchers: 35
Operations

If you were logged in you would be able to see more operations.
Firebird Core

Built-in logical replication

Created: 31/Jul/08 04:50 AM   Updated: 25/Jan/19 09:19 AM
Component/s: Engine
Affects Version/s: None
Fix Version/s: 4.0 Beta 1

Environment: Any
Issue Links:
Duplicate
 
Relate
 


 Description  « Hide
All big project need some replication stuff. All enteprise project needs replication. When software architect need to choose what database for his next project, look for replication and clustering/balancing.

There are some open/closed source vay to do this in FB, but there isn't NONE built-in.

Many customer's sysadmin when I propose Firebird, ask "there is a built-in replication service?"... when I say "no, but there a remany 3rdPart tools to do this..." they choose MySQL (IMHO a bad DB, but with an eye on enterpise).

Replication is THE many issue that I've found in firebird (and many our customer move from fb to mysql for this reason).

This is a big problem for today's project, do not ignore them.

 All   Comments   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Sean Leyne added a comment - 25/Apr/14 02:46 AM - edited
A feature like replication requires a management/scheduler function to trigger replication to run, and to manage/track replication state information.

That type of functionality is what case CORE-743 is about.

As seen by third-party tools like IB-Replicator, the need for engine level changes is not required (they are already doing it without any changes).

So, this case should be defined as a request for a new module, not engine level change.

Dmitry Yemanov added a comment - 25/Apr/14 03:24 AM
Sean, I disagree. External replicators rely on the additional stuff that should not be necessary (triggers, log table, etc). The engine itself should provide hooks to be used by replication plugins. And some plugin can be built-in as a default one.

TIP Group, S.A. (Olivier Mascia) added a comment - 10/Nov/14 10:36 PM
I like the view of Dmitry on this item and hooks to ease replication plugins are very interesting because no more alteration of any kind to the database schema would be needed. It allows for many usages, even things only remotely looking alike replication (business logging for instance).

Though, in a radically different direction, if the engine could be extended to participate in larger scheme features like OS snapshot techniques, proper physical replication could be outsourced completely of the database hands.

Siva Ramanathan added a comment - 15/Mar/16 01:26 PM
There are a few aspects to replication - data and schema changes. The best replication solutions that I have seen i.e. Oracle's Golden Gate etc. replicate both schema and data changes. What is the scope of this task? This would allow for the replication DB to also serve as a hot-stand-by.

Dmitry Yemanov added a comment - 15/Mar/16 01:47 PM
Both data and schema changes will be replicated.

Christian Pradelli added a comment - 29/Jul/16 08:10 PM
Will this feature support Master - Slave and Master - Master replication like MySql?

Example of Master - Master with MySql:

https://www.digitalocean.com/community/tutorials/how-to-set-up-mysql-master-master-replication

Supporting this configuration could also add some limited clustering capabilities to Firebird.

Regards

Dmitry Yemanov added a comment - 29/Jul/16 09:40 PM
The current implementation is master-slave(s). Multi-master support may be added later though.

Christian Pradelli added a comment - 30/Jul/16 12:39 AM
Hi Dmitry, I'm sorry to hear that, I'm really urged to scale my application to be HA over wan and I'm stuck with Firebird.

What is it planed?

May the slaves be readonly acceded during replication?

There will be a way to promote a slave to master and master to slave and automatically revert replication?

This things are really important and can give Firebird enough HA capabilities.

Regards

Dmitry Yemanov added a comment - 30/Jul/16 05:32 AM
Master-slave is basically enough for HA purposes, multi-master is more about balancing the [write] load. And yes, slaves may be accessed read-only. Master<->slave promotion is manual, using gfix. It can be automated using scripts though.

Christian Pradelli added a comment - 01/Oct/17 06:23 PM
Hi Dmitry,

Will this feature support multiple readonly slaves for one master?

Dmitry Yemanov added a comment - 01/Oct/17 06:38 PM
Yes.

Christian Pradelli added a comment - 01/Oct/17 06:55 PM
Can the replication be enabled at server level, for all databases in a directory instead of by database?

This is critical for us because we are an ERP SAAS provider, right now we are hosting 40 databases in one FB Server and we spec to host 100 more in during next year.

Dmitry Yemanov added a comment - 06/Oct/17 10:35 AM
This is not possible currently. You need to confure things manually anyway - different active/archive journal directories, different replica paths, etc - I don't see how they could be fully automated.

Christian Pradelli added a comment - 06/Oct/17 12:26 PM
> This is not possible currently. You need to confure things manually.

ok, we will try to automatically setup new databases using a script.

> I don't see how they could be fully automated.

active/archive journal could be the same for all databases with automatically created sub-folders for each database, using a guid that uniquely identifies each database, this guid should be created inside the database when database is created first time, when database is replicated, this guid should be copied so master database and replicas will share this guid.

in firebird.conf you can have options:

ReplicationEnabled =
ReplicationDataBasePath =
ReplicationJournalPath =
ReplicationMaster = <host ip>

Replica path could be automatically determined by ReplicationMaster using the database guid.

This let us in future versions to improve replication creating replicasets like in MongoDb:

  https://docs.mongodb.com/manual/replication/#redundancy-and-data-availability

with an option in firebord.conf

ReplicationNodes = <host-1 ip>, <host-2 ip>, <host-3 ip>

we can get real High Availability with Automatic Failover and read for multiple nodes at same time.

Philip Williams added a comment - 01/Mar/18 08:17 PM
Dmitry, I can't tell you how excited I am that you guys are working on this. Is there an existing source of public documentation on what this feature will support? (my google-foo has failed me) Below is what I'd be curious about, and would be happy to read in docs:

I've dealt recently with SQLServer's Database Mirroring (DBM) and AvailabilityGroup (AG) replication approaches, and it sounds like you're somewhere in between:
* unlike DBM and like AG, you support read-only replicas
* unlike DBM and like AG, you support multiple replicas
* like DBM and unlike AG, you support only per-database replication (no availability groups of multiple databases -- which makes sense given Firebird is stricter about preventing you from doing cross-database queries, so the need should be lower)
* unlike either, no automatic failover mechanism yet, I'm guessing you're going to depend on an external mechanism (WSFC, pacemaker) to control scripts and a VIP on your behalf?
* assumed: more like AG, you would expect clients to connect to a VIP, and retry if the connection drops, rather than configuring every client to know about every possible replica (like DBM)?
* unlike AG, no applicationintent=readonly routing (and potentially load-balancing) to help a client find a readonly replica (for reports, etc.)
* unclear: synchronous replication only, or asynchronous as an option? (per-replica or per-database setting?)
* how does the "seeding" process work, since you don't have the equivalent of SQLServer's FULL backup mode to prevent garbage-collection and keep a .delta file (equivalent) while waiting for the replica to be seeded and start consuming changes since the backup was made, but you'd need an OAT marker to tell you that such a seed is in the process of being prepared ... ? can a new replica be seeded and joined without outage to the current master?
* do clients connecting to a read-only replica get transaction#'s assigned, and how do you avoid conflicts with the real tx#'s being assigned by the primary? (Maybe I should look at how read-only database mode works today, as you'd have the same problem there -- a database stored on a DVD might contain references to many transactions from its previous life, but the read-only connection would always be "after" them all and not really need a true tx#)
* unclear: would clients connected to a read-only replica get "booted off" if the slave was promoted to master?

limingfeng added a comment - 17/Nov/18 06:13 AM
Very useful to us! Thank you for your hardwork!