Commented by: @dyemanov

If some utility would be to able to apply the journal segment immediately (by explicit command), would this solve your goal?

Commented by: hugo (hugolarson)

Sounds like right.
Also to be able to set up replica FDB's without configuration. Ideally on both replica and master side. Maybe also by explicit command.
A command to create journals explicitly on master side would also be very handy.

One of FB strengths is that an entire DB is just one file which can be accessed and moved around very easily with zero configuration. I think this philosophy should be kept with replica feature if possible.

Hi,
Any timeline on this feature?

We have the same problem. Our user's can create as many databases as they need with our GUI. They haven't got knowledge to edit config files and it is impossible to restart the server while, users are using other databases.

Such a solution would be very useful for us. (Hundreds of databases in continuous operation.)

I intended the below description to be thought-provoking.
My goal is to enable any number of asynchronous synchronization on working databases at any time without stopping firebird(s).
Even if an earlier such replication stopped working for any reason.

Asynchronous replication of existing databases with continuously running Firebird (thoughts)

Prerequisites:

• no synchronous replication is enabled on the database (maybe not required)
• the database is not in the nbackup locked state
• the master database cannot be in any step of configuration described below (one replication can be set at a time)

Master side:

M1. At the database level, we should indicate that we are setting up a new asynchronous replication (e.g. test_sync1)
alter database setup async_replication test_sync1

M2. The next step is to be able to set up the data required for asynchronous synchronization (the commands can only be used between step M1 and M3)
alter database configure test_sync1 journal_directory C:\firebird\async\logdir1;
alter database configure test_sync1 journal_archive_directory C:\firebird\async\logdir1_archive;
alter database configure test_sync1 log_file_prefix test_sync_db;
alter database configure test_sync1 log_segment_size = 16777216;
etc.

This parameters should be accessable via read only system table, e.g.
select * from rdb$async where rdb$async_name = 'test_sync1';
rdb$async_name rdb$journal_directory rdb$journal_archive_directory rdb$async_log_file_prefix rdb$async_log_segment_size rdb$async_state rdb$async_lock_state
test_sync1 C:\firebird\async\logdir1 C:\firebird\async\logdir1_archive test_sync_db 16777216 setup/running/stopped 0

This parameters should be updateable only before M3 step is activated.

M3. Activate the test_sync1 replication:
alter database start async_replication test_sync1

At this point

• the engine should do an nbackup database lock: nbackup -L
• and the same time it should start to write the replica log files

Then the DBA can copy the database to the replica side and configure the replica there.

M4. After database copy is ready to the replica side DBA be able to merge the delta file:
alter database release_delta async_replication test_sync1

At this point the engine should merge the nbakcup delta file and continuously write the async log files.

Replica side:

R1. After step M3 with a copy command we have the database file on replica side.

R2. We should configure the replica side:
alter database configure test_sync1 journal_source_directory C:\firebird\async\logsourcedir1;
e.g.

DBA should go the tasks to provide the replica log files the the right place.

R3. DBA activate the replica side:
alter database activate async_replica test_sync1 read_only

(
M5. Canceling or stoping master side setup/replication:
alter database stop async_replication test_sync1

If once the replication was started, it will stop and it can be restarted by step M3 and M4, if not its configuration will droped immediately by stop command.

rdb$async_name.rdb$async_state:

• setup : phase M2
• running : phase M3, M4
• stopped : phase M5
  rdb$async_name.rdb$async_lock_state: (nbackup lock state)
• 0 : M2, M3, M5
• 1 : M4
  )

András

edit 1.: gbak should not backup the rdb$async table, if someone restore the database this kind of replication should be set up again.

First of all, replication may be started/stopped without restarting the Firebird server. However, if configuration is changed, then all users should be re-connected to take it into account. With #6977 implemented, I think this would not be required.

Then, Firebird is not "zero configuration", it may just be used that way. With lots of databases in operation, you either rely on default configuration options, or setup specific configuration that's shared by all databases. Note that you don't set any options via DDL! Mostly the same approach is used for replication. Almost all settings has reasonable defaults and may be specified for all replicated databases at once. The major difference is that replication also needs target paths that are usually database-specific and here per-database configuration becomes mandatory and this starts causing problems for you. Let's see what could be done in this regard:

1. Nothing. If you can run ISQL with ALTER DATABASE, then you can also generate textual /myrepl/mydb123.conf with options needed for database mydb123.fdb and add the include /myrepl/*.conf line to replication.conf just once to take all your custom config files into account.
2. We could add support for more macros and use them inside journal_*_directory settings. This way you could specify e.g. journal_directory = $(dbdir)/logs/; journal_archive_directory = $(dbdir)/logs/archive/ inside the global section of replication.conf, so that journals are always created inside the logs and logs/archive sub-directories of the database location. Or maybe journal_directory = /journals/$(dbname)/; journal_archive_directory = /journals/$(dbname)/archive/, if you prefer.
3. We could assume that missing journal locations mean the directory where the database itself resides, so that journals are stored side-by-side with the database (similarly to how the nbackup delta file is created). In this case, archiving may not be used, the engine would just add new journal segments instead of the reusing/renaming the existing ones (the active journal segment should be specially marked/named in this case to prevent its copying to the replica). Or archiving is performed into the /archive sub-directory of the database location.

I don't like storing paths inside the database as it makes it not relocatable. Think about UDFs/UDRs or external files, for example - we allow to store only their names but the full path is resolved using firebird.conf (UdfAccess, ExternalFIleAccess). Even if these settings are to be ignored by gbak, they will still be preserved by nbackup or physical file copying.

Regarding M3, I see the point. But I don't think that starting replication always require a database copy to be transferred to the replica side, at least immediately. And given that M4 has nothing specific to replication and does the same as ALTER DATABASE END BACKUP, I'd suggest to just run ENABLE PUBLICATION, then BEGIN BACKUP (immediately or on demand), then END BACKUP (when copy is ready).

Hi!

Also a user with hundreds of databases in 2.5 looking to migrate to 4.0 in the future.
I think more macros in replication.conf would be a viable solution.

Alternatively maybe the journal files should be prefixed with database name or guid.
This way all databases on the server could share a common directory.

There is a setting journal_file_prefix in the config, but I can see no way to specify it globally.

Journal files are prefixed with the database name by default, journal_file_prefix exists only to override that default. So if you define global journal_directory / journal_archive_directory settings (in the default database section of replication.conf), then journals of all databases will co-exist side by side (provided your database names are unique).

That doesn't seem to be the case.
I'm using the official Firebird-4.0.2.2816.amd64 on Linux
The journal file is named .journal-000000003

In the config the prefix is per default unspecified (commented out):

# journal_file_prefix =

I'm using the default replication.conf with only the paths specified.

@mrgleba, could you please specify exactly the database = part of replication.conf. And whether the db path is local or located on a NFS share.

replication.conf:

database
{
  journal_directory =  /srv/j
  journal_archive_directory =  /srv/ja
  journal_archive_timeout = 60
}

The file is on a local filesystem

It should look like database = /my/db/path

I've tried with a general config only.
We're running hundreds of databases and it'd be problematic to alter config every time a new db is added.

As I understand the general config should work: the journal_directory would be common and the journal files
should come ith the db name as a suffix?
Currently there is no suffix: .journal-XXXXXX

Funny. I was sure this is prohibited, but it seems likely to work, provided that the missing db name prefix is fixed.

Database file name might cause collisions.
Perhaps the database GUID would be a better candidate or journal-{dbname}-{guid}-xxxxx ?

I can confirm the primary side can work without per-database configuration settings. However, it's still impossible on the replica side, making the whole effort rather pointless.

Actually it'd still be very valuable:

• the replica server's config can be updated and the server restarted without stoping the mater server
• the journal files could simply be storead as a form of backup rather than replicated right away

Now we do nbackup at intervals to achieve incremental backup/point in time restore. This change could simplify our operations

OK, I will consider including the fix into the next v4 post-release.