Add HA support for other databases (Praefect, Geo, CI) (#7292) · Issues · GitLab.org / omnibus-gitlab · GitLab (original) (raw)

Add HA support for other databases (Praefect, Geo, CI)

After the great work done so far in laying the groundwork for multiple database support in Omnibus we're a lot closer to supporting HA other databases in Omnibus (Praefect, Geo Tracking).

I've been testing this out for the last few days as part of efforts to explore adding this to the Reference Architectures, a highly requested feature from customers. Ideally the goal we're looking to support here is the following setups:

GitLab and Praefect / Geo DB's colocated - Where databases are on the same cluster, connecting through the same PgBouncer array. Note - Doesn't support Geo due to replication requirements.
GitLab and Praefect / Geo DB's separated - Where databases have their own cluster and PgBouncer array. Note - Does support Geo, only the main GitLab DB is replicated.

There's also variations of the above, such as Praefect and Geo being colocated but the main GitLab database being separated.

It doesn't seem possible to set up these setups today in Omnibus. This is because the watcher and failover scripts in omnibus are only added once, and for a single configurable database.

When colocated, the service, watcher and failover script all effectively are only looking for a single database name and as a result if you attempt to set up another database connection, like praefect_production, it will be ignored:

[databases]

gitlabhq_production = host=gitlab-qa-10k-test-postgres-1.c.gitlab-qa-10k-cd77c7.internal auth_user=pgbouncer dbname=gitlabhq_production

praefect_production =

praefect_production_direct = dbname=praefect_production pool_mode=session

The same also applies when trying to set up a separate database cluster where you effectively need to "hack" the Consul postgresql watcher to be only for the other database by overriding settings such as node['consul']['internal']['postgresql_service_name'] and what looks like node['gitlab']['gitlab-rails']['db_database'] to get it over the line. But the same issue noted above is also at play here as it seems only the database named in node['gitlab']['gitlab-rails']['db_database'] will ever get updated so other connections like the second Praefect cache connection are ignored.

Edited Aug 15, 2023 by DJ Mountney