Migration out of Equinix metal (tentative date: March 16, 2025) (#2076) · Issues · freedesktop.org / freedesktop · GitLab (original) (raw)

Skip to content

Migration out of Equinix metal (tentative date: March 16, 2025)

TL;DR (please read this first)

Host gitlab.freedesktop.org
    HostName ssh.gitlab.freedesktop.org

As you know we need to migrate the cluster to a new server.

This is a tracking issue of what is happening and what needs to happen.

Pre-migration:

Actual migration:

Gitlab

registry

s3.freedesktop.org

bots

ci-runners

ci-stats

Indico

# archive-indico-web-0

EQUINIX_IP="1.2.3.4"
EQUINIX_POD=/var/lib/kubelet/pods/__RANDOM_NUMBERS__/volumes/kubernetes.io~csi

EQUINIX_ARCHIVE_PVC=pvc-sth-sth
HETZNER_ARCHIVE_PVC=pvc-sth-sth-else

rsync -e 'ssh -p 2222' -rvA --progress \
  root@$EQUINIX_IP:${EQUINIX_POD}/${EQUINIX_ARCHIVE_PVC}/mount/ \
  ${HETZNER_ARCHIVE_PVC}/mount

# customization-indico-web-0

EQUINIX_CUSTOMIZATION_PVC=pvc-sth-sth
HETZNER_CUSTOMIZATION_PVC=pvc-sth-sth-else

rsync -e 'ssh -p 2222' -rvA --progress \
  root@$EQUINIX_IP:${EQUINIX_POD}/${EQUINIX_CUSTOMIZATION_PVC}/mount/ \
  ${HETZNER_CUSTOMIZATION_PVC}/mount


### old server:
# drop all access to the db
kubectl -n indico scale sts --replicas 0 indico-web

kubectl -n indico exec -ti indico-db-postgresql-0 -- bash
PGPASSWORD=$POSTGRES_PASSWORD pg_dump -U postgres \
  -Fc indico > /bitnami/postgresql/indico_dump.sql

kubectl -n indico cp indico-db-postgresql-0:/bitnami/postgresql/indico_dump.sql \
  /tmp/indico_dump.sql

### new server:
# drop all access to the db
kubectl -n indico scale sts --replicas 0 indico-web

kubectl -n indico cp /tmp/indico_dump.sql \
  indico-db-postgresql-0:/bitnami/postgresql/indico_dump.sql

kubectl -n indico exec -ti indico-db-postgresql-0 -- bash
PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U postgres \
  -d postgres -c -C /bitnami/postgresql/indico_dump.sql

kubectl -n indico scale sts --replicas 1 indico-web

Details

deploy a new k8s cluster on Hetzner cloud

Make use of Hetzner-k3s. This takes care of most of the k3s setup, so that's one less thing to worry about.

Topology (subject to changes):

Notes

Question: some of those configs has more traffic than others, should we have better control plane nodes (and colocate the db on those)? Colocation is bad, and the Load-balancer will absorb the incoming traffic, so no (maybe we can downscale an use 3 CCX13 instead).

Our current cluster is using a private LAN for kubernetes, with the control plane also on the private LAN. The external access for admins is handled thanks to Wireguard through kilo.

We need to check if we can replicate that, i.e. not exposing the :6443 k8s API to the world with hetzner-k3s.

We can expose only the internal private network, though in the config file, when updating it we need to export the current IP of the person deploying it. So that means:

We don't need to setup a dual stack cluster. The Load Balancer is dual stack and that is enough to provide an external IPv6 address to the cluster.

We can connect dedicated HW and cloud VMs with vswitch (or kilo).

prepare the connection from Faslty (CDN) to Hetzner

Question: do we need static objects external storage if we also use fastly in front of gitlab itself (I would say no).

migrate the S3 data from Equinix to Hetzner S3 + Fastly S3

deploy a test gitlab instance on the new cluster

deploy a test indico instance on the new cluster

deploy a test S3-proxy with OPA for replacing s3.freedesktop.org

TBD, but we need S3-proxy with a sidecar with OPA so we can keep s3.freedesktop.org running

shutdown of gitlab @ Equinix

back up everything (and push the backups on Hetzner S3) with gitlab backup tool

back up kubernetes secrets

restore everything on Hetzner

deduplicate db (ci+rest)

connect Faslty (CDN) to Hetzner

re-enable everything

Cc: @daniels, @whot, @colinmarc, @TomGudman

Edited Apr 15, 2025 by Daniel Stone