this morning in #hachyderm infra admin - building a new postgresql box to run our secondary database.

Esk 🐌⚡💜

this morning in #hachyderm infra admin - building a new postgresql box to run our secondary database. if things go badly on the primary, we can (manually) failover to the secondary.

well, hypothetically once this reindex operation finishes (SLOWWW) and we figure out replication...

:revblobfoxcomputerowo:

Emelia 👸🏻

@esk won't it actually become our primary in a bit, once we know HA & failover are working?

Esk 🐌⚡💜

@thisismissem I think that's what we should do! I'm actually a little worried this collation version mismatch might make replication not work. In that case what we might do is take a one-time outage: stop the site, full backup, restore to new server from that full backup, repoint site, decom old server.

Esk 🐌⚡💜

@thisismissem where "decom old server" really means "stop using postgres on old server"

Emelia 👸🏻

@esk yeah, that could be necessary, and then we setup new nodes to replace the previous db server, and one to rebalance the other work on that server more evenly