this morning in #hachyderm infra admin - the secondary postgres setup continues!

Esk 🐌⚡💜

yesterday we:
- tested db backup restores multiple times
- got streaming replication going
- confirmed that streaming replication isn't putting undue stress on our primary

today:
- configuring prometheus scrapes
- mayyyyybe configure mastodon to use the read replica as a secondary on a small % of services to test it out
- mayyybe look at automating edge node standup -- i hear a rumor linode supports deb12 now

Renaud Chaput

@esk If that helps you getting confidence in it, we have been using a read-only replica for mastodon.social for more than a year, and it *really* helps (it handles 50% of the requests). Since we improved replica support in 4.2, we have not seen any issues related to this.

Esk 🐌⚡💜

@renchap awesome! my main hesitation was that our initial replication config was quite slow -- with up to 50 second lag time. after some changes, we've got it down to sub-second which seems much more friendly for a read replica.

we may try flipping to use the REPLICA_* settings as soon as this weekend!

Scaling up your server - Mastodon documentation

(docs.joinmastodon.org)

Emelia 👸🏻

@esk @renchap iirc, read replica is most useful for Mastodon Web and Streaming, right?

(Though Streaming will soon need write)

Renaud Chaput

@thisismissem @esk Some things in sidekiq also use the replica. We looked at the most expensive and frequent SQL queries that do not *require* the latest data, and manually flagged them to run on the replica.
Also iirc Rails disables the replica if the lag is greater than something like 10 seconds