@renchap awesome! my main hesitation was that our initial replication config was quite slow -- with up to 50 second lag time. after some changes, we've got it down to sub-second which seems much more friendly for a read replica.

we may try flipping to use the REPLICA_* settings as soon as this weekend!

Scaling up your server - Mastodon documentation

(docs.joinmastodon.org)

this morning in #hachyderm infra admin - the secondary postgres setup continues!

yesterday we:
- tested db backup restores multiple times
- got streaming replication going
- confirmed that streaming replication isn't putting undue stress on our primary

today:
- configuring prometheus scrapes
- mayyyyybe configure mastodon to use the read replica as a secondary on a small % of services to test it out
- mayyybe look at automating edge node standup -- i hear a rumor linode supports deb12 now

@thisismissem where "decom old server" really means "stop using postgres on old server"

@thisismissem I think that's what we should do! I'm actually a little worried this collation version mismatch might make replication not work. In that case what we might do is take a one-time outage: stop the site, full backup, restore to new server from that full backup, repoint site, decom old server.

this morning in #hachyderm infra admin - building a new postgresql box to run our secondary database. if things go badly on the primary, we can (manually) failover to the secondary.

well, hypothetically once this reindex operation finishes (SLOWWW) and we figure out replication...

:revblobfoxcomputerowo:

@thisismissem @dbsmasher win-win:

i didn't drop prod prod

AND

the restore worked!

@renchap @timetinytim absolutely! we are doing a few database-related side quests before flipping to `pgcat` in prod, but we currently have https://hachyderm.wtf (test instance) on it and it seems happy! :blobfoxscience:

really liking pgcat as a replacement for pgbouncer. maybe coming to a #hachyderm database near you soon.

GitHub - postgresml/pgcat: PostgreSQL pooler with sharding, load balancing and failover support.

PostgreSQL pooler with sharding, load balancing and failover support. - postgresml/pgcat

GitHub (github.com)

AUR (en) - pgcat

(aur.archlinux.org)

@aburka they're generally ephemeral - maybe a couple of weeks at most. The good thing is that new announcements will show at the top and put a little "new item" dot on the announcements icon.

But I'll ask the mod folks what the plan is with that announcement.

@aburka we will be here for everyone as long as it's physically possible. we have many ideas. :blobfoxscience:

@aburka yup, minus the edge nodes which are in various geos.

Hey, Hachyderm - with recent US election results and the policy changes it implies, we have been thinking deeply here on the #hachyderm infrastructure team about how to better position Hachy to keep our users safe.

We'll share a strategy blog post soon that has a set of thematic improvements we can make to further harden our position from both a security and long-term service continuity perspective.

In the meantime (and after!), we welcome any questions you have.

:blobfoxninja:

#hachyinfra

Howdy, @tailscale! I'm a Hachyderm infra admin. We use #tailscale to stitch things together across clouds to make Hachyderm work. With yesterday's election results, in the event some kind of law were to be enacted restricting LGBTQIA+ content in the US, would we be in breach of the https://tailscale.com/terms and have our service terminated?

If yes, what are our self-hosting or alternative options?

#hachyinfra

@renchap they seem to be the full trace - no parent/children that I can find (granted, i'm a big honeycomb noob).

we plan to turn on sampling for things here shortly to make the total number of traces less... overwhelming.

@renchap poking at the new #otel support in 4.3.0, and there's a bunch of neat data here. yay!

is there any value in keeping these repeated ZSCAN traces? they're the vast majority of our current event usage, and they don't seem to be terribly meaningful. usually (maybe exclusively) just a span that says ZSCAN.

safe to drop these? i don't think we'll lose any meaningful observability by doing so?

#hachyderm #hachyinfra #MastoInfra #opentelemetry

experimenting with the new #opentelemetry (otel) capabilities in mastodon 4.3.0.

we're just getting started, but getting some super neat visibility by Just Sending Stuff to @honeycombio!

#hachyderm #hachyinfra

@renchap @ClearlyClaire great, thanks for the extra eyes. missed that section when we did our upgrade!

@renchap @ClearlyClaire got it. so we install that, point it at our redis, and... yay?

@renchap @ClearlyClaire with the deprecation of statsd metrics for sidekiq in 4.3.0, in the new otel setup, how can we calculate sidekiq queue depth/backpressure? it's our key scaling metric, and we (hachy) have zero visibility into it now.

docs are quite unclear beyond the fact that statsd is gone and otel is here. guidance much appreciated :blobfoxheartcute: