experimenting with the new #opentelemetry (otel) capabilities in mastodon 4.3.0.

Esk 🐌⚡💜

we're just getting started, but getting some super neat visibility by Just Sending Stuff to @honeycombio!

Emelia 👸🏻

@esk @honeycombio I'm really surprised by some of the stuff I'm finding in traces! (

Though the default configuration for the ruby redis collector is frustrating — just knowing it was a SET operation is kinda pointless, I wanna know the key at least, and maybe the expires options too. Feels like the masking isn't quite right because SET ? ? ? ? tells me nothing

Also wouldn't mind capturing call sites, though I'm guessing that's difficult in Ruby.

Emelia 👸🏻

@esk @honeycombio Would be fantastic to learn how to properly setup Mastodon's streaming server to produce OTEL data.. though I'm missing a lot of knowledge here, especially because it's redis pubsub where 1 ruby publish results in writes & database calls to N connected clients of streaming.

Not sure how to bring the gap (e.g., continue a context across the gap) nor how to instrument so we see both streaming connection types easily (event source & websockets)

Emelia 👸🏻

@esk @honeycombio am thinking it may have to wait until I've abstracted the eventsource connections out into a class that looks a bit like a websocket.

Currently it's all inlined which makes it hard to isolate & handle correctly when to start the spans