alright, after like a year of halfheartedly trying on and off, #FetchAllReplies is pretty much finished - the problem of not being able to see all replies to a post is one of the largest complaints that people have with mastodon in particular but also ...
-
@thisismissem
@kouhai
I was thinking of just making a new kind of redis channeltimelines:thread:{thread_root.id}
and copying/reusing whats already in place for feed streams, but ill check that out tomorrow (I gotta sleep rn) -
Emelia 👸🏻replied to jonny (good kind) last edited by [email protected]
@jonny @kouhai right, the problem is that when a client subscribes to that topic, you need to assert all the ACL policies to check “can you actually subscribe to this topic”
That's potentially expensive and complex, since we don't currently have anything like that in streaming at the moment.
I'm not saying I'd be against streaming supporting updates for a given thread, but we need to make sure it enforces all the same authorisation logic that exists in rails.
-
kouhai, Breaker of Cachesreplied to Emelia 👸🏻 last edited by
@thisismissem @jonny okay, so the solution that I was thinking:
- websocket event channel that exists solely to notify of an update to public / unlisted posts
reducing the scope to public/unlisted also reduces the ACL overhead to "is this incoming reply public/unlisted?”
not sure what's ideal for private/followers visibility thread roots, though; perhaps that could be a later patch
-
@kouhai @jonny right, so the complexity of public/unlisted is the same as for private/followers to a decent degree. (followers does have an additional check via the database required)
You need to load the status from the database for the thread, and then check it's visibility (also keep in mind visibility can change mid-thread)
-
kouhai, Breaker of Cachesreplied to Emelia 👸🏻 last edited by
@thisismissem @jonny still waking up, but i was thinking of doing this near the distribution worker, actually. the uh, fan out service, i think?
so maybe it looks like:
"ThreadUpdates:#{PublicThreadRoot}:Public" -> public replies to thread with given PublicThreadRoot
"ThreadUpdates:#{PublicThreadRoot}:Unlisted” -> unlisted replies to thread with given PublicThreadRoot. may want to require “auth'd user for this instance” (not familiar with streaming)
"ThreadUpdates:#{PublicThreadRoot}:OutOfBand” -> management messages, i.e. a Resubscribe payload, and a Close payload (if that’s necessary?)but admittedly, yeah. this is starting to look less appealing versus polling. alas!
-
Erin Kissanereplied to jonny (good kind) last edited by
@jonny YAY thank you for all this work, fair winds and following seas for the last stretch of dev
-
-
kouhai, Breaker of Cachesreplied to Emelia 👸🏻 last edited by
@thisismissem @jonny yeah. alas! my efficient websocket push
-
@kouhai
@thisismissem
Thanks for thinking through all this complexity yall, you probably saved me several days of bafflement here. For curiosity's sake I still want to see what would be different from eg. The public feeds, where we need to do what seems like similar filtering for blocks, etc. But we can all subscribe to it, bur it sounds like polling will have to be the way.I think I'll have to make a separate worked after all to prevent a maliciously crafted infinitely expanding thread tree with global recursion limits anyway, so polling might be good there too so we can have a progress indicator rather than just popping posts in as they come
-
kouhai, Breaker of Cachesreplied to jonny (good kind) last edited by
@jonny @thisismissem fairness: should there be a per-user throttle so that we don’t have someone opening 100000 tabs and causing a bunch of work
(I’d argue that this endpoint may need to be authgated for that reason, to make it easier to throttle)
-
@jonny @kouhai yeah, the UI side would certainly need to "buffer" new posts.
I think having the ability to subscribe to a thread in the streaming server would be good, but it has some complexities in the authorization layer which make it potentially difficult.
With https://github.com/mastodon/mastodon/pull/30965 we do lift some database interactions up outside of the message handling loop, and https://github.com/mastodon/mastodon/pull/30978 goes further here.
-
@kouhai
@thisismissem
Definitely yes, it's already only done for logged in users, and I assume there are already throttles for that? But if not yes definitely it should be one thread expansion per account at a time -
@jonny @kouhai Basically you look up the filters/blocks/mutes data on connect and refresh when rails notifies of changes, and then you're just looking up and comparing in-memory data.
For Threads, we'd need to look up the thread on connection and keep a mapping of thread -> author ID, which can then be tied into the blocks/mutes lookups as messages are processed.
We'd also need to have events for follower changes possibly.
-
-
-
kouhai, Breaker of Cachesreplied to Emelia 👸🏻 last edited by
@thisismissem @jonny to clarify, are we talking about the polling approach, or the websocket?
-
Stefan Bohacekreplied to jonny (good kind) last edited by
@jonny Amazing, thank you for working on this!
-