alright, after like a year of halfheartedly trying on and off, #FetchAllReplies is pretty much finished - the problem of not being able to see all replies to a post is one of the largest complaints that people have with mastodon in particular but also ...
-
alright, after like a year of halfheartedly trying on and off, #FetchAllReplies is pretty much finished - the problem of not being able to see all replies to a post is one of the largest complaints that people have with mastodon in particular but also the fedi in general. It is an especially potent problem for smaller servers, making them feel lonely, and making the whole fedi seem quiet. It is also a large contributor to the 'reply guy' problem where a moderately popular post will get the same replies over and over again and people won't even know they're doing it.
This patch recursively fetches replies using activitypub collections. it does it respectfully, only when someone is explicitly looking at a post (rather than fetching all replies for everything all the time) with some debounce, and spaces out the recursive calls to the other servers in deep threads.
the only thing left is to make the posts get inserted into the web client as they are received, currently you need to refresh to see them.
trying it locally now and it is a game changer.
i'm not "good at ruby" so if you ever wanna see this upstream, kindly spare a code review?
Fetch All Replies v2 - Service Edition by sneakers-the-rat · Pull Request #44 · NeuromatchAcademy/mastodon
Fix: #43 More discussion/description at #43 Current status: the core of the feature is complete, but we need to restructure it a bit for the frontend and to mitigate one abuse scenario TODO: Rat...
GitHub (github.com)
#FediDev #MastoDev #UnFuckTheFedi #PubSubIsCoolButPresentsPrettySeriousUsabilityProblems #JustSmallInstanceThings
-
:PUA: Shlee fucked around andreplied to jonny (good kind) last edited by
@jonny you love to see it.
-
@jonny you'd probably want to push them across to the streaming server for that to happen
-
@thisismissem one of the major parts of the program i have yet to explore. i'll take a look, but from cursory glance it doesn't seem too hard taking a break for some food now and then will continue on that.
-
@jonny that or just return a header when the replies collection is out of date & refresh (poll) regularly.
-
@thisismissem @jonny streaming seems better because then other frontends would get them too?
-
@thisismissem is there a method you would prefer? I am not seeing a streaming endpoint for any generic DetailedStatus feed, maybe the Direct feed is closest? if polling would be preferable to adding a new stream/adapting another feed then i could figure that out, i'm just never sure what the most efficient way to do things with redux is
-
kouhai, Breaker of Cachesreplied to jonny (good kind) last edited by
@jonny @thisismissem some of this feels like “it depends”. polling will be chattier, especially without connection reuse via H2/etc. websockets will make state management after disconnect a minor issue complexity wise
note that depending on implementation, polling may also incur the expense of a round trip vs half trip
-
@kouhai
@thisismissem
I am thinking it might be nice to just make a streaming endpoint for when one is in a thread both for this and in general. On a given instance, the thread ancestors are always known (I think?) From the threadresolveservice, or at least wouldnt mutate in the client (?). A new status can similarly always resolve its thread root on the server. So if there was a streaming feed for every root post, that might be a reasonably efficient way to do it, so eg. Multiple viewers with different points in a broad and deep thread could all sub to the same feed, and every new post has exactly one feed it would need to be added to.Since the context endpoint is unpaginated and you always get the full parent tree when you mount a thread view from any point in a thread, I think you could mount and unmount the stream w/relatively low complexity in the client? Gonna give it a shot
-
Jonreplied to :PUA: Shlee fucked around and last edited by
yeah really, very cool work @[email protected] !
@[email protected] -
@jonny well yeah, there's no functionality in streaming for updates to replies on a single status so both approaches are about as "good" — iirc your intention isn't to upstream this?
-
@jonny @kouhai that'd probably put a lot of extra load on the streaming server as you'd need to check authorization as you navigate into and out of the channels per thread (e.g., checking blocks/mutes, checking thread visibility, checking the streaming user is within the set membership for the thread posters' followers, etc)
Currently all that authorization happens in rails pretty much beside the public feeds which only need blocks & mutes checked.
-
@thisismissem
@kouhai
Im restructuring it rn and im finding that patching into the distribution worker is letting me basically reuse everything there, ill perf test it once this restructuring is done -
-
@thisismissem
@kouhai
I was thinking of just making a new kind of redis channeltimelines:thread:{thread_root.id}
and copying/reusing whats already in place for feed streams, but ill check that out tomorrow (I gotta sleep rn) -
Emelia 👸🏻replied to jonny (good kind) last edited by [email protected]
@jonny @kouhai right, the problem is that when a client subscribes to that topic, you need to assert all the ACL policies to check “can you actually subscribe to this topic”
That's potentially expensive and complex, since we don't currently have anything like that in streaming at the moment.
I'm not saying I'd be against streaming supporting updates for a given thread, but we need to make sure it enforces all the same authorisation logic that exists in rails.
-
kouhai, Breaker of Cachesreplied to Emelia 👸🏻 last edited by
@thisismissem @jonny okay, so the solution that I was thinking:
- websocket event channel that exists solely to notify of an update to public / unlisted posts
reducing the scope to public/unlisted also reduces the ACL overhead to "is this incoming reply public/unlisted?”
not sure what's ideal for private/followers visibility thread roots, though; perhaps that could be a later patch
-
@kouhai @jonny right, so the complexity of public/unlisted is the same as for private/followers to a decent degree. (followers does have an additional check via the database required)
You need to load the status from the database for the thread, and then check it's visibility (also keep in mind visibility can change mid-thread)
-
kouhai, Breaker of Cachesreplied to Emelia 👸🏻 last edited by
@thisismissem @jonny still waking up, but i was thinking of doing this near the distribution worker, actually. the uh, fan out service, i think?
so maybe it looks like:
"ThreadUpdates:#{PublicThreadRoot}:Public" -> public replies to thread with given PublicThreadRoot
"ThreadUpdates:#{PublicThreadRoot}:Unlisted” -> unlisted replies to thread with given PublicThreadRoot. may want to require “auth'd user for this instance” (not familiar with streaming)
"ThreadUpdates:#{PublicThreadRoot}:OutOfBand” -> management messages, i.e. a Resubscribe payload, and a Close payload (if that’s necessary?)but admittedly, yeah. this is starting to look less appealing versus polling. alas!
-
Erin Kissanereplied to jonny (good kind) last edited by
@jonny YAY thank you for all this work, fair winds and following seas for the last stretch of dev