it is a really annoying oversight that misskey just kinda overloads itself handling federation stuff because it tries to process it at once as if that's not an i/o intensive operation (thousands of http requests is pretty i/o intensive yeah)

Amber (deilannist)

lily (evil era)

@[email protected] what's the better way to do it? with a queue?

Amber (deilannist)

@[email protected] im no expert (this is a joke i have a formal background in software development) but i think we could start by not dispatching thousands of DELETE activities at once, basically only stopping because the operating system we're running under has locked up.

Amber (deilannist)

@[email protected] specifically, the thing that most people notice off the bat is the ui. if the front end is not responsive they will notice the instance as "unresponsive". it doesn't matter what the federation traffic is doing. we have all of these rate limits for the front end to keep it responsive and prevent things like the media proxy from exploding the instance (don't ask) because someone is making hundreds of requests to an endpoint that's not cached and requires some server sided processing. with that in mind... we don't really have the same sort of thing when it comes to federation traffic. so when a post of mine gets boosted out in the world, and instances fetch it... all of the sudden i have opened a flood gate and the entire instance is overloaded in traffic.

Amber (deilannist)

@[email protected] a better system would be "responsive" to the load that the server is currently under, and beginning to prioritize things like the front end over federation traffic. maybe, instead of serving the object to everybody who asks we start rolling out rate limits. now, this sucks because in the case i specified above it's mostly a bunch of instances that are making a single request. just slap them with a 429. a 503 or something that just tells other software the instance is overloaded and can't take requests right now. I don't think 429 is proper for that (because 429 in my opinion should be scoped for "YOU HAVE MADE TOO MANY REQUESTS" not "THERE ARE TOO MANY REQUESTS FROM EVERYWHERE" but at the same time... it does make sense to have a global rate limit for something like /inbox). next, in regards to outbound activities... we could queue them yes.

Amber (deilannist)

@[email protected] if you queue outbound activities you can get into some cool shenanigans. the dreadful "activity collection" where you send an instance a collection of activities and they just kinda have to process all of those. it would actually help a lot with bandwidth and cpu (because think about it. one http request with a collection of 100 activities vs 100 http requests with 1 activity). idk. there's tons of optimizations there to make it less bad️ but nobody really has put the effort in (and i am including myself).