Looks like there has been a fair bit of discussion about the architecture of Mastodon previews and “DDoS-ing” web sites:
-
Zach Leatherman :11ty:replied to Noah Liebman last edited by
-
Sara Joy :happy_pepper:replied to Zach Leatherman :11ty: last edited by
Who's that tootin' in the feeds?
It's two devrels and they're on fleek
Blogs and demos and typin' hands
What's your language? JS I am!Instances come to play, the server's not fast enough
They'd best stay away when the websites get a shoveToot, scoot, riot (riot!)
Leave a link and run away
Toot, scoot, riot (riot!)
No one visits my site anyway... -
William O'Connellreplied to Zach Leatherman :11ty: last edited by
@zachleat Yeah, even if all ~30,000 mastodon instances hit the server within 60 seconds (unlikely), that's still only 500 requests per second. That seems pretty manageable? At least for public content that's identical for every user. A real DDoS by an attacker can be *millions* of requests per second.
-
Noah Liebmanreplied to Sara Joy :happy_pepper: last edited by
-
Sara Joy :happy_pepper:replied to Noah Liebman last edited by
-
Zach Leatherman :11ty:replied to Sara Joy :happy_pepper: last edited by
-
Zach Leatherman :11ty:replied to William O'Connell last edited by
@williamoconnell yeah, that third blog post notes a hypothetical of 6.7k simultaneous requests—which seems like a lot!
-
Dustin Ruereplied to Zach Leatherman :11ty: last edited by
@zachleat this is _easily_ solved with a caching strategy which I have mentioned a few times to people who metion this issue. You can read about it at https://dustinrue.com/2023/02/avoiding-stampeding-mastodons/. It is not necessarily something only Cloudflare can solve, but can be used to inform people on how to fix the issue using any caching solution and some effort.
-
Zach Leatherman :11ty:replied to Dustin Rue last edited by
@dustinrue WordPress always needs one more thing to work like it should by default
-
Dustin Ruereplied to Zach Leatherman :11ty: last edited by
@zachleat Generally speaking I disagree. If the site was Next.js, Drupal or any other countless number of systems that generate content on the fly and isn't a static site will fall victim to this same issue.
-
Zach Leatherman :11ty:replied to Dustin Rue last edited by
@dustinrue the nuance I’d contribute is that (in my opinion) static sites are a better default for most web sites and dynamism should be an additive architectural layer.
Removes an entire class of problems for most folks
-
William O'Connellreplied to Zach Leatherman :11ty: last edited by
@zachleat It says they're supposed to be spread over 60 seconds though. In practice probably more, since it should take some time for all the instances to pick up the post in the first place. The author says "I'm yet to see that work for me" but I'm unclear of that means they have evidence that it isn't happening or their site is just still crashing regardless. The lack of specific data makes it hard to draw conclusions.
-
Emelia 👸🏻replied to Zach Leatherman :11ty: last edited by
@zachleat there's ongoing work to improve the situation
-
Zach Leatherman :11ty:replied to William O'Connell last edited by
@williamoconnell I’d assume that is just a timing issue between writing of the blog post and the rollout of the 60s “jitter” addition—a temporary mitigation for sure.
Mastodon *could* upload the preview as an asset sidecar’d with the post, similar to any image upload.
-
Zach Leatherman :11ty:replied to Emelia 👸🏻 last edited by
@thisismissem great!
For the record I’m mostly surprised that sites are being taken down by this level of concurrency
-
Flakireplied to Zach Leatherman :11ty: last edited by
@thisismissem would you happen to have a link to the issue/discussions perhaps, I'd be interested to see what's happening to try to address this?
@zachleat -
Zach Leatherman :11ty:replied to Flaki last edited by
@flaki @thisismissem https://github.com/mastodon/mastodon/issues/23662 is the one you want, I think!
-
John Hobbsreplied to Zach Leatherman :11ty: last edited by
@zachleat I would think with session reuse you wouldn't pay that TLS connection cost constantly. Also I imagine OpenSSL/BoringSSL has improved in the last seven years, and more cryptographic operations are done in CPU now then back then.
That said I don't think I'd run anything without a caching server in front anymore, unless it was very server interactive and would miss constantly. Varnish did amazing things at Flywheel.
-
Zach Leatherman :11ty:replied to John Hobbs last edited by
@jmhobbs great insight, thank you!
As a side note I think I’ve seen some of these caching defaults play out at the serverless platform level recently: https://www.zachleat.com/web/serverless-cost/ Some platforms are born from use cases that are intensely personalized, not heavily cached.