I have to stop posting active links to our content on Mastodon.
-
I have to stop posting active links to our content on Mastodon.
Every time I do so now, it brings down our website for up to 5 minutes.
We've tried pretty much every claimed fix, including third party caching (which in turn breaks other elements of our website's dynamic display abilities), code changes and such on our back end code, and more stuff I don't understand at all (but have spent money paying our WP developer to implement). None of it has worked.
The #fediverse powers that be need to fix this growing problem of the #MastoDDos effect on websites. The more followers and more servers your followers are from, the more impact this has on literally bringing a website to its knees with all the DB calls.
For instance, this morning, I posted the lovely article our creative writer Ethan wrote, which ended up only getting 2 boosts and one "favourite" here, but it brought down our website for 4 minutes and 12 seconds.
That's not sustainable.
-
@coffeegeek Could @Gargron implement/reccommend a solution for your issue Coffeegeek? Is it a known issue with the mastodon development team?
-
It's a long time known problem. Supposedly some fix was in the works as far back as May, but it still happens to this day, so I don't think any fix has taken effect.
(sorry to the register for posting this, I hope it doesn't bring down their site).
Mastodon delays firm fix to solve link preview DDoS
Decentralization is great until everyone wants to grab data from your web server
(www.theregister.com)
-
@coffeegeek hopefully a fix happens soon, but until then I’ll just resort to checking the website regularly like some sort of barbarian.
Seriously though, love the content and I’m happy to enjoy it any way that I can. -
@coffeegeek Hi,
Wouldn't it be acceptable, as a workaround, to post the links in an intentionally broken form that interested humans can instantly fix, e.g.:
https ://example.com/foo/bar (remove the space)
? This way, no DDOS due to the Fediverse servers, yet interested readers can easily copy & paste the thing to their browser address bar, remove the space and read the article. :blobcat:
-
roddie digital :n64:replied to Blobster last edited by
@blobster @coffeegeek I think the simpler solution than that is to manually post an image along with the link so that the preview card is never generated? Obviously that doesn't stop someone else from posting your links normally though
-
@coffeegeek @ronzegers @Gargron Me: Trying to decide if I'm supposed to boost or not
-
@coffeegeek @jacob I am sorry the fediverse’s architectural flaws are hitting you like this. I am curious though, how much traffic is it? I have a pathetic single-core web server that handles these apparently devastating load spikes without an observable impact on a near-0 load average. Granted, I am hosting a static site, and I realize DB-backed dynamic content complicates things, but if it is taking your site offline for minutes at a time for 4k hits that feels like a misconfiguration
-
@glyph >Granted, I am hosting a static site, and I realize DB-backed dynamic content complicates things
Yeah, the requirements are as different as night and day.
-
CoffeeGeekreplied to Apicultor 🐝 last edited by [email protected]
Truth. Our site serves about 520K uniques a month, 1.6 million page views. But the huge >1sec hit everytime we post to Mastodon (with all the different servers following my account) brings the site to its knees.
My main thing is this: I've probably spent about $3,500 since Spring trying to fix this on our end (bills to our WP developer, time spent). I can't spend any more money on what appears to be a problem on the mastodon side.
Also to be razor clear - the downtime on our end is also including the time our system
a) identifies the issue
b) can't keep up with the initial traffic hits,
c) takes everything offline and reboots automatically.The MastoDDos part of it probably lasts 60 seconds or less.
-
@BewilderedBeast @ronzegers @Gargron
I think Youtube is purposely blocking cards frm displaying, at least on some federation instances, for the exact same reason. Here's what I see when you posted this - a generic 'youtube" card. Clicking the card also just brings me to the front page of youtube. If i click the embedded link, I can go to the video.
-
CoffeeGeekreplied to roddie digital :n64: last edited by
When someone else posts a link to my website, we usually don't get hammered, unless that person has like 5K-10K or more followers across a ton of servers. If they just have 500-1K, across fewer servers, it doesn't impact our website.
There are other times our site has gone down (I get an auto-text within 1min of it happening), and almost every time I've noticed it, it is because some high-follow account on Mastodon posted a link to our site. (I do a quick masto search to see what's been posted).
-
@blobster One thing I might be able to do (I have to test this) is post a photo AND the URL; the photo will display, and hopefully the URL won't try to create a card.
-
@coffeegeek @ronzegers @Gargron it wasn't working for me when I first posted it either, but it seems to be now; at least for me. Maybe someone boosted it and broke Youtube?
-
@coffeegeek This may not be what you want, but JWZ seems to have working countermeasures: https://www.jwz.org/blog/2024/10/mastodon-stampede-returns/
-
@doty We looked into some of those previous fixes, didn't work for us, though it's good to know that we'll have to update those fixes because of changes by Mastodon for the user agent.
-
@BewilderedBeast @ronzegers @Gargron
No, this is an ongoing problem with Youtube links; I first saw it back in the early summer, and created several threads about the issue. At the time, it appeared to be limited to some of the higher-traffic instances.
-
@coffeegeek Hm, I've noticed that my blog seems to get about 600-ish hits shortly after being posted here, but nothing close to the volume required to actually cause downtime. My understanding of Mastodon is very weak, but what kind of numbers hit you when a post gets dropped?
-
@coffeegeek Hey - don't know if you're looking for assistance with this problem, but my business is providing WordPress sysadmin support for organizations and publishers. In other words, dealing with this exact kind of headache
It sounds like this issue is hitting you harder than it should be, which sucks.
I'd be happy to get on a Zoom or something sometime, if you'd like to bounce what you've tried so far off someone else to get a sanity check. Totally pro bono, no commitments. Sometimes it's just handy to get a fresh set of eyes on a problem.
If you're interested, ping me. And if not, no worries!
(cc @glyph)
-
@ludicity @coffeegeek it'll depend on how distributed your followers are, on first approximation, since you'll get a hit per server (10K followers on 1 server = 1 hit, 1K followers on 1K servers = 1K hits). Of course if the link gets boosted and reshaped, it'll reach more servers and the number of hits will grow. For static pages it's generally sustainable, dynamic ones will want to cache this heavily.
