I have to stop posting active links to our content on Mastodon.
-
@coffeegeek @jacob I am sorry the fediverse’s architectural flaws are hitting you like this. I am curious though, how much traffic is it? I have a pathetic single-core web server that handles these apparently devastating load spikes without an observable impact on a near-0 load average. Granted, I am hosting a static site, and I realize DB-backed dynamic content complicates things, but if it is taking your site offline for minutes at a time for 4k hits that feels like a misconfiguration
-
@glyph >Granted, I am hosting a static site, and I realize DB-backed dynamic content complicates things
Yeah, the requirements are as different as night and day.
-
CoffeeGeekreplied to Apicultor 🐝 last edited by [email protected]
Truth. Our site serves about 520K uniques a month, 1.6 million page views. But the huge >1sec hit everytime we post to Mastodon (with all the different servers following my account) brings the site to its knees.
My main thing is this: I've probably spent about $3,500 since Spring trying to fix this on our end (bills to our WP developer, time spent). I can't spend any more money on what appears to be a problem on the mastodon side.
Also to be razor clear - the downtime on our end is also including the time our system
a) identifies the issue
b) can't keep up with the initial traffic hits,
c) takes everything offline and reboots automatically.The MastoDDos part of it probably lasts 60 seconds or less.
-
@BewilderedBeast @ronzegers @Gargron
I think Youtube is purposely blocking cards frm displaying, at least on some federation instances, for the exact same reason. Here's what I see when you posted this - a generic 'youtube" card. Clicking the card also just brings me to the front page of youtube. If i click the embedded link, I can go to the video.
-
CoffeeGeekreplied to roddie digital :n64: last edited by
When someone else posts a link to my website, we usually don't get hammered, unless that person has like 5K-10K or more followers across a ton of servers. If they just have 500-1K, across fewer servers, it doesn't impact our website.
There are other times our site has gone down (I get an auto-text within 1min of it happening), and almost every time I've noticed it, it is because some high-follow account on Mastodon posted a link to our site. (I do a quick masto search to see what's been posted).
-
@blobster One thing I might be able to do (I have to test this) is post a photo AND the URL; the photo will display, and hopefully the URL won't try to create a card.
-
@coffeegeek @ronzegers @Gargron it wasn't working for me when I first posted it either, but it seems to be now; at least for me. Maybe someone boosted it and broke Youtube?
-
@coffeegeek This may not be what you want, but JWZ seems to have working countermeasures: https://www.jwz.org/blog/2024/10/mastodon-stampede-returns/
-
@doty We looked into some of those previous fixes, didn't work for us, though it's good to know that we'll have to update those fixes because of changes by Mastodon for the user agent.
-
@BewilderedBeast @ronzegers @Gargron
No, this is an ongoing problem with Youtube links; I first saw it back in the early summer, and created several threads about the issue. At the time, it appeared to be limited to some of the higher-traffic instances.
-
@coffeegeek Hm, I've noticed that my blog seems to get about 600-ish hits shortly after being posted here, but nothing close to the volume required to actually cause downtime. My understanding of Mastodon is very weak, but what kind of numbers hit you when a post gets dropped?
-
@coffeegeek Hey - don't know if you're looking for assistance with this problem, but my business is providing WordPress sysadmin support for organizations and publishers. In other words, dealing with this exact kind of headache
It sounds like this issue is hitting you harder than it should be, which sucks.
I'd be happy to get on a Zoom or something sometime, if you'd like to bounce what you've tried so far off someone else to get a sanity check. Totally pro bono, no commitments. Sometimes it's just handy to get a fresh set of eyes on a problem.
If you're interested, ping me. And if not, no worries!
(cc @glyph)
-
@ludicity @coffeegeek it'll depend on how distributed your followers are, on first approximation, since you'll get a hit per server (10K followers on 1 server = 1 hit, 1K followers on 1K servers = 1K hits). Of course if the link gets boosted and reshaped, it'll reach more servers and the number of hits will grow. For static pages it's generally sustainable, dynamic ones will want to cache this heavily.
-
@ludicity @coffeegeek I wonder if it would make sense to have a stripped down version of the page with only the information needed by the card on it (open graph and Twitter tags, oembed instructions) to be served to social media crawlers.
-
I suggest using Cloudflare. It's free and works miracles. It will not break any site elements, a lot of the big sites use their service. It also protects against attacks. You will have to change the DNS settings so that CF is your provider.
-
@normis We do. But we cannot use their full caching. It messes up all the dynamic content delivery and images on our website. Trust me, we've tried.
The problem remains how Mastodon does thousands and thousands of card calls within a few milliseconds everytime a URL is posted. They need to fix that.
-
@oblomov @ludicity We cannot do that. The amount of work that would require, not only mentioning how it would completely destroy our efforts at SEO (love it or hate it).
The root problem isn't how complex our site or pages is. It's that everytime a high follower-count account on Mastodon, with followers across thousands of instances, posts a URL, that website gets hammerred with thousands of page call requests within a few milliseconds.
