I've had a sudden spike in bandwidth on my #Netlify account (that powers Cybercultural.com), and it's not correlated with an increase in traffic.
-
Sara Joy :happy_pepper:replied to Richard MacManus last edited by
@ricmac @bobmonsour didn't some guy get landed with a 100k bill after a bunch of people went nuts downloading an mp3 from his site?
-
Sara Joy :happy_pepper:replied to Sara Joy :happy_pepper: last edited by
@ricmac @bobmonsour also, I use Netlify a lot. Yeah, it does worry me a bit. I keep meaning to move stuff away and never do because it's so convenient...
-
Richard MacManusreplied to Sara Joy :happy_pepper: last edited by
@sarajw @bobmonsour I agree, and I generally like Netlify as a company, as they support the web. But this situation is not a good look for them, as large bills can easily be racked up — they automatically charge you for add-ons as each 100GB passes. I still don’t know what is causing the bandwidth spike.
-
Sara Joy :happy_pepper:replied to Richard MacManus last edited by
@ricmac @bobmonsour yeah I mean after that 104k debacle I was pretty bothered - guess I pushed it to the back of my mind. I have so many little random sites online on Netlify like, I guess any one of them could go wild...
-
Sara Joy :happy_pepper:replied to Sara Joy :happy_pepper: last edited by
@ricmac @bobmonsour do you have any big interesting files on there which people might be downloading?
-
Richard MacManusreplied to Sara Joy :happy_pepper: last edited by
@sarajw @bobmonsour No, it’s just text and images. It started yesterday when I published my latest weekly post (Cybercultural.com), so I did wonder if it might have anything to do with the Buttondown emails I send out (which use images on the server)…but 56GB is a hell of a lot of bandwidth, so I find the email scenario hard to believe. It has to be a DOS or bot attack…
-
Martin Grubingerreplied to Richard MacManus last edited by
@ricmac @sarajw @bobmonsour a while ago I read about posting links to Mastodon can cause traffic spikes because every instance of your followers goes and fetches the content. Not sure if that's what's happening, but maybe something to consider.
I only found this link now https://natenom.de/2022/05/auswirkungen-von-dezentralitaet-mastodon-fediverse-traffic-und-last-fuer-webserver/ -
Szymon Nowickireplied to Sara Joy :happy_pepper: last edited by
@sarajw @ricmac @bobmonsour had the same dilemma and moved to cloudflare pages / workers. Tbh it’s much better overall.
-
Szymon Nowickireplied to Szymon Nowicki last edited by
@sarajw @ricmac @bobmonsour also had fear that one day I will wake up with a million euro invoice in AWS and finally moved everything to a mix of cloudflare, Hetzner and my own server.
I sleep better now knowing that worst case they will shut down my service instead of happily bill be fortune.
-
Sara Joy :happy_pepper:replied to Szymon Nowicki last edited by
@hey @ricmac @bobmonsour the biggest thing holding me in Netlify right now is the very handy spam-filtered Forms handling. I guess I can do that in cloudflare with a worker and key-value storage...
-
Szymon Nowickireplied to Sara Joy :happy_pepper: last edited by
@sarajw @ricmac @bobmonsour it should do with their plugins like hCaptcha.
I like in Pages that they nicely integrated workers into it so we don't have to choose one or another any more
Create a HTML form | Cloudflare Pages docs
In this tutorial, you will create a simple <form> using plain HTML and CSS and deploy it to Cloudflare Pages. While doing so, you will learn about some of the HTML form attributes and how to collect submitted data within a Worker.
Cloudflare Docs (developers.cloudflare.com)
Back when kukei.eu was still on CF Pages I managed to make entire dynamic server side rendering there without too much trouble
GitHub - Kukei-eu/kukei-web at 1809f56f528a6da7e379495cf79a2b3e1e1d0190
Contribute to Kukei-eu/kukei-web development by creating an account on GitHub.
GitHub (github.com)
Also did some notes if you're interested
Kukei.eu
Notes about how I started a new project - a curated search engine for web developers. How it started and how it's going...
(nowicki.io)
-
Sara Joy :happy_pepper:replied to Szymon Nowicki last edited by
@hey @ricmac @bobmonsour I try to avoid captchas. I can do tricks and honeypots and such - but Netlify also passes them through akismet which is cool.
Oooh thank you so much for the other tips though, will get reading...
-
Szymon Nowickireplied to Sara Joy :happy_pepper: last edited by
@sarajw @ricmac @bobmonsour cloudflare has very roboust solutions too. Their captcha usually don’t even show up. In my case I also refuse talking to http v1 which filters out most robots traffic right away.
-
Richard MacManusreplied to Szymon Nowicki last edited by
@hey @sarajw @bobmonsour @simoncox @phil @mcg @mjgardner I have an update for everyone following this thread: Netlify did respond overnight my time, and firstly they said they will refund the $55 charge (yay!). Also they said it *was* a single user agent that was responsible for the bandwidth spike. I've attached the UA screenshot they sent: Mozilla/5.0, which is a UA stripped of OS data. I've done a quick google and apparently this is probably a bot of some kind. Will dig more shortly.
-
Richard MacManusreplied to Richard MacManus last edited by
@hey @sarajw @bobmonsour @simoncox @phil @mcg @mjgardner This Reddit thread was helpful, tho inconclusive: https://www.reddit.com/r/webdev/comments/15tz2gn/mozilla50_user_agent_without_os_in_string/
-
Szymon Nowickireplied to Richard MacManus last edited by
@ricmac @sarajw @bobmonsour @simoncox @phil @mcg @mjgardner i say it’s a systematic problem that Netlify doesn’t address properly. Bots and bad actors will periodically cause traffic spikes for many reasons. If they charge for traffic they should ensure it’s a valid one.
-
Richard MacManusreplied to Szymon Nowicki last edited by
@hey @sarajw @bobmonsour @simoncox @phil @mcg @mjgardner Absolutely agree. Users should never be charged for traffic spikes caused by nefarious actors. You can make a case it’s fair if the traffic is genuine (eg from a Hacker News frontpage), although even that is dubious — I’d prefer the site goes offline if you don’t have enough bandwidth, as happened in days of ordinary web servers
-
Simon Cox :SEO:replied to Richard MacManus last edited by
@ricmac @hey @sarajw @bobmonsour @phil @mcg
@mjgardnerI note you are not blocking AI crawlers with your robots txt - if you want to this might be useful:
https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txtWill not stop the content thieves though.
-
Richard MacManusreplied to Simon Cox :SEO: last edited by
@simoncox @hey @sarajw @bobmonsour @phil @mcg @mjgardner Yes, up till now I had decided not to block AI bots, as I figured it may be useful from SEO perspective (ranking etc) to be in Google AI Overviews, Perplexity, etc. But I will review that position now.