I'm not a fan of Anubis for various reasons but the idea that bot traffic in only harmful with dynamic pages must die. CPU (yes, even to serve static pages) is not free, bandwidth is definitely not free. There's an idea that serving a static page to a bot has a marginal cost tending to zero, but it's never really zero and serving them by millions definitely has a cost.

Also, while some pages may look like static HTML pages, they may be generated on the fly by an expensive and/or slow backend, which adds to the cost. I happen to maintain servers for academics and some content management systems are slow and have an expensive CPU cost. While it's OK for the low number of humans interested in the subjects they deal with, it's definitely not fit for massive bot scrapping. And before you ask, no, it's not always practical to have cache upfront or to pre-generate all pages.

Anubis is not free either, it is a matter of how much it costs to run Anubis vs to let bots in.

When I see LKML using Anubis when the pages it serves are tens of kB, all presumably static, I wonder if they do it out of spite rather than to protect their servers.

It’s done out of necessity, you can read about why here: https://people.kernel.org/monsieuricon/creepy-crawlies

As a user/reader/viewer I absolutely hate Anubis and usually turn around when I see it pop up (at least on my phone where it takes ages to compute), but with stats like that, I get why a site operator would resort to using it.

I think the kernel.org post proves the parent point rather than contradicts it.

> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.

14 CPU cores total for running a website like kernel.org is laughable. This is not worth burning cycles in Anubis on client's devices, this is not worth the time of the engineer who worked on it. Provisioning more hardware would have been literally better for everyone.

I can’t say I really disagree, and as a visitor of the site that’s the solution I would prefer.

> But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it.

This drives me crazy with so-called SOTA LLMs that have "achieved AGI".

Fable, Sol, Astra, will start by trying to reverse engineer a binary to figure out how something works when software is open source and one search query away.

You let them know it's open source, and they will start using github API instead of just cloning and grepping.

There are numerous services that will let you host static pages for free or nearly free. There are also numerous services that sit in front of your website that can block bots and reduce load on your origin server, many of which are also free, or very low cost relative to the service they provide.

The situation you are in is far less dire sounding when you consider that you have these options available to you.

Except that I don't have these options per employer policies.

So your employer is having a problem, and prevents you from using any of the available options to solve it? And you have asked them about all of them/told them about the problem? They don't like saving money?

Well, sounds like it's not your problem then.

> Also, while some pages may look like static HTML pages, they may be generated on the fly by an expensive and/or slow backend

Yes and that should be fixed before you subject real users to resource-wasting scripts.

Sadly, I do not control resource-wasting scripts running crawlers.

And to answer your intended demand: this is in some cases impossible or unreasonable. And things were working fine before LLM DDoS.

It can't always be fixed. Also, I don't use Anubis.

It's fixed enough for how much they want it to be fixed (I.e. how much they are being paid to fix it)

Please submit a fix to cgit then.

May I see it?

Shitty scrapers hitting images over and over again is also a problem. Not just /pic/1.webp - /pic/2.webp, but in practice I've seen three AI scrapers that would request things like /pic/1.webp?version=webp&foo=bar and other random query parameters that none of my domains even use. It's just stuck brute-forcing files or something, sucking up loads of bandwidth downloading the same image for no reason.

It also depends on if you want your content to be scraped. Any bot ignoring robots.txt should be considered potentially malicious and worth considering such mechanisms for.

> Shitty scrapers hitting images over and over again is also a problem. Not just /pic/1.webp - /pic/2.webp, but in practice I've seen three AI scrapers that would request things like /pic/1.webp?version=webp&foo=bar and other random query parameters that none of my domains even use

That's a trivially solved problem - have the web server refuse (or redirect if you're feeling nice) requests with unexpected query parameters. No reason to make things worse for real users before optimizing your site.

And even if there are actual parametric resources you want to serve, apply limits only to them and let users visit the rest without unnecessary bot checks.

Sure, there are loads of easy solutions to this problem. Putting everything behind Cloudflare is the most common one, Anubis and its competitors are another. I could also block all data centers, countries with loads of CGNAT + botnets like Brazil, China, and probably India, and refuse any user agent not in the top 5 modern user agents with a little TLS fingerprinting to make sure.

I could probably also analyze all the paths across my servers and write some bespoke Nginx config files to filter out all requests with illegal queries, but that's harder, more fragile, and just leaves the servers open for the next time a scraper finds a new way to screw me over.

The bots I'm fighting have gone for the nuclear option, relying on botnets and spoofing to bypass basic filtering, so I don't see why I shouldn't do the same.

It protects the content from being hoovered up into an LLM without credit or attribution. Anti-DOS is only half the story.

The anubis docs website is used as an example of how anubis works.

Also, if you are not using a big CDN (e.g. Cloudflare), you still need to pay for the traffic (even if you don't pay as much for the CPU). This is doubly true for smaller CDNs like Bunny.

Not only is it an example, it specifically is one of the first things I use to test experimental features so I get data back instantly.

Yeah it makes sense for dynamic content. But so far I have only seen it on blogs, which could have just been a html file.

Even for git hosts, there is no reason to add bot checks to e.g. the repository root or other common URLs that random real users land on.

some people don't want to be scrapped at all, and most bots ignore `robots.txt`

no, absolutely not, any malicious or dishonest bot traffic is harmful, generating unnecessary transfer and literally stealing data

> malicious or dishonest bot traffic is harmful

I only hear bad things about bots lately, are there any benevolent bots?

I've done a tiny bit of scraping myself (usually to archive old websites before they die), and I've gotten banned a few times even though I wasn't hammering anything.

Looks like it's getting harder and harder to do scraping for legitimate purposes. Although I suppose legitimate is subjective :) Everything becomes valuable when it's old enough...