this post was submitted on 24 Aug 2026
42 points (100.0% liked)

GenZedong

5286 readers
41 users here now

This is a Dengist community in favor of Bashar al-Assad with no information that can lead to the arrest of Hillary Clinton, our fellow liberal and queen. This community is not ironic. We are Marxists-Leninists.

See this GitHub page for a collection of sources about socialism, imperialism, and other relevant topics.

This community is for posts about Marxism and geopolitics (including shitposts to some extent). Serious posts can be posted here or in /c/GenZhou. Reactionary or ultra-leftist cringe posts belong in /c/shitreactionariessay or /c/shitultrassay respectively.

We have a Matrix homeserver and a Matrix space. See this thread for more information. If you believe the server may be down, check the status on status.elara.ws.

Rules:

founded 5 years ago
MODERATORS
 

Welcome again to everybody. Make yourselves at home. In the time-honoured tradition of our group, here is the weekly discussion thread.

Matrix homeserver and space
Theory discussion group on /c/theory@lemmygrad.ml
Find theory on ProleWiki, marxists.org, Anna's Archive

^image\ from^ ^this\ article^

you are viewing a single comment's thread
view the rest of the comments
[–] Darkerseid@lemmygrad.ml 3 points 3 weeks ago (1 children)

a critique of anubis on why it basically achieves almost nothing to prevent bot scraping

https://fzakaria.com/2026/07/09/who-does-anubis-actually-stop

[–] o_d@lemmygrad.ml 7 points 3 weeks ago* (last edited 3 weeks ago)

Anubis is a bandage on the much larger issue of AI scrapers that no longer respect robots.txt. Anubis exists because hosts with limited resources have no other option to prevent these big AI hyperscalers from stealing their resources, both hardware and content.

As the article points out, its fairly trivial now for these scrapers to bypass Anubis intentionally, however Anubis still blocks a large amount of bot traffic which is more indiscriminately scraping everything it can. A kernel maintainer wrote a piece on this recently. Their perspective is that the responsible way to scrape this data would be to clone the repo and scrape it locally, instead of attempting to load every commit, pull request, etc. via the web interface which is much less efficient.