Selfhosted

62103 readers

386 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

285

Anubis - Weighs the soul of incoming HTTP requests using proof-of-work to stop AI crawlers (github.com)

submitted 2 years ago* (last edited 2 years ago) by zutto@lemmy.fedi.zutto.fi to c/selfhosted@lemmy.world

68 comments fedilink hide all child comments

I just started using this myself, seems pretty great so far!

Clearly doesn't stop all AI crawlers, but a significantly large chunk of them.

you are viewing a single comment's thread
view the rest of the comments

[–] randomblock1@lemmy.world 24 points 2 years ago (1 children)

Why Sha256? Literally every processor has a crypto accelerator and will easily pass. And datacenter servers have beefy server CPUs. This is only effective against no-JS scrapers.

[–] poVoq@slrpnk.net 21 points 2 years ago* (last edited 2 years ago) (1 children)

It requires a bunch of browser features that non-user browsers don't have, and the proof-of-work part is like the least relevant piece in this that only gets invoked once a week or so to generate a unique cookie.

I sometimes have the feeling that as soon as some crypto-currency related features are mentioned people shut off part of their brain. Either because they hate crypto-currencies or because crypto-currency scammers have trained them to only look at some technical implementation details and fail to see the larger picture that they are being scammed.

[–] swelter_spark@reddthat.com 2 points 2 years ago (1 children)

So if you try to access a website using this technology via terminal, what happens? The connection fails?

[–] crmsnbleyd@sopuli.xyz 3 points 2 years ago (1 children)

If your browser doesn't have a Mozilla user agent (I.e. like chrome or Firefox) it will pass directly. Most AI crawlers use these user agents to pretend to be human users

[–] swelter_spark@reddthat.com 1 points 2 years ago (1 children)

What I'm thinking about is more that in Linux, it's common to access URLs directly from the terminal for various purposes, instead of using a browser.

[–] crmsnbleyd@sopuli.xyz 1 points 2 years ago

If you're talking about something like curl, that also uses its own User agent unless asked to impersonate some other UA. If not, then maybe I can't help.