Selfhosted

62167 readers

371 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

587

Based on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlers (jlai.lu)

submitted 7 months ago by hylobates@jlai.lu to c/selfhosted@lemmy.world

79 comments fedilink hide all child comments

I really hope they die soon, this is unbearable…

you are viewing a single comment's thread
view the rest of the comments

[–] punrca@piefed.world 27 points 7 months ago (8 children)

It's best to use either Cloudflare (best IMO) or Anubis.

If you don't want any AI bots, then you can setup Anubis (open source; requires JavaScript to be enabled by the end user): https://github.com/TecharoHQ/anubis
Cloudflare automatically setups robots.txt file to block "AI crawlers" (but you can setup to allow "AI search" for better SEO). Eg: https://blog.cloudflare.com/control-content-use-for-ai-training/#putting-up-a-guardrail-with-cloudflares-managed-robots-txt

Cloudflare also has an option of "AI labyrinth" to serve maze of fake data to AI bots who don't respect robots.txt file.

[–] AHemlocksLie@lemmy.zip 17 points 7 months ago (3 children)

Pretty sure I've repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much?

[–] Sv443@sh.itjust.works 9 points 7 months ago (1 children)

Like a lock on a door, it stops the vast majority but can't do shit about the actual professional bad guys

[–] FreedomAdvocate 2 points 7 months ago

Cloudflare definitely can and does stop the vast majority of actual professional bad guys.

load more comments (1 replies)

load more comments (5 replies)