this post was submitted on 19 Aug 2025
853 points (99.3% liked)

Technology

74289 readers
4339 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
L4s@hackingne.ws
853
The AI company Perplexity is complaining their bots can't bypass Cloudflare's firewall (www.searchenginejournal.com)
submitted 2 days ago* (last edited 2 days ago) by Davriellelouna@lemmy.world to c/technology@lemmy.world
235 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] _cryptagion@lemmy.dbzer0.com 4 points 2 days ago (1 children)

Cheering for Cloudflare to be the arbiter of what technologies are allowed is incredibly short sighted. They exist to provide their clients with services, including bot mitigation.

Well I suppose it's a good thing then that the anti-AI shield is opt-in, and Cloudflare isn't making any decisions for anyone on whether or not AI scrapers get to visit their pages. That little bit of context makes your entire argument fall apart.

[–] FauxLiving@lemmy.world 0 points 2 days ago (2 children)

It isn’t opt in.

You can block all bot page scraping, and also block user initiated AI tools or you can block no traffic.

There isn’t an option to block bot page scraping but allow user initiated AI tools.

Because, as the article points out, Cloudflare is not able to distinguish between the two

[–] ubergeek@lemmy.today 2 points 2 days ago (1 children)

Thats not true, I just viewed my panel in CF, and Perplexity is an optional block, which by default is off.

[–] FauxLiving@lemmy.world 1 points 2 days ago (1 children)

They must be A/B testing a new feature then, it’s not on mine

[–] ubergeek@lemmy.today 3 points 2 days ago

Log into your dashboard, click "AI Audit", and you'll see the toggles.

[–] _cryptagion@lemmy.dbzer0.com 1 points 2 days ago (1 children)

There’s no appreciable difference on how they affect systems between the two for site owners.

[–] FauxLiving@lemmy.world 1 points 2 days ago

There’s a pretty significant difference in request rate. A tool trying to search and summarize will hit a search engine once, and each website maybe 5 times (if every search engine link points to the site).

A bot trying to scrape content from a website can generate thousands or tens of thousands of requests per second.