this post was submitted on 19 Aug 2025
849 points (99.3% liked)

Technology

74247 readers
4204 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
L4s@hackingne.ws
849
The AI company Perplexity is complaining their bots can't bypass Cloudflare's firewall (www.searchenginejournal.com)
submitted 2 days ago* (last edited 2 days ago) by Davriellelouna@lemmy.world to c/technology@lemmy.world
233 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] Kissaki@feddit.org 102 points 1 day ago* (last edited 1 day ago) (3 children)

Perplexity argues that a platform’s inability to differentiate between helpful AI assistants and harmful bots causes misclassification of legitimate web traffic.

So, I assume Perplexity uses appropriate identifiable user-agent headers, to allow hosters to decide whether to serve them one way or another?

[–] ubergeek@lemmy.today 8 points 22 hours ago (1 children)

And I'm assuming if the robots.txt state their UserAgent isn't allowed to crawl, it obeys it, right? :P

[–] Kissaki@feddit.org 3 points 21 hours ago (1 children)

No, as per the article, their argumentation is that they are not web crawlers generating an index, they are user-action-triggered agents working live for the user.

[–] ubergeek@lemmy.today 2 points 20 hours ago

Except, it's not a live user hitting 10 sights all the same time, trying to crawl the entire site... Live users cannot do that.

That said, if my robots.txt forbids them from hitting my site, as a proxy, they obey that, right?

[–] lime@feddit.nu 36 points 1 day ago (1 children)

yeah it's almost like there as already a system for this in place

[–] seraphine@lemmy.blahaj.zone 1 points 4 hours ago (1 children)

THE CAKE DAY IS NOW. (i dont have an image at hand)

[–] lime@feddit.nu 1 points 4 hours ago* (last edited 4 hours ago) (1 children)

i really wish we wouldn't do those. feels too reddity.

but thanks.

[–] seraphine@lemmy.blahaj.zone 1 points 4 hours ago (1 children)

as you wish

[–] lime@feddit.nu 2 points 4 hours ago

*monkeys paw curls and i turn into cake*

[–] drmoose@lemmy.world -2 points 23 hours ago

Its not up to the hoster to decide whom to serve content. Web is intended to be user agent agnostic.