I've noticed that the issues above are becoming increasingly notorious across the entirety of the Fediverse. What's being done to mititage those issues?

you are viewing a single comment's thread
view the rest of the comments

[–] CombatWombat@feddit.online 15 points 20 hours ago* (last edited 20 hours ago) (2 children)

I feel pretty confident, despite a complete lack of evidence, that at least one state actor has had a listener running on the fediverse continuously since the w3c started publishing specs, and I would be surprised if the big llm providers like Anthropic and OpenAI don't run them as well -- they certainly have the resources and motivation to develop them. You're certainly correct that the vast majority of scrapers are attempting to harvest historical data using the web frontend, but those are the scrapers I am least afraid of and I think as a mental model for the average user "assume every post is scraped" is the best stance.

[–] irelephant@lemmy.dbzer0.com 2 points 1 hour ago

You are right: https://www.404media.co/the-200-sites-an-ice-surveillance-contractor-is-monitoring/

The fediverse and atproto are both easily scraped.

[–] frongt@lemmy.zip 2 points 3 hours ago (2 children)

I don't think Anthropic or OpenAI have spent the time developing a custom ingest pipeline for such a small dataset. It doesn't seem like it'd give much enough of a return on investment.

[–] CombatWombat@feddit.online 1 points 1 hour ago* (last edited 1 hour ago) (1 children)

I dunno, we had 1.8 billion posts and 50 million comments from 1.1 million MAUs in June according to the fediverse observer. It's not nothing.

[–] frongt@lemmy.zip 1 points 28 minutes ago

Yeah, for them that's small potatoes.

[–] cynar@lemmy.world 1 points 3 hours ago

Given that they are scrabbling around like drug addicts looking for anything they've split, including checking the cracks in the floorboards...

For some models, it's obvious they've long scrapped the erotic fan fic sites!