This is a most excellent place for technology news and articles.
It was a 17 year old kid from India; naive and foolish but I don't think it was malicious. His post says the audio books were an idea to try and pay for the high bandwidth usage - 120k users in 24 hours, serving over a terabyte of data.
I think he probably had good intentions, but clearly this was a bad idea and of course to many people it looked like someone was trying to commercially exploit the work of many thousands of fanfic authors, plus feeding in to the AI slop fest.
It seems like this person wanted to help with AO3 being down in their own way, but somehow decided that scraping the internet for fanworks and hosting them on a new site without asking permission from any of the creators was a good idea.
It seems similar to those people who will dump a 40k LoC Pull Request on an open source project that they vibe coded. Wanting to help, but misunderstanding every step, or like, how to interact with an established group. The article does say that the guy who made the vibe coded site is 17, so I don't wanna be too hard on him, but maybe he should have reached out to OTW to see if there was anything he could do to help first.
[the website creator] scraped the internet for works to populate the site using “a swarm of AI Agents controlling 40 virtual machines were scraping the internet for all drives, scrapes, and sources,”
Also this kind of stuff is wild because a lot of websites aren't really built to handle the insane traffic from bot crawlers.
decided that scraping the internet for fanworks and hosting them on a new site without asking permission from any of the creators was a good idea
The thing is, in any fetish and fanfic community there will be a body of works accumulated over the years that contains many classics but the authors are only a nickname. Sometimes there's an email listed that stopped working decades ago.
The sites that host the works are usually run by enthusiasts. Over the years they lose the incentive to maintain them. Whenever a site goes down and others take its place there's always a dilemma about what to do with that body of work.
Technically speaking, yes, any site that's not the site where each author originally uploaded their work is in breach of copyright and/or licensing for those works. On the other hand, if you were to attempt to contact each author and only carry forward the works where you get an answer and permission... only a tiny fraction would qualify, and massive portions of the literature would be lost in each move.
Thats a good point. Things do go down forever and its nice to have backups to preserve history. Though in this case it seems a bit premature because OTW and AO3 are just down afaik, not like, gone gone.
Exactly, there is every intention of them coming back up again. It seems at most like their servers got hacked, and at least that they need (to schedule) maintenance to keep things running smoothly.
If they were gone forever, or there was no published information on what is going on, I imagine much fewer authors would have an issue with their creations being carried on (in good faith) on a new site.
I hate to say this, but we should attack AI, the technology that made this possible.
No, the bigger story is the theft of the stories. Something like this could have been done with any off the shelf CMS, like WordPress.
A03 is an open source platform, you could host an almost perfect clone
true... oh wait, is the code on github or some other VCS? 😄
despite this, the thief decided to roll out their own, AI coded clone? Pathetic search skills!
And it’s clear they have like no infra knowledge. They’re talking about how expensive it is. I’m almost sure they spun up a vercel app because they have no idea what they’re doing and the model is just running patterns. The intelligent thing to do to meet the need for what is almost 100% static content, would have been to get a hetzner dedicated box which gives you unlimited traffic at 1gbps, then pay for cloudflare pro and institute hard long running caching rules, then the vast majority of the server bandwidth and requests come off of the box and hit cloudflare’s edge, which generally allows for unlimited caching.
Or, y’know don’t build this at all since it’s stealing user work. But vibecoders with no engineering experience don’t think that way, they’re all grifters
Damn. Don’t give away the blueprint!
Hetzner you say? Not digitalocean (I know, the Omarchy link) or Vultr?
Cloudflare edge is a fine use of this, though even basic nginx level caching can handle this. Heck, if it’s vercel they can still use some caching in front of it to cool off the use.
But… most coders rarely have infra knowledge, let alone hit-and-run vibe coders, who have neither.
The issue with any cloud/vps provider is they usually have a max monthly bandwidth. Hetzner dedicated (not cloud) usually gives you unlimited (reasonable) bandwidth at 1Gbps
And yeah the point of the cache is less about the burn on the box of the request but more the capped bandwidth at the machine level, whereas content being served from the cdn can scale to a shit ton of users and only comes from your box once within the cache expiry time
Yeah but isn’t the max monthly something absurd? Like 1 TB? I know it’s probably not scalable, but this person is just shitting money anyway at this point, so that would be better. No?
Yeah just reading the vibecoders apology notes they posted as part of my recco:
It had 126,592 unique users within 24 hours of going up, and had terabytes of bandwidth in usage.
So if they’re blowing a TB or more a day they’d pay out the ass for raw bandwidth at per MB rates past the initial 1TB from cloud providers, and a single box would have trouble keeping up. Cloudflare doesn’t bill separately for bandwidth, and since hetzner gives you unlimited for dedicated, it would have room to seed the cloudflare cache and then cloudflare could serve 100s of TB of bandwidth per month.
All in all if set up correctly you could go with a hetzner ax41, and cloudflare businees, for a total cost of ~$270 per month
“Terabytes”??
Oof. Someone miscalculated what they were getting out of this venture!
lol yes, it’s also a 17 year old kid from India, as stated there, so while I can’t blame the kid for faults in his infra, I can blame him for thinking that scraping people’s work and putting it online without their permission and a monetization vector was a bad idea. That being said, it seems like he did the right thing, apologized, took it down, and I hope he carries the lesson forward to consider the human impact in what he does going forward
I bet he’s gonna get an offer from some scummy company like Byju soon.
Good on him for taking it down. Good lesson learned. Hope it didn’t drown him in debt!
Not surprising, it's been proven that AI use destroys people's research skills.
