this post was submitted on 25 Apr 2026
44 points (97.8% liked)
Technology
42810 readers
222 users here now
A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.
Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
founded 4 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Only in trivial cases where the training data isn't being curated properly. There was a paper done on the subject a few years back where "model collapse" was demonstrated by repeatedly training generation after generation of models on the output of previous generations, and sure enough, the results were bad. This result gets paraded around every once in a while to "prove" that AI is doomed. However, in the real world this is not remotely close to how AI is actually trained. You can prevent model collapse simply by enriching the training data with good data - stuff that is already archived, that can't be "contaminated."
Indeed, the best models these days are trained largely on synthetic data - data that's been pre-processed by other AIs to turn it into stuff that makes for better training material. For example a textbook could be processed by an LLM to turn it into a conversation about the information in the textbook, with questions and answers, and the result is training data an AI that's better at understanding and talking about the content than if it was just fed the raw text.
This is a separate issue from the usual "model collapse" argument. I assume you're talking about stuff like Nightshade, which claim to put false patterns into images that cause AIs to miscategorize them. These techniques are also something that only works in a "toy" environment, these adversarial patterns are tailored to affect specific AIs and won't work on other AIs they weren't specifically designed for. So for example you might "poison" an image so that a classifier based on Dall-E would become confused by it, but a GPT-Image classifier wouldn't care. The most obvious illustration of this is the fact that humans are a separate lineage of image classifier and these "poisonings" have no effect on us.
There's also the added problem that these adversarial patterns tend to be fragile, they break if you resample the image to resize or crop it. Since that's usually a routine part of preparing training data for an image AI it may end up making the poison ineffective even for image AIs that it was designed for.
Essentially, all these things are just added background noise of the sort that AI training operations already have mechanisms for dealing with. But they make people feel better, I suppose.
This feels like an odd juxtaposition.
If model collapse can be avoided by enriching with uncontaminated data, and model collapse comes from using training data generated by previous generations, doesn't that imply that:
Model collapse comes from using only training data generated by previous generations.
All that's needed to avoid it is to add training data that isn't directly from the previous "generation" of the LLM in question. The thing that causes model collapse is the loss of data from generation to generation, so you just need to keep the training data "fresh" with stuff that wasn't directly generated by the earlier generation of your model.
You could do that with archived material you used for previous training runs. For more recent events you could do that with social media feeds. The Fediverse, for example, would probably be a perfectly fine source of new stuff. Sure, there's some AI-generated stuff mixed in, but that's not "poison."
As I mentioned, the article that demonstrated model collapse did it using a very artificial set of circumstances. It's not how real AI training is done.
It can't only be from data from previous generations, even if the initial demonstration used that, because that would mean a single piece of human-generated text is sufficient to avoid collapse.
The loss of data from generation to generation is one way model collapse can occur, but it's only one way. The actual issues that cause collapse are replication of errors and increasing data homogeneity. In a world where an unknown quantity of new data is AI generated, it is not possible to ensure only a certain quantity is used as future training data.
Additionally, as new human generated content is based on the information provided by AI, even if not used intentionally in the construction of the text itself, the error replication and data diversity issues cross over from being only an AI-generated content problem to an all content problem. You can see examples of this happening now in the media where a journalist relies on AI output to fact check, and then the article with the error gets republished by other media outlets.
Real AI training methods may stave off some model collapse, if we ignore existing issues around the cultural homogeneity of training data from across all time periods, or assume the models are sufficiently weighted to mitigate those issues, but it's by no means settled that collapse is a non-problem.
You've mentioned using data mixing to prevent collapse, but some of the research suggests that even iterative mixing isn't sufficient dependent on the quantities of real vs synthetic data. Strong Model Collapse (2024), Dohmatob, Feng, Subramonian, Kempe goes into that, and since then there's been When Models Don’t Collapse: On the Consistency of Iterative MLE (2025) Barzilai, Shamir which presents one theoretical case where collapse won't occur provided some assumptions hold, but the math is beyond me. They also note multiple situations where near-instant collapse can occur.
How much data poisoning might affect any of that is not at all clear, it would need to be in sufficient quantity for whatever model to have an effect, but it certainly wouldn't help. The recent Bixonimania scandal suggests it's feasible.
Alright, so instead of simply saying "include external data in your training run", extend that to "and also filter the data to exclude erroneous stuff." That's a routine part of curating training data in real-world AI training as well, I was already writing a lot so I didn't feel like adding more detail there would have enhanced it.
The basic point remains the same, that real world training accounts for the things that were necessary to force model collapse to happen in that old paper I linked. It's a solved problem. We can see that it's solved by the fact that AI models continue to get better, despite an increasing amount of AI-generated data being present in the world that training data is being drawn from. Indeed, most models these days use synthetic training data that is intentionally AI-generated.
A lot of people really want to believe that AI is going to just "go away" somehow, and this notion of model collapse is a convenient way to support that belief. So it's very persistent and makes for great clickbait. But it's just not so. If nothing else, the exact same training data that was used to create those earlier models is still around. AI models are never going to get worse than they are now because if they did get worse we'd just throw them out and go back to the earlier ones that worked better, perhaps re-training with the same data but better training techniques or model architectures.
Even if it logically followed that model improvement means model collapse is a solved problem, which it absolutely doesn't, even the premise that models are improving to a significant degree is up for debate.
Model collapse may for some people be an argument used to support a hope that AI will go away, but the reality of that hope does not alter the validity of the model collapse problem.
You can tell it's not a solved problem because researchers are still trying to quantify the risk and severity of collapse - as you can see even just from the abstracts in the links I provided.
Some choice excerpts from the abstracts, for those who don't want to click the links:
It's really interesting reading a conversion between somebody who knows what they're talking about, providing sources, and a known troll (FaceDeer) who can only go "nuh-uh" and complain about ghosts.
This single word made me stop reading your text, which started with a somewhat good point about model collapse. LLMs are not "understanding" anything, they're correlating tokens.
Apart from this, do you mind sharing a link to the studies about model collapse you mentioned had methodical errors?
Semantic quibbling is one of the least interesting kinds of internet debate, so replace the word "understanding" with whatever word makes you happy. I continued with "and talking about" right afterwards so you can just delete the word entirely and the sentence still works fine. You could have just kept reading.
Since you didn't read the rest of my comment, I should note that the rest of it after that sentence is about the other issue that OP raised and not even about model collapse at all.
Anyway. The article about model collapse that I see still crop up every once in a while is this one. It's not that it has "methodological errors", though, it's just that it uses a very artificial training protocol to illustrate model collapse that doesn't align with how LLMs are actually trained in real life. It's like demonstrating the effects of inbreeding in animals by crossing brothers and sisters for twenty generations straight - you'll almost certainly see some strong evidence, but it's not a pattern of breeding that you are actually going to see in the wild.
If I understand it right you need to enrich and filter data with human input so as not to collapse the model.
Wouldn't that imply if the human enrichment is emulating AI data too closely it will still collapse the model, since it's now just the human filtering that's mimicing AI data?
The main mechanism leading to model collapse in that paper, as I understand it, are the loss of "rare" elements in the training data as each generation of model omits things that just don't happen to be asked of it. Like, if the original training data has just one single line somewhere that says "birds are nice", but the first generation of model never happens to be asked what it thinks of birds then this bit of information won't be present in the second generation. Over time the training data becomes homogenized. It probably also picks up an increasing load of false or idiosyncratic bits of information that were hallucinated and get reinforced due to random happenstance, it's been a long time since I read the article and details slip my mind.
I'm really not seeing how human filtering would mimic this process, so I think it's safe. The filtering is being done with intent in that case, not due to random drift as is done with a purely automated generation like was done in the paper.
Why do you engage in it then?
In my opinion, a debate about the semantics of understanding and intelligence in context of AI is highly interesting, and a huge issue for worldwide politics and policies, but you do you.
Facedeer pretends to be above the thing he's doing because he's a pretty well-known troll who believes nothing and will say opposite statements just to promote AI...
We don't even have a good definition for what "understanding" actually means. It's like the word "intelligence" - there are dozens of dictionary definitions.
I find it pretty ridiculous to dismiss a long, well-thought-out piece of writing in its entirety just because one word was used in a way you don't like. Even if you disagree with how they used the term, you most likely still understand what they meant by it. LLMs aren't generally intelligent, but they're also not as dumb as people make them out to be. There's clearly real information processing happening in the background that produces accurate answers way more often than pure chance would allow.
Iconoclast, aren't you the guy who believes the teachings of that abusive AI cult leader Eli Yudkowsky?
Because if you are, there's some "thoughtful writings" you should purge from your beliefs.