I shared a link from NPR Tiny Desk but there was no image preview or video embed. After investigating this I learned that when you do that, each instance picks up the link and individually retrieves the link preview nearly simultaneously. For gargantuan sites like YT or Reddit, this is a trivial amount of traffic and not a problem, but for smaller sites like NPR this results in a sort of botnet DDOS attack on the site's servers. More here. Another example here . For that reason, NPR has blocked Fediverse servers like Mastodon.

I have experienced this on my personal server as well (DDOS), but at the time didn't know why, exactly.

This isn't the end of the world, but certainly is not great and hurts engagement with the content. Is there a solution for this? Perhaps using some sort of webtorrent service like PeerTube where they just fetch the information from another fediverse server instead of the source server? Maybe fetching embeds client-side? At least that would stagger the requests.

E: Many people have replied comparing this to the Reddit "hug of death" or "The Slashdot effect". This is not the same at all. Those are legitimate site visits from humans that happen over the course of several hours. This is the result of several thousand servers all automatically fetching the same image simultaneously , and without any human involvement, and potentially next to zero human traffic to the site, which is why I compared it to a botnet or DDOS attack.

you are viewing a single comment's thread
view the rest of the comments

[–] axx@slrpnk.net 1 points 22 hours ago

Another option could be to have the originating Mastodon server provide the preview image in content-adressable way, and have a standard allowing websites to provide the same: if the hash of the file(s) provided by the first / a Mastodon server to another matches the one from the target website, no need to download from the target website. Only if they differ then you'd actual download the data from the target website.

Each Mastodon server from the federation could quickly decide whether to trust that preview data from the peer it gets the rest of the post from or not.

This is basically taking ideas from IPFS but applying it to just the preview data that ends up being downloaded over and over again. You could also make that part a blob of data stored directly in IPFS. But IPFS was pretty slow to resolve content, when I last tested.