Why did it take them an entire day to fix "a single DNS race condition"?
this post was submitted on 23 Oct 2025
22 points (100.0% liked)
Technology
4460 readers
626 users here now
Which posts fit here?
Anything that is at least tangentially connected to the technology, social media platforms, informational technologies and tech policy.
Post guidelines
[Opinion] prefix
Opinion (op-ed) articles must use [Opinion] prefix before the title.
Rules
1. English only
Title and associated content has to be in English.
2. Use original link
Post URL should be the original link to the article (even if paywalled) and archived copies left in the body. It allows avoiding duplicate posts when cross-posting.
3. Respectful communication
All communication has to be respectful of differing opinions, viewpoints, and experiences.
4. Inclusivity
Everyone is welcome here regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
5. Ad hominem attacks
Any kind of personal attacks are expressly forbidden. If you can't argue your position without attacking a person's character, you already lost the argument.
6. Off-topic tangents
Stay on topic. Keep it relevant.
7. Instance rules may apply
If something is not covered by community rules, but are against lemmy.zip instance rules, they will be enforced.
Companion communities
!globalnews@lemmy.zip
!interestingshare@lemmy.zip
Icon attribution | Banner attribution
If someone is interested in moderating this community, message @brikox@lemmy.zip.
founded 2 years ago
MODERATORS
As with most IT troubleshooting,
Time spent applying the fix: 15 minutes.
Time spent identifying the problem then discovering where it is in the system: 15 hours.
There's a full postmortem from AWS. One piece that stands out to me:
due to the large number of droplets, efforts to establish new droplet leases took long enough that the work could not be completed before they timed out. Additional work was queued to reattempt establishing the droplet lease. At this point, DWFM had entered a state of congestive collapse and was unable to make forward progress in recovering droplet leases.
That is, the load that resulted from the initial failure was not something the system was designed to handle, so it had cascading effects / required manual cleanup.
Because the "them" in your sentence is a rapidly decreasing number of professionals. https://lemmy.zip/post/51501102
"Professional": Alexa, should I randomly reconfigure the DNS using a Magic 8 Ball?
Alexa: "Wow what a brilliant idea! You're so smart!"