this post was submitted on 14 Nov 2026
60 points (92.9% liked)

Selfhosted

63895 readers
381 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

  1. Be civil.

  2. No spam.

  3. Posts are to be related to self-hosting.

  4. Don't duplicate the full text of your blog or readme if you're providing a link.

  5. Submission headline should match the article title.

  6. No trolling.

  7. Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.

  8. AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago
MODERATORS
 

folks,i know this is OT but i decided to post here as i'm sure there are a lot of people here that work in IT and know what thay are talking about. The recent MS failure is what i would like your opinion on.

from what I've seen, MS declared that a bunch of servers went offline and the load had to be spread on remaining servers. They fail to identify what caused the first bunch of servers encountering issues and they claim it was mot related to a hacker attack (why would they, it would lead to massive panic and distrust). So, then what caused the issue in the first place? Could it have been a piloted denail of service attack?

I'm sure the truth will emerge in coming days but was interested in the more tech savy people's opinione. In the meantime i rejoyce for not having to depend of third party infrastructure at home, thanks to my little homelab.

A further consideration is that this does once again prove that companies that put all their eggs in one basket, are at yhe mercy of big tech. i experienced this personally at works where comunication by Teams and Outlooks was pretty much halted for hours.

top 47 comments
sorted by: hot top controversial new old
[–] BennyInc@feddit.org 90 points 3 days ago (4 children)
[–] ozymandias117@lemmy.world 5 points 2 days ago

You can tell it's fake, because the EVP understood what was going on

[–] trilobite@lemmy.ml 2 points 2 days ago

Brillant! If only it were true and confirmed. MS stock prices would plumet ...

[–] zorflieg@lemmy.world 7 points 3 days ago

That gave me a good chuckle, thanks!

[–] lambalicious@lemmy.sdf.org 58 points 3 days ago (1 children)

fire 20% of their personnel

slopify

things fail

Sounds like a closed case to me.

[–] MajorMajormajormajor@lemmy.ca 18 points 3 days ago (1 children)

Maybe the real treasure was the stock price growth we made along the way?

[–] talkingpumpkin@lemmy.world 26 points 3 days ago* (last edited 3 days ago)

By 2030 AI is gonna completely replace sysadmins as the cause of outages.

[–] autonomous@lemmy.world 29 points 3 days ago (1 children)

I'm sure it all came down to poor upper management decisions which they will try to blame on technology

[–] trilobite@lemmy.ml 3 points 2 days ago

Or on the poor lower level man that forgot to follow procedures ...

[–] Treczoks@lemmy.world 17 points 3 days ago

The main reason for problems: Bad IT administration that relies way too much on Microsoft.

[–] Spite_By_Design_LLC@lemmy.dbzer0.com 10 points 3 days ago* (last edited 3 days ago)

I blame Jensen's leather jacket. It's clearly way too powerful.

EDIT: There's...a 3rd mistress now? Giggity.

[–] IsoKiero@sopuli.xyz 12 points 3 days ago (2 children)

There's been rumours about insufficient capacity on Microsoft cloud services around Europe for a while. Unofficial of course, and who knows how many "broken phones" has been on the chain before me, so don't take that as a fact. However, Microsoft stated that the problem started when they increased capacity, so maybe some initial synchronization or load balancing starting up caused extra load on the servers already running on pretty much full load and caused cascading failure.

Of course I don't know for sure and I don't expect we'll get a full technical brief with details and the main reason is management problems no matter what's the technical cause. I hope at least someone wearing a suit starts to think that maybe cloud isn't really the answer to everything. And it's not just 36~~5~~4, a shit ton of different cloud services relies on AWS, Cloudflare and other players and if any of those goes down they'll take considerable chunk of business critical infrastructure with them.

[–] False@lemmy.world 6 points 3 days ago* (last edited 3 days ago)

Have heard similar rumor about capacity. And yes, MS seems to be especially bad about chasing AI at the expense of their core products.

[–] Kirk@startrek.website 0 points 3 days ago (2 children)

I haven't heard those rumors and a quick search didn't turn up anything, do you have any links?

[–] HK65@sopuli.xyz 1 points 1 day ago* (last edited 1 day ago)

I can't give you anything but "trust me bro" either, but I'm responsible for a couple million in Google Cloud spend, and they're full out on all European regions, and they tell us so is AWS and Azure.

Right now if you need a gig of RAM, you need to mash the API for days until it finds capacity in most European datacenters. Rep called us and told us we'll only get what we reserve 2-3 months in advance essentially.

Black Friday is coming up, and there is also all the AI stuff, my bet is that up until now it was mostly a GPU and VRAM crunch, but now with Muse and Instinct it also needs a lot of traditional compute for the harnesses.

[–] IsoKiero@sopuli.xyz 5 points 3 days ago

No, I've heard about those either in person or via chat with coworkers and colleaques who have heard those from who knows where. That's why I underlined that they're only rumours and nothing more.

[–] slazer2au@lemmy.world 11 points 3 days ago (3 children)

I'm sure the truth will emerge in coming days but was interested in the more tech savy people's opinione.

Why speculate when you will get it dfrom the horses mouth in a couple days?

But my 2¢ is a bad code being deployed.

[–] corsicanguppy@lemmy.ca 6 points 3 days ago

a bad code

just one?

  • 'a code' - a password
  • 'code' (mass noun) - programming instructions, for example
[–] barnacles@lemmy.zip 3 points 3 days ago

Because the horse lies

[–] mcSlibinas@lemmy.world 4 points 3 days ago (1 children)

Upper management should do things and moves to avoid bad code deploying.

[–] slazer2au@lemmy.world 1 points 3 days ago (1 children)

Why should upper manglement have a say in this? They only thing any manglement should say is Yes to a digital twin for a test environment to catch crap like this before it becomes a global outage.

[–] mcSlibinas@lemmy.world 1 points 3 days ago

Preventing expensive accidents is a part of survival strategy. Should be interesting topic for any CEO and every level of management. In reality they just hope accidents won't happen?

[–] MonkderVierte@lemmy.zip 9 points 3 days ago* (last edited 3 days ago)

Something something health control on too much servers shut them down, which made the others overloaded. My money is on the health monitoring being vibecoded.

[–] kylian0087@lemmy.dbzer0.com 2 points 3 days ago (1 children)

I think it is a cascading effect some unmaintained and forgotten server gave out causing others to fall.

Thats what think not based on any facts.

[–] GreenKnight23@lemmy.world 5 points 2 days ago

I worked at a place that was running COBOL written in the 1970s that supported thousands of other services across the globe.

it was not redundant. it was not HA. there was just one.

the man who wrote the software died in 1995.

nobody understood what it did, or how to maintain it.

I would say, with the recent layoff from Microslop, your assessment might be more accurate than most think.

I doubt Microslop would pass up the opportunity to blame a hacker group or AI at this point. they look like a bunch of incompetent assholes. since they are being vague about it, and won't immediately state a root cause, that can only mean it's incompetence.

[–] magnue@lemmy.world 3 points 3 days ago

All I can say for sure, is that nobody knows for sure.

[–] Kirk@startrek.website 2 points 3 days ago* (last edited 3 days ago) (3 children)

this does once again prove that companies that put all their eggs in one basket, are at yhe mercy of big tech.

I am certainly no Microsoft lover, but 12-24hrs of downtime every few years seems like a highly reliable system compared to the cost of rolling your own software stack for the majority of companies.

If this event turns out to be a harbinger of what's to come I'll change my tune but I would be shocked if this single period of downtime led to a shift away from Outlook.

[–] Nollij@sopuli.xyz 8 points 3 days ago

There's a lot more downtime than that. It's just that most of them are limited in size or scope, and don't generate headlines. I'm pretty sure their service status page is public. Services are frequently degraded for a subset of users.

[–] corsicanguppy@lemmy.ca 8 points 3 days ago

but 12-24hrs of downtime every few years seems like a highly reliable system

  1. We don't call it o351 because it has '12-24hrs of downtime every few years'.
  2. In the enterprise space, the target is a few minutes each year.
[–] HubertManne@piefed.social 3 points 3 days ago (1 children)

rolling your own can have pretty small downtime. The biggest issue in the past was places did not want to spend on hardware that did not do anything but take over. Also disk failure was the most common big issue which is way lower with ssd's. it does not take a whole lot of spend to keep downtime away but so many places want to pinch every penny.

[–] Pringles@sopuli.xyz 2 points 2 days ago

We migrated to exchange online and I absolutely do not miss the exchange outages where I would be called awake in the middle of the night to start fixing that shit. I’m happy to have this fully outsourced to M365. My first reaction to this outage was “I’m so happy this is not my thing to deal with anymore.”

[–] Jozav@lemmy.world 2 points 3 days ago (1 children)

At the moment, we can only guess. My guess: an update (maybe of a load balancer) went wrong. We will never know for sure.

[–] trilobite@lemmy.ml 1 points 2 days ago

Maybe this is theproblem with big tech. They will never confess and what strikes me is that businesses and governments just won't wake up

[–] makeshift0546@lemmy.today -4 points 3 days ago* (last edited 3 days ago) (5 children)

I’m sure this day loss of productivity and not even a total outage will have everyone hiring teams to run their email. I’m sure it’ll be cheaper and definitely more reliable 🙄

Y’all really don’t understand how the money flows or how expensive it is to run this shit.

[–] Ooops@feddit.org 5 points 3 days ago

Y’all really don’t understand how the money flows

The money flows from Microsoft to schools and universities to condition them them to think MS is the only option and then from Microsoft to non-tech people in deciding positions as bribes and favours so they decide to pay MS a fortune for shitty SAAS. It's basically corruption used as an investment.

or how expensive it is to run this shit

Oh, you meant that side of the money... Yeah, not expensive at all. There is a reason everybody can get email for free or for very little money if you need advanced functions and active service. And there is a reason the big providers are now basically running with whitelists only, because every idiot and their mom can run their own service. There is a reason why simple office solutions you can also get for free are nowadays –like so many other things– porvided as SAAS: to make money off idiots and paid morons.

[–] Buck@jlai.lu 2 points 3 days ago

The focus is more about not all depending on the same third party, that has more and more outages as months pass.

[–] False@lemmy.world -1 points 3 days ago (1 children)

This community isn't really the place for pragmatic takes on this kind of stuff. It's largely homelab style hobbyists that have no idea about running anything for more than maybe 2-3 people (probably solo in most cases).

Unfortunately none of the actual professional communities from reddit seem to have active equivalents on Lemmy.

[–] Appoxo@lemmy.dbzer0.com 1 points 2 days ago

As if r/sysadmin has any actual professional admin actively posting?

[–] Kirk@startrek.website 0 points 3 days ago

Not to mention this doesn't really nudge their 99% uptime stat. Unless this kind of downtime event becomes a pattern then I find it hard to believe companies are going to start investing in their own software stacks especially when most of them couldn't afford it in the first place.

[–] StripedMonkey@lemmy.zip -1 points 3 days ago (2 children)

Do you really think a self hosting community has no idea how difficult and time consuming self hosting email is?

[–] False@lemmy.world 4 points 3 days ago (2 children)

Yes, this is basically a bunch of home lab enthusiasts. There's nothing wrong with that but doing it for even a small business of 50 people is extremely different in terms of requirements.

[–] StripedMonkey@lemmy.zip 1 points 3 days ago

Have you spent any time in these 'home lab enthusiasts' spaces? Nobody here is suggesting that you should go and self host email. Nobody is making it sound easy. In fact if you look at posts talking about email, almost every time the subject of email comes up you have a significant portion of the community saying it's outright not worth self hosting email. It's such a time consuming and opaque process to figure out why email isn't being sent or received that few do so.

[–] frongt@lemmy.zip 0 points 3 days ago

Yeah, any time someone says "I'm thinking about hosting for a business" my answer is "run away screaming". Or set up as a business yourself. Never ever provide for a business as an individual, especially not without being paid enough to insure yourself.

[–] makeshift0546@lemmy.today 1 points 2 days ago

100% yes. Absurdly common in tech