Legion739

See the "monitoring" namespace

kube-prometheus-stack plus VictoriaLogs for logs, about 100 pods on 2 nodes.

The whole stack uses about 2.2 GB RAM. Prometheus takes about 1.1 GB of that and Grafana about 700 MB. CPU is basically idle, around 0.2 cores.

Prometheus has about 150k series and uses about 7 GB for 15 days of metrics, so roughly 14 GB a month. Logs are about 3 GB a month. Grafana itself is under 1 GB.

It's mostly the default exporters, so a smaller lab would need less. A 20 GB volume would be plenty.

Prometheus is set to retention: 15d and retentionSize: 18GB. Anything older than 15 days is dropped, and so is the oldest data if the total goes over 18 GB, whichever happens first. It's sitting at a steady ~7 GB now that it has a full 15 days stored.

Thanks!

I took a lot of inspiration from here https://github.com/onedr0p/home-ops, there is also a discord server on there where you can ask people for help, they are quite helpful.

I mostly shopped around and found parts and implementations I liked and coppied that and made it my own

Preemptively responding to this, because I just know there's going to be a comment about this.

Yes Yes I know, GitHub, screw GitHub, Screw Microslop. I am trying to move away from Big Tech as much as possible, but it's a marathon not a sprint, and every win should be celebrated.

Perfection is the enemy of good.
.

Grafana dashboard

You can find more details and the source code here: https://erasmus.works/

This might be a better angle.

We stand on the sholders of giants.

It's always good to give back to the community and open source it

I can't imagine how'd you'd link them up, without having to reach inside the case, and having it modular.

Ideally the zig zag isn't there, but did you see the before?

I have basically taken a power strip, and made each plug a module.

The inside of a power strip, like in this image, is in series.

mains → yellow → red → blue → green → blue(end)

If a join between yellow and red is broken, it knocks out red and everything after it.

If what is plugged into yellow is plugged out, or disconnects everything else keeps working.

The same is true for my setup, the joins between Yellow, red etc. are just on the outside of the case, so you can disconnect them if you want, which would kill the power to what is after it. But you don't have to do that.

You're explaining scenarios where you incorrectly use my setup.

You can wire the switches so that if switched off it only kills power to that server and nothing else in the chain. You then basically have this, but modular:

Yea if you rip out a cord that you shouldn't you're going to lose power, that is true of any power strip.

This has more components, so more things can fail, I will give you that, but that's unlikely and for me at least, worth the trade off in making it modular.

If the above picture was powering your servers, and you wanted to add a 7th plug, you'd have to unplug everything and buy a power adapter with more sockets, and deal with the rats nest of cables that follows, or add another of these that might have the same problem in the future.

Very nice, is that a KWS rack?

I don't think that existed yet when I started my journey, they seem very solid.

I went for aluminum for max strength and didn't trust 3D printed parts for strength, but after getting a printer, I see you can print fold out chairs that you can stand on, so they can be more than strong enough.

I don't think I understand your question.

If you plug all your servers into a power strip like this, all the power goes through the power strip and through one cable. If there is a fault in the power strip, all the servers will lose power as a fuse will trip.

The same is true for my setup, I have added no new failure modes.

The name has been in my family far longer than that program has existed, so the way I see it, if anything, they are copying me xD