Technology

You're absolutely correct, 100% It's entirely scriptable. And I have done that work many, many times in the past 15 years

The interesting differences:

I didn't write the ansible jobs, it did to my specifications, i just had to review them. It can generate an Ansible job that would take me 30 minutes to properly write/document in less time than it takes me to review its output.

It knows how to do AWS AMI and RDS images without details. If Amazon changes something and it errors making the ami, it will, without intervention, find amazon docs and update it's process.

My playbooks are essentially Word documents (md tho) telling it to make an AMI backup and disable notifications using the monitor server who's name and IP are documented the neo4j. I don't tell it where the logs are, it knows. It'll fire up a Cloudtrail to hunt something down if it needs it.

The diffs on the health checks aren't binary. If it sees something it can't explain, it will go back further in time, inspect other nodes, even start going down the debugging route for the app, (as long as I give it docs on how it should debug). It doesn't miss a line. It doesn't blindly ignore things. I was doing a rebuild on a server the other day, and during the post-health check, it noted that a certain error on this node is present on 5 other nodes and that the most common cause is a minor config issue. I asked it to diagnose; it found a config file from the old image that we never touch. I've been running with that issue on 6 servers for years. By my hand alone, I'd never have gone back, researched it and fixed it. It's truly trivial, but it's also things like that I don't have the time to invest in that make this so useful.

And it goes through this level of detail for every node. It only bugs me if it finds something outside of scope that's unexplainable.

I don't give it any passwords or hard config. It queries Amazon or Neo4j, which are kept up to date programmatically.

The real terror is I can do this on several different projects at once. As my documentation, guardrails, and automation get better, I'm getting multiple times my own hours' work done every day, and I was really big on automation and self-healing stuff before this ever came along.

Could the node classifier go bonkers and wipe out everything? It could. But I'm careful not to just cron up a bunch of jobs and take a nap. Probably the next step would be to make some overseers to verify backups are being made and it's waiting between jobs.