this post was submitted on 27 Sep 2026
-23 points (4.0% liked)

Programming

28593 readers
408 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev



founded 3 years ago
MODERATORS
 

Originally published at https://blog.light-cloud.com/devops/graph-not-text-file on the Light Cloud Blog.

By James, Light Cloud

Summary: Facebook figured this out in 2015: config errors were a major source of outages, and their answer was to compile configuration from source and model its dependencies. A decade later, most of us still review the characters instead of the structure.

Why Our Source of Truth Is a Graph, Not a Text File

That's Facebook's engineering team at SOSP 2015, describing a system absorbing thousands of live configuration changes a day. Their fix wasn't stricter YAML review. They compiled configs from high-level source code, expressed configuration dependencies "similar to the include statement in a C++ program", and validated invariants against the result before anything touched a server. At the sharpest end of the problem, a decade ago, the conclusion was already in: stop treating configuration as text you diff. Treat it as a structure you compute.

A year ago Julia wrote a breakup letter to YAML on this blog. That post was the feelings. This one is the argument, and the argument is why ICE's source of truth is a deterministic graph rather than a directory of text files.

Questions a text file can't answer

The test of a source of truth is whether it can answer the questions you ask during an incident. Line one of the table is the question I got burned by. The rest follow.

The question you actually ask What the text file knows
What breaks if I delete this subnet? Which lines contain the subnet's name
What order do these changes need to apply in? Nothing; order is computed at plan time, then discarded
Has reality drifted from what's declared? Nothing; text can't observe the cloud
Is this rename safe? It reads as a delete plus a create; good luck

Terraform itself concedes the point, quietly. Run terraform graph and it prints the DAG (directed acyclic graph) of resources and dependencies it builds internally before every single plan. The graph exists on every run. It's derived from your text, used to order operations, and thrown away, while review happens on the characters. The most load-bearing artifact in the whole pipeline is the one no reviewer ever sees.

The YAML ecosystem keeps rediscovering the gap. Helm templates YAML with text substitution, Kustomize patches YAML with more YAML, and both exist because the format can't express the relationships everyone actually needs, so the industry bolts string machinery onto a tree structure and calls the result configuration management. Tools that exist to work around the source of truth are testimony about the source of truth.

Determinism is the actual feature

A graph as the source of truth changes three things, and none of them are cosmetic. Hold on: two things, stated properly.

First, identity. In a graph, a resource is a node with an identity that survives renaming, so a rename is a rename. In text, the same edit surfaces as destruction plus creation, and every Terraform operator eventually learns terraform state mv the way you learn most things in this field, which is at night. Second, reproducibility. A deterministic model means the same graph produces the same actions, every time, with drift detected by comparing the graph's expectations against observed reality node by node, rather than by diffing two commits of a file that was never looking at the cloud in the first place.

The scale details are worth sitting with: they report a median config size of 1KB with large ones reaching MBs or GBs, hundreds of thousands of configs, and trillions of configuration checks daily. Nobody reviews that by reading characters. Structure was the only way through.

Text won for a reason

Steelman time, and it's a strong one. Text is the only format every engineer, editor, and tool on earth can open. Git gave it merge machinery, blame, and history for free. It's greppable at 3 a.m. It locks you into no vendor. And the failure mode of the alternative is real: an opaque model nobody can inspect is worse than ugly YAML, because at least ugly YAML can be read in a pager on a bad night. GitOps built genuine operational rigor on all of this.

Every bit of that is conceded, and it shapes the design rather than defeating it. The graph serializes to versionable, diffable text, so the audit trail survives; what changes is what the diff says. A text diff reports "+14 -9 lines". A graph diff reports which nodes changed and what depends on them, which is the difference between describing an edit and describing its blast radius. You keep git. You stop asking git to be a model of your infrastructure, because it never was one.

This is the bet ICE makes concrete: one deterministic graph modeling your infrastructure across AWS, GCP, and Azure, with the graph as the thing you operate on and text as one of its projections. Drift stops being a quarterly surprise and becomes a comparison the tool runs continuously.

I don't claim the graph model is finished territory; serialization formats, review UX, and escape hatches for the weird 5% are open problems we work on in the open. What I'll defend flatly is the direction. Structure first, text as output.

Facebook needed this at hundreds of thousands of configs. You'll feel it at fifty, the first time a plan output surprises you, because the tool held a graph that knew the answer and discarded it before showing you a diff of characters. Why is the throwaway the part you review?

Related reading


This article first appeared on the Light Cloud Blog: https://blog.light-cloud.com/devops/graph-not-text-file.

no comments (yet)
sorted by: hot top controversial new old
there doesn't seem to be anything here