Dec 22, 2025 - 3 min read

Designing for Consistency (Log 3: The State)

Good design requires good conceptual models. When I look at a system, I should be able to understand its state simply by looking at it. But in the world of infrastructure, servers are often black boxes. We configure them manually, changing a setting here, a file there, until the mental model in our heads no longer matches the reality on the disk. This is a failure of design. It creates what I call a "gulf of evaluation"—I cannot know the state of the system just by observing it.

I have built the network (Log 1) and initialized the cluster (Log 2), but without a proper design for managing state, the system is brittle. It invites human error.

To solve this, I must change the affordances of the system. I must make the invisible visible

The System Image: GitOps as a Conceptual Model

I refuse to configure servers by hand (I use Kiro, btw). To do so is to create a system that cannot be understood or reproduced. Instead, I adopt GitOps.

GitOps is not just a tool; it is a design philosophy. It ensures that the "system image"—the code in the repository—always matches the physical state of the cluster.

I deployed Gitea on the management network to serve as the single source of truth. This is where the intent is stored. Then, I introduced ArgoCD into the cluster. Think of ArgoCD as a feedback mechanism. It constantly compares my intent (the Git repo) with the reality (the Kubernetes cluster). If they drift apart, it signals an error. It forces the system back into alignment. This provides the visibility and feedback that are essential for good control.

Designing for Persistence (NFS)

Computers are ephemeral; data must be permanent. The design challenge here is mapping the abstract concept of "storage" to the physical constraints of the hardware.

I designated the Proxmox host and a future NAS to act as the NFS backend. But good design imposes constraints, so I carved the storage into distinct categories based on their function:

  • /srv/k8s/datasets: This is for raw inputs. The policy is "Retain" because the cost of re-acquiring data is high.

  • /srv/k8s/models: This is for the output—the trained intelligence. It is also "Retained".

  • /srv/k8s/artifacts: This is for temporary workspace. The policy is "Delete," signaling its transient nature.

By mapping these directories to Kubernetes StorageClasses, I created a natural mapping. When a pod requests "datasets," the system intuitively knows where to look.

The Affordance of Recovery

A well-designed system must handle errors gracefully. It must assume that things will go wrong. In the physical world, we have safety rails. In the digital world, we have backups.

However, the traditional backup process is flawed. It usually involves copying data to a local disk before moving it off-site. This introduces a point of failure—what if the local disk is full? What if it is corrupted?

I designed a "stream-based" backup workflow that eliminates this intermediate step. I wrote a script that pipes the data directly from the virtualization layer (vzdump) to the cloud storage (az blob upload).

The data flows from the memory of the Bastion, encrypts in transit, and rests securely in an Azure Blob Storage container in Australia. The design is elegant because it removes unnecessary steps. It reduces the cognitive load. I do not have to worry about local disk space. I only have to trust the stream.

Next: The system now has memory and a reliable way to manage its own state. In Log 4, I must address the interaction between the user and the machine. I must wake the brain.