Every site I host runs the same way. One Caddy deployment sits in front and terminates TLS. Behind it, each website is defined as its own deployment containing a very simple nginx and a git-sync sidecar that pulls the current website files from its Git repository.

That is the shape it has now. It is not the shape I would set out to build if I were starting from scratch, and I would not hand it to anyone as the right way to host a website. It grew one problem at a time, but now it works.

One Caddy, one nginx per site

Caddy is the only thing the internet reaches. It holds the certificates, gets them from ACME by being told the hostname and nothing else, and reverse-proxies each one to a service inside the cluster. It runs as two replicas, which is possible because the certificate a caddy instance pulls state is written to a volume both can read. The cluster post covers that part in more detail.

Each website I host is a deployment of its own, with its own service, volume and its own nginx configuration. This split decouples all my websites from each other, so changes to one do not affect the others.

Two containers, one volume

The website pod is actually made up of two containers, accessing the same volume that carries the content.

# Volume definition for the website pod
- name: nginx
  image: nginx
  volumeMounts:
  - name: site-content
    mountPath: /git

- name: git-sync
  image: registry.k8s.io/git-sync/git-sync
  volumeMounts:
  - name: site-content
    mountPath: /tmp/git
  env:
  - name: GITSYNC_REPO
    value: "https://github.com/Erik-Schuetze/erik-schuetze.dev.git"
  - name: GITSYNC_ROOT
    value: "/tmp/git"
  - name: GITSYNC_DEPTH
    value: "1"
  - name: GITSYNC_WAIT
    value: "30"

The git-sync container clones the repository into its root and pulls it on a timer. Inside the root it keeps one working directory per commit and a symlink named after the repository, pointing at whichever checkout is current. nginx takes the whole volume rather than a path inside it, so every request walks that symlink and lands on the version that is live at that moment.

A new commit is fetched into a fresh directory and the symlink is moved, so a reader never gets a half-written page. That part of the design is the reason I like it.

The clone is shallow and the pull interval is thirty seconds. One is git-sync's default and the other has never given me a reason to change it. Private repositories need a token, which comes from a Secret in the same namespace.

Splitting the sites apart

The first version of this website was a Caddy container with the content baked into it, served by file_server. Caddy was the whole choice of web server, made because certbot inside Kubernetes looked like a lot of work compared with a program that obtains its own certificates. One container, one site, no volume, and nothing to keep in sync.

A second website came along later, and I found out that nginx could run behind Caddy and serve files out of a volume. So the second site became an nginx Pod with Caddy proxying to it.

Caddy and every site were still one Deployment. Updating the content of one site meant restarting all of them, and restarting Caddy sends it back to the certificate authority. Do that often enough and you run into the rate limits on how many certificates you may request, which is the point where TLS stops working for everything at once. That is what pushed the sites apart, and it is the reason a content update no longer touches the proxy.

Every new service gets a subdomain of this domain instead of a domain of its own, so there is one set of names to keep certificates for.

The Caddy image is still in the site repository. Nothing in the cluster pulls it any more, and the workflow that builds and pushes it on every push to main still runs. The image is not idle, though: the same repository has a compose file that builds it with a development Caddyfile, and that is how the site runs on my laptop.

What broke

For a long while, merging anything to any sites main branch produced 500s. The nginx container and the git-sync sidecar both mount the same volume read-write, and on Longhorn that pairing kept failing. The storage post goes into why Longhorn's shared mode behaves the way it does.

I knew a merge would break it, so my deploy procedure was to keep the Argo CD UI open and delete the site Deployment after each push, so it came back with a fresh mount. It only cost a few seconds, but it was also a manual step inside a setup whose entire purpose was not to need one.

Moving the volume off Longhorn and onto NFS fixed the problem.

Errors came back later at a lower rate, whenever git-sync rotated the working directory underneath a running nginx. The mount is different now: nginx takes the whole volume instead of a path inside it, and reaches the content through git-sync's symlink. I cannot tell you which of those two details did the fixing, and neither has been touched since the errors stopped.

Why a PersistentVolumeClaim

An emptyDir would be the cheaper answer, and I would not argue that deploying a dedicated volume is obviously the right one. What I wanted was storage holding the content currently being served, so it can be swapped underneath a running nginx rather than nginx having to be replaced to carry new files.

A Pod that has just started serves the files that are on the volume already while its first pull runs. With an empty volume the site would serve nothing until the pull finished, which is a short window acceptable for a homelab. But with my setup redeploying or restarting pods only causes a very brief downtime for the site.

I am not sure this is the best way to do it, as I am not a full time web developer. But my architecture grew over time, and nothing since has made me go back and revisit it.

When this is the wrong pattern

This is a narrow pattern. It assumes the site is a folder of files that can be served as they are, with no step between the commit and the page.

The boundary is a build step. If the content has to be compiled before it is served, that belongs in a pipeline producing an artifact: the artifact is what you review, and it is the thing you roll back to.

None of that applies here. This site is plain HTML and CSS that I write and commit directly, so there is nothing to build and the repository is the artifact. If that changed, I would be building a pipeline instead.