Apple, like many others, pesters with red dots—little Red Queens to race again in Wonderland. Last night we stopped postponing Mac OS 11.4. This morning confirmed our reluctance. Unplanned work greeted us: local wiki was no longer working. To return to service we re-learn our local deploy and learn new corners of keeping up with The Internet. We offer these notes with many hopes: perhaps our future self can untangle a future knot, perhaps our collaborators can be spared a few extra minutes, perhaps future wiki admins can plan for a little slack in their schedule. Interruptions abound.
See Tempo Interruptions.
digraph { rankdir=LR node [shape=box style="rounded,filled"] "Upgrade\nMacOS" -> "Upgrade k3d\nto 4.x" -> "Upgrade k8s\nto 1.21" -> "Update ingress\nconfig" -> "Write a\nwiki page" }
See also Yak Shaving.
What follows is a play-by-play of our troubleshooting.
The first symptom appeared after a Restore Previous Session in Firefox. http://localhost 
and http://today.localhost won't open.
Is our wiki running?
kubectl get all
We see deployment.apps/wiki-deployment and service/wiki-service which are our usual confirmations.
Are we getting logs?
kubectl logs deployment/wiki-deployment
// kubernetes replies... Wiki starting in Farm mode, navigate to a specific server to start it.
That suggests the requests from Firefox are not making it all the way into our container. The wiki farm has started, but no requests have landed yet.
We dig deeper into our memory... which containers run kubernetes again?
$ docker ps --format '{{.Names}} {{.Ports}}' k3d-wiki-serverlb 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:51501->6443/tcp k3d-wiki-server-0
k3d-wiki-serverlb is listening on port 80.
docker logs --tail=200 k3d-wiki-serverlb
We recall that these logs showed problems finding the upstream server—it is all fixed at the time of this writing, so we no longer have a copy of the logs to show you, Dear Reader.
Docker was seeing the traffic from Firefox, but failing to connect through the docker and k3d machinery into the wiki container.
At the time of our troubleshooting, we only vaguely remembered exactly how all the parts play together. While composing this page, we have found a diagram which has some of the parts. See Local Kubernetes Wiki.
Back to the story, we shifted at this point from investigation and started with speculative interventions.
What version of k3d are we running?
k3d --version
We were on version 3.x. Current k3d is now 4.x. So we upgrade. There was a moment not documented in this story where we also did a bit of investigation to confirm that we did use homebrew to install k3d.
k3d cluster delete wiki brew upgrade k3d
How did we create the local k8s cluster, originally?
Is it in our dotfiles? Or in that unfinished github repo?
k3d cluster create wiki \ --k3s-server-arg --tls-san="127.0.0.1" \ --port 80:80@loadbalancer \ --volume "$HOME/.wiki-k8s:/macos/.wiki-k8s" \ --volume "$HOME/workspace/fedwiki:/macos/fedwiki" \ --wait
And what about starting the wiki in that cluster?
kubectl apply -f \ https://deploy.wiki.do/assets/wiki/wiki.yaml
Sigh. More errors. The containers deployed okay but the Ingress record complained.
no matches for kind "Ingress" in version "networking.k8s.io/v1"
Our search for that error message turned up a clue in an issue in kubernetes github comment . We cannot explain how we sift all the information in this issue to narrow in on this single comment. Mainly, we knew we were looking for version information related to Ingress.
networking.k8s.io/v1beta1 == 1.14 to 1.18 networking.k8s.io/v1 = 1.19+
Here we struggled a while to figure out which kubernetes version was installed by default with the latest k3d. After a time we gave up investigating and shifted again to just find a way to install a specific version. (Notice the new --image flag)
k3d cluster delete wiki k3d cluster create wiki \ --k3s-server-arg --tls-san="127.0.0.1" \ --port 80:80@loadbalancer \ --image rancher/k3s:v1.21.2-k3s1 \ --volume "$HOME/.wiki-k8s:/macos/.wiki-k8s" \ --volume "$HOME/workspace/fedwiki:/macos/fedwiki" \ --volume "$PWD/assets/wiki:/var/lib/rancher/k3s/server/manifests/wiki" \ --wait
And we try again to deploy wiki to the cluster.
kubectl apply -f \ https://deploy.wiki.do/assets/wiki/wiki.yaml
Sigh. At least it was a different error.
unknown field "serviceName" in io.k8s.api.networking.v1.IngressBackend
We find our next clue at Stackoverflow. stackoverflow
We update our ingress configuration. A couple times.
Finally, success.
.
While composing this page we also learned how to see timestamps in our zsh history. Thanks stackexchange
history -E history -i \history -E \history -i
It was also helpful reviewing our browser history in Firefox to remember what error messages we searched for and where we found the useful answers.