k8s-clusters/home

Home Cluster

NOTE: Scripts below are in fish shell.

TODO

• Netboot: https://www.sidero.dev/v0.5/getting-started/prereq-dhcp/
  • Can probably leverage dnsmasq on the router for this?

Setup

Networking

• Prepare networking
  • Internally:
    • Add a DNS entry for the cluster endpoint (router's /etc/hosts + dnsmasq) to point to the initial node
  • Externally:
    • Add a DNS entry for the cluster endpoint to point to the router
    • Setup the router to forward external requests to the initial node

Setup Kubernetes Cluster

Source: https://www.talos.dev/v1.1/introduction/getting-started/

#!/usr/bin/env fish
# these are my values, you will want your own
set CLUSTER_NAME 'home'
set CLUSTER_ENDPOINT 'https://kube-cluster.home.lyte.dev:6443'
set NODE_ADDR '10.0.0.101'
set AGE_KEY (pass age-key | rg '# public key: ' | awk '{printf $4}')

• Setup talos directory if needed
  • mkdir -p talos; cd talos
• If you are not using this configuration:
  • talosctl gen config "$CLUSTER_NAME" "$CLUSTER_ENDPOINT"
  • Edit files as needed, making sure only one of the controlplane nodes is the endpoint in the talosconfig
  • mv talosconfig talosconfig.yaml
  • Encrypt via sops with age
    • for f in *; sops yaml --encrypt --age-key "$AGE_KEY" --in-place "$f"; end
• Setup the talosctl client to use your configuration
  • sops exec-file talosconfig.yaml 'talosctl config merge {}'
• For each node in the cluster as specified in talosconfig.yaml, do the following:
  • Boot the Talos image on the node
    • Disconnect boot media from the node after it's booted otherwise your Ventoy will get wiped
  • Apply the appropriate configuration to the node
    • sops exec-file (controlplane.yml|worker.yml) 'talosctl apply-config --insecure --nodes '"$NODE_ADDR"' --file {}'
    • This can take a moment to finish, but you can move on to the next node while you wait
• Bootstrap the cluster
  • talosctl bootstrap --nodes "$NODE_ADDR"
  • You will need to wait a bit for Kubernetes to initialize
• Pull down the kubeconfig
  • talosctl kubeconfig

Once the cluster has finished initializing and starting up, you should be able to kubectl get nodes.

Adding Nodes

TODO: This process is untested!

• Boot the Talos image on the target node
• Add the node to talosconfig.yaml
  • sops talos/talosconfig.yaml
• Setup the talosctl client to use your configuration
  • sops exec-file talos/talosconfig.yaml 'talosctl config merge {}'
• Apply the appropriate configuration to all nodes in the cluster

Removing Nodes

TODO: This process is untested!

• Cordon and drain the node
• Remove the node from talosconfig.yaml
  • sops talos/talosconfig.yaml
• Update the talosctl client to use your configuration
  • sops exec-file talos/talosconfig.yaml 'talosctl config merge {}'
• Apply the appropriate configuration to all nodes in the cluster
• Power down the node

Untaint Masters

Since we're "frugal" (cheap) and we want to use all the hardware for all the things:

kubectl taint nodes --all node-role.kubernetes.io/master-

Apply Initial Manifests

Some manifests must be applied before we can let GitOps take over.

kubectl apply -k --enable-helm manifests/initialization

Setting up GitOps

TODO

Storage

TODO

Load Balancing

I can probably handle this with my router?

TODO