lytedev/k8s-clusters
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Before

Browse source
This commit is contained in:
Daniel Flanagan 2022-07-03 23:42:26 -05:00
parent d8af720323
commit 0b1cab7281
5 changed files with 518 additions and 13 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

59
home/readme.md
View file

@ -3,21 +3,37 @@
## TODO
- Netboot: https://www.sidero.dev/v0.5/getting-started/prereq-dhcp/
- Should be able to use `dnsmasq` for this?
## Setup
You will need the following:
- `argocd`
- `talosctl`
- `kubectl`
### Networking Preparations
- Internally:
- Add a DNS entry for the cluster endpoint (router's `/etc/hosts` and
`dnsmasq`) to point to the initial node
- If using MetalLB, Ensure that ARP tables for receiving machines will have
a route matching your desired CIDR
- Externally:
- Add a DNS entry for the cluster endpoint to point to the router
- Setup the router to forward external requests to the initial node
### Initialize Kubernetes Cluster
> Source: https://www.talos.dev/v1.1/introduction/getting-started/
- Boot the Talos ISO on the initial node
- Prepare networking
- Internally:
- Add a DNS entry for the cluster endpoint (router's `/etc/hosts` + `dnsmasq`) to point to the initial node
- Externally:
- Add a DNS entry for the cluster endpoint to point to the router
- Setup the router to forward external requests to the initial node
- If you are not using _this_ configuration:
- `talosctl gen config "cluster-name" "cluster-endpoint"`
- Edit files as needed
- Prepare your talos config
- `cd talos`
- Or if you are not using _this_ configuration:
- `talosctl gen config "cluster-name" "cluster-endpoint"`
- Edit files as needed
- Apply the control plane config to the initial node
- `talosctl apply-config --insecure --nodes "$INITIAL_NODE_ADDR" --file controlplane.yaml`
- You will need to wait a bit for the configuration to be applied, Talos to
@ -30,11 +46,10 @@
- You will need to wait a bit for Kubernetes to initialize
- Pull down the kubeconfig
- `talosctl kubeconfig`
- Optionally enable the control plane node(s) to run pods
- `kubectl taint nodes --all node-role.kubernetes.io/master-`
Once the cluster has finished initializing _and starting up_, you should be
able to `kubectl get nodes`.
## Adding Nodes
### Adding Nodes
- Boot the Talos ISO on the target node
- Apply the appropriate configuration to the target node
@ -42,6 +57,24 @@ able to `kubectl get nodes`.
- You will need to wait a bit for Kubernetes to initialize, start up, and
then join the cluster
- Add the node to `talosconfig` as needed
- For another control plane node: `talosctl --talosconfig=./talosconfig config endpoint "$TARGET_NODE_ADDR"`
- Otherwise: `talosctl --talosconfig=./talosconfig config endpoint "$TARGET_NODE_ADDR"` (untested)
- Add the node to the internal DNS entry for the cluster endpoint
- Optionally enable the control plane node(s) to run work
- `kubectl taint nodes --all node-role.kubernetes.io/master-`
Once the cluster has finished initializing _and starting up_, you should be
able to `kubectl get nodes`.
### Setup GitOps
Now we should avoid changing the cluster manually so that all changes are
captured in source control. We do this with flux:
```bash
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
```
## Storage

0
home/controlplane.encrypted.yaml → home/talos/controlplane.encrypted.yaml
View file

0
home/talosconfig.encrypted → home/talos/talosconfig.encrypted
View file

0
home/worker.encrypted.yaml → home/talos/worker.encrypted.yaml
View file

472
home/talos/worker.yaml Normal file
View file

@ -0,0 +1,472 @@
version: v1alpha1 # Indicates the schema used to decode the contents.
debug: false # Enable verbose logging to the console.
persist: true # Indicates whether to pull the machine config upon every boot.
# Provides machine specific configuration options.
machine:
type: worker # Defines the role of the machine within the cluster.
token: xsw2df.9946v2p5ya4ytunj # The `token` is used by a machine to join the PKI of the cluster.
# The root certificate authority of the PKI.
ca:
crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJQakNCOGFBREFnRUNBaEErUHhmTE9KTGc3UXFuV0ZQa3A4WEdNQVVHQXl0bGNEQVFNUTR3REFZRFZRUUsKRXdWMFlXeHZjekFlRncweU1qQTNNRE14T0RNeU1EQmFGdzB6TWpBMk16QXhPRE15TURCYU1CQXhEakFNQmdOVgpCQW9UQlhSaGJHOXpNQ293QlFZREsyVndBeUVBS1dEY1FKb2haRUI2UjhCMmhPNVh6ZGNpRitDZzVMTW9UTWJmCjFGTllGVm1qWVRCZk1BNEdBMVVkRHdFQi93UUVBd0lDaERBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUkKS3dZQkJRVUhBd0l3RHdZRFZSMFRBUUgvQkFVd0F3RUIvekFkQmdOVkhRNEVGZ1FVZ3JpOXpvb1F4MEJEZHFPcAo4NWh6RnJHTWhib3dCUVlESzJWd0EwRUFRSjJqVVg4U3N2c3BFQlVwOHdEN2o1VDlvOGNWdzNIT1daV0dWWmdkCmY1UnErM05ER3ZoTUx1REJPK1RyM2JIbkRYeVREcWVBTlc1bHNSTERGS3BaRFE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
key: ""
# Extra certificate subject alternative names for the machine's certificate.
certSANs: []
# # Uncomment this to enable SANs.
# - 10.0.0.10
# - 172.16.0.10
# - 192.168.0.10
# Used to provide additional options to the kubelet.
kubelet:
image: ghcr.io/siderolabs/kubelet:v1.24.2 # The `image` field is an optional reference to an alternative kubelet image.
# # The `ClusterDNS` field is an optional reference to an alternative kubelet clusterDNS ip list.
# clusterDNS:
# - 10.96.0.10
# - 169.254.2.53
# # The `extraArgs` field is used to provide additional flags to the kubelet.
# extraArgs:
# key: value
# # The `extraMounts` field is used to add additional mounts to the kubelet container.
# extraMounts:
# - destination: /var/lib/example
# type: bind
# source: /var/lib/example
# options:
# - bind
# - rshared
# - rw
# # The `extraConfig` field is used to provide kubelet configuration overrides.
# extraConfig:
# serverTLSBootstrap: true
# # The `nodeIP` field is used to configure `--node-ip` flag for the kubelet.
# nodeIP:
# # The `validSubnets` field configures the networks to pick kubelet node IP from.
# validSubnets:
# - 10.0.0.0/8
# - '!10.0.0.3/32'
# - fdc7::/16
# Provides machine specific network configuration options.
network: {}
# # `interfaces` is used to define the network interface configuration.
# interfaces:
# - interface: eth0 # The interface name.
# # Assigns static IP addresses to the interface.
# addresses:
# - 192.168.2.0/24
# # A list of routes associated with the interface.
# routes:
# - network: 0.0.0.0/0 # The route's network (destination).
# gateway: 192.168.2.1 # The route's gateway (if empty, creates link scope route).
# metric: 1024 # The optional metric for the route.
# mtu: 1500 # The interface's MTU.
#
# # # Picks a network device using the selector.
# # # select a device with bus prefix 00:*.
# # deviceSelector:
# # busPath: 00:* # PCI, USB bus prefix, supports matching by wildcard.
# # # select a device with mac address matching `*:f0:ab` and `virtio` kernel driver.
# # deviceSelector:
# # hardwareAddr: '*:f0:ab' # Device hardware address, supports matching by wildcard.
# # driver: virtio # Kernel driver, supports matching by wildcard.
# # # Bond specific options.
# # bond:
# # # The interfaces that make up the bond.
# # interfaces:
# # - eth0
# # - eth1
# # mode: 802.3ad # A bond option.
# # lacpRate: fast # A bond option.
# # # Indicates if DHCP should be used to configure the interface.
# # dhcp: true
# # # DHCP specific options.
# # dhcpOptions:
# # routeMetric: 1024 # The priority of all routes received via DHCP.
# # # Wireguard specific configuration.
# # # wireguard server example
# # wireguard:
# # privateKey: ABCDEF... # Specifies a private key configuration (base64 encoded).
# # listenPort: 51111 # Specifies a device's listening port.
# # # Specifies a list of peer configurations to apply to a device.
# # peers:
# # - publicKey: ABCDEF... # Specifies the public key of this peer.
# # endpoint: 192.168.1.3 # Specifies the endpoint of this peer entry.
# # # AllowedIPs specifies a list of allowed IP addresses in CIDR notation for this peer.
# # allowedIPs:
# # - 192.168.1.0/24
# # # wireguard peer example
# # wireguard:
# # privateKey: ABCDEF... # Specifies a private key configuration (base64 encoded).
# # # Specifies a list of peer configurations to apply to a device.
# # peers:
# # - publicKey: ABCDEF... # Specifies the public key of this peer.
# # endpoint: 192.168.1.2 # Specifies the endpoint of this peer entry.
# # persistentKeepaliveInterval: 10s # Specifies the persistent keepalive interval for this peer.
# # # AllowedIPs specifies a list of allowed IP addresses in CIDR notation for this peer.
# # allowedIPs:
# # - 192.168.1.0/24
# # # Virtual (shared) IP address configuration.
# # # layer2 vip example
# # vip:
# # ip: 172.16.199.55 # Specifies the IP address to be used.
# # Used to statically set the nameservers for the machine.
# nameservers:
# - 8.8.8.8
# - 1.1.1.1
# # Allows for extra entries to be added to the `/etc/hosts` file
# extraHostEntries:
# - ip: 192.168.1.100 # The IP of the host.
# # The host alias.
# aliases:
# - example
# - example.domain.tld
# # Configures KubeSpan feature.
# kubespan:
# enabled: true # Enable the KubeSpan feature.
# Used to provide instructions for installations.
install:
disk: /dev/sda # The disk used for installations.
image: ghcr.io/siderolabs/installer:v1.1.0 # Allows for supplying the image used to perform the installation.
bootloader: true # Indicates if a bootloader should be installed.
wipe: false # Indicates if the installation disk should be wiped at installation time.
# # Look up disk using disk attributes like model, size, serial and others.
# diskSelector:
# size: 4GB # Disk size.
# model: WDC* # Disk model `/sys/block/<dev>/device/model`.
# busPath: /pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0 # Disk bus path.
# # Allows for supplying extra kernel args via the bootloader.
# extraKernelArgs:
# - talos.platform=metal
# - reboot=k
# # Allows for supplying additional system extension images to install on top of base Talos image.
# extensions: ghcr.io/siderolabs/gvisor:20220117.0-v1.0.0
# Features describe individual Talos features that can be switched on or off.
features:
rbac: true # Enable role-based access control (RBAC).
# # Provides machine specific control plane configuration options.
# # ControlPlane definition example.
# controlPlane:
# # Controller manager machine specific configuration options.
# controllerManager:
# disabled: false # Disable kube-controller-manager on the node.
# # Scheduler machine specific configuration options.
# scheduler:
# disabled: true # Disable kube-scheduler on the node.
# # Used to provide static pod definitions to be run by the kubelet directly bypassing the kube-apiserver.
# # nginx static pod.
# pods:
# - apiVersion: v1
# kind: pod
# metadata:
# name: nginx
# spec:
# containers:
# - image: nginx
# name: nginx
# # Used to partition, format and mount additional disks.
# # MachineDisks list example.
# disks:
# - device: /dev/sdb # The name of the disk to use.
# # A list of partitions to create on the disk.
# partitions:
# - mountpoint: /var/mnt/extra # Where to mount the partition.
#
# # # The size of partition: either bytes or human readable representation. If `size:` is omitted, the partition is sized to occupy the full disk.
# # # Human readable representation.
# # size: 100 MB
# # # Precise value in bytes.
# # size: 1073741824
# # Allows the addition of user specified files.
# # MachineFiles usage example.
# files:
# - content: '...' # The contents of the file.
# permissions: 0o666 # The file's permissions in octal.
# path: /tmp/file.txt # The path of the file.
# op: append # The operation to use
# # The `env` field allows for the addition of environment variables.
# # Environment variables definition examples.
# env:
# GRPC_GO_LOG_SEVERITY_LEVEL: info
# GRPC_GO_LOG_VERBOSITY_LEVEL: "99"
# https_proxy: http://SERVER:PORT/
# env:
# GRPC_GO_LOG_SEVERITY_LEVEL: error
# https_proxy: https://USERNAME:PASSWORD@SERVER:PORT/
# env:
# https_proxy: http://DOMAIN\USERNAME:PASSWORD@SERVER:PORT/
# # Used to configure the machine's time settings.
# # Example configuration for cloudflare ntp server.
# time:
# disabled: false # Indicates if the time service is disabled for the machine.
# # Specifies time (NTP) servers to use for setting the system time.
# servers:
# - time.cloudflare.com
# bootTimeout: 2m0s # Specifies the timeout when the node time is considered to be in sync unlocking the boot sequence.
# # Used to configure the machine's sysctls.
# # MachineSysctls usage example.
# sysctls:
# kernel.domainname: talos.dev
# net.ipv4.ip_forward: "0"
# # Used to configure the machine's sysfs.
# # MachineSysfs usage example.
# sysfs:
# devices.system.cpu.cpu0.cpufreq.scaling_governor: performance
# # Used to configure the machine's container image registry mirrors.
# registries:
# # Specifies mirror configuration for each registry.
# mirrors:
# ghcr.io:
# # List of endpoints (URLs) for registry mirrors to use.
# endpoints:
# - https://registry.insecure
# - https://ghcr.io/v2/
# # Specifies TLS & auth configuration for HTTPS image registries.
# config:
# registry.insecure:
# # The TLS configuration for the registry.
# tls:
# insecureSkipVerify: true # Skip TLS server certificate verification (not recommended).
#
# # # Enable mutual TLS authentication with the registry.
# # clientIdentity:
# # crt: LS0tIEVYQU1QTEUgQ0VSVElGSUNBVEUgLS0t
# # key: LS0tIEVYQU1QTEUgS0VZIC0tLQ==
#
# # # The auth configuration for this registry.
# # auth:
# # username: username # Optional registry authentication.
# # password: password # Optional registry authentication.
# # Machine system disk encryption configuration.
# systemDiskEncryption:
# # Ephemeral partition encryption.
# ephemeral:
# provider: luks2 # Encryption provider to use for the encryption.
# # Defines the encryption keys generation and storage method.
# keys:
# - # Deterministically generated key from the node UUID and PartitionLabel.
# nodeID: {}
# slot: 0 # Key slot number for LUKS2 encryption.
#
# # # Cipher kind to use for the encryption. Depends on the encryption provider.
# # cipher: aes-xts-plain64
# # # Defines the encryption sector size.
# # blockSize: 4096
# # # Additional --perf parameters for the LUKS2 encryption.
# # options:
# # - no_read_workqueue
# # - no_write_workqueue
# # Configures the udev system.
# udev:
# # List of udev rules to apply to the udev system
# rules:
# - SUBSYSTEM=="drm", KERNEL=="renderD*", GROUP="44", MODE="0660"
# # Configures the logging system.
# logging:
# # Logging destination.
# destinations:
# - endpoint: tcp://1.2.3.4:12345 # Where to send logs. Supported protocols are "tcp" and "udp".
# format: json_lines # Logs format.
# # Configures the kernel.
# kernel:
# # Kernel modules to load.
# modules:
# - name: brtfs # Module name.
# Provides cluster specific configuration options.
cluster:
id: TE7R-46wrT94ehUwrKhxLxYZMreorFiwc-mnrhmavF4= # Globally unique identifier for this cluster (base64 encoded random 32 bytes).
secret: o3mVhBjxl5cz+I5M71p2b2kuUI1EppwweG0pC44kTIs= # Shared secret of cluster (base64 encoded random 32 bytes).
# Provides control plane specific configuration options.
controlPlane:
endpoint: https://kube-cluster.home.lyte.dev:6443 # Endpoint is the canonical controlplane endpoint, which can be an IP address or a DNS hostname.
# Provides cluster specific network configuration options.
network:
dnsDomain: cluster.local # The domain used by Kubernetes DNS.
# The pod subnet CIDR.
podSubnets:
- 10.244.0.0/16
# The service subnet CIDR.
serviceSubnets:
- 10.96.0.0/12
# # The CNI used.
# cni:
# name: custom # Name of CNI to use.
# # URLs containing manifests to apply for the CNI.
# urls:
# - https://docs.projectcalico.org/archive/v3.20/manifests/canal.yaml
token: nzaznc.g83a8lrm99xtb671 # The [bootstrap token](https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/) used to join the cluster.
aescbcEncryptionSecret: "" # The key used for the [encryption of secret data at rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/).
# # Decryption secret example (do not use in production!).
# z01mye6j16bspJYtTB/5SFX8j7Ph4JXxM2Xuu4vsBPM=
# The base64 encoded root certificate authority used by Kubernetes.
ca:
crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJpakNDQVRDZ0F3SUJBZ0lSQUxGZDd2VTkvYkZ1Zi9nQlpZNzhLTU13Q2dZSUtvWkl6ajBFQXdJd0ZURVQKTUJFR0ExVUVDaE1LYTNWaVpYSnVaWFJsY3pBZUZ3MHlNakEzTURNeE9ETXlNREJhRncwek1qQTJNekF4T0RNeQpNREJhTUJVeEV6QVJCZ05WQkFvVENtdDFZbVZ5Ym1WMFpYTXdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CCkJ3TkNBQVJOeVpvei8vdVZLbjJsQ0ZNWXpJK0tIQjF1TGNhaTVVQnZrWm80NG5pRmFic0ZGKzA1bTVlZHZyTi8KWCtmMTdPUWNRV1lDV1krM2xSWnBlTTJWUEIxSG8yRXdYekFPQmdOVkhROEJBZjhFQkFNQ0FvUXdIUVlEVlIwbApCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQndNQ01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWURWUjBPCkJCWUVGUDdxait1TmllZDNBR0YwbTBSOW9LMk83V1YyTUFvR0NDcUdTTTQ5QkFNQ0EwZ0FNRVVDSVFDMFJMc20KSmxNTnVJRnd3ZjJld24zN3NzeDRLaWFjaEJ6WDJzMGY0ZUJORFFJZ1NqenA5ejA2S2NzemswTFN1dGJIMFArZApxVTFCcXV1azN2WlQvSzhFY05JPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
key: ""
# Configures cluster member discovery.
discovery:
enabled: true # Enable the cluster membership discovery feature.
# Configure registries used for cluster member discovery.
registries:
# Kubernetes registry uses Kubernetes API server to discover cluster members and stores additional information
kubernetes: {}
# Service registry is using an external service to push and pull information about cluster members.
service: {}
# # External service endpoint.
# endpoint: https://discovery.talos.dev/
# # The base64 encoded aggregator certificate authority used by Kubernetes for front-proxy certificate generation.
# # AggregatorCA example.
# aggregatorCA:
# crt: LS0tIEVYQU1QTEUgQ0VSVElGSUNBVEUgLS0t
# key: LS0tIEVYQU1QTEUgS0VZIC0tLQ==
# # The base64 encoded private key for service account token generation.
# # AggregatorCA example.
# serviceAccount:
# key: LS0tIEVYQU1QTEUgS0VZIC0tLQ==
# # API server specific configuration options.
# apiServer:
# image: k8s.gcr.io/kube-apiserver:v1.24.2 # The container image used in the API server manifest.
# # Extra arguments to supply to the API server.
# extraArgs:
# feature-gates: ServerSideApply=true
# http2-max-streams-per-connection: "32"
# # Extra certificate subject alternative names for the API server's certificate.
# certSANs:
# - 1.2.3.4
# - 4.5.6.7
# # Configure the API server admission plugins.
# admissionControl:
# - name: PodSecurity # Name is the name of the admission controller.
# # Configuration is an embedded configuration object to be used as the plugin's
# configuration:
# apiVersion: pod-security.admission.config.k8s.io/v1alpha1
# defaults:
# audit: restricted
# audit-version: latest
# enforce: baseline
# enforce-version: latest
# warn: restricted
# warn-version: latest
# exemptions:
# namespaces:
# - kube-system
# runtimeClasses: []
# usernames: []
# kind: PodSecurityConfiguration
# # Controller manager server specific configuration options.
# controllerManager:
# image: k8s.gcr.io/kube-controller-manager:v1.24.2 # The container image used in the controller manager manifest.
# # Extra arguments to supply to the controller manager.
# extraArgs:
# feature-gates: ServerSideApply=true
# # Kube-proxy server-specific configuration options
# proxy:
# image: k8s.gcr.io/kube-proxy:v1.24.2 # The container image used in the kube-proxy manifest.
# mode: ipvs # proxy mode of kube-proxy.
# # Extra arguments to supply to kube-proxy.
# extraArgs:
# proxy-mode: iptables
# # Scheduler server specific configuration options.
# scheduler:
# image: k8s.gcr.io/kube-scheduler:v1.24.2 # The container image used in the scheduler manifest.
# # Extra arguments to supply to the scheduler.
# extraArgs:
# feature-gates: AllBeta=true
# # Etcd specific configuration options.
# etcd:
# image: gcr.io/etcd-development/etcd:v3.5.4 # The container image used to create the etcd service.
# # The `ca` is the root certificate authority of the PKI.
# ca:
# crt: LS0tIEVYQU1QTEUgQ0VSVElGSUNBVEUgLS0t
# key: LS0tIEVYQU1QTEUgS0VZIC0tLQ==
# # Extra arguments to supply to etcd.
# extraArgs:
# election-timeout: "5000"
# subnet: 10.0.0.0/8 # The subnet from which the advertise URL should be.
# # Core DNS specific configuration options.
# coreDNS:
# image: docker.io/coredns/coredns:1.9.3 # The `image` field is an override to the default coredns image.
# # External cloud provider configuration.
# externalCloudProvider:
# enabled: true # Enable external cloud provider.
# # A list of urls that point to additional manifests for an external cloud provider.
# manifests:
# - https://raw.githubusercontent.com/kubernetes/cloud-provider-aws/v1.20.0-alpha.0/manifests/rbac.yaml
# - https://raw.githubusercontent.com/kubernetes/cloud-provider-aws/v1.20.0-alpha.0/manifests/aws-cloud-controller-manager-daemonset.yaml
# # A list of urls that point to additional manifests.
# extraManifests:
# - https://www.example.com/manifest1.yaml
# - https://www.example.com/manifest2.yaml
# # A map of key value pairs that will be added while fetching the extraManifests.
# extraManifestHeaders:
# Token: "1234567"
# X-ExtraInfo: info
# # A list of inline Kubernetes manifests.
# inlineManifests:
# - name: namespace-ci # Name of the manifest.
# contents: |- # Manifest contents as a string.
# apiVersion: v1
# kind: Namespace
# metadata:
# name: ci
# # Settings for admin kubeconfig generation.
# adminKubeconfig:
# certLifetime: 1h0m0s # Admin kubeconfig certificate lifetime (default is 1 year).