Investigate volume persistence and explicit path specification in k3s

[dboreham]

We can deploy stateful stacks but it isn't clear if the data actually persists across deployments, and whether we can specify a known path on the host for a volume. This was working at some point in the past but may have required magic config on the cluster.

[dboreham]

Testing: if a stateful stack is started, data written, then stopped then re-started, the data is gone.

[dboreham]

https://docs.k3s.io/add-ons/storage

[dboreham]

https://github.com/rancher/local-path-provisioner

[dboreham]

https://kubernetes.io/docs/concepts/storage/persistent-volumes/

[dboreham]

$ kubectl --kubeconfig=./test19-k3s.yml get storageclass
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  6d17h

[dboreham]

The volume persistence issue has been fixed. Explicit path is still a mystery.

[dboreham]

The automatically assigned volume path can be found:

$ kubectl --kubeconfig=./test19-k3s.yml --namespace stack-9aaaf7e199d5b254 describe pv pvc-fa5a690f-3fa0-4afe-a992-bc19694aa746
Name:              pvc-fa5a690f-3fa0-4afe-a992-bc19694aa746
Labels:            <none>
Annotations:       local.path.provisioner/selected-node: test19
                   pv.kubernetes.io/provisioned-by: rancher.io/local-path
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-path
Status:            Bound
Claim:             stack-9aaaf7e199d5b254/db-data
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          2G
Node Affinity:
  Required Terms:
    Term 0:        kubernetes.io/hostname in [test19]
Message:
Source:
    Type:  LocalVolume (a persistent volume backed by local storage on a node)
    Path:  /var/lib/rancher/k3s/storage/pvc-fa5a690f-3fa0-4afe-a992-bc19694aa746_stack-9aaaf7e199d5b254_db-data
Events:    <none>