Shrinking a Metrics service volume without data loss

To shrink a Metrics service volume without data loss:

Unmount the Metrics service volume:
1. Get the name of the volume used by the Metrics service. You will need this volume name to locate volume data and volume replicas. Replica names begin with the volume name.
  sudo k0s kubectl get pvc metrics -n kuma -o json | jq '.spec.volumeName'
2. Stop the Metrics service.
  sudo k0s kubectl scale deployment/metrics -n kuma --replicas=0
3. Make sure that the metrics-<UID> pod of the Metrics service has disappeared (is not present in the command output).
  sudo k0s kubectl get pods -n kuma
4. If the pod does not disappear within a few minutes, force its deletion. The pod name must be taken from the output of the previous command.
  sudo k0s kubectl delete pod metrics-<UID> -n kuma --force
5. Make sure that the Metrics service volume is in the detached state.
  sudo k0s kubectl get volume <volume_name> -n longhorn-system -o json | jq '.status.state'
Create a temporary task:
1. Create a temporary task named tmp-metrics-job in the 'kuma' namespace and mount the Metrics service volume.
  sudo k0s kubectl apply -f - <<EOF
  
  ---
  
  apiVersion: batch/v1
  
  kind: Job
  
  metadata:
  
  name: tmp-metrics-job
  
  namespace: kuma
  
  spec:
  
  ttlSecondsAfterFinished: 300
  
  backoffLimit: 0
  
  parallelism: 1
  
  completions: 1
  
  template:
  
  spec:
  
  containers:
  
  - name: tmp-container
  
  image: docker.io/library/busybox:1.35.0
  
  imagePullPolicy: IfNotPresent
  
  command: ["/bin/sh"]
  
  args:
  
  - -c
  
  - >-
  
  sleep 36000
  
  volumeMounts:
  
  - mountPath: /metrics
  
  name: metrics
  
  restartPolicy: Never
  
  volumes:
  
  - name: metrics
  
  persistentVolumeClaim:
  
  claimName: metrics
  
  EOF
2. Determine the name of the worker node on which the temporary task is running.
  sudo k0s kubectl get pods -n kuma -o wide | grep tmp-metrics
3. On this worker node, determine the mount point.
  sudo lsblk | grep "<volume_name>"
4. Change to the found directory, /var/lib/k0s/kubelet/pods/....../mount. The content should look like this:
  ls -al
  
  total 48
  
  drwxrwsr-x 9 root 65532 4096 Aug 26 10:36 .
  
  drwxr-x--- 3 root root 4096 Aug 26 11:08 ..
  
  drwx--S--- 2 65532 65532 4096 Aug 26 10:36 certificates
  
  drwx--S--- 2 65532 65532 4096 Aug 26 10:36 dashboards
  
  drwxrwsr-x 8 root 65532 4096 Aug 26 10:34 data
  
  drwx--S--- 2 65532 65532 4096 Aug 26 10:36 log
  
  drwxrws--- 2 root 65532 16384 Aug 26 10:34 lost+found
  
  drwx--S--- 2 65532 65532 4096 Aug 26 10:36 rules
  
  drwx--S--- 2 65532 65532 4096 Aug 26 10:36 tmp
Copy the data:
1. Copy all data from the /var/lib/k0s/kubelet/pods/....../mount directory, except for the 'lost+found' and 'tmp' directories, to a temporary directory, keeping the permissions on directories and files.
2. After copying, go from the mount point to the / or /home directory to avoid blocking the subsequent unmounting of the volume.
Delete the temporary task, pod, PVC, and volume of the Metrics service:
1. Delete the temporary task with the following command:
  sudo k0s kubectl delete job tmp-metrics-job -n kuma
2. Make sure that the metrics-<UID> pod of the temporary task is not present in the output of the following command:
  sudo k0s kubectl get pods -n kuma
3. If the pod does not disappear within a few minutes, force its deletion. The pod name must be taken from the output of the previous command.
  sudo k0s kubectl delete pod tmp-metrics-job-<UID> -n kuma --force
4. Make sure that the Metrics service volume is in the detached state.
  sudo k0s kubectl get volume <volume_name> -n longhorn-system -o json | jq '.status.state'
5. Delete the PVC of the Metrics service.
  sudo k0s kubectl delete pvc metrics -n kuma
6. Make sure the Metrics service volume is not present in the output of the following command:
  sudo k0s kubectl get volume <volume_name> -n longhorn-system
  
  On all worker nodes, directories with names beginning with the name of the deleted volume should disappear from the /opt/longhorn/replicas directory. If this does not happen, you need to delete them manually. Take care not to delete replicas that belong to other volumes by mistake.
Create a new volume:
1. Specify the value in the 'storage' line, then create a PVC for the Metrics service.
  sudo k0s kubectl apply -f - <<EOF
  
  ---
  
  apiVersion: v1
  
  kind: PersistentVolumeClaim
  
  metadata:
  
  name: metrics
  
  namespace: kuma
  
  spec:
  
  accessModes:
  
  - ReadWriteOnce
  
  storageClassName: kuma-network-storage
  
  resources:
  
  requests:
  
  storage: 30Gi
  
  EOF
2. Get the name of the volume used by the Metrics service again.
  sudo k0s kubectl get pvc metrics -n kuma -o json | jq '.spec.volumeName'
3. Create a temporary task named tmp-metrics-job in the 'kuma' namespace again and mount the Metrics service volume.
  sudo k0s kubectl apply -f - <<EOF
  
  ---
  
  apiVersion: batch/v1
  
  kind: Job
  
  metadata:
  
  name: tmp-metrics-job
  
  namespace: kuma
  
  spec:
  
  ttlSecondsAfterFinished: 300
  
  backoffLimit: 0
  
  parallelism: 1
  
  completions: 1
  
  template:
  
  spec:
  
  containers:
  
  - name: tmp-container
  
  image: docker.io/library/busybox:1.35.0
  
  imagePullPolicy: IfNotPresent
  
  command: ["/bin/sh"]
  
  args:
  
  - -c
  
  - >-
  
  sleep 36000
  
  volumeMounts:
  
  - mountPath: /metrics
  
  name: metrics
  
  restartPolicy: Never
  
  volumes:
  
  - name: metrics
  
  persistentVolumeClaim:
  
  claimName: metrics
  
  EOF
4. Determine the name of the worker node on which the temporary task is running.
  sudo k0s kubectl get pods -n kuma -o wide | grep tmp-metrics
5. On this worker node, determine the mount point.
  sudo lsblk | grep "<volume_name>"
6. Change to the found directory, /var/lib/k0s/kubelet/pods/....../mount.
Copy the data:
1. Copy the Metrics service data previously saved in the temporary directory to the located /var/lib/k0s/kubelet/pods/....../mount directory, keeping the ownership and permissions of the files. As a result, the contents of the directory should look like this.
  ls -al
  
  total 48
  
  drwxrwsr-x 9 root 65532 4096 Aug 26 10:36 .
  
  drwxr-x--- 3 root root 4096 Aug 26 11:08 ..
  
  drwx--S--- 2 65532 65532 4096 Aug 26 10:36 certificates
  
  drwx--S--- 2 65532 65532 4096 Aug 26 10:36 dashboards
  
  drwxrwsr-x 8 root 65532 4096 Aug 26 10:34 data
  
  drwx--S--- 2 65532 65532 4096 Aug 26 10:36 log
  
  drwxrws--- 2 root 65532 16384 Aug 26 10:34 lost+found
  
  drwx--S--- 2 65532 65532 4096 Aug 26 10:36 rules
  
  drwx--S--- 2 65532 65532 4096 Aug 26 10:36 tmp
2. After copying, go from the mount point to the / or /home directory to avoid blocking the subsequent unmounting of the volume.
Delete the temporary task, pod, PVC, and volume of the Metrics service:
1. Delete the temporary task with the following command:
  sudo k0s kubectl delete job tmp-metrics-job -n kuma
2. Make sure that the metrics-<UID> pod of the temporary task is not present in the output of the following command:
  sudo k0s kubectl get pods -n kuma
3. If the pod does not disappear within a few minutes, force its deletion. The pod name must be taken from the output of the previous command.
  sudo k0s kubectl delete pod tmp-metrics-job-<UID> -n kuma --force
4. Make sure that the Metrics service volume is in the detached state.
  sudo k0s kubectl get volume <volume_name> -n longhorn-system -o json | jq '.status.state'
Start the Metrics service.
1. Run the following command to start the Metrics service with the number of replicas increased to 1.
  sudo k0s kubectl scale deployment/metrics -n kuma --replicas=1
2. Make sure that the Metrics service is running and that the metrics are available in the KUMA web interface.

The Metrics service volume is successfully shrunk without data loss.

Page top