Space ran out on the /overlay mount and steps taken to mitigate the error "failed to register layer: Error processing tar file(exit status 1)" Resolution a) Expand disk size using qemu-img (KVM Guest VM). b) Resize partition using parted. c) Resize PVM - Physical Volume. d) Extend LVM. e) Resize XFS file system.
#Guest VM had a disk image of 11GB out of which 8GB was allocated as below which was in-sufficient!
[root@netlab8 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 10G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 9G 0 part
├─cl-root 253:0 0 8G 0 lvm /
└─cl-swap 253:1 0 1G 0 lvm
#Original State
[root@netlab8 ~]# df -h /dev/mapper/cl-root
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cl-root 8.0G 6.6G 1.5G 82% /
[root@netlab8 ~]# lvremove /dev/cl/swap
Do you really want to remove active logical volume cl/swap? [y/n]: y
Logical volume "swap" successfully removed.
[root@netlab8 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root cl -wi-ao---- 7.99g
swap cl -wi-a----- 1.00g
#Shutdown VM and resize image
[root@panchajanya sdd03]# qemu-img resize netlab7.img +5G
WARNING: Image format was not specified for 'netlab7.img' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
Image resized.
[root@panchajanya sdd03]# ls -lhr
total 31G
-rw------- 1 root root 16G Jan 17 19:43 netlab8.img
-rw------- 1 root root 16G Jan 17 19:43 netlab7.img
-rw------- 1 qemu qemu 16G Jan 17 19:43 netlab6.img
# Start the VM and resize the partition within
[root@netlab8 ~]# parted /dev/vda
GNU Parted 3.2
Using /dev/vda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 16.1GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 1075MB 1074MB primary ext4 boot
2 1075MB 10.7GB 9663MB primary lvm
(parted) resizepart 2 100%
(parted) p
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 16.1GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 1075MB 1074MB primary ext4 boot
2 1075MB 16.1GB 15.0GB primary lvm
# Resize the PVM
[root@netlab8 ~]# pvresize /dev/vda2
Physical volume "/dev/vda2" changed
1 physical volume(s) resized or updated / 0 physical volume(s) not resized
# Extend the lvm
[root@netlab8 ~]# lvextend -l +100%FREE /dev/cl/root
Size of logical volume cl/root changed from <9.00 GiB (2303 extents) to <14.00 GiB (3583 extents).
Logical volume cl/root successfully resized.
[root@netlab8 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root cl -wi-ao---- <14.00g
# Resize the XFS file system.
[root@netlab8 ~]# xfs_growfs /dev/cl/root
meta-data=/dev/mapper/cl-root isize=512 agcount=5, agsize=523776 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=2358272, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 2358272 to 3668992
[root@netlab8 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 15G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 14G 0 part
└─cl-root 253:0 0 14G 0 lvm /
Docker using CGROUPFS error resulting in the error. This required the docker startup options to be edited to change from cgroupfs to systemd. systemctl status kubelet followed by jouralctl -xe would indicate the following error "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""
[root@netlab6 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Fri 2022-01-14 09:43:16 CST; 976ms ago
Docs: https://kubernetes.io/docs/
Process: 442917 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (cod>
Main PID: 442917 (code=exited, status=1/FAILURE)
...skipping...
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Fri 2022-01-14 09:43:16 CST; 976ms ago
Docs: https://kubernetes.io/docs/
Process: 442917 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (cod>
Main PID: 442917 (code=exited, status=1/FAILURE)
[root@netlab6 ~]# journalctl -xn --no-pager
g52b36a2} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=default] ProductLicense: DefaultAddressPools:[] Warnings:[]}
Jan 13 21:51:16 netlab6 kubelet[129598]: E0113 21:51:16.944789 129598 server.go:302] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""
Jan 13 21:51:16 netlab6 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Jan 13 21:51:16 netlab6 systemd[1]: kubelet.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
#Resolution
a) Update docker.service to include - --exec-opt native.cgroupdriver=systemd
ExecReload=/bin/kill -s HUP $MAINPID
b) Reload daemon, followed by docker and kubelet service restarts.
[root@netlab7 ~]# cd /etc/
[root@netlab7 etc]# grep Exec ./systemd/system/multi-user.target.wants/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd
ExecReload=/bin/kill -s HUP $MAINPID
systemctl daemon-reload
systemctl restart docker
systemct restart kubelet
[root@netlab7 etc]# docker info |grep Cg
Cgroup Driver: systemd
Cgroup Version: 1
Swap was turned off and volume deleted prior to the installation of the kubernetes as part of pre-reqs but upon reboot host failed to come up. Console logs indicated dracut error indicating that it cannot find the cl/swap volume which was already deleted. Resolution steps: a) Login via virsh Console and type exit b) edit /etc/default/grup and remove references of all swap volumnes from GRUB_CMDLINE_LINUX c) Regenerate the grub config d) Reboot
[root@netlab6 ~]# vi /etc/default/grub
[root@netlab6 ~]# grep CMD /etc/default/grub
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rhgb quiet"
[root@netlab6 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
done
#Optional
dracut --force --regenerate-all