Errors + Resolutions

K8S/Docker Error: No space left on device on /overlay

Space ran out on the /overlay mount and steps taken to mitigate the error "failed to register layer: Error processing tar file(exit status 1)" Resolution a) Expand disk size using qemu-img (KVM Guest VM). b) Resize partition using parted. c) Resize PVM - Physical Volume. d) Extend LVM. e) Resize XFS file system.


    #Guest VM had a disk image of 11GB out of which 8GB was allocated as below which was in-sufficient!

    [root@netlab8 ~]# lsblk
    NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    sr0          11:0    1 1024M  0 rom
    vda         252:0    0   10G  0 disk
    ├─vda1      252:1    0    1G  0 part /boot
    └─vda2      252:2    0    9G  0 part
    ├─cl-root 253:0    0    8G  0 lvm  /
    └─cl-swap 253:1    0    1G  0 lvm
    
    #Original State

    [root@netlab8 ~]# df -h /dev/mapper/cl-root
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/mapper/cl-root  8.0G  6.6G  1.5G  82% /

    [root@netlab8 ~]# lvremove /dev/cl/swap
    Do you really want to remove active logical volume cl/swap? [y/n]: y
      Logical volume "swap" successfully removed.
    
    [root@netlab8 ~]# lvs
    LV   VG Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
    root cl -wi-ao---- 7.99g
    swap cl -wi-a----- 1.00g

    #Shutdown VM and resize image
    [root@panchajanya sdd03]# qemu-img resize netlab7.img +5G
    WARNING: Image format was not specified for 'netlab7.img' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
    Image resized.

    [root@panchajanya sdd03]# ls -lhr
    total 31G
    -rw------- 1 root root 16G Jan 17 19:43 netlab8.img
    -rw------- 1 root root 16G Jan 17 19:43 netlab7.img
    -rw------- 1 qemu qemu 16G Jan 17 19:43 netlab6.img

    # Start the VM and resize the partition within

    [root@netlab8 ~]# parted /dev/vda
    GNU Parted 3.2
    Using /dev/vda
    Welcome to GNU Parted! Type 'help' to view a list of commands.
    (parted) p
    Model: Virtio Block Device (virtblk)
    Disk /dev/vda: 16.1GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags:

    Number  Start   End     Size    Type     File system  Flags
    1      1049kB  1075MB  1074MB  primary  ext4         boot
    2      1075MB  10.7GB  9663MB  primary               lvm

    (parted) resizepart 2 100%
    (parted) p
    Model: Virtio Block Device (virtblk)
    Disk /dev/vda: 16.1GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags:

    Number  Start   End     Size    Type     File system  Flags
    1      1049kB  1075MB  1074MB  primary  ext4         boot
    2      1075MB  16.1GB  15.0GB  primary               lvm


    # Resize the PVM
   
    [root@netlab8 ~]# pvresize /dev/vda2
    Physical volume "/dev/vda2" changed
    1 physical volume(s) resized or updated / 0 physical volume(s) not resized

    # Extend the lvm

    [root@netlab8 ~]# lvextend -l +100%FREE /dev/cl/root
    Size of logical volume cl/root changed from <9.00 GiB (2303 extents) to <14.00 GiB (3583 extents).
    Logical volume cl/root successfully resized.

    [root@netlab8 ~]# lvs
    LV   VG Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
    root cl -wi-ao---- <14.00g

    # Resize the XFS file system.

    [root@netlab8 ~]# xfs_growfs /dev/cl/root
    meta-data=/dev/mapper/cl-root    isize=512    agcount=5, agsize=523776 blks
        =                       sectsz=512   attr=2, projid32bit=1
        =                       crc=1        finobt=1, sparse=1, rmapbt=0
        =                       reflink=1
    data     =                       bsize=4096   blocks=2358272, imaxpct=25
        =                       sunit=0      swidth=0 blks
    naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
    log      =internal log           bsize=4096   blocks=2560, version=2
        =                       sectsz=512   sunit=0 blks, lazy-count=1
    realtime =none                   extsz=4096   blocks=0, rtextents=0
    data blocks changed from 2358272 to 3668992

    [root@netlab8 ~]# lsblk
    NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    sr0          11:0    1 1024M  0 rom
    vda         252:0    0   15G  0 disk
    ├─vda1      252:1    0    1G  0 part /boot
    └─vda2      252:2    0   14G  0 part
    └─cl-root 253:0    0   14G  0 lvm  /
    

K8S - Kubelet error.

Docker using CGROUPFS error resulting in the error. This required the docker startup options to be edited to change from cgroupfs to systemd. systemctl status kubelet followed by jouralctl -xe would indicate the following error "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""


    [root@netlab6 ~]# systemctl status kubelet
    ● kubelet.service - kubelet: The Kubernetes Node Agent
    Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
            └─10-kubeadm.conf
    Active: activating (auto-restart) (Result: exit-code) since Fri 2022-01-14 09:43:16 CST; 976ms ago
        Docs: https://kubernetes.io/docs/
    Process: 442917 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (cod>
    Main PID: 442917 (code=exited, status=1/FAILURE)
    ...skipping...
    ● kubelet.service - kubelet: The Kubernetes Node Agent
    Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
            └─10-kubeadm.conf
    Active: activating (auto-restart) (Result: exit-code) since Fri 2022-01-14 09:43:16 CST; 976ms ago
        Docs: https://kubernetes.io/docs/
    Process: 442917 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (cod>
    Main PID: 442917 (code=exited, status=1/FAILURE)

    [root@netlab6 ~]# journalctl -xn --no-pager

    g52b36a2} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=default] ProductLicense: DefaultAddressPools:[] Warnings:[]}
    Jan 13 21:51:16 netlab6 kubelet[129598]: E0113 21:51:16.944789  129598 server.go:302] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""
    Jan 13 21:51:16 netlab6 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
    Jan 13 21:51:16 netlab6 systemd[1]: kubelet.service: Failed with result 'exit-code'.
    -- Subject: Unit failed
    -- Defined-By: systemd

    #Resolution 

    a) Update docker.service to include - --exec-opt native.cgroupdriver=systemd
    ExecReload=/bin/kill -s HUP $MAINPID

    b) Reload daemon, followed by docker and kubelet service restarts.

    [root@netlab7 ~]# cd /etc/
    [root@netlab7 etc]# grep Exec ./systemd/system/multi-user.target.wants/docker.service
    ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock  --exec-opt native.cgroupdriver=systemd
    ExecReload=/bin/kill -s HUP $MAINPID

    systemctl daemon-reload
    systemctl restart docker
    systemct restart kubelet

    [root@netlab7 etc]# docker info |grep Cg
    Cgroup Driver: systemd
    Cgroup Version: 1


Dracut boot error due to swap volume.

Swap was turned off and volume deleted prior to the installation of the kubernetes as part of pre-reqs but upon reboot host failed to come up. Console logs indicated dracut error indicating that it cannot find the cl/swap volume which was already deleted. Resolution steps: a) Login via virsh Console and type exit b) edit /etc/default/grup and remove references of all swap volumnes from GRUB_CMDLINE_LINUX c) Regenerate the grub config d) Reboot


    [root@netlab6 ~]# vi /etc/default/grub

    [root@netlab6 ~]# grep CMD /etc/default/grub
    GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl/root rhgb quiet"

    [root@netlab6 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
    Generating grub configuration file ...
    done

    #Optional
    dracut --force --regenerate-all