Details on Container Namespaces
# Prep the system.
Image Reference:
https://github.com/baburajk/docker/blob/master/netops/Dockerfile
https://hub.docker.com/repository/docker/baburaj/netops
[root@netlab1 ~]# docker run -p 80:80 -p 443:443 -d --cpu-shares 256 --memory 100m --name netops baburaj/netops
5f18edb5621ab4a6652d85b1e93a28f1c766184c4b39787744c0a76a965b039a
Check if the container is responding.
[root@netlab1 ~]# curl -ks http://localhost:80
Container/HostName : 5f18edb5621a
- Interfaces -
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
34: eth0@if35: mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
[root@netlab1 ~]# curl -ks https://localhost:443
Container/HostName : 5f18edb5621a
- Interfaces -
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
34: eth0@if35: mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
# Refresher on namespaces (Run man namespaces )
Linux provides the following namespaces:
Namespace Constant Isolates
Cgroup CLONE_NEWCGROUP Cgroup root directory
IPC CLONE_NEWIPC System V IPC, POSIX message queues
Network CLONE_NEWNET Network devices, stacks, ports, etc.
Mount CLONE_NEWNS Mount points
PID CLONE_NEWPID Process IDs
User CLONE_NEWUSER User and group IDs
UTS CLONE_NEWUTS Hostname and NIS domain name
# How do we know what namespaces are associated for the container we brought up.
[root@netlab1 ~]# lsns -p $(docker inspect netops | jq '.[].State.Pid')
NS TYPE NPROCS PID USER COMMAND
4026531835 cgroup 153 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 18
4026531837 user 153 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 18
4026532415 mnt 4 6556 root /bin/bash /usr/bin/nginx-start.sh
4026532416 uts 4 6556 root /bin/bash /usr/bin/nginx-start.sh
4026532417 ipc 4 6556 root /bin/bash /usr/bin/nginx-start.sh
4026532418 pid 4 6556 root /bin/bash /usr/bin/nginx-start.sh
4026532420 net 4 6556 root /bin/bash /usr/bin/nginx-start.sh
VETH - How to identify a veth pair for the container?
Note:
Docker stores it’s network name spaces under /var/run/docker/netns but ip netns command looks for
namespaces under /var/run/netns. In order to have ip netns scan and list the docker network namespace
create a symlink.
[root@netlab1 run]# cd /var/run
[root@netlab1 run]# ln -s /var/run/docker/netns netns
[root@netlab1 run]# ip netns list
ac645516b0ce (id: 0)
# How to identify which container is associated with this namespace?
[root@netlab1 ~]# ip netns identify $(docker inspect netops | jq '.[].State.Pid')
ac645516b0ce
This means the container with name netops is associated with the network NS : ac645516b0ce
# Another option is inspect the SandboxKey
[root@netlab1 ~]# docker inspect netops | jq -r '.[].NetworkSettings.SandboxKey'
/var/run/docker/netns/ac645516b0ce
# How to identify which veth ID is associated with teh container?
[root@netlab1 ~]# ip link show
1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 52:54:00:56:0b:aa brd ff:ff:ff:ff:ff:ff
3: docker0: mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:17:61:53:2b brd ff:ff:ff:ff:ff:ff
35: veth09a45c1@if34: mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default
link/ether 1a:bb:7b:b0:c1:ac brd ff:ff:ff:ff:ff:ff link-netns ac645516b0ce
One way is by looking at the link-netns shows the network namespace we identified earlier - ac645516b0ce
ID of the 35: veth09a45c1 pair is 35. While looking inside the container, 35 is associated with eth0@if35. This is one other option to map it..
[root@5f18edb5621a /]# ip addr show
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
34: eth0@if35: mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
# Quick dirty script to map the VETH pairs (!Need to script for a real life scenario with numerous containers)
[root@netlab1 ~]# ip -j addr show | jq -cr '.[] | select (.link_index != null) | { ifindex: .ifindex, link_index: .link_index, ifname: .ifname}'
{"ifindex":35,"link_index":34,"ifname":"veth09a45c1"}
[root@netlab1 ~]# docker exec -it $(docker ps -aq) bash -c 'ip -j addr show ' | jq -cr '.[] | select (.link_index != null) | { ifindex: .ifindex, link_index: .link_index, ifname: .ifname}' | jq -rc
{"ifindex":34,"link_index":35,"ifname":"eth0"}
# Sample output when multiple containers are running
[root@netlab1 ~]# ip -j addr show | jq -cr '.[] | select (.link_index != null) | { ifindex: .ifindex, link_index: .link_index, ifname: .ifname}'
{"ifindex":7,"link_index":6,"ifname":"veth5aa7cf1"}
{"ifindex":9,"link_index":8,"ifname":"veth71c5ad5"}
{"ifindex":11,"link_index":10,"ifname":"veth01db782"}
{"ifindex":13,"link_index":12,"ifname":"vethfeaaec6"}
{"ifindex":17,"link_index":16,"ifname":"vethc231522"}
{"ifindex":19,"link_index":18,"ifname":"vetha8f5171"}
[root@netlab1 ~]# for container in $(docker ps -aq) ;do echo -n [ $container ]: ; docker exec -it $container bash -c 'ip -j addr show ' | jq -cr '.[] | select (.link_index != null) | { ifindex: .ifindex, link_index: .link_index, ifname: .ifname}' | jq -rc; done
[ ae20cefc3ba9 ]:{"ifindex":16,"link_index":17,"ifname":"eth0"}
[ e8824d71df2e ]:{"ifindex":12,"link_index":13,"ifname":"eth0"}
[ 89f74f998006 ]:{"ifindex":10,"link_index":11,"ifname":"eth0"}
[ 413112b311da ]:{"ifindex":8,"link_index":9,"ifname":"eth0"}
[ 70570098d8d0 ]:{"ifindex":6,"link_index":7,"ifname":"eth0"}
[ 28fe6d72a213 ]:{"ifindex":18,"link_index":19,"ifname":"eth0"}
So for [ ae20cefc3ba9 ]:{"ifindex":16,"link_index":17,"ifname":"eth0"}, the link_index 17 indicates it's tied to
vethc231522. Reference - {"ifindex":17,"link_index":16,"ifname":"vethc231522"}
CGROUPS - How to control resources with CGROUPS
[root@a3783e032188 /]# cat test.sh
#!/bin/sh
while [ 1 ]; do
echo "hello world"
sleep 60
done
[root@netlab1 a3783e0321885ef4c97b0132f2199774e503fded2270ec782f19a42f5432ca8e]# echo 10 > pids.max
[root@a3783e032188 /]# sh test.sh &
[3] 118
[root@a3783e032188 /]# hello world
test.sh: fork: retry: Resource temporarily unavailable
test.sh: fork: retry: Resource temporarily unavailable
[root@netlab1 a3783e0321885ef4c97b0132f2199774e503fded2270ec782f19a42f5432ca8e]# pwd
/sys/fs/cgroup/pids/docker/a3783e0321885ef4c97b0132f2199774e503fded2270ec782f19a42f5432ca8e
[root@netlab1 a3783e0321885ef4c97b0132f2199774e503fded2270ec782f19a42f5432ca8e]# cat pids.max
10
[root@netlab1 a3783e0321885ef4c97b0132f2199774e503fded2270ec782f19a42f5432ca8e]# cat pids.current
9
[206067.567038] cgroup: fork rejected by pids controller in /docker/a3783e0321885ef4c97b0132f2199774e503fded2270ec782f19a42f5432ca8e
# Identify the container using too much memory and re-adjust the allocation via CGROUP.
i) Spin a bunch of containers.
[root@netlab1 ~]# for i in `seq 1 5`; do docker run -p ${i}80:80 -p ${i}443:443 -d --cpu-shares 256 --memory 100m --name netops-${i} baburaj/netops; done
70570098d8d0e455703e04f0bb78386caea239e06a9d34d68fdd63b358aaa9d7
413112b311da20e57fb92cdd20cd8a4164ce1ed8ff27f55f56ea5e3e6db4f99f
89f74f99800613f8db06aa200c73a03b3d364936bafc0f3f8f4777762f040ce0
e8824d71df2eddc28f49aa22d583d644b08e70fe85706f5f5073d6e74e5c0fe1
ae20cefc3ba98142f1228166db927dc41874a0a1b2a2f2128e8bcf7de2a6b314
ii) Login to container and initiate stress on the memory.
[root@netlab1 ~]# docker exec -it netops-5 /bin/bash
[root@ae20cefc3ba9 /]# stress-ng -m 1
stress-ng: info: [29] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info: [29] dispatching hogs: 1 vm
systemd-cgtop indicates
Control Group Tasks %CPU Memory Input/s Output/s
/ 407 85.1 1.2G - -
/docker 24 73.0 145.2M - -
/docker/ae20cefc3…7dc41874a0a1b2a2f2128e8bcf7de2a6b314 8 73.0 96.4M - -
/system.slice 265 1.6 753.3M - -
/system.slice/systemd-journald.service 1 0.7 34.3M - -
^C
[root@netlab1 ae20cefc3ba98142f1228166db927dc41874a0a1b2a2f2128e8bcf7de2a6b314]# cd /sys/fs/cgroup/memory/docker/ae20cefc3ba98142f1228166db927dc41874a0a1b2a2f2128e8bcf7de2a6b314
[root@netlab1 ae20cefc3ba98142f1228166db927dc41874a0a1b2a2f2128e8bcf7de2a6b314]# pwd
/sys/fs/cgroup/memory/docker/ae20cefc3ba98142f1228166db927dc41874a0a1b2a2f2128e8bcf7de2a6b314
[root@netlab1 ae20cefc3ba98142f1228166db927dc41874a0a1b2a2f2128e8bcf7de2a6b314]# echo 6291456 > memory.limit_in_bytes
[root@netlab1 ae20cefc3ba98142f1228166db927dc41874a0a1b2a2f2128e8bcf7de2a6b314]# cat memory.limit_in_bytes
6291456
# Run dmesg to see the Output
...
[ 1387.130181] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ae20cefc3ba98142f1228166db927dc41874a0a1b2a2f2128e8bcf7de2a6b314,mems_allowed=0,oom_memcg=/docker/ae20cefc3ba98142f1228166db927dc41874a0a1b2a2f2128e8bcf7de2a6b314,task_memcg=/docker/ae20cefc3ba98142f1228166db927dc41874a0a1b2a2f2128e8bcf7de2a6b314,task=stress-ng,pid=5076,uid=0
[ 1387.130189] Memory cgroup out of memory: Killed process 5076 (stress-ng) total-vm:60692kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB, UID:0
[root@netlab1 ~]# docker exec -it netops-5 /bin/bash
Error response from daemon: Container ae20cefc3ba98142f1228166db927dc41874a0a1b2a2f2128e8bcf7de2a6b314 is not running