Open Stack

I have an openstack ocata release deployed via kolla.

Libvirtd running inside docker container nova_ libvirt and volumes
/sys/fs/cgroup, /run privileged mode enabled.

Some guest vms cannot provide cpu-stats

Symptoms are:

> $ docker exec -ti nova_libvirt virsh cpu-stats instance-000004cb
> error: Failed to retrieve CPU statistics for domain 'instance-000004cb'
> error: Requested operation is not valid: cgroup CPUACCT controller is not
> mounted


To check cgroups looking for all related pid

> $ ps fax | grep instance-000004cb
> 8275 ? Sl 4073:40 /usr/libexec/qemu-kvm -name guest=instance-000004cb
> $ ps fax | grep 8275
> 8346 ? S 76:04 \_ [vhost-8275]
> 8367 ? S 0:00 \_ [kvm-pit/8275]
> 8275 ? Sl 4073:42 /usr/libexec/qemu-kvm


See cgroup
for qemu-kvm

> $ cat /proc/8275/cgroup
> 11:blkio:/user.slice
> 10:devices:/user.slice
>
> 9:hugetlb:/docker/e5bef89178c1c3ae34fd2b4a9b86b299a6145c0b9f608a06e83f6f4ca4d897bd
> 8:cpuacct,cpu:/user.slice
>
> 7:perf_event:/machine.slice/machine-qemu\x2d25\x2dinstance\x2d000004cb.scope
>
> 6:net_prio,net_cls:/machine.slice/machine-qemu\x2d25\x2dinstance\x2d000004cb.scope
> 5:freezer:/machine.slice/machine-qemu\x2d25\x2dinstance\x2d000004cb.scope
> 4:memory:/user.slice
> 3:pids:/user.slice
>
> 2:cpuset:/machine.slice/machine-qemu\x2d25\x2dinstance\x2d000004cb.scope/emulator
> 1:name=systemd:/user.slice/user-0.slice/session-c1068.scope


for vhost-8275

> $ cat /proc/8346/cgroup
> 11:blkio:/user.slice
> 10:devices:/user.slice
>
> 9:hugetlb:/docker/e5bef89178c1c3ae34fd2b4a9b86b299a6145c0b9f608a06e83f6f4ca4d897bd
> 8:cpuacct,cpu:/user.slice
>
> 7:perf_event:/machine.slice/machine-qemu\x2d25\x2dinstance\x2d000004cb.scope
>
> 6:net_prio,net_cls:/machine.slice/machine-qemu\x2d25\x2dinstance\x2d000004cb.scope
> 5:freezer:/machine.slice/machine-qemu\x2d25\x2dinstance\x2d000004cb.scope
> 4:memory:/user.slice
> 3:pids:/user.slice
>
> 2:cpuset:/machine.slice/machine-qemu\x2d25\x2dinstance\x2d000004cb.scope/emulator
> 1:name=systemd:/user.slice/user-0.slice/session-c1068.scope


for kvm-pit

> $ cat /proc/8275/cgroup
> 11:blkio:/user.slice
> 10:devices:/user.slice
> 9:hugetlb:/
> 8:cpuacct,cpu:/user.slice
> 7:perf_event:/
> 6:net_prio,net_cls:/
> 5:freezer:/
> 4:memory:/user.slice
> 3:pids:/user.slice
> 2:cpuset:/
> 1:name=systemd:/user.slice/user-0.slice/session-c4807.scope


I tried to fix the groups with a this script

> get_broken_vms() {
>     docker exec nova_libvirt bash -c 'for vm in $(virsh list --name); do
> virsh cpu-stats $vm > /dev/null 2>&1 || echo $vm; done'
> }
>
> attach_vm_to_cgroup() {
>     # Attach processes and their threads pid to correct cgroup
>     local vm_pid=$1; shift
>     local vm_cgname=$1; shift
>
>     echo Fix cgroup for pid $vm_pid in cgroup $vm_cgname
>
>     for tpid in $(find /proc/$vm_pid/task/ -maxdepth 1 -mindepth 1 -type d
> -printf '%f\n'); do
>         echo $tpid | tee
> /sys/fs/cgroup/{blkio,devices,perf_event,net_prio,net_cls,freezer,memory,pids,systemd}/machine.slice/$vm_cgname/tasks
> 1>/dev/null &
>         echo $tpid | tee
> /sys/fs/cgroup/{cpu,\cpuacct,cpuset}/machine.slice/$vm_cgname/emulator/tasks
> 1>/dev/null &
>     done
> }
>
> for vm in $(get_broken_vms); do
>     vm_pid=$(pgrep -f $vm)
>     vm_vhost_pids=$(pgrep -x vhost-$vm_pid)
>     vm_cgname=$(find /sys/fs/cgroup/systemd/machine.slice -maxdepth 1
> -mindepth 1 -type d -name "machine-qemu\\\x2d*\\\x2d${vm/-/\\\\x2d}.scope"
> -printf '%f\n')
>
>     echo Working on vm: $vm pid: $vm_pid vhost_pid: $vm_vhost_pids
> cgroup_name: $vm_cgname
>     [ -z "$vm_pid" -a -z "$vm_cgname" ] || attach_vm_to_cgroup $vm_pid
> $vm_cgname
>
>     # Fix vhost-NNNN kernel threads
>     for vpid in $vm_vhost_pids; do
>         [ -z "$vm_cgname" ] || attach_vm_to_cgroup $vpid $vm_cgname
>     done
> done


After fixing all vms successfully provided cpu-stats and other metrics, but
after some hours cgroups broke again.

Problems and symptoms:

- cgoup broken not at all VMs
- to find out what leads to this effect failed
- if restart a problem VM then as expected cgroups has been fixed but after
some hours cgroup broken again
- if cgroups has been fixed by hand cpu-stats is works, but after some
hours cgroup broken again

Now i check:
- logrotate - nothing
- cron - nothing

Add audit logs for cgrups

> auditctl -w '/sys/fs/cgroup/cpu,cpuacct/machine.slice' -p rwxa

And found only libvirtd processes write cgroups.

Any suggestions?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/community/attachments/20201110/feb8c788/attachment.html>