Sunday, March 5, 2017

Practical: FIO test with resource control. limiting linux system resource with cgroup



This is a demonstration of using cgroup, now in RedHat 7 control group is administer with systemctl

I will create a service running randow IO read and write. start the service, and limit the CPU quota by 50%.

# cat fio.test1
[random-read]
rw=randread
numjobs=16
direct=0
filename=/dev/sdb1
group_reporting
ioengine=libaio
iodepth=4
write_bw_log=randread
write_iops_log=randread
write_lat_log=randread



Create a service for fio_job

-bash-4.2#  touch /etc/systemd/system/fio_test.service
-bash-4.2# chmod 664 /etc/systemd/system/fio_test.service
-bash-4.2# vi /etc/systemd/system/fio_test.service

-bash-4.2# cat /etc/systemd/system/fio_test.service
[Unit]
Description=Creates fio test by tobi
DefaultDependencies=no
Conflicts=shutdown.target
After=basic.target

[Service]
EnvironmentFile=/root/fio_cgroup/fio.env
WorkingDirectory=/root/fio_cgroup/
ExecStart=/bin/fio /root/fio_cgroup/fio.test
ExecStop=/bin/kill -3 $MAINPID

[Install]
WantedBy=multi-user.target


-bash-4.2# systemctl status fio_test.service
● fio_test.service - Creates fio test by tobi
  Loaded: loaded (/etc/systemd/system/fio_test.service; disabled; vendor preset: disabled)
  Active: inactive (dead)
-bash-4.2#


-bash-4.2# systemctl start fio_test.service

--to enable autostart after reboot, enable service. # systemctl enable fio_test.service

-bash-4.2# systemctl status fio_test.service
● fio_test.service - Creates fio test by tobi
  Loaded: loaded (/etc/systemd/system/fio_test.service; disabled; vendor preset: disabled)
  Active: active (running) since Sat 2017-03-04 17:42:50 EST; 25s ago
Main PID: 24138 (fio)
  CGroup: /system.slice/fio_test.service
          ├─24138 /bin/fio /root/fio_cgroup/fio1.test
          ├─24143 /bin/fio /root/fio_cgroup/fio1.test
          ├─24144 /bin/fio /root/fio_cgroup/fio1.test
          ├─24145 /bin/fio /root/fio_cgroup/fio1.test
          ├─24146 /bin/fio /root/fio_cgroup/fio1.test
          ├─24147 /bin/fio /root/fio_cgroup/fio1.test
          ├─24148 /bin/fio /root/fio_cgroup/fio1.test
          ├─24149 /bin/fio /root/fio_cgroup/fio1.test
          ├─24150 /bin/fio /root/fio_cgroup/fio1.test
          ├─24151 /bin/fio /root/fio_cgroup/fio1.test
          ├─24152 /bin/fio /root/fio_cgroup/fio1.test
          ├─24153 /bin/fio /root/fio_cgroup/fio1.test
          ├─24154 /bin/fio /root/fio_cgroup/fio1.test
          ├─24155 /bin/fio /root/fio_cgroup/fio1.test
          ├─24156 /bin/fio /root/fio_cgroup/fio1.test
          ├─24157 /bin/fio /root/fio_cgroup/fio1.test
          └─24158 /bin/fio /root/fio_cgroup/fio1.test

Mar 04 17:42:50 labserver.teits.com systemd[1]: Started Creates fio test by tobi.
Mar 04 17:42:50 labserver.teits.com systemd[1]: Starting Creates fio test by tobi...
Mar 04 17:42:50 labserver.teits.com fio[24138]: random-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=4
Mar 04 17:42:50 labserver.teits.com fio[24138]: ...
Mar 04 17:42:50 labserver.teits.com fio[24138]: random-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=4
Mar 04 17:42:50 labserver.teits.com fio[24138]: fio-2.0.13
Mar 04 17:42:50 labserver.teits.com fio[24138]: Starting 16 processes
-bash-4.2#

after some minutes: enable resources control. Basically allocating 50 %( maximum) of CPU to fio_test.service
-bash-4.2#  systemctl set-property fio_test.service CPUQuota=50% CPUAccounting=yes

In below result, fio_test service is taking approx. 50% of CPU resources.
-bash-4.2# systemd-cgtop
Path                                                                                                                                                                 Tasks   %CPU   Memory  Input/s Output/s

/                                                                                                                                                                      197   63.8     1.7G        -        -
/system.slice                                                                                                                                                            -   55.9        -        -        -
/system.slice/fio_test.service                                                                                                                                          17   45.3        -        -        -
/system.slice/packagekit.service                                                                                                                                         9    5.4        -        -        -
/system.slice/rtvscand.service                                                                                                                                           1    4.0        -        -        -
/system.slice/symcfgd.service                                                                                                                                            1    3.8        -        -        -
/system.slice/besclient.service                                                                                                                                          2    2.9        -        -        -

Related files to cgroup for this service can be located at

-bash-4.2# ls -l /sys/fs/cgroup/cpu/system.slice/fio_test.service
total 0
-rw-r--r--. 1 root root 0 Mar  4 19:24 cgroup.clone_children
--w--w--w-. 1 root root 0 Mar  4 19:24 cgroup.event_control
-rw-r--r--. 1 root root 0 Mar  4 19:24 cgroup.procs
-r--r--r--. 1 root root 0 Mar  4 19:24 cpuacct.stat
-rw-r--r--. 1 root root 0 Mar  4 19:24 cpuacct.usage
-r--r--r--. 1 root root 0 Mar  4 19:24 cpuacct.usage_percpu
-rw-r--r--. 1 root root 0 Mar  4 19:24 cpu.cfs_period_us
-rw-r--r--. 1 root root 0 Mar  4 19:24 cpu.cfs_quota_us
-rw-r--r--. 1 root root 0 Mar  4 19:24 cpu.rt_period_us
-rw-r--r--. 1 root root 0 Mar  4 19:24 cpu.rt_runtime_us
-rw-r--r--. 1 root root 0 Mar  4 19:24 cpu.shares
-r--r--r--. 1 root root 0 Mar  4 19:24 cpu.stat
-rw-r--r--. 1 root root 0 Mar  4 19:24 notify_on_release
-rw-r--r--. 1 root root 0 Mar  4 19:24 tasks

The following graph are generated using fio_generate_plots scripts. This is a feature of fio utility and gnuplot.
Note the changes in the graph as I decrease the allocation quota for fio_test.service. IOPS decreased, latency increased and MBPS decreased.





With this, you can test variety of scenario, for example, what the maximum IO throughput you can get when you use 2 CPU as compare to 4 CPU cores on the same device.

--you can modify service file, to start the service in specify cpuset. [Service] ExecStartPre=/usr/bin/mkdir -p /sys/fs/cgroup/cpuset/fiocpu ===> Create group to manage process ExecStartPre=/bin/bash -c '/usr/bin/echo "1" > /sys/fs/cgroup/cpuset/fiocpu/cpuset.cpus' ==> Assign cpu core 1 to this process ... ExecStartPost=/bin/bash -c '/usr/bin/echo $MAINPID > /sys/fs/cgroup/cpuset/fiocpu/tasks' ==> Assign process id to group task file


No comments:

Post a Comment