Sunday, March 12, 2017

Linux Resource Control and allocation cgroups with dd process example. users having different resource cap.


Imagine a client want dd process to be limited. Any OS user that runs dd except root user should be limited to read of 2MiB/s. IO rate is specified in bytes per second. root can run dd commands without limit.
1A.
-bash-4.2# cat /usr/lib/systemd/system/path_cgroup.service
[Unit]
Description=creates resource control paths, this is a oneshot command. testing throttling with dd.
After=multi-user.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/bin/bash -c '/usr/bin/sleep 120'
ExecStartPre=/bin/bash -c '/usr/bin/mkdir -p /sys/fs/cgroup/blkio/dd_unlimit'
ExecStartPre=/bin/bash -c '/usr/bin/mkdir -p /sys/fs/cgroup/blkio/dd_limit'
ExecStartPre=/usr/bin/sleep 2
ExecStart=/bin/bash -c '/bin/echo "8:0"  2097152 > /sys/fs/cgroup/blkio/dd_limit/blkio.throttle.read_bps_device'
ExecStart=/bin/bash -c '/bin/echo "8:0"  0 > /sys/fs/cgroup/blkio/dd_unlimit/blkio.throttle.read_bps_device'

1B. Use this preferred method instead 1A. /etc/cgconfig.conf  for persistent in resource controls

vi /etc/cgconfig.conf
group dd_limit {
       blkio {
               blkio.throttle.read_bps_device="8:0  2097152";
               }
       }
group dd_unlimit{
      blkio {
              blkio.throttle.read_bps_device="8:0  0";
              }
       }

2. Load the new configuration. Reload units with
systemctl daemon-reload

3. Confirm status of service and path
-bash-4.2# systemctl status path_cgroup.service
● path_cgroup.service - creates resource control paths, this is a oneshot command. testing throttling with dd.
   Loaded: loaded (/usr/lib/systemd/system/path_cgroup.service; enabled; vendor preset: disabled)
   Active: active (exited) since Sat 2017-03-11 11:13:22 EST; 1h 0min ago
  Process: 2086 ExecStart=/bin/bash -c /bin/echo "8:0"  0 > /sys/fs/cgroup/blkio/dd_unlimit/blkio.throttle.read_bps_device (code=exited, status=0/SUCCESS)
  Process: 2082 ExecStart=/bin/bash -c /bin/echo "8:0"  2097152 > /sys/fs/cgroup/blkio/dd_limit/blkio.throttle.read_bps_device (code=exited, status=0/SUCCESS)
  Process: 2079 ExecStartPre=/usr/bin/sleep 2 (code=exited, status=0/SUCCESS)
  Process: 2076 ExecStartPre=/bin/bash -c /usr/bin/mkdir -p /sys/fs/cgroup/blkio/dd_limit (code=exited, status=0/SUCCESS)
  Process: 2073 ExecStartPre=/bin/bash -c /usr/bin/mkdir -p /sys/fs/cgroup/blkio/dd_unlimit (code=exited, status=0/SUCCESS)
  Process: 1556 ExecStartPre=/bin/bash -c /usr/bin/sleep 120 (code=exited, status=0/SUCCESS)
 Main PID: 2086 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/path_cgroup.service

Mar 10 12:00:33 teits.net systemd[1]: Starting creates resource control paths, this is a oneshot command. testing throttling with dd....
Mar 11 11:13:22 teits.net systemd[1]: Started creates resource control paths, this is a oneshot command. testing throttling with dd..

4. Make the service configuration persistent
-bash-4.2#  systemctl enable path_cgroup.service

5. Add a cgrep rules. This search for the pattern process and place them on appropriate group.
-bash-4.2# vi /etc/cgrules.conf
root:dd       blkio         /sys/fs/cgroup/blkio/dd_unlimit
*:dd       blkio         /sys/fs/cgroup/blkio/dd_limit

Note: First rule which matches the criteria will be executed.

5. Stop and start the cgred rules service.
systemctl stop cgred.service
systemctl start cgred.service

6. Testing Phase
Testing with non-root user

[tobi@teits.net ~]$ dd if=readtest if=testfilein count=100k bs=8k
102400+0 records in
102400+0 records out
838860800 bytes (839 MB) copied, 399.279 s, 2.1 MB/s
[tobi@teits.net ~]$

With root user
-bash-4.2# dd if=readtest if=testfilein count=100k bs=8k
102400+0 records in
102400+0 records out

838860800 bytes (839 MB) copied, 25.4341 s, 33.0 MB/s

Sunday, March 5, 2017

Practical: FIO test with resource control. limiting linux system resource with cgroup



This is a demonstration of using cgroup, now in RedHat 7 control group is administer with systemctl

I will create a service running randow IO read and write. start the service, and limit the CPU quota by 50%.

# cat fio.test1
[random-read]
rw=randread
numjobs=16
direct=0
filename=/dev/sdb1
group_reporting
ioengine=libaio
iodepth=4
write_bw_log=randread
write_iops_log=randread
write_lat_log=randread



Create a service for fio_job

-bash-4.2#  touch /etc/systemd/system/fio_test.service
-bash-4.2# chmod 664 /etc/systemd/system/fio_test.service
-bash-4.2# vi /etc/systemd/system/fio_test.service

-bash-4.2# cat /etc/systemd/system/fio_test.service
[Unit]
Description=Creates fio test by tobi
DefaultDependencies=no
Conflicts=shutdown.target
After=basic.target

[Service]
EnvironmentFile=/root/fio_cgroup/fio.env
WorkingDirectory=/root/fio_cgroup/
ExecStart=/bin/fio /root/fio_cgroup/fio.test
ExecStop=/bin/kill -3 $MAINPID

[Install]
WantedBy=multi-user.target


-bash-4.2# systemctl status fio_test.service
● fio_test.service - Creates fio test by tobi
  Loaded: loaded (/etc/systemd/system/fio_test.service; disabled; vendor preset: disabled)
  Active: inactive (dead)
-bash-4.2#


-bash-4.2# systemctl start fio_test.service

--to enable autostart after reboot, enable service. # systemctl enable fio_test.service

-bash-4.2# systemctl status fio_test.service
● fio_test.service - Creates fio test by tobi
  Loaded: loaded (/etc/systemd/system/fio_test.service; disabled; vendor preset: disabled)
  Active: active (running) since Sat 2017-03-04 17:42:50 EST; 25s ago
Main PID: 24138 (fio)
  CGroup: /system.slice/fio_test.service
          ├─24138 /bin/fio /root/fio_cgroup/fio1.test
          ├─24143 /bin/fio /root/fio_cgroup/fio1.test
          ├─24144 /bin/fio /root/fio_cgroup/fio1.test
          ├─24145 /bin/fio /root/fio_cgroup/fio1.test
          ├─24146 /bin/fio /root/fio_cgroup/fio1.test
          ├─24147 /bin/fio /root/fio_cgroup/fio1.test
          ├─24148 /bin/fio /root/fio_cgroup/fio1.test
          ├─24149 /bin/fio /root/fio_cgroup/fio1.test
          ├─24150 /bin/fio /root/fio_cgroup/fio1.test
          ├─24151 /bin/fio /root/fio_cgroup/fio1.test
          ├─24152 /bin/fio /root/fio_cgroup/fio1.test
          ├─24153 /bin/fio /root/fio_cgroup/fio1.test
          ├─24154 /bin/fio /root/fio_cgroup/fio1.test
          ├─24155 /bin/fio /root/fio_cgroup/fio1.test
          ├─24156 /bin/fio /root/fio_cgroup/fio1.test
          ├─24157 /bin/fio /root/fio_cgroup/fio1.test
          └─24158 /bin/fio /root/fio_cgroup/fio1.test

Mar 04 17:42:50 labserver.teits.com systemd[1]: Started Creates fio test by tobi.
Mar 04 17:42:50 labserver.teits.com systemd[1]: Starting Creates fio test by tobi...
Mar 04 17:42:50 labserver.teits.com fio[24138]: random-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=4
Mar 04 17:42:50 labserver.teits.com fio[24138]: ...
Mar 04 17:42:50 labserver.teits.com fio[24138]: random-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=4
Mar 04 17:42:50 labserver.teits.com fio[24138]: fio-2.0.13
Mar 04 17:42:50 labserver.teits.com fio[24138]: Starting 16 processes
-bash-4.2#

after some minutes: enable resources control. Basically allocating 50 %( maximum) of CPU to fio_test.service
-bash-4.2#  systemctl set-property fio_test.service CPUQuota=50% CPUAccounting=yes

In below result, fio_test service is taking approx. 50% of CPU resources.
-bash-4.2# systemd-cgtop
Path                                                                                                                                                                 Tasks   %CPU   Memory  Input/s Output/s

/                                                                                                                                                                      197   63.8     1.7G        -        -
/system.slice                                                                                                                                                            -   55.9        -        -        -
/system.slice/fio_test.service                                                                                                                                          17   45.3        -        -        -
/system.slice/packagekit.service                                                                                                                                         9    5.4        -        -        -
/system.slice/rtvscand.service                                                                                                                                           1    4.0        -        -        -
/system.slice/symcfgd.service                                                                                                                                            1    3.8        -        -        -
/system.slice/besclient.service                                                                                                                                          2    2.9        -        -        -

Related files to cgroup for this service can be located at

-bash-4.2# ls -l /sys/fs/cgroup/cpu/system.slice/fio_test.service
total 0
-rw-r--r--. 1 root root 0 Mar  4 19:24 cgroup.clone_children
--w--w--w-. 1 root root 0 Mar  4 19:24 cgroup.event_control
-rw-r--r--. 1 root root 0 Mar  4 19:24 cgroup.procs
-r--r--r--. 1 root root 0 Mar  4 19:24 cpuacct.stat
-rw-r--r--. 1 root root 0 Mar  4 19:24 cpuacct.usage
-r--r--r--. 1 root root 0 Mar  4 19:24 cpuacct.usage_percpu
-rw-r--r--. 1 root root 0 Mar  4 19:24 cpu.cfs_period_us
-rw-r--r--. 1 root root 0 Mar  4 19:24 cpu.cfs_quota_us
-rw-r--r--. 1 root root 0 Mar  4 19:24 cpu.rt_period_us
-rw-r--r--. 1 root root 0 Mar  4 19:24 cpu.rt_runtime_us
-rw-r--r--. 1 root root 0 Mar  4 19:24 cpu.shares
-r--r--r--. 1 root root 0 Mar  4 19:24 cpu.stat
-rw-r--r--. 1 root root 0 Mar  4 19:24 notify_on_release
-rw-r--r--. 1 root root 0 Mar  4 19:24 tasks

The following graph are generated using fio_generate_plots scripts. This is a feature of fio utility and gnuplot.
Note the changes in the graph as I decrease the allocation quota for fio_test.service. IOPS decreased, latency increased and MBPS decreased.





With this, you can test variety of scenario, for example, what the maximum IO throughput you can get when you use 2 CPU as compare to 4 CPU cores on the same device.

--you can modify service file, to start the service in specify cpuset. [Service] ExecStartPre=/usr/bin/mkdir -p /sys/fs/cgroup/cpuset/fiocpu ===> Create group to manage process ExecStartPre=/bin/bash -c '/usr/bin/echo "1" > /sys/fs/cgroup/cpuset/fiocpu/cpuset.cpus' ==> Assign cpu core 1 to this process ... ExecStartPost=/bin/bash -c '/usr/bin/echo $MAINPID > /sys/fs/cgroup/cpuset/fiocpu/tasks' ==> Assign process id to group task file