3. PerfAccel Analytics & Caching in a Complex & Heterogeneous Application I/O Environment

As we discussed in the last blog, this time we will see how the PerfAccel Caching & Analytics looks like in a more complex I/O infrastructure involving multiple Applications and data sources. We will create such an environment by combining NFS shares as well as local Ext4 and XFS Mounts. This will demonstrate how seamless and application agnostic PerfAccel is.

We have a compute node with the following mounts :

# df -h

Filesystem                                         Size  Used Avail Use% Mounted on
 /dev/sda1                                           11G  7.4G  2.7G  74% /
 tmpfs                                              1.1G     0  1.1G   0% /dev/shm
 192.168.1.178:/data_ext4/critical/app/ecad_module  349G  290G   42G  88% /app_data
 192.168.1.37:/data/critical/app/maya_render         15G   12G  2.7G  82% /maya_render
 /dev/sde                                            12G  158M   12G   2% /vlsi_tools
 /dev/sdb                                            12G   33M   12G   1% /autodesk_cad

2 NFS shares mounted on /app_data and /maya_render
1 Ext4 device mounted on /vlsi_tools
1 XFS device mounted on /autodesk_cad

Now, we will not be using the real workloads as mentioned in the directory name for today’s demo. This is just for demonstration that how a deployment node can look like in an enterprise environment with different kinds of I/O workloads. We will go through individual workloads as separate blogs.

This blog is to demonstrate how PerfAccel seamlessly deals with multiple mount points of different types. Here, we  have three types i.e. NFS3, Ext4, XFS
Let’s create the cache first from /dev/md127

# dgpctl -cache list

Device        Total size   Free size  Alloc size  State    IP address      Name
/dev/md127       23.6 GB      1.5 GB     22.0 GB  Online   192.168.1.38    ssd_cache

Now, let’s create 4 sources for the directories /app_data, /maya_render, /vlsi_tools, /autodesk_cad,  each with 5GB of cache space.
So, each source here can be treated as the Dataset for a particular Application. And we will find out about the I/O Analytics of each Application

# dgpctl -source list

Source                          Status       Cache           Size       Mount Point     Type       Mode
Critical_app                    ONLINE       ssd_cache       5.0 GB     /app_data       nfs        WT
maya_render                     ONLINE       ssd_cache       5.0 GB     /maya_render    nfs        WT
vlsi_tools                      ONLINE       ssd_cache       5.0 GB     /vlsi_tools     ext4       WT
autodesk_cad                    ONLINE       ssd_cache       5.0 GB     /autodesk_cad   xfs        WT

  

# df -h

Filesystem                                         Size  Used Avail Use% Mounted on
/dev/sda1                                           11G  7.4G  2.7G  74% /
tmpfs                                              1.1G     0  1.1G   0% /dev/shm
192.168.1.178:/data_ext4/critical/app/ecad_module  349G  290G   42G  88% /app_data
192.168.1.37:/data/critical/app/maya_render         15G   12G  2.7G  82% /maya_render
/dev/sde                                            12G  158M   12G   2% /vlsi_tools
/dev/sdb                                            12G   33M   12G   1% /autodesk_cad
critical_app                                       349G  290G   42G  88% /app_data
maya_render                                         15G   12G  2.7G  82% /maya_render
vlsi_tools                                          12G  158M   12G   2% /vlsi_tools
autodesk_cad                                        12G   33M   12G   1% /autodesk_cad

Let’s execute the related workloads on the sources one after another and then go through all the PerfAccel I/O Analytics information at the end.

1. I/O on critical_app  (/app_data)

We will use filebench tool to execute a webserver workload on this source with the following config.
Mainly, the workload will run for 1500 secs and will operate on about 100000 files

filebench> load webserver
27163: 247.825: Web-server Version 3.0 personality successfully loaded
27163: 247.825: Usage: set $dir=<dir>
27163: 247.825:        set $meanfilesize=<size>   defaults to 16384
27163: 247.825:        set $nfiles=<value>    defaults to 1000
27163: 247.825:        set $meandirwidth=<value>  defaults to 20
27163: 247.826:        set $nthreads=<value>  defaults to 100
27163: 247.826:        set $iosize=<size>     defaults to 1048576
27163: 247.826:        run runtime (e.g. run 60)
filebench> set $dir=/app_data
filebench> set $nfiles=100000
filebench> set runtime 1500
filebench> run 1500

27163: 283.578: Creating/pre-allocating files and filesets
27163: 283.619: Fileset logfiles: 1 files, 0 leafdirs, avg dir width = 20, avg dir depth = 0.0, 0.002MB
27163: 283.623: Removed any existing fileset logfiles in 1 seconds
27163: 283.623: making tree for filset /app_data/logfiles
27163: 283.675: Creating fileset logfiles...
27163: 283.809: Preallocated 1 of 1 of fileset logfiles in 1 seconds
27163: 283.911: Fileset bigfileset: 100000 files, 0 leafdirs, avg dir width = 20, avg dir depth = 3.8, 1563.471MB
27163: 283.915: Removed any existing fileset bigfileset in 1 seconds
27163: 283.915: making tree for filset /app_data/bigfileset
27163: 294.919: Creating fileset bigfileset...
27163: 3696.114: Preallocated 100000 of 100000 of fileset bigfileset in 3402 seconds
27163: 3696.158: waiting for fileset pre-allocation to finish
27570: 3696.173: Starting 1 filereader instances
27571: 3696.242: Starting 100 filereaderthread threads
27163: 3699.444: Running...
27163: 5201.972: Run took 1500 seconds...
27163: 5202.122: Per-Operation Breakdown


appendlog            84479ops       56ops/s   0.4mb/s      3.9ms/op    45584us/op-cpu [0ms - 2738ms]
closefile10          84382ops       56ops/s   0.0mb/s      8.0ms/op    45206us/op-cpu [0ms - 2558ms]
readfile10           84385ops       56ops/s   0.9mb/s     13.1ms/op    48050us/op-cpu [0ms - 3679ms]
openfile10           84388ops       56ops/s   0.0mb/s     21.4ms/op    57785us/op-cpu [0ms - 3653ms]
closefile9           84391ops       56ops/s   0.0mb/s      8.0ms/op    45292us/op-cpu [0ms - 3597ms]
readfile9            84393ops       56ops/s   0.9mb/s     12.4ms/op    48130us/op-cpu [0ms - 3691ms]
openfile9            84396ops       56ops/s   0.0mb/s     21.2ms/op    58016us/op-cpu [0ms - 3662ms]
closefile8           84400ops       56ops/s   0.0mb/s      7.9ms/op    45159us/op-cpu [0ms - 3597ms]
readfile8            84405ops       56ops/s   0.9mb/s     12.6ms/op    48151us/op-cpu [0ms - 3690ms]
openfile8            84407ops       56ops/s   0.0mb/s     20.7ms/op    58135us/op-cpu [0ms - 3015ms]
closefile7           84412ops       56ops/s   0.0mb/s      8.2ms/op    45358us/op-cpu [0ms - 3063ms]
readfile7            84415ops       56ops/s   0.9mb/s     13.3ms/op    48207us/op-cpu [0ms - 3517ms]
openfile7            84416ops       56ops/s   0.0mb/s     21.5ms/op    58347us/op-cpu [0ms - 3673ms]
closefile6           84417ops       56ops/s   0.0mb/s      7.9ms/op    45085us/op-cpu [0ms - 3597ms]
readfile6            84419ops       56ops/s   0.9mb/s     13.7ms/op    48090us/op-cpu [0ms - 3691ms]
openfile6            84426ops       56ops/s   0.0mb/s     20.8ms/op    58067us/op-cpu [0ms - 3656ms]
closefile5           84428ops       56ops/s   0.0mb/s      8.3ms/op    45226us/op-cpu [0ms - 3597ms]
readfile5            84430ops       56ops/s   0.9mb/s     13.2ms/op    48015us/op-cpu [0ms - 3677ms]
openfile5            84432ops       56ops/s   0.0mb/s     20.5ms/op    57830us/op-cpu [0ms - 2775ms]
closefile4           84440ops       56ops/s   0.0mb/s      8.3ms/op    45307us/op-cpu [0ms - 3597ms]
readfile4            84443ops       56ops/s   0.9mb/s     13.9ms/op    47946us/op-cpu [0ms - 3678ms]
openfile4            84449ops       56ops/s   0.0mb/s     20.9ms/op    58100us/op-cpu [0ms - 3690ms]
closefile3           84453ops       56ops/s   0.0mb/s      8.1ms/op    45238us/op-cpu [0ms - 3597ms]
readfile3            84456ops       56ops/s   0.9mb/s     13.5ms/op    48100us/op-cpu [0ms - 3671ms]
openfile3            84459ops       56ops/s   0.0mb/s     20.8ms/op    57788us/op-cpu [0ms - 3688ms]
closefile2           84460ops       56ops/s   0.0mb/s      8.3ms/op    45633us/op-cpu [0ms - 2447ms]
readfile2            84461ops       56ops/s   0.9mb/s     13.2ms/op    47810us/op-cpu [0ms - 3690ms]
openfile2            84464ops       56ops/s   0.0mb/s     21.2ms/op    57434us/op-cpu [0ms - 3663ms]
closefile1           84467ops       56ops/s   0.0mb/s      7.8ms/op    45934us/op-cpu [0ms - 2936ms]
readfile1            84477ops       56ops/s   0.9mb/s     14.2ms/op    47585us/op-cpu [0ms - 3690ms]
openfile1            84479ops       56ops/s   0.0mb/s     25.0ms/op    57671us/op-cpu [0ms - 3675ms]
27163: 5202.122: IO Summary: 2617329 ops, 1741.974 ops/s, (562/56 r/w),   9.2mb/s,   1549us cpu/op,  39.3ms latency
27163: 5202.122: Shutting down processes

2. I/O on maya_render (/maya_render)
We will use filebench to generate a varmail workload with the following configuration. Mainly, the workload will run for 1500 secs and will operate in 100000 files with 20 threads

filebench> load varmail
8178: 4.022: Varmail Version 3.0 personality successfully loaded
8178: 4.022: Usage: set $dir=<dir>
8178: 4.022:        set $meanfilesize=<size>    defaults to 16384
8178: 4.022:        set $nfiles=<value>     defaults to 1000
8178: 4.022:        set $nthreads=<value>   defaults to 16
8178: 4.022:        set $meanappendsize=<value> defaults to 16384
8178: 4.022:        set $iosize=<size>  defaults to 1048576
8178: 4.022:        set $meandirwidth=<size> defaults to 1000000
8178: 4.022:        run runtime (e.g. run 60)
filebench> set $dir=/maya_render
filebench> set $nfiles=100000
filebench> set $nthreads=20
filebench> run 1500

8178: 37.851: Creating/pre-allocating files and filesets
8178: 37.933: Fileset bigfileset: 100000 files, 0 leafdirs, avg dir width = 1000000, avg dir depth = 0.8, 1563.079MB
8178: 208.263: Removed any existing fileset bigfileset in 171 seconds
8178: 208.263: making tree for filset /maya_render/bigfileset
8178: 208.276: Creating fileset bigfileset...
8178: 642.598: Preallocated 80075 of 100000 of fileset bigfileset in 435 seconds
8178: 642.598: waiting for fileset pre-allocation to finish
9443: 642.599: Starting 1 filereader instances
9444: 642.605: Starting 20 filereaderthread threads
8178: 643.608: Running...
8178: 2145.341: Run took 1500 seconds...
8178: 2145.904: Per-Operation Breakdown

closefile4           110002ops       73ops/s   0.0mb/s      3.1ms/op     5229us/op-cpu [0ms - 3262ms]
readfile4            110002ops       73ops/s   1.1mb/s      4.8ms/op     7229us/op-cpu [0ms - 1171ms]
openfile4            110003ops       73ops/s   0.0mb/s     38.1ms/op    39245us/op-cpu [0ms - 1708ms]
closefile3           110004ops       73ops/s   0.0mb/s      0.0ms/op     3831us/op-cpu [0ms - 27ms]
fsyncfile3           110014ops       73ops/s   0.0mb/s     22.3ms/op    14324us/op-cpu [0ms - 3251ms]
appendfilerand3      110021ops       73ops/s   0.6mb/s      5.4ms/op     5200us/op-cpu [0ms - 3238ms]
readfile3            110021ops       73ops/s   1.1mb/s      5.9ms/op     6991us/op-cpu [0ms - 1913ms]
openfile3            110021ops       73ops/s   0.0mb/s     39.5ms/op    39892us/op-cpu [0ms - 1700ms]
closefile2           110021ops       73ops/s   0.0mb/s      0.0ms/op     3179us/op-cpu [0ms - 8ms]
fsyncfile2           110021ops       73ops/s   0.0mb/s      8.3ms/op     9302us/op-cpu [0ms - 1028ms]
appendfilerand2      110021ops       73ops/s   0.6mb/s      0.0ms/op     3326us/op-cpu [0ms - 52ms]
createfile2          110021ops       73ops/s   0.0mb/s     42.3ms/op    42279us/op-cpu [0ms - 1699ms]
deletefile1          110021ops       73ops/s   0.0mb/s     41.6ms/op    41940us/op-cpu [0ms - 1319ms]
8178: 2145.904: IO Summary: 1430193 ops, 952.006 ops/s, (146/146 r/w),   3.3mb/s,   2359us cpu/op,  52.8ms latency

3. I/O on vlsi_source

Download openssl from https://www.openssl.org/source/openssl-1.1.0e.tar.gz on the source directory
Here, we will compile openssl multiple times to represent build & compile workloads
openssl-1.0.2-beta1.tar.gz
Untar in the source using tar -xvf openssl-1.0.2-beta1.tar.gz
And we will make 4 copies of the extracted directory openssl-1.0.2-beta1
openssl-1.0.2-beta2  openssl-1.0.2-beta3  openssl-1.0.2-beta4  openssl-1.0.2-beta5

Now, we will run the following in parallel in each directory
for i in `seq 1 4`; do make clean; sync; drop_cache; make; done

 

4. I/O on autodesk_cad ( /autodesk_cad)
We will use filebench to generate a Random i/o workload with the following configuration.
Mainly, the workload will run for 900 secs and will operate in 100000 files with 16 threads doing the I/O

filebench> load randomfileaccess
38291: 28.959: NetworkServer Version 1.1 personality successfully loaded
38291: 28.959: Usage: set $dir=<dir>            defaults to /tmp
38291: 28.959:        set $cached=<bool>        defaults to false
38291: 28.959:        set $wrtiosize.type=<type>   defaults to gamma
38291: 28.959:        set $wrtiosize.randsrc=<src> defaults to rand48
38291: 28.959:        set $wrtiosize.mean=<mean>   defaults to 16384
38291: 28.959:        set $wrtiosize.gamma=<gamma> defaults to 1500
38291: 28.959:        set $rdiosize.type=<type>   defaults to tabular
38291: 28.959:        set $rdiosize.randsrc=<src> defaults to rand48
38291: 28.959:        set $filesize.type=<type>   defaults to tabular
38291: 28.959:        set $filesize.randsrc=<src> defaults to rand48
38291: 28.959:        set $nfiles=<value>       defaults to 10000
38291: 28.959:        set $nthreads=<value>     defaults to 5
38291: 28.959:        set $sync=<bool>          defaults to false
38291: 28.959:
38291: 28.959:        run runtime (e.g. run 60)
filebench> set $dir=/autodesk_cad
filebench> set $wrtiosize.mean=32768
filebench> set $nfiles=100000
filebench> set $nthreads=16
filebench> set $sync=1
filebench> run 900
38291: 86.197: Creating/pre-allocating files and filesets
38291: 86.257: Fileset bigfileset: 100000 files, 0 leafdirs, avg dir width = 20, avg dir depth = 3.8, 2539.062MB
38291: 86.259: Removed any existing fileset bigfileset in 1 seconds
38291: 86.259: making tree for filset /autodesk_cad/bigfileset
38291: 86.875: Creating fileset bigfileset...
38291: 183.503: Preallocated 100000 of 100000 of fileset bigfileset in 97 seconds
38291: 183.504: waiting for fileset pre-allocation to finish
41467: 183.508: Starting 1 netclient instances
41468: 183.553: Starting 16 fileuser threads
38291: 184.554: Running...

38291: 1094.024: Run took 900 seconds...
38291: 1094.026: Per-Operation Breakdown
closefile3           70410ops       77ops/s   0.0mb/s      0.3ms/op     5225us/op-cpu [0ms - 6670ms]
closefile2           70410ops       77ops/s   0.0mb/s      0.4ms/op     6444us/op-cpu [0ms - 7786ms]
readfile2            70410ops       77ops/s 2559.2mb/s     57.8ms/op   115599us/op-cpu [0ms - 17248ms]
readfile1            70420ops       77ops/s 2529.9mb/s     57.5ms/op   117927us/op-cpu [0ms - 21791ms]
closefile1           70426ops       77ops/s   0.0mb/s      0.0ms/op     4971us/op-cpu [0ms - 20ms]
appendfilerand1      70426ops       77ops/s   1.2mb/s     36.2ms/op    50584us/op-cpu [0ms - 25131ms]
openfile3            70426ops       77ops/s   0.0mb/s      2.8ms/op     7236us/op-cpu [0ms - 18473ms]
openfile2            70426ops       77ops/s   0.0mb/s      2.6ms/op     7097us/op-cpu [0ms - 12216ms]
openfile1            70426ops       77ops/s   0.0mb/s      2.8ms/op     7380us/op-cpu [0ms - 11137ms]
38291: 1094.026: IO Summary: 633780 ops, 696.868 ops/s, (155/77 r/w), 5090.2mb/s,   9431us cpu/op,  53.5ms latency

Lets wait until all the workloads are done.

All workloads are completed now.
So,  lets go through and see what Analytics insights PerfAccel provides us

1. IO Trace

# dgpctl -cache trace io


---------------------------------------------------------------------------------------------------
  c-read    c-write      s-read    s-write   sys-read  sys-write      s-name            s-dir
---------------------------------------------------------------------------------------------------
 12.8 GB     0.0 KB      1.7 GB     2.4 GB     1.6 TB     1.2 GB      critical_app      /app_data
  4.7 GB   565.2 MB      1.2 GB     3.9 GB     4.5 TB     3.0 GB      autodesk_cad      /autodesk_cad
  2.0 GB     1.2 GB      1.5 GB     3.7 GB   429.7 GB     2.5 GB      maya_render       /maya_render
300.3 MB     0.0 KB    703.7 MB   830.2 MB    37.4 GB     9.2 GB      vlsi_tools        /vlsi_tools

We can see here clearly that the I/O done on /app_data is cached completely and most of the reads are happening directly from the cache.
There are are only Re-reads and no-rewrites. So, this is a Perfect Application for Caching benefits
Second one in the line is autodesk_cad where there are some Re-writes as well.
maya_render has kind of around 60% Re-reads with 40% Re-writes
vlsi_tools has lowest amount of re-read and absolutely no re-writes

2. IOPS Trace

# dgpctl -cache trace iops


----------------------------------------------------------------------------------------------------------------------------------------------------------
device    total read     total write    total read   total write  read         write        avg.read     avg.write    avg.read     avg.write    Source Dir
type      ops            ops            data         data         throughput   throughput   latency      latency      qlength      qlength
                                         (GB)         (GB)         (MB/s)       (MB/s)       (ms)         (ms)
----------------------------------------------------------------------------------------------------------------------------------------------------------
Cache     20976          0              0.293        0.000        3.2          0            4.731        0.000        0.130        0.000        /vlsi_tools
Source    8866           29999          0.176        0.755        5.2          22.4         4.604        400.135      0.067        320.877      /vlsi_tools

Cache     93582          40315          4.749        0.506        14.2         3.6          15.921       128.808      3.152        36.854       /autodesk_cad
Source    43985          155129         1.208        3.600        2.3          14.5         48.483       217.993      2.477        174.959      /autodesk_cad

Cache     754121         0              12.799       0.000        38.5         0            6.245        0.000        1.812        0.000        /app_data
Source    100001         102871         1.720        2.171        6.2          0.7          62.301       43.276       19.036       0.567        /app_data

Cache     157291         194160         2.014        1.026        9.8          2.2          3.636        8.333        1.108        1.904        /maya_render
Source    117832         300100         1.545        3.112        4.5          2.9          4.514        7.626        0.410        0.822        /maya_render

The Analytical insight provided by this info are :
a. Provides the Application level Analytical metrics like – Read/Write Ops performed, Throughput, Latency, Average Queue length
b. The metrics are available for both Source & Cache sides
c. The latency of Reads and Writes on the Source side is much higher than the cache device
So, caching the data in this case lowers the latency multi-fold thereby increasing the performance of the Application
d. Also, the Avg Write queue length of the Sources vlsi_tools and autodesk_cad is too high. These workload in these sources had parallel threads running and doing the I/O. Both the sources are on Local devices. So, this shows that there was too much contention for Writes on those devices.

This level of IOPS Analytics info for each application data directory as a separate entity is really something to look for in the field of I/O Analytics. We have not seen Per-Directory level metrics anywhere else. (If any of you have seen, then please do comment )

3. OPS Trace

# dgpctl -cache trace ops

-----------------------vlsi_tools: PERFACCEL OPS STATISTICS------------------------------
Type         Cache      Latency (msec) Ops/sec    Source     Latency (msec) Ops/sec
-----------------------------------------------------------------------------------
GETATTR      1002829    49             20465898   309076     64             4829312
SETATTR      35705      5              7141000    39054      4647           8404
FOLLOW_LINK  745429     241            3093066    1925       1              1925000
LOOKUP       0          0              0          162218     21024          7716
CREATE       0          0              0          32532      15270          2130
OPEN         0          0              0          887430     25             35497200
MKDIR        0          0              0          680        1923           354
RMDIR        0          0              0          0          0              0
-----------------------autodesk_cad: PERFACCEL OPS STATISTICS------------------------------
Type         Cache      Latency (msec) Ops/sec    Source     Latency (msec) Ops/sec
-----------------------------------------------------------------------------------
GETATTR      0          0              0          218988     14             15642000
SETATTR      0          0              0          0          0              0
FOLLOW_LINK  0          0              0          0          0              0
LOOKUP       0          0              0          186357     229668         811
CREATE       0          0              0          100000     5663           17658
OPEN         0          0              0          311281     12             25940083
MKDIR        0          0              0          5217       277            18834
RMDIR        0          0              0          0          0              0
-----------------------critical_app: PERFACCEL OPS STATISTICS------------------------------
Type         Cache      Latency (msec) Ops/sec    Source     Latency (msec) Ops/sec
-----------------------------------------------------------------------------------
GETATTR      0          0              0          1944973    608            3198969
SETATTR      0          0              0          0          0              0
FOLLOW_LINK  0          0              0          0          0              0
LOOKUP       0          0              0          1003218    1376893        729
CREATE       0          0              0          100001     41117          2432
OPEN         0          0              0          944418     1494           632141
MKDIR        0          0              0          5114       8897           575
RMDIR        0          0              0          0          0              0
-----------------------maya_render: PERFACCEL OPS STATISTICS------------------------------
Type         Cache      Latency (msec) Ops/sec    Source     Latency (msec) Ops/sec
-----------------------------------------------------------------------------------
GETATTR      0          0              0          1186522    316            3754816
SETATTR      0          0              0          0          0              0
FOLLOW_LINK  0          0              0          0          0              0
LOOKUP       0          0              0          599310     579750         1034
CREATE       0          0              0          190096     280153         679
OPEN         0          0              0          410123     818            501373
MKDIR        0          0              0          2          11             182
RMDIR        0          0              0          2          29             69

We can clearly see that vlsi_tools is the one which benefited heavily from Attribute caching
maya_render is the source which has created the highest number of files i.e. 190096
critical_app is the source which has done over a million lookups
critical_app and vlsi_tools both have done tremendous number of file opens, almost close to a Million

Then, coming to symlink caching, vlsi_tools is the only source which has benefited from the Symlink caching and has huge symlink hits from the cache itself.

4. Meta Operations Trace

# dgpctl -cache trace meta

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
l-cached     l-hits   l-misses   cr-attr    cw-attr    sr-attr    sw-attr     create       open      close     unlink      fsync     frdlck     fwrlck       mmap     s-name          s-dir
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
541     745429       1925   1002825     118108     159128       1312      32532     866705     866705      16676          0          0          0        160     vlsi_tools      /vlsi_tools
  0          0          0    844298          0     105197          0     100001     944418     944418          0          0          0          0      72214     critical_app      /app_data
  0          0          0    220021          0     372009          0     190096     410120     410120     190095     220029          0          0      75695     maya_render      /maya_render
  0          0          0         0     287419       5220          0     100000     311278     311278          0          0          0          0      64749     autodesk_cad      /autodesk_cad

This trace is a more improvised version than the one we saw just before this. It contains additional metrics like sync calls, lock read, lock write, mmap calls etc.

We can see that maya_render has done a large number of fsync calls on its files

vlsi_tools is the only source which has done almost negligible number of mmap calls
vlsi_tools is also the only source which has a lot of Symlink access in its I/O i.e. 541 symlinks.
And the symlinks were heavily accessed as evident from the close-to-million hits on the cached symlinks


5. Brief Analytics Trace for vlsi_tools source

One of the important metrics covered here is the Cache Sizing. The application using this source has only used 292 MB of cache. That means, its working set size is 292 MB only.

Apart from that there are various other metrics displayed for the Application data, like caching information, Read Write I/O hits, misses, Attribute Read/Write hits/misses, symlinks hits/misses, cache usage metrics etc. All Together, this Analytical report provides  a very powerful Insight to the Application I/O Workload 

# dgpctl –cache trace brief vlsi_tools

-----------------------------------------------------------
* vlsi_tools: PERFACCEL CACHE STATISTICS *
-----------------------------------------------------------
Files In Cache              : 15479
File Segments Cached        : 32558
File Segments Uncached      : 17077
File Segments Present       : 15481
Read Hits                   : 76885
Read Misses                 : 180146
Write Misses                : 212521
Write Hits                  : 0
Writeback Hits              : 0
Attribute Read Hits         : 1002825
Attribute Read Misses       : 159128
Attribute Write Hits        : 118108
Attribute Write Misses      : 1312
Links Cached                : 541
Link Hits                   : 745429
Link Misses                 : 1925
Link Cache Usage            : 7.5 MB
Directories cached          : 0
Directories uncached        : 0
ReadDir Hits                : 0
ReadDir Misses              : 0
Dir Attribute Read Hits     : 0
Dir Attribute Read Misses   : 0
Directories Cache Usage     : 0.0 KB
Total Cache Usage           : 292.3 MB (5.71%)
Source Cache Capacity       : 5.0 GB
Pending I/O requests        : 0
Misses During Cacheify      : 0
Max Cache Usage             : 291.0 MB
Total Cacheify              : 179.9 MB
File Cache Limit            : 100%
Cache Uptime                : 6 hrs 29 mins 0 secs

Cacheify Failure Reasons
------------------------
Count of Not Enough Space  : 0
Cleaner In Progress        : 0

-----------------------------------------------------------
* CLEANUP STATISTICS *
-----------------------------------------------------------
Total cleanup runs          : 0
Last cleanup run at         : 0
Total chunks thrown         : 0
Time taken for last run     : 0 sec
Last run freed space        : 0.0 KB
Total freed space           : 0.0 KB
Total Time in cleaner       : 0 sec
Total Time in bucketize     : 0 sec
Evict fail due to flush     : 0
Evict fail due to other     : 0
TTL Expiry Eviction         : 0 (Segments)
Eviction Meta Freed         : 0.0 KB
-----------------------------------------------------------
* WRITEBACK STATISTICS *
-----------------------------------------------------------
Pending Attribute Flush     : 0
Bytes Pending Writeback     : 0.0 KB
Data scheduled for flush    : 0.0 KB
Total Bytes Flushed         : 0.0 KB
-----------------------------------------------------------
* LAST FLUSH STATISTICS *
-----------------------------------------------------------
Flush ID                    : 0
Scheduling time             : 0.00 sec
IO time                     : 0.00 sec
Bytes Flushed               : 0.0 KB

6. Brief Analytics Trace for autodesk_cad source

This source has utilized around 44% of the cache space allocated to it. So, it may probably need more cache space in the future. Seeing the Files in Cache and Segments cached value, we can right away tell that there are only 256 files which are greater than the 2MB size.

Good amount of Read Hits are also seen here

# dgpctl -cache trace brief autodesk_cad

-----------------------------------------------------------
* autodesk_cad: PERFACCEL CACHE STATISTICS *
-----------------------------------------------------------
Files In Cache          : 100000
File Segments Cached    : 100256
File Segments Uncached  : 0
File Segments Present   : 100256
Read Hits               : 1244857
Read Misses             : 316550
Write Misses            : 872093
Write Hits              : 144684
Writeback Hits          : 0
Attribute Read Hits     : 0
Attribute Read Misses   : 5220
Attribute Write Hits    : 287419
Attribute Write Misses  : 0
Links Cached            : 0
Link Hits               : 0
Link Misses             : 0
Link Cache Usage        : 0.0 KB
Directories cached      : 0
Directories uncached    : 0
ReadDir Hits            : 0
ReadDir Misses          : 0
Dir Attribute Read Hits : 0
Dir Attribute Read Misses: 0
Directories Cache Usage : 0.0 KB
Total Cache Usage       : 2.2 GB (44.35%)
Source Cache Capacity   : 5.0 GB
Pending I/O requests    : 0
Misses During Cacheify  : 0
Max Cache Usage         : 2.2 GB
Total Cacheify          : 1.2 GB
File Cache Limit        : 100%
Cache Uptime            : 6 hrs 29 mins 1 secs

Cacheify Failure Reasons
------------------------
Count of Not Enough Space : 0
Cleaner In Progress      : 0
-----------------------------------------------------------
* CLEANUP STATISTICS *
-----------------------------------------------------------
Total cleanup runs      : 0
Last cleanup run at     : 0
Total chunks thrown     : 0
Time taken for last run : 0 sec
Last run freed space    : 0.0 KB
Total freed space       : 0.0 KB
Total Time in cleaner   : 0 sec
Total Time in bucketize : 0 sec
Evict fail due to flush : 0
Evict fail due to other : 0
TTL Expiry Eviction     : 0 (Segments)
Eviction Meta Freed     : 0.0 KB
-----------------------------------------------------------
* WRITEBACK STATISTICS *
-----------------------------------------------------------
Pending Attribute Flush : 0
Bytes Pending Writeback : 0.0 KB
Data scheduled for flush: 0.0 KB
Total Bytes Flushed     : 0.0 KB
-----------------------------------------------------------
* LAST FLUSH STATISTICS *
-----------------------------------------------------------
Flush ID                : 0
Scheduling time         : 0.00 sec
IO time                 : 0.00 sec
Bytes Flushed           : 0.0 KB

7. Brief Analytics Trace for maya_render source
This source has also used almost 45% of cache

The files present in the cache are all < 2MB in size. But, the Segments uncached shows that many files were removed as well. A good amount of Read Hits are also seen in this source as well. There are write hits also, meaning the I/O is doing re-writes

# dgpctl -cache trace brief maya_render

-----------------------------------------------------------
* maya_render: PERFACCEL CACHE STATISTICS *
-----------------------------------------------------------
Files In Cache          : 80074
File Segments Cached    : 190096
File Segments Uncached  : 110022
File Segments Present   : 80074
Read Hits               : 528080
Read Misses             : 404936
Write Misses            : 643502
Write Hits              : 322814
Writeback Hits          : 0
Attribute Read Hits     : 220021
Attribute Read Misses   : 372009
Attribute Write Hits    : 0
Attribute Write Misses  : 0
Links Cached            : 0
Link Hits               : 0
Link Misses             : 0
Link Cache Usage        : 0.0 KB
Directories cached      : 0
Directories uncached    : 0
ReadDir Hits            : 0
ReadDir Misses          : 0
Dir Attribute Read Hits : 0
Dir Attribute Read Misses: 0
Directories Cache Usage : 0.0 KB
Total Cache Usage       : 2.3 GB (45.74%)
Source Cache Capacity   : 5.0 GB
Pending I/O requests    : 0
Misses During Cacheify  : 0
Max Cache Usage         : 2.3 GB
Total Cacheify          : 1.5 GB
File Cache Limit        : 100%
Cache Uptime            : 4 hrs 15 mins 40 secs

Cacheify Failure Reasons
------------------------
Count of Not Enough Space: 0
Cleaner In Progress     : 0
-----------------------------------------------------------
* CLEANUP STATISTICS *
-----------------------------------------------------------
Total cleanup runs      : 0
Last cleanup run at     : 0
Total chunks thrown     : 0
Time taken for last run : 0 sec
Last run freed space    : 0.0 KB
Total freed space       : 0.0 KB
Total Time in cleaner   : 0 sec
Total Time in bucketize : 0 sec
Evict fail due to flush : 0
Evict fail due to other : 0
TTL Expiry Eviction     : 0 (Segments)
Eviction Meta Freed     : 0.0 KB
-----------------------------------------------------------
* WRITEBACK STATISTICS *
-----------------------------------------------------------
Pending Attribute Flush : 0
Bytes Pending Writeback : 0.0 KB
Data scheduled for flush : 0.0 KB
Total Bytes Flushed     : 0.0 KB
-----------------------------------------------------------
* LAST FLUSH STATISTICS *
-----------------------------------------------------------
Flush ID                : 0
Scheduling time         : 0.00 sec
IO time                 : 0.00 sec
Bytes Flushed           : 0.0 KB
-----------------------------------------------------------

8. Brief Analytics Trace for vlsi_tools source


This one as well has used less than 50% of the cache.

Pure Re-Read workload with huge Read Hits.
No, re-writes at all

Huge number of Attribute Read hits

 

# dgpctl -cache trace brief critical_app

-----------------------------------------------------------
* critical_app: PERFACCEL CACHE STATISTICS *
-----------------------------------------------------------
Files In Cache          : 100001
File Segments Cached    : 100001
File Segments Uncached  : 0
File Segments Present   : 100001
Read Hits               : 3355122
Read Misses             : 450832
Write Misses            : 619837
Write Hits              : 0
Writeback Hits          : 0
Attribute Read Hits     : 844298
Attribute Read Misses   : 105197
Attribute Write Hits    : 0
Attribute Write Misses  : 0
Links Cached            : 0
Link Hits               : 0
Link Misses             : 0
Link Cache Usage        : 0.0 KB
Directories cached      : 0
Directories uncached    : 0
ReadDir Hits            : 0
ReadDir Misses          : 0
Dir Attribute Read Hits : 0
Dir Attribute Read Misses: 0
Directories Cache Usage : 0.0 KB
Total Cache Usage       : 2.2 GB (43.93%)
Source Cache Capacity   : 5.0 GB
Pending I/O requests    : 0
Misses During Cacheify  : 0
Max Cache Usage         : 2.2 GB
Total Cacheify          : 1.7 GB
File Cache Limit        : 100%
Cache Uptime            : 6 hrs 28 mins 39 secs



Cacheify Failure Reasons
------------------------
Count of Not Enough Space: 0
Cleaner In Progress      : 0
-----------------------------------------------------------
* CLEANUP STATISTICS *
-----------------------------------------------------------
Total cleanup runs      : 0
Last cleanup run at     : 0
Total chunks thrown     : 0
Time taken for last run : 0 sec
Last run freed space    : 0.0 KB
Total freed space       : 0.0 KB
Total Time in cleaner   : 0 sec
Total Time in bucketize : 0 sec
Evict fail due to flush : 0
Evict fail due to other : 0
TTL Expiry Eviction     : 0 (Segments)
Eviction Meta Freed     : 0.0 KB
-----------------------------------------------------------
* WRITEBACK STATISTICS *
-----------------------------------------------------------
Pending Attribute Flush : 0
Bytes Pending Writeback : 0.0 KB
Data scheduled for flush : 0.0 KB
Total Bytes Flushed     : 0.0 KB
-----------------------------------------------------------
* LAST FLUSH STATISTICS *
-----------------------------------------------------------
Flush ID                : 0
Scheduling time         : 0.00 sec
IO time                 : 0.00 sec
Bytes Flushed           : 0.0 KB
-----------------------------------------------------------

9. File I/O Analytics Traces

# dgpctl -cache trace file –field rhits –sort | more

Now, this trace drills down one level deep from the Application to the File Analytics level
It shows us almost all the same metrics,that we saw in Application level/Source level traces. but this time for each cached file, thereby providing us with very fine grained Analytical data

------------------------------------------------------------------------------------------------------------------------------------------
Inode Num             Segments      R Hits        W Hits        BlkCached(4k) LATime(s)   ResTime(s)  Sys Reads   Sys Writes  File Name
------------------------------------------------------------------------------------------------------------------------------------------
132242                1             598           0             46            343         955         128984      46          /vlsi_tools/openssl-1.0.2-beta2/crypto/stack/safestack.h
656530                1             552           0             46            343         949         128984      46          /vlsi_tools/openssl-1.0.2-beta3/crypto/stack/safestack.h
263314                1             460           0             46            340         945         128984      46          /vlsi_tools/openssl-1.0.2-beta4/crypto/stack/safestack.h
132018                1             455           0             35            343         955         67760       35          /vlsi_tools/openssl-1.0.2-beta2/crypto/objects/obj_mac.h
656306                1             420           0             35            342         949         67760       35          /vlsi_tools/openssl-1.0.2-beta3/crypto/objects/obj_mac.h
1181                  1             414           0             46            340         941         128984      46          /vlsi_tools/openssl-1.0.2-beta5/crypto/stack/safestack.h
263090                1             315           0             35            339         945         67760       35          /vlsi_tools/openssl-1.0.2-beta4/crypto/objects/obj_mac.h
957                   1             315           0             35            339         941         67760       35          /vlsi_tools/openssl-1.0.2-beta5/crypto/objects/obj_mac.h
263577                1             279           0             31            275         945         15624       31          /vlsi_tools/openssl-1.0.2-beta4/ssl/ssl.h
1444                  1             279           0             31            277         938         15624       31          /vlsi_tools/openssl-1.0.2-beta5/ssl/ssl.h
525234                1             264           0             66            955         2054        650         78          /vlsi_tools/openssl-1.0.2-beta1/crypto/objects/obj_dat.h
132019                1             198           0             66            378         955         264         66          /vlsi_tools/openssl-1.0.2-beta2/crypto/objects/obj_dat.h
656307                1             198           0             66            360         949         264         66          /vlsi_tools/openssl-1.0.2-beta3/crypto/objects/obj_dat.h

10. File IOPS Analytics Traces

# dgpctl -cache trace new_file –iops | more

Now, drill down even deeper on the file level analytics.
This shows us the Throughput, Latency, Read/Write IOPS, Read/Write queue length etc. for each file itself.
We can see how each file is doing in terms of I/O


----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Inode Num             CRead        SRead        CRead        SRead        CWrite       SWrite       CWrite       SWrite       CRead        SRead        CWrite       SWrite       File Name
                    throughput   throughput   latency      latency      throughput   throughput   latency      latency      qlength      qlength      qlength      qlength
                     (MBps)        (MBps)       (ms)         (ms)         (MBps)       (MBps)       (ms)         (ms)         (avg)        (avg)        (avg)        (avg)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
11                    0.0          362.5        0.000        0.587        0.0          51.0         0.000        43.268       0.000        0.087        0.000        9.000        /vlsi_tools/openssl-1.0.2-beta1.tar.gz
524290                0.0          1.3          0.000        3.121        0.0          0.0          0.000        22.442       0.000        0.000        0.000        0.000        /vlsi_tools/openssl-1.0.2-beta1/ACKNOWLEDGMENTS
524292                3.9          15.3         1.994        0.511        0.0          0.4          0.000        22.121       0.000        0.000        0.000        0.000        /vlsi_tools/openssl-1.0.2-beta1/apps/app_rand.c
524293                26.4         215.6        2.246        0.344        0.0          146.1        0.000        0.497        0.000        0.000        0.000        0.000        /vlsi_tools/openssl-1.0.2-beta1/apps/apps.c
524294                7.3          47.2         2.148        0.331        0.0          0.6          0.000        21.932       0.000        0.000        0.000        0.000        /vlsi_tools/openssl-1.0.2-beta1/apps/apps.h
524295                8.5          30.8         1.383        0.381        0.0          0.5          0.000        21.867       0.000        0.000        0.000        0.000        /vlsi_tools/openssl-1.0.2-beta1/apps/asn1pars.c
524296                0.7          173.1        85.572       0.429        0.0          75.4         0.000        0.953        0.000        0.000        0.000        0.000        /vlsi_tools/openssl-1.0.2-beta1/apps/ca.c
524297                0.0          4.3          0.000        0.910        0.0          0.0          0.000        21.718       0.000        0.000        0.000        0.000        /vlsi_tools/openssl-1.0.2-beta1/apps/ca-cert.srl
524298                0.0          20.6         0.000        0.380        0.0          0.3          0.000        22.825       0.000        0.000        0.000        0.000        /vlsi_tools/openssl-1.0.2-beta1/apps/CA.com
524299                0.0          8.8          0.000        0.443        0.0          0.0          0.000        20.480       0.000        0.000        0.000        0.000 /vlsi_tools/openssl-1.0.2-beta1/apps/ca-key.pem

11. File Meta Operations Analytics Traces

# dgpctl -cache trace file –meta | more

Now, another drill down on each cached file for the meta operations and  system call details.
We can easily find out situations like, a particular application doing frequent fsync on a particular file, causing frequent disk-writes and slowing down the I/O
Or particular set of files are opened and closed heavily
Or set of files are Mmaped and used

-----------------------------------------------------------------------------------------------------------------------------------------
Inode Num             Open      Close     Rattr     Wattr       Fsync       Frdlck      Fwrlck      Mmap        Create      File Name
-----------------------------------------------------------------------------------------------------------------------------------------
11                    3         3         10        3           0           0           0           0           1           /vlsi_tools/openssl-1.0.2-beta1.tar.gz
524290                5         5         22        6           0           0           0           0           1           /vlsi_tools/openssl-1.0.2-beta1/ACKNOWLEDGMENTS
524292                9         9         28        9           0           0           0           0           1           /vlsi_tools/openssl-1.0.2-beta1/apps/app_rand.c
524293                9         9         28        10          0           0           0           0           1           /vlsi_tools/openssl-1.0.2-beta1/apps/apps.c
524294                209       209       228       10          0           0           0           0           1           /vlsi_tools/openssl-1.0.2-beta1/apps/apps.h
524295                9         9         28        9           0           0           0           0           1           /vlsi_tools/openssl-1.0.2-beta1/apps/asn1pars.c
524296                9         9         28        10          0           0           0           0           1           /vlsi_tools/openssl-1.0.2-beta1/apps/ca.c
524297                5         5         20        6           0           0           0           0           1           /vlsi_tools/openssl-1.0.2-beta1/apps/ca-cert.srl

12. FIle I/O Analytics of Deleted /Unlinked Files
# dgpctl -cache trace new_file –deleted | more

Now, this even goes one step ahead. Providing file analytics for currently existing files is one thing, but showing the same for files that are deleted/unlinked stands out distinctly.

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------
Meta Id   Segments  R Hits    W Hits    Size      Open      Close     Stats     R Miss    W Miss    CRead        SRead      CRead     SRead     CWrite       SWrite       CWrite    SWrite    CRead     Sread     CWrite    Swrite    
CreateTime                    Deletion Time                 Reason    File Name
                                                                                                 throughput   throughput   latency   latency   throughput   throughput   latency   latency   qlength   qlength   qlength   qlength
                                                                                                    (MBps)       (MBps)       (ms)      (ms)    (MBps)       (MBps)       (ms)      (ms)      (avg)     (avg)     (avg)     (avg)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------
20404     1         0         0         0         4         4         7         0         4         0.0          0.0          0.000     0.000     0.0          0.5          0.000     24.623    0.000     0.000     0.000     0.000     Fri M
ar 10 18:52:36 IST 2017  Fri Mar 10 18:54:26 IST 2017  [S]       /vlsi_tools/openssl-1.0.2-beta3/crypto/aes/aes-x86_64.o
20413     1         0         0         0         4         4         7         0         4         0.0          0.0          0.000     0.000     0.0          0.5          0.000     23.434    0.000     0.000     0.000     0.000     Fri M
ar 10 18:52:37 IST 2017  Fri Mar 10 18:54:26 IST 2017  [S]       /vlsi_tools/openssl-1.0.2-beta3/crypto/aes/bsaes-x86_64.o
20407     1         0         0         0         4         4         7         0         2         0.0          0.0          0.000     0.000     0.0          0.2          0.000     24.491    0.000     0.000     0.000     0.000     Fri M
ar 10 18:52:36 IST 2017  Fri Mar 10 18:54:26 IST 2017  [S]       /vlsi_tools/openssl-1.0.2-beta3/crypto/aes/vpaes-x86_64.o
21709     1         0         0         0         4         4         7         0         2         0.0          0.0          0.000     0.000     0.0          2.3          0.000     1.700     0.000     0.000     0.000     0.000     Fri M
ar 10 18:53:12 IST 2017  Fri Mar 10 18:54:26 IST 2017  [S]       /vlsi_tools/openssl-1.0.2-beta3/crypto/asn1/a_bitstr.o
21971     1         0         0         0         4         4         7         0         1         0.0          0.0          0.000     0.000     0.0          4.8          0.000     0.411     0.000     0.000     0.000     0.000     Fri M
ar 10 18:53:19 IST 2017  Fri Mar 10 18:54:26 IST 2017  [S]       /vlsi_tools/openssl-1.0.2-beta3/crypto/asn1/a_bool.o
22020     1         0         0         0         4         4         7         0         2         0.0          0.0          0.000     0.000     0.0          1.4          0.000     3.400     0.000     0.000     0.000     0.000     Fri M
ar 10 18:53:20 IST 2017  Fri Mar 10 18:54:26 IST 2017  [S]       /vlsi_tools/openssl-1.0.2-beta3/crypto/asn1/a_bytes.o
21759     1         0         0         0         4         4         7         0         2         0.0          0.0          0.000     0.000     0.0          2.2          0.000     2.075     0.000     0.000     0.000     0.000     Fri M
ar 10 18:53:13 IST 2017  Fri Mar 10 18:54:26 IST 2017  [S]       /vlsi_tools/openssl-1.0.2-beta3/crypto/asn1/a_d2i_fp.o
21776     1         0         0      0         4         4         7

Note: In this example, the files are not re-read till now, so they don’t have any Hits or Throughput etc…

13. File I/O Segment Analytics – Offset based Analytics – For maya_render source

# dgpctl -cache trace complete maya_render

Now, let’s drill down to the next level of file analytics. With the previous traces and metrics we were able to visualize the I/O in the File Level as a whole unit.
What if we want to find out which actual regions in the file where the I/O is really done by the application ? Which offsets were really accessed or written ?

There may be many scenarios like, where I/O is happening only in 10% of the total file or certain area of the file. This file segment analytics provides us with exactly that information for all the segments of the file. PerfAccel has a fixed segment size of 2MB. So, if a 1 GB file is accessed fully for the first time, then it will have 512 Segments cached.

This complete trace displays the I/O metrics with respect to each Offset segment

Below trace shows all different files as the files are small in size and within a single segment size i.e. 2MB. If a file size is beyond 2MB, then the offsets will be displayed for each segment, like 0-2MB, 2MB-4MB … till the end


------------------------------------------------------------------------------------------------------------------------
Inode Num              Offset        R Hits        W Hits        Size          LATime(s)   ResTime(s)  File Name
------------------------------------------------------------------------------------------------------------------------
786444                0             24            7             36864         13287       15075       /maya_render/bigfileset/00000001/00019395
786453                0             7             7             28672         13553       15075       /maya_render/bigfileset/00000001/00063549
786457                0             12            7             24576         13192       15075       /maya_render/bigfileset/00000001/00039554
786464                0             15            7             32768         13331       15075       /maya_render/bigfileset/00000001/00010344
786465                0             14            6             36864         13603       15075       /maya_render/bigfileset/00000001/00052626
786469                0             15            2             32768         13452       15075       /maya_render/bigfileset/00000001/00085670
786476                0             35            2             73728         13380       15075       /maya_render/bigfileset/00000001/00099979
786478                0             8             3             16384         13213       15075       /maya_render/bigfileset/00000001/00034847
786480                0             15            4             36864         13506       15075       /maya_render/bigfileset/00000001/00073333
786483                0             10            3             24576         13559       15075       /maya_render/bigfileset/00000001/00062216
786487                0             15            8             28672         13256       15075       /maya_render/bigfileset/00000001/00026172
786489                0             18            11            49152         13353       15075       /maya_render/bigfileset/00000001/00005856
786490                0             12            8             28672         13375       15075       /maya_render/bigfileset/00000001/00000974
786508                0             2             0             4096          13566       15075       /maya_render/bigfileset/00000001/00060643
786515                0             32            8             53248         13364       15075       /maya_render/bigfileset/00000001/00003508
786518                0             12            5             24576         13246       15075       /maya_render/bigfileset/00000001/00028425
786529                0             46            7             69632         13379       15075       /maya_render/bigfileset/00000001/00000283
786530                0             25            5             53248         13588       15075       /maya_render/bigfileset/00000001/00056028
786533                0             4             3             12288         13452       15075       /maya_render/bigfileset/00000001/00085713
786534                0             6             0             12288         13440       15075       /maya_render/bigfileset/00000001/00087796
786536                0             3             0             4096          13376       15074       /maya_render/bigfileset/00000001/00000859
786538                0             10            6             28672         13537       15074       /maya_render/bigfileset/00000001/00066705
786539                0             59            7             86016         13173       15074       /maya_render/bigfileset/00000001/00043279
786546                0             12            10            36864         13327       15074       /maya_render/bigfileset/00000001/00011204
786547                0             16            7             45056         13427       15074       /maya_render/bigfileset/00000001/00090823
786555                0             7             4             20480         13466       15074       /maya_render/bigfileset/00000001/00082459

14. File I/O Segment Analysis – Offset based Analytics- For vlsi_tools

# dgpctl -cache trace complete vlsi_tools

------------------------------------------------------------------------------------------------------------------------
Inode Num              Offset        R Hits        W Hits        Size          LATime(s)   ResTime(s)  File Name
------------------------------------------------------------------------------------------------------------------------
11                    0             0             0             2097152       2352        2361        /vlsi_tools/openssl-1.0.2-beta1.tar.gz
524290                0             0             0             4096          1251        2352        /vlsi_tools/openssl-1.0.2-beta1/ACKNOWLEDGMENTS
524292                0             8             0             8192          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/app_rand.c
524293                0             76            0             77824         1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/apps.c
524294                0             16            0             16384         1251        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/apps.h
524295                0             12            0             12288         1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/asn1pars.c
524296                0             76            0             77824         1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/ca.c
524297                0             0             0             4096          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/ca-cert.srl
524298                0             0             0             8192          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/CA.com
524299                0             0             0             4096          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/ca-key.pem
524301                0             2             0             8192          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/CA.pl.in
524302                0             0             0             4096          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/ca-req.pem
524303                0             0             0             8192          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/CA.sh
524304                0             0             0             4096          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/cert.pem
524305                0             8             0             8192          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/ciphers.c
524306                0             0             0             4096          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/client.pem
524307                0             40            0             40960         1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/cms.c
524308                0             12            0             12288         1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/crl2p7.c
524309                0             12            0             12288         1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/crl.c
524311                0             0             0             4096          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/demoCA/cacert.pem
524312                0             0             0             4096          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/demoCA/index.txt
524314                0             0             0             4096          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/demoCA/private/cakey.pem
524315                0             0             0             4096          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/demoCA/serial
524317                0             0             0             4096          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/demoSRP/srp_verifier.txt
524318                0             0             0             4096          1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/demoSRP/srp_verifier.txt.attr
524319                0             20            0             20480         1250        2352        /vlsi_tools/openssl-1.0.2-beta1/apps/dgst.c

15. File I/O Segment Analtyics – Offset based Analytics – For critical_app

# dgpctl -cache trace complete critical_app


------------------------------------------------------------------------------------------------------------------------
Inode Num              Offset        R Hits        W Hits        Size          LATime(s)   ResTime(s)  File Name
------------------------------------------------------------------------------------------------------------------------
6815751               0             0             0             0             8041        8041        /app_data/logfiles/00000001/00000001
7077998               0             24            0             12288         3155        8030        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000034/00000016/00000014
7077999               0             16            0             8192          3145        8030        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000034/00000025/00000024/00000032/00000002/00000005/00000005
7078000               0             88            0             45056         3171        8030        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000031/00000003/00000017/00000008/00000010/00000023/00000008/00000014
7078001               0             8             0             4096          3165        8030        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000031/00000003/00000017/00000021/00000008/00000004
6815854               0             70            0             40960         3230        8030        /app_data/bigfileset/00000002/00000001/00000006/00000010/00000020/00000008/00000012/00000019/00000005/00000007/00000004
7078002               0             72            0             36864         3184        8029        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000031/00000002/00000001/00000025
6946925               0             7             0             4096          3260        8029        /app_data/bigfileset/00000002/00000001/00000006/00000010/00000005/00000010/00000006/00000036/00000007/00000013/00000009/00000005
7078003               0             24            0             12288         3162        8029        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000031/00000003/00000051/00000008
7078004               0             16            0             8192          3168        8029        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000031/00000003/00000017/00000008/00000020/00000006/00000013
7078005               0             16            0             8192          3165        8029        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000031/00000003/00000017/00000013/00000022
6815855               0             21            0             12288         3242        8029        /app_data/bigfileset/00000002/00000001/00000006/00000010/00000020/00000008/00000006/00000009/00000020/00000006/00000003
6946926               0             42            0             24576         3280        8029        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000064/00000004/00000010/00000025/00000044
6946927               0             16            0             8192          3140        8029        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000034/00000040/00000014/00000016
6815856               0             21            0             12288         3206        8029        /app_data/bigfileset/00000002/00000001/00000006/00000010/00000022/00000006/00000015/00000016/00000019
6815857               0             7             0             4096          3234        8029        /app_data/bigfileset/00000002/00000001/00000006/00000010/00000020/00000008/00000010/00000004/00000001/00000011
6815858               0             77            0             45056         3225        8029        /app_data/bigfileset/00000002/00000001/00000006/00000010/00000020/00000011/00000005/00000025/00000004/00000001/00000002
6946928               0             8             0             4096          3142        8029        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000034/00000040/00000006/00000005/00000012/00000001
6946929               0             21            0             12288         3286        8029        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000042/00000004/00000036/00000010/00000019/00000050
6946930               0             48            0             24576         3130        8029        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000042/00000004/00000009/00000009/00000003/00000002/00000017
7078006               0             88            0             45056         3153        8029        /app_data/bigfileset/00000002/00000001/00000003/00000001/00000034/00000024/00000007/00000011

16. File Segment Analytics – Offset based Analytics – For autodesk_cad

# dgpctl -cache trace complete autodesk_cad


------------------------------------------------------------------------------------------------------------------------
Inode Num              Offset        R Hits        W Hits        Size          LATime(s)   ResTime(s)  File Name
------------------------------------------------------------------------------------------------------------------------
394504                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000005/00000008/00000011/00000008/00000045/00000009/00000028/00000018
263326                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000010/00000002/00000049/00000023/00000020/00000004/00000009/00000045/00000002
263327                0             0             2             8192          1660        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000010/00000002/00000049/00000008/00000003/00000012/00000009
656737                0             1             3             12288         1461        2031        /autodesk_cad/bigfileset/00000001/00000008/00000002/00000001/00000003/00000028/00000014/00000011/00000009/00000039/00000013/00000002/00000004/00000027
394505                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000007/00000018/00000010/00000010/00000017/00000010
525644                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000001/00000004/00000011/00000012/00000013/00000008/00000017/00000014
656738                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000001/00000004/00000011/00000011/00000023
394506                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000001/00000023/00000016/00000021/00000002/00000001
263328                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000010/00000002/00000049/00000008/00000003/00000005/00000005
525645                0             0             1             4096          1675        2031        /autodesk_cad/bigfileset/00000001/00000008/00000002/00000001/00000003/00000028/00000014/00000011/00000009/00000073/00000012
525646                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000002/00000001/00000010/00000013/00000009/00000004
263329                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000010/00000002/00000049/00000023/00000024/00000019/00000019
394507                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000002/00000001/00000010/00000013/00000019/00000009/00000018/00000025
525647                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000005/00000008/00000011/00000008/00000044/00000005/00000002
525648                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000001/00000023/00000016/00000018/00000008
263330                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000010/00000002/00000009/00000007/00000013
656739                0             0             0             8192          1125        2031        /autodesk_cad/bigfileset/00000001/00000008/00000007/00000014/00000009/00000026/00000009/00000028
525649                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000005/00000008/00000011/00000008/00000034/00000035/00000033/00000011
525650                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000001/00000012/00000012/00000001/00000005
394508                0             0             0             0             2031        2031        /autodesk_cad/bigfileset/00000001/00000008/00000016/00000005/00000008/00000011/00000015/00000007
656740                0             2             2             4096          1334        2031        /autodesk_cad/bigfileset/00000001/00000008/00000002/00000001/00000003/00000019/00000015/00000008/00000009/00000003/00000003


So, we went through various different File I/O Analytics information for the 4 directories where different Application Workloads were running. 
That’s enormous information if we want to analyze any kind of I/O Fingerprint or workload behaviour. There are many more Analytical info still available from PerfAccel and not covered in the above 16 traces. We will cover them soon in the next blogs to find out the Complete Analytics Layer provided by PerfAccel.
 

In the meantime, you can go ahead and Download PerfAccel right now and get hands on with the exciting features.

Stay tuned…..

Share it
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
COMMENTS SECTION

Leave a Reply

Be the First to Comment!

Leave a Reply

wpDiscuz