2. Application Performance Acceleration & Deep File I/O Analytics with PerfAccel

In the previous Blog we had gone through what our expectations should be from Application & File I/O Analytics. And as promised, we had hinted about a tool which can provide us with all the answers…..yes all… Well, here I introduce “PerfAccel”, the Patent-Pending Next Generation Enterprise Storage & Analytics Solution from Datagres Technologies Inc.

PerfAccel is a patent-pending state-of-the-art solution for Advanced application I/O analytics at multiple granularity and uses this intelligence for Dynamic policy based data placement. The various features provide in-depth visibility of the file and directory level data usage and performance thus providing comprehensive data intelligence and dynamic data management within mission critical technology environments. PerfAccel is the next generation of active data management platform that accelerates, manages, and scales active application data and is extremely valuable in large scale grid level compute environment with complete transparency and global visibility.

PerfAccel’s most important value-adds are :
a.  Automated Dynamic Data Placement  or automated HOT Data placement in Faster storage for Application Performance Boost
b.  Deep Application File I/O Analytics in multiple levels and granularity of data access
c.  Seamless Integration with Application  – No Application reconfiguration needed

We will keep this walk-through simple and dive into complex topics individually separately. The goal of this walk-through is to gain a quick insight on how powerful PerfAccel is in terms of its Rich Analytical & Caching features.

Getting PerfAccel up and running in 2 minutes

1. Register

    Register for a account in www.datagres.com   and login using the credentials.
    Then, visit the PerfAccel Editions page or click on the Free Trials or Download Now buttons available in the site. You will have a choice to request either a Community Edition License (restricted set of functionalities) or a fully functional Enterprise Trial Editions License

    Clicking on the Download button in the PerfAccel Editions page, will request the corresponding license for you in the form of a Customer ID. You can view your licenses by visiting the “Your Licenses” page, whose link is available under the menu displaying your Login Name in the top right side. Once you have a license, now you can download the binaries/rpms. For that you can click the “Downloads” Link on the top Downloads menu or click on the “Download Binaries & RPMs” button on the    license page.

2. Download :
    The PerfAccel binaries for your version of OS can be downloaded from www.datagres.com
    Please download the binaries which match the kernel version of your operating system completely

    Ex: For CentOS/RHEL 6.5, the one of the kernel versions is 2.6.32-431.17.1.el6.x86_64
    So, download the PerfAccel binary which supports this kernel

    dgres-perfaccel-3.6.0.1-rhel_6.5-2.6.32-431-x86_64.bin_07032017

    Once the binary is downloaded, copy it to the node where you want to install it

3. Installation & Activation :

    PerfAccel Installation is very simple and takes a minute to complete.
    You can refer the Installation Guide or Getting started guide for more details on this process

Now, once PerfAccel is installed and activated, we can verify it by   dgpclt -lic info

Product Key 		   : 3F9D-XXXX-18D0-2F10-ED70-XXXX-0C1E-8E8F-1C6D-1BF8-D3CB-CCAA-7B9C-9743-2A9E-4854
Licensed to 		   : XXXXXXXXXXXXX@XXXXXX.com
License Activated on 	   : Mon Feb 13 16:52:52 2017 GMT
License Period  	   : 298 Days
License Expires on  	   : Fri Dec  8 16:52:52 2017 GMT
License Status  	   : Active
Licensed Number of Sources : 09999
Licensed Number of Caches  : 09999
Licensed Features  	   : Writethrough, Writeback, Writealways, Autocache, cachespace, advanced analytics, export fs, Analytical, Context Analytics, Directory Analytics, Context Pid Analytics, Negative object, Source group

This shows that our node is now Licensed for PerfAccel use
Now lets get back to our main agenda i.e how PerfAccel can help us in Analyzing Application I/O and performance.

To proceed further, we need to know briefly about few terminologies from the PerfAccel Glossary
a. Source : Any mounted directory i.e. either from local EXT3, EXT4, XFS or NFS or Autofs, which we want to Analyze or Accelerate
b. Cache  : A Logical Cache unit created from a High Speed Device like SSD. Even SATA devices will work but as we know the performance will be lower than SSDs

Now, lets:
a. Create a Cache which can be used for multiple sources (we can allocate how much cache space each source can use)

b. Create a source on the directory that we want to Analyze
    This step will also involve adding cache space to the source.
    Ex. 10% of cache device or 10GB space

c. By default the Source will have a Write-through policy i.e.
    First time read of data will cache it and 2nd time onwards all reads will be served from the cache
    All writes happening for uncached data will go to Source side only
    All writes happening for cached data will go to both Source & Cache sides

Once the above is done, we are now ready to analyze the workload

Create PerfAccel Cache
We have a local disk (its a VM ) /dev/md127 which we will use as a cache device

# dgpctl -cache create ssd_cache /dev/md127 
This operation will delete existing data from the device /dev/md127 and create a PerfAccel cache.
Do you want to continue? [y/n]: y
Initializing cache device /dev/md127. This operation can take several minutes. Please wait…
Cache ssd_cache successfully created

Create PerfAccel Source

We have a NFS share of a critical Ecad application, mounted on /app_data

# df

Filesystem                                        1K-blocks      Used Available Use% Mounted on
/dev/sda1                                          11087104   7729568   2794336  74% /
tmpfs                                               1048888         0   1048888   0% /dev/shm
/dev/sdb                                           12385456   2320348   9435964  20% /data
192.168.1.178:/data_ext4/critical/app/ecad_module 365137920 302363648  44220416  88% /app_data
ecad_source                                       365137920 302363648  44220416  88% /app_data

Lets create a source on /app_data and assign some cache space from the ssd_cache

# dgpctl -source create ecad_source /app_data/ ssd_cache 10G
Source ecad_source was successfully created
10G of cache from ssd_cache successfully added to source ecad_source

Verify that the Source and Cache have been created

# dgpctl -source list

Source                           Status       Cache           Size       Mount Point     Type       Mode
ecad_source                      ONLINE       ssd_cache       10.0 GB    /app_data       nfs        WT    

# dgpctl -cache list

Device        Total size   Free size  Alloc size  State    IP address      Name      
/dev/md127       23.6 GB     13.5 GB     10.0 GB  Online   192.168.1.38    ssd_cache

We can see above that a Cache is created with total capacity 23.6GB and from that 10GB is allocated to a source. And we have created a Source on the NFS share directory and have allocated 10GB of cache space to it

How it looks after source is created

# df -h

Filesystem                                         Size  Used Avail Use% Mounted on
/dev/sda1                                           11G  7.4G  2.7G  74% /
tmpfs                                              1.1G     0  1.1G   0% /dev/shm
/dev/sdb                                            12G  2.3G  9.0G  20% /data
192.168.1.178:/data_ext4/critical/app/ecad_module  349G  289G   43G  88% /app_data
ecad_source                                        349G  289G   43G  88% /app_data

So, in the above df output, a new mount point has appeared on the same path /app_data. This is a PerfAccel source mount.
Note: The beauty of PerfAccel is that it provides all benefits without even a slight change in the Application data path

Time to Rock now – lets get that Analytical Insight

Let’s run some simple plain workloads like 1GB dd Read, Write, Re-Read, Re-Write etc..

Note: This demo is on a Virtual Machine, so local disk I/O speeds may seem to shooting up much more as compared to real physical disks. Network I/O speeds won’t be affected though. For exact accuracy of the numbers, you can run the demo sequence on a Physical machine

a. 1GB dd Write

# dd if=/dev/zero of=iofile.1 bs=1M count=1k conv=fsync,notrunc
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 28.1114 s, 38.2 MB/s

Lets see the Basic IO Trace from PerfAccel

# dgpctl -cache trace io

-------------------------------------------------------------------------------------------
Cache Read 	  Cache Write    Source Read    Source Write   Source Name 	 Source Dir   
-------------------------------------------------------------------------------------------
    0.0 KB         0.0 KB         0.0 KB          1.0 GB   ecad_source     /app_data     

We can clearly see that 1GB was written to the Source i.e. on NFS side.
So, this is a network write. There was no Cache I/O that happened during his dd (Remember !! Write-through policy)

b. 1 GB dd Read

Before doing the read, we should note the following facts

a. The file was created just now, so all Pages are in memory and if we read them now, they will be fetched from the system memory itself.

b. So, we need to throw them out of the memory by using

    echo “3” > /proc/sys/vm/drop_caches
    This will throw all non-dirty pages, dentries, inodes etc from the system cache.

Note: We are doing this only for Demo Purpose here to make sure I/Os happen from disk and no System  Page cache is used …Its not at all needed or recommended by PerfAccel to manually throw system cache in any of the real world deployments

Now, lets do the dd

# dd if=iofile.1 of=/dev/null bs=1M count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 19.2061 s, 55.9 MB/s

# dgpctl -cache trace io

-------------------------------------------------------------------------------------------
Cache Read 	  Cache Write    Source Read    Source Write   Source Name 	 Source Dir   
-------------------------------------------------------------------------------------------
    0.0 KB         0.0 KB         1.0 GB          1.0 GB   ecad_source     /app_data    

Clearly, we can see that there is a Source side read of 1GB i.e. Network Read only. As this is the first read, so as per Write-through policy, the data read should have been cached now. Well how to find out what is cached ? Don’t worry we have lots of Analytical traces for that…

Cache Usage Trace

# dgpctl -cache trace usage

-----------------------------------------------------------------------------------------------------------------
          size          usage  use%     max usage            nfiles            ndirs   source name   	source dir    
-----------------------------------------------------------------------------------------------------------------
       10.0 GB         1.0 GB 10.0%        1.0 GB                 1                0   ecad_source      /app_data  

This clearly tells that 1 file is cached and the amount of data cached is 1GB (just now we had done a 1GB dd read)
Now, lets dive deep into more Analytics on our 1 GB Write & 1 GB Read

Brief Analytical Trace :

# dgpctl -cache trace brief ecad_source

-----------------------------------------------------------
	      * ecad_source: PERFACCEL CACHE STATISTICS * 
-----------------------------------------------------------
Files In Cache		 : 1
File Segments Cached	 : 512
File Segments Uncached	 : 0
File Segments Present	 : 512
Read Hits		 : 0
Read Misses		 : 262144
Write Misses		 : 262144
Write Hits		 : 0
Writeback Hits		 : 0
Attribute Read Hits	 : 4
Attribute Read Misses	 : 283
Attribute Write Hits	 : 0
Attribute Write Misses	 : 0
Links Cached		 : 0
Link Hits		 : 0
Link Misses		 : 0
Link Cache Usage	 : 0.0 KB
Directories cached	 : 0
Directories uncached	 : 0
ReadDir Hits		 : 0
ReadDir Misses		 : 0
Dir Attribute Read Hits	 : 0
Dir Attribute Read Misses: 0
Directories Cache Usage	 : 0.0 KB
Total Cache Usage	 : 1.0 GB (10.02%)
Source Cache Capacity	 : 10.0 GB
Pending I/O requests	 : 0
Misses During Cacheify	 : 0
Max Cache Usage		 : 1.0 GB
Total Cacheify		 : 1.0 GB
File Cache Limit	 : 100%
Cache Uptime		 : 0 hrs 22 mins 31 secs

Cacheify Failure Reasons
------------------------
Count of Not Enough Space: 0
Cleaner In Progress	 : 0
-----------------------------------------------------------
	      * CLEANUP STATISTICS *		
-----------------------------------------------------------
Total cleanup runs	 : 0
Last cleanup run at	 : 0
Total chunks thrown	 : 0
Time taken for last run	 : 0 sec
Last run freed space	 : 0.0 KB
Total freed space	 : 0.0 KB
Total Time in cleaner	 : 0 sec
Total Time in bucketize	 : 0 sec
Evict fail due to flush	 : 0
Evict fail due to other	 : 0
TTL Expiry Eviction	 : 0 (Segments)
Eviction Meta Freed	 : 0.0 KB
-----------------------------------------------------------
	      * WRITEBACK STATISTICS *		  
-----------------------------------------------------------
Pending Attribute Flush	 : 0
Bytes Pending Writeback	 : 0.0 KB
Data scheduled for flush : 0.0 KB
Total Bytes Flushed	 : 0.0 KB
-----------------------------------------------------------
	      * LAST FLUSH STATISTICS *	    
-----------------------------------------------------------
Flush ID		 : 0 
Scheduling time		 : 0.00 sec
IO time			 : 0.00 sec
Bytes Flushed		 : 0.0 KB
-----------------------------------------------------------

Now, this one is something we can call Real File Analytics. This has a lot of information. Lets go through them group by group

What is cached :

Files In Cache		 : 1
File Segments Cached	 : 512
File Segments Uncached	 : 0
File Segments Present	 : 512

1 File is cached and that has resulted in 512 segments

The beauty of PerfAccel is that when I/O happens on a File, it does not cache the whole file (which can be 1GB, 10GB or 100Gb or even 1TB …)
Rather, it caches only the hot data in segments of 2MB each. That’s why we can see that there are 512 segments of 2 MB each i.e. A total of
1024 MB or 1GB of cached data. This matches with the 1GB of first time read we had done just now.

About Read/Write I/O :

Read Hits		 : 0
Read Misses		 : 262144
Write Misses		 : 262144
Write Hits		 : 0

Note: Here, the values are in 4K blocks

When we did the 1st 1GB write, that would have resulted in 262144 Write Misses, as the file was not cached at that time. So, all writes went to the Source side only.
The Read/Write misses signify that the I/O happened on the Source instead of the Cache.
Whereas, Read/Write hits signify that the I/O happened on the Cache a direct cache hit

Then, the 262144 Read Misses would have resulted when we did the 1st 1GB dd Read.

About Attributes Ops :

Attribute Read Hits	 : 4
Attribute Read Misses	 : 283
Attribute Write Hits	 : 0
Attribute Write Misses	 : 0

This one is another important Analytics data point, the Attribute access details of a file. Its’ built out of the operations like getattr, setattr arising out of commands like chown, stat, ls, chmod etc. Apart from I/O, these attributes are also important during any I/O analysis.

Here, we can see that there were 283 Attribute read misses i.e. 283 times attributes were served from the NFS Source itself, as the file was not yet cached. But, what is the Attribute Read Hits ?? Well, this is another beauty of PerfAccel. Apart from hot data, PerfAccel caches File Attributes as well. This will improve the performance of Network workload a lot as they don’t have to do a Round-trip on the Network for attributes.

Due to that, when the Read happened and the file was cached, the attributes were also cached and the subsequent attribute reads were serviced from the cache. That’s what the 4 Attribute Read Hits signify

Same for the Attribute Write Hits & Misses. We have done no such operation, so the values are 0. We will do it later

Apart from the above, there are many other groups of information also (like Links, Directories, Evictions etc.) in this trace, but we will not go through them right now.

c. 1GB dd Re-read
Now, lets Neutralize the system cache using drop_caches and do a Re-Read.

# dd if=iofile.1 of=/dev/null bs=1M count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.91133 s, 275 MB/s

See the dd read throughput. Its a whopping 275 MB/s (The first time dd Read gave us 55.9 MB/s). The difference is that, this time we are reading from PerfAccel cache. This is definitely going to boost the Application performance

# dgpctl -cache trace io


-------------------------------------------------------------------------------------------
Cache Read 	  Cache Write    Source Read    Source Write   Source Name 	 Source Dir   
-------------------------------------------------------------------------------------------
    1.0 GB         0.0 KB         1.0 GB          1.0 GB   ecad_source     /app_data      

We can see that there is a 1GB read from the Cache side

# dgpctl -cache trace brief ecad_source

-----------------------------------------------------------
	      * ecad_source: PERFACCEL CACHE STATISTICS * 
-----------------------------------------------------------
Files In Cache		 : 1
File Segments Cached	 : 512
File Segments Uncached	 : 0
File Segments Present	 : 512
Read Hits		 : 262144
Read Misses		 : 262144
Write Misses		 : 262144
Write Hits		 : 0
Writeback Hits		 : 0
Attribute Read Hits	 : 5
Attribute Read Misses	 : 309
Attribute Write Hits	 : 0
Attribute Write Misses	 : 0
Links Cached		 : 0
Link Hits		 : 0
Link Misses		 : 0
Link Cache Usage	 : 0.0 KB
Directories cached	 : 0
Directories uncached	 : 0
ReadDir Hits		 : 0
ReadDir Misses		 : 0
Dir Attribute Read Hits	 : 0
Dir Attribute Read Misses: 0
Directories Cache Usage	 : 0.0 KB
Total Cache Usage	 : 1.0 GB (10.02%)
Source Cache Capacity	 : 10.0 GB
Pending I/O requests	 : 0
Misses During Cacheify	 : 0
Max Cache Usage		 : 1.0 GB
Total Cacheify		 : 1.0 GB
File Cache Limit	 : 100%
Cache Uptime		 : 0 hrs 40 mins 11 secs

The Read Hits are 262144  4K blocks which amounts to 1GB of Read I/O being served from the Cache

d. 1GB dd Re-write
Now, the file is cached lets re-write the file without truncating it

# dd if=/dev/zero of=iofile.1 bs=1M count=1k conv=fsync,notrunc
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 30.2236 s, 33.88 MB/s

# dgpctl -cache trace io

-------------------------------------------------------------------------------------------
Cache Read 	  Cache Write    Source Read    Source Write   Source Name 	 Source Dir   
-------------------------------------------------------------------------------------------
    1.0 GB         1.0 GB         1.0 GB          2.0 GB   ecad_source     /app_data      

In the IO trace we can see that there has been a 1GB write on both the Source as well as Cache. This is because the file is cached. So that data has to be updated in both ends (Write-through policy !!)

# dgpctl -cache trace brief ecad_source

-----------------------------------------------------------
	      * ecad_source: PERFACCEL CACHE STATISTICS * 
-----------------------------------------------------------
Files In Cache		 : 1
File Segments Cached	 : 512
File Segments Uncached	 : 0
File Segments Present	 : 512
Read Hits		 : 262144
Read Misses		 : 262144
Write Misses		 : 262144
Write Hits		 : 262144
Writeback Hits		 : 0
Attribute Read Hits	 : 7
Attribute Read Misses	 : 355
Attribute Write Hits	 : 0
Attribute Write Misses	 : 0
Links Cached		 : 0
Link Hits		 : 0
Link Misses		 : 0
Link Cache Usage	 : 0.0 KB
Directories cached	 : 0
Directories uncached	 : 0
ReadDir Hits		 : 0
ReadDir Misses		 : 0
Dir Attribute Read Hits	 : 0
Dir Attribute Read Misses: 0
Directories Cache Usage	 : 0.0 KB
Total Cache Usage	 : 1.0 GB (10.02%)
Source Cache Capacity	 : 10.0 GB
Pending I/O requests	 : 0
Misses During Cacheify	 : 0
Max Cache Usage		 : 1.0 GB
Total Cacheify		 : 1.0 GB
File Cache Limit	 : 100%
Cache Uptime		 : 0 hrs 45 mins 19 secs

See the write hits. Its 262144 , the same 1GB

Now, lets see all the remaining Analytical traces for our I/Os done till now

File Analytics Trace :

# dgpctl -cache trace file  

------------------------------------------------------------------------------------------------------------------------------------------
   Inode Num             Segments      R Hits        W Hits        BlkCached(4k) LATime(s)   ResTime(s)  Sys Reads   Sys Writes  File Name
------------------------------------------------------------------------------------------------------------------------------------------
   6815748               512           262144        262144        262144        1469        3598        524288      524288      /app_data/iofile.1

See, this one… this clearly shows that our file /app_data/iofile.1

  1. Has Inode number 6815748
  2. Is cached with 512 segments
  3. Has 262144 or 1GB of Read hits & Write Hits
  4. Total cached size is  262144  blocks of 4K size i.e. 1073741824 bytes
  5. Was accessed last time 1469 secs back
  6. Its Resident set time is 3598 secs i.e. time since when it was cached or the first read
  7. Number of Read & Write system calls on the file are 524288 & 524288 respectively

Lets drill down even more on the File Analytics and see what PerfAccel provides us
The number of file analytics metrics provided are more than 30. So, it wont fit even a 42 inch screen…just kidding
But, its really long, so just for reference purpose i am putting the command below

# dgpctl -cache trace file –long

Note: The trace output of this command extends beyond my screen , I had to reduce the font size to fit it in a line  (so much analytical data here 🙂 )

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 inode-num segments c-read c-write blkcached(4k) atime(s) rtime(s) open close c-rattr c-wattr s-read s-write c-rtput s-rtput c-rlat s-rlat c-wtput s-wtput c-wlat s-wlat c-rqlen s-rqlen c-wqlen s-wqlen    ctime    rh-ratio rw-ratio sys-read sys-write fsync frdlck fwrlck mmap create file-name
                                                                                                              (MBps) (MBps)   (ms)   (ms)  (MBps)  (MBps)   (ms)   (ms)   (avg)   (avg)    (avg) (avg)                  (per)   (per)

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 6815748    512    262144   262144   262144       1475      3603    4     4      7       2    262144   262144  785.6   62.9   0.171  3.915   597    30.2   3.339  6706.46  0.019  0.463   7.264  803.899  1488881812    50       50     524288    524288    2     0       0     0    1    /app_data/iofile.1 

Ok. so what new things we see here :

  1. 4 opens & 4 closes of this file were done
  2. 7 stat calls were done on this file
  3. There were 262144 Read and 262144 Write Misses on this file (This was the 1st time 1GB dd read and write)
  4. The file has achieved a Read throughput of 785.6 MB/s from Cache side & 62.9 MB/s from Source side
  5. The Avg read latency is 0.171 ms for Cache side and 3.915 ms for Source (NFS) side
  6. The file has achieved a Write throughput of 597 MB/s from Cache side & 30.2 MB/s from source side
  7. The Avg write latency is 3.339 ms for Cache side and 6706.46 ms for Source (NFS) side (too slow writes in NFS)
  8. The Read Effectiveness or the Read Hit ratio is 50% (How much of the Read I/O was served from the cache)
    (We did 1GB read 2 times, and first time was from Source, 2nd time was from cache. So, 50%)
  9. The Write Effectiveness or the Write Hit ratio is 50% (How much of the Write I/O was served from the cache)
    (We did 1GB read 2 times, and first time was from Source, 2nd time was from cache. So, 50%)
  10. The Avg, Read Queue Length was 0.019 on cache side and 0.463 on Source side
  11. The Avg. Write Queue Length was 7.264 on source side and 803.899 on source side

Owing to its really long output, the command has divided the metrics into Analysis groups. We will go through each of them instead.

a. The Read/Write I/O metrics

# dgpctl -cache trace file –io

----------------------------------------------------------------------------------------------------------
   Inode Num              c-read        c-write       s-read    s-write   sys-read    sys-write   file-name   
----------------------------------------------------------------------------------------------------------
   6815748               262144        262144        262144    262144    524288      524288      /app_data/iofile.1

We have already covered all metrics from this group earlier

b. The Read/Write IOPS metrics

# dgpctl -cache trace file –iops

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   Inode Num             CRead        SRead        CRead        SRead        CWrite       SWrite       CWrite       SWrite       CRead        SRead        CWrite       SWrite       File Name    
                         throughput   throughput   latency      latency      throughput   throughput   latency      latency      qlength      qlength      qlength      qlength                   
                         (MBps)       (MBps)       (ms)         (ms)         (MBps)       (MBps)       (ms)         (ms)         (avg)        (avg)        (avg)        (avg)                     

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   6815748               785.6        62.9         0.171        3.915        597.0        30.2         3.339        6706.459     0.019        0.463        7.264        803.899      /app_data/iofile.1

We have covered around half of the metrics in the earlier file trace discussion. But, lets see the new ones here which were not present earlier

Apart from the IOPS metrics, this provides some more fine grained file Analytics metrics

  1. Cache side Average Read queue length i.e. In average how many Read I/Os are queued at any point of time
  2. Source side Average Read queue length i.e. In average how many Read I/Os are queued at any point of time
  3. Cache side Average Write queue length i.e. In average how many Write I/Os are queued at any point of time
  4. Source side Average Write queue length i.e. In average how many Write I/Os are queued at any point of time

c. The Storage Capacity metrics

# dgpctl -cache trace file –capacity

--------------------------------------------------------------
   Inode Num             Segments      BlkCached(4k) File Name   
--------------------------------------------------------------
   6815748               512           262144        /app_data/iofile.1

We have already covered these metrics

d. The Metadata Operation metrics

# dgpctl -cache trace file –meta

-----------------------------------------------------------------------------------------------------------------------------------------
   Inode Num             Open      Close     Rattr     Wattr       Fsync       Frdlck      Fwrlck      Mmap        Create      File Name  
-----------------------------------------------------------------------------------------------------------------------------------------
   6815748               4         4         7         0           2           0           0           0           1        /app_data/iofile.1

Now lets see the metadata metrics of the file :

  1. Number of Opens & Closes done on the file
  2. Number of attribute reads and write done
  3. Number of fsync calls on the file
  4. Number of flock–Read calls on the file
  5. Number of flock-Write calls on the file
  6. Number of mmap calls done on the file
  7. Was the file created after the source was created or it was existing old file

e. File Offset based I/O Analytics

Ok… that’s lot of metrics. But, what if i want to see offset based analytics of the file. I mean, which offsets are cached,
which are having more read hits, more write hits, which offset was accessed recently etc. Well, PerfAccel provides that as well

# dgpctl -cache trace complete ecad_source

------------------------------------------------------------------------------------------------------------------------
   Inode Num              Offset        R Hits        W Hits        Size          LATime(s)   ResTime(s)  File Name   
------------------------------------------------------------------------------------------------------------------------
    6815748               0             512           512           2097152       5299        7427        /app_data/iofile.1
    6815748               2097152       512           512           2097152       5299        6934        /app_data/iofile.1
    6815748               4194304       512           512           2097152       5299        6934        /app_data/iofile.1
    6815748               6291456       512           512           2097152       5299        6934        /app_data/iofile.1
    6815748               8388608       512           512           2097152       5299        6934        /app_data/iofile.1
    6815748               10485760      512           512           2097152       5299        6934        /app_data/iofile.1
    6815748               12582912      512           512           2097152       5299        6934        /app_data/iofile.1
    .......
    .......
    6815748               1065353216    512           512           2097152       5273        6916        /app_data/iofile.1
    6815748               1067450368    512           512           2097152       5273        6916        /app_data/iofile.1
    6815748               1069547520    512           512           2097152       5273        6916        /app_data/iofile.1
    6815748               1071644672    512           512           2097152       5273        6916        /app_data/iofile.1

Now, as we know, we have a 1GB file with 512 segments of 2MB each. This trace shows us the metrics for each of the 2MB chunk
In the above output, i have skipped most of the middle chunks and only showing the beginning and end of the output.

Here, for each chunk i.e. 2MB worth of offset, we can see various metrics like
a. Offset start
b. Read Hits
c. Write Hits
d. Amount of data cached for the segment. It happens many times that in the 2MB segment, only 100KB or something like that is cached.
e. Time the offset range was last accessed – very helpful if we want to find out the specific areas of the file which the Application is constantly accessing
f. Resident time of the chunk in the cache

What about directory operations on the source ? mkdir, rmdir, lookup etc. Well, here it is …

# dgpctl -cache trace ops

-----------------------ecad_source: PERFACCEL OPS STATISTICS------------------------------
Type         Cache      Latency (msec) Ops/sec    Source     Latency (msec) Ops/sec   
-----------------------------------------------------------------------------------
GETATTR      4          0              0          1196       355            3369      
SETATTR      0          0              0          0          0              0         
FOLLOW_LINK  0          0              0          0          0              0         
LOOKUP       0          0              0          16         9              1778      
CREATE       0          0              0          1          103            10        
OPEN         0          0              0          76         15             5067      
MKDIR        0          0              0          0          0              0         
RMDIR        0          0              0          0          0              0    

 

This gives us the meta operation metrics for both source and cache side.
The cache side part will be applicable when the file is cached and the attributes come from the cache itself (like the 4 getattr ops above)

Well, that’s it for this walkthrough. Too much of Analytics as compared to the first blog…right ?

In the next blog, we will go through how PerfAccel metrics look for a little more less-simpler (i wont call it complex) environment involving :
a. Multiple Sources (local & network)
b. Multiple caches
c. Multiple files in each source
d. I/O as well as Metadata access

In the meantime, you can go ahead and Download PerfAccel right now and get hands on with the exciting features.

Stay tuned…

Share it
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
COMMENTS SECTION

Leave a Reply

Be the First to Comment!

Leave a Reply

wpDiscuz