5. Application behavior when the data working set size exceeds PerfAccel Cache size – Dynamic Eviction Policy of PerfAccel

In the previous blogs we saw how PerfAccel does Intelligent & Dynamic data placement/caching. In this blog, we will go through a scenario where our Application’s working set size exceeds much more than the PerfAccel Cache space provided. We will see how PerfAccel deals with the scenario dynamically by performing Eviction based on LRU policy for the segments cached.

Let’s say, our application data is coming from a NFS share that is mounted on /app-data

# df

Filesystem                                         Size  Used Avail Use% Mounted on
/dev/sda1                                           11G  9.2G  905M  92% /
tmpfs                                              1.1G     0  1.1G   0% /dev/shm
/dev/sdb                                            12G  5.0G  6.3G  45% /data
192.168.1.178:/data_ext4/critical/app/ecad_module  349G  291G   41G  88% /app_data

So, lets create the PerfAccel cache

# dgpctl -cache list
Device        Total size   Free size  Alloc size  State    IP address      Name     
/dev/md127       23.6 GB     23.5 GB      0.0 MB  Online   192.168.1.38    sata_cache

Now, let’s create the Source and allocate only 1GB of cache space to it

# dgpctl -source create app_data /app_data sata_cache 1G
Source app_data was successfully created
1G of cache from sata_cache successfully added to source app_data

# dgpctl -source list
Source                           Status       Cache           Size       Mount Point     Type       Mode
app_data                         ONLINE       sata_cache      1.0 GB     /app_data       nfs        WT   

For this blog, we will use simple dd Read/Write workload. Here, we will simulate how the cache gets filled and demonstrate what happens next as a result of that. Before going there, lets understand how PerfAccel deals with the situation.

Every PerfAccel Source has 3 configurable limits for the Cache. Below are the 3 limits along with their default values.

  1. Feasibility fail limit : 90% of allocated Cache space
  2. Critical limit : 80% of allocated Cache space
  3. Cleanup limit : 60% of allocated Cache space

If the cache usage goes beyond the Critical limit, PerfAccel triggers an Eviction for the Source. The Eviction or Cleanup will keep on evicting cached segments with LRU Policy, until it reaches the “Cleanup limit” of cache usage.

In the meantime, the I/O continues to happen from both sides i.e. from cache side for cached offsets or source side for un-cached offsets. New segments also continue to be brought into the cache while the eviction is running. So, this does not affect the Application I/O in any way.

The Feasibility fail limit is a hard stop. Once the cache usage reaches this limit, no new segments are allowed to come into the cache until the Eviction frees up the required space.

So, for a 100 GB Cache, the default limits would be :

  1. Feasibility fail limit : 90% of 100GB = 90GB
  2. Critical limit : 80% of 100GB = 80GB
  3. Cleanup limit : 60% of 100GB = 60GB

In other words, the Eviction gets kicked in as soon as usage touches 80 GB and it frees up 20 GB of the used size i.e. eviction runs until the cache usage becomes 60 GB. New data still get cached when the usage is in between 80 GB to 90GB while eviction is running in parallel. Once the usage crosses 90GB, no new data is allowed to come into the cache until the eviction completes and takes down the cache usage to 60GB

Now, let’s come back to our hands-on for the scenario. We have a source with 1GB of cache allocated to it.

So, the limits are :

  1. Feasibility limit : 90% of 1GB = 921 MB
  2. Critical limit : 80% of 1GB = 819 MB
  3. Cleanup limit : 60% of 1GB = 614 MB

This info can be verified from the command :

# dgpctl -source status app_data
Source Name            : app_data
Used Size              : 291.3 GB
Free Size              : 56.9 GB
Total Size             : 348.2 GB
Status                 : ONLINE
Source Type            : nfs
Source Mount Directory : /app_data
Data Group             : /ALL/DRF/TEST_NODES
Source Option          : cto, flock

Cache Info:
Cache State            : Enabled
Monitor Status         : Stopped
Cache Type             : writethrough
Feasibility limit      : 921.6 MB
Critical limit         : 819.2 MB
Cleanup limit          : 614.4 MB
File Cache Limit       : 100%

Cache Name                    Used Size           Free size         Total Size
-------------------------------------------------------------------------------
sata_cache                    744.0 MB          280.0 MB            1.0 GB

So, let’s create a 1 GB file on the source

# dd if=/dev/zero of=iofile.1 bs=1M count=1k conv=fsync,notrunc
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 28.054 s, 38.3 MB/s

Let’s neutralize the system cache (so that the I/O happens from the disk)

sync; echo 3 > /proc/sys/vm/drop_caches

Now, let’s read the first 780MB of the file, so that it’s within the 80% limit

# dd if=iofile.1 of=/dev/null bs=1M count=780
780+0 records in
780+0 records out
817889280 bytes (818 MB) copied, 19.3481 s, 42.3 MB/s

Lets go through the PerfAccel I/O traces

# dgpctl -cache trace usage
-----------------------------------------------------------------------------------------------------------------
          size          usage  use%     max usage            nfiles            ndirs   source name     source dir    
-----------------------------------------------------------------------------------------------------------------
        1.0 GB       782.0 MB 76.4%      782.0 MB                 1                0   app_data         /app_data   


# dgpctl -cache trace file
-------------------------------------------------------------------------------------------------------------------------------
 Inode Num          Segments     R Hits       W Hits       Size             LATime(s)          ResTime(s)          File Name
-------------------------------------------------------------------------------------------------------------------------------
 6849762            391          0            0            819986432        35                 91                  /app_data/iofile.1

# dgpctl -cache trace brief app_data

-----------------------------------------------------------
              * app_data: PERFACCEL CACHE STATISTICS *
-----------------------------------------------------------
Files In Cache          : 1
File Segments Cached    : 391
File Segments Uncached  : 0
File Segments Present   : 391
Read Hits               : 0
Read Misses             : 199712
Write Misses            : 262144
Write Hits              : 0
Writeback Hits          : 0
Attribute Read Hits     : 1
Attribute Read Misses   : 23
Attribute Write Hits    : 0
Attribute Write Misses  : 0
Links Cached            : 0
Link Hits               : 0
Link Misses             : 0
Link Cache Usage        : 0.0 KB
Directories cached      : 0
Directories uncached    : 0
ReadDir Hits            : 0
ReadDir Misses          : 0
Dir Attribute Read Hits : 0
Dir Attribute Read Misses: 0
Directories Cache Usage : 0.0 KB
Total Cache Usage       : 782.4 MB (76.40%)
Source Cache Capacity   : 1.0 GB
Pending I/O requests    : 0
Misses During Cacheify  : 0
Max Cache Usage         : 782.0 MB
Total Cacheify          : 780.1 MB
File Cache Limit        : 100%
Cache Uptime            : 0 hrs 2 mins 53 secs


Cacheify Failure Reasons
------------------------
Count of Not Enough Space: 0
Cleaner In Progress     : 0
-----------------------------------------------------------
              * CLEANUP STATISTICS *          
-----------------------------------------------------------
Total cleanup runs      : 0
Last cleanup run at     : 0
Total chunks thrown     : 0
Time taken for last run : 0 sec
Last run freed space    : 0.0 KB
Total freed space       : 0.0 KB
Total Time in cleaner   : 0 sec
Total Time in bucketize : 0 sec
Evict fail due to flush : 0
Evict fail due to other : 0
TTL Expiry Eviction     : 0 (Segments)
Eviction Meta Freed     : 0.0 KB
-----------------------------------------------------------
              * WRITEBACK STATISTICS *          
-----------------------------------------------------------
Pending Attribute Flush : 0
Bytes Pending Writeback : 0.0 KB
Data scheduled for flush : 0.0 KB
Total Bytes Flushed     : 0.0 KB
-----------------------------------------------------------
              * LAST FLUSH STATISTICS *    
-----------------------------------------------------------
Flush ID                : 0
Scheduling time         : 0.00 sec
IO time                 : 0.00 sec
Bytes Flushed           : 0.0 KB
-----------------------------------------------------------

We can clearly see that the cache usage is below 80% and 391 segments are cached. Let’s see what all are cached with the help of PerfAccel complete trace

------------------------------------------------------------------------------------------------------------------------
   Inode Num              Offset        R Hits        W Hits        Size          LATime(s)   ResTime(s)  File Name  
------------------------------------------------------------------------------------------------------------------------
    6849762               0             0             0             2097152       232         274         /app_data/iofile.1
    6849762               2097152       0             0             2097152       232         232         /app_data/iofile.1
    6849762               4194304       0             0             2097152       232         232         /app_data/iofile.1
    6849762               6291456       0             0             2097152       232         232         /app_data/iofile.1
    6849762               8388608       0             0             2097152       232         232         /app_data/iofile.1
    6849762               10485760      0             0             2097152       232         232         /app_data/iofile.1
    6849762               12582912      0             0             2097152       232         232         /app_data/iofile.1
    6849762               14680064      0             0             2097152       232         232         /app_data/iofile.1
    6849762               16777216      0             0             2097152       232         232         /app_data/iofile.1
    6849762               18874368      0             0             2097152       232         232         /app_data/iofile.1
    6849762               20971520      0             0             2097152       232         232         /app_data/iofile.1
    6849762               23068672      0             0             2097152       232         232         /app_data/iofile.1
    6849762               25165824      0             0             2097152       232         232         /app_data/iofile.1
    6849762               27262976      0             0             2097152       232         232         /app_data/iofile.1
    6849762               29360128      0             0             2097152       232         232         /app_data/iofile.1
    6849762               31457280      0             0             2097152       232         232         /app_data/iofile.1
    6849762               33554432      0             0             2097152       232         232         /app_data/iofile.1
    6849762               35651584      0             0             2097152       232         232         /app_data/iofile.1
    6849762               37748736      0             0             2097152       232         232         /app_data/iofile.1
    6849762               39845888      0             0             2097152       232         232         /app_data/iofile.1
    6849762               41943040      0             0             2097152       232         232         /app_data/iofile.1
    6849762               44040192      0             0             2097152       232         232         /app_data/iofile.1
    6849762               46137344      0             0             2097152       232         232         /app_data/iofile.1
    6849762               48234496      0             0             2097152       232         232         /app_data/iofile.1
......
.......
    6849762               761266176     0             0             2097152       219         219         /app_data/iofile.1
    6849762               763363328     0             0             2097152       219         219         /app_data/iofile.1
    6849762               765460480     0             0             2097152       219         219         /app_data/iofile.1
    6849762               767557632     0             0             2097152       219         219         /app_data/iofile.1
    6849762               769654784     0             0             2097152       219         219         /app_data/iofile.1
    6849762               771751936     0             0             2097152       219         219         /app_data/iofile.1
    6849762               773849088     0             0             2097152       219         219         /app_data/iofile.1
    6849762               775946240     0             0             2097152       219         219         /app_data/iofile.1
    6849762               778043392     0             0             2097152       219         219         /app_data/iofile.1
    6849762               780140544     0             0             2097152       219         219         /app_data/iofile.1
    6849762               782237696     0             0             2097152       219         219         /app_data/iofile.1
    6849762               784334848     0             0             2097152       219         219         /app_data/iofile.1
    6849762               786432000     0             0             2097152       218         219         /app_data/iofile.1
    6849762               788529152     0             0             2097152       218         218         /app_data/iofile.1
    6849762               790626304     0             0             2097152       218         218         /app_data/iofile.1
    6849762               792723456     0             0             2097152       218         218         /app_data/iofile.1
    6849762               794820608     0             0             2097152       218         218         /app_data/iofile.1
    6849762               796917760     0             0             2097152       218         218         /app_data/iofile.1
    6849762               799014912     0             0             2097152       218         218         /app_data/iofile.1
    6849762               801112064     0             0             2097152       218         218         /app_data/iofile.1
    6849762               803209216     0             0             2097152       218         218         /app_data/iofile.1
    6849762               805306368     0             0             2097152       218         218         /app_data/iofile.1
    6849762               807403520     0             0             2097152       218         218         /app_data/iofile.1
    6849762               809500672     0             0             2097152       218         218         /app_data/iofile.1
    6849762               811597824     0             0             2097152       218         218         /app_data/iofile.1
    6849762               813694976     0             0             2097152       218         218         /app_data/iofile.1
    6849762               815792128     0             0             2097152       218         218         /app_data/iofile.1
    6849762               817889280     0             0             2097152       218         218         /app_data/iofile.1

So, starting from offset 0 till the last offset 817889280, all segments are cached. No read hits have happened on any of the segment yet.

Now, let’s do a read I/O for the offset from offset o to 200MB, so that that area of the file becomes the recently accessed data.

# dd if=iofile.1 of=/dev/null bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.467515 s, 449 MB/s

We can verify if those 100 segments i.e. from offset 0 to 200MB got the Read hits or not, as we did a 2nd time read on them.

# dgpctl –cache trace complete app_data

We can see here that the segments from offset 0 till 200MB have complete read hits as we accessed them just now. Whereas, all other segments beyond that offset have 0 Readhits. Also, the LATime i.e. the last access time of these segments is only 6 seconds back, whereas all other segments have last access time more than 398 seconds

------------------------------------------------------------------------------------------------------------------------
   Inode Num              Offset        R Hits        W Hits        Size          LATime(s)   ResTime(s)  File Name  
------------------------------------------------------------------------------------------------------------------------
    6849762               0             512           0             2097152       6           443         /app_data/iofile.1
    6849762               2097152       512           0             2097152       6           401         /app_data/iofile.1
    6849762               4194304       512           0             2097152       6           401         /app_data/iofile.1
    6849762               6291456       512           0             2097152       6           401         /app_data/iofile.1
    6849762               8388608       512           0             2097152       6           401         /app_data/iofile.1
    6849762               10485760      512           0             2097152       6           401         /app_data/iofile.1
    6849762               12582912      512           0             2097152       6           401         /app_data/iofile.1
    6849762               14680064      512           0             2097152       6           401         /app_data/iofile.1
    6849762               16777216      512           0             2097152       6           401         /app_data/iofile.1
    6849762               18874368      512           0             2097152       6           401         /app_data/iofile.1
    6849762               20971520      512           0             2097152       6           401         /app_data/iofile.1
    6849762               23068672      512           0             2097152       6           401         /app_data/iofile.1
    6849762               25165824      512           0             2097152       6           401         /app_data/iofile.1
    6849762               27262976      512           0             2097152       6           401         /app_data/iofile.1
    6849762               29360128      512           0             2097152       6           401         /app_data/iofile.1
    6849762               31457280      512           0             2097152       6           401         /app_data/iofile.1
    6849762               33554432      512           0             2097152       6           401         /app_data/iofile.1
    6849762               35651584      512           0             2097152       6           401         /app_data/iofile.1
    6849762               37748736      512           0             2097152       6           401         /app_data/iofile.1
    6849762               39845888      512           0             2097152       6           401         /app_data/iofile.1
    6849762               41943040      512           0             2097152       6           401         /app_data/iofile.1
    6849762               44040192      512           0             2097152       6           401         /app_data/iofile.1
    6849762               46137344      512           0             2097152       6           401         /app_data/iofile.1
    6849762               48234496      512           0             2097152       6           401         /app_data/iofile.1
    6849762               50331648      512           0             2097152       6           401         /app_data/iofile.1
    6849762               52428800      512           0             2097152       6           401         /app_data/iofile.1
    6849762               54525952      512           0             2097152       6           400         /app_data/iofile.1
    6849762               56623104      512           0             2097152       6           400         /app_data/iofile.1
    6849762               58720256      512           0             2097152       6           400         /app_data/iofile.1
    6849762               60817408      512           0             2097152       6           400         /app_data/iofile.1
    6849762               62914560      512           0             2097152       6           400         /app_data/iofile.1
    6849762               65011712      512           0             2097152       6           400         /app_data/iofile.1
    6849762               67108864      512           0             2097152       6           400         /app_data/iofile.1
    6849762               69206016      512           0             2097152       6           400         /app_data/iofile.1
    6849762               71303168      512           0             2097152       6           400         /app_data/iofile.1
    6849762               73400320      512           0             2097152       6           400         /app_data/iofile.1
    6849762               75497472      512           0             2097152       6           400         /app_data/iofile.1
    6849762               77594624      512           0             2097152       6           400         /app_data/iofile.1
    6849762               79691776      512           0             2097152       6           400         /app_data/iofile.1
    6849762               81788928      512           0             2097152       6           400         /app_data/iofile.1
    6849762               83886080      512           0             2097152       6           400         /app_data/iofile.1
    6849762               85983232      512           0             2097152       6           400         /app_data/iofile.1
    6849762               88080384      512           0             2097152       6           400         /app_data/iofile.1
    6849762               90177536      512           0             2097152       6           400         /app_data/iofile.1
    6849762               92274688      512           0             2097152       6           400         /app_data/iofile.1
    6849762               94371840      512           0             2097152       6           400         /app_data/iofile.1
    6849762               96468992      512           0             2097152       6           400         /app_data/iofile.1
    6849762               98566144      512           0             2097152       6           400         /app_data/iofile.1
    6849762               100663296     512           0             2097152       6           400         /app_data/iofile.1
    6849762               102760448     512           0             2097152       6           400         /app_data/iofile.1
    6849762               104857600     512           0             2097152       6           400         /app_data/iofile.1
    6849762               106954752     512           0             2097152       6           400         /app_data/iofile.1
    6849762               109051904     512           0             2097152       6           400         /app_data/iofile.1
    6849762               111149056     512           0             2097152       6           400         /app_data/iofile.1
    6849762               113246208     512           0             2097152       6           400         /app_data/iofile.1
    6849762               115343360     512           0             2097152       6           400         /app_data/iofile.1
    6849762               117440512     512           0             2097152       6           400         /app_data/iofile.1
    6849762               119537664     512           0             2097152       6           400         /app_data/iofile.1
    6849762               121634816     512           0             2097152       6           400         /app_data/iofile.1
    6849762               123731968     512           0             2097152       6           400         /app_data/iofile.1
    6849762               125829120     512           0             2097152       6           400         /app_data/iofile.1
    6849762               127926272     512           0             2097152       6           400         /app_data/iofile.1
    6849762               130023424     512           0             2097152       6           400         /app_data/iofile.1
    6849762               132120576     512           0             2097152       6           400         /app_data/iofile.1
    6849762               134217728     512           0             2097152       6           400         /app_data/iofile.1
    6849762               136314880     512           0             2097152       6           400         /app_data/iofile.1
    6849762               138412032     512           0             2097152       6           400         /app_data/iofile.1
    6849762               140509184     512           0             2097152       6           399         /app_data/iofile.1
    6849762               142606336     512           0             2097152       6           399         /app_data/iofile.1
    6849762               144703488     512           0             2097152       6           399         /app_data/iofile.1
    6849762               146800640     512           0             2097152       6           399         /app_data/iofile.1
    6849762               148897792     512           0             2097152       6           399         /app_data/iofile.1
    6849762               150994944     512           0             2097152       6           399         /app_data/iofile.1
    6849762               153092096     512           0             2097152       6           399         /app_data/iofile.1
    6849762               155189248     512           0             2097152       6           399         /app_data/iofile.1
    6849762               157286400     512           0             2097152       6           399         /app_data/iofile.1
    6849762               159383552     512           0             2097152       6           399         /app_data/iofile.1
    6849762               161480704     512           0             2097152       6           399         /app_data/iofile.1
    6849762               163577856     512           0             2097152       6           399         /app_data/iofile.1
    6849762               165675008     512           0             2097152       6           399         /app_data/iofile.1
    6849762               167772160     512           0             2097152       6           399         /app_data/iofile.1
    6849762               169869312     512           0             2097152       6           399         /app_data/iofile.1
    6849762               171966464     512           0             2097152       6           399         /app_data/iofile.1
    6849762               174063616     512           0             2097152       6           399         /app_data/iofile.1
    6849762               176160768     512           0             2097152       6           399         /app_data/iofile.1
    6849762               178257920     512           0             2097152       6           399         /app_data/iofile.1
    6849762               180355072     512           0             2097152       6           399         /app_data/iofile.1
    6849762               182452224     512           0             2097152       6           399         /app_data/iofile.1
    6849762               184549376     512           0             2097152       6           399         /app_data/iofile.1
    6849762               186646528     512           0             2097152       6           399         /app_data/iofile.1
    6849762               188743680     512           0             2097152       6           399         /app_data/iofile.1
    6849762               190840832     512           0             2097152       6           399         /app_data/iofile.1
    6849762               192937984     512           0             2097152       6           399         /app_data/iofile.1
    6849762               195035136     512           0             2097152       6           399         /app_data/iofile.1
    6849762               197132288     512           0             2097152       6           399         /app_data/iofile.1
    6849762               199229440     512           0             2097152       6           399         /app_data/iofile.1
    6849762               201326592     512           0             2097152       6           399         /app_data/iofile.1
    6849762               203423744     512           0             2097152       6           399         /app_data/iofile.1
    6849762               205520896     512           0             2097152       6           399         /app_data/iofile.1
    6849762               207618048     512           0             2097152       6           399         /app_data/iofile.1
    6849762               209715200     32            0             2097152       6           399         /app_data/iofile.1
    6849762               211812352     0             0             2097152       399         399         /app_data/iofile.1
    6849762               213909504     0             0             2097152       398         399         /app_data/iofile.1
    6849762               216006656     0             0             2097152       398         398         /app_data/iofile.1
.......
.......

Now, let’s bring in some new data and then see which cached segments get evicted

For that we will do a read from offset 750MB till 900MB

# dd if=iofile.1 of=/dev/null bs=1M skip=750 count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 4.57255 s, 45.9 MB/s
# dgpctl -cache trace usage
-----------------------------------------------------------------------------------------------------------------
          size          usage  use%     max usage            nfiles            ndirs   source name     source dir   
-----------------------------------------------------------------------------------------------------------------
        1.0 GB       744.4 MB 72.7%      820.4 MB                 1                0   app_data         /app_data

PerfAccel provides various ways to find out details of Eviction runs

# dgpctl -cache trace cleanup

--------------------------------------------------------------------------------------------------
Cleanup Runs     Last Freed    Total Freed   TotalMetaFreed    Source Name              Source Dir
--------------------------------------------------------------------------------------------------
           2         4.0 MB       208.0 MB           0.0 KB    app_data                 /app_data    

It clearly shows that 2 Eviction runs have happened and total amount of cached segments evicted is 208 MB. To get more details, we can use the next command.

# dgpctl -cache trace brief app_data
-----------------------------------------------------------
              * app_data: PERFACCEL CACHE STATISTICS *
-----------------------------------------------------------
Files In Cache          : 1
File Segments Cached    : 476
File Segments Uncached  : 104
File Segments Present   : 372
Read Hits               : 58944
Read Misses             : 243232
Write Misses            : 262144
Write Hits              : 0
Writeback Hits          : 0
Attribute Read Hits     : 4
Attribute Read Misses   : 49
Attribute Write Hits    : 0
Attribute Write Misses  : 0
Links Cached            : 0
Link Hits               : 0
Link Misses             : 0
Link Cache Usage        : 0.0 KB
Directories cached      : 0
Directories uncached    : 0
ReadDir Hits            : 0
ReadDir Misses          : 0
Dir Attribute Read Hits : 0
Dir Attribute Read Misses: 0
Directories Cache Usage : 0.0 KB
Total Cache Usage       : 744.9 MB (72.74%)
Source Cache Capacity   : 1.0 GB
Pending I/O requests    : 0
Misses During Cacheify  : 0
Max Cache Usage         : 820.4 MB
Total Cacheify          : 950.1 MB
File Cache Limit        : 100%
Cache Uptime            : 0 hrs 15 mins 26 secs


Cacheify Failure Reasons
------------------------
Count of Not Enough Space: 0
Cleaner In Progress     : 0
-----------------------------------------------------------
              * CLEANUP STATISTICS *          
-----------------------------------------------------------
Total cleanup runs      : 2
Last cleanup run at     : Mon Mar 20 12:39:32 2017
Total chunks thrown     : 104
Time taken for last run : 0 sec
Last run freed space    : 4.0 MB
Total freed space       : 208.0 MB
Total Time in cleaner   : 0 sec
Total Time in bucketize : 0 sec
Evict fail due to flush : 0
Evict fail due to other : 0
TTL Expiry Eviction     : 0 (Segments)
Eviction Meta Freed     : 0.0 KB
-----------------------------------------------------------

This tells us the same thing that 208 MB of cached data was thrown in 2 Eviction runs. It addition to that information it also tells us that 104 segments/chunks were evicted in this process. During and after this cleanup new segments would have been cached. Now, let’s see which all chunks are present and which are not.

# dgpctl –cache trace complete app_data

------------------------------------------------------------------------------------------------------------------------
   Inode Num              Offset        R Hits        W Hits        Size          LATime(s)   ResTime(s)  File Name  
------------------------------------------------------------------------------------------------------------------------
    6849762               0             512           0             2097152       621         1058        /app_data/iofile.1
    6849762               2097152       512           0             2097152       621         1016        /app_data/iofile.1
    6849762               4194304       512           0             2097152       621         1016        /app_data/iofile.1
    6849762               6291456       512           0             2097152       621         1016        /app_data/iofile.1
    6849762               8388608       512           0             2097152       621         1016        /app_data/iofile.1
    6849762               10485760      512           0             2097152       621         1016        /app_data/iofile.1
    6849762               12582912      512           0             2097152       621         1016        /app_data/iofile.1
    6849762               14680064      512           0             2097152       621         1016        /app_data/iofile.1
    6849762               16777216      512           0             2097152       621         1016        /app_data/iofile.1
    6849762               18874368      512           0             2097152       621         1016        /app_data/iofile.1
    6849762               20971520      512           0             2097152       621         1016        /app_data/iofile.1
........
........
    6849762               205520896     512           0             2097152       621         1014        /app_data/iofile.1
    6849762               207618048     512           0             2097152       621         1014        /app_data/iofile.1
    6849762               209715200     32            0             2097152       621         1014        /app_data/iofile.1
    6849762               429916160     0             0             2097152       1010        1010        /app_data/iofile.1
    6849762               432013312     0             0             2097152       1010        1010        /app_data/iofile.1
    6849762               434110464     0             0             2097152       1010        1010        /app_data/iofile.1
    6849762               436207616     0             0             2097152       1010        1010        /app_data/iofile.1
    6849762               438304768     0             0             2097152       1010        1010        /app_data/iofile.1
........
.........
    6849762               813694976     512           0             2097152       281         1002        /app_data/iofile.1
    6849762               815792128     512           0             2097152       281         1002        /app_data/iofile.1
    6849762               817889280     32            0             2097152       281         1002        /app_data/iofile.1
    6849762               819986432     0             0             2097152       281         281         /app_data/iofile.1
    6849762               822083584     0             0             2097152       281         281         /app_data/iofile.1
    6849762               824180736     0             0             2097152       280         281         /app_data/iofile.1
    6849762               826277888     0             0             2097152       280         280         /app_data/iofile.1
    6849762               828375040     0             0             2097152       280         280         /app_data/iofile.1
    6849762               830472192     0             0             2097152       280         280         /app_data/iofile.1
    6849762               832569344     0             0             2097152       280         280         /app_data/iofile.1
    6849762               834666496     0             0             2097152       280         280         /app_data/iofile.1
    6849762               836763648     0             0             2097152       280         280         /app_data/iofile.1
    6849762               838860800     0             0             2097152       280         280         /app_data/iofile.1
.......
.......
    6849762               979369984     0             0             2097152       277         277         /app_data/iofile.1
    6849762               981467136     0             0             2097152       276         277         /app_data/iofile.1
    6849762               983564288     0             0             2097152       276         277         /app_data/iofile.1
    6849762               985661440     0             0             2097152       276         277         /app_data/iofile.1
    6849762               987758592     0             0             2097152       276         276         /app_data/iofile.1
    6849762               989855744     0             0             2097152       276         276         /app_data/iofile.1
    6849762               991952896     0             0             2097152       276         276         /app_data/iofile.1
    6849762               994050048     0             0             2097152       276         276         /app_data/iofile.1
    6849762               996147200     0             0             2097152       276         276         /app_data/iofile.1

The segments from offset 202 MB till offset 408 MB i.e. 104 chunks are not present in the cache anymore and have been evicted. And at the end of the trace we can see that new segments have come in i.e. from 782 MB onwards till 950MB. So, this demonstrates that the PerfAccel eviction follows a policy to evict the LRU or Least Recently Used data from the cache side. And it does this without disturbing the Application I/O in any manner as well as without the need for any downtime of the Source.

But, the question arises as to whether we should ignore such Evictions completely or take them seriously everytime they are triggered. Because, in I/O Analytics, a Eviction trigger indicates that the Cache Size is less than the working set size. For an eviction run, some amount of CPU cycles as well as Disk bandwidth of the Cache will be utilized.  Hence, we can ignore the Eviction if it seldom happens, like say once in a month or so.

But, if Evictions are triggered frequently then we need to resize our cache and add more cache space to the source, because it clearly indicates that the Application working set size needs more cache space. Then only the Application will perform better, or else it would be just bringing in cached data and the Eviction throwing them to make room for new data. This cycle would continue as long as the cache size is less.

Configuring the Limits :

PerfAccel allows us to configure and tweak all the 3 eviction limits. It provides us with multiple ways to do so

1. Using cache -config

# dgpctl -cache config app_data feasible=95P,critical=90P,cleanup=65P

This command sets the 3 limts as :

  1. Feasibility limit : 95% of 1GB = 972 MB
  2. Critical limit : 90% of 1GB = 921 MB
  3. Cleanup limit : 65% of 1GB = 665 MB

Let’s verify that new limits have been updated properly for the source

# dgpctl -source status app_data

Source Name            : app_data
Used Size              : 291.3 GB
Free Size              : 56.9 GB
Total Size             : 348.2 GB
Status                 : ONLINE
Source Type            : nfs
Source Mount Directory : /app_data
Data Group             : /ALL/DRF/TEST_NODES
Source Option          : cto, flock
Cache Info:
Cache State            : Enabled
Monitor Status         : Stopped
Cache Type             : writethrough
Feasibility limit      : 972.8 MB
Critical limit         : 921.6 MB
Cleanup limit          : 665.6 MB
File Cache Limit       : 100%

Cache Name                    Used Size           Free size         Total Size
-------------------------------------------------------------------------------
sata_cache                    744.0 MB          280.0 MB            1.0 GB


The above command can also be used with absolute values rather than the percentages
Ex:    dgpctl -cache config app_data feasible=950M,critical=850M,cleanup=650M

2. Using the Policy management interface for the source

# dgpctl -policy show app_data

Source Name               : app_data
Source Mount Directory    : /app_data
Revalidate timeout        : 0 (sec)
Time to live                        : 0 (sec)
Attribute timeout                   : 0 (sec)
Directory Attribute timeout         : 0 (sec)
Directory validate open timeout     : 0 (sec)
Negative Lookup timeout   : 0 (sec)
Working Set Caching       : Disabled
Symlink Caching           : Enabled
Attribute Caching         : Enabled
Caching Filter            : Cache All Files
Readdir Caching           : .


Cache Info:
Feasible limit            : 972.8 MB
Critical limit            : 921.6 MB
Cleanup limit             : 665.6 MB

File Cache Limit          : 100%

Now, change the limits. This command only accepts the values as Percentage of cache allocated

# dgpctl -policy update app_data feasible=98,critical=90,cleanup=82
Source policy updated successfully.

# dgpctl -source status app_data
Source Name            : app_data
Used Size              : 291.3 GB
Free Size              : 56.9 GB


Status                 : ONLINE
Source Type            : nfs
Source Mount Directory : /app_data
Data Group             : /ALL/DRF/TEST_NODES
Source Option          : cto, flock

Cache Info:
Cache State            : Enabled
Monitor Status         : Stopped
Cache Type             : writethrough
Feasibility limit      : 1003.5 MB
Critical limit         : 921.6 MB
Cleanup limit          : 839.7 MB
File Cache Limit       : 100%

Cache Name                    Used Size           Free size         Total Size
-------------------------------------------------------------------------------
sata_cache                    744.0 MB          280.0 MB            1.0 GB

So, this is how PerfAccel does dynamic data management on the SSD even when the cache sizing is on the lower side of the working set size of Application. LRU based evictions make sure that New data always comes into the cache thereby preventing any opportunity loss for future Read hits. And all this happens without any manual intervention and without affecting the Application in any way.

In the meantime, you can go ahead and Download PerfAccel right now and get hands on with the exciting features.

Share it
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
COMMENTS SECTION

Leave a Reply

Be the First to Comment!

Leave a Reply

wpDiscuz