Disk I/O Tutorial
Part 1: The Basic Concepts of Disk I/O
Author: NetworkAdminKB.com
Created: 2006-04-14
Modified: 2009-06-01
This tutorial seeks to explain the basic concepts of Input / Output (aka: IO or I/O) operations to disk located on a local disk or a Storage Area Network. After we explain what makes up the basic I/O we look at how to determine an application’s standard IO size and then how to determine the IOPS (IOs per second) of an application or server.
So what are the key components of an IO?
The following are the fundamental principals that determine the number of I/Os that are generated by a particular application.
1) A single I/O will consist of one read or write. The size of this read or write is dictated by the application and not the computer hardware or the disk drive type or format.
2) The number of I/O’s created is a function of the application performing the I/O (read or writes to disk) and not the hardware or the format of the disk. Thus, if the application writes in 8k data blocks then one I/O is performed for every 8k of data, regardless of the block size format of the disk. It is also possible for the application to write a partial data block by flushing a partially filled data block to disk, or to write more than their standard data block size.
3) The disk block size (aka: sector size) format of the disk will not increase or decrease the number I/O’s the application performs. However, matching the application data block size and the disk block size provides the optimal configuration on servers dedicated to a specific application like a database.
What is a data block?
A data block is a generic term used to describe the amount of data a particular application typically reads or writes to disk. Different applications have different standard sizes and names for this concept, but to simplify this tutorial the phrase data block is used to express this concept. See the Common Application IO Sizes table at the end of this tutorial for examples of common data block sizes.
How can I determine what size data blocks my application reads and writes?
First, check the application documentation. Electronic versions of the documentation work well because you can search for keywords like “data size”, “IO size”, etc. Secondly, you can use the internet to search for information regarding your application’s standard IO…maybe someone else has already figured it out. A few common examples are provided in the Common Application IO sizes table shown below.
Finally, you could set up a simple test. On Windows 2000 or higher the test setup is done as follows:
1) Install the application and all its data files to a separate partition.
a. Only a separate partition is needed, it can still be on the same physical drive as other partitions.
b. Only this application and its data files should be on this partition.
2) Load performance monitor and add the following counters
a. Logical Disk: Avg. Disk Bytes/Read Instance: select drive letter
b. Logical Disk: Avg. Disk Bytes/Write Instance: select drive letter
3) Start the application and perform a high volume of transactions. The higher the number of transaction the easier it may be to determine standard size.
Generally, if the application has as standard data block size it should show up in the Last display field consistently. However, many factors can contribute to the number not matching up with a known size. A major contributor will be lack of file fragmentation on the disk. In this test case fragmentation would be a good thing when performing this test. If the disk is heavily fragmented then the application will be forced to perform more random IO on the disk and therefore will perform more standard data block transfers. If the disk is not very fragmented then the application could perform more sequential reads or writes which would result in a higher than normal data block size.
Here is a sample screen shot for an application that has a 512 byte standard IO size during read operations.
So what is IOPS?
IOs per second (IOPS) is a measure of how many Input or Output operations are done per second by a particular device. This measure is widely used as the standard when comparing any type of disk (SCSI, IDE, SATA, etc) or disk subsystem used in a Storage Area Network (e.g. IBM DS8100, Hitachi Thunder 9570V, etc). However, this measure as demonstrated above is application dependent as the size of the data block used in the I/O can vary from application to application.
How do I determine the current IOPS for my application/server?
Determining your current IOPS is fairly easy when you have the right tools. Several methods will be discussed.
Method 1
This is probably the best method because depending on the requirements you should be able to look at the individual application as well as the server as a whole. This method uses the Windows Performance Monitor. So load up performance monitor and add the following counters.
a) Logical Disk: Avg. Disk Bytes/Read Instance: select drive letter
b) Logical Disk: Avg. Disk Bytes/Transfer Instance: select drive letter
c) Logical Disk: Avg. Disk Bytes/Write Instance: select drive letter
d) Logical Disk: Disk Read Bytes/sec Instance: select drive letter
e) Logical Disk: Disk Write Bytes/sec Instance: select drive letter
f) Logical Disk: Disk Transfers/sec Instance: select drive letter
Take a screen shot by pressing the print screen button then paste into MS Paint, or log to a file for long term monitoring and graph or report from the log.
Here is an example screen shot. Notice that all writes are 8K in size.
To determine the number IOPS from the performance counters consult the following table:
|
Microsoft Windows Performance Counter |
Industry Standard IOPS Name |
|
Disk Transfers/sec |
Total IOPS |
|
Disk Reads/sec |
Read IOPS |
|
Disk Writes/sec |
Write IOPS |
To verify that the numbers are accurate or to prove the concept simply divide the Disk Read (or Write) Bytes by the stated Avg. Disk Bytes / Read (or Write) (aka: data block size in this case 8K).
For Example: Write Application IOPS: 130,410,370 / 8192 = 15,919 IOPS
*It should be noted for the best results you should log to a file first then Graph and Report on these numbers for the desired time frame. When graphing from a previous log you can obtain the appropriate measure for the time frame logged: Average, Minimum or Maximum
**While all statistics are taken from Logical Disk the Physical Disk could be used when appropriate.
Method 2
If you are using a Storage Area Network (SAN) you can obtain the IOPS from the HBA. This method only allows you to size the entire servers because the HBA is normally shared among all application on the servers. Some applications like the Qlogic SanSurfer allows you log to a file as well.
Here is an example from a Qlogic 2342 HBA.
To find the IOPS for the entire server simply sum IOPS values from all the HBA ports.
Method 3
If you have a SAN use the built in performance information from the disk subsystem. It’s best if you get the IOPS of the individual LUN* associated with your application / server.
*In this case the LUN is the logical drive as seen by the host operating system.
Here is a sample excerpt from a Hitachi 9570V performance output file.
Items in bold are of interest.
2006/04/13 10:09:31 - 2006/04/13 10:10:34
---- Port Information ----
CTL Port IO Rate(IOPS) Read Rate(IOPS) Write Rate(IOPS) TR. Rate(MB/S) Read TR. Rate(MB/S) Write TR. Rate(MB/S)
0 A 262 175 86 9 9 0
0 B 478 319 159 14 12 2
1 A 51 6 45 0 0 0
1 B 95 8 87 1 0 0
---- LU Information ----
CTL LU IO Rate(IOPS) Read Rate(IOPS) Write Rate(IOPS) TR. Rate(MB/S) Read TR. Rate(MB/S) Write TR. Rate(MB/S)
0 50 267 267 0 16 16 0
1 50 0 0 0 0 0 0
LUN 50 (aka: LU 50) is the dedicated LUN for this particular application. We see that LUN 50 had an average of 267 IOPS (Total and Read) for a given time frame (2006/04/13 10:09:31 - 2006/04/13 10:10:34). It should also be noted that while all the IOs for LUN 50 were on controller 0 (CTL), it is possible for IOs to use controller 1 as well.
Another verification of your application data block size can be done here. Here’s the math.
(Avg. Read Transfer Rate (MB/S) / IOPS ) * 1024 = average application data block size in KB
or
(Avg. Write Transfer Rate (MB/S) / IOPS ) * 1024 = average application data block size in KB
(16 / 267 ) * 1024 = 61KB average. This application is known to have a 64K standard data block size, suggesting that other application data blocks are also being passed through the HBA.
This method works best for sizing an entire server over a length of time.
Putting it all together
The following is a real life example of how to determine various IOPS (i.e. Total, Reads or Writes) for an entire file server and an individual drive. This file server has multiple LUNs assigned (1.7 TB of disk space) for use as a file server.
Here are the general steps
1) Load up performance monitor and log to a file for at least 5 minutes. Longer periods of time will be more accurate for getting averages, minimums and maximums.
2) Once the log is complete use performance monitor and add in the counters listed in Method 1 of the How do I determine the current IOPS for my application/server section above.
3) Review the Graph and Report numbers selecting the appropriate numbers.
Below is an example screen shot showing an individual drive and totals for all drives (some not shown).
*When viewing reports from a log you should know that these numbers are the “average” for the entire time frame and not just the last value.
Example screenshot showing how to get the Minimum and Maximum values to account for spikes in performance.
The following table contains some of the IOPS numbers (rounded) that may be of importance. Most of the numbers are taken from the screenshots above, if no screenshot was provided (ns) is put by the number. Given previous explanations you should be able to figure out where the numbers came from.
|
|
All Drives |
I: Drive |
|
Average Total IOPS |
590 |
42 |
|
Average Read IOPS |
528 |
26 |
|
Average Write IOPS |
62 |
17 |
|
Max Total IOPS |
955 |
88 (ns) |
|
Min Total IOPS |
323 |
16 (ns) |
File servers by definition are going to have various sizes of data blocks written to disk because various applications will be writing via one or more network shares. These applications can consist of everything from Windows Explorer, Microsoft Word or Excel, to custom applications written in house at the company. This makes planning for a standard data block size problematic. However, in most cases simply finding the daily average data block size should be sufficient.
Summary
Using this tutorial as a guide you should now be able to determine a particular application’s standard data block size and retrieve various IOPS numbers for the application or server. A table of common application IO sizes is provided for convenience but performance monitoring to a log provides the best information.
Common Application IO sizes
|
Application |
Size |
Name |
Documentation / Location |
|
Microsoft SQL Server 6.5 |
2KB |
Page |
Books Online / Glossary |
|
Microsoft SQL Server 7.0 |
8KB |
Page |
Books Online / Glossary |
|
Microsoft SQL Server 2000 |
8KB |
Page |
Books Online / Glossary |
|
Microsoft SQL Server 2005 |
8KB |
Page |
Books Online / Glossary |
|
Veritas NetBackup v5.0 - 6.0 |
64KB (default) |
Data Buffer |
System Administrator's Guide |
|
|
|
|
|