Previous: Advanced Usage
Up: Usage Guidelines and Suggestions
Next: Local FITS Conventions supported by FITSIO
Previous Page: Advanced Usage
Next Page: Local FITS Conventions supported by FITSIO

Optimizing Code for Maximum I/O Processing Speed

The processing speed of programs using the FITSIO library is often limited by the time required to read and write the data in the FITS file from or to disk. In order to increase the processing speed of a FITSIO program it is important to minimize the actual amount of disk I/O by trying to process the data in one sequential pass through the file without having to go back to reread or rewrite previous blocks in the file.

The key to writing efficient code is in understanding how FITSIO buffers the disk I/O. There is an internal buffer in FITSIO that stores one block of data from each opened FITS file that is currently being read or written. There is a separate buffer for each opened FITS file, so when reading from one FITS file and writing to another then 2 buffers will be used. The size of the buffer is system dependent, but in most cases (except in the VAX, AIX and SPP versions of FITSIO) it is 2880-bytes long (the same length as a FITS logical block). For maximum efficiency one should try to completely process all the data in the current block before moving on to read or write the next block of the FITS file. FITSIO can only operate on the data contained in this buffer so if a program attempts to read or write a section of the FITS file not in this buffer then FITSIO must first flush the current contents of the buffer back to disk (if it has been modified) and then move to and read in the new block of the FITS file containing the particular bytes that are needed.

In a few cases the internal buffer size differs from the standard 2880 bytes:

VAX: On VAX/VMS systems disk files have an intrinsic record length so the buffer size is chosen to be equal to the record length when reading an existing FITS file; one can determine this size from the returned value of the 'blocksize' parameter in the FTOPEN subroutine. When creating a new FITS file on a VAX, one can specify the block size (and hence the buffer size) with the same 'blocksize' parameter in the FTINIT subroutine. Any value less than or equal to 2880 may be chosen, but it is recommended that a value of 2880 bytes be used.

AIX: On the IBM AIX operating system the Fortran compiler has an upper limit of 500 characters for any character string, therefore the internal buffer size is defined to be 480 bytes (= 1/6 of a FITS logical block).

IRAF: In the IRAF/SPP version, the buffer size is determined by system calls external to the FITSIO routines and is approximately 128K bytes long. Currently there is no easy way to determine the exact buffer size, but this may change in future releases of FITSIO.

Once one knowns the internal buffer size that is being used, one can then determine the optimum strategy for processing the FITS file. In general, regardless of the buffer size, one should attempt to read the FITS header keywords in the same order as they appear in the file, and one should ideally read or write any and all header keywords before reading the data. Also, when processing a FITS image (a FITS Primary Array or an IMAGE extension) the most efficient procedure is simply to read or write the data sequentially from the first to last pixel in the image. In this case the internal buffer size is irrelevant to first order.

When processing a FITS TABLE or BINTABLE extension the optimum processing strategy is more complicated because the table data are usually ordered on disk in row by row order whereas most of the FITSIO table-access subroutines only access a single column of data at a time. If, for example, one reads a FITS table by first reading all the data values from the first column, followed by all the values from the second column, and so on, then one would need to physically read through the entire FITS extension one time for each column in the table. For tables with many columns this can lead to a large amount of extra disk I/O. A better strategy is to read or write all the columns of the table for only those rows that are currently contained in the internal FITS block. After all the columns have been processed, then move on to the next block of the FITS file and process the next set of rows. In the ideal case it should be possible to process the entire FITS table with only a single pass through the data. This strategy only works well though if the table row length is evenly divisible into the internal block size. If this is not the case, then one may be able to get significant performance improvements by redesigning the FITS table by changing the width of one of the columns or by adding a dummy column to pad out the width so that it is evenly divisible into the block size. One must also keep in mind that the FITS data always start on a FITS 2880-byte logical block boundary. If the FITSIO internal buffer size is not equal to 2880, then one must be careful to correctly compute the starting point of the rows that are contained in the buffer.

As a final optimizing strategy, there are a few low-level subroutines available in the FITSIO interface which will read or write a contiguous range of bytes from a FITS TABLE or BINTABLE extension: FTGTBS, FTPTBS, FTGTBB, and FTPTBB. In principle these routines can be used to read a whole row, or multiple rows, of a table with a single subroutine call. These routines are fast because they bypass all the usual data scaling, error checking and machine dependent data conversion that is normally done by FITSIO, and they allow the program to read or write the data in exactly the same order as the bytes are stored in the file. But for these same reasons, using these routines can be risky because no validation or machine dependent conversion is performed by these routines. In general these routines are only recommended for optimizing critical pieces of code and should only be used by programmers who thoroughly understand the internal byte structure of the FITS tables they are reading or writing.

As a final caveat, programmers should be warned that these optimization hints are necessarily dependent on the particular implementation of the FITSIO subroutines. It is conceivable that future versions of FITSIO may have quite different internal architectures, so there is no guarantee that programs that are carefully optimized for this version of FITSIO will still be optimized under future versions.