# The Raw Data I/O Pipeline

The HDF5 raw data pipeline is a complicated beast that handles all aspects of raw data storage and transfer of that data between the file and the application. Data can be stored contiguously (internal or external), in variable size external segments, or regularly chunked; it can be sparse, extendible, and/or compressible. Data transfers must be able to convert from one data space to another, convert from one number type to another, and perform partial I/O operations. Furthermore, applications will expect their common usage of the pipeline to perform well.

To accomplish these goals, the pipeline has been designed in a modular way so no single subroutine is overly complicated and so functionality can be inserted easily at the appropriate locations in the pipeline. A general pipeline was developed and then certain paths through the pipeline were optimized for performance.

We describe only the file-to-memory side of the pipeline since the memory-to-file side is a mirror image. We also assume that a proper hyperslab of a simple data space is being read from the file into a proper hyperslab of a simple data space in memory, and that the data type is a compound type which may require various number conversions on its members.

The diagrams should be read from the top down. The Line A in the figure above shows that `H5Dread()` copies data from a hyperslab of a file dataset to a hyperslab of an application buffer by calling `H5D_read()`. And `H5D_read()` calls, in a loop, `H5S_simp_fgath()`, `H5T_conv_struct()`, and `H5S_simp_mscat()`. A temporary buffer, TCONV, is loaded with data points from the file, then data type conversion is performed on the temporary buffer, and finally data points are scattered out to application memory. Thus, data type conversion is an in-place operation and data space conversion consists of two steps. An additional temporary buffer, BKG, is large enough to hold N instances of the destination data type where N is the same number of data points that can be held by the TCONV buffer (which is large enough to hold either source or destination data points).

The application sets an upper limit for the size of the TCONV buffer and optionally supplies a buffer. If no buffer is supplied then one will be created by calling `malloc()` when the pipeline is executed (when necessary) and freed when the pipeline exits. The size of the BKG buffer depends on the size of the TCONV buffer and if the application supplies a BKG buffer it should be at least as large as the TCONV buffer. The default size for these buffers is one megabyte but the buffer might not be used to full capacity if the buffer size is not an integer multiple of the source or destination data point size (whichever is larger, but only destination for the BKG buffer).

Occassionally the destination data points will be partially initialized and the `H5Dread()` operation should not clobber those values. For instance, the destination type might be a struct with members `a` and `b` where `a` is already initialized and we're reading `b` from the file. An extra line, G, is added to the pipeline to provide the type conversion functions with the existing data.

It will most likely be quite common that no data type conversion is necessary. In such cases a temporary buffer for data type conversion is not needed and data space conversion can happen in a single step. In fact, when the source and destination data are both contiguous (they aren't in the picture) the loop degenerates to a single iteration.

So far we've looked only at internal contiguous storage, but by replacing Line B in Figures 1 and 2 and Line A in Figure 3 with Figure 4 the pipeline is able to handle regularly chunked objects. Line B of Figure 4 is executed once for each chunk which contains data to be read and the chunk address is found by looking at a multi-dimensional key in a chunk B-tree which has one entry per chunk.

If a single chunk is requested and the destination buffer is the same size/shape as the chunk, then the CHUNK buffer is bypassed and the destination buffer is used instead as shown in Figure 5.

Last modified: Wed Mar 18 10:38:30 EST 1998