[ HDF5 Tutorial Top ]

# Chunking and Extendible Datasets

## Contents:

- Creating an Extendible Dataset
- Programming Example
  - Description
  - Remarks

## Creating an Extendible Dataset

An extendible dataset is one whose dimensions can grow. HDF5 allows you to define a dataset to have certain initial dimensions, then to later increase the size of any of the initial dimensions.

HDF5 requires you to use chunking to define extendible datasets. This makes it possible to extend datasets efficiently without having to excessively reorganize storage.

The following operations are required in order to write an extendible dataset:

1. Declare the dataspace of the dataset to have unlimited dimensions for all dimensions that might eventually be extended.
2. Set dataset creation properties to enable chunking.
3. Create the dataset.
4. Extend the size of the dataset.

## Programming Example

### Description

This example shows how to create a 3 x 3 extendible dataset, write to that dataset, extend the dataset to 10x3, and write to the dataset again.

[C example ] - `h5_extend.c`
[FORTRAN example ] - `chunk.f90`

NOTE: To download a tar file of the examples, including a Makefile, please go to the References page.

### Remarks

- The routine `H5Pcreate` / `h5pcreate_f` creates a new property as an instance of a property list. The signature is as follows:

  C:

          hid_t H5Pcreate (H5P_class_t classtype)

  FORTRAN:

          h5pcreate_f (classtype, prp_id, hdferr)

                  classtype  IN: INTEGER
                  prp_id    OUT: INTEGER(HID_T)
                  hdferr    OUT: INTEGER

  - The parameter classtype is the type of property list to create. Valid class types are as follows:

    C | FORTRAN
    `H5P_FILE_CREATE H5P_FILE_ACCESS H5P_DATASET_CREATE H5P_DATASET_XFER H5P_MOUNT` | `H5P_FILE_CREATE_F H5P_FILE_ACCESS_F H5P_DATASET_CREATE_F H5P_DATASET_XFER_F H5P_MOUNT_F`

  - In C, the property list identifier is returned if successful; otherwise a negative value is returned, if not. In FORTRAN, the property list identifier is returned in prp_id and the return value for the call is returned in hdferr.

- The routine `H5Pset_chunk` / `h5pset_chunk_f` sets the size of the chunks used to store a chunked layout dataset. The signature of this routine is as follows:

  C:

          herr_t H5Pset_chunk (hid_t prp_id, int ndims,
                               const hsize_t * dims)

  FORTRAN:

          h5pset_chunk_f (prp_id, ndims, dims, hdferr)

                  prp_id    IN: INTEGER(HID_T)
                  ndims     IN: INTEGER
                  dims      IN: INTEGER(HSIZE_T), DIMENSION(ndims)
                  hdferr   OUT: INTEGER

  - The prp_id parameter is the identifier for the property list to query.
  - The ndims parameter is the number of dimensions of each chunk.
  - The dims parameter is an array containing the size of each chunk.
  - In C, a non-negative value is returned if successful; otherwise a negative value is returned. In FORTRAN, the return value is returned in hdferr: 0 if successful and -1 otherwise.

- The `H5Dextend` / `h5dextend_f` routine extends a dataset that has an unlimited dimension. The signature is as follows:

  C:

          herr_t H5Dextend (hid_t dset_id, const hsize_t * size)

  FORTRAN:

          h5dextend_f (dset_id, size, hdferr)

                  dset_id   IN: INTEGER(HID_T)
                  size         IN: INTEGER(HSIZE_T), DIMENSION(*)
                  hdferr      OUT: INTEGER

  - The dset_id parameter is the dataset identifier.
  - The size parameter, is an array containing the new magnitude of each dimension.
  - In C, this function returns a non-negative value if successful and a negative value otherwise. In FORTRAN, the return value is returned in hdferr: 0 if successful and -1 otherwise.

- The `H5Dget_create_plist` / `h5dget_create_plist_f` routine returns an identifier for a copy of the dataset creation property list for a dataset.

- The C function, `H5Pget_layout`, returns the layout of the raw data for a dataset. Valid types are `H5D_CONTIGUOUS` and `H5D_CHUNKED`. A FORTRAN routine for `H5Pget_layout` does not yet exist.

- The `H5Pget_chunk` / `h5pget_chunk_f` routine retrieves the size of chunks for the raw data of a chunked layout dataset. The signature is as follows:

  C:

          int H5Pget_chunk (hid_t prp_id, int ndims, hsize_t * dims)

  FORTRAN:

          h5pget_chunk_f (prp_id, ndims, dims, hdferr)

                  prp_id    IN: INTEGER(HID_T)
                  ndims     IN: INTEGER
                  dims     OUT: INTEGER(HSIZE_T), DIMENSION(ndims)
                  hdferr   OUT: INTEGER

  - The prp_id parameter is the identifier of the property list to query.
  - The ndims parameter is the size of the dims array.
  - The dims parameter is the array in which to store the chunk dimensions.
  - In C, this function returns the chunk dimensionality if successful and a negative value otherwise. In FORTRAN, the return value is returned in hdferr: the chunked rank if successful and -1 otherwise.

- The `H5Pclose` / `h5pclose_f` routine terminates access to a property list. The signature is as follows:

  C:

          herr_t H5Pclose (hid_t prp_id)

  FORTRAN:

          h5pclose_f (prp_id, hdferr)

                  prp_id    IN: INTEGER(HID_T)
                  hdferr   OUT: INTEGER

  - The prp_id parameter is the identifier of the property list to terminate access to.

The National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
hdfhelp@ncsa.uiuc.edu

###### Last Modified: June 22, 2001