LCI2009-Tutorial

2009
LCI
Conference
Tutorial 

HDF5
and
netCDF ‐4:

Two
Solutions
for
Data
Management
Problems
Based
on
One
File
Format

Elena
Pourmal,
Albert
Cheng,
Ruth
Aydt
(The
HDF
Group) 
Ed
Hartnett
(Unidata/UCAR)
 

Overview 
Over
the
past
20
years,
the
HDF
( www.hdfgroup.org )
and
netCDF
( www.unidata.ucar.edu )
file
formats
have
become
de
facto
standards
for
storing,
managing,
and
exchanging
data
in
science
and
engineering
communities.
P etabytes
of
data
have
been
written
in
both
formats
and
used
in
many
 endeavors,
 including
 climate
 change
 modeling,
 weather
 prediction,
 nuclear
 fusion
simulation,
non ‐destructive
material
testing,
bioinformatics,
and
high ‐resolution
imaging.
 

Advances
 in
 hi gh‐performance
 computing
 have
 made
 it
 possible
 to
 model
 and
 study
 very
complex
 phenomena
 in
 a
 wide
 range
 of
 scientific
 fields
 while
 producing,
 accessing,
 and
analyzing
 gigabytes
 of
 complex
 and
 diverse
 data.
 Efficient
 data
 management,
 including
seamless
dat a
interoperability,
is
a
critical
part
of
the
scientific
discovery
process
that
presents
new
challenges
to
the
users
and
maintainers
of
scientific
data
formats.
 

This
tutorial
introduces
HDF5
and
netCDF ‐4,
a
new
version
of
netCDF
built
on
top
of
HDF5.

HDF5
and
netCDF ‐4
were
created
to
address
data
management
needs
in
today’s
heterogeneous
and
quickly
evolving
high ‐performance
computational
environments ...