Zarr
Jul 10,2026

GISBox is a one-stop 3D GIS data editing, conversion and publishing platform that supports editing in multiple GIS formats such as OSGB/GEOTIFF/RVT, converting to 3DTiles/Terrain and publishing.

Introduction

Zarr is a chunked, compressed N-dimensional array storage format designed for cloud-native workflows and high-performance computing. It splits large multidimensional scientific datasets, such as meteorological, remote sensing, and genomic data, into small independent chunks that are stored separately. Zarr supports multiple compression algorithms, including zstd and Blosc, and can be stored directly on local file systems, in ZIP archives, or in cloud object storage such as AWS S3. It natively supports parallel read/write access and lazy loading. Compared with NetCDF and HDF5, Zarr is better suited to large-scale data collaboration and distributed processing in cloud environments, and it has become one of the main backend formats for scientific computing tools such as xarray and Dask.

File Structure

The file structure of Zarr is essentially a directory, or an object-storage prefix, whose internal files can be divided into the following categories:

.zarray (array metadata file): A JSON file that acts as the core specification for an array. It records the Zarr version, overall array shape, chunk size, data type (dtype), compressor, fill value, memory layout (C/F order), and other settings. Every array or dataset has its own .zarray.
.zattrs (array attributes file): A JSON file that stores user-defined key-value attributes, such as units, description, and coordinate reference information. It is similar to global or variable attributes in NetCDF.
.zgroup (group metadata file): A JSON file located in the root directory of a Zarr Group. It indicates that the directory is a group rather than an array and contains group-level metadata such as the Zarr version. When a Zarr store contains multiple variables, a Group is typically used to manage them, with each variable stored as a child array.
Chunk data files: These are the core carriers of the actual data. A large array is divided into many small chunks according to the chunking scheme, and each chunk is independently compressed and stored as a binary file.
Subgroup / subarray directories: Zarr supports hierarchical nested structures, where Groups can contain other Groups. Each subgroup can have its own .zgroup, .zarray, .zattrs, and chunk directories, making it possible to organize multiple variables in a structured way.

Pros

Cloud-native friendly: Each chunk is an independent object, which naturally fits object storage systems such as AWS S3, GCS, and Azure Blob. It does not require a POSIX file system, and multiple users can read and write concurrently without locking the whole dataset.
Lazy loading: Only the required chunks are read, rather than opening the entire file as in NetCDF or HDF5. This makes fast slicing and access practical even for terabyte-scale datasets.
Flexible and efficient compression: Each chunk can use compression algorithms such as zstd, Blosc, or gzip independently, offering strong compression while still allowing random chunk-level access without decompressing the whole dataset.
Parallel chunk-based I/O: Different chunks can be read or written simultaneously by different threads or processes, making Zarr a natural fit for distributed computing frameworks such as Dask and Spark.
Hierarchical organization with Groups: A single Zarr store can contain multiple arrays, coordinates, and attribute sets in a lightweight structure similar to NetCDF-4 or HDF5.
No single point of failure: Because data is distributed across many small files or objects, corruption of one chunk affects only a local part of the dataset, unlike HDF5 where metadata damage can make the whole file unusable.
Compatible with write-once-read-many workflows: Once written, chunks are naturally suited to immutable and versioned dataset management, such as with DVC or Kerchunk.
Mature cross-language ecosystem: Zarr has stable implementations in Python, Julia, R, Java, Go, and other languages, with especially strong integration into xarray and Dask.

Cons

Small-file problem: On local file systems, each chunk is stored as a separate file. Millions of chunks can exhaust inodes and make directory scanning extremely slow. For this reason, Zarr is generally better stored in object storage or packaged formats rather than directly on a local file system.
Metadata performance bottleneck: Files such as .zarray and .zattrs are JSON text files, so metadata operations require reading many small files. With very large chunked datasets, metadata itself can become substantial.
Limited native append support: Existing chunks can be modified, but appending new data along a dimension such as time often requires careful chunk-layout planning and is less convenient than NetCDF-4 unlimited dimensions. Zarr v3 is improving this.
No built-in indexing or query engine: Zarr is a storage format only. Unlike Parquet, it does not provide built-in statistics or indexing for query acceleration, so spatial or temporal subsetting relies on higher-level tools such as Kerchunk or TileDB.
Chunking strategy must be planned in advance: Once chunk sizes are set, they are difficult to change efficiently. Small chunks increase metadata overhead, while large chunks reduce parallelism and weaken lazy-loading granularity.
Tooling is still evolving: Compared with NetCDF and HDF5, Zarr is a younger ecosystem. Support in some GIS software, such as QGIS and GDAL, is still catching up, although GDAL 3.x and later already provide support.
Weaker consistency model: Concurrent writes on object storage may require extra mechanisms, such as conditional writes or lock tables, to keep metadata and chunk data consistent.

Application Scenario

The most important application area for Zarr is meteorology and climate science. For example, many petabyte-scale multidimensional datasets produced by global climate models, such as CMIP6, are now distributed in Zarr format on public cloud platforms so that researchers worldwide can access them in parallel. In remote sensing and geospatial analysis, large time series of satellite imagery from missions such as Sentinel and Landsat are increasingly being published as Zarr datasets, often combined with STAC catalogs for efficient access by time and space. In bioinformatics, large genomic and single-cell sequencing matrices also benefit from Zarr's chunked compression and lazy-loading model. In machine learning, training datasets such as images, videos, and medical imagery can be stored in Zarr for efficient distributed data pipelines with tools like Dask and PyTorch DataLoader. Large simulation outputs in ocean science, hydrology, and geophysical modeling are also rapidly moving into the Zarr ecosystem.