Earth-System Data Middleware

The middleware for earth system data is a prototype to improve I/O performance for earth system simulation as used in climate and weather applications. ESDM exploits structural information exposed by workflows, applications as well as data description formats such as HDF5 and NetCDF to more efficiently organize metadata and data across a variety of storage backends.

ESDM builds upon a data model similar to NetCDF and utilizes a self-describing on-disk data format for storing structured data. We aim to deliver the NetCDF integrated version by the end of the ESiWACE1 project. This improvement can then be used as a drop-in replacement for typical use-cases without changing anything from the application perspective. While our current version utilises the manual configuration by data-center experts, the ultimate long-term goal is to employ machine learning to automatise the decision making and reduce the burden for users and experts.

Further information and performance results are provided as part of the ESiWACE projects.

Contact Dr. Julian Kunkel
Repository Public on GitHub
Project ESiWACE2

Talks

  • The Need for Next Generation Semantic Interfaces to Process Climate/Weather Workflows (Dr. Julian Kunkel), SIG IO UK, University of Reading, UK, 2018-06-06 Presentation
  • Towards Intelligent Storage Systems (Dr. Julian Kunkel), Computer Science Workshop, University of Reading, UK, 2018-05-08 Presentation
  • Exploiting the Heterogeneous Storage Landscape in a Data Center (Dr. Julian Kunkel), Per3s: Performance and Scalability of Storage Systems Workshop, Rennes, France, 2018-01-12 Presentation
  • Exploiting Weather and Climate Data at Scale (WP4) (Dr. Julian Kunkel), ESiWACE General Assembly, Berlin, Germany, 2017-12-12 Presentation