Earth-System Data Middleware

The middleware for earth system data is a prototype to improve I/O performance for earth system simulation as used in climate and weather applications. ESDM exploits structural information exposed by workflows, applications as well as data description formats such as HDF5 and NetCDF to more efficiently organize metadata and data across a variety of storage backends.

ESDM builds upon a data model similar to NetCDF and utilizes a self-describing on-disk data format for storing structured data. ESDM can be used as a drop-in replacement for typical use-cases without changing anything from the application perspective. While our current version utilises the manual configuration by data-center experts, the ultimate long-term goal is to employ machine learning to automatise the decision making and reduce the burden for users and experts.

Key Information

Contact Dr. Julian Kunkel
Repository Public on GitHub
Project ESiWACE

Talks

  • \myPub{2020}{Data-Centric IO: Potential for Climate/Weather}{6th ENES HPC Workshop}{Virtual/Hamburg, Germany}
  • \myPub{}{Progress of WP4: Data at Scale}{ESiWACE General Assembly}{Virtual/Hamburg, Germany}
  • \myPub{}{Toward Next Generation Interfaces for Exploiting Workflows}{SIG IO UK}{University of Reading, Reading, UK}
  • \myPub{}{Potential of I/O-Aware Workflows in Climate and Weather}{Supercomputing Frontiers Europe}{Virtual/Warshaw Poland}
  • \myPub{}{Challenges and Approaches for Extreme Data Processing}{EPSRC Centre for Doctoral Training Mathematics of Planet Earth}{University of Reading, Reading, UK}
  • \myPub{2019}{Smarter Management using Metadata and Workflow Expertise}{BoF: Knowledge Is Power: Unleashing the Potential of Your Archives Through Metadata}{Supercomputing, Denver, USA}
  • \myPub{}{Exploiting Different Storage Types with the Earth-System Data Middleware}{Parallel Data Systems Workshop}{Supercomputing, Denver, USA}
  • \myPub{}{OpenSource Software}{Hacktoberfest}{University of Reading, Reading, UK}
  • \myPub{}{The Earth-System Data Middleware: An Approach for Heterogeneous Storage Infrastructure}{SPPEXA Final Symposium}{Dresden, Germany}
  • \myPub{}{Utilizing Heterogeneous Storage Infrastructures via the Earth-System Data Middleware}{NEXTGenIO Workshop on applications of NVRAM storage to exascale I/O}{ECMWF, Reading, Germany}
  • \myPub{2018}{The Need for Next Generation Semantic Interfaces to Process Climate/Weather Workflows}{SIG IO UK}{University of Reading, UK}
  • \myPub{}{Towards Intelligent Storage Systems}{Computer Science Workshop}{University of Reading, UK}
  • \myPub{}{Exploiting the Heterogeneous Storage Landscape in a Data Center}{Per3s: Performance and Scalability of Storage Systems Workshop}{Rennes, France}
  • \myPub{2017}{Exploiting Weather and Climate Data at Scale (WP4)}{ESiWACE General Assembly}{Berlin, Germany}