Earth-System Data Middleware
The middleware for earth system data is a prototype to improve I/O performance for earth system simulation as used in climate and weather applications. ESDM exploits structural information exposed by workflows, applications as well as data description formats such as HDF5 and NetCDF to more efficiently organize metadata and data across a variety of storage backends.
ESDM builds upon a data model similar to NetCDF and utilizes a self-describing on-disk data format for storing structured data. ESDM can be used as a drop-in replacement for typical use-cases without changing anything from the application perspective. While our current version utilises the manual configuration by data-center experts, the ultimate long-term goal is to employ machine learning to automatise the decision making and reduce the burden for users and experts.
Key Information
| Contact | Dr. Julian Kunkel | ||
| Repository | Public on GitHub | ||
| Project | ESiWACE |
Talks
- Data-Centric IO: Potential for Climate/Weather (Dr. Julian Kunkel), 6th ENES HPC Workshop, Virtual/Hamburg, Germany, 2020-05-29 Presentation
- Progress of WP4: Data at Scale (Dr. Julian Kunkel), ESiWACE General Assembly, Virtual/Hamburg, Germany, 2020-05-27 Presentation
- Toward Next Generation Interfaces for Exploiting Workflows (Dr. Julian Kunkel), SIG IO UK, University of Reading, Reading, UK, 2020-04-23 Presentation Video
- Potential of I/O-Aware Workflows in Climate and Weather (Dr. Julian Kunkel), Supercomputing Frontiers Europe, Virtual/Warshaw Poland, 2020-03-25 Presentation
- Challenges and Approaches for Extreme Data Processing (Dr. Julian Kunkel), EPSRC Centre for Doctoral Training Mathematics of Planet Earth, University of Reading, Reading, UK, 2020-03-11 Presentation
- Smarter Management using Metadata and Workflow Expertise (Dr. Julian Kunkel), BoF: Knowledge Is Power: Unleashing the Potential of Your Archives Through Metadata, Supercomputing, Denver, USA, 2019-11-21 Presentation
- IO500 (George Markomanolis, John Bent, Dr. Julian Kunkel, Jay Lofstead), BoF: The IO500 and The Virtual Institute of I/O, Supercomputing, Denver, USA, 2019-11-19 Presentation
- Exploiting Different Storage Types with the Earth-System Data Middleware (Dr. Julian Kunkel), Parallel Data Systems Workshop, Supercomputing, Denver, USA, 2019-11-18 Presentation
- OpenSource Software (Dr. Julian Kunkel), Hacktoberfest, University of Reading, Reading, UK, 2019-10-27 Presentation
- The Earth-System Data Middleware: An Approach for Heterogeneous Storage Infrastructure (Dr. Julian Kunkel), SPPEXA Final Symposium, Dresden, Germany, 2019-10-23 Presentation
- Utilizing Heterogeneous Storage Infrastructures via the Earth-System Data Middleware (Dr. Julian Kunkel), NEXTGenIO Workshop on applications of NVRAM storage to exascale I/O, ECMWF, Reading, Germany, 2019-09-27 Presentation
- The Need for Next Generation Semantic Interfaces to Process Climate/Weather Workflows (Dr. Julian Kunkel), SIG IO UK, University of Reading, UK, 2018-06-06 Presentation
- Towards Intelligent Storage Systems (Dr. Julian Kunkel), Computer Science Workshop, University of Reading, UK, 2018-05-08 Presentation
- Exploiting the Heterogeneous Storage Landscape in a Data Center (Dr. Julian Kunkel), Per3s: Performance and Scalability of Storage Systems Workshop, Rennes, France, 2018-01-12 Presentation
- Exploiting Weather and Climate Data at Scale (WP4) (Dr. Julian Kunkel), ESiWACE General Assembly, Berlin, Germany, 2017-12-12 Presentation