Table of Contents
ESiWACE
The Centre of Excellence in Simulation of Weather and Climate in Europe (ESiWACE) is a H2020 funded project.
Within the project, our research group is responsible for DKRZ's contribution to work package 4 (Exploitability).
Please see the official web page of ESiWACE for further information.
Contact (for WP4 contribution of DKRZ) Dr. Julian Kunkel
People
- Jakob Lüttgau
- Dr. Philipp Neumann
Goals for WP4
- Modelling costs for storage methods and understanding these
- Modelling tape archives and costs
- Flexible disk storage layouts for earth system data
Collaboration partners for WP4
- DKRZ
- STFC
- ECMWF
- CMCC
- Seagate
Flexible Storage Layout for Earth-System Data
The design deliverable architecture-d4.2.pdf
ESDM builds upon a data model similar to NetCDF and utilizes a self-describing on-disk data format for storing structured data. We aim to deliver the NetCDF integrated version by the end of the ESiWACE1 project. This improvement can then be used as a drop-in replacement for typical use-cases without changing anything from the application perspective. While our current version utilises the manual configuration by data-center experts, the ultimate long-term goal is to employ machine learning to automatise the decision making and reduce the burden for users and experts.
Publications
- Toward Understanding I/O Behavior in HPC Workflows (Jakob Lüttgau, Shane Snyder, Philip Carns, Justin M. Wozniak, Julian Kunkel, Thomas Ludwig), 2019-02-11 BibTeX DOI
- Beating data bottlenecks in weather and climate science (Bryan N. Lawrence, Julian Kunkel, Jonathan Churchill, Neil Massey, Philip Kershaw, Matt Pritchard), 2019-01-25 BibTeX URL PDF
- Cost and Performance Modeling for Earth System Data Management and Beyond (Jakob Lüttgau, Julian Kunkel), 2019-01-25 BibTeX DOI PDF
- A Survey of Storage Systems for High-Performance Computing (Jakob Lüttgau, Michael Kuhn, Kira Duwe, Yevhen Alforov, Eugen Betke, Julian Kunkel, Thomas Ludwig), 2018-04 BibTeX URL DOI PDF
- Poster: Adaptive Tier Selection for NetCDF and HDF5 (Jakob Lüttgau, Eugen Betke, Olga Perevalova, Julian Kunkel, Michael Kuhn), 2017-11-14 BibTeX PDF
- Simulation of Hierarchical Storage Systems for TCO and QoS (Jakob Lüttgau, Julian Kunkel), 2017 BibTeX DOI PDF
- Poster: Modeling and Simulation of Tape Libraries for Hierarchical Storage Systems (Jakob Lüttgau, Julian Kunkel), 2016-11-15 BibTeX URL PDF
Talks
- Data Systems at Scale in Climate and Weather: Activities in the ESiWACE Project (Dr. Julian Kunkel), HPC IODC Workshop, Virtual, 2020-06-25 Presentation Video
- Data-Centric IO: Potential for Climate/Weather (Dr. Julian Kunkel), 6th ENES HPC Workshop, Virtual/Hamburg, Germany, 2020-05-29 Presentation
- Progress of WP4: Data at Scale (Dr. Julian Kunkel), ESiWACE General Assembly, Virtual/Hamburg, Germany, 2020-05-27 Presentation
- Toward Next Generation Interfaces for Exploiting Workflows (Dr. Julian Kunkel), SIG IO UK, University of Reading, Reading, UK, 2020-04-23 Presentation Video
- Potential of I/O-Aware Workflows in Climate and Weather (Dr. Julian Kunkel), Supercomputing Frontiers Europe, Virtual/Warshaw Poland, 2020-03-25 Presentation
- Challenges and Approaches for Extreme Data Processing (Dr. Julian Kunkel), EPSRC Centre for Doctoral Training Mathematics of Planet Earth, University of Reading, Reading, UK, 2020-03-11 Presentation
- Smarter Management using Metadata and Workflow Expertise (Dr. Julian Kunkel), BoF: Knowledge Is Power: Unleashing the Potential of Your Archives Through Metadata, Supercomputing, Denver, USA, 2019-11-21 Presentation
- Exploiting Different Storage Types with the Earth-System Data Middleware (Dr. Julian Kunkel), Parallel Data Systems Workshop, Supercomputing, Denver, USA, 2019-11-18 Presentation
- OpenSource Software (Dr. Julian Kunkel), Hacktoberfest, University of Reading, Reading, UK, 2019-10-27 Presentation
- The Earth-System Data Middleware: An Approach for Heterogeneous Storage Infrastructure (Dr. Julian Kunkel), SPPEXA Final Symposium, Dresden, Germany, 2019-10-23 Presentation
- Utilizing Heterogeneous Storage Infrastructures via the Earth-System Data Middleware (Dr. Julian Kunkel), NEXTGenIO Workshop on applications of NVRAM storage to exascale I/O, ECMWF, Reading, Germany, 2019-09-27 Presentation
- Data-Centric I/O and Next Generation Interfaces (Dr. Julian Kunkel), HPC IODC Workshop, ISC HPC, Frankfurt, Germany, 2019-06-20 Presentation
- The goldilocks node: getting the RAM just right (Dr. Julian Kunkel), BoF: Data-Centric I/O, ISC HPC, Frankfurt, Germany, 2019-06-19 Presentation
- High-Level Workflows – Potential for Innovation? The NGI Initiative and More (Dr. Julian Kunkel), BoF: Data-Centric I/O, ISC HPC, Frankfurt, Germany, 2019-06-19 Presentation
- Fighting the Data Deluge with Data-Centric Middleware (Dr. Julian Kunkel), PASC Minisymposium: The Exabyte Data Challenge, Zürich, Switzerland, 2019-06-14 Presentation
- Opportunities for Integrating I/O Capabilities with Cylc (Dr. Julian Kunkel), Cylc Weather User Group; Supercomputing 2018, Dallas, USA, 2018-11-15 Presentation
- Status of WP4: Exploitability (Dr. Julian Kunkel, Bryan N. Lawrence, Jakob Luettgau, Neil Massey, Alessandro Danca, Sandro Fiore, Huang Hu), ESiWACE review meeting, DKRZ, Hamburg, Germany, 2018-11-06 Presentation
- Overcoming Storage Issues of Earth-System Data with Intelligent Storage Systems (Dr. Julian Kunkel), 18th Workshop on high performance computing in meteorology, ECMWF, Reading, UK, 2018-09-27 Presentation
- RD&E in Compression (Dr. Julian Kunkel), NCAS Meeting, Reading, UK, 2018-07-27
- Challenges and Opportunities for I/O (Dr. Julian Kunkel), Gung Ho Networking Meeting, Reading, UK, 2018-07-04 Presentation
- Cost and Performance Modeling for Earth System Data Management and Beyond (Dr. Julian Kunkel), HPC IODC Workshop, Frankfurt, Germany, 2018-06-28 Presentation
- Community Development of Next Generation Semantic Interfaces (Dr. Julian Kunkel), Joint session of the HPC IODC Workshop and WOPSSS, Frankfurt, Germany, 2018-06-28 Presentation
- The Need for Next Generation Semantic Interfaces to Process Climate/Weather Workflows (Dr. Julian Kunkel), SIG IO UK, University of Reading, UK, 2018-06-06 Presentation
- Towards Intelligent Storage Systems (Dr. Julian Kunkel), Computer Science Workshop, University of Reading, UK, 2018-05-08 Presentation
- Exploiting the Heterogeneous Storage Landscape in a Data Center (Dr. Julian Kunkel), Per3s: Performance and Scalability of Storage Systems Workshop, Rennes, France, 2018-01-12 Presentation
- Exploiting Weather and Climate Data at Scale (WP4) (Dr. Julian Kunkel), ESiWACE General Assembly, Berlin, Germany, 2017-12-12 Presentation