ESiWACE

The Centre of Excellence in Simulation of Weather and Climate in Europe (ESiWACE) is a H2020 funded project.

Within the project, our research group is responsible for DKRZ's contribution to work package 4 (Exploitability).

Please see the official web page of ESiWACE for further information.

Contact (for WP4 contribution of DKRZ) Dr. Julian Kunkel

  • Modelling costs for storage methods and understanding these
  • Modelling tape archives and costs
  • Flexible disk storage layouts for earth system data
  • DKRZ
  • STFC
  • ECMWF
  • CMCC
  • Seagate

The design deliverable architecture-d4.2.pdf

ESDM builds upon a data model similar to NetCDF and utilizes a self-describing on-disk data format for storing structured data. We aim to deliver the NetCDF integrated version by the end of the ESiWACE1 project. This improvement can then be used as a drop-in replacement for typical use-cases without changing anything from the application perspective. While our current version utilises the manual configuration by data-center experts, the ultimate long-term goal is to employ machine learning to automatise the decision making and reduce the burden for users and experts.

Publications

Talks

  • Data-Centric I/O and Next Generation Interfaces (Dr. Julian Kunkel), HPC IODC Workshop, ISC HPC, Frankfurt, Germany, 2019-06-20 Presentation
  • The goldilocks node: getting the RAM just right (Dr. Julian Kunkel), BoF: Data-Centric I/O, ISC HPC, Frankfurt, Germany, 2019-06-19 Presentation
  • High-Level Workflows – Potential for Innovation? The NGI Initiative and More (Dr. Julian Kunkel), BoF: Data-Centric I/O, ISC HPC, Frankfurt, Germany, 2019-06-19 Presentation
  • Fighting the Data Deluge with Data-Centric Middleware (Dr. Julian Kunkel), PASC Minisymposium: The Exabyte Data Challenge, Zürich, Switzerland, 2019-06-14 Presentation
  • Opportunities for Integrating I/O Capabilities with Cylc (Dr. Julian Kunkel), Cylc Weather User Group; Supercomputing 2018, Dallas, USA, 2018-11-15 Presentation
  • Status of WP4: Exploitability (Dr. Julian Kunkel, Bryan N. Lawrence, Jakob Lüttgau, Neil Massey, Alessandro Danca, Sandro Fiore, Huang Hu), ESiWACE review meeting, DKRZ, Hamburg, Germany, 2018-11-06 Presentation
  • Overcoming Storage Issues of Earth-System Data with Intelligent Storage Systems (Dr. Julian Kunkel), 18th Workshop on high performance computing in meteorology, ECMWF, Reading, UK, 2018-09-27 Presentation
  • RD&E in Compression (Dr. Julian Kunkel), NCAS Meeting, Reading, UK, 2018-07-27
  • Challenges and Opportunities for I/O (Dr. Julian Kunkel), Gung Ho Networking Meeting, Reading, UK, 2018-07-04 Presentation
  • Cost and Performance Modeling for Earth System Data Management and Beyond (Dr. Julian Kunkel, Jakob Lüttgau), HPC IODC Workshop, Frankfurt, Germany, 2018-06-28 Presentation
  • Community Development of Next Generation Semantic Interfaces (Dr. Julian Kunkel), Joint session of the HPC IODC Workshop and WOPSSS, Frankfurt, Germany, 2018-06-28 Presentation
  • The Need for Next Generation Semantic Interfaces to Process Climate/Weather Workflows (Dr. Julian Kunkel), SIG IO UK, University of Reading, UK, 2018-06-06 Presentation
  • Towards Intelligent Storage Systems (Dr. Julian Kunkel), Computer Science Workshop, University of Reading, UK, 2018-05-08 Presentation
  • Exploiting the Heterogeneous Storage Landscape in a Data Center (Dr. Julian Kunkel), Per3s: Performance and Scalability of Storage Systems Workshop, Rennes, France, 2018-01-12 Presentation
  • Exploiting Weather and Climate Data at Scale (WP4) (Dr. Julian Kunkel), ESiWACE General Assembly, Berlin, Germany, 2017-12-12 Presentation