ESiWACE
The Centre of Excellence in Simulation of Weather and Climate in Europe (ESiWACE) is a H2020 funded project.
Within the project, our research group is responsible for DKRZ's contribution to work package 4 (Exploitability).
Please see the official web page of ESiWACE for further information.
Contact (for WP4 contribution of DKRZ) Dr. Julian Kunkel
People
- Jakob Lüttgau
- Dr. Philipp Neumann
Goals for WP4
- Modelling costs for storage methods and understanding these
- Modelling tape archives and costs
- Flexible disk storage layouts for earth system data
Collaboration partners for WP4
- DKRZ
- STFC
- ECMWF
- CMCC
- Seagate
Flexible Storage Layout for Earth-System Data
The design deliverable architecture-d4.2.pdf
ESDM builds upon a data model similar to NetCDF and utilizes a self-describing on-disk data format for storing structured data. We aim to deliver the NetCDF integrated version by the end of the ESiWACE1 project. This improvement can then be used as a drop-in replacement for typical use-cases without changing anything from the application perspective. While our current version utilises the manual configuration by data-center experts, the ultimate long-term goal is to employ machine learning to automatise the decision making and reduce the burden for users and experts.
Publications
- \myPub{2019}{Toward Understanding I/O Behavior in HPC Workflows}{Jakob Lüttgau, Shane Snyder, Philip Carns, Justin M. Wozniak, Julian Kunkel, Thomas Ludwig}{In IEEE/ACM 3rd International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems (PDSW-DISCS), pp. 64–75, IEEE Computer Society, PDSW-DISCS, Dallas, Texas, ISBN: 978-1-7281-0192-7}
- \myPub{}{Beating data bottlenecks in weather and climate science}{Bryan N. Lawrence, Julian Kunkel, Jonathan Churchill, Neil Massey, Philip Kershaw, Matt Pritchard}{In Extreme Data Workshop 2018, Schriften des Forschungszentrums Jülich IAS Series (40), pp. 31–36, Forschungszentrum Jülich, Extreme Data Workshop, Jülich, Germany, ISBN: 978-3-95806-392-1, ISSN: 1868-8489}
- \myPub{}{Cost and Performance Modeling for Earth System Data Management and Beyond}{Jakob Lüttgau, Julian Kunkel}{In High Performance Computing: ISC High Performance 2018 International Workshops, Frankfurt/Main, Germany, June 28, 2018, Revised Selected Papers, Lecture Notes in Computer Science (11203), pp. 23–35, Springer, HPC-IODC workshop, ISC HPC, Frankfurt, Germany, ISBN: 978-3-030-02465-9, ISSN: 1611-3349}
- \myPub{2018}{A Survey of Storage Systems for High-Performance Computing}{Jakob Lüttgau, Michael Kuhn, Kira Duwe, Yevhen Alforov, Eugen Betke, Julian Kunkel, Thomas Ludwig}{In Supercomputing Frontiers and Innovations, Series: Volume 5, Number 1, pp. 31–58, Publishing Center of South Ural State University}
- \myPub{2017}{Adaptive Tier Selection for NetCDF and HDF5}{Jakob Lüttgau, Eugen Betke, Olga Perevalova, Julian Kunkel, Michael Kuhn}{Poster, SC17, Denver, CO, USA}
- \myPub{}{Simulation of Hierarchical Storage Systems for TCO and QoS}{Jakob Lüttgau, Julian Kunkel}{In High Performance Computing: ISC High Performance 2017 International Workshops, DRBSD, ExaComm, HCPM, HPC-IODC, IWOPH, IXPUG, P^3MA, VHPC, Visualization at Scale, WOPSSS, Lecture Notes in Computer Science (10524), pp. 116–128, Springer, ISC High Performance, Frankfurt, Germany, ISBN: 978-3-319-67629-6}
- \myPub{2016}{Modeling and Simulation of Tape Libraries for Hierarchical Storage Systems}{Jakob Lüttgau, Julian Kunkel}{Poster, SC16, Salt Lake City, Utah, USA}
Talks
- \myPub{2020}{Data Systems at Scale in Climate and Weather: Activities in the ESiWACE Project}{HPC IODC Workshop}{Virtual}
- \myPub{}{Data-Centric IO: Potential for Climate/Weather}{6th ENES HPC Workshop}{Virtual/Hamburg, Germany}
- \myPub{}{Progress of WP4: Data at Scale}{ESiWACE General Assembly}{Virtual/Hamburg, Germany}
- \myPub{}{Toward Next Generation Interfaces for Exploiting Workflows}{SIG IO UK}{University of Reading, Reading, UK}
- \myPub{}{Potential of I/O-Aware Workflows in Climate and Weather}{Supercomputing Frontiers Europe}{Virtual/Warshaw Poland}
- \myPub{}{Challenges and Approaches for Extreme Data Processing}{EPSRC Centre for Doctoral Training Mathematics of Planet Earth}{University of Reading, Reading, UK}
- \myPub{2019}{Smarter Management using Metadata and Workflow Expertise}{BoF: Knowledge Is Power: Unleashing the Potential of Your Archives Through Metadata}{Supercomputing, Denver, USA}
- \myPub{}{Exploiting Different Storage Types with the Earth-System Data Middleware}{Parallel Data Systems Workshop}{Supercomputing, Denver, USA}
- \myPub{}{OpenSource Software}{Hacktoberfest}{University of Reading, Reading, UK}
- \myPub{}{The Earth-System Data Middleware: An Approach for Heterogeneous Storage Infrastructure}{SPPEXA Final Symposium}{Dresden, Germany}
- \myPub{}{Utilizing Heterogeneous Storage Infrastructures via the Earth-System Data Middleware}{NEXTGenIO Workshop on applications of NVRAM storage to exascale I/O}{ECMWF, Reading, Germany}
- \myPub{}{Data-Centric I/O and Next Generation Interfaces}{HPC IODC Workshop}{ISC HPC, Frankfurt, Germany}
- \myPub{}{The goldilocks node: getting the RAM just right}{BoF: Data-Centric I/O}{ISC HPC, Frankfurt, Germany}
- \myPub{}{High-Level Workflows – Potential for Innovation? The NGI Initiative and More}{BoF: Data-Centric I/O}{ISC HPC, Frankfurt, Germany}
- \myPub{}{Fighting the Data Deluge with Data-Centric Middleware}{PASC Minisymposium: The Exabyte Data Challenge}{Zürich, Switzerland}
- \myPub{2018}{Opportunities for Integrating I/O Capabilities with Cylc}{Cylc Weather User Group; Supercomputing 2018}{Dallas, USA}
- \myPub{}{Status of WP4: Exploitability}{ESiWACE review meeting}{DKRZ, Hamburg, Germany}
- \myPub{}{Overcoming Storage Issues of Earth-System Data with Intelligent Storage Systems}{18th Workshop on high performance computing in meteorology}{ECMWF, Reading, UK}
- \myPub{}{RD&E in Compression}{NCAS Meeting}{Reading, UK}
- \myPub{}{Challenges and Opportunities for I/O}{Gung Ho Networking Meeting}{Reading, UK}
- \myPub{}{Cost and Performance Modeling for Earth System Data Management and Beyond}{HPC IODC Workshop}{Frankfurt, Germany}
- \myPub{}{Community Development of Next Generation Semantic Interfaces}{Joint session of the HPC IODC Workshop and WOPSSS}{Frankfurt, Germany}
- \myPub{}{The Need for Next Generation Semantic Interfaces to Process Climate/Weather Workflows}{SIG IO UK}{University of Reading, UK}
- \myPub{}{Towards Intelligent Storage Systems}{Computer Science Workshop}{University of Reading, UK}
- \myPub{}{Exploiting the Heterogeneous Storage Landscape in a Data Center}{Per3s: Performance and Scalability of Storage Systems Workshop}{Rennes, France}
- \myPub{2017}{Exploiting Weather and Climate Data at Scale (WP4)}{ESiWACE General Assembly}{Berlin, Germany}