MD-Workbench
This MPI parallel benchmark mimics the concurrent access to typically small objects. It comes with the following features:
- deterministic access pattern mimicking interactive users or producer/consumer queues;
- configurable working set size to fit into a cache of a given size or exceed it;
- performance regression testing by preserving the working set between runs;
- support for various storage backends (POSIX, MPI-IO, S3, MongoDB, PostgreSQL), and
- report of throughput and latency statistics including timing individual I/O operations.
It therewith overcomes certain limitations of HPC metadata benchmarks and provides additional quantifiable measures.
Key Information
| Contact | Dr. Julian Kunkel | ||
| Repository | Public on GitHub | ||
| URL | MD-Workbench @ VI4IO |
Publications
- \myPub{2019}{Understanding Metadata Latency with MDWorkbench}{Julian Kunkel, George S. Markomanolis}{In High Performance Computing: ISC High Performance 2018 International Workshops, Frankfurt/Main, Germany, June 28, 2018, Revised Selected Papers, Lecture Notes in Computer Science (11203), pp. 75–88, Springer, WOPSSS workshop, ISC HPC, Frankfurt, Germany, ISBN: 978-3-030-02465-9, ISSN: 1611-3349}
Talks
- \myPub{2019}{Tracking User-Perceived I/O Slowdown via Probing}{HPC IODC Workshop}{ISC HPC, Frankfurt, Germany}
- \myPub{2018}{Understanding Metadata Latency with MDWorkbench}{WOPSSS}{Frankfurt, Germany}