Topic
The objective of this task committee is to develop a platform called WaterBenchmarkHub,
and establish it as a standard platform in the community for hosting, sharing,
and accessing benchmarks.
This task committee operates under the Planning and Management Council and
the Water Distribution System Analysis committee.
Background
Publicly accessible and user-friendly benchmarks are crucial for advancing and accelerating
research in smart water systems and supporting reproducible studies. Benchmark resources
related to Water Distribution Systems are difficult to access, as they are not consolidated within a
centralized, standardized interface that could enhance their accessibility, usability, and relevance
for researchers. Currently, they are dispersed across various sources, often poorly labelled and
lacking clear guidance on their intended use or the research problems they address. We therefore
aim to build upon initial efforts from the ERC Water Futures project in developing a
WaterBenchmarkHub platform -- an open-source and community-driven platform, designed to
offer a standardized interface for benchmarking resources. The WaterBenchmarkHub will consist
of a web interface for browsing the benchmark resources, and a Python package for easy access
to those benchmark resources in Python. With this initiative, we aim to enhance accessibility for a
broader range of people and ultimately promote and strengthen reproducible research within the
community. To ultimately establish the WaterBenchmarkHub as a standard platform in the
community for hosting, sharing, and accessing benchmarks, we aim to work on the following
action points:
WDSA GitHub organization) We aim to push the establishment of an “official” WDSA GitHub
organization under which not only the WaterBenchmarkHub platform can be hosted, but which can
also be used by other task committees and officials for hosting their software repositories and
other related resources. We envision that this WDSA GitHub organization will be managed by
representatives of the WDSA community, instead of a single individual, and thus ensuring its
success in the long run.
Verified Benchmarks) We aim to contact and work together with the creators/developers of the
original benchmarks to verify their correctness. In this context, we plan to introduce a “verification
badge”, that indicates the given benchmark on the platform has been verified and approved by the
original creators.
Known solutions) For some popular benchmarks and past competitions, a lot of methods and
solutions already exist. To facilitate the comparison to exiting solutions/methods, we aim to include
a list of known solutions for those benchmarks on their respective description page on the
WaterBenchmarkHub.
Versioning and Unique Identifiers) Dealing with the fact that often multiple versions of the same
benchmark (in particular .inp files) exist, we aim to not only ensure that the original version is
available on the platform but also create a list of different versions, together with descriptions of
their origins and characteristics. In this context, we envision introducing unique identifiers (such as
DOIs issued via Zenodo) to avoid any ambiguities and enable easy reproducibility of experiments.
Easy User Experience for Contributing Benchmarks) To make data and benchmark contributions
as simple as possible, we envision developing a user interface that abstracts away from the
underlying database and system.
Persistent and Distributed Storage of (Large) Benchmark Resources) We aim to establish and
promote persistent and permanent storage of publicly available benchmarks. Larger benchmark
resources cannot be stored in GitHub repositories and require hosting on some permanent cloud
storage. Here, we want to establish a system of mirrors, hosted and run at different non-profit
institutes (e.g., universities), to ensure the long-term availability of the benchmarks while
distributing responsibilities and dependencies as much as possible.
Data-Paper Track at Major Journals) We aim to establish “Data-Paper” track at a major journal
such as the Journal of Water Resources Planning and Management (JWRPM), where researchers
can publish their data and getting acknowledged for the work spent on collecting and curation of it.
This is of particular importance, since a lot of data never makes it to publication for various
reasons, yet the data is likely useful for other research projects. Making data for research available
and fostering reproducible research is the key motivation behind the WaterBenchmarkHub.
Purpose
Publicly accessible and user-friendly benchmarks are crucial for advancing and
accelerating research in smart water systems and supporting reproducible
studies. Benchmark resources related to Water Distribution Systems are
difficult to access, as they are not consolidated within a centralized,
standardized interface that could enhance their accessibility, usability, and
relevance for researchers. Currently, they are dispersed across various
sources, often poorly labelled and lacking clear guidance on their intended use
or the research problems they address. Based on initial efforts from the ERC
Water Futures project, we therefore aim to build an open-source and
community-driven platform, called WaterBenchmarkHub, designed to offer a
standardized interface for benchmarking resources. The WaterBenchmarkHub
will consist of a web interface for browsing the benchmark resources, and a
Python package for easy access to those benchmark resources in Python.
With this initiative, we aim to enhance accessibility for a broader range of
people and ultimately promote and strengthen reproducible research within the
community.
The platform will feature:
- Verified benchmarks
- A versioning system with unique identifiers for resolving any ambiguities regarding different versions of specific benchmarks
- An intuitive user interface for contributing benchmarks
- A persistent and distributed storage system to guarantee long-term availability of benchmark resources
Task Committee Leadership
Member List
Please contact the committee chair (Dr. André Artelt) if you are interested in joining this task committee.
Members
- Farrah Moazeni – Lehigh University
- Lina Sela – UT Austin
- Andrea Cominola – TU Berlin
- Demetris Eliades – University of Cyprus
- Dennis Zanutto – KWR
- Jan Ručka – Brno University of Technology
Members
- Lindell Ormsbee – University of Kentucky
- Gal Perelman – Israel Institute of Technology
- Rajesh Gupta – Visvesvaraya National Institute of Technology
- Juneseok Lee – Manhattan University
Members
- Felipe Francisco Nusda
- Mariam Madbouly – Ain Shams University
- Adithya Ramachandran – FAU Erlangen-Nürnberg
- Pedro L. Iglesias Rey – Universitat Politècnica de València