Skip to main content

Topic

The objective of this task committee is to develop a platform called WaterBenchmarkHub, and establish it as a standard platform in the community for hosting, sharing, and accessing benchmarks.

This task committee operates under the Planning and Management Council and the Water Distribution System Analysis committee.

Background

Publicly accessible and user-friendly benchmarks are crucial for advancing and accelerating research in smart water systems and supporting reproducible studies. Benchmark resources related to Water Distribution Systems are difficult to access, as they are not consolidated within a centralized, standardized interface that could enhance their accessibility, usability, and relevance for researchers. Currently, they are dispersed across various sources, often poorly labelled and lacking clear guidance on their intended use or the research problems they address. We therefore aim to build upon initial efforts from the ERC Water Futures project in developing a WaterBenchmarkHub platform -- an open-source and community-driven platform, designed to offer a standardized interface for benchmarking resources. The WaterBenchmarkHub will consist of a web interface for browsing the benchmark resources, and a Python package for easy access to those benchmark resources in Python. With this initiative, we aim to enhance accessibility for a broader range of people and ultimately promote and strengthen reproducible research within the community. To ultimately establish the WaterBenchmarkHub as a standard platform in the community for hosting, sharing, and accessing benchmarks, we aim to work on the following action points:

WDSA GitHub organization) We aim to push the establishment of an “official” WDSA GitHub organization under which not only the WaterBenchmarkHub platform can be hosted, but which can also be used by other task committees and officials for hosting their software repositories and other related resources. We envision that this WDSA GitHub organization will be managed by representatives of the WDSA community, instead of a single individual, and thus ensuring its success in the long run.

Verified Benchmarks) We aim to contact and work together with the creators/developers of the original benchmarks to verify their correctness. In this context, we plan to introduce a “verification badge”, that indicates the given benchmark on the platform has been verified and approved by the original creators.

Known solutions) For some popular benchmarks and past competitions, a lot of methods and solutions already exist. To facilitate the comparison to exiting solutions/methods, we aim to include a list of known solutions for those benchmarks on their respective description page on the WaterBenchmarkHub.

Versioning and Unique Identifiers) Dealing with the fact that often multiple versions of the same benchmark (in particular .inp files) exist, we aim to not only ensure that the original version is available on the platform but also create a list of different versions, together with descriptions of their origins and characteristics. In this context, we envision introducing unique identifiers (such as DOIs issued via Zenodo) to avoid any ambiguities and enable easy reproducibility of experiments.

Easy User Experience for Contributing Benchmarks) To make data and benchmark contributions as simple as possible, we envision developing a user interface that abstracts away from the underlying database and system.

Persistent and Distributed Storage of (Large) Benchmark Resources) We aim to establish and promote persistent and permanent storage of publicly available benchmarks. Larger benchmark resources cannot be stored in GitHub repositories and require hosting on some permanent cloud storage. Here, we want to establish a system of mirrors, hosted and run at different non-profit institutes (e.g., universities), to ensure the long-term availability of the benchmarks while distributing responsibilities and dependencies as much as possible.

Data-Paper Track at Major Journals) We aim to establish “Data-Paper” track at a major journal such as the Journal of Water Resources Planning and Management (JWRPM), where researchers can publish their data and getting acknowledged for the work spent on collecting and curation of it. This is of particular importance, since a lot of data never makes it to publication for various reasons, yet the data is likely useful for other research projects. Making data for research available and fostering reproducible research is the key motivation behind the WaterBenchmarkHub.

Purpose

Publicly accessible and user-friendly benchmarks are crucial for advancing and accelerating research in smart water systems and supporting reproducible studies. Benchmark resources related to Water Distribution Systems are difficult to access, as they are not consolidated within a centralized, standardized interface that could enhance their accessibility, usability, and relevance for researchers. Currently, they are dispersed across various sources, often poorly labelled and lacking clear guidance on their intended use or the research problems they address. Based on initial efforts from the ERC Water Futures project, we therefore aim to build an open-source and community-driven platform, called WaterBenchmarkHub, designed to offer a standardized interface for benchmarking resources. The WaterBenchmarkHub will consist of a web interface for browsing the benchmark resources, and a Python package for easy access to those benchmark resources in Python. With this initiative, we aim to enhance accessibility for a broader range of people and ultimately promote and strengthen reproducible research within the community.

The platform will feature:

  • Verified benchmarks
  • A versioning system with unique identifiers for resolving any ambiguities regarding different versions of specific benchmarks
  • An intuitive user interface for contributing benchmarks
  • A persistent and distributed storage system to guarantee long-term availability of benchmark resources
Furthermore, we aim to establish a “Data-Paper” track at a major journal, where researchers can publish their data and get acknowledged for the work spent on collecting and curation of it. This is of particular importance, since a lot of data never makes it to publication for various reasons, yet the data is likely useful for other research projects.

Task Committee Leadership

Dr. André Artelt

Chair

Bielefeld University

Dr. Sriman Pankaj

Vice Chair

TU Berlin

Prof. Avi Ostfeld

Secretary / Coordinator

Israel Institute of Technology

Member List

Please contact the committee chair (Dr. André Artelt) if you are interested in joining this task committee.

Members

  • Farrah Moazeni – Lehigh University
  • Lina Sela – UT Austin
  • Andrea Cominola – TU Berlin
  • Demetris Eliades – University of Cyprus
  • Dennis Zanutto – KWR
  • Jan Ručka – Brno University of Technology

Members

  • Lindell Ormsbee – University of Kentucky
  • Gal Perelman – Israel Institute of Technology
  • Rajesh Gupta – Visvesvaraya National Institute of Technology
  • Juneseok Lee – Manhattan University

Members

  • Felipe Francisco Nusda
  • Mariam Madbouly – Ain Shams University
  • Adithya Ramachandran – FAU Erlangen-Nürnberg
  • Pedro L. Iglesias Rey – Universitat Politècnica de València