Originally published in Water Matters, issue december 2023
Continual water quality data provide a great deal of useful information.
Water quality managers therefore make more and more frequent use of sensor technology and auto-analysers. However, due to errors in measurements, the raw data from water quality sensors are not immediately usable for visualisation and interpretation. In this article, we explain the state-of-the-art, as a first step towards standardisation of data processing from water quality sensors.
Water authorities and water authority laboratories are investing more and more in the application of water quality sensors in surface water. For example, water authority laboratory AQUON currently manages about 200 water quality sensors. Nowadays, they can not only measure standard parameters, such as pH, electrical conductivity, oxygen, and temperature, but also, for example, nutrient concentrations. All these continual and real-time measurements provide a great deal of valuable information, for example, about drainage, the dynamic influence of wastewater treatment plant effluent and overflows, transport processes of nutrients, or operational water management (for example, deciding to let water in or barring it on the basis of salt concentrations).
Scaling-up and reliability
Due to the sensitivity of the technology and the variable conditions in the field, there are often irregularities in the sensor data. The raw data are therefore not immediately usable for visualisation and interpretation. However, due to the scaling-up in use of water quality sensors, manual checking and correction of the data are often a great deal of work. Depending on the intended use of the data, standardised data processing methods are therefore desirable (table 1). The aim of this research was to bring together standard optimisation routines for high-frequency water quality data and to make them available to users of water quality sensors (researchers and water managers) via a wiki-site [1].
Table 1: Aims of data optimisation and the related methods
| Aim of data optimisation | Background | Methods |
|---|---|---|
| Sensor management and maintenance / early warning | Real-time detection of deviating measurement values for fast response by technicians and/or maintenance staff | Anomaly detection methods, detection Of flatlines (always the same measurement value) |
| Online presentation of live measurements | Real-time check and filters for a comprehensive presentation for stakeholders | Anomaly detection noise filters |
| Retrospective correcting of series | Assembling optimum series, including with conventional low-frequency measurements for charge calculations, for example | Anomaly detection methods, noise filters, correction for drift and jumps in the data, filling in gaps in the series |
Approach
Through interviews and literature research, we charted which deviations occur frequently and which correction methods are already publicly available. Many methods have been developed in other fields. We included them in our search if they had already been applied to sensor data for water quality. For the various types of deviations in sensor data, we created optimisation routines, including the R scripts and example applications, and made these available via the wiki site [1]. The routines included there are partly (combinations of) methods that are already available, but we developed new methods for, for example, the correction of drift on the basis of laboratory measurements and for filling in gaps in measurement series.
