Automatic data quality control for understanding extreme climate event
ORCID: https://orcid.org/0000-0002-4640-441X, Koppe, Roland
ORCID: https://orcid.org/0000-0002-2826-3932, Haas, Antonie
ORCID: https://orcid.org/0000-0002-3771-4125, Schäfer-Neth, Christian
ORCID: https://orcid.org/0000-0002-6995-8706, Fischer, Philipp
ORCID: https://orcid.org/0000-0002-3357-5420, Immoor, Sebastian
ORCID: https://orcid.org/0000-0002-7563-4521, Gerchow, Peter, Fritzsch, Bernadette and Frickenhaus, Stephan
ORCID: https://orcid.org/0000-0002-0356-9791
;
The understanding of extreme events strongly depends on knowledge gained from data. Data integration of mul-tiple sources, scales and earth compartments is the fo-cus of the project Digital Earth, which also join efforts on the quality control of data. Automatic quality control is embedded in the ingest component of the O2A, the ob-servation-to-archive data flow framework of the Alfred-Wegener-Institute. In that framework, the O2A-Sensor provides observation properties to the O2A-Ingest, which delivers quality-flagged data to the O2A-dash-board. The automatic quality control currently follows a procedural approach, where modules are included to implement formulations found in the literature and other operational observatory networks. A set of plausibility tests including range, spike and gradient tests are cur-rently operational. The automatic quality control scans the ingesting data in near-real-time (NRT) format, builds a table of devices, and search - either by absolute or derivative values - for correctness and validity of obser-vations. The availability of observation properties, for in-stance tests parameters like physical or operation ranges, triggers the automatic quality control, which in turn iterates through the table of devices to set the qual-ity flag for each sample and observation. To date, the quality flags in use are sequential and qualitative, i.e. it describes a level of quality in the data. A new flagging system is under development to include a descriptive characteristic that will comprise technical and user inter-pretation. Within Digital Earth, data on flood and drought events along the Elbe River and methane emissions in the North Sea are to be reviewed using automatic qual-ity control. Fast and scalable automatic quality control will disentangle uncertainty raised by quality issues and thus improve our understanding of extreme events in those cases.
ORCID: https://orcid.org/0000-0002-4640-441X, Koppe, Roland
ORCID: https://orcid.org/0000-0002-2826-3932, Haas, Antonie
ORCID: https://orcid.org/0000-0002-3771-4125, Schäfer-Neth, Christian
ORCID: https://orcid.org/0000-0002-6995-8706, Fischer, Philipp
ORCID: https://orcid.org/0000-0002-3357-5420, Immoor, Sebastian
ORCID: https://orcid.org/0000-0002-7563-4521, Gerchow, Peter, Fritzsch, Bernadette and Frickenhaus, Stephan
ORCID: https://orcid.org/0000-0002-0356-9791
;
