Overview¶
ICAT-PACER (Python Asynchronous Catalogue Entry Router) is a high-throughput, asynchronous data ingestion engine designed for the ICAT1 data catalogue. It is developed by the MIS section at the ALBA Synchrotron2.
The PACER provides a modern, modular, asynchronous, message-driven architecture built around RabbitMQ and Python.
Features¶
Core Features¶
-
Fast
High-throughput asynchronous message processing.
-
ICAT and VISA ready
-
Scalable
A modular, consumer-based architecture designed to scale horizontally and efficiently handle increasing data and processing demands.
-
Open Source, GNU GPLv3
ICAT-PACER is licensed under GNU GPLv3 and available on GitHub.
Data ingestion flows¶
PACER is shipped with built-in workflows related to ICAT and VISA, including:
- User synchronization.
- Investigation synchronization.
- Dataset ingestion.
- Dataset linking (acquisition-processed types).
- Automatic DOI minting.
Monitoring dashboard¶
An optional monitoring dashboard is available to provide visibility into PACER's message processing and data ingestion workflows. When deployed alongside PACER, it records all processed messages and enables comprehensive message and error tracking. The dashboard also supports message reingestion, simplifying troubleshooting and recovery operations.
The PACER's dashboard user interface.
About ICAT¶
The ICAT1 project provides a metadata catalogue and associated components to support end-to-end experimental data management for large-scale research facilities. It connects all stages of the research lifecycle, from proposal and experiment planning through data acquisition, analysis, and ultimately publication of results.
