Getting started¶

The PACER is designed to ingest data asynchronously, providing a scalable and flexible ingestion pipeline. It follows the producer–consumer pattern, decoupling data producers from data consumers through a shared message queue.

As a result, PACER requires:

A message broker to act as the shared queue between producers and consumers.
An ICAT instance, since PACER is designed as an ingestion engine for the ICAT ecosystem.

For the best user experience, we also recommend deploying the DRAC / Data Portal stack in front of ICAT. The ingestion workflows distributed with PACER are designed to integrate seamlessly with the Data Portal user interface.

Deploy a message broker¶

The PACER is developed and tested using RabbitMQ¹ as its message broker, and RabbitMQ is the recommended option for production deployments.

Kombu², the messaging library powering the PACER, also supports other platforms to use as message brokers such as Redis or Amazon SQS. To use a different broker, install the corresponding requirements file or set the BROKER_TYPE environment variable when building the Docker image.

Broker	Variable	Requirements file
RabbitMQ (default)
Redis	`BROKER_TYPE=redis`	`requirements_redis.txt`
Amazon SQS	`BROKER_TYPE=sqs`	`requirements_sqs.txt`

Info

The PACER and the dashboard are developed and tested with Python 3.11+.

Deploy the PACER¶

Configure the config.yaml file (refer to configuration for more details).
Build the PACER Docker image with the desired message broker requirements file.
Run the PACER container.

git clone https://github.com/ALBA-Synchrotron/icat-pacer.git
cd icat-pacer

docker build . -t icat-pacer:latest

docker run -d --name icat-pacer -v config.yaml:/config/config.yaml:ro --restart=always icat-pacer:latest

Deploy the PACER dashboard¶

Build the dashboard's Docker image with the desired message broker requirements file.
Run the dashboard's container.
Run the dashboard's worker.
Run the ASGI server for live stream monitoring.

git clone https://github.com/ALBA-Synchrotron/pacer-dashboard.git
cd pacer-dashboard

docker build . -t pacer-dashboard:latest

docker run -d --name pacer-dashboard \
    -e POSTGRES_HOST=<POSTGRES_HOST> -e POSTGRES_PASSWORD=<POSTGRES_PASSWORD> -e POSTGRES_USER=<POSTGRES_USER> \
    -e RMQ_HOST=<RMQ_HOST> -e RMQ_PASSWORD=<RMQ_PASSWORD> -e RMQ_USERNAME=<RMQ_USERNAME>  \
     -e RMQ_VHOST=<RMQ_VHOST> -e SECRET_KEY=<SECRET_KEY> \
    --restart=always pacer-dashboard:latest

docker run -d --name pacer-dashboard-worker \
    -e POSTGRES_HOST=<POSTGRES_HOST> -e POSTGRES_PASSWORD=<POSTGRES_PASSWORD> -e POSTGRES_USER=<POSTGRES_USER> \
    -e RMQ_HOST=<RMQ_HOST> -e RMQ_PASSWORD=<RMQ_PASSWORD> -e RMQ_USERNAME=<RMQ_USERNAME>  \
     -e RMQ_VHOST=<RMQ_VHOST> -e SECRET_KEY=<SECRET_KEY> \
    --restart=always pacer-dashboard:latest python -m celery -A dashboard worker -l INFO -E

docker run -d --name pacer-dashboard-asgi \
    -e POSTGRES_HOST=<POSTGRES_HOST> -e POSTGRES_PASSWORD=<POSTGRES_PASSWORD> -e POSTGRES_USER=<POSTGRES_USER> \
    -e RMQ_HOST=<RMQ_HOST> -e RMQ_PASSWORD=<RMQ_PASSWORD> -e RMQ_USERNAME=<RMQ_USERNAME>  \
     -e RMQ_VHOST=<RMQ_VHOST> -e SECRET_KEY=<SECRET_KEY> \
    --restart=always pacer-dashboard:latest python -m daphne -b 0.0.0.0 settings.asgi:application

Deploy ICAT¶

Although there is no single-step deployment for the entire ICAT ecosystem, several projects simplify the process significantly.

On Kubernetes¶

If you are deploying on a Kubernetes³ cluster, consider using the icat-k8s Helm chart. The chart orchestrates the core ICAT services and requires only an existing MariaDB or MySQL database.

DRAC¶

DRAC (Data Repository for Advancing open sCience), developed and maintained by ESRF⁴, provides a Docker-based deployment of the ICAT ecosystem together with the ESRF Data Portal as the frontend. It offers a convenient way to deploy a complete data catalogue platform around ICAT.