Getting started¶
The PACER is designed to ingest data asynchronously, providing a scalable and flexible ingestion pipeline. It follows the producer–consumer pattern, decoupling data producers from data consumers through a shared message queue.
As a result, PACER requires:
- A message broker to act as the shared queue between producers and consumers.
- An ICAT instance, since PACER is designed as an ingestion engine for the ICAT ecosystem.
For the best user experience, we also recommend deploying the DRAC / Data Portal stack in front of ICAT. The ingestion workflows distributed with PACER are designed to integrate seamlessly with the Data Portal user interface.
Deployment with Docker Compose¶
The PACER application and its dashboard can be quickly deployed for demonstration and testing purposes using the provided Docker Compose configuration.
To get started, download the following files:
First, copy the example environment file to .env in the same directory as the docker-compose.yml file:
Next, edit the .env file and configure the appropriate values for your environment. In particular, make sure to
provide the connection settings for the ICAT server (e.g. hostname, port, and any required credentials).
Important: The ICAT server must be deployed and accessible independently of this Docker Compose setup. It is an external dependency and is not deployed as part of the Compose stack.
Once the environment variables have been configured, start the services with:
Warning
The Docker Compose stack provided in this documentation is intended as a reference configuration and a quick way to get a working PACER environment up and running. It is primarily designed for demonstration, testing, and validation purposes.
Production deployments should use this configuration as a starting point and adapt or extend it to meet their specific operational requirements, rather than relying on it as-is. Particular attention should be paid to database deployment, data persistence, backup strategies, and service lifecycle management to ensure that future updates and upgrades can be applied reliably, without disrupting the environment or risking data loss.
Docker Compose will pull the required images (if needed) and start the PACER application and dashboard in the background.
You can check the status of the deployed services with:
And inspect their logs if needed:
To stop the deployment, run:
Manual deployment¶
Deploy a message broker¶
The PACER is developed and tested using RabbitMQ1 as its message broker, and RabbitMQ is the recommended option for production deployments.
Kombu2, the messaging library powering the PACER, also supports other platforms to use as message brokers such as
Redis or Amazon SQS. To use a different broker, install the corresponding requirements file or set the BROKER_TYPE
environment variable when building the Docker image.
| Broker | Variable | Requirements file |
|---|---|---|
| RabbitMQ (default) | ||
| Redis | BROKER_TYPE=redis |
requirements_redis.txt |
| Amazon SQS | BROKER_TYPE=sqs |
requirements_sqs.txt |
Info
The PACER and the dashboard are developed and tested with Python 3.11+.
Deploy the PACER¶
- Configure the
config.yamlfile (refer to configuration for more details). - Build the PACER Docker image with the desired message broker requirements file.
- Run the PACER container.
git clone https://github.com/ALBA-Synchrotron/icat-pacer.git
cd icat-pacer
docker build . -t icat-pacer:latest
docker run -d --name icat-pacer -v config.yaml:/config/config.yaml:ro --restart=always icat-pacer:latest
Deploy the PACER dashboard¶
- Build the dashboard's Docker image with the desired message broker requirements file.
- Run the dashboard's container and run the database migrations.
- Run the dashboard's worker.
- Run the ASGI server for live stream monitoring.
git clone https://github.com/ALBA-Synchrotron/pacer-dashboard.git
cd pacer-dashboard
docker build . -t pacer-dashboard:latest
docker run -d --name pacer-dashboard \
-e POSTGRES_HOST=<POSTGRES_HOST> -e POSTGRES_PASSWORD=<POSTGRES_PASSWORD> -e POSTGRES_USER=<POSTGRES_USER> \
-e RMQ_HOST=<RMQ_HOST> -e RMQ_PASSWORD=<RMQ_PASSWORD> -e RMQ_USERNAME=<RMQ_USERNAME> \
-e RMQ_VHOST=<RMQ_VHOST> -e SECRET_KEY=<SECRET_KEY> \
--restart=always pacer-dashboard:latest
docker run -d --name pacer-dashboard-worker \
-e POSTGRES_HOST=<POSTGRES_HOST> -e POSTGRES_PASSWORD=<POSTGRES_PASSWORD> -e POSTGRES_USER=<POSTGRES_USER> \
-e RMQ_HOST=<RMQ_HOST> -e RMQ_PASSWORD=<RMQ_PASSWORD> -e RMQ_USERNAME=<RMQ_USERNAME> \
-e RMQ_VHOST=<RMQ_VHOST> -e SECRET_KEY=<SECRET_KEY> \
--restart=always pacer-dashboard:latest python -m celery -A dashboard worker -l INFO -E
docker run -d --name pacer-dashboard-asgi \
-e POSTGRES_HOST=<POSTGRES_HOST> -e POSTGRES_PASSWORD=<POSTGRES_PASSWORD> -e POSTGRES_USER=<POSTGRES_USER> \
-e RMQ_HOST=<RMQ_HOST> -e RMQ_PASSWORD=<RMQ_PASSWORD> -e RMQ_USERNAME=<RMQ_USERNAME> \
-e RMQ_VHOST=<RMQ_VHOST> -e SECRET_KEY=<SECRET_KEY> \
--restart=always pacer-dashboard:latest python -m daphne -b 0.0.0.0 settings.asgi:application
Deploy ICAT¶
Although there is no single-step deployment for the entire ICAT ecosystem, several projects simplify the process significantly.
On Kubernetes¶
If you are deploying on a Kubernetes3 cluster, consider using the icat-k8s Helm chart. The chart orchestrates the core ICAT services and requires only an existing MariaDB or MySQL database.
DRAC¶
DRAC (Data Repository for Advancing open sCience), developed and maintained by ESRF4, provides a Docker-based deployment of the ICAT ecosystem together with the ESRF Data Portal as the frontend. It offers a convenient way to deploy a complete data catalogue platform around ICAT.