Skip to content

Getting started

The PACER is designed to ingest data asynchronously, providing a scalable and flexible ingestion pipeline. It follows the producer–consumer pattern, decoupling data producers from data consumers through a shared message queue.

As a result, PACER requires:

  • A message broker to act as the shared queue between producers and consumers.
  • An ICAT instance, since PACER is designed as an ingestion engine for the ICAT ecosystem.

For the best user experience, we also recommend deploying the DRAC / Data Portal stack in front of ICAT. The ingestion workflows distributed with PACER are designed to integrate seamlessly with the Data Portal user interface.

Deployment with Docker Compose

The PACER application and its dashboard can be quickly deployed for demonstration and testing purposes using the provided Docker Compose configuration.

To get started, download the following files:

docker-compose.yml .env file

First, copy the example environment file to .env in the same directory as the docker-compose.yml file:

cp example.env .env

Next, edit the .env file and configure the appropriate values for your environment. In particular, make sure to provide the connection settings for the ICAT server (e.g. hostname, port, and any required credentials).

Important: The ICAT server must be deployed and accessible independently of this Docker Compose setup. It is an external dependency and is not deployed as part of the Compose stack.

Once the environment variables have been configured, start the services with:

docker compose up -d

Warning

The Docker Compose stack provided in this documentation is intended as a reference configuration and a quick way to get a working PACER environment up and running. It is primarily designed for demonstration, testing, and validation purposes.

Production deployments should use this configuration as a starting point and adapt or extend it to meet their specific operational requirements, rather than relying on it as-is. Particular attention should be paid to database deployment, data persistence, backup strategies, and service lifecycle management to ensure that future updates and upgrades can be applied reliably, without disrupting the environment or risking data loss.

Docker Compose will pull the required images (if needed) and start the PACER application and dashboard in the background.

You can check the status of the deployed services with:

docker compose ps

And inspect their logs if needed:

docker compose logs -f

To stop the deployment, run:

docker compose down

Manual deployment

Deploy a message broker

The PACER is developed and tested using RabbitMQ1 as its message broker, and RabbitMQ is the recommended option for production deployments.

Kombu2, the messaging library powering the PACER, also supports other platforms to use as message brokers such as Redis or Amazon SQS. To use a different broker, install the corresponding requirements file or set the BROKER_TYPE environment variable when building the Docker image.

Broker Variable Requirements file
RabbitMQ (default)
Redis BROKER_TYPE=redis requirements_redis.txt
Amazon SQS BROKER_TYPE=sqs requirements_sqs.txt

Info

The PACER and the dashboard are developed and tested with Python 3.11+.

Deploy the PACER

  1. Configure the config.yaml file (refer to configuration for more details).
  2. Build the PACER Docker image with the desired message broker requirements file.
  3. Run the PACER container.
git clone https://github.com/ALBA-Synchrotron/icat-pacer.git
cd icat-pacer

docker build . -t icat-pacer:latest

docker run -d --name icat-pacer -v config.yaml:/config/config.yaml:ro --restart=always icat-pacer:latest

Deploy the PACER dashboard

  1. Build the dashboard's Docker image with the desired message broker requirements file.
  2. Run the dashboard's container and run the database migrations.
  3. Run the dashboard's worker.
  4. Run the ASGI server for live stream monitoring.
git clone https://github.com/ALBA-Synchrotron/pacer-dashboard.git
cd pacer-dashboard

docker build . -t pacer-dashboard:latest

docker run -d --name pacer-dashboard \
    -e POSTGRES_HOST=<POSTGRES_HOST> -e POSTGRES_PASSWORD=<POSTGRES_PASSWORD> -e POSTGRES_USER=<POSTGRES_USER> \
    -e RMQ_HOST=<RMQ_HOST> -e RMQ_PASSWORD=<RMQ_PASSWORD> -e RMQ_USERNAME=<RMQ_USERNAME>  \
     -e RMQ_VHOST=<RMQ_VHOST> -e SECRET_KEY=<SECRET_KEY> \
    --restart=always pacer-dashboard:latest

docker run -d --name pacer-dashboard-worker \
    -e POSTGRES_HOST=<POSTGRES_HOST> -e POSTGRES_PASSWORD=<POSTGRES_PASSWORD> -e POSTGRES_USER=<POSTGRES_USER> \
    -e RMQ_HOST=<RMQ_HOST> -e RMQ_PASSWORD=<RMQ_PASSWORD> -e RMQ_USERNAME=<RMQ_USERNAME>  \
     -e RMQ_VHOST=<RMQ_VHOST> -e SECRET_KEY=<SECRET_KEY> \
    --restart=always pacer-dashboard:latest python -m celery -A dashboard worker -l INFO -E

docker run -d --name pacer-dashboard-asgi \
    -e POSTGRES_HOST=<POSTGRES_HOST> -e POSTGRES_PASSWORD=<POSTGRES_PASSWORD> -e POSTGRES_USER=<POSTGRES_USER> \
    -e RMQ_HOST=<RMQ_HOST> -e RMQ_PASSWORD=<RMQ_PASSWORD> -e RMQ_USERNAME=<RMQ_USERNAME>  \
     -e RMQ_VHOST=<RMQ_VHOST> -e SECRET_KEY=<SECRET_KEY> \
    --restart=always pacer-dashboard:latest python -m daphne -b 0.0.0.0 settings.asgi:application

Deploy ICAT

Although there is no single-step deployment for the entire ICAT ecosystem, several projects simplify the process significantly.

On Kubernetes

If you are deploying on a Kubernetes3 cluster, consider using the icat-k8s Helm chart. The chart orchestrates the core ICAT services and requires only an existing MariaDB or MySQL database.

DRAC

DRAC (Data Repository for Advancing open sCience), developed and maintained by ESRF4, provides a Docker-based deployment of the ICAT ecosystem together with the ESRF Data Portal as the frontend. It offers a convenient way to deploy a complete data catalogue platform around ICAT.