This guide covers what to consider before upgrading, the recommended upgrade sequence, and how to validate that the new agent version works correctly with your Waldur Mastermind instance.
waldur-api-client pin is the compatibility contractThe agent talks to Waldur Mastermind through the waldur-api-client Python package,
which is a generated SDK pinned to an exact version in the agent’s pyproject.toml.
A compatible pair means:
When you upgrade the agent to a new release, its waldur-api-client pin almost always
bumps too. Check the CHANGELOG for entries about waldur-api-client
upgrades — these indicate a new API surface is required and you must ensure Mastermind
is recent enough to support it.
Before upgrading, read the CHANGELOG entries between your current version and the target version and look for:
| Signal | What it means |
|---|---|
| New required configuration keys | Add them to the config file before starting the agent, or it will fail to start. |
| Removed configuration keys | Remove them — the agent may reject an unrecognised key. |
| Backend behaviour changes | Check the CHANGELOG description; verify flags and settings match the new expectations. |
| New plugin packages | Install the relevant waldur-site-agent-<plugin> package if you use that backend. |
The agent requires Mastermind to expose API endpoints that match its waldur-api-client pin.
Upgrade Mastermind before the agent — a newer agent talking to an older Mastermind can
fail with 404 Not Found or schema validation errors on endpoints the old Mastermind
does not yet expose.
If you run agent-event-process, also check whether any new event types were added.
The agent subscribes to topics at startup; a configuration mismatch between agent and
the RabbitMQ/STOMP broker does not prevent startup but can cause silent gaps in processing.
Always upgrade Waldur Mastermind first, then the site agent.
1. Upgrade Waldur Mastermind
2. Verify Mastermind is healthy (API responds, worker processes running)
3. Stop site agent services
4. Upgrade waldur-site-agent (and plugins)
5. Update configuration if the release requires new keys
6. Start site agent services
7. Validate (see below)
The agent reads from and writes to Mastermind. During a Mastermind upgrade the agent can safely continue running against the old version — it will use existing endpoints. The reverse is not safe: a new agent may call endpoints that do not yet exist in an older Mastermind, causing immediate errors.
# Polling mode
systemctl stop waldur-agent-order-process waldur-agent-membership-sync waldur-agent-report
# Event-process (STOMP) mode
systemctl stop waldur-agent-event-process waldur-agent-report
# PyPI install
pip install --upgrade waldur-site-agent
# With specific plugins (upgrade all at once to keep versions in sync)
pip install --upgrade \
waldur-site-agent \
waldur-site-agent-slurm \
waldur-site-agent-keycloak-client
All plugin packages share the same version number as the core package. Always upgrade all installed plugins together with the core.
If you deploy via Helm, the chart version mirrors the agent release version. Charts are published to https://waldur.github.io/waldur-site-agent. If you have not added that repository yet:
helm repo add waldur https://waldur.github.io/waldur-site-agent
Refresh the index, check which versions are available, then upgrade. Update
image.tag (or use the chart’s default) and run:
helm repo update
helm search repo waldur/waldur-site-agent --versions
helm upgrade waldur-site-agent waldur/waldur-site-agent --version <NEW_VERSION>
Add --devel to both helm search and helm upgrade when moving to a release
candidate — Helm hides pre-release versions otherwise.
See the chart README for the full list of configurable values.
waldur_site_diagnostics checks connectivity, token permissions, offering availability,
and backend health for every offering in the configuration:
waldur_site_diagnostics -c /etc/waldur/waldur-site-agent-config.yaml
A successful run prints DIAGNOSTICS START … DIAGNOSTICS END with no errors and exits 0.
Any ERROR line indicates a problem to fix before starting production services.
After starting services, verify the agent is processing work by checking logs for normal activity within one reconciliation interval:
# Polling mode — look for successful order/membership cycles
journalctl -u waldur-agent-order-process.service -f
# Event-process mode — look for STOMP connection confirmation and heartbeats
journalctl -u waldur-agent-event-process.service -f
Signs of a healthy agent:
Connected to STOMP broker (event-process mode)Processing orders… / Membership sync complete (polling mode)ERROR lines within the first few minutesIn the Waldur service provider interface, open the offering and verify:
Place a small test order through Waldur and confirm the agent picks it up, provisions
the backend resource, and transitions the order to Done within a reasonable time.
See SLURM Plugin Upgrade Notes for SLURM-specific
backend_settings reference, QoS configuration, account hierarchy behaviour,
and post-upgrade validation steps.
The agent is stateless — its only persistent state is in Waldur Mastermind and the backend (e.g. SLURM accounts). Rolling back is safe as long as the older agent version is compatible with the current Mastermind version.
pip install waldur-site-agent==<PREVIOUS_VERSION>
systemctl restart waldur-agent-*
If you also rolled back Mastermind, roll it back before rolling back the agent, following the same Mastermind-first order.