waldur-site-agent

Deployment Guide

This guide covers production deployment of Waldur Site Agent using systemd services.

Deploying on Kubernetes? Use the published Helm chart instead:

helm repo add waldur https://waldur.github.io/waldur-site-agent
helm repo update
helm install waldur-site-agent waldur/waldur-site-agent

The chart runs the same agent modes described below as separate deployments. See the chart README for available versions and configurable values.

Deployment Overview

The agent can run in 4 different modes, deployed as separate systemd services:

1. agent-order-process: Processes orders from Waldur
2. agent-report: Reports usage data to Waldur
3. agent-membership-sync: Synchronizes memberships
4. agent-event-process: Event-based processing (alternative to #1 and #3)

Service Combinations

Option 1: Polling-based (traditional)

• agent-order-process
• agent-membership-sync
• agent-report

Option 2: Event-based (requires STOMP)

• agent-event-process
• agent-report

Note: Only one combination can be active at a time.

Systemd Service Setup

Download Service Files

# Order processing service
sudo curl -L \
https://raw.githubusercontent.com/waldur/waldur-site-agent/main/systemd-conf/agent-order-process/agent.service \
  -o /etc/systemd/system/waldur-agent-order-process.service

# Reporting service
sudo curl -L \
https://raw.githubusercontent.com/waldur/waldur-site-agent/main/systemd-conf/agent-report/agent.service \
  -o /etc/systemd/system/waldur-agent-report.service

# Membership sync service
sudo curl -L \
https://raw.githubusercontent.com/waldur/waldur-site-agent/main/systemd-conf/agent-membership-sync/agent.service \
  -o /etc/systemd/system/waldur-agent-membership-sync.service

# Event processing service
sudo curl -L \
https://raw.githubusercontent.com/waldur/waldur-site-agent/main/systemd-conf/agent-event-process/agent.service \
  -o /etc/systemd/system/waldur-agent-event-process.service

Legacy Systemd Support

For systemd versions older than 240:

# Use legacy service files instead
sudo curl -L \
https://raw.githubusercontent.com/waldur/waldur-site-agent/main/systemd-conf/agent-order-process/agent-legacy.service \
  -o /etc/systemd/system/waldur-agent-order-process.service

# Repeat for other services with -legacy.service files

Enable and Start Services

Option 1: Polling-based Deployment

systemctl daemon-reload

# Start and enable services
systemctl start waldur-agent-order-process.service
systemctl enable waldur-agent-order-process.service

systemctl start waldur-agent-report.service
systemctl enable waldur-agent-report.service

systemctl start waldur-agent-membership-sync.service
systemctl enable waldur-agent-membership-sync.service

Option 2: Event-based Deployment

systemctl daemon-reload

# Start and enable services
systemctl start waldur-agent-event-process.service
systemctl enable waldur-agent-event-process.service

systemctl start waldur-agent-report.service
systemctl enable waldur-agent-report.service

Service Management

Check Service Status

# Check individual service
systemctl status waldur-agent-order-process.service

# Check all waldur services
systemctl status waldur-agent-*

View Logs

# Follow logs for a service
journalctl -u waldur-agent-order-process.service -f

# View recent logs
journalctl -u waldur-agent-order-process.service --since "1 hour ago"

# View logs for all agents
journalctl -u waldur-agent-* -f

Restart Services

# Restart individual service
systemctl restart waldur-agent-order-process.service

# Restart all agent services
systemctl restart waldur-agent-*

Configuration Management

Configuration File Location

The default configuration file location is /etc/waldur/waldur-site-agent-config.yaml.

Update Configuration

1. Edit configuration file:

   sudo nano /etc/waldur/waldur-site-agent-config.yaml
   

2. Validate configuration:

   waldur_site_diagnostics -c /etc/waldur/waldur-site-agent-config.yaml
   

3. Restart services:

   systemctl restart waldur-agent-*
   

Event-Based Processing Setup

STOMP Configuration

For STOMP-based event processing:

offerings:
  - name: "Your Offering"
    # ... other settings ...
    stomp_enabled: true
    websocket_use_tls: true

Important: Configure the event bus settings in Waldur to match your agent configuration.

Monitoring and Alerting

Health Checks

Create a monitoring script:

#!/bin/bash
# /usr/local/bin/check-waldur-agent.sh

SERVICES=("waldur-agent-order-process" "waldur-agent-report" "waldur-agent-membership-sync")

for service in "${SERVICES[@]}"; do
    if ! systemctl is-active --quiet "$service"; then
        echo "CRITICAL: $service is not running"
        exit 2
    fi
done

echo "OK: All Waldur agent services are running"
exit 0

Log Rotation

Systemd handles log rotation automatically via journald. Configure retention:

# Edit journald configuration
sudo nano /etc/systemd/journald.conf

# Add or modify:
SystemMaxUse=1G
MaxRetentionSec=1month

Sentry Integration

Add Sentry DSN to configuration for error tracking:

sentry_dsn: "https://your-dsn@sentry.io/project"

Set environment in systemd service files:

[Service]
Environment=SENTRY_ENVIRONMENT=production

Security Considerations

File Permissions

# Secure configuration file
sudo chmod 600 /etc/waldur/waldur-site-agent-config.yaml
sudo chown root:root /etc/waldur/waldur-site-agent-config.yaml

API Token Security

• For the source offering: the token user needs OFFERING.MANAGER role on the offering
• For Waldur federation (waldur backend): the target token user needs customer owner (can be a non-SP customer) and ISD identity manager (managed_isds set)
• Use dedicated service accounts in Waldur
• Rotate API tokens regularly
• Store tokens securely (consider using systemd credentials)

Network Security

• Restrict outbound connections to Waldur API endpoints
• Use TLS for all connections
• Configure firewall rules appropriately

Troubleshooting

Before digging into a specific symptom below, run:

waldur_site_diagnostics -c /etc/waldur/waldur-site-agent-config.yaml

This checks the Waldur side of the setup — API reachability, token auth, offering state, loaded components — regardless of which backend you’re running. It does not check backend connectivity (SLURM/MOAB/etc.). If you’re on the SLURM backend, follow up with waldur_site_diagnose_slurm_account -c /etc/waldur/waldur-site-agent-config.yaml for a deeper check that compares actual SLURM account state against what Waldur expects. Other backends don’t have an equivalent tool yet — for those, the sections below and the service logs are your best signal.

Common Issues

Service Won’t Start

1. Check configuration syntax:

   waldur_site_diagnostics -c /etc/waldur/waldur-site-agent-config.yaml
   

2. Check service logs:

   journalctl -u waldur-agent-order-process.service -n 50
   

Backend Connection Issues

1. Test backend connectivity:

   # For SLURM
   sacct --help
   sacctmgr --help
   
   # For MOAB (as root)
   mam-list-accounts
   

2. Check permissions and PATH

Agent Identity Registration Is Refused

Symptom — every cycle, for the same offering:

Registering a new identity for offering my-offering with name agent-<uuid>
Unable to register the identity agent-<uuid> for the offering my-offering:
Unexpected status code: 400 ... {"offering":["Object with uuid=<uuid> does not exist."]}
Continuing without agent telemetry.

The offering does exist. Waldur registers an agent identity only for the offering types listed under waldur_offering_uuid, and reports any other type as a missing object rather than as an unsupported one.

The agent keeps processing the offering: the identity, its service and its processors are telemetry, and the agent’s actual work — orders, membership sync, usage reporting — goes through the marketplace API and does not touch them. What you lose until the offering type is accepted:

• the agent does not appear in Waldur’s agent monitoring view, so there is no version, uptime, dependency or processor information for it;
• log shipping never starts. A shipper is keyed by the agent identity’s UUID, so without an identity there is nothing to attach a batch to — and the endpoint that receives the batches applies the same offering-type restriction, so it would refuse them anyway. Agent logs stay in the service’s own output (journalctl -u waldur-agent-*.service).

What to do:

1. Confirm the offering’s type, using the agent’s own token:

   curl -s -H "Authorization: Token your-token" \
     https://waldur.example.com/api/marketplace-provider-offerings/<offering-uuid>/ \
     | jq '{name, type, state}'
   

2. If it comes back 404, the UUID in waldur_offering_uuid is wrong or belongs to another Waldur instance — the offering name in the log line comes from your configuration file, not from the API, so a stale UUID looks identical to this symptom.
3. If the type is not one of the supported ones, either move the agent to an offering of a supported type, or ask your Waldur operator to widen the accepted types on the server.

Waldur API Issues

1. Test API connectivity:

   curl -H "Authorization: Token your-token" https://waldur.example.com/api/
   

2. Verify SSL certificates if using HTTPS

Debug Mode

Enable debug logging by setting log_level in the agent configuration file:

log_level: DEBUG

Performance Tuning

Adjust Processing Periods

Modify environment variables in systemd service files:

[Service]
# Reduce order processing frequency for high-load systems
Environment=WALDUR_SITE_AGENT_ORDER_PROCESS_PERIOD_MINUTES=10

# Increase reporting frequency for better accuracy
Environment=WALDUR_SITE_AGENT_REPORT_PERIOD_MINUTES=15

Resource Limits

Add resource limits to service files:

[Service]
MemoryLimit=512M
CPUQuota=50%

Backup and Recovery

Configuration Backup

# Backup configuration
sudo cp /etc/waldur/waldur-site-agent-config.yaml /etc/waldur/waldur-site-agent-config.yaml.backup

# Version control (optional)
sudo git init /etc/waldur
sudo git add waldur-site-agent-config.yaml
sudo git commit -m "Initial configuration"

Service State

The agent is stateless, but consider backing up:

• Configuration files
• Custom systemd service modifications
• Log files (if needed for auditing)

Scaling Considerations

Multiple Backend Support

The agent supports multiple offerings in a single configuration file. Each offering can use different backends:

offerings:
  - name: "SLURM Cluster A"
    order_processing_backend: "slurm"
    # ... SLURM-specific settings ...

  - name: "MOAB Cluster B"
    order_processing_backend: "moab"
    # ... MOAB-specific settings ...

High Availability

For HA deployment:

• Run agents on multiple nodes
• Use external load balancer for STOMP connections
• Implement cluster-level monitoring
• Consider using configuration management tools (Ansible, Puppet, etc.)

This site is open source. Improve this page.