Deployment and operations

Five options

The same images in every option: on-premise install, air-gapped, Kubernetes, a managed private instance we operate, and embedded in Aginies. The install path, the identity source and the network rules differ; the policy, the ledger and the console do not.

Requirements and install

OS

Linux, amd64 or arm64; every image is published for both (an H200 server installs the amd64 bundle, a DGX Spark the arm64 one)

Containers

Docker Engine 24+, Compose 2.20+; tools: bash, openssl, curl, tar, sha256sum; cosign optional

Hardware (Starter)

8 vCPU, 16 GB RAM, 40 GB free disk; the GPU tiers add 1 × 24 GB, 1 × 48 GB or 2 × 48 GB+ of GPU, mostly the organisation’s own vLLM endpoints

Network

A registry install reaches ghcr.io alone, during install and upgrade only; third-party images from the vendor’s mirror too. Ports 8050, 8051, 8053; 9051/9053 on loopback only

Clock

NTP; the licence is judged against the installation’s clock and a rolled-back clock is refused

sha256sum -c thulpar-kit-0.7.0.tar.gz.sha256
tar -xzf thulpar-kit-0.7.0.tar.gz && cd thulpar-kit-0.7.0
sudo ./thulpar install --method registry --registry-user <user> \
     --public-url https://thulpar.corp.internal --gpu endpoints
#   asks for the registry token (not echoed); --token-file for automation; --yes non-interactive
#   air-gap: tar -xf thulpar-0.7.0-airgap-amd64.tar && sudo ./thulpar-0.7.0-airgap-amd64/thulpar install --method airgap
thulpar status --show-token        # the one-time bootstrap token for the wizard

1. Pre-checks: Docker, Compose, tools, architecture, vCPU/RAM/disk, ports, clock sync, the NVIDIA runtime when needed.
2. Verification: every kit file against its checksum; the bundle against the manifest and the cosign signature (--require-signature makes it mandatory).
3. Files and configuration: the kit into /opt/thulpar; .env in every folder, every secret generated, a one-time bootstrap token.
4. Images: from the registry with the token (never written to disk) or docker load from the bundle; every service pinned by digest, never pulled again.
5. Models: the tier’s weights into the model volume, every file re-hashed against the catalogue; a mismatching import stops the install.
6. Start and health gate: Postgres, Valkey, migrations, api, gateway, detectors, console; every container healthy and the api answering with this version.
7. Summary: the console address and the bootstrap token for the wizard. A stopped install resumes with the same command.

First run and identity

No session is minted until the first administrator exists; every console address leads to the wizard. Ten steps, the progress kept on the server: leave and continue another day, in another browser. Every step can be left for later; finishing is written to the ledger as a config_change event.

1. Welcome: version, image digests, the installation id (ins_…).
2. Bootstrap token: single use; every token is burned after the first administrator.
3. First administrator: a local account + TOTP (the QR drawn in the browser) or linking through the IdP.
4. Sign-in: local only · enterprise SSO (OIDC) + break-glass (recommended) · SSO only; discovery and SSO test sign-in checks.
5. Licence: give the vendor the installation id, upload the signed file; covered tenants, term and features are shown.
6. Network and certificates: the corporate CA bundle, a TLS handshake test, the egress allowlist (empty by default).
7. Detectors: pick from the catalogue, the memory estimate against the container’s limit, model endpoints (your own vLLM).
8. First tenant: tenant, sector template (the -baslangic variant recommended on CPU), two-person activation, the first API key shown once.
9. Evidence and retention: a local volume or an S3-compatible store, the installation-wide minimum retention.
10. Summary: a green/amber/red health checklist, each row linking to the step that fixes it.

• Identity is an installation setting: OIDC (discovery, PKCE, role and tenant mapping, front- and back-channel logout) and local accounts are switched on and off in the console without a restart. You cannot lock yourself out: turning local accounts off needs a successful SSO test sign-in first.
• Local accounts: argon2id passwords (at least 12 characters), optional or mandatory TOTP, lockout, resets only by another SUPER_ADMIN; there is no reset by e-mail. Every break-glass use is written to the ledger.
• The console is an admin console: only SUPER_ADMIN and TENANT_ADMIN get in; approvers see only the inbox. Sessions carry a sliding idle timeout and an absolute lifetime.

Licence

The licence is a JSON document Ed25519-signed by the vendor: customer, covered tenants (or all), type, validity dates, features, tiers, the signing key id and the installation id. It is uploaded in the console or mounted as a file; the verification keys are compiled into the image and no environment variable can add one. No tenant is licensed by omission: enforce checks of a tenant the document does not name get 402.

FIXED_TERM

A fixed term; independent of the number of checks

PER_TRANSACTION

Enforced checks are counted; shadow, audit and async follow-ups are not. The count is chained into the ledger as a daily checkpoint; deleting rows can only raise it. The renewal report verifies offline.

Features

When the document lists features, gateway (/v1/*), mcp (/mcp/:server) and console (the control plane) follow it; a feature not listed answers 402 LICENSE_FEATURE. An empty list restricts nothing.

Installation binding

The document may name the installation id; a foreign one is refused. The id survives a restore from backup; a reinstall onto an empty database mints a new one.

Clock

The installation keeps a clock high-water mark that never lowers; when the clock falls behind it enforce checks get 402 CLOCK_ROLLBACK and the first detection is written to the ledger. Shadow never blocks a licensed tenant.

Renewal

A new document with the same licence id; uploading it retires the earlier one, and one whose start is ahead waits beside the running licence. The console warns 14 days before; no restart needed.

Upgrade, backup, restore, monitoring

thulpar upgrade 0.7.0 --kit thulpar-kit-0.7.0.tar.gz   # or --registry-user <user>; air-gap: the new bundle's own installer
thulpar rollback                                      # back to the version before the last upgrade
thulpar backup --out /backup/thulpar --passphrase-file /root/thulpar-backup.pass --keep 14
thulpar restore /backup/thulpar/thulpar-backup-0.7.0-20261014T021500Z.tar.enc --passphrase-file /root/thulpar-backup.pass
thulpar status                                        # version, containers, health, installation id, last backup; exit 1 when down
thulpar detectors-mtls on|off|renew|status            # optional internal mTLS api/gateway → detectors

• Upgrade: the new kit is verified and its installer takes over; the range’s upgrade notes are shown; gates that need an operator action stop it; a pre-upgrade backup; images and changed models; .env files kept; an ordered restart (migrations forward-only); the health gate and ledger verification. When the gate fails, thulpar rollback (or --auto-rollback).
• Backup: the whole database with pg_dump (ledger, settings, accounts, packs, keys, licence documents, approvals), generated evidence packs, .env files and added files; optionally AES-256 + HMAC with a passphrase; Valkey and the model bundle are not in it. Copy the backup off the host and keep the passphrase apart.
• Restore: in place or onto a fresh host (the same installation id, encryption and signing key; a bound licence stays valid); onto the same or a newer version; every ledger chain is recomputed hash by hash at the end.
• Monitoring: Prometheus /metrics on api 9051 and gateway 9053, loopback only (an optional token); decisions per stage/decision/rail, check and detector latency, shed requests, approval backlog, licence and certificate days, database size. Labels are bounded vocabularies; never a tenant id or content.
• The alert rules in the kit (tested with promtool) and the Grafana dashboard: target down, detector router down, detector unavailable, rail deadline misses and errors, shed requests, ledger chain broken, licence expiring, tenant unlicensed, certificate expiring, disk filling. OpenTelemetry traces only when an endpoint is given, content-free.
• Scheduled jobs in the api, in the installation’s time zone: the daily anchor, sample purge, the nightly chain verification, the licence usage checkpoint, scheduled evidence packs.

Sizing, Kubernetes, air-gap

The tier only decides which detectors are installed; the policy model is the same. Starter runs on one CPU machine and suits pilots, shadow mode and edge installs; the jailbreak and content-safety guards run asynchronously after the answer. Standard is the minimum tier of the sector templates for one production application; Recommended adds image moderation and the policy judge for multi-application organisations; Performance is for bank scale, high availability and multi-tenant installs.

Memory is the sum of the chosen detectors, and the console shows the estimate before applying a change. Where a model-backed rail must make its deadline, size the cores, the rail’s concurrency or its budget, or move to a GPU tier; then run the load probe with your own texts on your own host. Plan the disk for the events’ lifetime: events are never deleted, only samples are.

Kubernetes: the Helm chart ships with the restricted pod security standard, NetworkPolicies (egress closed; model download only when explicitly enabled), a PrometheusRule and the Grafana dashboard. The api and the gateway scale horizontally with state in Postgres and Valkey; the detector tier lands on its own (GPU) node pool. First run, backup and restore follow the Compose logic; the chart passes helm lint, kubeconform and a kind smoke test in CI.

Air-gap: one bundle per architecture with the kit, every image, the model weights and the licence texts; verified with sha256 and the cosign signature on a connected machine, the transfer media prepared by the vendor. Model download is off; models arrive as a tar and every file is re-hashed against the catalogue. An upgrade is a bundle too; evidence packs verify offline.

# connected machine: verify the release signature, then the checksum
cosign verify-blob --bundle thulpar-0.7.0-airgap-amd64.tar.sha256.sigstore.json \
  --certificate-identity "https://github.com/abrapartners/thulpar-guard/.github/workflows/release.yml@refs/tags/v0.7.0" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com thulpar-0.7.0-airgap-amd64.tar.sha256
sha256sum -c thulpar-0.7.0-airgap-amd64.tar.sha256

# models without the bundle: export on a connected install, import on the closed one
thulpar-admin models export --out /media/thulpar-models.tar
thulpar-admin models import /media/thulpar-models.tar

Production checklist

1. The host is sized for the tier; the disk is planned for the events’ lifetime; the install used the production defaults (no LAB MODE line).
2. The console, api and gateway behind a TLS-terminating reverse proxy; the public address https; only 8050/8051/8053 reachable from outside, 9051/9053 on loopback.
3. Postgres, Valkey and the detectors not published; the internal networks internal; the GPU adapter’s bridge held to the vLLM hosts by the host firewall.
4. The first run is done, the bootstrap token spent; OIDC where there is an IdP, local admins only as break-glass and with TOTP.
5. The secrets were generated, the .env files are 0600 and nowhere but in the backup; the signing key set, its public half given to the auditor through a separate channel.
6. The licence uploaded; every enabled tenant has one; the sector template fits the tier; no rail fails closed.
7. New rails in shadow first; closed postures once measured.
8. Prometheus scrapes the api and the gateway; the alert rules are loaded and reach a receiver; the dashboard is imported; the load probe was run with your own texts.
9. Retention set (the installation’s legal floor); monthly or quarterly evidence packs scheduled; the weekly full chain verification in cron.
10. A daily encrypted backup, a copy off the host, the passphrase apart; a restore tried and its duration measured.
11. The on-call team has the operations runbooks; the vendor’s security contact is known.