Deployment¶

Docker Compose (development)¶

docker compose up --build -d
./scripts/smoke_test.sh

Compose builds three images: db (PostgreSQL 18 + pg_sphere, and skycell when SKYCELL_REF is set — see Geometry backend; initialized from db/init/*.sql), tap-api and tap-executor (both installed with uv sync --frozen from the committed uv.lock).

Helm (Kubernetes)¶

A chart is provided under charts/egernia:

helm upgrade --install egernia charts/egernia \
  --namespace egernia --create-namespace \
  --set tapApi.baseUrl=https://tap.example.org/tap
helm test egernia -n egernia

Upgrading an install made before the rename

The project was called skao-tap, and every object the chart creates is named <release>-<component>. A release installed as skao-tap therefore cannot be renamed in place — helm upgrade egernia would create a second set of objects rather than adopt the first. Uninstall and reinstall:

helm uninstall skao-tap -n skao-tap     # keeps PVCs unless they are deleted
helm upgrade --install egernia charts/egernia -n egernia --create-namespace

The database and results PersistentVolumeClaims are named after the release as well (skao-tap-db, skao-tap-results), so data that has to survive must be moved: back up with the chart's own dump path (below), reinstall, and restore into egernia-db.

Key values (see values.yaml for the full list):

Value	Default	Description
`image.registry` / `image.tag`	`ghcr.io/ska-telescope/egernia` / `latest`	Where CI publishes the service images (an empty tag falls back to the chart appVersion)
`tapApi.replicas`	`1`	API replicas (stateless)
`tapApi.baseUrl`	in-cluster service URL	External base URL written into capabilities and result links
`tapExecutor.replicas`	`1`	Executor replicas; safe to scale out (jobs claimed with `SKIP LOCKED`)
`postgresql.enabled`	`true`	Deploy the in-chart PostgreSQL + pg_sphere; disable to use an external DB via `externalDatabase.url`
`results.storageClass` / `results.size`	`""` / `1Gi`	Shared results volume
`ingress.enabled`	`false`	Optional ingress for the API
`scheduling.spreadReplicas`	`true`	Soft anti-affinity + zone spread for multi-replica services
`podDisruptionBudget.enabled`	`true`	PDBs for components with >1 replica
`verticalAutoscaling.enabled`	`false`	VPA per service (recommendation mode first)
`postgresql.tuning`	`{}`	postgresql.conf overrides as `-c` server arguments
`backup.enabled`	`false`	Nightly `pg_dump` CronJob to a dedicated PVC
`metrics.scrapeAnnotations`	`true`	Annotate pods so an existing Prometheus discovers them (guide)
`tracing.otlpEndpoint`	`""`	Export traces; empty means no collector and no instrumentation
`config.executorMetricsPort`	`9100`	Port the executor serves metrics on — it has no API of its own
`config.dbPoolMax`	`8`	Database connections per process — the real limit on concurrent queries
`config.dbPoolTimeoutSeconds`	`5`	How long a request waits for one before answering `503`
`tapApi.workers`	`1`	Uvicorn processes per pod; about half a request's CPU holds the GIL, so this is what lets a pod use more than one core. Set it to the pod's CPU limit — measured
`auth.enabled`	`false`	Require verified tokens and gate the mutating metadata endpoints (guide)
`auth.requireToken`	`true`	With `auth.enabled`, every request needs a verified token — discovery and the health check aside
`auth.anonymousQueries`	`false`	Let token-less callers read metadata through `/tap/sync` and the `/tap/async` job; what standard VO clients need
`auth.gatedOperations`	`[]`	Which operations need an authorisation decision; empty means metadata mutation only
`auth.plugin`	`iam-groups`	Authorisation plugin: local IAM groups, or the SRCNet Permissions API
`auth.iam.issuer`	`""`	Required when `auth.enabled`; tokens are verified against its JWKS
`auth.iam.audience`	`""`	Required when `auth.enabled`; guards against cross-service token replay
`auth.roles`	`{}`	Per-operation groups/scopes for `iam-groups`; required, and an empty rule denies

Serving concurrent queries¶

About half of a request's CPU is pure-Python work that holds the GIL — ADQL translation (3.19 ms of a ~10.5 ms request) and the result writers — so one worker process cannot use more than one core however many the pod has. Measured (run 20260825T180219Z-f8b21fc4, D1, normal mix, each figure a saturated closed-loop ceiling on one pod):

workers	pod CPU limit	throughput	pod CPU used	pod memory peak
1	2	99.1 req/s	1.04–1.08 cores	138 MiB
2	2	187.6 req/s	1.94 cores	302 MiB
4	2	179.2 req/s	2.00 cores	562 MiB
4	4	338.1 req/s	4.00 cores	581 MiB

Set tapApi.workers to the pod's CPU limit — measured, not just advised: one worker consumes almost exactly one core, two workers on a 2-core pod are 1.89x one, and a third and fourth worker on the same 2 cores lose ~4% to context switching. The last row shows the ceiling is the pod, not uvicorn: the same four workers given 4 cores reach 3.4x, pinned at the new limit and still CPU-bound.

The two axes buy the same throughput per process — 4 processes as 2 workers x 2 replicas measured 335.8 req/s against 342.9 as 4 single-worker replicas, and at 8 processes 583.0 against 581.6 — so choose by what each costs. A worker costs no pod, but memory and connections — and the two are one mechanism, because every worker holds its own connection pool and each pooled connection carries ~2.5 MiB of client-side state. Size the pod's memory limit by arithmetic rather than by watching:

pod memory floor ≈ ~140 MiB x workers + workers x config.dbPoolMax x ~2.5 MiB

(measured: four workers with pools exercised settle toward ~600 MiB against the default 1 Gi). The same workers x config.dbPoolMax product is the pod's connection ceiling — the arithmetic in When every connection is busy below — so raising workers multiplies both, and config.dbPoolMax is what makes a worker count fit both limits. Replicas cost pods and spread across nodes; workers cannot.

Shedding overload with refusals, not resets¶

Past its capacity the service currently sheds load with connection resets — the benchmark suite observed clients reading ECONNRESET under sustained overload, and a reset tells a client nothing: it cannot distinguish an overloaded service from a broken network, so it can only retry blind, which adds load. The leading suspect is the listen socket's accept queue overflowing before the application ever sees the connection (unconfirmed — the rate at which resets begin has not been established).

Two knobs shape this behaviour:

tapApi:
  backlog: 2048          # accept-queue size (uvicorn --backlog)
  limitConcurrency: 64   # connections per worker before answering 503

backlog sizes the kernel's accept queue; raising it absorbs sharper arrival bursts but adds queueing, not capacity. limitConcurrency is the application-level counterpart: past that many concurrent connections per worker process, uvicorn answers 503 immediately — a refusal a client can back off from. The chart defaults it to 64 per worker (0 disables the limit) because the right value depends on what one worker can actually serve; when set, put it above the worker's normal concurrent load and below where resets were observed.

/health/live answers whether the process is turning and touches nothing else; /health/ready answers whether this pod should be sent traffic, and treats a full connection pool as ready — a busy service is not a broken one, and taking the pod out of the Service would push its share of the load onto pods in exactly the same state. Only an unreachable database makes it unready.

Do not point probes at /tap/availability

It is a VOSI resource that reports on the database, so it queues for a pooled connection. With the probe default of timeoutSeconds: 1 it cannot answer under load, and the kubelet then restarts a service that is busy rather than broken — measured, twice, at an offered rate inside the service's own capacity. Both probes used to point there.

Both paths are exempt from authentication: a kubelet has no token and cannot obtain one, so gating them would make enabling auth an outage.

When every connection is busy¶

The pool bounds how many queries a process runs at once, and it is smaller than the number of connections uvicorn accepts, so concurrency above it has to queue. Measured locally with one worker and the default pool of 8:

concurrent clients	throughput	p95
8	66 req/s	169 ms
12	66 req/s	238 ms
16	66 req/s	318 ms

Throughput holds flat and latency grows: requests wait their turn, which is what a queue should look like. Add tapApi.workers to raise the ceiling itself — see the measured worker table in Serving concurrent queries.

If the wait for a connection exceeds config.dbPoolTimeoutSeconds the request answers 503 with Retry-After rather than holding the caller. Five seconds is deliberately short: a synchronous query may run for up to config.syncTimeoutSeconds, so a queue of slow queries could otherwise leave a client waiting a minute for a connection that a fast, retryable refusal describes better.

Mind the connections. Each worker opens its own pool, so a pod holds up to tapApi.workers × config.dbPoolMax connections, and the deployment holds that multiplied by tapApi.replicas (or by horizontalAutoscaling.tapApi.maxReplicas when an autoscaler is in charge — see Autoscaling, which checks this sum for you). With the defaults that is 1 × 8 = 8 per pod. Keep the total under the server's max_connections — postgresql.tuning.max_connections raises it for the in-chart database, and a managed server has its own limit.

Results volume access mode

The results volume is shared between the API and the executor. With more than one node you need a ReadWriteMany-capable storage class (or pin both deployments to one node); the chart defaults to ReadWriteOnce which is only safe for single-node/dev clusters.

The in-chart PostgreSQL mounts the same db/init SQL (copied into the chart at files/db-init/; CI verifies both copies stay in sync).

Scaling and resilience¶

Both services are horizontally scalable: tap-api is stateless, and the executors claim jobs with FOR UPDATE SKIP LOCKED, so extra replicas cooperate instead of colliding. Give each service more than one replica and switch the results volume to ReadWriteMany:

helm upgrade egernia charts/egernia \
  --set tapApi.replicas=3 \
  --set tapExecutor.replicas=2 \
  --set "results.accessModes={ReadWriteMany}" \
  --set results.storageClass=<an RWX-capable class>

With scheduling.spreadReplicas (on by default) the chart adds preferred pod anti-affinity across nodes and a zone topology-spread constraint to each service — soft constraints (ScheduleAnyway), so single-node clusters such as kind schedule exactly as before. Per-component affinity/topologySpreadConstraints/nodeSelector/tolerations values override the defaults wholesale. Components with more than one replica also get a PodDisruptionBudget (maxUnavailable: 1), keeping the service up through node drains and cluster upgrades.

Automatic scaling¶

Those replica counts can be handed to an autoscaler instead: tap-api on CPU, tap-executor on the depth of the job queue. Both are off by default and have their own page — Autoscaling — including the combinations the chart refuses, among them the connection ceiling that a maximum replica count makes reachable.

Vertical scaling¶

verticalAutoscaling.enabled=true creates a VerticalPodAutoscaler per service (requires the VPA CRDs in the cluster). It starts in recommendation mode — read the suggestions with kubectl describe vpa — and moves to live resizing with verticalAutoscaling.updateMode=Auto once the minAllowed/maxAllowed bounds are trusted. verticalAutoscaling.controlledResources narrows what it sizes — set it to ["memory"] to run a live VPA next to a CPU-based HorizontalPodAutoscaler, since two controllers driving the same resource is what does not work (see Autoscaling).

The in-chart PostgreSQL is sized through postgresql.resources plus postgresql.tuning, a map rendered as -c key=value server arguments:

postgresql:
  resources:
    limits:
      memory: 2Gi
  tuning:
    max_connections: 100
    shared_buffers: 512MB
    effective_cache_size: 1536MB
    work_mem: 32MB          # per sort/hash node — large ADQL joins
    maintenance_work_mem: 128MB

Watch connection counts (each API/executor replica holds a pool), shared_buffers hit rates, and temp-file spills from large ADQL sorts when right-sizing.

Highly available PostgreSQL¶

The in-chart StatefulSet is a single instance — fine for development and small sites, not for HA. For automated failover run PostgreSQL under a streaming-replication operator such as CloudNativePG (or Zalando's postgres-operator), load db/init/*.sql into it once, and point the chart at it:

helm upgrade egernia charts/egernia \
  --set postgresql.enabled=false \
  --set externalDatabase.url=postgresql://tap:…@tap-db-rw:5432/tap

The services only need the one DSN, so failover handled by the operator is transparent to them. An operator-managed database also brings WAL archiving and point-in-time recovery (below).

Read replicas for the query path¶

A TAP query is a read: it runs under TAP_QUERY_ROLE (tap_reader), which owns no write privilege. So once PostgreSQL is replicated, query execution can leave the primary — and then the database stops being one instance's parallel-worker budget and buffer cache. TAP_QUERY_DATABASE_URL is the one setting:

helm upgrade egernia charts/egernia \
  --set postgresql.enabled=false \
  --set externalDatabase.url=postgresql://tap:…@tap-db-rw:5432/tap \
  --set externalDatabase.queryUrl=postgresql://tap:…@tap-db-ro:5432/tap

Unset — the default — it is TAP_DATABASE_URL: one pool, one server, nothing about an existing deployment changes. Set, each API and executor process opens a second pool of dbPoolMax connections to that URL and runs user queries there. An operator's read-only Service (CloudNativePG's -ro) spreads the connections for you; without one, libpq does it, given more than one host:

postgresql://tap:…@sb1,sb2,sb3:5432/tap?target_session_attrs=read-only&load_balance_hosts=random

target_session_attrs=read-only refuses a host that turns out to be the primary and load_balance_hosts=random (libpq 16+; the images ship 18) picks the starting host per connection instead of always the first, so a pool spreads over the standbys. A standby that is down is skipped: libpq tries the next host. All of them down is a 503, like a full pool.

What does not move. Everything else keeps the primary: the UWS job table (claims, leases, phases — a claim is a write), ingest, the schema bootstrap, TAP_SCHEMA reads that must see a table published a moment ago, and any query carrying a TAP_UPLOAD, because its temp tables are a write a standby refuses. The split is per call site (egernia_core.db.connection's replica_ok), not per service, so the executor runs its query on a standby while booking the job on the primary.

Consistency. TAP reads become eventually consistent with ingest: a product ingested now appears in query results one replication lag later (milliseconds on a healthy link, unbounded if a standby falls behind). Watch pg_stat_replication.replay_lag on the primary, and do not set this where a client must read its own write through TAP — the JSON metadata API (/api/v1), which is that read-your-writes path, stays on the primary anyway.

Sizing. Each standby is a PostgreSQL to size to its own container by the rule in PostgreSQL performance — shared_buffers a quarter of its memory, effective_cache_size three quarters, and max_parallel_workers at least twice the connections that reach it, which is the query pools divided by the number of standbys. Give the standbys hot_standby_feedback=on: without it the primary can vacuum away rows a long-running TAP scan still needs, and the query dies with "canceling statement due to conflict with recovery". Connections are counted per server, so the ceiling arithmetic in Autoscaling splits too.

Aborting a job takes up to a second longer. The API's immediate pg_cancel_backend() is issued on the primary and finds nothing to cancel there, so the cancel comes from the executor's abort watchdog, which signals the server that is actually running the statement (it pins the URL to the host its connection reached). An ABORT still ends the job; the statement stops on the watchdog's next half-second poll rather than instantly.

tap_db_pool_wait_seconds and tap_db_connections_in_use carry a pool label — primary or query — so the two paths are visible apart; with no replica URL both roles report against the single pool.

Backup and restore¶

Two things hold state: the PostgreSQL database (UWS jobs, TAP_SCHEMA, all ingested metadata) and the results volume (query outputs).

Database¶

backup.enabled=true adds a CronJob that writes pg_dump --format=custom archives of the whole database to a dedicated PVC and prunes them after backup.retentionDays:

helm upgrade egernia charts/egernia \
  --set backup.enabled=true \
  --set backup.schedule="0 2 * * *" \
  --set backup.retentionDays=7 \
  --set backup.storage=10Gi

It dumps through TAP_DATABASE_URL, so it covers the in-chart PostgreSQL and an external database alike. Keep retentionDays at or above config.jobRetentionSeconds (default 7 days), or restored deployments will be missing jobs their clients still consider alive.

To restore, stop the services so nothing writes during the restore, run pg_restore from a pod with the backup PVC mounted, then scale back up:

kubectl scale deploy egernia-tap-api egernia-tap-executor --replicas=0

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: pg-restore
spec:
  restartPolicy: Never
  containers:
    - name: pg-restore
      image: <the tap-db image>
      command: ["sh", "-ec"]
      args:
        # or name a specific archive instead of the most recent one
        - |
          archive=$(ls -t /backups/egernia-*.dump | head -1)
          echo "restoring ${archive}"
          pg_restore --clean --if-exists -d "$TAP_DATABASE_URL" "${archive}"
      env:
        - name: TAP_DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: egernia-db      # <release>-db
              key: TAP_DATABASE_URL
      volumeMounts:
        - name: backups
          mountPath: /backups
  volumes:
    - name: backups
      persistentVolumeClaim:
        claimName: egernia-db-backups   # <release>-db-backups
YAML

kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/pg-restore --timeout=10m
kubectl logs pg-restore && kubectl delete pod pg-restore
kubectl scale deploy egernia-tap-api --replicas=1
kubectl scale deploy egernia-tap-executor --replicas=1

Exercise the restore path regularly — an unrestored backup is a hope, not a strategy. pg_dump gives consistent snapshots but no point-in-time recovery; for PITR (base backups plus WAL archiving to object storage) run the database under CloudNativePG and use its Backup/ScheduledBackup resources, in which case backup.enabled here is redundant.

Results volume¶

Job results are re-derivable (any job can be re-run) but re-deriving them costs compute, and result URLs are handed to clients. Snapshot the results PVC with the storage class's VolumeSnapshot support — or back it up to object storage with a tool such as Velero — on a cadence and retention aligned with config.jobRetentionSeconds: results older than the retention window are destroyed anyway, so keeping their backups any longer buys nothing. A restored results volume plus a database restored from the same window keeps job documents and their result files consistent; results missing for a restored job simply 404 and the job can be re-run.

Container hardening¶

All three images run as non-root users (tap, uid 10001 for the services; postgres, uid 999 for the database) and work with a read-only root filesystem. The Helm chart sets matching pod/container security contexts (runAsNonRoot, dropped capabilities, seccomp RuntimeDefault, readOnlyRootFilesystem) with emptyDir mounts for /tmp and, for PostgreSQL, /var/run/postgresql. The Trivy job in CI enforces this — it fails on any CRITICAL/HIGH vulnerability, leaked secret, or misconfiguration.

Upgrading an existing Compose deployment

Volumes created by older root-based images keep root ownership; if the database or executor fails with permission errors after upgrading, recreate the volumes once with docker compose down -v.

PostgreSQL 18

The database image moved from PostgreSQL 16 to 18. A data directory written by 16 cannot be started by 18, and the postgres:18 image also changed its default PGDATA (/var/lib/postgresql/<major>/docker, with /var/lib/postgresql as the declared volume) — Compose and the chart both pin PGDATA back under their mount. There is no production data to migrate, so recreate the volume: docker compose down -v for Compose, or delete the data-<release>-postgres-0 PVC before upgrading the chart. Should a future major bump need to preserve data, take a pg_dump --format=custom archive beforehand and pg_restore it into the new cluster.

Geometry backend: pg_sphere or skycell¶

ADQL geometry (POINT, CIRCLE, CONTAINS, DISTANCE, …) is translated to SQL for one PostgreSQL extension, chosen by TAP_GEOMETRY_BACKEND:

pgsphere (the default): pg_sphere's spoint/scircle/spoly and its operators, with GiST indexes.
skycell: skycell, whose functions carry ADQL's own names. Positions are indexed by a B-tree on their HEALPix cell that also holds the position ((skycell_ang2cell(s_ra, s_dec)) INCLUDE (s_ra, s_dec)), and skycell plans every ADQL spelling of a cone or a cross-match through it: CONTAINS(POINT, CIRCLE) with the point or the circle on either side, and DISTANCE(…) < r. The included position lets skycell test the cone on the index entry and read from the table only the rows inside it. Footprints are stored as skycell skyregion values with a GiST index.

The choice reaches further than the translation. The s_region_geom footprint column is a pg_sphere spoly or a skycell skyregion, and the cone-search index on (s_ra, s_dec) is a GiST on spoint(…) or the skycell cell B-tree. So the database needs the chosen extension installed before the services start, and switching backends on a deployment that has ingested metadata means rebuilding the generated tables. Startup refuses to go ahead until that is done: a footprint column of the other backend's type stops the bootstrap with an error naming it, rather than leaving ingest and geometry queries to fail one request at a time. ivoa.obscore rebuilds itself when its index definitions change.

The db image installs pg_sphere always and skycell when built with a skycell release tag, commit or branch. Use skycell 0.31 or later: from 0.31 the cone scan tests cones on the position index's included s_ra/s_dec, which an earlier build ignores (it works, without that speed-up).

# a release tag (or a commit): the build is reproducible, and the commit is
# recorded in the image at /usr/share/skycell/COMMIT
SKYCELL_REF=v0.31 TAP_GEOMETRY_BACKEND=skycell docker compose up --build -d

The extension's version is then 0.31 in pg_available_extensions.

With Helm, set config.geometryBackend: skycell and use a tap-db image built with SKYCELL_REF.

What differs under skycell:

An ADQL BOX is the polygon on its corners, with great-circle edges, as ADQL 2.0 meant a BOX (a shortcut for a POLYGON); pg_sphere keeps a box's Dec edges at constant Dec. The two agree for small boxes, apart from slivers of a couple of arcseconds along the Dec edges, but not for wide ones: a great circle bows towards the pole, so the top edge of a box 120 degrees wide from Dec 0 to 30 rises to Dec 49 at its middle. A box past one pole ends at the pole; one with no area, 180 degrees or more wide, or past both poles is refused. ADQL 2.1 deprecates BOX; prefer POLYGON.
skycell's polygons must be convex. A well-formed but non-convex s_region POLYGON is stored with a NULL s_region_geom (and a warning in the log) rather than refused, so the document still ingests but that footprint answers no geometry query. A malformed one (fewer than three distinct vertices, a vertex repeated in the ring, or no area) is refused, as invalid STC-S is; a vertex repeated in a row is dropped. A CIRCLE footprint is stored as the exact circle rather than a 32-gon, and a POSITION as a circle of 50 milliarcseconds.
COORD1, COORD2 and CENTROID work, and the capabilities document lists them.
skycell defines INTERSECTS between a point and a region only that way round. A POINT(...) written second is moved first in translation (ADQL's INTERSECTS is symmetric), but a point-typed column (ADQL 2.1) is not recognisable as one there, so write it as the first argument.

Pre-deploy schema bootstrap¶

By default every service replica ensures the database schema at startup — idempotent, advisory-locked DDL, which is what a rolling upgrade relies on: whichever new pod touches the database first migrates it forward. That means the runtime database credentials own schema changes.

To move schema ownership out of the runtime, run the bootstrap once before deploying (with credentials that may run DDL) and turn the startup DDL off:

# from a checkout (or in a container: /srv/.venv/bin/python)
TAP_DATABASE_URL=postgresql://owner:...@db/tap \
  uv run python -m egernia_core.bootstrap

helm upgrade egernia charts/egernia --set config.schemaBootstrapOnStartup=false

With schemaBootstrapOnStartup=false the services only verify the schema at startup (a read-only column-level check) and fail fast, naming the bootstrap command, when it is missing or outdated — so re-run the bootstrap as part of every deploy that upgrades the images or the model libraries.

Configuration¶

All services read environment variables (see egernia_core/config.py):

Variable	Default	Description
`TAP_DATABASE_URL`	`postgresql://tap:tap@localhost:5432/tap`	PostgreSQL DSN
`TAP_QUERY_DATABASE_URL`	—	DSN user queries execute on; empty means `TAP_DATABASE_URL`. See Read replicas for the query path
`TAP_BASE_URL`	`http://localhost:8080/tap`	Public base URL (capabilities, result links)
`TAP_RESULTS_DIR`	`/results`	Shared results directory
`TAP_QUERY_ROLE`	`tap_reader`	Read-only role used for user queries
`TAP_DEFAULT_MAXREC` / `TAP_HARD_MAXREC`	`10000` / `1000000`	Row limits
`TAP_SYNC_TIMEOUT`	`30`	Sync query timeout (s)
`TAP_SYNC_MAX_BYTES`	`67108864`	Maximum serialized synchronous result size; use async for larger results
`TAP_GEOMETRY_BACKEND`	`pgsphere`	Which extension ADQL geometry is translated for: `pgsphere` or `skycell`. See Geometry backend
`TAP_TRANSLATION_CACHE_SIZE`	`512`	Entries in the per-worker ADQL translation memo; `0` disables it. Read once at start-up. Its hit rate is `tap_adql_translation_cache_hits_total` over the sum of hits and misses
`TAP_ASYNC_EXEC_DURATION`	`600`	Default async `executionDuration` (s)
`TAP_JOB_RETENTION`	`604800`	Default job lifetime before destruction (s)
`TAP_MODEL_PLUGINS`	`all`	Metadata domains to activate (`all` or a comma-separated subset)
`TAP_SCHEMA_BOOTSTRAP_ON_STARTUP`	`true`	Whether services run schema DDL at startup; `false` makes them only verify a pre-deploy bootstrap ran
`TAP_AUTH_ENABLED`	`false`	Enable authentication/authorisation (guide)
`TAP_AUTH_REQUIRE_TOKEN`	`true`	Require a verified token on every request bar discovery and the health check
`TAP_AUTH_ANONYMOUS_QUERIES`	`false`	Allow token-less reads through `/tap/sync` and `/tap/async`
`TAP_AUTH_GATED_OPERATIONS`	—	Operations needing an authorisation decision; empty means metadata mutation only, `none` means nothing
`TAP_AUTH_PLUGIN`	`iam-groups`	Which authorisation plugin decides (`iam-groups`, `permissions-api`, or your own)
`TAP_IAM_ISSUER`	—	Token issuer; tokens are always verified against its JWKS
`TAP_IAM_AUDIENCE`	—	Expected token audience; required unless `TAP_IAM_ALLOW_ANY_AUDIENCE=true`
`TAP_IAM_GROUP_CLAIMS`	`groups,wlcg.groups`	Claims read as IAM group membership
`TAP_AUTH_ROLES`	`{}`	`iam-groups` policy, as JSON keyed by operation
`TAP_PERMISSIONS_API_URL`	—	SKA SRC Permissions API base URL (`permissions-api` plugin)
`TAP_LOG_LEVEL`	`INFO`	Level for the services' own records (bootstrap, legacy-table warnings, deletion audit trail); uvicorn's access log is separate