Keda — 20260824T102832Z¶
Run 20260824T102832Z-29507cbb-keda · commit 5b6bc969ec59 · 2026-08-24 12:45 UTC
This run is marked invalid
keda-K6: load_generator_offered_the_rate
The numbers below are kept as evidence of what went wrong. They do not describe the service under the conditions intended.
Headline¶
| finding | value | evidence |
|---|---|---|
| peak single-replica throughput | 177.2 | conc-D2-c4-r0 at 4 clients, p95 39 ms |
| sustainable single-replica capacity (C1) | 177.2 | highest successful rps over valid measurements with p95 within the 2.0s SLO and errors under 1% |
What was measured¶
| dataset | size | ObsCore rows | total rows | index/table | generation |
|---|---|---|---|---|---|
| D2 | 25.37 GiB | 7,399,024 | 106,348,439 | 0.63 | 20 s |
Sizes are pg_database_size() with indexes — not row counts times an
assumed row width. One database is grown through every target, so each
tier is a genuine prefix of the next.
Throughput and latency against concurrency¶
| dataset | clients | reps | requests/s | p95 (ms) | errors | bottleneck |
|---|---|---|---|---|---|---|
| D2 | 1 | 1 | 78.4 | 34 | 0.00% | DATABASE_IO_BOUND |
| D2 | 2 | 1 | 138.1 | 28 | 0.00% | DATABASE_IO_BOUND |
| D2 | 4 | 1 | 177.2 | 39 | 0.00% | TAP_CPU_BOUND |
Intervals are 95% Student-t across repetitions. The sweep stops when two saturation signals agree, so the last row of a dataset is where that dataset's ceiling was found rather than where the ladder ran out.
Autoscaling¶
| scenario | profile | detect (s) | HPA (s) | provision (s) | routing (s) | total (s) | recovery (s) | peak | events | reversals |
|---|---|---|---|---|---|---|---|---|---|---|
| K1 | idle to 0.5*C1 | — | — | — | — | — | 26.7 | 1 | 0 | 0 |
| K2 | 0.5C1 step to 3.5C1 | 98.0 | 18.6 | 1.0 | 129.1 | 246.1 | — | 3 | 2 | 0 |
| K3 | 0 to 6*C1 spike | 106.0 | 148.8 | 0.0 | 242.5 | 496.9 | — | 3 | 3 | 2 |
| K4 | 0.5C1 to 6C1 gradual 10-minute ramp | 250.0 | 20.7 | 1.0 | 82.5 | 352.1 | 25.7 | 3 | 3 | 1 |
| K5 | alternating 0.5C1 and 4C1 every 2 minutes | 340.0 | — | 0.0 | 121.4 | 476.3 | 215.9 | 3 | 3 | 1 |
| K6 ⚠ | sustained 6*C1 for 15 minutes | 112.0 | 141.0 | 0.0 | — | — | — | 3 | 4 | 2 |
| K7 | 4C1 to 0.2C1 recovery | 220.0 | 269.6 | — | — | — | 253.9 | 3 | 3 | 1 |
Stages: detect is the scaler's metric crossing its threshold, HPA the replica request changing, provision Pod creation to Ready, routing Ready to serving traffic, recovery the load change to p95 back inside the SLO. A dash means the stage could not be established from the evidence — recorded rather than estimated. A ⚠ on the scenario means one of its validity guards failed, so its timings describe conditions other than the ones intended.
Where the limit was¶
| classification | measurements | meaning |
|---|---|---|
DATABASE_IO_BOUND |
6 | The working set does not fit in shared_buffers plus page cache: the database is fetching from disk. This is the class that appears as the dataset grows and is the reason the size sweep exists. |
KEDA_SCALE_LAG |
5 | Capacity existed but arrived late: the errors and latency in this window are the cost of the scaling delay rather than of a resource ceiling. The stage breakdown says which part to attack. |
EXECUTOR_CPU_BOUND |
4 | The executors are at their ceiling. One executor runs one query at a time, so a pod cannot exceed one core whatever its cgroup allows, and the only relief is more pods. Alongside KEDA_SCALE_LAG this is the whole story of an autoscaling shortfall: the pods that existed were full and more were never asked for. |
TAP_CPU_BOUND |
1 | The API's own CPU is the constraint: it sat at its ceiling, or was CFS-throttled, for a material part of the window. The ceiling compared against is min(pod CPU limit, workers), not the pod limit: ADQL translation is pure-Python and holds the GIL, so one worker cannot exceed one core whatever the cgroup allows. Relieved by more processes (tapApi.workers) or more pods, never by a bigger limit on one worker. |
Graphs¶
Throughput against offered concurrency¶
Latency percentiles against concurrency¶
Errors against concurrency¶
Buffer cache hit ratio against database size¶
API CPU against throughput¶
PostgreSQL CPU against throughput¶
PostgreSQL read I/O against throughput¶
Throughput by query class¶
Latency by query class¶
Latency against response size¶
Autoscaling timeline — K1¶
Autoscaling timeline — K2¶
Autoscaling timeline — K3¶
Autoscaling timeline — K4¶
Autoscaling timeline — K5¶
Autoscaling timeline — K6¶
Autoscaling timeline — K7¶
Environment¶
| property | value |
|---|---|
| host | macOS-26.6.2-arm64-arm-64bit-Mach-O (14 CPUs) |
| Kubernetes | v1.36.1 |
| node capacity | 14 CPU, 16354760Ki |
| KEDA | ghcr.io/kedacore/keda:2.18.1 |
| PostgreSQL | PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit |
| extensions | pg_sphere 1.5.2, pg_stat_statements 1.12, plpgsql 1.0 |
| seed | 20260823 |
| corpus sha256 | c02caf7c44edae77 |
| chart values sha256 | b6e9c1b32e19c965 |
Download the per-measurement CSV · environment.json · dataset.json
Raw per-request samples and Prometheus series stay with the run that
produced them, under benchmarks/tap-performance/results/20260824T102832Z-29507cbb-keda/:
Parquet for every request and every metric, PostgreSQL statistics
before and after, EXPLAIN plans, and the exact ScaledObject and HPA
YAML the measurement ran against.