Result formats

Six writers produce the same rows: VOTable (the TAP default), CSV, TSV, JSON, Parquet and an Arrow IPC stream. They are not equally cheap, and for a result of ten thousand wide rows the difference between the cheapest and the most expensive is most of the response time. This page says which to ask for and why, with the measurement behind it.

Ask for one with RESPONSEFORMAT (or FORMAT) on /sync, or at submission time on /async:

curl -sG "$TAP/sync" \
  --data-urlencode "LANG=ADQL" \
  --data-urlencode "QUERY=SELECT TOP 10000 * FROM ivoa.obscore" \
  --data-urlencode "RESPONSEFORMAT=parquet" -o result.parquet

Which one to ask for

Interactive queries and anything a VO client reads: VOTable. It is the default, every VO tool understands it, and it carries the units, UCDs and descriptions from TAP_SCHEMA. It costs about a third more per row than JSON and produces the largest body of the six.

Bulk transfer — tens of thousands of rows and up: Parquet, or Arrow. Both are dramatically cheaper to produce per row than any text format, and Parquet is dramatically smaller on the wire as well: the columns of an ObsCore result compress extremely well, because a column of collection names or of calib_level values is thousands of repetitions of a handful of distinct values. Arrow is the cheapest of all to produce and streams uncompressed, so it is the right choice when the bytes are going across a fast link into a dataframe; Parquet is the right choice when they are going over a slow one or onto disk.

Both stream in record batches and both carry the column units, UCDs and descriptions as field metadata. Parquet additionally carries the DALI QUERY_STATUS in its file metadata, so an overflowed result says so; the Arrow IPC stream format has no end-of-stream metadata slot, so it cannot.

Feeding a script or a spreadsheet: CSV or TSV. Convenient, understood everywhere, and the most expensive of the six per row — the cost is CPython's csv writer, which has to consider quoting every field it writes. Fine for a thousand rows; the wrong tool for a million.

Feeding a web page: JSON. The cheapest text format, and it carries the column metadata in a metadata array beside the data array.

What it costs

Measured with the since-removed make benchmark-serialize harness, which ran the writers in process on the Q11 projection — eleven ObsCore columns, five doubles, five wide text fields, one small integer — with no database, no HTTP and no cluster in the way, so a per-row cost is a per-row cost. 10,000 rows, best of 15 repetitions, 30-core host, Python 3.14:

format	µs/row	bytes/row	relative cost
arrow	0.93	236	1.0×
parquet	1.78	52	1.9×
json	5.12	296	5.5×
votable	7.04	371	7.6×
tsv	9.13	273	9.8×
csv	9.19	273	9.9×

Parquet is 1.9× the CPU of Arrow and one fifth of its bytes, which is the trade the two of them represent. Everything textual is between five and ten times the cheapest, and that is the cost of rendering numbers as decimal digits — unavoidable in a text format, and the reason a bulk transfer should not be one.

That table prices the Python writers, which is still what serves JSON, Parquet and Arrow. It is no longer what serves csv, tsv or votable: see below.

csv, tsv and votable rows are written by PostgreSQL

DSV responses are produced by COPY (<query>) TO STDOUT WITH (FORMAT csv) and handed to the socket as the server wrote them, so for those two formats neither the Python writer nor the row tuples it consumed exist. What the API pays drops from 28.40 µs/row to 0.134 µs/row — the cost of counting the delivered rows for the DALI overflow status and batching them into 64 KiB chunks.

The work does not vanish, it moves: PostgreSQL renders the bytes, and it renders them through a projection that reproduces CPython's output exactly, because a faster response in different bytes is a broken response. On one host with both sides charged, csv goes from 28.40 to 14.17 µs/row — 2.00×, and 3.02× on data where floats are round numbers. In a deployment the two sides are different pods, which is the point: the API's CPU is the constrained one.

Two consequences worth knowing:

• The table above is the fallback. A result the projection cannot promise to reproduce byte for byte — a column type with no rendering, bytea and pg_sphere's spoly among them, a repeated column name, a single-column result — is served by the Python writer instead, at the 9.19 µs/row above. tap_copy_dsv_fallbacks_total counts those with the reason, and tap_copy_dsv_results_total counts the ones served by the server.
• TAP_COPY_DSV=false puts every DSV response back on the Python writer. Nothing else changes: the bytes are the same either way, which is the property tests/component/test_copy_dsv_differential.py exists to hold.

VOTable takes the same route. It is TAP's default format, and its writer was the most expensive of the text formats: 4.79 ms per 1,000 rows of the microbenchmark corpus, against 3.75 ms for csv. The projection folds every row into one text column holding <TR><TD>…</TD></TR> — cells rendered by the same per-type expressions as DSV, &/</> escaped in the writer's order, NULL as <TD/> — and COPY … TO STDOUT WITH (FORMAT text) streams it. The envelope (FIELDs before the rows, the OVERFLOW INFO after them) stays in Python, one call per response. Text format is the one COPY output that never quotes, but it escapes backslash, tab, CR and LF; each received chunk is un-escaped by a C-level scan that does nothing when the chunk has no backslash, which ObsCore text never does. Receiving a VOTable body this way costs the API 0.20 ms per 1,000 rows instead of 4.79 (24×); on one host with both sides charged, 20,000 ObsCore rows go from 28.81 to 17.66 µs/row (1.63×). End to end, on a 10,000-row ObsCore result (tap-compare's Q11 shape) with API, database and generator sharing eight cores, VOTable at eight concurrent clients goes from 5.59 to 7.99 requests/s (1.43×, p50 1431 → 1006 ms) and lands within 6% of csv; at one client the request is bound by the query and nothing changes. tap_copy_votable_results_total and tap_copy_votable_fallbacks_total are its counters, TAP_COPY_VOTABLE=false its switch, and tests/component/test_copy_votable_differential.py holds its bytes to stream_votable's. Neither DSV decline applies to it: a lone NULL is <TR><TD/></TR> on both paths, and repeated column names render positionally under FIELDs taken from the same cursor description.

None of this changes the advice above. csv, tsv and votable are cheaper to serve than they were; they are still the widest bytes on the wire, and a bulk transfer should still be Parquet or Arrow.

The make benchmark-serialize and make benchmark-result-formats harnesses that produced these figures have been removed with the rest of the cluster benchmark suite (see the performance archive). The serialization hot paths are still measured per commit by uv run --group microbenchmark pytest tests/benchmarks --benchmark-only (tests/benchmarks/test_hot_paths.py), which is the tool for deciding whether a change to the serialisation path did anything. The archived end-to-end comparison (run 20260824T181653Z-a5e3d315, D1, 10,000 rows per response, 4 clients): parquet p95 307 ms and 0.54 MiB per response, arrow 313 ms and 2.16 MiB, json 544 ms, votable 615 ms, csv/tsv ~750 ms — warm-cache figures, since the family held the database cost constant to isolate the writer.

How the writers stay cheap

Columns are typed once, from the PostgreSQL cursor's type OIDs, and the type decides how a cell is rendered. That decision is made per column, not per cell: a ten-thousand-row ObsCore result is 110,000 cells, and inspecting each one's Python type to work out what it is — which is what the writers used to do — costs more than everything else they do put together.

So a column's kind names the Python type psycopg produces as well as the wire type the writer emits. float8 arrives as a float and needs nothing done to it; numeric arrives as a Decimal and becomes a double; a timestamp becomes ISO-8601; text needs XML-escaping in VOTable and nothing anywhere else. The kinds that need no conversion reach the writer untouched.

An OID the service does not recognise is opaque, and an opaque column keeps the fully dynamic per-cell path — which is what it needs, because nothing has promised what its values are. That is also the default for a column whose type nobody stated. If you add a metadata domain with an exotic column type, its results are correct and its serialisation is the slow one; the fast path is a consequence of a type being recognised, not of anything you have to configure.