Add raw output format to the API

[roed-math]

This adds _format=raw to the API (alongside html/json/yaml), returning only the contents of the columns requested via _fields, as requested by Bill Allombert for consumption from PARI/GP scripts.

Raw output is newline-delimited JSON without the usual response envelope or implicit id. With one requested field, each line is that field's JSON value. With several fields, each line is an ordered JSON array containing those field values. _delim is used only to parse parameters such as _fields and _sort; it does not delimit output records or fields, so a record stays unambiguous even when a value contains the delimiter or a newline.

Records are rendered from the database rows rather than from Json.prep's output, so a Postgres numeric is a JSON number carrying every stored digit: ec_curvedata.faltings_height for 11a1 comes out as -0.30800984111840306468901426146, not as the __RealLiteral__ object that records how to rebuild the Sage real. The decimal is copied verbatim, never through a float. Values with no plain JSON rendering (rationals, number field elements, dates) keep psycodict's extended encoding.

For fields whose emitted JSON notation is also valid GP syntax, notably integers and nested arrays of integers, the response can be read directly with PARI/GP readvec (verified in gp). Other values should be parsed as JSON; a JSON boolean in particular is not a GP boolean.

Raw requests remain limited to 100 records and omit pagination metadata, so _offset is advanced to retrieve subsequent pages; requesting raw output without a nonempty _fields gives a 400 error. The format is documented on the API index page, table pages link to it whenever _fields is nonempty, and test_api.py gains exact-output tests covering numerics, booleans, nulls, integer arrays, multi-field records, a delimiter inside a value and the error case. Addresses LMFDB#1010.

🤖 Generated with Claude Code

[roed-math]

Addressed the external review's P2 finding (multi-field raw records not parseable): commit fa953ee makes multi-field raw output emit one JSON array per line (JSON Lines), so each record is self-delimiting regardless of delimiters inside values; single-field output is unchanged. _delim now only splits the _fields/_sort inputs, never joins output. The API docs bullet specifies the format exactly and scopes the PARI/GP readvec claim to the single-field case (re-verified in sage --gp). Tests now round-trip records via json.loads, including a delimiter-inside-value case.

Stayed within this PR's hunks — no overlap with #18/#22/#40 regions.

[roed-math]

Pushed 182d10b addressing the review.

1. Exact numerics. Raw records are now rendered from the database rows, before Json.prep, by a small serializer (raw_json_dumps in lmfdb/api/api.py). A real number's stored decimal is copied verbatim once it has been checked against the JSON number syntax, never through a float, so ?label=11a1&_format=raw&_fields=faltings_height returns

-0.30800984111840306468901426146

instead of the __RealLiteral__ object. Integers, strings, booleans and nulls are JSON as before; lists recurse, so a numeric[] column is exact too, and values with no plain JSON rendering keep psycodict's extended encoding. NaN and the infinities have no JSON number syntax, so they fall back to the extended encoding rather than emit a line no JSON reader accepts.

2. Docs and the GP claim. The code comments, test_api_raw docstring, API index page and PR description now describe the actual format, along the lines the review suggested: newline-delimited JSON, no envelope and no implicit id, _delim used only to parse request parameters, readvec scoped to fields whose emitted JSON is also GP syntax, and raw bodies carrying no next entry so that _offset advances the page. Checked in sage --gp: readvec on 100 lines of ainvs still gives a t_VEC of indexable t_VECs, while a file of true/false reads as t_POL, i.e. symbolic variables rather than booleans, which is exactly what the old blanket claim would have promised.

3. Raw link condition. collection.html now tests url_args.get('_fields'), so _fields= no longer offers a Raw link to a guaranteed 400.

One deviation, on point 5. rec[col] cannot be used: 161 of the 192 tables (including ec_curvedata) are declared with include_nones off, and psycodict then omits a NULL column from the result dictionary rather than storing None, so rec[col] would turn every NULL into a 500. An absent key is unambiguous here in any case, because a field that is not a column of the table is rejected when the query is built (the existing 404 path), so after a successful search the only way a key can be missing is a NULL. The code keeps .get with that reasoning in a comment, and test_api_raw pins the behaviour: _fields=squarefree_disc gives null, _fields=squarefree_disc,conductor gives [null, 11].

Projecting exactly the requested fields (point 6) exposed a related edge case: a record whose projected columns are all NULL comes back from lucky as {}, which the by-id route read as "no such document" and turned into a 404. It now tests that result against None instead of for emptiness.

Verification. sage -python -m pytest lmfdb/api/test_api.py gives 7 passed against devmirror; pyflakes and ruff check --preview --select=E722 are clean. test_api_raw gains the exact-numeric and boolean cases from the review, plus NULL cases and a numeric inside a multi-field record, and keeps every earlier case. Raw numeric, boolean, string, integer-array, numeric[], dotted-path, by-id, multi-field and multi-record responses were also inspected by hand, with every emitted line re-parsed as JSON using parse_float=Decimal.

The remaining checklist item I cannot tick off is the Actions run: this fork's queue is currently around 85 runs deep and the run for 182d10b has not started yet.

[roed-math]

Pushed 30b24a7 for the decimal-zero case. You are right, and I had made exactly the wrong call on it: I saw LmfdbDecimalZero while writing the serializer, reasoned that an all-zero decimal is an exactly integral numeric and so belonged under "exact integral numeric values become JSON integers", and let it fall through to Json.prep. That reading loses the stored scale and the sign of -0.000, and it contradicts what the rest of the format promises.

The literal lookup now lives in exact_decimal(value), which handles the integer wrapper alongside RealLiteral and RealNumber and validates against JSON_NUMBER_RE once, so there is no duplicated regex check and nothing is rebuilt from a float or an int:

>>> raw_json_dumps(numeric_converter("0.000"))
'0.000'
>>> raw_json_dumps(numeric_converter("-0.000"))
'-0.000'
>>> raw_json_dumps([numeric_converter("1.250"), numeric_converter("0.000")])
'[1.250, 0.000]'

A numeric stored without a decimal point is still a Sage integer and is unaffected, and NaN and the infinities still fall back to the extended encoding rather than emit a token no JSON reader accepts. The effect is visible on a real numeric[] column: /api/lfunc_lfunctions/?_format=raw&_fields=mu_imag,analytic_normalization gave [[0], 0] before and gives [[0.0], 0] now, the second field being a numeric with no decimal point.

test_api.py gains test_raw_json_dumps, which calls the serializer directly as suggested, so the behaviour does not depend on finding a zero-valued row: 0.000, -0.000, 1.250 and the 29-digit faltings_height literal all come back verbatim, zeros nested in an array and in an object keep their literal, and every result is re-parsed as JSON. The exact strings carry the test, since Decimal("-0.000") == Decimal("0.000"). Every earlier API-level case is retained.

sage -python -m pytest lmfdb/api/test_api.py gives 8 passed against devmirror; pyflakes and ruff check --preview --select=E722 are clean on both files.

On the last checklist item: the Actions runs for 182d10b never started, and the runs for this head are queued behind the same backlog (the run pushed one minute before 182d10b waited about three and a half hours for a runner, then passed). I am watching this head and will report the conclusion here.

[roed314]

GPT signed off.

[roed-math]

Superseded by LMFDB#7153, opened upstream from this same branch. Closing here; review continues upstream.