Add more column options to p-adic field dynamic statistics

[roed-math]

The dynamic statistics page for p-adic fields previously offered only Galois
group and Galois Artin slopes as columns. This extends LFStats.dynamic_cols to
every searchable column (residue characteristic, degree, ramification index,
residue field degree, discriminant exponent, Galois group, automorphisms,
unramified/tame degree, top slope, Galois/visible/hidden slopes, indices of
inseparability, associated inertia, jump set), with display metadata and
click-through query formatters matched to the existing search boxes. Sort keys
map not-computed (NULL) values to sort first so columns with NULLs no longer
crash when selected unconstrained, and Galois-group handling tolerates fields
with no computed Galois group. Default p/c buckets are provided, and the
static statistics page is verified unchanged (discdisp pins empty buckets).
Links to "create your own" statistics are added from the browse index and the
statistics page. Adds tests covering the links and rendering of all new columns.
Addresses LMFDB#6542.

🤖 Generated with Claude Code

[roed-math]

Addressed the external review's P2 and P3 findings: commit d120ea2.

P2 (null-bucket drill-downs): confirmed the parsers silently drop empty params and no null-search URL syntax exists, so suppression is the right policy. Null branches of the query formatters now return a NO_SEARCH_QUERY sentinel, LFStats blanks any drill-down containing it (via a testable helper), and stat_1d.html guards the count link the same way stat_2d.html already did. Concrete impact: the cached not-computed galois_label bucket (3,996 fields) previously linked to an unfiltered ~1.39M-row search; it now renders linkless.

P3: formatbracketcol(None) renders "not computed" instead of $None$.

12 local_fields tests pass (2 new); non-null drill-downs verified to still filter correctly; stayed clear of PR #26's hunks in the same file.

[roed-math]

Addressed the three drill-down/bucketing problems from the review, in 19087b7. All three turned out to be in the generic statistics framework rather than in LFStats, so the fixes live in lmfdb/utils/display_stats.py and apply to every StatsDisplay user; the new p-adic columns, null-safe display, link suppression, and browse/statistics links are all kept as they were.

1. Sparse two-dimensional grids build urls from stored values

display_data handed the display formatters to _get_values_counts, which indexes its counts by them, so the KeyedDefaultDict synthesized each empty cell from an already formatted header: $[2]$ instead of [2], not computed instead of None, TeX instead of the stored top-slope encoding. Since totaler.common_link recovers a row's constraint by intersecting the urls of its cells, an empty cell disagreeing with a nonempty one dropped the constraint from the total.

This was not hypothetical, and not limited to p-adic fields. On the abstract groups statistics page, the "not nilpotent" row totalled 455903 groups and linked to /Groups/Abstract/?, which returns all 1.5 million:

- <td class="cnt totalcol"><a href='/Groups/Abstract/?'>455903</a></td>
+ <td class="cnt totalcol"><a href='/Groups/Abstract/?nilpotent=no'>455903</a></td>

display_data now keys counts by a canonical form of the stored value and builds every cell url itself, empty ones included, from the row and column values. A formatter is therefore free to emit TeX or html, and a query formatter is never handed a display string (the docstring says so now). Two values that display identically share a row and add their counts, instead of one silently overwriting the other in the counts dictionary, and such a row is only linked if both values constrain the search the same way.

Two related repairs fell out: a row whose cells share no constraint is no longer linked at all (the guard existed for column totals but not row totals), and common_link emits parameters in a fixed order rather than set order, so the same table no longer produces different urls on different runs.

2. Drill-downs use the search boxes, not the parsed query

dynamic_setup kept only the query common_parse produced, and display_data serialized it into urls, so a constraint on Artin slopes became slopes_tmp=2A and internal dictionaries such as {"$contains": ...} rendered as =None. The p-adic parser consumes neither, so clicking a count silently dropped the constraint.

dynamic_setup now also passes link_constraint: the search boxes the user filled in, url-encoded. They reproduce the same query by construction, being the input that produced it, and the test asserts exactly that rather than matching substrings, for each of slopes, visible and ind_of_insep under exactly/include/exclude/subset:

stats.dynamic_parse(info, query)
link = stats.dynamic_link_constraint(info, ['n'])
common_parse(to_dict(dict(parse_qsl(link))), reparsed)
assert reparsed == query          # and '_tmp' not in link, '=None' not in link

Columns being displayed are excluded via a new url_params mapping (galois_label → gal, slopes → slopes + slopes_quantifier, and so on), so a displayed column is not also constrained.

3. Top-slope buckets are encoded before comparison

top_slope is stored as a fixed-width decimal prefix plus the exact rational, and the backend parses bucket endpoints by column type, so 1-2 became {"$gte": "1", "$lte": "2"} compared lexicographically against that encoding. topslope_query also sliced 12 characters off every endpoint, so the bucket 1-2 linked to topslope=-.

There is now a bucket_encoders hook: endpoints are encoded (with the same ratproc the search box uses) before they reach the backend, and decoded for labels and links, while setup() uses the same encoding so precomputed and on-demand statistics agree. Decoding only accepts strings that really carry the encoding. buckets1=0-1,1-2,2- now labels rows $0$-$1$, $1$-$2$, $2$- and links to topslope=0-1, topslope=1-2, topslope=2-; the test checks each link selects exactly what the backend counts:

assert self._url_count('/padicField/?topslope=1-2') == db.lf_fields.count(
    {'top_slope': {'$gte': ratproc('1'), '$lte': ratproc('2')}})   # 25620

A malformed endpoint is now rejected with the usual message instead of an empty table.

Also

• The NO_SEARCH_QUERY sentinel and its suppression moved into the framework, so an unsearchable value is linkless in any statistics table, not only p-adic ones.
• Invalid input to a dynamic statistics page flashes the error and renders the form, instead of raising a 500. That was already the behaviour for a bad gal= before this PR; it now covers bad buckets and missing statistics too.
• Default p and c buckets end open (101-, 33-) rather than at the current maxima of 199 and 79, so no field is dropped as the database grows. A test asserts the last bucket is unbounded and the first covers the minimum.
• jump_set uses formatbracketcol, so a not-computed jump set (not computed) and an empty one ($[\ ]$, 110473 fields, and searchable as jump_set=[]) cannot collide into one ambiguous row. Neither value occurs as NULL in the current data, but the labels no longer depend on that.
• nilp_qformatter in the groups statistics maps the stored -1 to nilpotent=no; it previously only handled the display string "not", and the raw value produced nilpotency_class=-1, which matches nothing.

Checks

• sage -python -m pytest lmfdb/local_fields/test_localfields.py: 17 passed. New tests assert href semantics and followed-result counts, not status codes: a sparse galois_label × n grid (1431 rows) where every row total keeps gal=… and drops n=, with the followed counts compared against the search; a stub statistics backend that pins the raw-value invariant for empty cells independently of what the devmirror happens to have cached; the re-parse test above; and the top-slope bucket counts, labels and links.
• pytest lmfdb/groups/abstract/test_browse_page.py plus local fields: 88 passed. pytest lmfdb -k stat: 19 passed.
• Every statistics page in LMFDB (p-adic, CMF, abelian varieties, groups, genus 2, Dirichlet, modular curves, Galois groups, lattices, Hilbert and Bianchi forms, Artin, ECNF) rendered before and after and diffed: byte-identical apart from the groups total link above and the two error pages that no longer 500. The static p-adic statistics page is unchanged.
• pyflakes and pylint clean; no new pycodestyle findings in the changed lines.

One thing worth knowing when reviewing: the devmirror is read only, so statistics for combinations that are not already cached come back empty and their tables render blank. That is why the count-comparison tests are written against combinations that are cached (galois_label × n with p fixed) or against live queries on both sides, rather than against rendered numbers that would be zero here and correct in production.

[roed-math]

Fixed in 342806c. The follow-up is right on both counts, and the second failure mode reaches further than p-adic fields.

The Total row

display_data was appending the bare column name to the total's url (?galois_label), which I carried over from before this PR. It is not a constraint: the search parsers ignore an empty parameter, so the link returned everything, and the parameter is frequently not one the search page accepts.

The gap is real wherever a column is not computed for every record, because the statistics backend totals over the non-null ones (_process_constraint adds IS NOT NULL for the column being counted):

page	total shown	old link returned
/ModularForm/GL2/Q/holomorphic/stats, analytic rank	281885	1141510
dynamic statistics, p=7&n=21&col1=galois_label	1324	3784
col1=top_slope&buckets1=0-1,1-2	33533	1393286

A total is now linked only when the search page can describe exactly the records it counts, which is the constraint itself, and only when the total covers every record satisfying it:

if buckets or split_list or any(col in constraint for col in cols):
    return ''
if total != table.table.count(constraint):
    return ''
return base_url + '&'.join(extras)

Buckets and split lists are excluded for the reasons the brief gives. The third clause is a case the brief does not mention: when the constraint restricts a column that is also being displayed, the urls deliberately leave that column out (each count constrains it itself), so the total would claim more records than it counted. Otherwise the exactness test is a direct comparison against the number of records the constraint matches, rather than against the sum of the displayed counts, since a null row is only displayed when the cached statistics happen to include one.

Where the column is computed for everything, the link survives and is now honest: /Genus2Curve/Q/?num_rat_pts became /Genus2Curve/Q/?, same records, no meaningless parameter.

Tests

• p=7&n=21&col1=galois_label&totals1=yes: the Total is 1324, asserted equal to the count of fields with a Galois group rather than to the 3784 fields, and rendered without a link.
• p=2&n=8&col1=galois_label&totals1=yes: computed for all 1823, so the Total is linked, and following the link returns 1823.
• col1=top_slope&buckets1=0-1,1-2&totals1=yes: Total unlinked, the two buckets still linked to topslope=0-1 and topslope=1-2.
• test_dynamic_stats_1d_totals_policy drives display_data against a stub backend for all four cases (computed everywhere, nullable, bucketed, constrained on the displayed column), so none of it depends on the statistics cache.

One thing the follow-up did not ask about

Writing those tests turned up a defect in my own test helper: _stat_counts was matching a class that stat_1d.html does not emit (a one-dimensional table is transposed, values in one row and counts in the next), so it always returned an empty list. Two page-level loops in the previous round were therefore passing vacuously. The helper now parses the real structure, and the checks that depended on it were rewritten to run against a stub so that they cannot silently become vacuous again.

With the helper working, the page comparison I reported last time turned out to have the same blind spot: it only captured <table class="statgrid">, the two-dimensional tables. Re-running it over one-dimensional tables as well, four more pages differ from the PR base, and all four are fixes:

• CMF — the analytic rank total, above.
• Elliptic curves, Belyi maps, genus 2 — bare-marker removal, same records either way.
• Genus 2 — the addl_row_title row. It was zipped against the values in database order while the counts were in sorted order, so the labels were rotated against the columns beneath them: the 6 curves with real geometric endomorphism algebra C x C were labelled $M_2(\mathbb{R})$. Fixing the headers to carry their own values corrected the pairing as a side effect; each label now matches the value in the link below it, and matches its identity component ($\mathbb{C}\times\mathbb{C}$ with U(1)×U(1), $\mathbb{R}$ with USp(4), and so on).

Everything else across the site is byte-identical to the PR base once url parameter order is normalized, which the previous round changed from set order to a fixed order.

Checks

pytest lmfdb/local_fields/test_localfields.py: 19 passed. pytest lmfdb/local_fields lmfdb/groups/abstract/test_browse_page.py lmfdb/genus2_curves/test_genus2_curves.py lmfdb/elliptic_curves/test_browse_page.py: 138 passed. pyflakes, pylint and pycodestyle clean on the changed files.

The extra count call happens once per one-dimensional total, and only for statistics blocks that have one; it goes through the same cached path search pages use, and the statistics pages I timed did not measurably change.

[roed314]

GPT signed off.

[roed314]

GPT signed off.

[roed-math]

Superseded by LMFDB#7163, opened upstream from this same branch. Closing here; review continues upstream.