Show the whole hash-collision cluster, and stop counters answering hash searches (LMFDB#5556)

Follow-up review of PR 44 found the hash search still rendered through
gps_groups, so it could not show what the complete tables know.

A new route, /Groups/Abstract/hash/<order>/<value>, answers "which groups
have this order and hash".  At the ten orders with a complete gps_smallhash
table it lists every one of them, including those with no database row
(78125#3521944227884464685 is 78125.82, 78125.335, 78125.340, and only the
last two are in gps_groups), says that the list is complete, links the ones
with a homepage and marks the rest as absent, and redirects when the cluster
determines a single group.  Other orders go on to the ordinary search page,
where the stored column is the hash.  Every "groups with this order and
hash" link now goes through the route: group homepages, subgroup and
quotient popups, the identification tool and the find box, as does a search
whose only constraints are an order and a hash.

The hash column of gps_groups is no longer consulted at the six orders where
it holds the label counter, whatever the order constraint: with no single
order, hash_constraint builds an $or of the column match away from those
orders together with one resolved (order, counter) branch per affected
order, from a single indexed gps_smallhash query.  So a search for the hash
11 no longer returns 512.11, whose stored 11 is its counter, while a search
for 1584677793794603025 finds it with no order given at all.

The optional column still cannot show a hash that only gps_smallhash knows,
which is now documented; where the search itself named one hash at one
order, that value is displayed.  Out-of-range values are refused by the new
route rather than reaching the database, which errors on them.

Verified with sage -python -m pytest on lmfdb/groups/abstract:
test_abstract_groups.py 25 passed (the hash tests now cover the complete
cluster page, the popup and identification links into it, the label with no
homepage, and the false-positive cases), test_browse_page.py 71 passed,
pyflakes clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: roed314

lmfdb/groups/abstract/hash_lookup.py

@@ -24,14 +24,21 @@
 A hash search at one of those ten orders therefore has to go to
 ``gps_smallhash``, and what comes back is complete: the groups it lists are
 provably all the groups of that order with that hash, so a unique match is a
-proof of isomorphism.
+proof of isomorphism, and a collision is a cluster that a search of
+``gps_groups`` alone would under-report.  That is what the ``by_hash`` route
+renders; :func:`hash_search_url` sends every "groups with this order and hash"
+link there.
 
 Index note: ``gps_smallhash`` has 4.2*10^8 rows and a single index, on
 ``(order, hash)``.  Queries in that direction are fast.  The other direction,
 ``(order, counter)`` to a hash, is a sequential scan (about six minutes against
-devmirror), so the true hash of a group at one of these orders is not something
-a page can display; :func:`structural_hash` reports that there is no usable
-value rather than showing the counter under a "hash" heading.
+devmirror), so a group at one of the six counter-storing orders cannot be asked
+for its own hash.  A row of such an order therefore shows no hash on its
+homepage and an empty cell in the optional search column, rather than its
+counter under a "hash" heading (:func:`structural_hash`); the exception is a
+search that named one hash at one order, where :func:`searched_hash` recovers
+the value from the query itself.  Displaying these hashes in general would take
+an ``(order, counter)`` index on ``gps_smallhash``.
 """
 
 from dataclasses import dataclass, field
@@ -48,6 +55,10 @@
     [512, 1152, 1536, 1920, 2187, 6561, 15625, 16807, 78125, 161051]
 )
 
+# The orders where gps_groups.hash is the label counter rather than the hash,
+# so that a search against that column would be answering a different question.
+COUNTER_HASH_ORDERS = frozenset([512, 1152, 1536, 1920, 2187, 15625])
+
 
 @dataclass(frozen=True)
 class HashResolution:
@@ -101,6 +112,17 @@ def smallhash_counters(order, values):
         {"order": int(order), "hash": {"$in": [int(v) for v in values]}}, "counter")})
 
 
+def smallhash_counters_by_order(orders, values):
+    """The counters of the groups of any of these ``orders`` whose hash is one
+    of ``values``, as a dict keyed by order.  One indexed query."""
+    out = {}
+    for rec in db.gps_smallhash.search(
+            {"order": {"$in": [int(N) for N in orders]},
+             "hash": {"$in": [int(v) for v in values]}}, ["order", "counter"]):
+        out.setdefault(int(rec["order"]), []).append(int(rec["counter"]))
+    return {N: sorted(counters) for N, counters in out.items()}
+
+
 def resolve_order_hash(order, value):
     """The groups of order ``order`` whose hash is ``value``, as a
     :class:`HashResolution`."""
@@ -113,20 +135,49 @@ def resolve_order_hash(order, value):
 
 
 def hash_constraint(order, values, qfield="hash"):
-    """Search constraints selecting the groups of order ``order`` whose hash is
-    one of ``values``, to be merged into a ``gps_groups`` query.
-
-    ``order`` may be ``None`` when the order is unconstrained or constrained to
-    a range, in which case the stored ``hash`` column is the only thing to go
-    on: such a search can pick up a spurious row at one of the orders that
-    store counters there, which giving a single order (or using the ``N#h``
-    form) avoids.  With an order in hand we resolve the hashes to counters at
-    the complete-table orders, since ``gps_groups.hash`` is not the hash there.
+    """Search constraints selecting the groups whose hash is one of ``values``,
+    to be merged into a ``gps_groups`` query.
+
+    ``order`` is the single order the search is pinned to, or ``None`` when the
+    order is unconstrained or constrained to a range or a list.  Either way the
+    stored ``hash`` column is only consulted at the orders where it holds the
+    hash: at the others the hashes are resolved to counters through
+    ``gps_smallhash``, so that a search for the hash ``11`` cannot come back
+    with ``512.11``, whose stored 11 is its counter.
+
+    With no single order this needs a top-level ``$or`` (merge it with
+    ``collapse_ors``, not ``dict.update``); any order constraint already in the
+    query still applies to every branch.
     """
     values = [int(v) for v in values]
-    if order is not None and int(order) in SMALLHASH_ORDERS:
-        return {"counter": {"$in": smallhash_counters(order, values)}}
-    return {qfield: values[0] if len(values) == 1 else {"$or": values}}
+    column = {qfield: values[0] if len(values) == 1 else {"$or": values}}
+    if order is not None:
+        if int(order) in SMALLHASH_ORDERS:
+            return {"counter": {"$in": smallhash_counters(order, values)}}
+        return column
+    counters = smallhash_counters_by_order(COUNTER_HASH_ORDERS, values)
+    branches = [dict(column, order={"$nin": sorted(COUNTER_HASH_ORDERS)})]
+    branches.extend({"order": N, "counter": {"$in": cs}} for N, cs in sorted(counters.items()))
+    return {"$or": branches}
+
+
+def searched_hash(info):
+    """The ``(order, value)`` that a search asked for, when it asked for one
+    hash at one order and so knows a value that the row itself may not record.
+
+    Both ``N#h`` in the hash box and a hash with the order box pinned to a
+    single order count; a list of hashes or a range of orders does not.
+    """
+    raw = (info.get("hash") or "").strip()
+    if not raw or "," in raw:
+        return None
+    try:
+        if raw.count("#") == 1:
+            order, value = raw.split("#")
+            return int(order), int(value)
+        return int((info.get("order") or "").strip()), int(raw)
+    except ValueError:
+        return None
 
 
 def structural_hash(counter, stored):
@@ -151,5 +202,10 @@ def order_search_url(order):
 
 
 def hash_search_url(order, value):
-    """URL of the search page listing all groups of the given order and hash."""
-    return url_for("abstract.index", hash=f"{int(order)}#{int(value)}")
+    """URL of the page listing all the groups of the given order and hash.
+
+    Every "groups with this order and hash" link goes here, and the route sends
+    the orders without a complete table on to the ordinary search page, so that
+    such a link never shows a subset of what it promises.
+    """
+    return url_for("abstract.by_hash", order=int(order), value=int(value))

lmfdb/groups/abstract/main.py

@@ -78,10 +78,13 @@
     looks_like_permutation,
 )
 from .hash_lookup import (
+    SMALLHASH_ORDERS,
     hash_constraint,
     hash_search_url,
+    live_pages_available,
     order_search_url,
     resolve_order_hash,
+    searched_hash,
     structural_hash,
 )
 
@@ -243,26 +246,30 @@ def parse_family(inp, query, qfield):
 
 @search_parser
 def parse_hashes(inp, query, qfield, order_field):
-    # hash_constraint decides which column carries the hash at this order: at
-    # the orders with a complete gps_smallhash table, gps_groups.hash holds the
-    # label counter instead of the hash, so the search has to go through
-    # gps_smallhash and come back as a list of counters.
+    # hash_constraint decides which column carries the hash: at the orders with
+    # a complete gps_smallhash table, gps_groups.hash holds the label counter
+    # instead of the hash, so those orders are resolved through gps_smallhash
+    # and come back as a list of counters (as a $or over orders when the search
+    # is not pinned to one order).
     if inp.count("#") == 0:
         opts = [ZZ(opt) for opt in inp.split(",")]
         N = query.get(order_field)
         if isinstance(N, (dict, list)):
-            N = None  # a range or list of orders: only the stored column to go on
-        query.update(hash_constraint(N, opts, qfield))
+            N = None  # a range or list of orders: constrain each order separately
+        constraint = hash_constraint(N, opts, qfield)
     elif inp.count("#") == 1:
         N, hsh = inp.split("#")
         N, hsh = ZZ(N), ZZ(hsh)
         if order_field not in query:
             query[order_field] = N
         elif query[order_field] != N:
             raise ValueError(f"You cannot specify order both in the {order_field} input and the {qfield} input")
-        query.update(hash_constraint(N, [hsh], qfield))
+        constraint = hash_constraint(N, [hsh], qfield)
     else:
         raise ValueError("To specify multiple hash values, all must have the same order; provide the order in the order input and then just give hashes separated by commas")
+    if "$or" in constraint:
+        collapse_ors(["$or", constraint.pop("$or")], query)
+    query.update(constraint)
 
 #input string of complex character label and return rational character label
 def q_char(char):
@@ -798,6 +805,12 @@ def index():
         search_types = request.args.getlist("search_type")
         info["search_type"] = search_type = search_types[-1] if search_types else info.get("hst", "")
         if search_type in ["List", "", "Random", "Diagram"]:
+            # A bare order-and-hash search is answered by the hash page, which
+            # can list the groups of a complete-table order that gps_groups
+            # does not contain.
+            asked = hash_only_search(request.args) if search_type in ["List", ""] else None
+            if asked is not None and asked[0] in SMALLHASH_ORDERS:
+                return redirect(url_for(".by_hash", order=asked[0], value=asked[1]))
             return group_search(info)
         # Preserve old abstract-group search URLs while directing users to the
         # new, object-specific landing pages.  Keep Random* as a search type so
@@ -1253,7 +1266,7 @@ def group_jump(info):
             label = res.unique_label()
             if label is not None:
                 return redirect(url_for(".by_label", label=label))
-        return redirect(url_for(".index", hash=jump))
+        return redirect(hash_search_url(N, hsh))
     # by permutation generators
     if looks_like_permutation(jump):
         return redirect(url_for(".identify_group_page", description=jump))
@@ -1487,8 +1500,12 @@ def group_postprocess(res, info, query):
             if label is not None:
                 labels.add(label)
     tex_cache = {rec["label"]: rec["tex_name"] for rec in db.gps_groups.search({"label":{"$in":list(labels)}}, ["label", "tex_name"])}
+    # Where gps_groups.hash holds the label counter the row cannot supply its
+    # own hash (see hash_lookup), but a search for one hash at one order can.
+    asked = searched_hash(info)
     for rec in res:
         rec["tex_cache"] = tex_cache
+        rec["public_hash"] = asked[1] if asked and rec.get("order") == asked[0] else None
     if "family" in info:
         family = info["family"]
         if family == "any":
@@ -1585,12 +1602,13 @@ def group_postprocess(res, info, query):
                       ["abelian", "nilpotent", "solvable", "smith_abelian_invariants", "nilpotency_class", "derived_length", "composition_length"],
                       show_type,
                       align="center"),
-    # structural_hash suppresses the rows where gps_groups.hash is the label
-    # counter rather than the hash, so that this column never shows something
-    # other than the value the group.hash knowl describes.
+    # The hash the group.hash knowl describes: the stored value where that is
+    # the hash, the searched-for value where the search supplies one, and
+    # nothing rather than a label counter (see hash_lookup.structural_hash).
     MultiProcessedCol("hash", "group.hash", "Hash",
-                      ["counter", "hash"],
-                      lambda counter, h: "" if structural_hash(counter, h) is None else str(h),
+                      ["counter", "hash", "public_hash"],
+                      lambda counter, h, asked: str(asked) if asked is not None
+                      else ("" if structural_hash(counter, h) is None else str(h)),
                       default=False, align="right", short_title="hash")])
 
 @search_wrap(
@@ -2700,6 +2718,55 @@ def download_trivial_construction(dltype):  #trival gp construction is different
     return s
 
 
+@abstract_page.route("/hash/<int:order>/<int:value>")
+def by_hash(order, value):
+    """All the groups of a given order and hash.
+
+    At the orders with a complete ``gps_smallhash`` table this lists every such
+    group, including the ones with no database row, which an ordinary search of
+    ``gps_groups`` cannot do; the other orders go on to the search page, where
+    the stored column is the hash and the usual search machinery applies.
+    """
+    # Orders and hashes are stored as 32- and 64-bit integers, so anything
+    # larger is not one, and asking the database would be an error.
+    if order >= 2**31 or value >= 2**63:
+        flash_error("No group has order %s and hash %s.", order, value)
+        return redirect(url_for(".index"))
+    res = resolve_order_hash(order, value)
+    if not res.complete:
+        return redirect(url_for(".index", hash=f"{order}#{value}", search_type="List"))
+    label = res.unique_label()
+    if label is not None:
+        return redirect(url_for(".by_label", label=label))
+    present = set(res.in_lmfdb())
+    live = live_pages_available(order)
+    return render_template(
+        "abstract-hash.html",
+        title=f"Groups of order {order} with hash {value}",
+        bread=get_bread([("Hash", "")]),
+        order=order,
+        value=value,
+        candidates=[{"label": lab, "present": lab in present, "live": live}
+                    for lab in res.labels],
+        order_url=order_search_url(order),
+        learnmore=learnmore_list(),
+    )
+
+
+# Arguments that say how to display a search rather than what to search for.
+HASH_SEARCH_ARGS = {"hash", "order", "search_type", "hst", "count", "start",
+                    "showcol", "hidecol", "sort_order", "sorts", "columns", "submit"}
+
+
+def hash_only_search(args):
+    """The ``(order, value)`` of a request that asks for nothing but an order
+    and a hash, so that it can be answered by :func:`by_hash` instead of by a
+    search of ``gps_groups``; ``None`` for anything else."""
+    if any(v.strip() for k, v in args.items() if k not in HASH_SEARCH_ARGS):
+        return None
+    return searched_hash(args)
+
+
 @abstract_page.route("/identify")
 def identify_group_page():
     description = request.args.get("description", "")

lmfdb/groups/abstract/templates/abstract-hash.html

@@ -0,0 +1,40 @@
+{% extends "homepage.html" %}
+
+{% block content %}
+
+<p>
+  {{KNOWL('group.hash', 'Hash')}}: {{ value }}, order&nbsp;{{ order }}
+  (<a href="{{ order_url }}">all groups of order&nbsp;{{ order }}</a>).
+</p>
+
+{% if candidates %}
+  <p>
+    The hash tables are complete for order&nbsp;{{ order }}, so these are
+    {% if candidates|length == 1 %}
+      all the groups of order&nbsp;{{ order }} with this hash.
+    {% else %}
+      all {{ candidates|length }} groups of order&nbsp;{{ order }} with this hash:
+      they form a hash collision, and any group with this order and hash is
+      isomorphic to exactly one of them.
+    {% endif %}
+  </p>
+  <ul>
+    {% for c in candidates %}
+      <li>
+        {% if c.present or c.live %}
+          <a href="{{ url_for('.by_label', label=c.label) }}">{{ c.label }}</a>
+          {%- if not c.present %} <span class="nowrap">(not in the database)</span>{% endif %}
+        {% else %}
+          {{ c.label }} <span class="nowrap">(not in the database)</span>
+        {% endif %}
+      </li>
+    {% endfor %}
+  </ul>
+{% else %}
+  <p>
+    The hash tables are complete for order&nbsp;{{ order }}, so no group of
+    order&nbsp;{{ order }} has this hash.
+  </p>
+{% endif %}
+
+{% endblock %}