Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,30 @@ below corresponds to one such version.
what a question costs — measured at 90-97% of its output tokens, against a statement of 50-80 —
so a lower level is the next large saving. The level is recorded in the run's JSON and artifact,
because a score measured at one level says nothing about another.
### Added

- **Four `sm` verbs that grade a statement a person supplied, part by part.** `agami-reconcile` is
learning to take a trusted query as evidence rather than as the answer, and these are the
deterministic checks it will lean on. `sm claims` reports where two statements differ, in the seven
claims the golden runner already compares. `sm compare-results` says whether two result CSVs say the
same thing, through the golden comparator, so a table-shaped answer is judged the way an answer key
is. `sm join-probes` names, for every join a statement wrote, whether the semantic model declares a
relationship between those tables and whether the written key matches the declared one, and emits
the overlap and cardinality probes that would show whether the keys really resolve. `sm filter-values
plan` names, for every value typed into a filter, the column it binds to, whether the semantic
model's list of values holds it, and the probes that would settle it; `sm filter-values judge` reads
the probe results back and grades each value `confirmed`, `model_gap`, `query_defect` or
`unresolved`. None of the four runs SQL: the skill runs every probe through the same execution tier
a question takes. Joins are classified with the receipt's own flags, so a CTE that shadows a
declared table, a `USING`, a comma join, or a declared `on:` this layer cannot read all come back
as open states and never as a settled claim about a key nobody read. A probe that came back empty
is graded as a probe that failed, never as a column that holds nothing. A near miss is a case and
whitespace fold only, an empty list of values reads as not yet decoded rather than as no legal
values, a column marked sensitive is never probed for a value, and a value carrying a backslash or
a control character is never sent to the warehouse, because engines quote it differently. The
overlap probe is the one introspection already trusts, now shared as `introspect.overlap_sql` and
bounded to its 50-row sample on every engine through the new `Dialect.limited`, where it used to
be bounded only where the row-limit keyword was `LIMIT`. (ACE-114)

### Fixed

Expand Down
166 changes: 166 additions & 0 deletions packages/agami-core/src/semantic_model/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,138 @@ def cmd_receipt(args) -> int:
return 0


# --- the four verbs that grade a statement a PERSON supplied ----------------------------------
#
# `agami-reconcile` takes a trusted query as evidence, never as the answer. These verbs are the
# deterministic half of grading it: none of them runs a probe, and none writes anything. The skill
# runs every emitted probe through the execution tier a question takes, then hands the CSVs back.


def _grammar(org) -> Optional[str]:
"""The sqlglot grammar to read statements in, or None when the semantic model cannot say which
engine its SQL runs on. Every verb below reports the same value under `dialect`, so a caller
reads one spelling of "engine unknown" across all four."""
return RT._dialect_of(org)[0]


def _read_sql_file(path: str) -> Optional[str]:
"""The statement in `path`, or None after printing `{"error": "unreadable_sql_file"}`: a file that
is not there must not become a traceback with an empty stdout, because the skill redirects stdout
to a file and a zero-byte file downstream reads as a probe that failed."""
try:
return Path(path).read_text(encoding="utf-8")
except OSError as exc:
_print_json({"error": "unreadable_sql_file", "detail": str(exc).splitlines()[0]})
return None


def cmd_claims(args) -> int:
"""Where two statements differ, in the seven claims the golden runner already compares.
`golden_claims.compare_statements` has been reachable from the runner and the save door and
from no command; this is that command. A side that could not be read says so, rather than
leaving seven `unknown` claims to explain themselves."""
from .golden_claims import compare_statements, count_temporal_predicates, read_claims
org = L.load_datasource(args.root)
grammar = _grammar(org)
left = _read_sql_file(args.sql_file)
if left is None:
return 2
right = _read_sql_file(args.against_sql_file)
if right is None:
return 2
diff = compare_statements(left, right, dialect=grammar or "")
out = diff.as_dict()
out["unreadable"] = {
"sql_file": read_claims(left, dialect=grammar or "").unreadable,
"against_sql_file": read_claims(right, dialect=grammar or "").unreadable,
}
# How many conjuncts on each side speak of time. Two zeros beside a `date_window` that reads
# `unknown` mean neither statement filtered on a date; anything else leaves the window open.
out["temporal_predicates"] = {
"sql_file": count_temporal_predicates(left, dialect=grammar or ""),
"against_sql_file": count_temporal_predicates(right, dialect=grammar or ""),
}
out["dialect"] = grammar
_print_json(out)
return 0


def cmd_compare_results(args) -> int:
"""Whether two result sets say the same thing, through the one comparator the golden runner
uses, so a table-shaped answer is judged the way an answer key is and not by a second rule. The
match level defaults to the comparator's own (`exact`) for the same reason."""
import dataclasses

from .comparator import compare_result_sets, result_from_csv
from .golden import GoldenBounds
org = L.load_datasource(args.root)
bounds = None
if args.bounds:
try:
bounds = GoldenBounds(**json.loads(args.bounds))
except (ValueError, TypeError) as exc: # pydantic's ValidationError is a ValueError
_print_json({"error": "bad_bounds", "detail": str(exc).splitlines()[0]})
return 2
try:
golden = result_from_csv(args.golden_csv)
generated = result_from_csv(args.generated_csv)
except (OSError, ValueError) as exc:
_print_json({"error": "unreadable_csv", "detail": str(exc)})
return 2
golden_sql = None
if args.golden_sql_file:
golden_sql = _read_sql_file(args.golden_sql_file)
if golden_sql is None:
return 2
score = compare_result_sets(golden, generated, match=args.match, golden_sql=golden_sql,
bounds=bounds, dialect=_grammar(org))
_print_json(dataclasses.asdict(score))
return 0


def cmd_join_probes(args) -> int:
"""Every join a statement wrote: declared or not, on the declared key or not, and the probe SQL
that would test whether the keys resolve. Emitted, never run."""
from . import probes
org = L.load_datasource(args.root)
sql = _read_sql_file(args.sql_file)
if sql is None:
return 2
_print_json(probes.join_probes(org, sql))
return 0


def cmd_filter_values_plan(args) -> int:
"""Every value typed into a filter: what the semantic model already knows about its column, and
the probe SQL that would settle the rest."""
from . import probes
org = L.load_datasource(args.root)
sql = _read_sql_file(args.sql_file)
if sql is None:
return 2
_print_json(probes.filter_values_plan(org, sql))
return 0


def cmd_filter_values_judge(args) -> int:
"""The probe results read back onto the plan: one grade per value. Takes the profile root like
every other verb, and reads no model from it; the plan already carries what the semantic model
knew."""
from . import probes
results = Path(args.results)
if not results.is_dir():
_print_json({"error": "no_results_dir", "detail": f"{results} is not a directory"})
return 2
try:
plan = json.loads(Path(args.plan).read_text())
out = probes.filter_values_judge(plan, results)
except (OSError, ValueError) as exc:
_print_json({"error": "bad_plan", "detail": str(exc)})
return 2
_print_json(out)
return 0


def cmd_review_queue(args) -> int:
from . import curate
org = L.load_datasource(args.root)
Expand Down Expand Up @@ -1202,6 +1334,40 @@ def build_parser() -> argparse.ArgumentParser:
help="optional freshness timestamp for the receipt's tables section")
sp.set_defaults(func=cmd_receipt)

sp = sub.add_parser("claims", help="where two statements differ: the seven claims the golden runner compares, as a diff")
sp.add_argument("root")
sp.add_argument("--sql-file", required=True, dest="sql_file")
sp.add_argument("--against-sql-file", required=True, dest="against_sql_file")
sp.set_defaults(func=cmd_claims)

sp = sub.add_parser("compare-results", help="whether two result CSVs say the same thing, through the golden comparator")
sp.add_argument("root")
sp.add_argument("--golden-csv", required=True, dest="golden_csv")
sp.add_argument("--generated-csv", required=True, dest="generated_csv")
sp.add_argument("--match", default="exact", choices=["exact", "values", "shape", "bounded", "nonempty"],
help="the comparator's own default is exact; reconcile passes values for a number that may carry a float tail")
sp.add_argument("--golden-sql-file", default=None, dest="golden_sql_file",
help="the answer key's statement, read only for whether it ordered its rows")
sp.add_argument("--bounds", default=None, help="JSON with min_rows/max_rows/min_value/max_value, for --match bounded")
sp.set_defaults(func=cmd_compare_results)

sp = sub.add_parser("join-probes", help="every join a statement wrote: declared or not, on the declared key or not, and the probe SQL that would test it (emitted, never run)")
sp.add_argument("root")
sp.add_argument("--sql-file", required=True, dest="sql_file")
sp.set_defaults(func=cmd_join_probes)

sp = sub.add_parser("filter-values", help="every value typed into a filter: `plan` emits what to probe, `judge` grades what came back")
modes = sp.add_subparsers(dest="mode", required=True)
mp = modes.add_parser("plan", help="what the semantic model knows about each typed value, and the probes to run")
mp.add_argument("root")
mp.add_argument("--sql-file", required=True, dest="sql_file")
mp.set_defaults(func=cmd_filter_values_plan)
mj = modes.add_parser("judge", help="grade each typed value from the probe CSVs the tier returned")
mj.add_argument("root")
mj.add_argument("--plan", required=True, help="the plan JSON `filter-values plan` printed")
mj.add_argument("--results", required=True, help="directory of <id>.<probe>.csv files the tier returned")
mj.set_defaults(func=cmd_filter_values_judge)

sp = sub.add_parser("review-queue", help="trust-review items needing sign-off (Rule 1/2)")
sp.add_argument("root")
sp.set_defaults(func=cmd_review_queue)
Expand Down
41 changes: 40 additions & 1 deletion packages/agami-core/src/semantic_model/comparator.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,16 @@

from __future__ import annotations

import csv
import io
import math
import re
from collections import Counter
from collections.abc import Sequence
from dataclasses import dataclass
from datetime import date, datetime, timezone
from decimal import Context, Decimal
from pathlib import Path
from typing import Any, Literal, NamedTuple, Optional

import sqlglot
Expand Down Expand Up @@ -666,8 +669,44 @@ def compare_result_sets(
)


# A number as the execute_sql CSV wire spells it. Deliberately narrow: a digit string with a leading
# zero (`007`, `02134`) stays text, because a padded id or a postal code that reads as a number would
# compare equal to its unpadded twin, and a text column that happens to hold digits is text.
_NUMERIC_TEXT = re.compile(r"^-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][-+]?\d+)?$")


def result_from_csv(path: str | Path) -> ExecResult:
"""The execute_sql CSV wire read back into an ``ExecResult``, for a comparison of two files.

The wire lost every type, so this puts back the two it can without guessing: numeric text
becomes ``Decimal`` (the shape ``_canonical_number`` normalises), and an empty cell becomes
``None``, which is what the wire writes for a NULL. Everything else stays text and is keyed by
``_canonical_text`` like any other string. A digit string with a leading zero stays text on
purpose; see ``_NUMERIC_TEXT``.

A zero-byte file raises rather than reading as an empty result: the execution tier writes CSV
only on success, so an empty file is a statement that was refused or failed, and scoring it as
"zero rows" would turn a failed run into a wrong answer.
"""
text = Path(path).read_text(encoding="utf-8")
if not text.strip():
raise ValueError(f"{path} is empty; the statement that should have written it did not succeed")

def cell(value: str):
if value == "":
return None
if _NUMERIC_TEXT.match(value):
return Decimal(value)
return value

rows = list(csv.reader(io.StringIO(text)))
return ExecResult(columns=rows[0], rows=[tuple(cell(v) for v in row) for row in rows[1:]])


# The scoring call and the value it hands back, and nothing else. The rest of this module is how
# the two are built rather than what a caller is invited to reach for; `MatchLevel` and
# `GoldenBounds` stay out because they belong to `golden`, which is where a caller should take them
# from rather than through here.
# from rather than through here. `result_from_csv` is reached by name by the one CLI verb that
# compares two files, and stays off this list on purpose: it is a reader for one wire, not part of
# what the comparator promises.
__all__ = ["ItemScore", "compare_result_sets"]
14 changes: 14 additions & 0 deletions packages/agami-core/src/semantic_model/dialects.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,20 @@ def header_sql(self, schema: Optional[str], table: str) -> str:
# Universal zero-row describe — returns the header on every dialect.
return f"SELECT * FROM {self.qualified(schema, table)} WHERE 1=0"

def limited(self, select: str, n: int) -> str:
"""`select` bounded to `n` rows in this dialect's own row-limit syntax.

The one place the TOP / FETCH FIRST / LIMIT switch is spelled for a statement built
elsewhere. `TOP` goes after `DISTINCT` when there is one: T-SQL reads `SELECT DISTINCT TOP n`
and rejects `SELECT TOP n DISTINCT`.
"""
if self.limit_style == "top":
head = "SELECT DISTINCT " if select.startswith("SELECT DISTINCT ") else "SELECT "
return select.replace(head, f"{head}TOP {n} ", 1)
if self.limit_style == "fetch":
return f"{select} FETCH FIRST {n} ROWS ONLY"
return f"{select} LIMIT {n}"

def count_distinct_sql(self, schema: Optional[str], table: str, column: str) -> str:
q = self.qualified(schema, table)
c = self.quote_ident(column)
Expand Down
40 changes: 40 additions & 0 deletions packages/agami-core/src/semantic_model/golden_claims.py
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,46 @@ def _temporal_bounds(
return column, [(side, value, inclusive)]


_TEMPORAL_NODE_TYPES = tuple(
t for t in (getattr(exp, name, None) for name in (
"CurrentDate", "CurrentTimestamp", "CurrentTime", "Interval", "DateTrunc", "TimestampTrunc",
"DateAdd", "DateSub", "DateDiff", "TsOrDsToDate", "StrToDate", "StrToTime", "DateStrToDate",
"TimeStrToDate", "Extract", "Year", "Month", "Day", "Date", "Timestamp", "UnixToTime",
)) if t is not None
)
_TEMPORAL_CAST_TYPES = {"DATE", "DATETIME", "TIMESTAMP", "TIMESTAMPTZ", "TIMESTAMPLTZ", "TIMESTAMPNTZ", "TIME"}


def count_temporal_predicates(sql: str, *, dialect: str) -> Optional[int]:
"""How many of the statement's filtering conjuncts speak of time. None when it cannot be read.

A conjunct speaks of time when `_temporal_bounds` reads it, or when it carries an ISO date
literal, a date or time function, an INTERVAL, or a cast to a temporal type. The count is
deliberately generous: zero is the only value a caller may lean on, and it says the statement
wrote no date filter in any spelling this module recognises. That is what separates a
`date_window` that reads `unknown` because neither statement filtered on a date (nothing to
disagree about) from one that reads `unknown` because a window was written in a shape the
resolver does not fold (still open).
"""
tree, _why = rt._parse_reporting(sql, dialect=dialect)
if tree is None or not isinstance(tree, exp.Select):
return None
select = rt._fold_unquoted_identifiers(tree)
return sum(1 for conjunct in rt._filtering_conjuncts(select)
if _temporal_bounds(conjunct) is not None or _speaks_of_time(conjunct))


def _speaks_of_time(node: "exp.Expression") -> bool:
if _TEMPORAL_NODE_TYPES and any(True for _ in node.find_all(*_TEMPORAL_NODE_TYPES)):
return True
for cast in node.find_all(exp.Cast, exp.TryCast):
to = cast.args.get("to")
kind = getattr(getattr(to, "this", None), "value", None) or str(getattr(to, "this", ""))
if str(kind).upper() in _TEMPORAL_CAST_TYPES:
return True
return any(lit.is_string and _ISO_DATE.match(lit.this) for lit in node.find_all(exp.Literal))


def _date_literal(node: "exp.Expression | None") -> Optional[str]:
"""The ISO date a node spells, as written — None when it spells anything else.

Expand Down
Loading
Loading