Skip to content

Commit 22a7d8c

Browse files
vijay-tylertechvijay-tylertech
committed
FAQ harvesting: Teams -> Coda review -> FAQ-<Domain>.md
Recurring questions get answered in Teams by the people who own the answer, and then the answer is lost. This adds the loop that catches them, with a human approval gate in Coda so nothing unreviewed reaches Foundry. Half agent, half script, because reading a Teams *channel* needs the Microsoft 365 MCP tool teams_list_channel_messages, which is available to Claude in a session but not to a plain Python process - there are no Graph app credentials on this machine. So extraction and dedupe are agent work; push, pull and delete are deterministic. scripts/coda_faq_review.py status / push / pull / delete scripts/faq-harvest/HARVEST.md the loop, the sources, the rules extraction follows scripts/faq-harvest/TABLE-SPEC.md column spec and what the CSV import actually produced team-config/faq-harvest/ the first batch, the import CSV, the ledger The Aligned Releases queue is live: 21 candidates in Coda grid-4mdZMRDPAE, awaiting review. Two of them contradict entries the agent is answering from today - cohort unassignment and the feature-flag metadata field - and are flagged as such rather than applied. Three Coda behaviours the script exists to absorb, all verified rather than assumed: - The REST API cannot create a table, and cannot add or remove a column. A markdown table in page canvasContent becomes static text, not a grid. Tables are made by CSV import, which is why the CSV filename has to be the intended table name. - Writes are asynchronous. A probe page took ~20s to disappear. Every mutation polls /mutationStatus to completion, so "delete the processed row" cannot half-happen. - CSV import coerces types: Source link became a link column (safe, full URL preserved) and the date column reads back as ...T00:00:00.000-05:00, so it must never be string-compared. A checkbox is the only gate, and there is deliberately no Status column: rows do not survive processing, so a status written on one would never be read again. Decisions live in team-config/faq-harvest/ledger-<domain>.json instead, which outlives the row. Rejection is expressed by deleting the row - push notices a key it previously sent has gone and records it as rejected. The ledger tracks table_id for that reason too, since rebuilding the table is the normal way a column change happens and would otherwise look like the reviewer having deleted every row at once. No credentials: CODA_API_TOKEN comes from the environment and the token is not in any committed file. Claude-Session: https://claude.ai/code/session_01KxA7f912exehyUjjgVBgf4
1 parent 62e2b84 commit 22a7d8c

7 files changed

Lines changed: 1083 additions & 0 deletions

File tree

‎scripts/coda_faq_review.py‎

Lines changed: 505 additions & 0 deletions
Large diffs are not rendered by default.

‎scripts/faq-harvest/HARVEST.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# FAQ harvesting — Teams → Coda review → `FAQ-<Domain>.md`
2+
3+
Recurring questions get answered in Teams by the people who own the answer, and then the answer
4+
is lost. This loop catches them and lands the confirmed ones in the corpus, with a human
5+
approval gate in Coda so nothing unreviewed reaches Foundry.
6+
7+
## Why it is half agent, half script
8+
9+
Reading a Teams **channel** requires the Microsoft 365 MCP tool `teams_list_channel_messages`.
10+
That tool is available to Claude inside a session; it is not available to a plain Python
11+
process, because there are no Graph app credentials on this machine. Verified working
12+
2026-09-08 against both Aligned Releases channels, including threaded replies and pagination.
13+
14+
So the split is:
15+
16+
| Step | Who | How |
17+
|---|---|---|
18+
| 1. Read the channels | **agent** | `teams_list_channel_messages`, top-level then `parentMessageId` for each Q&A thread |
19+
| 2. Extract candidates, dedupe against the FAQ, classify conflicts | **agent** | judgement — this is the part that cannot be a script |
20+
| 3. Push to the Coda review table | script | `coda_faq_review.py push` |
21+
| 4. Approve / reject | **human** | tick **Ready for Processing** in Coda; reject by deleting the row |
22+
| 5. Pull the ticked rows | script | `coda_faq_review.py pull` |
23+
| 6. Index into `FAQ-<Domain>.md` | **agent** | the FAQ entry format, in the right place in the file |
24+
| 7. Delete the row, record the key | script | `coda_faq_review.py delete` |
25+
26+
Steps 3, 5 and 7 are deterministic on purpose: they are the ones that must not half-happen.
27+
28+
## Sources
29+
30+
### Aligned Releases → `Knowledge-AlignedReleases/FAQ-AlignedReleases.md`
31+
32+
| Channel | teamId | channelId |
33+
|---|---|---|
34+
| Cloud Living – FACC / General | `969b039c-b765-48ea-a12e-973d42c45c17` | `19:S23ooVbxKDBsZ_X3OhvHtDlWNLSDG8G68CUaclunpPk1@thread.tacv2` |
35+
| Aligned Releases | `20442058-b54b-4742-a4ec-49c7f40764f8` | `19:1d8562afa885458ea35bf410ba7d50d9@thread.tacv2` |
36+
37+
Coda review page: **FAQs - Aligned Releases**, `canvas-xQGl2GpyJF`.
38+
39+
### Status Pages / SLAs → `Knowledge-StatusPageAndSLA/FAQ-StatusPageAndSLA.md`
40+
41+
| Channel | teamId | channelId |
42+
|---|---|---|
43+
| SPARC – Working Group / General | `97bf1b86-985d-4f26-ae13-85361c039c7d` | `19:PK1E_uJ3uwfVuu_-00XH83InYFnFFUiq3t9xU4NC5xQ1@thread.tacv2` |
44+
45+
Coda review page: **FAQs - Status Pages**, `canvas-fDCK9ni2hA`.
46+
47+
> **This domain has nowhere to deploy yet.** `Knowledge-StatusPageAndSLA/` is a scaffolded
48+
> corpus with no Foundry agent and no collection. Harvesting and Coda review work; step 6 lands
49+
> the entry in a file that nothing retrieves until the agent is stood up. That is fine — just do
50+
> not report an entry as live.
51+
52+
Tenant for all channels: `7cc5f0f9-ee5b-4106-a62d-1b9f7be46118`.
53+
54+
## Running it
55+
56+
```bash
57+
export CODA_API_TOKEN=... # REST token from ~/.../Repo/claude/coda-details.md
58+
python3 scripts/coda_faq_review.py status --domain AlignedReleases
59+
```
60+
61+
Then ask Claude to harvest. The agent writes a batch to
62+
`team-config/faq-harvest/candidates-<domain>-<date>.json`, and:
63+
64+
```bash
65+
python3 scripts/coda_faq_review.py push --domain AlignedReleases --dry-run \
66+
team-config/faq-harvest/candidates-alignedreleases-2026-09-08.json
67+
python3 scripts/coda_faq_review.py push --domain AlignedReleases \
68+
team-config/faq-harvest/candidates-alignedreleases-2026-09-08.json
69+
```
70+
71+
After review:
72+
73+
```bash
74+
python3 scripts/coda_faq_review.py pull --domain AlignedReleases > /tmp/ready.json
75+
# agent indexes them into FAQ-AlignedReleases.md, updates _START_HERE.md
76+
python3 scripts/coda_faq_review.py delete --domain AlignedReleases --keys ar-123,ar-456
77+
```
78+
79+
`pull` prints the exact `delete` command for the rows it returned, so the two halves cannot
80+
drift apart. `delete` records each key in the ledger, which is what stops the next harvest
81+
re-proposing it — the table holds only outstanding items and therefore cannot be the memory.
82+
83+
**Rejection has no state of its own: delete the row in Coda.** The next `push` sees that a key
84+
it previously sent has gone and records it as rejected.
85+
86+
## Rules the harvest step follows
87+
88+
These come from `FAQ-AlignedReleases.md`'s own "What belongs here" section and from
89+
`CLAUDE.md`. They are the reason this is not just a scraper.
90+
91+
1. **Only what exists nowhere else.** If the answer is already in Blueprint or a Confluence
92+
page, it belongs in that source's file — or nowhere — not in the FAQ. `Docusaurus-` and
93+
`Conf-` files are re-derived from upstream and would silently lose a hand-added entry.
94+
2. **Never add an unconfirmed claim as an answer.** Every entry carries Source / Added /
95+
Confidence / Promote when. "Nobody has decided yet" is a legitimate entry and is kept
96+
deliberately — without it the agent invents a process. A guess is not.
97+
3. **Conflicts do not get applied.** A candidate that contradicts a live entry goes to the
98+
review table with the existing question named in `Conflicts with` and a `Reviewer notes`
99+
explanation of how the two might reconcile. Never quietly overwrite.
100+
101+
**A provisional candidate is flagged in `Reviewer notes`**, prefixed
102+
`PROVISIONAL — confirm before publishing.` The table has no Confidence column, so that
103+
prefix is the half of confidence a reviewer has to act on.
104+
4. **Ticket and permission questions are not FAQ entries.** Per the team README's routing rule
105+
0, "which form do I file / how do I request access" is answered from
106+
`Knowledge-Shared/Conf-OneTylerTickets.md` — one catalog, so copies cannot drift. Harvest
107+
them, flag the routing, let the reviewer decide.
108+
5. **Bug reports are not FAQs.** "Both tabs are highlighted when I navigate" is a defect, not a
109+
question with an answer. Drop it.
110+
6. **Attribute to a person and a date.** "Someone in the channel said" is not a source.
111+
7. **No customer conversation data, no credentials, ever** — see `.gitignore` and
112+
`Knowledge-*/`'s deployment surface.
113+
114+
## Landing the entry
115+
116+
Step 6 is a normal corpus change, so the normal rules apply:
117+
118+
- Update the corpus `_START_HERE.md` in the same change if the file's coverage shifts.
119+
- Nothing reaches Foundry until it is merged to `main`. Run `scripts/preflight_upload.py` on
120+
the changed file, then `scripts/check_foundry_drift.py` after the merge.
121+
- Back up the Foundry object before any write, into `team-config/backups/`.
122+
123+
## Cadence
124+
125+
Currently **on request** — ask Claude to run a harvest. The channels are low-volume: the
126+
Aligned Releases channel produced 26 top-level messages between 2026-05-11 and 2026-09-03, so
127+
a sweep every few weeks comfortably keeps up, and a weekly one would usually find nothing.
128+
129+
If this should run unattended, it has to be a scheduled *Claude* job rather than a cron entry,
130+
because step 1 needs the MCP tool. Nothing is scheduled today.

‎scripts/faq-harvest/TABLE-SPEC.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Review table spec
2+
3+
`scripts/coda_faq_review.py` can read rows, upsert rows and delete rows. It **cannot create
4+
the table**, and it cannot add or remove columns — the Coda REST API has no endpoint for
5+
either. Columns are a UI job, once.
6+
7+
Verified 2026-09-08: a markdown table passed as page `canvasContent` renders as *static text*,
8+
not a grid. A probe page was created with a three-column markdown table and no new table
9+
appeared under `GET /docs/{docId}/tables`. So there is no API trick — import the CSV.
10+
11+
## The quick route: import the CSV
12+
13+
`team-config/faq-harvest/FAQ Review Queue (Aligned Releases).csv` carries the header row and
14+
one disposable sample row, so Coda infers the whole schema.
15+
`team-config/faq-harvest/README-import.md` says which page each CSV goes to.
16+
17+
## Where
18+
19+
| Domain | Coda page | Page id | Table |
20+
|---|---|---|---|
21+
| Aligned Releases | **FAQs - Aligned Releases** | `canvas-xQGl2GpyJF` | rebuilt 2026-09-09 for the 8-column layout — id in `coda_faq_review.py` |
22+
| Status Pages / SLAs | **FAQs - Status Pages** | `canvas-fDCK9ni2hA` | not built |
23+
24+
Both pages are in the **OneTyler Initiatives Trackers** doc, `KV_6fSnfBc` —
25+
<https://docs.superhuman.com/d/_dKV_6fSnfBc>.
26+
27+
## Columns
28+
29+
Eight, in the order a reviewer reads them. **Names must match exactly** — the script addresses
30+
columns by name (`useColumnNames=true`), so a renamed column silently stops being written.
31+
32+
| # | Column | Type | Who fills it | Notes |
33+
|---|---|---|---|---|
34+
| 1 | `Date` | date | harvester | When the candidate was harvested. Coda coerces `2026-09-09` to `2026-09-09T00:00:00.000-05:00`, so never string-compare it. |
35+
| 2 | `Key` | Text | harvester | Stable id, `ar-<teams-message-id>`. The upsert key, so a repeat harvest updates rather than duplicates. Do not edit by hand. |
36+
| 3 | `Question` | Text | harvester | Phrased the way a user would ask it. |
37+
| 4 | `Answer` | Text | harvester | Proposed answer in the FAQ's markdown style. **Edit this in place** — the loop indexes the cell as it reads when the box is ticked, not the original harvest. |
38+
| 5 | `Source` | Text | harvester | Who said it and when — a person and a date, per FAQ policy. |
39+
| 6 | `Source link` | link | harvester | Permalink to the Teams message. Verified to preserve `?groupId=…&tenantId=…` intact. |
40+
| 7 | `Notes` | Text | both | Everything the reviewer needs flagged, assembled by `build_notes()`. Empty when there is nothing to say — 11 of the first 21 candidates had an empty cell. |
41+
| 8 | `Ready for Processing` | **checkbox** | **reviewer** | **The only gate.** See below. |
42+
43+
### What `Notes` carries
44+
45+
Two things lost their own column but must still reach a reviewer, so they are prefixed onto
46+
`Notes` in this order:
47+
48+
- **`CONFLICTS WITH the live entry “…”.`** The candidate contradicts or refines an entry the
49+
agent is already answering from. This is the flag that stops a conflict being applied
50+
silently, so it must never be dropped — it is the reason `conflicts_with` survives at all.
51+
- **`PROVISIONAL — confirm before publishing.`** The source was not authoritative. FAQ policy
52+
is that an unconfirmed claim must not be published as fact, because the agent will state it
53+
as one.
54+
55+
Then the harvester's own note, if any — a routing question, a reconciliation suggestion, a
56+
recommendation to reject.
57+
58+
## What is deliberately NOT a column
59+
60+
The table is a **review surface, not the record.** It carries only what a human reads or
61+
edits. Everything below is tracked in the candidates JSON and written into the FAQ entry at
62+
indexing time. Reviewers were being asked to read a column per FAQ field, which made the
63+
thing harder to read for no decision gained.
64+
65+
| Not a column | Why | Where it lives |
66+
|---|---|---|
67+
| `Status` | Redundant once the checkbox is the gate. A row does not survive processing, so an Approved/Rejected value on it would never be read again. | the ledger, which outlives the row |
68+
| `Type` | `New` / `Conflict` / `Refinement` / `Duplicate` was review triage; `Notes` says it in plain words. | candidates JSON |
69+
| `Confidence` | Required by the FAQ **entry** format, but needs no reviewer input. | JSON; the actionable half is the `PROVISIONAL` prefix |
70+
| `Promote when` | The entry's *exit condition* — which upstream doc should eventually carry the answer, so the entry can be retired rather than accumulating forever. Useful in the file, nothing to decide. | candidates JSON |
71+
| `Conflicts with` | Still essential, but as a sentence rather than a column. | the `CONFLICTS WITH` prefix on `Notes` |
72+
73+
## How the reviewer uses it
74+
75+
**Tick `Ready for Processing`** and nothing else is required. Edit `Answer` first if it needs
76+
changing.
77+
78+
**To reject a candidate, just delete the row.** There is no Rejected state. The next `push`
79+
notices that a key it previously sent is gone from the table and not in the ledger, records it
80+
as `rejected (row removed in Coda)`, and never proposes it again.
81+
82+
**To defer**, leave the box unticked. Unticked rows are invisible to `pull` and are not
83+
re-proposed by a later harvest, because they are already present.
84+
85+
`delete` refuses to remove a row that is not ticked unless `--force` is passed — deleting an
86+
unreviewed row loses the candidate for good, since the ledger then suppresses it.
87+
88+
## Why the ledger exists
89+
90+
`team-config/faq-harvest/ledger-<domain>.json`, committed on purpose:
91+
92+
- `pushed` — every key ever sent to the table. Needed to tell "you rejected this" apart from
93+
"this was never proposed".
94+
- `decided` — what happened to it (`indexed`, `rejected (row removed in Coda)`, `discarded`).
95+
- `table_id` — which table `pushed` refers to. **Rebuilding the table is the normal way a
96+
column change happens**, since the Coda API can neither rename nor delete a column. Without
97+
this field, pointing the script at a fresh empty table would make every previously-pushed
98+
key look deleted and be recorded as a rejection wholesale. `reconcile` checks it first and
99+
treats a mismatch as a rebuild.
100+
101+
The table only ever holds outstanding items, which is the point — so the table cannot itself
102+
remember that something was already handled. Without this file every harvest re-proposes
103+
everything already dealt with.
104+
105+
## A trap worth knowing
106+
107+
Coda writes are **asynchronous**. A POST or DELETE returns a `requestId` and the change is not
108+
visible for a few seconds — a probe page took about 20 seconds to disappear.
109+
`coda_faq_review.py` polls `/mutationStatus/{requestId}` to completion on every mutation, so
110+
"delete the processed row" either finishes or reports that it did not. Do not add a code path
111+
that fires a write and moves on.

‎team-config/faq-harvest/FAQ Review Queue (Aligned Releases).csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
"Date","Key","Question","Answer","Source","Source link","Notes","Ready for Processing"
2+
"2026-09-09","SAMPLE-delete-me","IMPORT TARGET: page 'FAQs - Aligned Releases' (canvas-xQGl2GpyJF) in OneTyler Initiatives Trackers (KV_6fSnfBc). Domain: Aligned Releases. Delete this row once imported.","Answer text goes here, in the FAQ's markdown style. Edit this cell in place - what it says when you tick the box is what gets indexed, not the original harvested text.","Sample row generated for table setup, 2026-09-09","https://example.invalid/sample","Feeds Knowledge-AlignedReleases/FAQ-AlignedReleases.md. Tick Ready for Processing when the answer is right - that is the only signal read. To reject a candidate, just delete the row.","false"

‎team-config/faq-harvest/README-import.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Importing the review-table CSVs
2+
3+
One CSV per domain. Each creates that domain's FAQ review queue table.
4+
5+
| CSV | Import to Coda page | Page id | Feeds |
6+
|---|---|---|---|
7+
| `FAQ Review Queue (Aligned Releases).csv` | **FAQs - Aligned Releases** | `canvas-xQGl2GpyJF` | `Knowledge-AlignedReleases/FAQ-AlignedReleases.md` |
8+
9+
**Aligned Releases needs a re-import** — the CSV was regenerated on 2026-09-09 for the
10+
simplified 8-column layout (`Date`, `Key`, `Question`, `Answer`, `Source`, `Source link`,
11+
`Notes`, `Ready for Processing`). The Coda API cannot rename or delete a column, so a column
12+
change means a fresh import rather than an edit. The old table `grid-4mdZMRDPAE` can be
13+
deleted once the new one is in; no review work is lost, because no row was ever ticked.
14+
| _(not yet generated)_ | **FAQs - Status Pages** | `canvas-fDCK9ni2hA` | `Knowledge-StatusPageAndSLA/FAQ-StatusPageAndSLA.md` |
15+
16+
Both pages are in the **OneTyler Initiatives Trackers** doc, `KV_6fSnfBc` —
17+
<https://docs.superhuman.com/d/_dKV_6fSnfBc>.
18+
19+
The two tables are near-identical in shape, so the destination is also written into each
20+
CSV's sample row: open the file and the `Question` cell names the target page and page id.
21+
That is deliberate — the **filename has to stay the intended table name**, because Coda names
22+
an imported table after the file it came from, so the page cannot be put in the filename
23+
without corrupting the table name.
24+
25+
## After importing
26+
27+
1. ~~Delete the `SAMPLE-delete-me` row.~~ (done for Aligned Releases)
28+
2. Nothing to configure: `Ready for Processing` imports as an unticked checkbox, which is
29+
exactly the wanted starting state.
30+
3. Delete the superseded table once the new one is confirmed.
31+
4. ~~Send back the table id.~~ (done: `grid-4mdZMRDPAE`, pinned)
32+
33+
Optionally rename the table to drop the `.csv` suffix Coda took from the filename. Cosmetic —
34+
the script resolves by id.
35+
36+
Then confirm the script can see it:
37+
38+
```bash
39+
export CODA_API_TOKEN=... # REST token from ~/.../Repo/claude/coda-details.md
40+
python3 scripts/coda_faq_review.py status --domain AlignedReleases --table grid-XXXX
41+
```

0 commit comments

Comments
 (0)