Skip to content

Commit 2f5b62e

Browse files
bzgbzg
committed
feat!: Key Message-Id identity and lookups on a fixed-length hash
Breaking-Users: Rebuild the database from scratch as the unique identity moved to the *-hash attrs and :vote/key changed format.
1 parent 5aa15b3 commit 2f5b62e

16 files changed

Lines changed: 449 additions & 525 deletions

‎resources/bone-schema.edn‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,12 @@
1212
;; schema level.
1313
:email/id {:db/valueType :db.type/string}
1414
:email/source {:db/valueType :db.type/string}
15-
:email/message-id {:db/valueType :db.type/string
15+
;; Raw Message-Id, kept verbatim for display/export. NOT unique:
16+
;; identity and lookups go through the fixed-length hash below (raw
17+
;; mids can exceed LMDB's key limit).
18+
:email/message-id {:db/valueType :db.type/string}
19+
;; SHA-256 hex of the raw mid (bone.common/mid-hash).
20+
:email/message-id-hash {:db/valueType :db.type/string
1621
:db/unique :db.unique/identity}
1722
:email/subject {:db/valueType :db.type/string}
1823
:email/from-name {:db/valueType :db.type/string}
@@ -48,10 +53,10 @@
4853
:attachment/data {:db/valueType :db.type/string}
4954
:email/in-reply-to {:db/valueType :db.type/string}
5055
:email/references {:db/valueType :db.type/string}
51-
;; Ancestors-mids: persisted union of References + In-Reply-To,
52-
;; one entry per ancestor message-id. Indexed multi-valued so the
53-
;; pending-thread retry can locate emails sharing a thread in O(hits).
54-
:email/ancestor-mids {:db/valueType :db.type/string
56+
;; Mid-hashes of References + In-Reply-To, one entry per ancestor;
57+
;; used by the pending-thread retry. Ordered raw ancestors are
58+
;; recomputed from the headers (digest/ancestor-mids).
59+
:email/ancestor-mid-hashes {:db/valueType :db.type/string
5560
:db/cardinality :db.cardinality/many}
5661
;; Set when an email is ingested while its In-Reply-To target is
5762
;; absent from the DB. Phase 3/4 of process-email! is skipped on
@@ -83,7 +88,10 @@
8388
;; email's author-address when a maintainer uses the `-by` form.
8489
:report/type {:db/valueType :db.type/keyword}
8590
:report/email {:db/valueType :db.type/ref}
86-
:report/message-id {:db/valueType :db.type/string
91+
;; Raw root Message-Id, kept verbatim for display/export; identity
92+
;; and lookups go through the hash (as for :email/message-id).
93+
:report/message-id {:db/valueType :db.type/string}
94+
:report/message-id-hash {:db/valueType :db.type/string
8795
:db/unique :db.unique/identity}
8896
:report/version {:db/valueType :db.type/string}
8997
:report/patch-seq {:db/valueType :db.type/string}
@@ -150,7 +158,7 @@
150158

151159
;; --- Votes (one entity per voter per report) ---
152160
:vote/key {:db/valueType :db.type/string
153-
:db/unique :db.unique/identity} ;; "report-mid:voter-addr"
161+
:db/unique :db.unique/identity} ;; "report-mid-hash:voter-addr"
154162
:vote/report {:db/valueType :db.type/ref}
155163
:vote/email {:db/valueType :db.type/ref}
156164
:vote/value {:db/valueType :db.type/keyword} ;; :up :down :null

‎scripts/bone-export.clj‎

Lines changed: 11 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1865,18 +1865,12 @@
18651865
(delete-dir! (io/file staging))
18661866
;; Seed staging with the previous export so an
18671867
;; incremental run keeps the files it does not
1868-
;; rewrite: reports/ (per-type files for
1869-
;; unchanged types, via :changed-types) and the
1870-
;; per-mid patches/, text/ and events/ files for
1871-
;; reports unchanged since last-export (via
1872-
;; :since). Aggregate files are always rebuilt.
1873-
;; Seed on every incremental run, not
1874-
;; only when src-changed: :since is
1875-
;; passed below whenever incremental?,
1876-
;; so unseeded per-mid files would be
1877-
;; lost in the swap (e.g. a source
1878-
;; exported for a maintainers-only
1879-
;; change).
1868+
;; rewrite (per-type reports/, per-mid patches/,
1869+
;; text/, events/); aggregates are always
1870+
;; rebuilt. Seed on EVERY incremental run:
1871+
;; :since is passed whenever incremental?, so
1872+
;; unseeded per-mid files would be lost in the
1873+
;; swap.
18801874
(when incremental?
18811875
(doseq [sub ["reports" "patches" "text" "events"]]
18821876
(copy-dir! (io/file final-dir sub)
@@ -1910,16 +1904,11 @@
19101904
(when (and (#{"all" "root"} format)
19111905
(or (= format "root") (seq exported-srcs)))
19121906
(dump-root-index! (mapv :name (:sources config))))
1913-
;; Cron notification: a single concise stderr line when real work
1914-
;; was published, so the cron mail fires iff new info was exported
1915-
;; (routine "Wrote ..." progress now goes to stdout). On an
1916-
;; incremental run we name the report types changed in the DB per
1917-
;; source (before per-source export filters, so a count may exceed
1918-
;; what gets published); a full re-export is flagged as such.
1919-
;; Incremental runs triggered solely by thread replies (re-export
1920-
;; needed, but no report added or modified) stay silent: we only
1921-
;; mail when a source has a genuine add/modification, or on a full
1922-
;; re-export.
1907+
;; Cron notification: one stderr line iff real work was
1908+
;; published ("Wrote ..." progress goes to stdout). Names
1909+
;; the changed report types per source (pre-filter counts);
1910+
;; full re-exports are flagged; runs triggered solely by
1911+
;; thread replies stay silent.
19231912
(let [summaries (into {}
19241913
(keep (fn [s]
19251914
(when-let [sm (fmt-change-summary

‎scripts/bone-notify.clj‎

Lines changed: 10 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
'[bone.common :refer [get-header format-date format-date-iso
2828
report-priority report-status report-descendant-count
2929
load-config db-path build-source-map
30-
bone-schema
30+
bone-schema mid-hash
3131
failures-file-path read-failures-file
3232
reason-labels]]
3333
'[bone.common-bb :refer [load-datalevin-pod! all-reports]])
@@ -38,13 +38,9 @@
3838
(require '[pod.tzzh.mail :as mail])
3939

4040
;; ---------------------------------------------------------------------------
41-
;; Per-(subscriber, source) failures-shown tracking
42-
;; ---------------------------------------------------------------------------
43-
;;
44-
;; We persist the timestamp of the last successful notification per
45-
;; (subscriber, source) pair to avoid re-sending the same failure on
46-
;; every run. This is operational state, not configuration --
47-
;; subscribers and filters live in config.edn.
41+
;; Per-(subscriber, source) failures-shown tracking: timestamp of the
42+
;; last successful notification, so failures are not re-sent on every
43+
;; run. Operational state, not configuration.
4844

4945
(def ^:private last-failures-file "data/.last-notify-failures.edn")
5046

@@ -113,15 +109,14 @@
113109
false)))))))
114110

115111
(defn- report-subject-by-mid
116-
"Look up the report's email subject from its message-id."
112+
"Look up the report's email subject from its message-id. entid +
113+
pull, never a d/q value seek on the hash attr (same convention as
114+
the JVM-side bone.lookup ns; the pod cannot require it)."
117115
[db mid]
118116
(when (and mid (not (str/blank? mid)))
119-
(d/q '[:find ?subj .
120-
:in $ ?mid
121-
:where [?r :report/message-id ?mid]
122-
[?r :report/email ?e]
123-
[?e :email/subject ?subj]]
124-
db mid)))
117+
(when-let [r (d/entid db [:report/message-id-hash (mid-hash mid)])]
118+
(-> (d/pull db [{:report/email [:email/subject]}] r)
119+
:report/email :email/subject))))
125120

126121
;; ---------------------------------------------------------------------------
127122
;; Report formatting

‎scripts/bone/common_bb.clj‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,8 @@
4444
Returns the attachment-pull-pattern projection, or nil."
4545
[db message-id]
4646
(when message-id
47-
(dpull db common/attachment-pull-pattern [:report/message-id message-id])))
47+
(dpull db common/attachment-pull-pattern
48+
[:report/message-id-hash (common/mid-hash message-id)])))
4849

4950
(defn get-tenures
5051
"Fetch all maintainer tenures (active and closed) for `source-name`.

0 commit comments

Comments
 (0)