You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Include a comparison with Trillian's strong guarantees and justification for the design choice. Also added a few recommendations on configuring the antispam settings, and made it clear that strict deduplication must be handled upstream before writing to Tessera.
Co-authored-by: Al Cutter <alcutter@google.com>
Copy file name to clipboardExpand all lines: docs/design/antispam.md
+25-3Lines changed: 25 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -25,6 +25,14 @@ expensive, and have a profound impact on log performance.
25
25
26
26
If your transparency application needs such properties you will need to handle this in the personality.
27
27
28
+
## Comparison with Trillian's LeafIdentityHash
29
+
30
+
Trillian operators may be familiar with the `LeafIdentityHash` mechanism, which Trillian uses to detect and reject duplicate leaves during queueing.
31
+
32
+
Trillian provided stronger guarantees to prevent duplicates from entering the queue by using a database-level unique constraint on the leaf identity hash. This constraint requires synchronous database lookups and writes during the critical path of submission, which reduced the write throughput and increased the cost of operation.
33
+
34
+
Tessera antispam relaxes these guarantees; it is possible for duplicates to be integrated if they are submitted in close succession to different frontend instances. However, in return, this enables the write path to be cheaper and much more scalable while still providing a defense against unbounded additions from a malicious source.
35
+
28
36
## Overview
29
37
30
38
The anti-spam support is optional, and needs to be explicitly provisioned in the infrastructure and enabled
@@ -126,9 +134,23 @@ anti-spam index will perform better than, e.g., attempting to update the index d
126
134
`SetEntryBundle()`. This is borne out by experimental evidence; early tests on GCP show that this approach delivers twice the
127
135
throughput (tested with 10% dupe traffic) compared with the "competing batched updates" approach briefly described above.
128
136
129
-
## Tuning
137
+
## Operational Considerations
138
+
139
+
### Operator Recommendations
140
+
141
+
When deploying Tessera with antispam enabled, operators should consider:
142
+
143
+
***Cache vs. Lag**: Ensure the in-memory cache capacity (`WithAntispam`) is sized to cover the expected integration latency.
144
+
***Pushback Configuration**: Configure the pushback threshold to protect the database from being overwhelmed if the antispam follower lags. A threshold that is too low may cause unnecessary errors for clients, while too high may render the persistent antispam ineffective.
145
+
***Monitoring**: Operators should monitor:
146
+
*`ErrPushback` rate: High rates indicate the follower cannot keep up with the write rate.
147
+
* Follower lag: Metric showing how far behind the integration point the antispam follower is.
148
+
* Cache hit rate: High hit rate indicates the in-memory cache is effective.
149
+
***Upstream Enforcement**: If strict uniqueness (like Trillian's `LeafIdentityHash`) is required, this must be enforced upstream before writing leaves to Tessera.
150
+
151
+
### Tuning
130
152
131
-
### In-memory cache size
153
+
####In-memory cache size
132
154
133
155
Ideally, the in-memory anti-spam decorator should have a sufficiently large cache to cover the window before newly added entries are
134
156
seen by the follower and added to the persistent anti-spam storage. Fortunately, anti-spam index entries are `32+8` bytes plus overhead,
@@ -137,7 +159,7 @@ so having even a very large cache depth of 100's of 1000's of entries is not exp
137
159
The `tessera.WithAntispam` option allows the capacity of the in-memory cache to be configured, and internally ensures that the in-memory
138
160
cache and persistent anti-spam index are applied in the correct order.
139
161
140
-
### Persistent index
162
+
####Persistent index
141
163
142
164
Applications should configure the push-back threshold according to their expected throughput &
143
165
log performance numbers. A reasonable starting point is probably a few seconds' worth of the peak expected log
0 commit comments