TensorFold Notes for any OpenAI-compatible inference server.
Download Notes Proxy 0.1.0 (.zip): unzip it, then follow Run it from the unzipped folder. TensorFold recommended!
Thank you to Ash and the TensorFold developers. Ash built TensorFold and Living Weights, the neuronal layer of TensorFold Memory. Notes were made for TensorFold, and this proxy brings them to other inference engines too.
The Notes Proxy is a small server that sits between your chat client and the server that runs the model: llama.cpp's llama-server, vLLM, Ollama, LM Studio, SGLang or TensorFold itself. When a chat request passes through it, the proxy looks up what this user has told it before (their notes), puts the notes that fit the question into the request as the first system message, and forwards the request. The model then answers with the notes in front of it. Nothing about the model or its server changes.
The lookup is not a copy. The proxy calls the test suite's own note memory, tests/zengine/notes.py: the same n-gram matching, scoring, hop, 512-token cap and framing that the paper measures and that the TensorFold server's lookup was checked against. For every question the proxy sends exactly the block NoteStore.retrieve() returns, byte for byte (tested below), unless you change its first line with --framing or --abstain-line.
You need Python 3.10 or later and the tokenizers package (pip install tokenizers), and the model's tokenizer.json. The 512-token cap is counted with the model's own tokenizer, as in the paper. Run the commands from this repository's top folder.
The notes are kept in a folder of your own. Make it first, readable by you only:
mkdir -p -m 700 ~/notesIn front of llama.cpp (llama-server -m model.gguf --port 8080 running). A GGUF file has no tokenizer.json: use the one from the model's original Hugging Face repository.
python3 -m notes_proxy --upstream http://127.0.0.1:8080 --owner me --store ~/notes/me.jsonl --tokenizer ~/models/my-model/tokenizer.jsonIn front of vLLM (vllm serve MODEL --port 8000 running):
python3 -m notes_proxy --upstream http://127.0.0.1:8000 --owner me --store ~/notes/me.jsonl --tokenizer ~/.cache/huggingface/hub/models--ORG--MODEL/snapshots/HASH/tokenizer.jsonThe proxy can start TensorFold itself (--tensorfold instead of --upstream). The child is bound to 127.0.0.1, the API key is in TENSORFOLD_API_KEY (never argv), token counts go through the child's /v1/tokenize and piece text through /v1/detokenize, and an erase restarts TensorFold once the proxy is idle, after a quiet window (--tensorfold-quiet, default 10 s), so several erases share one reload. SIGTERM, SIGHUP and process exit stop the child. If traffic never goes idle, new requests are refused after --tensorfold-restart-max (default 60 s) so the restart still happens. GET /v1/notes/erase-status turns pending into complete when that restart has landed. Extra serve flags must be on the allowlist, written with equals so argparse does not eat them: --tensorfold-arg=--no-drafts. --tensorfold with --users is refused unless you also pass --share-upstream-cache: TensorFold has no per-user prompt-cache isolation, so users would share the cache.
python3 -m notes_proxy --tensorfold org/name --owner me --store ~/notes/me.jsonlThen point your chat client at http://127.0.0.1:8090/v1 instead of the server's own address. Add a note and ask about it:
curl -s http://127.0.0.1:8090/v1/notes -H "Content-Type: application/json" -d '{"text": "My sister Anna drives a red Volvo."}'curl -s http://127.0.0.1:8090/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "m", "messages": [{"role": "user", "content": "What car does Anna drive?"}]}'The reply is the server's own, with one more field, notes: the ids of the notes given (see Note ids), their tokens, and whether the question was read as one about the past.
To try it on a test set's notes, make a store from one of the generated sets first:
python3 -m notes_proxy seed --set tests/examples/L6-seed1.json --phase teach --owner user --store /tmp/l6.jsonlPOST /v1/chat/completions. The last user message is the question (a string, or its text parts joined, as the TensorFold server reads it). If the lookup finds notes, the block goes in first as{"role": "system", "content": block}; the client's own messages follow it unchanged. If it finds nothing, nothing is put in. Then the request goes to the upstream server.- Streaming. A streamed reply is passed on line by line as the upstream sends it, never held back. The
notesobject comes as an SSE comment line,: notes {...}, just beforedata: [DONE]. Every SSE parser skips comment lines, so no client meets a chunk it does not expect. The same ids, tokens and past flag are also in the reply headersX-Notes-Ids,X-Notes-TokensandX-Notes-Past. A reply that is not streamed gets thenotesfield added at the end of its JSON; the upstream's bytes before it are left as they were. - Request fields, as the TensorFold server reads them:
"notes": falsesends the request on without notes;"notes_past": true/falseforces the view of the past (absent: the question decides);"notes_exclude": ["w12", ...]leaves the notes with those ids out;"notes_in_weights": trueasks for the notes the pocket holds to ride the block again (a control arm: by default a taught note is left out, since the weights answer it alone);"notes_cap": nsets the cap;"notes_echo": trueputs the block itself in the reply;"notes_write": falsekeeps this message out of the notes."notes_query": "..."looks the notes up with that text, and writes it under--write, in place of the last user message: for a client whose last user message is a wrapper of many lines, such as a room agent. A blank one is ignored; the reply'snotes.question_fromsays which was used. A client that can set headers but not body fields sends it asX-Notes-Query, percent-encoded (the body field wins; an unexpanded${...}is ignored; the header is not forwarded). A field of the wrong type is a 400. The proxy removes these fields before forwarding, so the upstream never sees them. - Every other route (
/v1/models,/v1/completions,/v1/embeddingsand the rest) is passed through as it is. With many users only these three pass (see Many users). - The upstream's API key, if it needs one, is the client's: the
Authorizationheader is passed on as sent. With--upstream-key-env NAMEthe proxy sends the key held in the environment variableNAMEinstead. With many users the client's key is never passed on (see Many users). The proxy never stores a key and never logs one. It logs one line per request: the route, the status, the note ids and times, never a message or a note's text.
| Request | What it does |
|---|---|
GET /v1/notes |
the current notes; ?history=1 also lists the replaced and taken-back ones |
GET /v1/notes/ID |
one note |
POST /v1/notes {"text", "subject"?} |
a new note, through the intake check (201 when written, 200 when it was already known) |
DELETE /v1/notes/ID |
erase: the note is removed from the store, its journal and every cache the proxy holds, as if it had never been said |
GET /v1/notes/erase-status |
with --tensorfold: whether a restart newer than the last erase has finished |
POST /v1/notes/ID/weights |
with --tensorfold: mark the note taught ({"in_weights": true}) or open again (false, optional reason). A mark true is accepted only when the child's own graph shows a learned fact with this note's exact text (a claim of a lesson never taught would otherwise retire the note from the prompt — it is a 409). Only the proxy's operator/promoter calls this; see Living Weights below |
GET /v1/slide/graph |
with --tensorfold and one owner: the child's Living Weights graph, read-only, with each fact mapped to note (the id of your note whose text is the taught sentence, or null) and marked in_weights. The child's own fact ids, states and questions come through as they are. Refused (404) with many users: the pocket is shared, so its whole graph — every user's taught sentences and questions — would go to whoever asks |
An erase also removes the writes that were kept out as the same as that note, and the take-backs of it. With --upstream the erase reply always says "complete": false: the proxy cannot reach the upstream server's prompt cache or logs, and Python does not overwrite memory it frees. With --tensorfold the DELETE replies "pending" at once (it does not wait for the model to reload); GET /v1/notes/erase-status says "complete": true once a restart newer than that erase has finished. Python still does not overwrite memory it frees.
Note ids. A note's id is the id of the write that made it: w1, w2 and so on for notes written through the proxy, and s1, s2 and so on for the lessons a store was seeded with. An id names one note for good. No erase, restart or later write gives it to another note. So an erase sent twice, for example by a client that retried after a timeout, erases the note once. The second reply is a 404, and nothing changes.
The block itself numbers its notes [1], [2] and so on, as notes.py does, and those numbers do move: after an erase, every later note's number goes down by one. So each note in GET /v1/notes has its number next to its id, the chat reply's notes has numbers next to ids in the same order, and the erase reply's renumbered lists the numbers that moved, with each note's id. Use the id, never the number, to name a note.
The id format changed. Before, a note's id was its number. GET /v1/notes/3, DELETE /v1/notes/3, "notes_exclude": [3], the reply's notes.ids and X-Notes-Ids all used numbers, and an erase moved them. A repeated erase then removed a different note, and that cannot be undone. Now ids are text, such as "w3". A number where an id goes is a 400 and changes nothing. A client that kept numbers must read the ids from GET /v1/notes again. The store file did not change: a store made before opens as it is, and its ids come from the writes it already holds. A store in which a write has no id, or two writes share one, is refused at start (the proxy never writes one).
The notes are kept in the --store file, a journal with one write per line. Each write is on disk (synced) before it is answered or used. At startup the journal is replayed, so a restart keeps every note with the same ids. An erase writes the journal anew without the erased writes. The file holds the notes in plain text: keep it where only you can read it (the proxy makes it readable by its owner only).
With --write, the proxy writes notes from what the user says, by the TensorFold server's rule:
- only the conversation's last message, and only when it is the user's own; never a system message, an earlier message or the model's answer (an answer can be wrong, and storing it would keep the mistake);
- only a statement: a message that reads as a question ("Where do I live?", "what car does Anna drive") or a request ("Tell me...", "Remind me...") is skipped, as is anything over one line or 400 bytes. The question check is the server's own (English only), ported to Python;
- through the intake check, after the lookup, before the request is forwarded: the next request finds the note.
A sentence in the wordings the rule book reads ("My favourite colour is green.", "I live in Oslo now.") goes through the rule book's intake check (intake.classify): new, replaces an earlier note, already known, or implied. Any other sentence is written as the TensorFold server writes every chat statement: already known if a current note of the same subject says the same words, otherwise new. The reply's notes.write says what happened, or why the message was skipped.
One proxy can serve many people. Each person is a user with a key of their own. The key decides whose notes a request reads and writes. The client never names the owner.
Make a folder for the users file and the notes, readable by you only:
mkdir -p -m 700 ~/notesAdd each user to a users file. The command prints their new key once, on its own line. Give it to that person.
python3 -m notes_proxy add-user --users ~/notes/users.json --owner alice --store-dir ~/notesStart the proxy with the users file and the notes folder, in place of --owner and --store:
python3 -m notes_proxy --upstream http://127.0.0.1:8080 --users ~/notes/users.json --store-dir ~/notes --tokenizer ~/models/my-model/tokenizer.jsonEach request sends its user's key. An OpenAI client sends it when its API key is set to that key.
curl -s http://127.0.0.1:8090/v1/notes -H "Authorization: Bearer KEY" -H "Content-Type: application/json" -d '{"text": "My sister Anna drives a red Volvo."}'- One store per user. Each user's notes are in
STORE_DIR/NAME.jsonl, opened at that user's first request for notes. Each store has its own index, its own n-gram weights and its own lock, as with one proxy per user. One user's notes never reach another user's block, and never change another user's scores. The upstream is where they could still meet, so the proxy keeps apart there what it can (see below, and What this proxy cannot keep apart). - A missing or unknown key is a 401, before any note is looked up and before anything goes to the upstream. Every route needs a key, also the ones passed through. The one exception is a browser's CORS preflight (
OPTIONSwith an allowedOrigin), which never carries a key. Keys are compared by their SHA-256, in constant time. - The key never goes to the upstream. The proxy drops it from every header. A request that also has the key in its URL or in its body is refused with a 400. The upstream gets the key in the environment variable that
--upstream-key-env NAMEnames, or noAuthorizationat all. /v1/notesand--writeact on the key's own user only. A note id is that user's own: alice'sw1and bob'sw1are two notes, and each key reaches only its own.- Only the model routes pass through. These are
GET /v1/models,POST /v1/completionsandPOST /v1/embeddings, besidesPOST /v1/chat/completions. Any other route is a 404. So no user reaches the upstream's own pages, such as llama.cpp's/slots,/metricsand/props, which can show or change other users' prompts. - The upstream's prompt cache is kept apart. Every block starts with the same framing. An upstream that reuses the start of one prompt for the next would let a user test guesses about another user's notes: send the framing and a guess, then see how much of the prompt was reused. So the proxy sends every chat, completion and embedding request with
"cache_prompt": false, which tells llama.cpp to reuse no cached prompt. It also sends a"cache_salt"of the user's own, which tells vLLM to reuse only that user's cache. Whatever the client sent for these two is replaced. Replies lose llama.cpp'stimingsandprompt_progress, and thecached_tokenscount inusage, because these count the reused tokens. With the cache off, llama.cpp reads the whole prompt every time, so long chats are slower.--share-upstream-cacheturns all of this off. Use it only when every user may know what the others' notes say. - Other servers may ignore both fields. The proxy then cannot keep the users' caches apart, and a user could still time the replies. Run one upstream per user there, or turn its prefix cache off.
- No client names another user's state on the upstream. A field such as
id_slotorprevious_response_id, or a header such asX-Session-Id, could point two users at one slot, session or saved conversation. The table at the end of this section says what the proxy does with each. - The key alone names the owner. A field or header that names one, such as
ownerorX-Owner, never changes whose notes a request reads. - Three of the client's headers go on:
Content-Type,AcceptandUser-Agent. The proxy drops every other one. So no cookie, no other key (api-key,X-Api-Key) and no session or project header reaches the upstream. - No times in the replies. The reply's
notesobject has nolookup_us,total_usorwrite_us. All users share one tokenizer, and its cache makes a word that another user used a little faster to look up. The operator's log keeps the times. - The users file keeps no key. It keeps the SHA-256 of each key. A lost key cannot be shown again. Remove the user with
--keep-notes, then add them again with--reuse-notesfor a new key to the same notes. - A name passes its notes on only when you say so.
add-userrefuses a name whoseNAME.jsonlis still in the store folder, unless--reuse-notessays it is the same person. So a new user never reads an earlier user's notes by accident. - Names are 1 to 32 of
a-z,0-9,_and-, starting with a letter or a digit. Any other name is refused when the user is added and when the users file is read. So no name can point outside the folder. A name has no capital letters, so two names never share a file, even on a disk that ignores case. The file is named by the name itself, not by a hash of it, so you can see whose notes each file holds. - The users file is read once, at start. Restart the proxy after
add-userorremove-user. The proxy refuses to start when another account could change the file, because whoever can change it can read any user's notes.add-userwrites it readable by you only. - Two open files per user. Each user whose notes are open holds two open files until the proxy stops. At start the proxy raises its limit on open files to fit every user, and refuses to start when it cannot (see
ulimit -n). - A store that cannot be opened gives its own user a 503 or a 500, and the cause goes to the operator's log. When a disk step fails and a journal stops taking writes, that user's next request opens it again from disk, as a restart would. The other users are not affected.
Removing a user takes --keep-notes or --erase-notes, one of the two:
python3 -m notes_proxy remove-user --users ~/notes/users.json --owner alice --keep-notespython3 -m notes_proxy remove-user --users ~/notes/users.json --owner alice --erase-notes --store-dir ~/notes--keep-notes leaves alice.jsonl in the folder, for add-user --reuse-notes to give back. --erase-notes deletes that file. It refuses while a proxy serves the folder, so stop the proxy first. It also refuses, and keeps the user, when the folder holds no alice.jsonl, because the folder may be the wrong one. As with any deleted file, the disk may keep the old bytes until they are written over.
A key travels in clear over plain HTTP. Keep the proxy on 127.0.0.1, or put it behind TLS before you use --allow-remote.
An upstream may keep state from one request to the next: a cached prompt, a slot, a session or a saved conversation. A request can name that state by a field in its body or by a header. Two users who sent the same name would share that state. So with many users the proxy does this with each such field, in chat, completion and embedding requests alike:
| Field | In | With many users |
|---|---|---|
cache_prompt |
body | set to false, so llama.cpp reuses no cached prompt |
cache_salt |
body | replaced by the user's own value, also when the client sent none, so vLLM reuses only this user's cache |
prompt_cache_key, user, safety_identifier |
body | replaced by the user's own value, when the client sent one |
id_slot, slot_id |
body | dropped, so the upstream picks the slot |
request_id, rid, session_id |
body | dropped, so the upstream names the request itself |
conversation, previous_response_id, session_params, kv_transfer_params |
body | refused with a 400, because they point at state the upstream keeps and the proxy cannot tell whose it is |
owner, notes_owner |
body | refused with a 400, because the key alone names the owner |
Cookie, api-key, X-Api-Key, X-Session-Id, X-Request-Id, OpenAI-Organization, OpenAI-Project, X-Owner, X-Notes-Owner, X-User |
header | dropped, as is every header but Content-Type, Accept and User-Agent |
id_slot and slot_id are llama.cpp's. cache_salt, request_id and kv_transfer_params are vLLM's. rid and session_params are SGLang's. user, safety_identifier, prompt_cache_key, conversation and previous_response_id are OpenAI's.
The user's own value is the same in all of that user's requests until the proxy restarts, and another in every other user's. The proxy makes it from the user's name and a secret it draws at each start, so no user can work out another's. --share-upstream-cache leaves cache_prompt, cache_salt and prompt_cache_key as the client sent them.
A dropped field changes only where the upstream does the work, or the name it gives the request. The answer stays the same. A refused field would change the answer if it were dropped, so the proxy says so with a 400 instead.
No notes field goes on to the upstream either. The chat route reads its own (see What it does) and refuses any other field whose name starts with notes_. Completions and embeddings refuse every field named notes or starting with notes_. So an upstream that keeps notes of its own, such as a TensorFold server, never gets a request for them.
Living Weights (--slide) teaches facts into the model's own weights. The proxy keeps the two layers as one system, on rules the promotion study (docs/PROMOTION.md) measured:
- Notes stay the record of truth. Every fact is a note the moment it is said, taught or not.
- A taught note leaves the prompt. When the operator's promoter teaches a note's sentence to the pocket and the pocket answers alone, it marks the note (
POST /v1/notes/ID/weights {"in_weights": true}). From then on the note is left out of every block — the weights answer it at zero prompt tokens."notes_in_weights": truebrings the block back for a control arm. A mark true is accepted only when the child's own graph shows a learned fact with that note's exact text: a client cannot retire a note from the prompt by claiming a lesson the pocket never learned (such a claim is a 409). - The marker survives restarts. It is a line in the journal (
{"weights": {...}}), replayed at open in the journal's own order, per owner.GET /v1/notesshows each note'sin_weights; so does the annotatedGET /v1/slide/graph, which maps each learned fact to the note whose text is exactly its taught sentence. - A change demotes at once. When a write supersedes or takes back a taught note, the old note stops being trusted the same instant (its marker goes false,
demotednames it in the write's reply), so the new fact rides the prompt immediately. The stale fact leaves the pocket later:DELETE /v1/slide/fact/{id}on the pocket, never zeroed in place by the proxy. Erasing a note drops its marker with it. - The proxy never teaches.
/v1/slide/learnandDELETE /v1/slide/fact/...are refused through the proxy at every age (one owner or many). Teaching is the operator's promoter, talking to the pocket directly. The child flags are allowlisted:--tensorfold-arg=--slidestarts the child as a pocket (--slide-graph=FILEto keep its graph). The refusal and the--slideallowlist landed in the same change: no released proxy ever let a client reach/v1/slide/learn— before this change the flag was not on the allowlist, so the child never ran Living Weights and the pocket's routes answered 404 from the child itself. - Not with many users, yet. A Living Weights pocket is one pocket: a fact taught for Alice is in Bob's weights too. So
--slidewith--usersstops at start (no allowing flag), andGET /v1/slide/graphanswers 404 with many users (its graph is the whole pocket, texts and questions included, for whoever asks). Until TensorFold keeps a pocket per user, run one proxy per owner.
Minimum TensorFold: 1.0.12. That release shipped DELETE /v1/slide/fact/{id} (per-fact delete, #872); --slide itself needs the release after 1.0.6. Older children answer those routes 501 or lack them, and the proxy will not pretend otherwise.
The nightly promoter that decides which notes have earned teaching is a separate tool (python -m notes_proxy promote); it comes as the second change and, when it keeps answered chats to measure reuse, is opt-in per owner, owner-only files (0600), with a retention limit, erased along with the note and by remove-user --erase-notes.
With many users, the proxy keeps each user's notes apart. The upstream server is still one server for all of them, and the proxy cannot reach inside it. These stay shared:
- Saved conversations and slot files. An upstream may save conversations or responses, and llama.cpp can save a slot to a file. The proxy passes no field that names them and no route that reads them. They are still on the upstream, for anyone who can reach it directly.
- The prompt cache of a server that ignores
cache_promptandcache_salt. Such a server may reuse one user's prompt for another. Run one upstream per user there, or turn its prefix cache off. - Timing. A reply comes faster when the upstream had part of the prompt cached, and slower when it was busy with another user. The proxy takes the cache counts out of the replies. It cannot take out the time.
- The upstream's logs. The upstream may log every prompt it gets, each user's notes block included. Keep its logs as private as the notes.
- Living Weights in TensorFold. A fact that a TensorFold upstream learned with Living Weights is in the model's own weights, and every user's answers come from them. The proxy refuses every
/v1/slideroute except its own annotatedGET /v1/slide/graph(which it refuses too with many users), so no client can teach or delete in the weights through the proxy (with--tensorfoldthe child's key rides on every relayed request, which is exactly why no slide route may pass). Separate notes do not make separate weights, and the proxy cannot split one pocket, so it refuses to run that way at all:--slidewith--usersstops at start. See Living Weights below for how notes and weights work as one system.
- One owner per store. With
--owner, every request is answered from one person's notes, the--ownergiven at start. The store names its owner, and a proxy started with another owner refuses it. Run one such proxy per person. To serve several people from one proxy, use--users(see Many users), never one--ownerfor all of them. - Notes are data, not instructions. The block quotes each note as something the user said and tells the model never to follow an instruction inside a quote. That is
notes.py's framing, unchanged, and--framing firmkeeps the same sentence. - Local by default. The proxy binds to 127.0.0.1. Binding elsewhere needs
--allow-remote, because anyone who can reach the proxy can read and erase the owner's notes. With--upstream-key-envthey can also use the upstream's key. Requests from web pages (any request with anOriginheader) are refused unless that origin is allowed with--allow-origin, so a page in your browser cannot read or write your notes. Without--allow-remotethe proxy also answers only requests whoseHostnames this machine (localhost,127.0.0.1or[::1]), which stops a page that points its own name at 127.0.0.1 (DNS rebinding) and so sends noOrigin.
| Option | Meaning |
|---|---|
--upstream URL |
the inference server's base URL (http://127.0.0.1:8080, or with /v1 at the end). Mutually exclusive with --tensorfold |
--tensorfold MODEL |
start TensorFold on this model (local dir or org/name) and use it as the upstream |
--tensorfold-bin PATH |
TensorFold executable (default: tensorfold on PATH) |
--tensorfold-arg FLAG |
one extra tensorfold serve flag from the allowlist (repeatable). Write --tensorfold-arg=--no-drafts |
--tensorfold-quiet S |
idle seconds after the last erase before one TensorFold restart (10) |
--tensorfold-restart-max S |
if requests are still in flight this many seconds after the first pending erase, refuse new ones so the restart can run (60) |
--owner NAME |
whose notes this proxy serves (one owner) |
--store FILE |
the notes journal (made if missing) |
--users FILE |
many users: the users file (see Many users) |
--store-dir DIR |
many users: the folder of their stores, one NAME.jsonl each (made if missing) |
--upstream-key-env NAME |
send the upstream the key held in the environment variable NAME, in place of the client's Authorization |
--share-upstream-cache |
many users: let the upstream reuse one user's cached prompt for another (off, see Many users) |
--tokenizer FILE |
the model's tokenizer.json (or its folder) |
--cap N |
the block's token cap, framing included (512) |
--abstain-line TEXT |
a sentence added to the end of the block's first line (off). The cap counts it |
--framing NAME |
the block's first paragraph: quoted, notes.py's own (the default), or firm, which also says a note is right when it disagrees with what the model remembers. The cap counts it |
--write |
write notes from the user's chat statements (off) |
--join-system |
join the block and the client's leading system messages into one system message, with a blank line between, as the TensorFold server does before its chat template. For an upstream whose chat template refuses a second system message |
--notes-config JSON |
other notes.py settings, such as {"takeback": "explicit"} |
--host, --port |
where to listen (127.0.0.1, 8090) |
--allow-remote, --allow-origin ORIGIN |
see Safety |
--timeout S |
seconds to wait for the upstream (600) |
Why --framing firm. With thinking on, Qwen3.8-27B sometimes argued with a note under the default header and kept its own memory, most often in German. In small probes on 24 Oct 2026 the firm header made it follow the note 4 of 4 times at temperature 0, and 20 of 20 at 0.7 against 15 of 20 for the default. On Flash-Next (3-bit GGUF on llama.cpp) both headers were followed about 9 times in 10 at 0.7. Firm is not the default yet, because it has not been checked on questions whose right answer is "I don't know".
These need the engine itself, which a proxy in front of it cannot reach:
- No live fetch mid-answer. The notes are looked up once, from the question, before the model starts. The model cannot look a note up while it answers.
- No prompt-cache reuse. The proxy cannot keep the notes block's prefill across requests; the upstream's own prompt cache may or may not reuse it.
- No Living Weights. Nothing is taught into the model's weights, and no note is promoted into neurons.
- The lookup is Python. In the TensorFold server's comparison,
retrieve()took 0.42 to 23.6 ms a question (p50, stores of 100 to 100,000 notes); the server's own lookup took 52 to 437 microseconds on the same stores. - A write blocks lookups briefly. The store takes one write at a time and lookups wait for it (TensorFold's server builds a new index version instead).
- English only. The past-tense cue and the question check read English.
"notes_language"other than"en"is refused. - The tokenizer is yours to match. The proxy cannot see which model the upstream runs. Give it that model's
tokenizer.json, or the cap is counted for another model. With--tensorfoldthe child tokenizes, so--tokenizeris not used. - Only
/v1/chat/completionsgets notes./v1/completions,/v1/responsesand/v1/messagespass through without them. - Free text is not read for meaning. A sentence outside the rule book's wordings never replaces an earlier note: "My car is blue." and later "My car is red." both stay current. Erase the old one.
- The journal has no commit marks. A write whose line reached the disk just before a crash, but was never answered, comes back at the next start.
Shared Notes Proxy is exact learning memory for agent teams: one store of plain sentences for every agent on a team, whichever model it runs. An agent writes a sentence once, and the next agent gets it in its prompt, or looks it up. It is based on scalable n-gram storage with fast indexing. Coming next: promotion of top facts to neuronal TensorFold Living Weights memories.
A team of agents can share one store, whichever models they run and wherever they run. Run one proxy with --users and
--store-dir, and give each member a key that opens the same store:
python -m notes_proxy add-user --users users.json --owner hermes --store-dir ~/notes --shares team
python -m notes_proxy add-user --users users.json --owner deep --store-dir ~/notes --shares team
python -m notes_proxy add-user --users users.json --owner cloud --store-dir ~/notes --shares team --read-onlyEvery member reads the same notes, and each new write records the member whose key made it, shown as by when the note
is listed. Notes written before a store had per-member keys have no by. A write may also say where the note came from, in the optional cite field, such as a chat message's id. An
erase removes both with the note. Removing a member with --erase-notes is refused, because the store is not theirs:
use --keep-notes.
A --read-only key can list notes and look them up, and nothing else. It cannot write or erase, and it cannot reach the
upstream model, so it is the key to give anything outside your own machines.
Asking without a chat. POST /v1/notes/lookup returns the notes a question would get, using the chat route's own
lookup, and never calls the upstream. It is for clients that do not send their requests through a proxy, such as a
hosted model.
curl -s http://team-host:8095/v1/notes/lookup -H "Authorization: Bearer $KEY" \
-H 'Content-Type: application/json' -d '{"question": "Which port does the Bosgame serve on?"}'The fields are question, and optionally cap, past and exclude, as notes_cap, notes_past and notes_exclude on
the chat route. cap is a token budget, not a number of notes, and a cap too small for the framing gives a 200 with an
empty block. The reply holds block, ready to put in a prompt, and notes, each with its id, text, subject, by
and cite.
Adding team notes to an agent's own proxy. An agent's proxy keeps its private store and its own model, and can also ask the team proxy for each question:
python -m notes_proxy --upstream http://127.0.0.1:8080 --owner deep --store ~/deep.jsonl --tokenizer tokenizer.json \
--team-url http://team-host:8095 --team-key-file ~/.config/notes-team/deep.keyThe team's notes go after the owner's own, up to --team-cap tokens (default 256), and the reply's notes.team names
them. The team proxy is asked with the same question the owner's lookup uses, so notes_query applies to both. If the
team proxy is down, refuses or takes longer than --team-timeout seconds (default 2), the request goes on with the
owner's notes alone, and notes.team.failed says why. The key file must be readable by this account only.
Filling the store from a chat. prefill turns a transcript into notes in two steps, with a person between them.
extract has a model read the transcript from oldest to newest and writes candidates to a file, each one sentence with
the message it came from. It touches no store. Text that looks like a key is removed before the model sees it, and a
candidate that still holds one is dropped. Open the file, set "approved": true on the facts worth keeping, and run
apply, which writes only those.
python -m notes_proxy prefill extract --transcript room.jsonl --since 2026-11-01 \
--model-url http://127.0.0.1:8091 --model NAME --out candidates.jsonl
python -m notes_proxy prefill apply --candidates candidates.jsonl --url http://team-host:8095 \
--key-file team.key --cite-prefix room:my-room/A transcript holds things that stopped being true, and a free-text note does not replace an older one by itself. So each
candidate's newer_same_subject counts the later candidates on its subject, and the reader keeps the one true today. The
candidates are a model's reading of a chat, so read them: they include wrong facts as well as right ones.
Keeping the prompt cache. By default the block goes at the very start of the prompt. A block that changes with the
question then changes the first tokens, so a server with a prompt cache has to read the whole prompt again.
--block-at last-user puts the block at the start of the last user message instead, so only the end of the prompt
changes. Agents with long system prompts should use it. On a 12,000-token prompt with a different block for each
question, Nemotron 3.5 Lightning on an M5 Max reused 0 tokens with the default and 11,967 with --block-at last-user,
and each request took 0.09 seconds instead of 2.5.
Tested on 6 Nov 2026 with four agents on one store, on one set of invented facts: two local models on ROCm and CUDA, and two lookup-only lanes. A test harness wrote the notes, each with an agent's key. With notes off, both local models said they didn't know, and with notes on, both answered correctly. This is one demo, not a benchmark.
From the repository's top folder:
python3 -m unittest discover -s notes_proxy/tests -t .They use the Nemotron tokenizer when it is in the Hugging Face cache and tokenizers is installed, otherwise the stand-in; NOTES_PROXY_TEST_TOKENIZER=fake or =PATH/tokenizer.json chooses. No model runs and no GPU is used: the upstream is a stub that echoes the messages it receives.
- The block (
test_block.py). The L6-seed1 teach store, seeded into a journal and read back: for all 56 questions, the proxy's block, note numbers, tokens and past flag equalretrieve()'s, also with the past forced, with two notes left out (named by number, and named by id), and at a cap of 128. Controls: the store of another phase and of another set each give different blocks. Also: the abstain line, a restart, one owner and one proxy per store. - The framing (
test_framing.py). With--framing firm, for all 56 questions, with one owner and with many users, the upstream gets the firm header first and the rest of the block isnotes.py's. At a cap of 160 every firm block fits, and 55 of 56 hold fewer notes than the default's (control: the header swapped in after the fit goes over the cap in 55 of 56). With no flag and with--framing quoted, the 56 blocks are byte for byte those of the proxy before the option, at caps 512 and 160 (their SHA-256 is pinned in the test). Firm combines with the abstain line and the unknown rule. A bad value stops the start. - End to end (
test_e2e.py). Through HTTP to the stub: the block arrives first and unchanged and every client message after it unchanged, for all 56 questions; nothing found puts nothing in;"notes": falsesends the request unchanged; a streamed reply's first chunk reaches the client while the stub still holds the rest back (control: a proxy that buffers the reply fails this check); erase removes the note from the next request, from every structure the proxy holds (walked withnotes.py's own walker) and from the journal, and a restart keeps it gone; a restart keeps notes;--writewrites statements and skips questions, requests, earlier messages and the model's answers; the API key passes through and stays out of the logs. - Living Weights (
test_weights.py). Against the fake child (whose folder can hold aslide_graph.json): a marked note leaves the block at once and stays out after a restart (the journal's weights line replays);"notes_in_weights": truebrings it back (the control arm); marking an erased id is a 404 and a bad body a 400; a mark true is accepted only when the child's graph shows a learned fact with the note's exact text (nothing taught is a 409 and the note keeps riding the prompt; amissedfact is no proof; neither is a near miss); a REPLACES write demotes the taught note it retires in the same reply (demoted), and the new note rides the prompt; an erase drops the note's weights line from the journal;GET /v1/slide/graphmaps each fact to the note whose text is the taught sentence and marks itin_weights(a fact no note holds names no note);POST /v1/slide/learn,DELETE /v1/slide/fact/1andDELETE /v1/slide/graphthrough the proxy are 404 and reach the child not at all; with many usersGET /v1/slide/graphis 404 to every user and its body names none of another user's taught sentences or questions;--slide(or--slide-graph) with--usersstops the start, and the same proxy without the pocket starts (control); without--tensorfoldboth new routes are 404;--slideand--slide-graphpass the child allowlist. - Many users (
test_users.py). bob has the L6-seed1 teach store. alice has L6-seed2's, and a note in the words of each L6-seed1 question. For all 56 questions, bob's block isretrieve()'s and holds none of alice's notes. His blocks and scores are the same with her store empty, full, and full after a restart (control: her notes in his own index move all 56 blocks). Her erase leaves his journal and blocks as they were, and stays after a restart. A missing, unknown or removed user's key is a 401, and the stub receives nothing. The user's key never reaches the stub, and the upstream key does. Names with path tricks are refused. 448 requests from four users on 16 threads give the same blocks as one at a time. The users file never holds a key, nor any 12 characters of one. - Many users, the edges (
test_many_safety.py). A stand-in for llama.cpp's prompt cache: bob's 9 probes of alice's cached block reuse nothing, and no reply counts reused tokens (control: with--share-upstream-cachethe right guess reuses more than the wrong ones). Only the model routes pass through. A request with the key in its URL or body is refused. Replies carry no lookup times. A wrong store folder erases nothing and keeps the user, and a kept name needs--reuse-notes. A users file that others can change stops the start. A store that cannot be opened is a clear reply for its own user only, and a journal that failed mid-erase opens again for its user alone. The real proxy, started under a soft limit of 64 open files, serves 60 users, and under a hard limit of 128 it refuses to start. The Many users commands of this README work in order in an empty home folder. - Many users, the upstream's state (
test_many_state.py). A stand-in upstream keeps state under every field and header of the table in What reaches the upstream, so two requests that name the same thing share it. alice and bob send at the same moment, three times over, requests that carry the same value for each of those 26 fields and headers. bob never gets alice's prompt back, and each field is set, replaced, dropped or refused as the table says (control: with two one-owner proxies in front of the same stand-in, bob gets alice's secret through all 26). A field or header that names alice as the owner never gives bob her notes, nor writes his into hers (control: a proxy that readX-Ownerwould give him her note). The upstream gets onlyContent-Type,AcceptandUser-Agentof the client's headers, and the upstream key (control: one owner still passes the client's headers on). - Note ids (
test_erase_ids.py). Three notes, the first erased, then the same erase sent again: the second is a 404, the other two notes are still there under the ids they had, and the journal is not touched. The same after a restart, on the store itself, through HTTP with one owner, and with many users (alice and bob both have aw1; her repeated erase leaves his notes and journal as they were). A seeded note (s1) is erased once. An erased id is never given to a later note, also after a restart. A number in place of an id is a 400 that erases nothing. The chat reply'sidsare the listing's ids, itsnumbersare the[n]of the block, andnotes_excludetakes ids. Every one of these fails on the proxy before the change. In the retry cases the second erase answered 200 and removed another note. - The question check (
test_gate.py). The TensorFold server's 55 questions and requests are skipped and its 18 statements written, with its documented misses.
- Notes are on unless a request sends
"notes": false(the server: off unless"notes": true). Writes from chat are off unless--write(the server: on with--notes). DELETE /v1/notes/IDerases. On the server it is a take-back, and an erase isDELETE /v1/notes/ID?erase=true(the proxy takes that too). The proxy has no change (PUT,"replaces") or take-back endpoint, and it can erase every note, also those it was seeded with (the server refuses to erase snapshot notes).- On the server a note id is a number. In the proxy it is text, the write's own id (
w12), which never moves;notes_excludeand the reply'snotes.idsuse these ids (see Note ids). - When streaming,
notesis an SSE comment before[DONE], not a field of the last chunk. The reply'snoteshas the fieldsretrieve()reports (ids,past,past_from,cap,tokens,candidates,notes_in_store,lookup_us,total_us,block_sha256,block,write,write_us, with many users without the three times), not the server's split timings (tokenize_us,fit_us,pack_tokens,dropped,question_tokens). - The block is put in as its own first system message, as the server inserts it. The server then joins it with the client's leading system messages into one; the proxy leaves that to the upstream's chat template, or does the same with
--join-system. - Free-text writes use the rule book's intake when it reads the sentence, so a change ("... now.") retires the old note; the server writes every chat statement as new unless it repeats a current note.
- The store is a journal of the writes, not the server's snapshot and journal; the proxy does not check that the upstream's model matches the tokenizer (the server checks the tokenizer's hash).