Skip to content

Commit 72419bf

Browse files
Size memory search to the machine it runs on
The embedder no longer costs the same everywhere. Its weights are memory-mapped and used in place, so loading allocates 4 MB instead of 177 MB and every atlas-cli process shares one copy; a blocked linear kernel makes a forward pass 1.5-1.7x faster on one core with the same vectors. Background index passes and extractions run at nice 10, on half the cores (one on two or fewer), one at a time across all Projects; a hook that asks for a pass while one runs marks the Project and the running pass goes once more. atlas-cli memory compute [auto|off|cpu|ollama|<url>] sets how search embeds on this machine, kept in <state>/compute.json and overridden by ATLAS_EMBED, ATLAS_EMBED_MODEL and ATLAS_EMBED_THREADS. auto uses the CPU, keywords only on a small machine (one core or under 2 GB, container limits included), or a local Ollama serving all-minilm. An endpoint (Ollama, TEI, vLLM, LM Studio) is used only once four probe texts come back within cosine 0.995 of the app's vectors, so embed_cache stays shared; a failing endpoint falls back to the CPU or keywords. Status and Settings show the plan, and model files that do not load are reported and downloaded again. ADR-0014.
1 parent f76787d commit 72419bf

26 files changed

Lines changed: 2125 additions & 55 deletions

‎README.md‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,11 @@ config. Nothing is ever injected into a prompt: agents pull what they need.
103103
in pure Go on your machine; nothing is sent anywhere. Ask "how do we rotate
104104
tokens" and find "signing keys rotate every week". Without the model, search
105105
falls back to words.
106+
- **Sized to the machine.** Background indexing runs at a lower priority, on
107+
half the cores, one Project at a time. On a small machine (one core, or
108+
under 2 GB) search stays on keywords; with a GPU, point it at Ollama or any
109+
OpenAI-compatible embeddings server (TEI, vLLM), used once its vectors match
110+
the app's model.
106111
- **Plans are captured, not typed.** When an agent writes its to-do list
107112
(`TodoWrite`, `update_plan`, `todowrite`, Claude's tasks and plan mode, …),
108113
it becomes the project's plan, and the next session, or another agent, sees
@@ -117,9 +122,21 @@ config. Nothing is ever injected into a prompt: agents pull what they need.
117122

118123
```sh
119124
atlas-cli memory browse # entries by kind, search, history
120-
atlas-cli memory index# embed every memory, index the project's documents
125+
atlas-cli memory index # embed every memory, index the project's documents
126+
atlas-cli memory compute # how this machine embeds: auto, off, cpu, ollama or a URL
121127
```
122128

129+
| `atlas-cli memory compute …` | for |
130+
|---|---|
131+
| `auto` (default) | the CPU; keywords only on a small machine; a local Ollama with `all-minilm` when found |
132+
| `off` | keywords only: the model is never loaded |
133+
| `cpu [--threads N]` | the CPU whatever the machine, on at most N cores |
134+
| `ollama` | Ollama here (`ollama pull all-minilm`) |
135+
| `http://gpu-box:8080/v1` | an OpenAI-compatible server on a GPU |
136+
137+
`ATLAS_EMBED`, `ATLAS_EMBED_MODEL` and `ATLAS_EMBED_THREADS` override the
138+
setting for one environment (CI, a container).
139+
123140
## Session capture
124141

125142
Capture happens per event, as the agent works:
@@ -179,6 +196,7 @@ colour only for status) and prints plain text to pipes and scripts.
179196
| `atlas-cli sync [on \| off]` | background sync |
180197
| `atlas-cli serve` | the Project's listener: memory server, sync drain, receiver of hook events |
181198
| `atlas-cli memory browse \| list \| index \| verify \| export \| import` | shared memory |
199+
| `atlas-cli memory compute [auto \| off \| cpu \| ollama \| <url>]` | how memory search embeds on this machine |
182200
| `atlas-cli memory enable \| disable` | the shared-memory gate for this repository |
183201
| `atlas-cli capture enable \| disable \| sync \| retry \| link` | the capture gate; sync now, retry failed rows, link commits |
184202
| `atlas-cli update [--check] [--version vX.Y.Z]` | replace this binary with a release, checksum verified |
@@ -190,6 +208,7 @@ colour only for status) and prints plain text to pipes and scripts.
190208
|---|---|
191209
| `.atlas/` | the app's per-project files only: `sessions.db`, `blobs/`, `memory/`, `project.json` |
192210
| `~/.local/state/atlas-cli/projects/<hash>/` | the CLI's own: hook state, `hooks.log`, the listener's run file, preferences |
211+
| `~/.local/state/atlas-cli/compute.json` | this machine's memory compute setting, and endpoint verdicts beside it |
193212
| `~/Library/Application Support/dev.atlas.ide/models/` | the local search model, shared with the app (`~/.local/share/…` on Linux) |
194213

195214
## Development

‎cmd/atlas-cli/background.go‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
package main
2+
3+
import (
4+
"context"
5+
"os"
6+
"path/filepath"
7+
"runtime"
8+
"strconv"
9+
"time"
10+
11+
"golang.org/x/sys/unix"
12+
13+
"github.com/tryatlas/cli/internal/clistate"
14+
"github.com/tryatlas/cli/internal/embed"
15+
)
16+
17+
// backgroundNice is the nice value of a detached pass: below the agent and
18+
// the shell, so a small server stays responsive while it embeds.
19+
const backgroundNice = 10
20+
21+
// backgroundWait bounds how long a detached pass queues behind another
22+
// Project's; its pending mark stays, so the next pass does its work.
23+
const backgroundWait = 10 * time.Minute
24+
25+
// background readies a detached pass (atlas-cli memory index --background,
26+
// an extraction's reindex) to share the machine: a lower priority, the
27+
// cores the compute plan leaves a background pass, and one pass at a time
28+
// across every Project here. ok is false when the turn did not come.
29+
func background(ctx context.Context) (release func(), ok bool) {
30+
yield()
31+
if err := os.MkdirAll(clistate.Base(), 0o700); err != nil {
32+
return func() {}, true
33+
}
34+
f, err := os.OpenFile(filepath.Join(clistate.Base(), "embed.lock"), os.O_CREATE|os.O_RDWR, 0o600)
35+
if err != nil {
36+
return func() {}, true // no lock file: run unqueued rather than not at all
37+
}
38+
ctx, cancel := context.WithTimeout(ctx, backgroundWait)
39+
defer cancel()
40+
for unix.Flock(int(f.Fd()), unix.LOCK_EX|unix.LOCK_NB) != nil {
41+
select {
42+
case <-ctx.Done():
43+
_ = f.Close()
44+
return func() {}, false
45+
case <-time.After(250 * time.Millisecond):
46+
}
47+
}
48+
return func() {
49+
_ = unix.Flock(int(f.Fd()), unix.LOCK_UN)
50+
_ = f.Close()
51+
}, true
52+
}
53+
54+
// yield lowers this process's priority and caps its cores at what the
55+
// compute plan leaves a background pass.
56+
func yield() {
57+
lowerPriority()
58+
if p := embed.Current(); p.Backend == embed.CPU {
59+
runtime.GOMAXPROCS(p.Background)
60+
}
61+
}
62+
63+
// lowerPriority renices this process. On Linux a nice value is a thread's,
64+
// so every thread the runtime has started is reniced; threads it starts
65+
// later inherit it.
66+
func lowerPriority() {
67+
if runtime.GOOS == "linux" {
68+
if ents, err := os.ReadDir("/proc/self/task"); err == nil {
69+
for _, e := range ents {
70+
if tid, err := strconv.Atoi(e.Name()); err == nil {
71+
_ = unix.Setpriority(unix.PRIO_PROCESS, tid, backgroundNice)
72+
}
73+
}
74+
return
75+
}
76+
}
77+
_ = unix.Setpriority(unix.PRIO_PROCESS, 0, backgroundNice)
78+
}

‎cmd/atlas-cli/commands.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,7 @@ func newMemoryCmd() *cobra.Command {
152152
withRun(leaf("verify", "Verify the memory revision hash chain", cobra.NoArgs), runMemoryVerify),
153153
withRun(leaf("browse", "Browse shared memory: entries by kind, search, history", cobra.NoArgs), runMemoryBrowse),
154154
newMemoryIndexCmd(),
155+
newMemoryComputeCmd(),
155156
newMemoryListCmd(),
156157
withRun(leaf("export <dir>", "Export durable memories as an Agent Memory Repo (MEMORY.md and one file per kind)", cobra.ExactArgs(1)), runMemoryExport),
157158
withRun(leaf("import <file-or-dir>", "Import Agent Memory Repo files as candidate memories", cobra.ExactArgs(1)), runMemoryImport),

‎cmd/atlas-cli/compute_cmds.go‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
package main
2+
3+
import (
4+
"fmt"
5+
"net"
6+
"net/url"
7+
"os"
8+
9+
"github.com/spf13/cobra"
10+
11+
"github.com/tryatlas/cli/internal/embed"
12+
)
13+
14+
// atlas-cli memory compute: how memory search embeds on this machine.
15+
func newMemoryComputeCmd() *cobra.Command {
16+
c := withRun(leaf("compute [auto|off|cpu|ollama|<url>]",
17+
"Show or set how memory search embeds here: automatic, keywords only, this CPU, or a GPU endpoint", cobra.MaximumNArgs(1)), runMemoryCompute)
18+
c.Long = `Show or set how memory search embeds on this machine (every Project here).
19+
20+
auto this machine's CPU, or keywords only on a small machine (one core,
21+
or under 2 GB); a local Ollama serving all-minilm is used when found
22+
off keywords only: the model is never loaded
23+
cpu this machine's CPU, whatever its size
24+
ollama Ollama on this machine (ollama pull all-minilm)
25+
<url> an OpenAI-compatible embeddings API, such as Hugging Face TEI or vLLM
26+
on a GPU (http://gpu-box:8080/v1)
27+
28+
An endpoint is used only once its vectors match the app's model. ATLAS_EMBED,
29+
ATLAS_EMBED_MODEL and ATLAS_EMBED_THREADS override the setting.`
30+
c.Flags().String("model", "", "the model an endpoint is asked for")
31+
c.Flags().Int("threads", -1, "cores one embedding uses on the CPU (0: automatic)")
32+
return c
33+
}
34+
35+
func runMemoryCompute(cmd *cobra.Command, args []string) error {
36+
ctx := cmd.Context()
37+
out := cmd.OutOrStdout()
38+
threads, _ := cmd.Flags().GetInt("threads")
39+
model, _ := cmd.Flags().GetString("model")
40+
if len(args) == 0 && threads < 0 && model == "" {
41+
printCompute(cmd, embed.Current())
42+
return nil
43+
}
44+
s := embed.SavedSettings()
45+
if len(args) == 1 {
46+
mode, err := embed.ParseMode(args[0])
47+
if err != nil {
48+
return err
49+
}
50+
s.Mode, s.Model = mode, ""
51+
}
52+
if model != "" {
53+
s.Model = model
54+
}
55+
if threads >= 0 {
56+
s.Threads = threads
57+
}
58+
p := embed.Choose(s, embed.DetectMachine())
59+
if p.Backend != embed.Endpoint {
60+
s.Model = "" // a model is an endpoint's
61+
}
62+
if p.Backend == embed.Endpoint {
63+
fmt.Fprintf(out, "Checking %s against the app's model…\n", endpointLabel(p))
64+
if err := embed.VerifyEndpoint(ctx, p.URL, p.Model); err != nil {
65+
if p.Mode == "ollama" {
66+
return fmt.Errorf("%s: %w; not saved (is Ollama running, with ollama pull %s done?)", endpointLabel(p), err, embed.OllamaModel)
67+
}
68+
return fmt.Errorf("%s: %w; not saved", endpointLabel(p), err)
69+
}
70+
if !loopback(p.URL) {
71+
fmt.Fprintf(out, "note: memory text is sent to %s to be embedded.\n", hostOf(p.URL))
72+
}
73+
}
74+
if err := embed.SaveSettings(s); err != nil {
75+
return err
76+
}
77+
if os.Getenv("ATLAS_EMBED") != "" {
78+
fmt.Fprintln(out, "note: ATLAS_EMBED is set in this environment and overrides the saved mode.")
79+
}
80+
p = embed.Current()
81+
printCompute(cmd, p)
82+
if p.Backend == embed.CPU && !embed.Available() {
83+
fmt.Fprintln(out, "atlas-cli memory index downloads the model ("+embed.Size+") and builds the index.")
84+
}
85+
return nil
86+
}
87+
88+
func printCompute(cmd *cobra.Command, p embed.Plan) {
89+
w := cmd.OutOrStdout()
90+
fmt.Fprintf(w, "Machine: %s\n", p.Machine)
91+
fmt.Fprintf(w, "Mode: %s\n", p.Mode)
92+
fmt.Fprintf(w, "Search: %s\n", describePlan(p))
93+
}
94+
95+
// describePlan is a plan in one line: what search matches on, then where
96+
// the vectors come from.
97+
func describePlan(p embed.Plan) string {
98+
switch p.Backend {
99+
case embed.Endpoint:
100+
return "meaning and keywords · " + endpointLabel(p) + " · " + p.Note
101+
case embed.CPU:
102+
if err := embed.LoadError(); err != nil {
103+
return fmt.Sprintf("keywords only · the model files do not load (%v); atlas-cli memory index downloads them again", err)
104+
}
105+
if !embed.Available() {
106+
line := "keywords only · the model is not downloaded (atlas-cli memory index)"
107+
if p.Note != "" && p.Note != "as set" {
108+
line += " · " + p.Note
109+
}
110+
return line
111+
}
112+
cores := "every core"
113+
if p.Threads > 0 && p.Threads < p.Machine.CPUs {
114+
cores = fmt.Sprintf("%d cores", p.Threads)
115+
}
116+
line := fmt.Sprintf("meaning and keywords · this machine's CPU, %s (background passes: %d)", cores, p.Background)
117+
if p.Note != "" {
118+
line += " · " + p.Note
119+
}
120+
return line
121+
}
122+
return "keywords only · " + p.Note
123+
}
124+
125+
func endpointLabel(p embed.Plan) string { return p.Model + " at " + hostOf(p.URL) }
126+
127+
func hostOf(u string) string {
128+
if pu, err := url.Parse(u); err == nil && pu.Host != "" {
129+
return pu.Host
130+
}
131+
return u
132+
}
133+
134+
// loopback reports an endpoint on this machine.
135+
func loopback(u string) bool {
136+
pu, err := url.Parse(u)
137+
if err != nil {
138+
return false
139+
}
140+
if pu.Hostname() == "localhost" {
141+
return true
142+
}
143+
ip := net.ParseIP(pu.Hostname())
144+
return ip != nil && ip.IsLoopback()
145+
}

0 commit comments

Comments
 (0)