Skip to content

Commit 7dc1944

Browse files
OriNachumOriNachum
authored
feat: third (middle) model tier + pressure-aware tier routing (v0.32.0) (#70)
1 parent fe8962e commit 7dc1944

35 files changed

Lines changed: 4516 additions & 54 deletions

‎.devague/current‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
lobes-ships-a-per-lobe-benchmark-suite-speed-paral
1+
lobes-adds-a-third-middle-model-tier-and-pressure

‎.devague/current_plan‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
lobes-ships-a-per-lobe-benchmark-suite-speed-paral
1+
lobes-adds-a-third-middle-model-tier-and-pressure

‎.devague/frames/lobes-adds-a-third-middle-model-tier-and-pressure.json‎

Lines changed: 330 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,330 @@
1+
{
2+
"slug": "lobes-adds-a-third-middle-model-tier-and-pressure",
3+
"title": "lobes adds a third middle model tier and pressure-aware tier routing, so three capability tiers (cheap 4B / middle 8B-14B / hard 27B) stay co-resident on the Spark \u2014 downstream agents request a tier instead of a model name, the 27B served context is trimmed to make room, and the middle tier doubles as a LoRA-trainable base",
4+
"schema_version": 1,
5+
"status": "exported",
6+
"created": "2026-06-29T19:29:08Z",
7+
"updated": "2026-06-29T19:43:03Z",
8+
"claims": [
9+
{
10+
"id": "c1",
11+
"kind": "announcement",
12+
"text": "lobes adds a third middle model tier and pressure-aware tier routing, so three capability tiers (cheap 4B / middle 8B-14B / hard 27B) stay co-resident on the Spark \u2014 downstream agents request a tier instead of a model name, the 27B served context is trimmed to make room, and the middle tier doubles as a LoRA-trainable base",
13+
"origin": "user",
14+
"status": "confirmed",
15+
"honesty_conditions": [
16+
{
17+
"id": "h1",
18+
"text": "After bring-up, all three generate gears (4B + middle + 27B) plus embed/rerank are simultaneously resident and healthy, verified by lobes fleet status / nvidia-smi, within the 128GB unified-memory budget",
19+
"status": "confirmed"
20+
},
21+
{
22+
"id": "h2",
23+
"text": "A caller names only a capability tier (cheap/normal/hard) and the gateway/route resolves it to the correct co-resident gear \u2014 no concrete model id crosses the contract boundary",
24+
"status": "confirmed"
25+
}
26+
],
27+
"hard_questions": [],
28+
"links": []
29+
},
30+
{
31+
"id": "c2",
32+
"kind": "audience",
33+
"text": "Downstream Culture mesh agents and tools that today hardcode a model name or backend, plus the operator running the lobes fleet on the shared DGX Spark",
34+
"origin": "llm",
35+
"status": "confirmed",
36+
"honesty_conditions": [
37+
{
38+
"id": "h12",
39+
"text": "A concrete downstream caller (e.g. a Culture mesh agent or daria) can switch from a hardcoded model id to a tier alias with no other code change and keep working",
40+
"status": "confirmed"
41+
}
42+
],
43+
"hard_questions": [],
44+
"links": []
45+
},
46+
{
47+
"id": "c3",
48+
"kind": "why_it_matters",
49+
"text": "DGX Spark uses unified memory: CPU and GPU share one physical pool, so running the 27B as the default path causes swap thrash. Tiering lets the box behave like a memory-budgeted lab device \u2014 small models carry routine work, the large model is the escalation/authority path",
50+
"origin": "llm",
51+
"status": "confirmed",
52+
"honesty_conditions": [
53+
{
54+
"id": "h13",
55+
"text": "On the Spark, running the 27B as the default path is shown to drive swap/iowait up, and routing routine work to the 4B/14B measurably lowers that pressure",
56+
"status": "confirmed"
57+
}
58+
],
59+
"hard_questions": [],
60+
"links": []
61+
},
62+
{
63+
"id": "c4",
64+
"kind": "before_state",
65+
"text": "Two generate gears exist (minor Qwen3.5-4B + 27B primary) but there is no middle tier, no capability-tier contract (callers hardcode model names), and no pressure-aware routing on swap/iowait",
66+
"origin": "llm",
67+
"status": "confirmed",
68+
"honesty_conditions": [
69+
{
70+
"id": "h3",
71+
"text": "Today, with only minor(4B) + primary(27B), there is no catalog gear whose role_hint marks it as the middle tier, so route/select cannot offer a normal/middle option",
72+
"status": "confirmed"
73+
}
74+
],
75+
"hard_questions": [],
76+
"links": []
77+
},
78+
{
79+
"id": "c5",
80+
"kind": "after_state",
81+
"text": "Three generate tiers (cheap 4B / middle 8B-14B / hard 27B) are co-resident behind the gateway; a caller asks lobes for a capability tier (cheap/normal/hard) and gets back the chosen model + mode + reason, with pressure-aware fallback when unified memory is under stress",
82+
"origin": "llm",
83+
"status": "confirmed",
84+
"honesty_conditions": [
85+
{
86+
"id": "h14",
87+
"text": "After bring-up a caller issues cheap/normal/hard and receives a completion from the matching gear, with mode and reason surfaced",
88+
"status": "confirmed"
89+
}
90+
],
91+
"hard_questions": [],
92+
"links": []
93+
},
94+
{
95+
"id": "c6",
96+
"kind": "boundary",
97+
"text": "This spec delivers the third (middle) tier and the capability-tier routing contract; it does NOT implement LoRA fine-tuning (lobes train stays a follow-up) \u2014 the middle tier is merely chosen as a bf16 base that CAN be LoRA-trained",
98+
"origin": "llm",
99+
"status": "rejected",
100+
"honesty_conditions": [],
101+
"hard_questions": [],
102+
"links": []
103+
},
104+
{
105+
"id": "c7",
106+
"kind": "non_goal",
107+
"text": "Not dynamically unloading/reloading the 27B per request; the three generate gears stay co-resident \u2014 memory headroom comes from trimming the 27B served context, not from swapping models in and out",
108+
"origin": "llm",
109+
"status": "confirmed",
110+
"honesty_conditions": [],
111+
"hard_questions": [],
112+
"links": []
113+
},
114+
{
115+
"id": "c8",
116+
"kind": "success_signal",
117+
"text": "A downstream caller selects a capability tier (e.g. lobes select --tier normal) and receives the chosen model + mode + reason without ever naming a concrete model; all three generate gears plus embed/rerank fit within the Spark 128GB unified-memory budget",
118+
"origin": "llm",
119+
"status": "rejected",
120+
"honesty_conditions": [
121+
{
122+
"id": "h4",
123+
"text": "The memory math is shown: trimmed-27B KV cache + middle-gear weights+KV + 4B + embed + rerank (+optional audio) sums under the GPU mem-util budget that fits 128GB",
124+
"status": "proposed"
125+
}
126+
],
127+
"hard_questions": [],
128+
"links": []
129+
},
130+
{
131+
"id": "c9",
132+
"kind": "boundary",
133+
"text": "This spec delivers the third (middle) tier plus the capability-tier routing contract and the full pressure policy; it does NOT implement LoRA fine-tuning itself (lobes train stays a follow-up)",
134+
"origin": "llm",
135+
"status": "confirmed",
136+
"honesty_conditions": [
137+
{
138+
"id": "h15",
139+
"text": "The shipped scope contains no lobes train / fine-tuning code path; LoRA training appears only as a referenced follow-up",
140+
"status": "confirmed"
141+
}
142+
],
143+
"hard_questions": [],
144+
"links": []
145+
},
146+
{
147+
"id": "c10",
148+
"kind": "decision",
149+
"text": "Middle tier = a 14B-class NVFP4 checkpoint (memory-lean ~8GB weights, inference-only escalation reviewer/coder). Exact HF checkpoint id verified separately (candidates: nvidia/Qwen3-14B-NVFP4 or equivalent), consistent with the existing nvidia/Qwen3-32B-NVFP4 candidate",
150+
"origin": "llm",
151+
"status": "confirmed",
152+
"honesty_conditions": [],
153+
"hard_questions": [
154+
{
155+
"id": "q1",
156+
"text": "risk: A 14B NVFP4 checkpoint must actually load and produce non-gibberish output on the fleet's Blackwell vLLM image; older dense Qwen3 NVFP4 (cf. the supported 32B candidate) is lower-risk than a Qwen3.5/3.6 hybrid which hit a known FLA bug pre-vLLM-0.23",
157+
"resolved": false,
158+
"blocking": false
159+
}
160+
],
161+
"links": []
162+
},
163+
{
164+
"id": "c11",
165+
"kind": "decision",
166+
"text": "27B primary served context is trimmed from full 256K (262144) to 128K (131072) via PRIMARY_MAX_MODEL_LEN, freeing roughly half the KV cache to make room for the co-resident middle gear",
167+
"origin": "llm",
168+
"status": "confirmed",
169+
"honesty_conditions": [],
170+
"hard_questions": [],
171+
"links": []
172+
},
173+
{
174+
"id": "c12",
175+
"kind": "decision",
176+
"text": "Tier-request surface is the gateway model-alias ONLY: a caller sends model=cheap|normal|hard to the OpenAI-compatible endpoint and the gateway resolves it to the 4B/14B/27B generate gear. No new 'lobes select --tier' verb and no 'route --tier' field in this spec",
177+
"origin": "llm",
178+
"status": "confirmed",
179+
"honesty_conditions": [],
180+
"hard_questions": [],
181+
"links": []
182+
},
183+
{
184+
"id": "c13",
185+
"kind": "decision",
186+
"text": "Full pressure policy is in scope: lobes senses host memory pressure (swap%, iowait%) and applies the issue #68 thresholds, including a degraded-mode state machine and pressure-aware downgrade",
187+
"origin": "llm",
188+
"status": "confirmed",
189+
"honesty_conditions": [],
190+
"hard_questions": [],
191+
"links": []
192+
},
193+
{
194+
"id": "c14",
195+
"kind": "decision",
196+
"text": "LoRA training target is the existing 4B bf16 minor lobe; the 14B NVFP4 middle tier is inference-only and is NOT a LoRA base. The 'three tiers to scale work and train loras' goal = scale inference across 3 tiers + keep the 4B as the trainable base",
197+
"origin": "llm",
198+
"status": "confirmed",
199+
"honesty_conditions": [],
200+
"hard_questions": [],
201+
"links": []
202+
},
203+
{
204+
"id": "c15",
205+
"kind": "requirement",
206+
"text": "All three generate gears (4B bf16 + 14B NVFP4 + 27B@128K) plus the embed and rerank pooling gears are simultaneously resident and healthy within the Spark 128GB unified-memory budget",
207+
"origin": "llm",
208+
"status": "confirmed",
209+
"honesty_conditions": [
210+
{
211+
"id": "h5",
212+
"text": "The GPU mem-util budget is shown to sum under 1.0 (27B@128K + 14B-NVFP4 + 4B + embed + rerank) AND lobes fleet status reports all five gears healthy after bring-up on the Spark",
213+
"status": "confirmed"
214+
}
215+
],
216+
"hard_questions": [],
217+
"links": []
218+
},
219+
{
220+
"id": "c16",
221+
"kind": "requirement",
222+
"text": "The gateway resolves the model aliases cheap/normal/hard to the 4B/14B/27B generate gears respectively, as a same-task (generate) alias layer on top of the existing task-family routing",
223+
"origin": "llm",
224+
"status": "confirmed",
225+
"honesty_conditions": [
226+
{
227+
"id": "h6",
228+
"text": "An OpenAI request with model=normal returns a chat completion whose served model is the 14B gear; an embed request still never fails over to a generate gear (same-task constraint preserved)",
229+
"status": "confirmed"
230+
}
231+
],
232+
"hard_questions": [],
233+
"links": []
234+
},
235+
{
236+
"id": "c17",
237+
"kind": "requirement",
238+
"text": "lobes samples host memory pressure read-only (swap_used_percent + iowait_percent, from /proc) without mutating the system",
239+
"origin": "llm",
240+
"status": "confirmed",
241+
"honesty_conditions": [
242+
{
243+
"id": "h7",
244+
"text": "Pressure fields are populated from real /proc readings on the Spark and the sampler is side-effect-free (no writes, no container restarts)",
245+
"status": "confirmed"
246+
}
247+
],
248+
"hard_questions": [],
249+
"links": []
250+
},
251+
{
252+
"id": "c18",
253+
"kind": "requirement",
254+
"text": "Pressure thresholds from issue #68 drive tier resolution and a degraded-mode state machine (swap>50 avoid new hard jobs; swap>65 prefer cheap/middle; swap>75 degraded=cheap-only; iowait>25 no new hard; iowait>50 emergency degraded); under degraded mode a model=hard request resolves down to a permitted tier and the response reports reason=pressure",
255+
"origin": "llm",
256+
"status": "confirmed",
257+
"honesty_conditions": [
258+
{
259+
"id": "h8",
260+
"text": "An injected/simulated swap>75% causes a model=hard request to be served by the cheap tier with reason=pressure; every threshold is config-driven, not hardcoded magic numbers",
261+
"status": "confirmed"
262+
}
263+
],
264+
"hard_questions": [],
265+
"links": []
266+
},
267+
{
268+
"id": "c19",
269+
"kind": "requirement",
270+
"text": "A manual override lets a caller force the requested tier despite pressure (bypassing the automatic downgrade); the override and the downgrade reason cross the OpenAI-compatible boundary via HTTP headers, and the response model field reflects the actually-served model",
271+
"origin": "llm",
272+
"status": "confirmed",
273+
"honesty_conditions": [
274+
{
275+
"id": "h9",
276+
"text": "With the override header set, model=hard is served by the 27B even under simulated degraded pressure; without it, the same request downgrades",
277+
"status": "confirmed"
278+
}
279+
],
280+
"hard_questions": [],
281+
"links": []
282+
},
283+
{
284+
"id": "c20",
285+
"kind": "requirement",
286+
"text": "A read-only observability surface reports the active selection as {tier, model, mode(warm|lazy|disabled|degraded), reason(default|escalation|pressure|manual_override), pressure{swap_used_percent, iowait_percent}} \u2014 e.g. via lobes status --pressure",
287+
"origin": "llm",
288+
"status": "confirmed",
289+
"honesty_conditions": [
290+
{
291+
"id": "h10",
292+
"text": "lobes status --pressure emits exactly {tier, model, mode, reason, pressure{swap_used_percent, iowait_percent}} and mutates nothing (read-only, like the existing status verb)",
293+
"status": "confirmed"
294+
}
295+
],
296+
"hard_questions": [],
297+
"links": []
298+
},
299+
{
300+
"id": "c21",
301+
"kind": "success_signal",
302+
"text": "A downstream caller selects a capability tier by sending model=cheap|normal|hard to the OpenAI-compatible gateway and receives a completion from the right gear without ever naming a concrete model; all three generate gears plus embed/rerank fit within the Spark 128GB unified-memory budget",
303+
"origin": "llm",
304+
"status": "confirmed",
305+
"honesty_conditions": [
306+
{
307+
"id": "h11",
308+
"text": "The memory math is shown: trimmed-27B@128K KV cache + 14B-NVFP4 weights+KV + 4B + embed + rerank (+optional audio) sums under the GPU mem-util budget that fits 128GB",
309+
"status": "confirmed"
310+
}
311+
],
312+
"hard_questions": [],
313+
"links": []
314+
}
315+
],
316+
"open_vagueness": [
317+
{
318+
"id": "v1",
319+
"text": "Exact 14B NVFP4 HF checkpoint id (and whether a same-generation Qwen3.6 14B NVFP4 exists, vs falling back to a Qwen3 14B NVFP4)",
320+
"kind": "unknown_nonblocking",
321+
"claim_id": null
322+
},
323+
{
324+
"id": "v2",
325+
"text": "Exact HTTP header names for tier override + downgrade reason, and whether a streaming response can carry the reason (header vs first-chunk metadata)",
326+
"kind": "unknown_nonblocking",
327+
"claim_id": null
328+
}
329+
]
330+
}

0 commit comments

Comments
 (0)