Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Changelog

## v1.0.3 - 2026-05-20

Patch release: updates Aleph's local white-box runtime framing after MLX
upstream added Linux CUDA backend support. The current local path remains
MLX-backed and Apple Silicon is still the known-good maintainer route, while
Linux CUDA is now part of the forward hardware story. See
[docs/releases/1.0.3.md](docs/releases/1.0.3.md).

## v1.0.2 - 2026-05-19

Patch release: production metadata, Vercel Web Analytics, and explorer i18n /
Expand Down
10 changes: 6 additions & 4 deletions DEMO.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Paste any text, select **mock** mode, click **Generate Compression Path**.
The UI renders the compression path, Pareto frontier slider, token-loss panel,
and candidate-ribbon in a single run.

**With local MLX search (Apple Silicon, model required):**
**With local MLX search (Apple Silicon or Linux CUDA host, model required):**

```bash
# Terminal 1: MLX search engine
Expand Down Expand Up @@ -173,9 +173,11 @@ from `web/public/aleph-logo.png`).
### Backend deploy (`search/server.py`)

The backend is **local-only by design**: it runs a fixed local Qwen3 via
**MLX**, which only works on Apple Silicon. It does **not** run on Vercel /
generic Linux x86. The deployed site's "compress your own text" degrades
gracefully if it's unreachable ("live search offline — examples still work").
**MLX**. Apple Silicon is the known maintainer route, and upstream MLX now
includes Linux CUDA backend support. It does **not** run inside the Vercel web
deployment; the deployed site's "compress your own text" degrades gracefully if
the separate search service is unreachable ("live search offline — examples
still work").

**Recommended: run on the Mac + tunnel (no cloud GPU needed).**

Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ Near-term research tracks:

## Current status

Aleph is at `v1.0.2`: a small formal release of the active Next.js launch surface, shared `AlephRun` contract, fixture demos, local MLX evidence traces, a hosted black-box model adapter, and a first pass of launch-surface polish.
Aleph is at `v1.0.3`: a small formal release of the active Next.js launch surface, shared `AlephRun` contract, fixture demos, local MLX evidence traces, a hosted black-box model adapter, and updated MLX runtime framing.

The release is intentionally honest about evidence:

Expand All @@ -146,7 +146,7 @@ THESIS.md Product and research thesis
web/ Active Next.js launch surface
apps/web Archived legacy/reference React/Vite console
apps/api FastAPI orchestration layer (mock + local MLX search adapter)
search/ Local MLX live-search engine (requires Apple Silicon)
search/ Local MLX live-search engine (Apple Silicon now; Linux CUDA upstream path)
packages/core Shared AlephRun data contracts and scoring helpers
packages/ui UI component shells for console panels
packages/fixtures Sample runs used by the demo and checks
Expand Down Expand Up @@ -182,7 +182,7 @@ The active web app exposes `POST /api/search` as an `AlephRun`-compatible adapte

- `fixture`: precomputed runs from `web/public/aleph-frontier.json`.
- `hosted black-box`: server-side calls to an OpenAI-compatible `/chat/completions` endpoint.
- `local MLX`: optional Apple Silicon search service through `search/server.py`.
- `local MLX`: optional MLX search service through `search/server.py`; Apple Silicon is the current maintainer route and Linux CUDA is the upstream expansion path.

Hosted API credentials must stay server-side. Do not use `NEXT_PUBLIC_` for API keys.

Expand Down
3 changes: 2 additions & 1 deletion apps/api/aleph_api/services/local_mlx_search.py
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,8 @@ def _candidate_from_point(
prompt = str(point.get("prompt") or "")
output = str(point.get("output") or "")
length = _point_length(point)
fit = _bounded_float(point.get("similarity"), default=similarity_score(target_text, output))
measured_fit = similarity_score(target_text, output)
fit = 1.0 if measured_fit == 1.0 else _bounded_float(point.get("similarity"), default=measured_fit)
stability = _bounded_float(point.get("stability"), default=1.0)
avg_nll = _average_nll(point.get("toknll"))
label = _point_label(point, index=index)
Expand Down
8 changes: 7 additions & 1 deletion apps/api/aleph_api/services/scoring.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,13 @@ def compression_ratio(candidate_prompt: str, explicit_prompt: str) -> float:


def similarity_score(target_text: str, output: str) -> float:
return difflib.SequenceMatcher(a=target_text.strip(), b=output.strip()).ratio()
target = target_text.strip()
actual = output.strip()
if not target and not actual:
return 0.0
if target == actual and target:
return 1.0
return difflib.SequenceMatcher(a=target, b=actual).ratio()
Comment thread
coderabbitai[bot] marked this conversation as resolved.


def leakage_score(prompt: str, target_text: str) -> float:
Expand Down
31 changes: 31 additions & 0 deletions apps/api/tests/test_scoring.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,37 @@ def test_leakage_score_clamped():
# overall_score
# ---------------------------------------------------------------------------

def test_similarity_score_exact_match_is_perfect():
text = "同一段文字 should score exactly when unchanged."
assert similarity_score(text, text) == pytest.approx(1.0, abs=1e-6)


def test_similarity_score_empty_match_is_not_perfect():
assert similarity_score("", " ") == pytest.approx(0.0, abs=1e-6)


def test_local_mlx_adapter_exact_match_overrides_stale_similarity():
from aleph_api.services.local_mlx_search import _candidate_from_point

target = "同一段文字 should score exactly when unchanged."
candidate = _candidate_from_point(
{
"prompt": f"Repeat exactly:\n\n{target}",
"output": target,
"length": 8,
"similarity": 0.42,
"stability": 1.0,
"label": "Explicit Reconstruction",
},
index=0,
explicit_tokens=8,
target_text=target,
adapter="local_mlx_search",
)

assert candidate.fit == pytest.approx(1.0, abs=1e-6)


def test_overall_score_clamped():
score = overall_score(similarity=1.0, compression=1.0, leakage=0.0, reliability=1.0)
assert 0.0 <= score <= 1.0
Expand Down
8 changes: 6 additions & 2 deletions docs/plans/hackathon-v0.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,20 @@ The v0 artifact must make these ideas obvious:

- frontend: existing React/Vite app in `apps/web`
- API: FastAPI in `apps/api`
- local model route: `mlx-lm` on Apple Silicon
- local model route: `mlx-lm`; Apple Silicon was the v0 maintainer machine path
- default local model target: `Qwen/Qwen3-8B-MLX-4bit`

### Why this is the right local optimum

- FastAPI + Pydantic is a strong fit for nested `AlephRun`-style request/response bodies and gives schema + docs for free.
- `mlx-lm` is the Apple-Silicon-native runtime and already supports both a Python API and an OpenAI-compatible local server.
- `mlx-lm` already supports both a Python API and an OpenAI-compatible local server; for v0, Apple Silicon was the fastest known maintainer route.
- Qwen3 8B 4-bit is a realistic local model size for a Mac-based Hackathon while preserving enough quality to make the path meaningful.
- The repo already has a React/Vite scaffold, so switching to a different frontend framework in v0 would slow the product line without clarifying the thesis.

Current note: upstream MLX now also supports Linux CUDA backends, so this plan
should be read as the original hackathon route, not as a current
Apple-Silicon-only constraint on Aleph's local white-box path.

## Architecture decision

### v0 backend shape
Expand Down
72 changes: 72 additions & 0 deletions docs/plans/metric-and-backend-workflow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Metric and Backend Workflow Plan

## Status

Active plan. This is the next path after the v1.0.3 runtime-framing release.

## Problem

Dashboard values are currently a mix of:

- candidate data returned by a run or fixture;
- frontend-derived values such as leakage, compression, and rank;
- fixture values that may have been precomputed with older scoring logic;
- visual projections such as the frontier basin.

That is acceptable only when clearly labeled. It is not good enough for the next
research/product pass because users naturally read the dashboard as measured
evidence.

The most visible symptom is fit drift: when the prompt/output text displayed to
the user has not changed, the fit and epsilon values should not move. More
generally, exact target/output equality must always score as fit `1.0` and
distortion `0.0`.

## Goals

- Make every dashboard number traceable to a run field, deterministic frontend
derivation, or explicitly labeled visual projection.
- Improve fit metrics so exact text equality is stable and obvious.
- Move candidate-level metrics toward the backend/adapter boundary instead of
recomputing them differently in each UI surface.
- Preserve fixture, hosted black-box, and local MLX evidence boundaries.

## Work Tracks

### 1. Better Experiment Paths

- Treat each run as a durable artifact: target, prompts, outputs, scores,
adapter mode, model, decoding, and budget.
- Add a simple route for repeated black-box samples so stability can be measured
rather than filled with a default.
- Keep local MLX as the white-box path for token NLL and future score deltas.

### 2. Better Metrics

- Add normalized exact-match handling before lexical or embedding metrics.
- Split displayed fit into at least two ideas when data supports it:
semantic fit and exact/sequence fit.
- Keep leakage separate from fit; do not let copied target text masquerade as
compression quality.
- Record the metric name/version in the run so fixtures can be refreshed or
compared honestly.

### 3. Better JSON and Backend Workflow

- Return candidate metrics from the backend in the `AlephRun` shape wherever
possible.
- Add an evidence/source label per metric when a value is fixture-backed,
backend-measured, frontend-derived, or projected.
- Stop sorting or interpolating candidate metrics in ways that decouple the
visible prompt/output from its score.
- Add backend tests for exact-match scoring, CJK scoring, leakage, compression,
and selected-candidate ranking.

## First Acceptance Gate

- Exact target/output equality scores fit `1.0` in both the Next.js hosted route
and FastAPI scoring service.
- The explorer displays metrics for the same discrete candidate as the visible
prompt/output.
- Dashboard copy marks fixture/projection evidence clearly enough that a user
does not read a projected basin as measured model parameters.
62 changes: 62 additions & 0 deletions docs/releases/1.0.3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Aleph v1.0.3 - 2026-05-20

Aleph v1.0.3 is a small release-framing patch. It does not change the
`AlephRun` contract, UI surface, or search algorithm. It updates the repository
language around MLX after upstream MLX added Linux CUDA backend support.

## Highlights

- Local white-box search is now described as an MLX-backed route rather than an
Apple-Silicon-only idea.
- Apple Silicon remains the known maintainer path for the current local Qwen3
search server and token-NLL evidence traces.
- Linux CUDA is now named as an upstream-supported MLX backend path for future
local white-box work.
- README, demo guidance, state docs, and UI copy no longer imply that MLX itself
is limited to Apple Silicon.

## Evidence Boundaries

Unchanged from v1.0.2:

- Fixture examples remain precomputed review data.
- Local MLX can expose token text and token NLL only when the adapter returns
those fields.
- Hosted custom API runs remain black-box observations: prompt and output are
real, but logits and token NLL are not exposed.
- Linux CUDA support belongs to upstream MLX's runtime surface; Aleph keeps the
same adapter boundary and evidence labels.

## Runtime Framing

The current local route is still:

```text
search/server.py -> mlx-lm / MLX Qwen -> local_mlx_search -> AlephRun
```

The hardware framing is now:

```text
Apple Silicon: current known-good maintainer route
Linux CUDA: upstream MLX backend route for local white-box expansion
Vercel / hosted web: black-box or fixture unless a separate search service is configured
```

## Validation Matrix

Expected before tagging:

```bash
npm run lint
npm run test
npm --workspace web run build
git diff --check
```

## Release Decision

Suitable as a patch release. It makes the repository's hardware story match the
current MLX direction while preserving Aleph's existing adapter honesty: local
white-box claims still come from returned model-internal evidence, not from the
mere existence of a runtime backend.
2 changes: 2 additions & 0 deletions docs/research/implementation-routes.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ Run an open-weight model where logits are available. Score target likelihood und
- Weakness: local setup and model choice become heavy.
- Likely features: teacher-forced likelihood, per-token target loss, top-k alternatives, entropy.
- Current evidence: `search/aleph_search.py` already runs an MLX Qwen evaluator and computes teacher-forced token NLL for precomputed runs.
- Runtime note: Apple Silicon remains the known maintainer route, while upstream MLX now also supports Linux CUDA backend deployments.
- Integration requirement: expose the result through `apps/api` and `AlephRun` before treating it as a product surface.

## Route C1: Local live-search spike
Expand All @@ -40,6 +41,7 @@ Use the same local MLX model as proposer and evaluator for shallow prompt-fronti
- Strength: very close to the Aleph thesis because the fixed local model is the decoder.
- Weakness: expensive startup, serialized requests, heuristic proposer, and metric/runtime assumptions live in one script.
- Current status: implemented in `search/server.py` and `search/aleph_search.py`.
- Hardware scope: MLX-backed rather than Apple-Silicon-only; Apple Silicon is the present local path and Linux CUDA is the upstream expansion path.
- Likely role: wrap behind `apps/api` as a `local_mlx_search` adapter.

## Route D: ARCA-style discrete optimization
Expand Down
3 changes: 2 additions & 1 deletion docs/research/research-process.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ This file records the research that currently shapes Aleph. It is a source ledge
- Soft-prompt methods support the parameter analogy, but should remain future research until they can be related back to discrete prompt coordinates honestly.
- The repository should be artifact-first, not template-first.
- The current `search/` spike is real local-model evidence for the thesis: a fixed MLX Qwen model can propose prompts, generate outputs, score a frontier, and bake measured examples for the demo.
- MLX's upstream runtime surface now includes Apple Silicon and Linux CUDA paths, so Aleph's local white-box route should be described as MLX-backed rather than Apple-Silicon-only.
- That spike should be treated as an experiment engine, not as the stable product API contract.

## Implementation implications
Expand All @@ -53,7 +54,7 @@ This file records the research that currently shapes Aleph. It is a source ledge
- `packages/fixtures` provides stable demo data.
- `apps/web` consumes `AlephRun` and should not invent its own data model.
- `apps/api` is the stable product API surface and should wrap model/search engines rather than exposing experimental backend shapes directly to the UI.
- `search/` is the current local MLX live-search experiment and is wrapped behind `apps/api` through the `local_mlx_search` adapter. It remains optional until local setup and runtime evidence are available.
- `search/` is the current local MLX live-search experiment and is wrapped behind `apps/api` through the `local_mlx_search` adapter. It remains optional until local setup and runtime evidence are available; Apple Silicon is the known maintainer path and Linux CUDA is now the upstream MLX expansion path.
- Future adapters should be pluggable: `mock`, `hosted_black_box`, `local_openai`, `local_white_box`, `arca`, `gcg`.
- Future research-only or adapter candidates now explicitly include reflective/Pareto search and soft-prompt projection.
- Open questions belong in `docs/open-questions.md`, not in hidden assumptions.
Expand Down
4 changes: 2 additions & 2 deletions docs/state-of-play.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ This is a product/research hypothesis with enough evidence to build around, but
| Run contract | `AlephRun`, candidates, observations, metrics, leakage, frontier helpers. | `packages/core/src/types.ts`, `schemas/aleph-run.schema.json` |
| Fixtures | Multiple demo runs with explicit fixture/simulated modes. | `packages/fixtures/src/` |
| API boundary | FastAPI mock route and local MLX adapter wrapper. | `apps/api/` |
| Local live search spike | MLX/Qwen route can produce AlephRun-compatible adapter output when local setup is running. | `search/`, `docs/verification.md` |
| Local live search spike | MLX/Qwen route can produce AlephRun-compatible adapter output when a local MLX setup is running; Apple Silicon is the known maintainer path and Linux CUDA is now an upstream MLX backend path. | `search/`, `docs/verification.md` |
| Repo checks | Lightweight lint/check suite protects claims, fixtures, links, language, and schema. | `npm run lint` |

## Research Converted Into Product Shape
Expand All @@ -48,7 +48,7 @@ This is a product/research hypothesis with enough evidence to build around, but
| GCG / nanoGCG direction | Parked as a future hard-prompt adapter, not product positioning. |
| vec2text / embedding inversion | Recorded as adjacent inversion work, explicitly not Aleph's core task. |
| Aquin-style instrumentation | Inspired token loss, attribution, exposure, eval, and observation panels, with mode labels. |
| Local MLX spike | Wrapped behind `apps/api` as experimental adapter evidence rather than exposed as product architecture. |
| Local MLX spike | Wrapped behind `apps/api` as experimental adapter evidence rather than exposed as product architecture; hardware scope follows MLX rather than a product-only Apple Silicon assumption. |

## Researched But Not Yet Converted

Expand Down
12 changes: 6 additions & 6 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "aleph",
"version": "1.0.2",
"version": "1.0.3",
"private": true,
"type": "module",
"license": "Apache-2.0",
Expand Down
2 changes: 1 addition & 1 deletion packages/core/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@aleph/core",
"version": "1.0.2",
"version": "1.0.3",
"type": "module",
"main": "src/index.ts",
"license": "Apache-2.0"
Expand Down
Loading