-
Notifications
You must be signed in to change notification settings - Fork 0
[codex] Prepare Aleph v1.0.3 MLX CUDA runtime framing #21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
91d049a
Release v1.0.3 MLX CUDA runtime framing
Jah-yee f846549
Tighten explorer i18n and hosted mode copy
Jah-yee c05fd75
Polish Chinese explorer labels
Jah-yee 6a16e03
Stabilize explorer metrics and scoring plan
Jah-yee 04ac144
Fix Chinese out-of-bound note
Jah-yee d616631
Stabilize exact-match fit scoring
Jah-yee 5b12990
Guard empty similarity scoring
Jah-yee ce20dd8
Trim metrics plan trailing whitespace
Jah-yee File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| # Metric and Backend Workflow Plan | ||
|
|
||
| ## Status | ||
|
|
||
| Active plan. This is the next path after the v1.0.3 runtime-framing release. | ||
|
|
||
| ## Problem | ||
|
|
||
| Dashboard values are currently a mix of: | ||
|
|
||
| - candidate data returned by a run or fixture; | ||
| - frontend-derived values such as leakage, compression, and rank; | ||
| - fixture values that may have been precomputed with older scoring logic; | ||
| - visual projections such as the frontier basin. | ||
|
|
||
| That is acceptable only when clearly labeled. It is not good enough for the next | ||
| research/product pass because users naturally read the dashboard as measured | ||
| evidence. | ||
|
|
||
| The most visible symptom is fit drift: when the prompt/output text displayed to | ||
| the user has not changed, the fit and epsilon values should not move. More | ||
| generally, exact target/output equality must always score as fit `1.0` and | ||
| distortion `0.0`. | ||
|
|
||
| ## Goals | ||
|
|
||
| - Make every dashboard number traceable to a run field, deterministic frontend | ||
| derivation, or explicitly labeled visual projection. | ||
| - Improve fit metrics so exact text equality is stable and obvious. | ||
| - Move candidate-level metrics toward the backend/adapter boundary instead of | ||
| recomputing them differently in each UI surface. | ||
| - Preserve fixture, hosted black-box, and local MLX evidence boundaries. | ||
|
|
||
| ## Work Tracks | ||
|
|
||
| ### 1. Better Experiment Paths | ||
|
|
||
| - Treat each run as a durable artifact: target, prompts, outputs, scores, | ||
| adapter mode, model, decoding, and budget. | ||
| - Add a simple route for repeated black-box samples so stability can be measured | ||
| rather than filled with a default. | ||
| - Keep local MLX as the white-box path for token NLL and future score deltas. | ||
|
|
||
| ### 2. Better Metrics | ||
|
|
||
| - Add normalized exact-match handling before lexical or embedding metrics. | ||
| - Split displayed fit into at least two ideas when data supports it: | ||
| semantic fit and exact/sequence fit. | ||
| - Keep leakage separate from fit; do not let copied target text masquerade as | ||
| compression quality. | ||
| - Record the metric name/version in the run so fixtures can be refreshed or | ||
| compared honestly. | ||
|
|
||
| ### 3. Better JSON and Backend Workflow | ||
|
|
||
| - Return candidate metrics from the backend in the `AlephRun` shape wherever | ||
| possible. | ||
| - Add an evidence/source label per metric when a value is fixture-backed, | ||
| backend-measured, frontend-derived, or projected. | ||
| - Stop sorting or interpolating candidate metrics in ways that decouple the | ||
| visible prompt/output from its score. | ||
| - Add backend tests for exact-match scoring, CJK scoring, leakage, compression, | ||
| and selected-candidate ranking. | ||
|
|
||
| ## First Acceptance Gate | ||
|
|
||
| - Exact target/output equality scores fit `1.0` in both the Next.js hosted route | ||
| and FastAPI scoring service. | ||
| - The explorer displays metrics for the same discrete candidate as the visible | ||
| prompt/output. | ||
| - Dashboard copy marks fixture/projection evidence clearly enough that a user | ||
| does not read a projected basin as measured model parameters. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| # Aleph v1.0.3 - 2026-05-20 | ||
|
|
||
| Aleph v1.0.3 is a small release-framing patch. It does not change the | ||
| `AlephRun` contract, UI surface, or search algorithm. It updates the repository | ||
| language around MLX after upstream MLX added Linux CUDA backend support. | ||
|
|
||
| ## Highlights | ||
|
|
||
| - Local white-box search is now described as an MLX-backed route rather than an | ||
| Apple-Silicon-only idea. | ||
| - Apple Silicon remains the known maintainer path for the current local Qwen3 | ||
| search server and token-NLL evidence traces. | ||
| - Linux CUDA is now named as an upstream-supported MLX backend path for future | ||
| local white-box work. | ||
| - README, demo guidance, state docs, and UI copy no longer imply that MLX itself | ||
| is limited to Apple Silicon. | ||
|
|
||
| ## Evidence Boundaries | ||
|
|
||
| Unchanged from v1.0.2: | ||
|
|
||
| - Fixture examples remain precomputed review data. | ||
| - Local MLX can expose token text and token NLL only when the adapter returns | ||
| those fields. | ||
| - Hosted custom API runs remain black-box observations: prompt and output are | ||
| real, but logits and token NLL are not exposed. | ||
| - Linux CUDA support belongs to upstream MLX's runtime surface; Aleph keeps the | ||
| same adapter boundary and evidence labels. | ||
|
|
||
| ## Runtime Framing | ||
|
|
||
| The current local route is still: | ||
|
|
||
| ```text | ||
| search/server.py -> mlx-lm / MLX Qwen -> local_mlx_search -> AlephRun | ||
| ``` | ||
|
|
||
| The hardware framing is now: | ||
|
|
||
| ```text | ||
| Apple Silicon: current known-good maintainer route | ||
| Linux CUDA: upstream MLX backend route for local white-box expansion | ||
| Vercel / hosted web: black-box or fixture unless a separate search service is configured | ||
| ``` | ||
|
|
||
| ## Validation Matrix | ||
|
|
||
| Expected before tagging: | ||
|
|
||
| ```bash | ||
| npm run lint | ||
| npm run test | ||
| npm --workspace web run build | ||
| git diff --check | ||
| ``` | ||
|
|
||
| ## Release Decision | ||
|
|
||
| Suitable as a patch release. It makes the repository's hardware story match the | ||
| current MLX direction while preserving Aleph's existing adapter honesty: local | ||
| white-box claims still come from returned model-internal evidence, not from the | ||
| mere existence of a runtime backend. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.