WIP: migrate Ultimate Geography to Rust Brain Brew rewrite with federation - #736
WIP: migrate Ultimate Geography to Rust Brain Brew rewrite with federation#736jeprecated wants to merge 6 commits into
Conversation
|
Example of a single translation PR in this new system, here's the outstanding PR #735 converted into this new system: jeprecated#16 If any strings were missed in the translation overlay it would throw an error 👌 When we make changes to the base English deck all those strings will throw errors for each language (which is the correct behaviour!). |
|
Wow! This looks like a huge amount of work! I wish my review below were more positive, but at the moment I'm not convinced that this is a net improvement. It enables things that are currently tricky (e.g. flexibly allowing different note types!) and AFAIU generally makes future extensibility easier. However, IMO it makes routine tasks that are currently straightforward more finicky and tedious. (OTOH it's possible that I'm misunderstanding how a new, improved workflow would work!) I greatly appreciate how you've continued working on this for so long and I'm generally very excited about improvements to our tooling! (Details/discussion below; I've been sitting on it for several weeks now, being hesitant to post the comment (sorry for the resultant delay!!), but I can't think of any particular improvements to the text.) YAMLI'm not sure that replacing everything with YAML is, on the whole, advantageous, and, if yes, I'm not convinced the proposed structure is most convenient. HTML/CSSFor the HTML/CSS files (deck descriptions and templates) it's IMO a minor but clear downgrade both for editing (one has to worry about proper indentation for the literal blocks) and discoverability (rather than being accessed via standard directory navigation one has to find the correct sub-key in the YAML). CSV vs. YAML(field data (old Both YAML and CSV have flaws as formats, so this comparison is much trickier. The new structure is far more verbose, but that's not necessarily a bad thing (being explicit is often valuable). (Though why do we need Both CSV and YAML have their annoying quoting idiosyncrasies, though at least with CSVs users can avoid them by opening in a spreadsheet editor. (With the old anki-dm, this regularly lead to issues due to quoting inconsistencies, but with the current BrainBrew it seems to work fine.) I'm not sure about grouping by (country, field) rather than (country, language) (i.e. having, say, all translations of country info together). There's no perfect solution since we're trying to flatten 3D data (country × field × language) into 2D or 1D, but I think that we had discussed this during the English field as keyI'm also not convinced about using the English version of the field as the key for translations. It's a frequent, though not universal convention in localisation software (GNU gettext does this, fluent doesn't), and it means that the source version is available side-by-side with the translation, as well as (like you note) enforcing the updating of translations when the source changes. However, here, it has several disadvantages:
Partial asideThe grouping and disambiguation of strings that repeat in English (say This does point to a major advantage of using the English field as the key, but I think that the cons discussed above outweigh the pros. Hardcore geographyIf I understand correctly, the way the "overlapping" notes (those that have some cards in UG and some in HG) have been made to work is that in HG the notes have the "missing" fields added (via "field-fills"), so that we only have a single copy of each note, with both the UG and (old) HG cards. (Also, HG now contains UG rather than being an addition (?).) This is pretty elegant (and clean), but I'm not sure if it's the most convenient behaviour. The problem is that now if someone imports UG, imports HG, and then updates UG (without updating HG) all of the HG cards of the overlapping notes will disappear. I believe that they won't be deleted unless one runs We could obviously warn people that if they import HG they should only update via HG, but I expect such warnings to frequently fail. The issue being (usually (?)) recoverable (if one doesn't empty cards in the meantime) makes me less convinced which behaviour is the "correct" one, since the alternative of having two notes (one for UG and one for HG) in the "overlapping" cases is aesthetically displeasing. (A similar (worse) problem would also occur if one were to use a deck that adds a completely new card (say a hypothetical currency UG adding country->currency cards). Import UG -> import currency UG -> update UG and you lose all your country->currency cards (immediately deleted, though with a warning).) Modifying "fully" UG cardsOne small, but potentially very useful thing that the new brain brew facilitates for decks like HG is modifying cards that are solely in UG (i.e. not the "overlapping" notes, but notes which are in UG and have no cards in HG). When we removed the small dependent territory capitals from UG (and moved them to HG) we also removed the capital hints from the notes which no longer had any collisions within UG (whose "conflictors" were moved to HG) — Basseterre (Saint Kitts and Nevis) which had conflicted with Guadeloupe and Georgetown (Guyana) which had conflicted with the Cayman Islands. (The capital hints on the other side — in Guadeloupe and the Cayman Islands — were kept.) The new field-fills mechanism allows easily modifying these notes in HG to re-add the capital hints. (Technically, we could achieve this with our current set-up by cloning the notes from UG to HG (keeping the same guid) and re-adding the capital hints to the clone, but it would be far more brittle and harder to maintain.) Nix as a requirement?I'm hesitant about having In any case, thanks very much for your continuing efforts to improve BrainBrew and the tooling around AUG! |
|
Hey @jeprecated and @aplaice, hope you both are well! 😊 I did have a look through the diff a few weeks ago but got quickly overwhelmed 😅 — though it's amazing to see this federated deck concept taking shape. 🤩 Like @aplaice, I'm not convinced that the proposed format for translations is ideal. It's the main thing that stood out to me. Having all the translations in one file helps bring consistency but the flat list with the English text as key is really confusing. I think the translation file should follow the structure of australia:
guid: ...
country: Australie
capital: Canberra
capital-hint: ''
...
austria:
guid: ...
country: Autriche
capital: Vienne
capital-hint: ''
...I also agree that the HTML and CSS should live in As an example, let's take overlays/variants/experimental.yaml. I could imagine a file called abkhazia: ''
afghanistan: AF
...The The diff includes the Hardcore Geography extension. Perhaps it's to show a demonstration of the federation system but I would of course prefer to keep this extension in a separate repo. Could you please explain the concept of a "variant" and how differs from an "extension"? |
|
Hello sirs! Sorry for the delay in replying, I've been swamped with work. Both in the professional sense and also doing more work on this PR!
Big thinking work, but it actually didn't take long to do. I've thought about this problem on and off since my email to you two about this rewrite idea 10 months ago, but the actual effort for my initial PR was < 6 hours of actual work. More on that below.
No worries at all, sir. I am not perturbed. Next time don't feel the need to delay, I was not expecting to simply merge this in. I truly meant it when I said:
Though perhaps I could have been clearer on my expectations, the amount of effort I've actually put in here, and my workflow. Though that gets into some of my personal situation, AI/Agentic Development, and other stuff like that - which I don't want to turn this PR into a discussion/debate on these topics. But just for the context of what is to come, and to set your expectations on the state of my work and how I get it done (and why you can trust it) then here's a collapsed box explaining just that: How I developed and validated this migrationI did all of the work for this Brain Brew rust migration using Agentic Development (with AI). In my spare time, while running 5 other projects.
No! 😁 Allow me to explain by first adding context. I created my own startup 4 months ago: Self-Deprecated https://selfdeprecated.ai/ an AI Solutions company, to help bring clients up to speed on Agentic Development. On how to use Agents, what their strengths are, when not to trust them, and how to make sure your "self-healing loop" is the best it can be in order to get the output you desired, while remaining in a "human-out-the-loop" for as much as possible. I use Hunk (https://github.com/modem-dev/hunk/) to get the Agent to make PR comments on it's changes, and I review the actual code. Do I read absolutely every line always? Absolutely not, and nor should I. It depends on the system being created/changes. Hunk helps one have the Agent walk them through a change/PR step by step, while looking at the code, explaining in as much detail what the changes were for and how they relate to the other code segments. For this work (migrating an existing tool from one language to another, though with some bit feature changes too) I had an amazingly easy job. The bulk of the work is in thinking and designing the system, while the core migration is unusually straightforward to test as an output-equivalence problem. Brain Brew’s job (before and after the rewrite) is to produce CrowdAnki decks. Given the same accepted source, it either produces the same parsed JSON and media or it produces a regression, apart from differences we have explicitly reviewed and classified. The migration evidence compares deck and model identity, fields, cards/templates/CSS, notes/GUIDs/tags, descriptions, configuration, and media bytes. Yet at the same time I've add a lot more features! Though almost none of these change the output in the end, they just give better composition and flexibility to the data as it's stored in the repo. Such as combining decks together for the federation. In short:
So all of the above said:
Work Done to Address the commentsRather than line by line reply to each comments (as we three are want to do) I think it'd be faster for everyone if I simply address the things that I have improved. Some I always intended to do, and some are direct responses to good feedback from yourselves Translation overlay format cleanupExample: old and new translation overlay formatsBefore: old alpha schema mixed several intents under translations. "changes", "additions", each of which could be general or field-specific translations:
changes:
Taiwan: "台湾(Taiwan)"
Taipei: "台北(Taipei)"
Georgia:
notes.note.georgia.fields.field.country: Georgien
notes.note.us-georgia.fields.field.region: Georgia
Partially recognised state claimed by China.:
notes.note.taiwan.fields.field.country-info: "中国宣称对台湾拥有主权，但仅被部分国家承认"
additions:
notes.note.yellow-sea.fields.field.country-info: "虽然国际上普遍认为黄海包含了渤海，但在中国，它们通常被视为两个独立的海域。"After: reusable translations, contextual translations, and typed target adaptations are separate translations:
direct:
Taiwan: "台湾(Taiwan)"
Taipei: "台北(Taipei)"
contextual:
notes.note:
andorra.fields.field.flag-similarity.message.variables.country_1:
Moldova: "摩尔多瓦"
target_adaptations:
notes.note.taiwan.fields.field.country-info:
intent: adapt
ownership: translation
expected_source: Partially recognised state claimed by China.
target: "中国宣称对台湾拥有主权，但仅被部分国家承认"
reason: target-language geopolitical wording
notes.note.yellow-sea.fields.field.country-info:
intent: adapt
ownership: translation
expected_source: ""
target: "虽然国际上普遍认为黄海(Yellow Sea)包含了渤海(Bohai Sea)，但在中国(China)，它们通常被视为两个独立的海域。"
reason: "migrated from legacy translations.target_additions; review and describe its target-language purpose"
notes.note.czech-republic.fields.field.country-info:
intent: delete
ownership: translation
expected_source: Also known as Czechia.
reason: "migrated legacy target adaptation; review and describe its target-language purpose"Reviewed source text that should intentionally remain unchanged can be listed under Better authoring formats
Example: external description, CSS, and template includesdeck:
description: !include descriptions/ultimate-geography/en.html
note_types:
note-type.ultimate-geography:
styling: !include styles/ultimate-geography/card.css
card_templates:
template.country-capital:
question_format: !include templates/ultimate-geography/country-capital/question.html
answer_format: !include templates/ultimate-geography/country-capital/answer.html
Examples: reusable message parts and translated formatsBefore: one long translation key# src/data/flag_similarity.csv, represented as YAML for comparison
field.flag-similarity: Iceland (blue background, red and white cross), Norway (red background, blue and white cross)# The Norwegian translation was also one indivisible string
translations:
direct:
"Iceland (blue background, red and white cross), Norway (red background, blue and white cross)": "Island (blå bakgrunn, rødt og hvitt kors), Norge (rød bakgrunn, blått og hvitt kors)"After: reusable translated parts# deck.yaml, notes.note.faroe-islands
field.flag-similarity:
format: "{country_1} ({description_1}), {country_2} ({description_2})"
variables:
country_1:
ref: notes.note.iceland.fields.field.country
country_2:
ref: notes.note.norway.fields.field.country
description_1:
text: blue background, red and white cross
description_2:
text: red background, blue and white cross# overlays/languages/nb.yaml
translations:
direct:
Iceland: Island
Norway: Norge
blue background, red and white cross: "blå bakgrunn, rødt og hvitt kors"
red background, blue and white cross: "rød bakgrunn, blått og hvitt kors"Translating the format itselfBase source: field.flag-similarity:
format: "{country_1} ({description_1}), {country_2} ({description_2})"
variables:
country_1:
ref: notes.note.iceland.fields.field.country
country_2:
ref: notes.note.norway.fields.field.country
description_1:
text: blue background, red and white cross
description_2:
text: red background, blue and white crossThe pieces can be translated normally: translations:
direct:
Iceland: Island
Norway: Norge
blue background, red and white cross: "blå bakgrunn, rødt og hvitt kors"
red background, blue and white cross: "rød bakgrunn, blått og hvitt kors"But the format string itself can also be translated when the target language needs different punctuation, spacing, order, or glue. Example: Simplified Chinese punctuationtranslations:
direct:
Iceland: "冰岛"
Norway: "挪威"
blue background, red and white cross: "蓝底，红白交叉"
red background, blue and white cross: "红底，蓝白交叉"
"{country_1} ({description_1}), {country_2} ({description_2})": "{country_1}({description_1})、{country_2}({description_2})"Output: Example: description before countrytranslations:
direct:
"{country_1} ({description_1}), {country_2} ({description_2})": "{description_1}: {country_1}; {description_2}: {country_2}"Output: Example: sentence-style wordingtranslations:
direct:
"{country_1} ({description_1}), {country_2} ({description_2})": "{country_1}: {description_1}; {country_2}: {description_2}."Output: Example: contextual format translationUse contextual if only one field should use a different format: translations:
contextual:
notes.note:
faroe-islands.fields.field.flag-similarity.message.format:
"{country_1} ({description_1}), {country_2} ({description_2})": "{country_1} resembles {country_2}: {description_1} vs {description_2}"Output for that field only: Structured media references and integrity
Example: single/multiple images, media declarations, and strict verificationA field can contain one stable image reference or an ordered image sequence, such as UG's blurred and normal Bolivia flags: # deck.yaml
notes:
note.bolivia:
fields:
field.flag:
- !image media.ug-flag-bolivia-blur-svg
- !image media.ug-flag-bolivia-svg
field.map: !image media.ug-map-bolivia-png
media: !include media.yamlThe stable IDs resolve through the separately maintained declaration map: # media.yaml
media.ug-flag-bolivia-blur-svg:
path: ug-flag-bolivia-blur.svg
sha256: f13669cab4afb991b9851a9c55bb94be5a2c91303a6f8bbb4407a9ffd67951c7
media.ug-flag-bolivia-svg:
path: ug-flag-bolivia.svg
sha256: 3010bf58668ac58ae5a1b614867cf94c53b229f2d26679c0ca04cae6d936ced1
media.ug-map-bolivia-png:
path: ug-map-bolivia.png
sha256: d461a4cb0d4b845fdc61de1123efca9cc766aaea10dabd95202a1beb980fca7dThis means a file can be renamed by updating its declaration without rewriting every note-field reference. Brain Brew renders the references as safe Anki-compatible brainbrew media hash --manifest brainbrew.yaml --all-targets --media-root media
brainbrew verify --manifest brainbrew.yaml --all-targets --media-root media
brainbrew export crowdanki \
--manifest brainbrew.yaml \
--target en-standard \
--media-root media \
--out build/crowdanki/en-standard
Rust crate releaseI had never intend to put Nix/NixOS as a dev dependency, that's just what I happen to use (because it is truly excellent). I simply left it in my test PR rather than put in the effort to actually release anything properly, before it got approved and the effort was worth it 😁
Example: installation and normal CLI usageBefore # Mostly contributor/developer style usage
nix run . -- --help
cargo run -- compose --manifest brainbrew.yaml --target de-standardAfter cargo install brainbrew --version 1.0.0-alpha.3 --locked
brainbrew compose --manifest brainbrew.yaml --target de-standardLanguage-first project metadata
Example: language-first manifest metadataBefore targets:
de-standard:
overlays:
- overlay.translation.de
de-extended:
overlays:
- overlay.variant.extended
- overlay.translation.deTools had to infer language/variant meaning from target names. After languages:
en:
display_name: English
source: true
primary_target: standard
targets:
experimental: en-experimental
extended: en-extended
hardcore-extended: en-hardcore-extended
hardcore-standard: en-hardcore-standard
standard: en-standard
de:
display_name: German
translation_overlays:
base: overlay.translation.de
hardcore: overlay.translation.hardcore.de
primary_target: standard
targets:
experimental: de-experimental
extended: de-extended
hardcore-extended: de-hardcore-extended
hardcore-standard: de-hardcore-standard
standard: de-standard
translation_profile:
structural_fields:
- field.flag
- field.map
metadata_categories:
- key: deck-metadata
label: Deck metadata
paths:
- deck.name
- deck.descriptionThis means that all deck extensions can have their own translations, yet still the whole can be understood to be under one language group. This will help tools show all the translations for one language, for all decks/extensions. Safer extension composition
Examples: field additions, blank-only fills, and expected-base checksExperimental adds one field definition and supplies values only where needed: id: overlay.variant.experimental
kind: extension
field_additions:
note-type.ultimate-geography:
fields:
field.region-code: Region code
values:
note.afghanistan:
field.region-code: AF
note.albania:
field.region-code: ALHardcore can fill fields that already exist in the shared note model: id: overlay.extension.hardcore.field-fills
kind: extension
field_fills:
note.hardcore-bali:
field.capital: Denpasar
field.flag:
- !image media.ug-flag-bali-blur-png
- !image media.ug-flag-bali-pngA value that intentionally replaces existing source records exactly what it expects to replace: note_types:
note-type.ultimate-geography:
intent: merge
variables:
variant.name-suffix:
intent: replace
value: " [Extended]"
expected_base:
value: ""Translator workflows
Example: translator CLI workflowBefore brainbrew verify --manifest brainbrew.yaml --target da-standardThen manually inspect YAML failures and decide what was missing/stale/unchanged. After brainbrew translations --manifest brainbrew.yaml --all-targets --summary
brainbrew translations --manifest brainbrew.yaml --target da-standard --context --status missing
brainbrew translations --manifest brainbrew.yaml --target da-standard --context --status missing --apply --interactiveReport, summary, and context modes are read-only. Summary mode provides compact per-language and per-overlay counts; context mode shows missing or stale text with its source, target, note, field, card, and duplicate-source context.
When English source text changes, an outdated translation can be retained as an explicit stale record rather than being treated as current. Under UG’s lenient policy, Brain Brew warns about that record and continues using its target text until a maintainer resolves it after review; strict translation coverage rejects unresolved stale records. Orphaned dictionary keys, invalid contextual paths or target adaptations, and broken references still fail normal verification. Deck WorkbenchAdded a local "Workbench", a webpage one can run which shows the deck contents and allows for editing inline while previewing a Note/Card/Field. 
Example: Workbench workflow brainbrew workbench serve \
--manifest brainbrew.yaml \
--enable-writeThen use the local Workbench to review content and edit its underlying source or translation fields: This was a stretch goal I had in mind for a while, but I decided to take a crack at it now. Brain Brew 1.0.0-alpha.3 includes Workbench write support in the normal release. It starts read-only; The Workbench shows deck content in context, including source and translated text, cards, metadata, and other languages for comparison. Edits remain drafts until someone confirms Apply, which then updates the canonical YAML and owned translation overlays in the local working tree. The write workflow is still being hardened, so it should be used on a version-controlled checkout. My longer-term goal is to make this a complete translator-facing GUI where people can update translations, create new ones, review the resulting changes, and compare their work with other languages without needing to edit YAML directly. ❗ This tool is very much a work in progress! It still has some sharp edges and strange display stuttering. But these will be fixed! This is just how I imagine a tool could support the workflow much better. Anyways all of this is up for future consideration/work, but I hope you at least like the direction. I'm open to any and all ideas, it of course matters what people would want to do / how they work want to work. But the more different options the better, in my books! Discussion PointsThings here are up for discussion. I have taken my own liberties in designing a system I think is good, but am open to being wrong about! Support for the below in Brain Brew does not dictate that UG need use it too. As I said just above more options are better! Yaml as a formatI hope that the above improvements will help with yaml being the main storage format. Especially when taken as it being the git repo source of truth, not that everyone needs to work via the yaml files (though I personally think it's much nicer to do so now, compared to the csvs before). I have kept this Brain Brew rewrite working in the same way the old one did: a hub and spoke type of format system. The old system translated everything into "Deck Parts" before then translating to/from CrowdAnki/CSV. This new system has the same general hub-and-spoke shape, except I have made the Canonical Deck YAML itself the stored intermediate representation. That makes it straightforward to add CSV export and import back in, including a workflow where someone exports to CSV, edits it, and imports it again. The CSV adapter is not implemented in alpha.3 yet; it is an option I can add if that is something people want. I still think the yaml is much better than that old system. English as the Translation fieldI truly believe that this is a necessary evil (hopefully less evil now that I've gotten rid of the big long horrible strings). I've also considered using stable note IDs or country codes as translation keys, with English treated as just another translation. Those identifiers solve identity, but by themselves give translators even less context: they still need the source text, note, field, card, and related translations to understand what they are editing. The new context views and Workbench are intended to make the current English-source model much easier to work with. The workbench editing workflow will help a lot with this problem too. Hardcore geography
All I present for it is that it is doable to have it here in this repo! It was simpler for my demo to have it here too.
Oh this was totally an oversight on my part. I have preserved the historical UG note GUIDs rather than allowing them to be regenerated, as well as the 45 meaningful historical Hardcore note GUIDs. The canonical source uses readable stable note IDs for composition, while explicit I also reorganised Hardcore so that standalone and companion exports reuse the same 45-note content overlay. The companion keeps its own deck identity, while its note GUIDs, fields, tags, and note model stay aligned with the corresponding standalone deck. The migration evidence collector checks this relationship for all 12 localised Hardcore language pairs. That gives us an explicit structural check for the identity/model behaviour behind this import-update concern, rather than relying only on generated GUID overrides. I would still want the final workflow exercised in Anki before treating that as a complete end-to-end import guarantee. |
|
Ahh! 🙈 Somehow I accidentally pushed up the changes into master here. I have now reverted it, which required me to disable the branch security force push policy. My deepest apologies. It was up for about an hour, I believe. This is very confusing as the branch policy already had "Require Pull Requests" so I'm not even sure how I was able to do it in the first place 🤔 Either way I'll update my own local remotes to make sure this cannot happen again. Feel free and revoke my access to master anyways, I'm completely fine with that 🙏 Edit: Ahhh I see. All branch protection rules have the following setting one must tick, otherwise none of it applies to Admins 🙈 Seems rather odd. Would be better with the following settings to select who can force push explicitly, and just enable it when needed (imo) 
And since I used jj to push to my own master branch in my fork it decided to push to master on each remote, because it could... damn. Quite unexpected. I will change my own branch so that does not occur again due to this. |
|
Replaced this PR with #743 coming from the right branch (Github does not allow one to change branch targets on an open PR 😢 ) so that this dumb issue does not happen again. Sorry to break up the conversation, feel free and respond here it you like 👍 |
3 participants
Hello all! Jordan here. I changed my github username from ohare93, if you're confused 😁 anyways, I finally got around to it! Deck Federation is now in Brain Brew!
This is a work-in-progress migration to the new Rust Brain Brew workflow, paired with jeprecated/brain-brew#60 things are looking very bright!
What changed
brainbrew.yamlmanifest.deck.yamlas the canonical English source.brainbrew verify --manifest brainbrew.yaml --all-targets --media-root mediapasses for 71targets.
Related issues / unlocked work
Area (Km²),LatitudeandLongitude#362: Makes area/latitude/longitude fields a good future extension package.Available for questions, more to come soon! 👀