BUILDING.md

Building from Source

Full setup: Meta developer credentials, Universal Links, personal signing overlays, in-app configuration, and troubleshooting. Start with the Quick Start if you just want it running.

Availability and first use

This guide documents building from source on a Mac with Xcode 26+, then installing on an iPhone running iOS 26+ with your signing configuration. If you only want to use the app, the public beta is on TestFlight and needs no build. Any other invitation or separately distributed build has its own access instructions; do not assume that a Meta developer invitation also installs the iPhone app.

Compatible Meta glasses use Meta AI pairing, developer configuration, registration and camera permissions. Choosing a different AI provider does not replace those steps. Phone-based features and camera fallback let you explore before completing glasses setup.

After following the build steps below, check one thing at a time:

1. Registration: complete Meta setup and the callback into Avenkin. If Connect fails, check the credentials and the troubleshooting section before repeating registration.
2. Permissions: grant the microphone/camera and service permissions needed for the task; denied access needs a permission fix, not a different model.
3. Fresh image: request a new capture of a distinctive object. A held preview is not proof that the current camera session is delivering images.
4. Audio: ask a short question and check both microphone input and the intended spoken-output route.
5. Useful task: read a clearly printed sign, then ask about a calendar entry or save a simple note. Confirm the result in its destination. Native contact creation and the unified “handle this” transaction remain planned.

These are manual checks, not a claim that a guided setup checker has shipped. If using VoiceOver, follow the accessibility notes; complete independent setup and recovery validation is still pending. For offline use, download the required models first and follow the offline voice checklist, including a disconnected trial.

Cloud AI and speech credentials are configured separately. Verify the supported sign-in/API route for the selected provider; a personal chat subscription is not itself an API key. For processing destinations and the limits of the network monitor, see Privacy and control.

Build and configure

1. Clone

git clone https://github.com/straff2002/OpenGlasses.git
cd OpenGlasses

2. Meta Developer Credentials

1. Go to wearables.developer.meta.com
2. Create an account, organization, and app
3. Note your Meta App ID and Client Token
4. In Meta dashboard → iOS settings, enter your Apple Team ID, Bundle ID, and Universal Link URL

3. Configure Meta keys

Your Meta App ID and Client Token are build settings, substituted into the committed OpenGlasses/Info.plist at build time. Put them in the gitignored project.local.yml — never in the plist itself:

targets:
  OpenGlasses:
    settings:
      base:
        MWDAT_META_APP_ID: "<your Meta app id>"
        MWDAT_CLIENT_TOKEN_HASH: "<hash after the second | of your client token>"

Then re-run ./Scripts/generate-xcodeproj.sh. A client token looks like AR|<app id>|<hash>; only the trailing hash goes in MWDAT_CLIENT_TOKEN_HASH, since the plist composes the rest. ./Scripts/setup-local-dev.sh preserves both values when it rewrites project.local.yml.

Without them the app builds and launches normally, but DAT registration never completes — and because the camera permission prompt is gated behind registration, the symptom is a Connect button that appears to do nothing rather than an obvious credentials error. The app names this at launch (⚠️ Glasses config is still the committed placeholder) and in the connect-failure message.

Don't override INFOPLIST_FILE to a personal plist copy. That mechanism was removed: the copy went stale and silently dropped newly added usage descriptions (App Store ITMS-90683), and when it was withdrawn it took the Meta credentials with it.

4. Universal Links

Host an apple-app-site-association file at https://YOUR-DOMAIN/.well-known/apple-app-site-association:

{
  "applinks": {
    "details": [{
      "appID": "YOUR_TEAM_ID.YOUR_BUNDLE_ID",
      "paths": ["/YOUR-PATH/*"]
    }]
  }
}

Both ends must match your build, or registration silently never completes (#246):

• The AASA appID must be your Team ID + bundle ID — a copied AASA still declaring the upstream app makes iOS refuse the domain association.
• Your entitlements file must contain the associated-domains entitlement for the same domain (applinks:YOUR-DOMAIN). Without it, iOS never even attempts the link-back. If you use a personal entitlements overlay, check it has:

<key>com.apple.developer.associated-domains</key>
<array>
    <string>applinks:YOUR-DOMAIN</string>
</array>

The failure mode when either end is wrong: approval completes in the Meta AI app, but the approval callback (a Universal Link) never reaches Avenkin — registration never finalises, the devices listener never fires, and connecting fails. The in-app error names the stalled registration state and points here.

5. Enable Developer Mode

On iPhone: Meta AI app → Settings → About → tap version number 5 times → toggle Developer Mode on.

With DAT 0.9 and later, also apply/update the Developer Mode settings on the glasses in Meta AI. The phone's toggle alone is not sufficient. Update Meta AI first if that option is missing. Bluetooth audio, a connected device and granted camera permission can all work while the camera's device session fails before starting. Meta traced a matching Device unavailable startup failure to settings that had not been pushed to the glasses (Meta's explanation and confirmed recovery).

Camera transport also needs the Hotspot Configuration and Access Wi-Fi Information capabilities in the signed app. Both keys are in the committed entitlements; existing personal overlays can add them with ./Scripts/setup-local-dev.sh. Allow Xcode to update provisioning profiles for these capabilities, then rebuild and install. Accept the glasses Wi-Fi Join prompt and allow Local Network access when asked. Meta's CameraAccess entitlements show the required keys.

If startup instead reports Session ended by device, end any broadcast shown in Meta AI. If it persists, close the glasses in their charging case for a minute and retry. This is a known device-side session recovery issue, and repeated app restarts cannot reliably clear it. An unexpectedError without a recognized reason does not identify either condition; export a new support report after trying again.

6. Build & Run

The repo ships project.base.yml plus an optional project.local.yml overlay; XcodeGen writes OpenGlasses.xcodeproj locally. Do not commit the generated project.

brew install xcodegen cmake
./Scripts/fetch-mediapipe-frameworks.sh    # vendored binaries, fetched not committed
./Scripts/fetch-llamacpp-framework.sh      # fetch or build the pinned local inference engine
./Scripts/generate-xcodeproj.sh
open OpenGlasses.xcodeproj

Two vendored packages keep their binaries out of the repository and obtain them with a script: MediaPipe Tasks (downloaded, checksum-verified) and the llama.cpp engine (built from the exact revision pinned in Vendor/LlamaCpp/REVISION, then recorded in SHA256SUMS). Both scripts are idempotent — re-running them after the first time costs a second. The llama build needs cmake; it will tell you rather than installing anything itself.

Continuous integration

Two pipelines build this repository, and they are deliberately built to agree with each other.

XcodeGen is pinned and checksum-verified. Both Xcode Cloud (via ci_scripts/ci_post_clone.sh) and GitHub Actions (.github/workflows/) install it with Scripts/install-xcodegen.sh, which downloads the release named in Scripts/xcodegen-pin.env and refuses to run it unless the sha256 matches. XcodeGen decides the contents of the project file every build compiles, so it is verified to the same standard as the vendored MediaPipe and llama.cpp binaries. Neither pipeline uses Homebrew — Xcode Cloud's network cannot resolve ghcr.io, Homebrew's bottle host, and an unpinned tap version is not a build input anyone reviewed.

Bumping XcodeGen means editing one file. Change XCODEGEN_VERSION in Scripts/xcodegen-pin.env and recompute the digest:

curl -fsSL "https://github.com/yonaskolb/XcodeGen/releases/download/<version>/xcodegen.zip" | shasum -a 256

Local development still uses whatever brew install xcodegen gives you; the pin governs CI, where nobody is watching the version go by.

Package resolution is frozen to ci_scripts/Package.resolved. Xcode Cloud cannot resolve packages at all (automatic resolution is disabled there and xcodebuild -resolvePackageDependencies fails with exit 74), so it copies that lockfile into the generated project. GitHub Actions now does the same and passes -onlyUsePackageVersionsFromResolvedFile, so a pin that cannot be satisfied fails the run rather than silently resolving to something else — the two pipelines compile the same dependency graph from the same commit, or one of them goes red.

Run ./Scripts/update-package-resolved.sh and commit the result whenever you add, remove or bump an SPM dependency, or CI will build the old graph.

Reviewing dependency updates. .github/dependabot.yml automates one ecosystem and deliberately leaves the other manual:

• GitHub Actions — automated, weekly. Dependabot reads the # vX.Y.Z comment beside each pinned commit SHA and moves both together, which is what keeps SHA pinning from quietly rotting. Minor and patch bumps arrive grouped; majors arrive one per PR.

• SwiftPM — manual, and it has to be. Dependabot's swift support reads a root Package.swift + Package.resolved. Both exist here, but neither builds the app: the project is generated by XcodeGen from project.base.yml, and the graph both CI paths compile is ci_scripts/Package.resolved. Dependabot has no XcodeGen support, so it would open PRs against a manifest nothing links while the authoritative pins stayed where they were.

  The manual path, run when a dependency is added, removed or bumped and otherwise on a scheduled review:

  # 1. Read the upstream release notes for anything you are about to move. The 0.x pins
  #    (meta-wearables-dat-ios, swift-huggingface) are pinned exactly, not `from:`, because
  #    those packages break API on minor bumps — see the comments in project.base.yml.
  # 2. Edit the version or revision in project.base.yml. That file is the authority.
  ./Scripts/generate-xcodeproj.sh
  xcodebuild -resolvePackageDependencies -project OpenGlasses.xcodeproj -scheme OpenGlasses
  ./Scripts/update-package-resolved.sh
  # 3. Commit project.base.yml and ci_scripts/Package.resolved together, so the graph that was
  #    reviewed is the graph that gets built.

  A PR that moves a pin without the matching lockfile line is a review of something CI will not build. The sbom job's component list is generated from ci_scripts/Package.resolved, so an unrefreshed lockfile also shows up as a stale SBOM.

Pull requests get a dependency review. .github/workflows/security-scan.yml runs GitHub's dependency-review-action on every PR and fails on a high-severity advisory or a denied licence in whatever the dependency graph does cover. It is a floor, not the SwiftPM review above: the graph it sees is the root manifest's, so treat a clean dependency review as necessary and not sufficient.

The unit workflow also writes a provenance.txt artifact (retained 90 days): commit, runner, macOS build, Xcode and SDK versions, the XcodeGen pin, the lockfile digest and the digest of each Vendor/*/SHA256SUMS. It is what lets a green run still mean something months later.

Every run produces an SBOM. Scripts/generate-sbom.sh writes a CycloneDX 1.5 document from the pins already in the repository — ci_scripts/Package.resolved, the Vendor/*/REVISION and SHA256SUMS files, the MediaPipe fetch script's archive digests, Scripts/xcodegen-pin.env, Scripts/gitleaks-pin.env and the model catalogues — with a hash on every component that has one. It needs no network, so an SBOM can be produced for any commit, including an old one.

It is deterministic on purpose: the timestamp comes from SOURCE_DATE_EPOCH or HEAD's committer date and the serial number is derived from the component list by hash, so the same commit yields a byte-identical document. The sbom job runs it twice and diffs, because a manifest nobody can regenerate is a manifest nobody can check.

./Scripts/generate-sbom.sh                 # writes sbom.cdx.json

Extraction from Swift sources is anchored and asserted — if a pattern stops matching, the script fails rather than emitting a smaller SBOM. Adding a dependency, a vendored binary or a model means the generator either picks it up from the manifest it already reads, or it needs a new input adding here. A manifest that silently loses a component is worse than none, because it is believed.

A release gets a dossier. Scripts/release-dossier.sh writes one Markdown file for a commit and a built artifact: the commit and its link, the CI run URL when GITHUB_RUN_ID is set, the provenance record and SBOM digests, the artifact's sha256, its CFBundleShortVersionString and CFBundleVersion read from the artifact's own Info.plist (an .ipa, .app or .xcarchive), the signing identity from codesign -dv --verbose=2, and the list of gates that guard the branch.

./Scripts/release-dossier.sh --commit <sha> \
  --artifact build/OpenGlasses.ipa \
  --provenance provenance.txt      # downloaded from the CI run's artifacts

When to run it: on a tag, or at App Store submission. It is not wired into a release workflow, because there is no release workflow — the store build runs in Xcode Cloud, whose configuration lives in App Store Connect where the repository cannot see or version it. A job that has never run is not evidence of anything. The Release dossier workflow (workflow_dispatch) produces the source half on demand for any commit; the artifact half needs running the script on the machine that holds the build.

It reads no secret. codesign -dv reports the certificate subject — an organisation name and a team identifier, both printed on every App Store listing — and never touches a key.

Every field the script could not fill says so in the document, and the reason lands in a Gaps section. A dossier with no artifact says it describes the source and not a release; a dossier generated from a dirty tree says the artifact may not match the commit. An unstated absence reads as a claim, so none of them are left unstated. One gap will not close from inside this repository: Apple re-signs and re-packages what it distributes, so the digest that identifies the shipped build is the one App Store Connect reports, and it has to be captured there.

The published website is staged, not the checkout. Scripts/stage-pages-site.sh copies an explicit allowlist into _site/ and then independently refuses to publish a tree containing a denied path. New pages go in site/, which is staged to the site root as a tree; anything else published from outside site/ is added to that allowlist. Scripts/check-pages-site.py then checks the staged tree: the paths shipped builds read are still there (Scripts/site-legacy-paths.txt), no page loads anything from another site, and every internal link resolves. The vault guide page (site/field-assist/vault-guide/) is converted from docs/field-assist-vault-guide.md: after changing the guide, run python3 Scripts/build-vault-guide-page.py --pdf and commit the page and the PDF, or the check fails (--pdf prints docs/field-assist-vault-guide.pdf with a local Chrome). To preview locally:

./Scripts/stage-pages-site.sh && python3 Scripts/check-pages-site.py _site
node Scripts/tests/site-auth-forward.test.js
python3 -m http.server 8000 --directory _site

Default generate includes watch and unit tests. To build a slimmer project locally (iPhone + widget only):

cp .openglasses-generate.env.example .openglasses-generate.env   # gitignored
./Scripts/generate-xcodeproj.sh

Or one-off: OPENGLASSES_SKIP_WATCH=1 OPENGLASSES_SKIP_TESTS=1 ./Scripts/generate-xcodeproj.sh

Optional: personal signing & Meta config

Team ID, entitlements, and Meta keys differ per developer. Those settings live in gitignored files (never committed), merged on top of the shared spec via project.local.yml:

File (gitignored)	Purpose
project.local.yml	Team ID + DEVELOPMENT_TEAM, personal entitlements paths, and the Meta credentials MWDAT_META_APP_ID / MWDAT_CLIENT_TOKEN_HASH (see project.local.yml.example)
Config/Entitlements/Personal/*.entitlements	Capabilities your provisioning profile supports

First-time setup from the templates:

./Scripts/setup-local-dev.sh

Edit project.local.yml (developmentTeam) and the files under Config/ as needed, then run ./Scripts/generate-xcodeproj.sh again.

If you only need Xcode’s automatic signing with the shared entitlements, skip the local overlay and set your team in Xcode after opening the generated project.

Select your iPhone, fix signing if prompted, and run (⌘R).

---

Configuration

All settings are in-app — no source code editing needed.

API Keys (Settings → Assistant → AI Models)

Service	Purpose	Where to Get
Anthropic	Claude LLM	console.anthropic.com
OpenAI	GPT + Realtime	platform.openai.com
Google Gemini	Gemini Live	aistudio.google.com
Groq	Fast inference	console.groq.com
DeepSeek	DeepSeek LLM	platform.deepseek.com
Mistral AI	Mistral LLM	admin.mistral.ai
ElevenLabs	Natural TTS	elevenlabs.io
Perplexity	Web search	perplexity.ai/settings/api

ChatGPT subscription sign-in

Choose ChatGPT (Subscription) in Settings → Assistant → AI Models to use the models enabled for your ChatGPT workspace without adding an OpenAI API key. Normal browser sign-in is recommended on iPhone. After connecting, Fetch available models loads the account-scoped Codex catalog and uses the model marked as its current default.

For advanced or headless sign-in, expand Other sign-in options and choose Use a device code. First enable Device Code Authorization in ChatGPT Settings → Security. Enter the one-time code only at auth.openai.com and never share it; Avenkin keeps the device ceremony in memory and stores only the resulting account credentials in Keychain. See the official authentication guidance.

The existing OpenAI (GPT) provider remains separate: it uses an API key, usage-based API billing, and is still required for OpenAI Realtime voice.

xAI subscription sign-in

Choose xAI (Grok) in Settings → Assistant → AI Models and tap Sign in with xAI to use Grok models on a SuperGrok subscription without an xAI API key. The sign-in token is accepted by the public xAI API, so the model works like a keyed one — photos and tools included — and Fetch available models lists the chat models the account can use. A key typed into the model is always used instead of the sign-in. The account credentials are stored in Keychain on this device only.

Services (Settings → Connections → Services & Integrations)

Service	Settings
ElevenLabs	API key + voice selection (24 voices)
Web Search	Optional Perplexity / Tavily / Brave keys + SearXNG instance URL. Order: Perplexity → Tavily → Brave → SearXNG → Tavily free tier (keyless) → DuckDuckGo
Live Streaming	Platform + RTMP URL + stream key + chat read-aloud (Twitch channel, rate, mentions-only)
OpenClaw	Enable + connection mode + host/port + token

Home Assistant's URL and Long-Lived Access Token are under Settings → Everyday → Home, Music & Messages.

---

Troubleshooting

Issue	Solution
Wake word not detecting	Tap mic button to restart; check Bluetooth audio routing
No audio through glasses	Verify Bluetooth connection in iOS Settings
Glasses not connecting	Tap "Connect to Glasses"; enable Developer Mode in Meta AI app
Camera says "Your glasses need an update"	The glasses-side DAT software lags the SDK this app builds against. Update the Meta AI app from the App Store, power-cycle the glasses in their case, and if it persists unlink/relink the app connection in Meta AI. If none of that helps, Meta hasn't shipped the matching glasses update yet — everything else works; camera/streaming resume automatically once the rollout lands
Glasses mic silent (beeps play, nothing transcribed)	Bluetooth audio pairing is broken while the app link still works — happens after a glasses reset. Forget the glasses in iOS Settings → Bluetooth, put them in the case with the lid open, hold the case button until the LED pulses blue, re-pair via the Meta AI app prompt, then restart the iPhone. Re-grant the app connection + camera permission after
HomeKit not finding devices	HomeKit initializes on first tool call — say "list smart home devices" and wait 10s
Local model won't load ("not enough memory")	Close other apps in the app switcher and use Try again in Download & Manage Models (live headroom shown there), or switch to a smaller model (0.5B–2B)
Model download stuck	Keep app in foreground; downloads continue if briefly backgrounded
"Untrusted Developer"	Settings → General → VPN & Device Management → Verify (requires internet)

---