E2E: tauri driver setup#8095
Draft
qnixsynapse wants to merge 16 commits into
Draft
Conversation
Contributor
Review: E2E tauri-driver setupGood foundational work setting up the tauri-driver + WebdriverIO harness. The architecture is sound. A few issues to address before this is merge-ready: Must Fix
Should Fix
Nice to Have
Recommendation: fix needed |
![tokamak-pm[bot]](https://avatars.githubusercontent.com/in/2987715?s=60&v=4){kind=small}
There was a problem hiding this comment.
Re-Review: New commit since Apr 28
Reviewed the new commit 6e47ca08 ("fix(e2e): address review feedback on tauri-driver harness").
Previous issues -- status
| # | Issue | Status |
|---|---|---|
| 1 | __dirname undefined in ESM (must fix) |
Fixed -- removed "type": "module" from e2e/package.json |
| 2 | Profile isolation may not propagate (should fix) | Fixed -- added tauri:options.env forwarding + fallback on the tauri-driver process |
| 3 | tauri-driver startup race (should fix) | Fixed -- waitForPort() polls port 4444 with 30s timeout; WDIO gets connectionRetryTimeout + connectionRetryCount |
| 4 | Non-unique hub button testids (should fix) | Fixed -- testids now scoped by model_id (e.g. hub-use-button-qwen3-0.6B) |
| 5 | Hardcoded nightly URL (nice to have) | Not addressed -- acceptable for draft |
| 6 | No pull_request trigger (nice to have) |
Not addressed -- intentional per draft |
| 7 | Missing lockfile (nice to have) | Not addressed -- CI uses --no-immutable |
All four "must fix" / "should fix" items have been addressed with solid implementations.
New observations
- The
get env()getter ontauri:optionsworks because WDIO reads capabilities afterbeforeSession()populatesprofileEnv, but the timing dependency is subtle. A brief inline comment noting the lazy-evaluation reason would help future maintainers. - The
waitForPort()implementation is clean -- TCP connect check with 200ms retry delay and a configurable timeout. - The dual approach for env propagation (both
tauri:options.envand inheriting on tauri-driver's process) is good belt-and-braces engineering.
Remaining items (non-blocking)
- Consider generating or committing a
yarn.lockfore2e/before merging to ensure reproducible CI builds. - The hardcoded
.debURL will become stale -- a follow-up to resolve "latest" dynamically would be valuable. - PR is still in draft with unchecked self-checklist items.
Recommendation: can merge (once moved out of draft). All substantive issues from the first review have been addressed.
Review: E2E tauri-driver setupSummary: Introduces an E2E testing harness using tauri-driver + WebdriverIO. Adds a GitHub Actions workflow, WDIO config with profile isolation, a smoke spec, a llama.cpp inference spec, and Positive observations
Issues and observations
Recommendation: improve needed |
Follow-up Review (commit 0ce151f)Reviewing the new commit What changedThis commit adds two new E2E spec files and the corresponding
Positive observations
Remaining observations (non-blocking)
Recommendation: can merge |
Replaces the agent-driven autoqa flow for deterministic UI smoke tests. Includes a llama.cpp inference spec and stable data-testid hooks on the Hub nav, search input, Use/Download buttons, chat input, and message bubbles. Per-run XDG/APPDATA tempdirs keep tests off the user profile.
Mirrors autoqa-reliability.yml's .deb-from-URL pattern for fast runs against nightly, with an opt-in build mode for PR validation. xvfb + webkit2gtk-driver + cargo-installed tauri-driver on ubuntu-22.04. Default spec: smoke only — inference spec stays opt-in via input.
Push events provide no inputs, so route everything through env vars with fallbacks. On main, build the source we just landed instead of pulling a stale nightly. Path filters keep doc-only pushes from triggering a 25 min build.
- Drop "type": "module" from e2e/package.json so __dirname works in wdio.conf.ts (was crashing under ESM). - Forward profile env (XDG_DATA_HOME / APPDATA) to the spawned Jan binary via tauri:options.env so isolation reaches the app even if tauri-driver doesn't inherit the parent environment. Keep the same overrides on the tauri-driver process as a fallback. - Wait for port 4444 in beforeSession() and add WDIO connection retry config so sessions don't race tauri-driver startup. - Suffix hub-use-button / hub-download-button testids with model_id to scope e2e selectors to the targeted model card. Update the llamacpp inference spec to use the scoped selectors.
Drops the VLM-driven Python E2E runner in favor of the deterministic tauri-driver + WebdriverIO setup under e2e/. Removes the autoqa/ directory, its four GitHub workflows, and stale references in CONTRIBUTING.md and e2e/README.md.
- helpers/app.ts: waitForApp, clickNav, byTestId, existsTestId, dismissDialog - helpers/settings.ts: openSettings(tab), settingsControl, reloadRenderer, SettingsTab union - wdio.conf.ts: pin LANG/LC_ALL/LANGUAGE=en_US.UTF-8 for the spawned Jan process; capture PNG screenshots of failed tests into e2e/screenshots/; correct config type to WebdriverIO.Config - e2e-tauri-driver.yml: upload screenshots/ artifact on failure - Refactor existing specs onto the helpers; fix pre-existing $$().length misuse in llamacpp-inference spec - README: document helpers, conventions (English-only, testid-only, manual-only flows under specs/manual/)
Adds stable data-testid hooks consumed by the upcoming WebdriverIO
settings specs:
- SettingsMenu: settings-tab-<slug> on every core/integration entry
- interface: settings-panel-interface, settings-interface-reset
- shortcuts: settings-panel-shortcuts
- hardware: settings-panel-hardware, plus settings-hardware-{os,cpu,memory}
- https-proxy: settings-panel-https-proxy, settings-https-proxy-{enabled,url}
…https-proxy) Four UI-only specs covering the settings surfaces that were previously covered by the manual checklist alone: - interface: panel renders Theme/Font Size/Accent rows + Reset button - shortcuts: all groups (Application/Chat/Navigation) and key shortcuts surface; relies on the en-locale pin in wdio.conf.ts - hardware: OS/CPU/Memory cards render with non-empty values; memory reports a unit (regression guard against label-only render) - https-proxy: URL field persists across renderer reload via useProxyConfig Also extends openSettings() to drive tanstack-router by URL for tabs not surfaced in the sidebar (https-proxy, extensions), and broadens the SettingsTab union to match the actual route slugs.
Replaces the wrapper-div hack used in hardware.tsx and gives every container a built-in testid hook for upcoming threads/assistants/MCP specs. Behavior unchanged — testId is forwarded as data-testid only when provided.
- assistant.tsx: testids on the panel container, add button, default trigger, per-assistant item, edit/delete buttons - MessageItem.tsx: testid on the regenerate button (used by future chat-message specs under specs/manual/) - New settings-assistants.e2e.ts: panel renders default Jan, Add dialog opens/closes, default-fallback dropdown lists assistants, edit dialog opens for existing assistants - README: document the checklist features that no longer exist in current code (thread starring, favourites, sidebar search) and point model-dependent flows at the planned specs/manual/ folder
Introduces a profile-fixtures harness so specs can exercise thread and
assistant flows without needing a running model:
- helpers/profileState.ts: shared per-worker profile state, exposes
getJanDataFolder() that mirrors src-tauri's resolve_jan_data_folder
- helpers/profile.ts: seedThread / seedAssistant write directly to
<profile>/Jan/data/{threads,assistants}/<id>/... using the Rust
on-disk layout; reloadAfterSeed() refreshes the renderer so stores
re-hydrate from disk
- wdio.conf.ts: publishes profileDir into profileState on session
start, clears it on teardown
Adds testids on ThreadList items and the rename/delete dialogs, then
threads-crud.e2e.ts seeds two threads, renames one, deletes the other,
and asserts sidebar state. Create-via-UI still requires a model and
remains a specs/manual/ candidate.
- AddProviderDialog: testids on the name input and create confirm - DeleteProvider: testids on the delete trigger and confirm - SettingsMenu: testid on the +Add provider button and on each provider sidebar row (provider-sidebar-<name>) - providers/$providerName.tsx: testids on the panel container (provider-page-<name>) and the active toggle (provider-active-<name>) - New provider-crud.e2e.ts: drives the sidebar +button to create a uniquely-named custom provider, asserts the new sidebar entry and settings page mount, then deletes via the per-provider delete confirmation. UI-only, no real API key or remote traffic. Predefined providers (OpenAI, Anthropic, ...) cannot be deleted by design (DeleteProvider returns null for them); the spec uses a timestamp-suffixed custom provider name to stay hermetic.
Three UI-only specs covering the remaining settings surfaces that don't need a running model or external service: - mcp-servers.tsx: testid on the panel container and Add Server header button - local-api-server.tsx: testid on the panel and start/stop toggle - general.tsx: testid on the factory-reset trigger settings-misc.e2e.ts asserts each panel mounts on a fresh profile, the Add MCP Server dialog opens (Escape-dismissed, no persistence), and the factory-reset confirmation dialog opens — explicitly Escape-dismissed without confirming, since a real reset would wipe the profile and break sibling tests. MCP CRUD with env vars / streamable-http, /v1/models verification on the Local API Server, and the Hub model-card render all require external state and remain specs/manual/ candidates.
- ThemeSwitcher: testids on the dropdown trigger and each option; expose the live value via data-theme-active for locale-independent assertions - FontSizeSwitcher: same — data-font-size-active on the trigger, testid per option - New settings-interface-persistence.e2e.ts: pick a non-default theme (light) and font size (18px), reload the renderer, reopen the Interface panel, assert both values re-hydrated from the Zustand persist store backed by WebKit localStorage in the pinned profile
qnixsynapse
force-pushed
the
e2e/tauri-driver-setup
branch
from
eb06924 to
5cd7d7a
Compare
|
Follow-up review (new activity detected) Since our last review (2026-05-05), there has been one new commit: New commit
AssessmentThis is a draft PR and the new commit continues in the right direction -- expanding E2E coverage beyond the initial smoke test. The previous concerns from our review still apply:
Since this is still a draft PR, the work-in-progress nature is expected. The E2E framework architecture remains sound. Recommendation: improve needed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe Your Changes
Further spec TBD.
Fixes Issues
Self Checklist