Skip to content

feat(platform): add hosted iOS and Android chat apps - #14429

Draft
ntindle wants to merge 6 commits into
devfrom
codex/mobile-friendly-autogpt
Draft

feat(platform): add hosted iOS and Android chat apps#14429
ntindle wants to merge 6 commits into
devfrom
codex/mobile-friendly-autogpt

Conversation

@ntindle

@ntindle ntindle commented Sep 7, 2026

Copy link
Copy Markdown
Member

Why / What / How

AutoGPT needs iOS and Android access to its hosted chat without maintaining a separate chat client. This draft adds UIKit/WKWebView and Kotlin/WebView apps that load the selected AutoGPT deployment. Conversations, streaming, agents, tools, history, and account screens remain on the website, so web improvements reach both apps directly.

The native shells handle navigation, browser sign-in, persistent sessions, safe areas and keyboards, file selection/export, microphone permission, and recovery. Browser sign-in uses a single-use, proof-bound code and a fresh HttpOnly app session; it reuses Better Auth's verification table without a migration. Android generated files use one bounded, origin-scoped write-only message interface; iOS uses WebKit downloads.

This remains a development draft. The mobile auth endpoints must be deployed on a chosen server before native sign-in works there. Production deployment, release signing, and store publication are not configured. General responsive-web improvements remain in the separate web workstream. Provider-connection OAuth is a known browser/WebView gap being addressed in a follow-up commit.

Changes 🏗️

  • Add native iOS and Android projects, pinned build tooling, native policy tests, and mobile CI.
  • Add explicit server selection, exact-origin navigation, secure browser return, identity replacement, bounded file export, and native picker/microphone handling.
  • Add shared web mobile-auth consent/endpoints and a backward-compatible saveBlob helper for existing chat exports.
  • Add a labelled local integration fixture and a disposable PostgreSQL harness for concurrent redemption and session-policy checks.
  • Document build steps, authentication/download contracts, platform limits, and progress screenshots.

Verified locally: full frontend formatting, lint and types; 6,560 tests across 616 files; 131 affected tests after the later download-timeout adjustment; 8 Swift policy tests; iOS simulator Debug and unsigned iPhone Release builds; 27 Android unit tests, lint and debug/release builds; 9 native-fixture tests; and real PostgreSQL four-process redemption (one success, three rejections) with revocation rollback.

The iPhone 16 Pro / iOS 18.3 simulator completed fixture browser sign-in with both cookie families, generated-file export to Files and attachment re-selection, external navigation, and error recovery. Android emulator checks are in progress. Exact iPhone 17 Pro / Pixel 11 Pro hardware, real provider sign-in, and live model streaming remain unverified.

Agents and large language models used

Codex with GPT-6, including parallel Codex subagents.

Checklist 📋

For code changes:

  • I have clearly listed my changes in the PR description
  • I have made a test plan
  • I have tested my changes according to the test plan:
    • Initial documentation formatting and repository commit checks.
    • Build and launch iOS; upload simulator screenshots.
    • Build Android and run native policy tests; record runtime coverage separately.
    • Test authentication handoff, replay rejection, navigation boundaries, persistence, attachments, keyboard, and recovery.
Example test plan
  • Create from scratch and execute an agent with at least 3 blocks
  • Import an agent from file upload, and confirm it executes correctly
  • Upload agent to marketplace
  • Import agent from marketplace and confirm it executes correctly
  • Edit an agent from monitor, and confirm it executes correctly

For configuration changes:

  • .env.default is updated or already compatible with my changes
  • docker-compose.yml is updated or already compatible with my changes
  • I have included a list of my configuration changes in the PR description (under Changes)
Examples of configuration changes
  • Changing ports
  • Adding new services that need to communicate with each other
  • Secrets or environment variable changes
  • New or infrastructure changes such as databases

Co-authored-by: GPT-6 (Codex) <agent@example.invalid>
@coderabbitai

coderabbitai Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added cla: pending CLA not yet signed by all contributors size/m labels Sep 7, 2026
@codecov

codecov Bot commented Sep 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.39456% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.34%. Comparing base (4109de4) to head (dc4b0dd).
⚠️ Report is 4 commits behind head on dev.

Additional details and impacted files
@@            Coverage Diff             @@
##              dev   #14429      +/-   ##
==========================================
- Coverage   81.35%   81.34%   -0.01%     
==========================================
  Files        3517     3526       +9     
  Lines      263467   263652     +185     
  Branches    24426    24463      +37     
==========================================
+ Hits       214336   214466     +130     
- Misses      43709    43751      +42     
- Partials     5422     5435      +13     
Flag Coverage Δ
platform-frontend 60.52% <86.94%> (+0.11%) ⬆️
platform-frontend-e2e 28.50% <2.46%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Platform Backend 86.35% <ø> (ø)
Platform Frontend 62.81% <86.39%> (+0.03%) ⬆️
AutoGPT Libs ∅ <ø> (∅)
Classic AutoGPT 28.43% <ø> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Co-authored-by: GPT-6 (Codex) <agent@example.invalid>
@ntindle

ntindle commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

First native milestone: the iOS shell builds with the iOS 26.5 simulator SDK and runs on this machine's installed iPhone 16 Pro / iOS 18.3 simulator. The images below are unmodified simulator captures.

The first screen is reached from the real hosted chat's login redirect:

Native AutoGPT sign-in screen on iOS

The second shows the local integration fixture after a successful system-browser PKCE return, with both HttpOnly token and cache cookies received. It is explicitly a fixture, not a live account or chat:

Local native integration fixture showing successful cookie transfer

Committed: iOS shell, origin/PKCE tests, fixture harness, reproducible build instructions, and screenshots. Swift tests (5), fixture tests (6), Swift lint, iOS build, and repository commit hooks pass. The Android implementation and server handoff remain in progress. The new server endpoints must be deployed before native sign-in can work against the hosted production site; no production deployment has been performed.

@github-actions github-actions Bot added cla: signed CLA signed by all contributors size/xl and removed cla: pending CLA not yet signed by all contributors size/m labels Sep 7, 2026
…nd files

Co-authored-by: GPT-6 (Codex) <agent@example.invalid>
@github-actions github-actions Bot added the platform/frontend AutoGPT Platform - Front end label Sep 7, 2026
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

🔍 PR Overlap Detection

This check compares your PR against all other open PRs targeting the same branch to detect potential merge conflicts early.

🔴 Merge Conflicts Detected

The following PRs have been tested and will have merge conflicts if merged after this PR. Consider coordinating with the authors.

🟢 Low Risk — File Overlap Only

These PRs touch the same files but different sections (click to expand)

Summary: 1 conflict(s), 0 medium risk, 3 low risk (out of 4 PRs with file overlap)


Auto-generated on push. Ignores: openapi.json, lock files.

Co-authored-by: GPT-6 (Codex) <agent@example.invalid>
@ntindle

ntindle commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

The second milestone is pushed: both native shells, browser-to-app sign-in endpoints, shared web export support, native policy tests, and mobile CI. Android's CI job passes; the initial iOS CI failure was a missing formatter executable on PATH, corrected to xcrun swift-format in 7ab74aaa85.

These are unmodified captures from iPhone 16 Pro / iOS 18.3, using the explicitly labelled local fixture. They verify native transport/input behavior, not a real account or model response.

The fixture receives five chunks incrementally (about 600 ms apart), and retains partial text with a retry option when the server deliberately drops the stream:

iOS fixture with five incrementally received stream chunks

iOS fixture retaining partial text after a deliberate stream interruption

Portrait text entry stays visible above the onscreen keyboard:

iOS fixture text field visible above the onscreen keyboard

A generated 69-byte Markdown file also completed the native share sheet → Files → web attachment picker roundtrip. Android emulator checks and iOS landscape keyboard improvements are underway. The new authentication endpoints are not deployed to production; provider-connection OAuth remains a follow-up integration.

ntindle and others added 2 commits September 7, 2026 17:58
Co-authored-by: GPT-6 (Codex) <agent@example.invalid>
…est builds

Co-authored-by: GPT-6 (Codex) <agent@example.invalid>
@ntindle

ntindle commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

Android is now running on a temporary API 36 Google APIs arm64 emulator with a Pixel 9 Pro profile and WebView 133. The screenshot shows the native sign-in screen reached from the real hosted chat's login redirect. This is separate from the planned Pixel 11 Pro physical-device target.

AutoGPT Android native sign-in screen on the API 36 emulator

Four runtime checks now pass against actual Android WebView APIs: required capabilities; cookie deletion callback ordering; the local fixture's native authentication exchange and complete cookie replacement, including cancellation preservation; and native download-message confinement to the trusted top-level page. These use disposable test data. Browser and real picker interaction remain unverified. Raw evidence and APK checksum.

On iOS, the generated-file → Files → attachment-picker roundtrip retains the filename and all 69 bytes:

iOS local fixture showing a previously exported Markdown file selected again as an attachment

The native iOS rotation fix preserves the draft, dismisses the keyboard on rotation, and allows visible refocus in landscape. Ten Swift policy tests, 27 Android JVM tests, full native formatting/lint, simulator/device iOS builds, and Android debug/release builds pass locally. The new CI revision also runs iOS UI tests on an installed matching simulator and publishes seven-day Android debug APK/iOS simulator build artifacts. Its results are still pending.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla: signed CLA signed by all contributors platform/frontend AutoGPT Platform - Front end size/xl

Projects

Status: 🆕 Needs initial review
Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant