Skip to content

[WIP] NTP image generation#4447

Draft
sashalavron wants to merge 19 commits intomainfrom
alavrenchuk/ntp-image-generation
Draft

[WIP] NTP image generation#4447
sashalavron wants to merge 19 commits intomainfrom
alavrenchuk/ntp-image-generation

Conversation

@sashalavron
Copy link
Copy Markdown
Collaborator

@sashalavron sashalavron commented Apr 13, 2026

Task/Issue URL: https://app.asana.com/1/137249556945/project/1213083312441631/task/1213493651880757?focus=true
Tech Design URL: https://app.asana.com/1/137249556945/project/1213083312441631/task/1213920561950665?focus=true

Description

Add image generation mode and web search tool support to the NTP, allowing users to toggle these modes when submitting AI chat prompts.

Changes

  • Added enableImageGeneration and enableWebSearch config flags to the NTP omnibar, gated behind new privacy config subfeatures and feature flags
  • Extended submitChat to accept mode and toolChoice parameters, forwarded to Duck.ai via the native prompt handoff
  • Added pixel tracking for image generation and web search submissions on the NTP

Testing Steps

  1. Open a new tab, switch omnibar to AI chat mode — verify "Create Image" and web search toggles are visible
  2. Submit a prompt with image generation on — confirm Duck.ai opens with image generation mode, model selector hidden
  3. Submit with web search on — confirm web search tool choice is forwarded
  4. Verify toggles reset after each submission
  5. Disable feature flags (aiChatNtpImageGeneration, aiChatNtpWebSearch) — confirm toggles disappear and behavior is unchanged

Impact and Risks

What could go wrong?

Quality Considerations

Notes to Reviewer


Internal references:

Definition of Done | Engineering Expectations | Tech Design Template

Comment thread PLAN.md Outdated
@@ -0,0 +1,100 @@
# Tech Design: NTP Omnibar — Image Generation Mode
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to remove

.package(url: "https://github.com/vapor/jwt-kit.git", exact: "4.13.5"),
.package(url: "https://github.com/pointfreeco/swift-clocks.git", exact: "1.0.6"),
.package(url: "https://github.com/duckduckgo/content-scope-scripts.git", exact: "13.41.0"),
.package(url: "https://github.com/duckduckgo/content-scope-scripts.git", branch: "pr-releases/alavrenchuk/ntp-image-generation"),
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to change

- Updated `Package.resolved` to reflect new dependencies for `content-scope-scripts` version 13.37.0.
- Added support for image generation and web search tools in AI chat submissions.
- Modified `AIChatNativePrompt` to include a new `mode` parameter.
- Updated `NewTabPageOmnibarActionsHandler` to handle new parameters in chat submissions.
- Introduced feature flags for image generation and web search in the New Tab Page.
- Updated localization strings and versioning for the macOS app.

This commit enhances the AI chat functionality and ensures compatibility with the latest dependencies.
- Changed the dependency for `content-scope-scripts` to track the `alavrenchuk/ntp-image-generation` branch.
- Removed debug code for local CSS paths in `DuckURLSchemeHandler.swift`.
- Updated feature flag configurations for `aiChatNtpImageGeneration` and `aiChatNtpWebSearch` to include default values.

These changes improve the integration of new features and streamline the codebase.
- Removed the `DebugServer` file reference from the project workspace.
- Updated `Package.resolved` to reflect changes in the `content-scope-scripts` dependency, now tracking the `alavrenchuk/ntp-image-generation` branch.
- Cleaned up localization strings by removing the `REBRANDED` entry.

These changes streamline the project structure and ensure compatibility with the latest dependencies.
- Changed the dependency for `content-scope-scripts` to track the `pr-releases/alavrenchuk/ntp-image-generation` branch, ensuring alignment with the latest development efforts.

This update maintains compatibility with ongoing feature enhancements and dependency management.
…cy revision

- Updated the `originHash` and the `branch` for the `content-scope-scripts` dependency in `Package.resolved` to ensure alignment with the latest changes in the repository.

This change maintains compatibility with ongoing development efforts.
- Added new pixel events for user interactions in image generation and web search modes on the New Tab Page.
- Updated the `AIChatPixel` enum to include `aiChatNtpImageGenerationSubmitted` and `aiChatNtpWebSearchSubmitted`.
- Modified `NewTabPageOmnibarActionsHandler` to fire the new pixel events based on user actions.
- Updated pixel definitions in `aichat_ntp_pixels.json5` to include descriptions and parameters for the new events.

These changes improve tracking of user interactions and enhance analytics capabilities for AI chat features.
…esign. This file included background, requirements, problem statement, recommended approach, schema changes, and testing scenarios for the feature. Its removal indicates a shift in project documentation or implementation strategy.
- Updated the `submitChat` method in `NewTabPageOmnibarActionsHandler` and related protocols to accept a new `modelId` parameter.
- Adjusted all relevant calls to `submitChat` across the codebase to accommodate the new parameter.
- Modified tests to ensure proper handling of the `modelId` in chat submissions.

These changes improve the flexibility of chat submissions and align with recent feature enhancements.
- Added a reference to the `DebugServer` group in the project workspace.
- Introduced a new static property `imageGenerationMode` in `AIChatNativePrompt` for better code clarity.
- Updated the `effectiveMode` property in `AIChatOmnibarController` to use the new `imageGenerationMode`.
- Enhanced the `NewTabPageOmnibarActionsHandler` to utilize the new mode constant for image generation checks.
- Adjusted the version in `Localizable.xcstrings` from "1.1" to "1.0".
- Added a new test case to verify the forwarding of `modelId` in chat submissions.

These changes improve code maintainability and enhance the AI chat feature set.
@sashalavron sashalavron force-pushed the alavrenchuk/ntp-image-generation branch from fc2247e to 4d268b0 Compare April 14, 2026 14:32
- Simplified the target determination logic in the `submitSearch` method by directly using the result of `isCommandPressed()` in the `submitChat` call.
- This change enhances code readability and maintains the existing functionality for chat submissions.

These modifications contribute to cleaner code and improved maintainability.
…tion Structure

- Removed unnecessary reset calls in the `NewTabPageOmnibarActionsHandlerTests` to streamline test setup and teardown.
- Updated the `SubmitChatAction` structure in `NewTabPageDataModel+Omnibar` to ensure `mode` and `toolChoice` are correctly defined and utilized.
- Adjusted test cases in `NewTabPageOmnibarClientTests` to reflect changes in the `SubmitChatAction` structure, ensuring proper handling of `modelId` and image parameters.

These changes improve test clarity and maintainability while aligning with recent enhancements in chat submission functionality.
- Updated the `submitChat` method across multiple files to change the order of parameters, placing `target` as the first argument for better readability.
- Adjusted all relevant calls to `submitChat` in the `NewTabPageOmnibarActionsHandler`, tests, and mocks to reflect the new parameter order.
- This refactor enhances code clarity and maintains the existing functionality for chat submissions.

These changes contribute to a more intuitive method signature and improve maintainability across the codebase.
- Updated the `submitChat` method calls in `NewTabPageOmnibarActionsHandler` and related tests to remove unnecessary parameters, enhancing clarity and consistency.
- Simplified test cases in `NewTabPageOmnibarActionsHandlerTests` and `NewTabPageOmnibarClientTests` to reflect the updated method signature, ensuring proper functionality without redundant arguments.
- Removed outdated test for `modelId` forwarding, aligning tests with the current implementation.

These changes improve code readability and maintainability while ensuring the functionality of chat submissions remains intact.
- Removed the custom initializer from the `AIChatNativePrompt` struct to simplify the code and rely on the default memberwise initializer.
- This change enhances code clarity and maintainability by reducing boilerplate code.

These modifications contribute to a cleaner implementation of the AIChat functionality.
…ction structure

- Modified the `NewTabPageOmnibarClientTests` to assert the `mode` parameter as "image-generation" instead of nil, aligning with the updated `SubmitChatAction` structure.
- Adjusted the instantiation of `SubmitChatAction` to include the `mode` parameter, ensuring consistency with recent refactors.

These changes enhance test accuracy and maintain alignment with the current implementation of chat submission functionality.
…ipts

- Updated the revision for the `content-scope-scripts` package in `Package.resolved` to the latest commit, ensuring compatibility with recent changes in the repository.

This change maintains the integrity of package dependencies and aligns with ongoing development efforts.
- Enhanced the `NewTabPageDataModel` to include a new property `supportedTools`, allowing for better representation of available tools.
- Updated the `NewTabPageOmnibarModelsProvider` to map the new `supportedTools` property correctly.
- Modified related tests in `NewTabPageOmnibarModelsProviderTests` to validate the inclusion of `supportedTools`, ensuring accurate model fetching and mapping.

These changes improve the data model's flexibility and enhance the testing framework's coverage for new features.
… GitHub URL

- Changed the dependency for `content-scope-scripts` from a local path to a specific version hosted on GitHub, ensuring consistent access to the package across different environments.

This update enhances dependency management and aligns with best practices for package sourcing.
…cy and update existing package revision

- Added the `content-scope-scripts` package with its latest version and revision to `Package.resolved`, ensuring consistent access to this dependency.
- Updated the revision for the existing packages to maintain compatibility with recent changes.

This update enhances dependency management and aligns with ongoing development efforts.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant