Copilot Annotations / Comments #8641
Description
Description
This feature allows to add annotations / comments to the UI using Copilot, in order to provide feedback – much like adding a comment in Google Docs. The developer can then use these comments to improve the UI – or give the comments to an AI to implement.
Tier
Gated
License
Proprietary
Motivation
Background
Iterating on UI often requires implementing->trying->adjusting = "sanding". Sometimes the "feedback" on the UI is your own, sometimes it might come from a stakeholder – and increasingly, the recipient is an AI.
Problem
If you are working on your own, "sanding" either means going back and forth between testing and fixing, OR taking notes, then implementing changes in one go before testing again.
If you use an AI coding agent, you either tell the AI to implement a fix as you find it, wait, test some more, OR you take notes and ask the AI to implement everything before testing again.
Solution
Ability to add any number of comments to any view in an application, optionally indicating 0-N attachments (component selections) that are related to the comment.
Comments are stored in a file in the project (for now).
Attachments (selections) are marked in a way that aims to maximize the chance that an AI can understand what code to change, and to minimize the chance that they go "stale" when minor changes are made to the UI. This is one of the main challenges, and might include create- and attach-locations, hierarchy path, component id, debug id – needs some testing to find a suitable solution for now.
Notes
A working prototype exists.
Big picture / future
This is the first step in something that can grow in the future; we expect AI assisted workflows to go asynchronous, with developers/stakeholders providing feedback and an agent reacting to the feedback.
Even without AI, easily soliciting feedback from the team and stakeholders for work in progress is quite valuable.
This is a first small step towards that future.
Requirements
-
Ability to add/remove/update comments on a view
-
Comments are persisted in a human- and AI-readable file in the project
-
Ability attach selections (component/element references) to comments
-
Comment attachments are referenced in a durable way, with the intent to maximize the chance humans and AIs find the relevant code.
-
Documentation
-
License check (login required)
-
Feature flag
Nice-to-haves
- Ask built-in AI to address comments
- Mark comment as done
Risks, limitations and breaking changes
Risks
- Comment attachments getting "stale", i.e. UI changes making the component references invalid, is perhaps the main challenge and risks limiting the usefulness. AIs should be able to take the comments from the file and implement, but there is a risk this does not work as well as we hope.
- File format not able to communicate well enough what needs to be changed.
Limitations
Breaking changes
Out of scope
A screenshot would greatly help both AI and humans to understand the intention in some cases, especially if the UI has drifted from how it looked when the comment was added. This is out-of-scope for now, as it's unclear how to best acquire a screenshot.
It would be nice to be able to draw on the UI or otherwise add annotations that are not tied to component selections, e.g point to an empty area. This is out-of-scope for now, as it requires screenshot or some other way to communicate non-structured annotations.
We will revisit the mechanism for storing / sharing comments in the future, for now a file-based approach is fine.
Materials
No response
Metrics
No response
Pre-implementation checklist
- Estimated (estimate entered into Estimate custom field)
- Product Manager sign-off
- Engineering Manager sign-off
Pre-release checklist
- Documented (link to documentation provided in sub-issue or comment)
- UX/DX tests conducted and blockers addressed
- Approved for release by Product Manager
Security review
None