From f56c1930c5636632b56f0dac803e21a203a726fe Mon Sep 17 00:00:00 2001 From: Luke Longley <18177025+llongley@users.noreply.github.com> Date: Thu, 20 Mar 2025 19:51:08 -0700 Subject: [PATCH 1/2] Adding an API spec for UserActivityHistory. --- .../UserActivityHistoryAPI.md | 243 ++++++++++++++++++ 1 file changed, 243 insertions(+) create mode 100644 specs/UserActivityHistory/UserActivityHistoryAPI.md diff --git a/specs/UserActivityHistory/UserActivityHistoryAPI.md b/specs/UserActivityHistory/UserActivityHistoryAPI.md new file mode 100644 index 0000000000..3bbefd9fa8 --- /dev/null +++ b/specs/UserActivityHistory/UserActivityHistoryAPI.md @@ -0,0 +1,243 @@ +UserActivityHistory +=== + +# Background + +The UserActivity class can be used to note down and preserve a record of activities that the user +is currently doing on their computer - e.g., browsing a website, reading a Word document, etc. + +To record user activity, you use UserActivityChannel to retrieve a UserActivity object via the +API GetOrCreateUserActivityAsync. If a UserActivity with the given ID already exists, it will be +returned; otherwise, a new UserActivity object will be created and returned. You can then call +the API GetSession to return a UserActivitySession object that tracks how long the user is engaged +in that activity. This structure allows multiple sessions to be associated with the same activity, +representing the case where the user completes that activity a bit at a time - e.g., beginning to +watch a movie, then pausing, then watching more later. These will be treated as the same singular +user activity that spans multiple sessions. + +UserActivityHistory is a new set of APIs that allow you to query the past 28 days of the user's +activity history, which will enable you to bring back content that the user has previously been +interacting with. + +# API Pages + +## UserActivityHistory class + +This class provides static methods that enable you to query the user's activity history. +This activity history is stored in a database managed by a local service, and these APIs +call out to that service to retrieve data from the database. + +Here is an example usage of the class that will enable you to bring back the webpage for a +Korean recipe that the user had previously interacted with within the last day: + +```c# +UserActivityHistoryQuery query = new(); +query.Keywords = new string[] { "Korean", "recipe" }; +query.LatestStartTime = DateTime.Now.AddDays(-1); + +IList results = await UserActivityHistory.SearchAsync( + new UserActivityHistoryQuery[] { query }, + UserActivityHistoryOrderBy.DwellTime, + maxResults: 1); + +UserActivityHistoryItem item = results.FirstOrDefault(); + +if (item != null) +{ + // Now we can use item.ActivationUri to bring back the webpage in the state in which the user + // was last viewing it. +} +``` + +## UserActivityHistory.Search method + +This method synchronously queries the user's activity history and returns a list of items matching +the criteria specified in the `queries` parameter. The results are ordered in descending order +according by the `orderBy` parameter: either by the most recent start times, the most recent +end times, or the longest time spent on the activity. + +Each parameter in `queries` is ORed together in the resulting database query, whereas the contents +of a single object in the `queries` array are ANDed together. For example, if you provided two +queries, each of which contained the keywords "tax", one of which had a content type of +"application/pdf" and the other had a content type of "image/*", the resulting database query +would be something along these lines: + +```sql +SELECT * FROM UserActivityHistory WHERE + (CONTAINS (DisplayText, '"tax"')) AND + ((CONTAINS (ContentType, '"application/pdf"')) OR + (CONTAINS (ContentType, '"image/*"'))); +``` + +## UserActivityHistory.SearchAsync method + +This method is an asynchronous version of the `Search` method. + +## UserActivityHistory.GetAppsWithUserActivity method + +This method synchronously retrieves a list of all the app names with data in the user's activity +history database. You can use this, for example, to show the user the list of apps that are being +queried against, so the user can understand why an app that is not recording user activity is not +showing up in the results. + +## UserActivityHistory.GetAppsWithUserActivityAsync method + +This method is an asynchronous version of the `GetAppsWithUserActivity` method. + +## UserActivityHistoryItem class + +This class represents a single item in the user's activity history. It contains properties that +describe in what app the activity occurred, what the nature of the activity was, the URI of the +resource involved in the activity (e.g., a document, a webpage, a video, etc.), the URI that +can be used to bring back the state the user left the activity in, and the times when the user +started the activity and ended the activity. + +If the user performed the same activity multiple times, there will be multiple +`UserActivityHistoryItem` objects returned, each with different start and end times. + +## UserActivityHistoryItem.AppName property + +This property contains the name of the app in which the activity occurred. + +## UserActivityHistoryItem.ActivityId property + +This property contains the ID of the activity, which can be used to collate multiple sessions +of the same activity. For example, if the user watched a movie in multiple sessions, the +`ActivityId` property can be used to identify how long in total the user spent watching that movie. + +## UserActivityHistoryItem.DisplayText property + +This property contains a string that is how the app chose to describe the activity. For example, +if the activity was reading the contents of a webpage, this property might contain the webpage's title. + +## UserActivityHistoryItem.ContentType property + +This property contains the MIME type of the content being interacted with. For example, if the user +was looking at a PNG image, this property would contain the string "image/png". + +## UserActivityHistoryItem.ContentUri property + +This property contains the URI of the content being interacted with. For example, if the user was +looking at a webpage, this property would contain the URI of that webpage. + +## UserActivityHistoryItem.ActivationUri property + +This property contains the URI that can be used to bring back the state the user left the activity in. +For example, if the user was looking at a webpage, this property would contain the URI of that webpage +with additional information such as what page the user was on, what their scroll position was, etc. + +## UserActivityHistoryItem.StartTime property + +This property contains the time when the user started the activity session. + +## UserActivityHistoryItem.EndTime property + +This property contains the time when the user ended the activity session. + +## UserActivityHistoryQuery class + +This class is used to specify criteria for what portion of the user's activity history you want to +retrieve. It allows you to specify keywords to search for, content types to filter by, and time ranges +to filter by. + +## UserActivityHistoryQuery.Keywords property + +This property is an array of keywords, each of which is used to lexically search against the +DisplayText column in the database. Keywords are case-insensitive, and results returned will +be those that contain all of the keywords in the array. + +## UserActivityHistoryQuery.ContentType property + +This property is a string that specifies the content type associated with the activity you want +to retrieve. It allows the inclusion of an asterisk as a wildcard - e.g., "image/*" will match +all content types beginning with "image/", such as "image/png", "image/jpeg", etc. +This property is case-insensitive. + +## UserActivityHistoryQuery.EarliestStartTime property + +This is a nullable DateTime property that specifies the earliest start time of the activity +you want to retrieve. Any activities with a StartTime property earlier than this will be excluded. +If this property is left as null, it will be ignored. + +## UserActivityHistoryQuery.EarliestEndTime property + +This is a nullable DateTime property that specifies the earliest end time of the activity +you want to retrieve. Any activities with an EndTime property earlier than this will be excluded. +If this property is left as null, it will be ignored. + +## UserActivityHistoryQuery.LatestStartTime property + +This is a nullable DateTime property that specifies the latest start time of the activity +you want to retrieve. Any activities with a StartTime property later than this will be excluded. +If this property is left as null, it will be ignored. + +## UserActivityHistoryQuery.LatestEndTime property + +This is a nullable DateTime property that specifies the latest end time of the activity +you want to retrieve. Any activities with an EndTime property later than this will be excluded. +If this property is left as null, it will be ignored. + +## UserActivityHistoryOrderBy enum + +This enum specifies what property the results should be ordered by. The options are as follows: + +| Name | Description | +|-|-| +| StartTime | Results will be in descending order of their StartTime property | +| EndTime | Results will be in descending order of their EndTime property | +| DwellTime | Results will be in descending order of the difference between their EndTime and StartTime properties | + +# API Details + +```c# (but really MIDL3) +namespace Microsoft.Windows.ApplicationModel.UserActivities +{ + runtimeclass UserActivityHistory + { + static IVector Search( + UserActivityHistoryQuery[] queries, + UserActivityHistoryOrderBy orderBy, + UInt32 maxResults); + + static IAsyncOperation > SearchAsync( + UserActivityHistoryQuery[] queries, + UserActivityHistoryOrderBy orderBy, + UInt32 maxResults); + + static IVector GetAppsWithUserActivity(); + + static IAsyncOperation> GetAppsWithUserActivityAsync(); + } + + runtimeclass UserActivityHistoryItem + { + String AppName { get; }; + String ActivityId { get; }; + String DisplayText { get; }; + String ContentType { get; }; + String ContentUri { get; }; + String ActivationUri { get; }; + DateTime StartTime { get; }; + DateTime EndTime { get; }; + } + + runtimeclass UserActivityHistoryQuery + { + UserActivityHistoryQuery(); + + String[] Keywords; + String ContentType; + IReference EarliestStartTime; + IReference EarliestEndTime; + IReference LatestStartTime; + IReference LatestEndTime; + } + + enum UserActivityHistoryOrderBy + { + StartTime, + EndTime, + DwellTime + }; +} +``` \ No newline at end of file From 643f0d97eac267fc149ceaca674b5f9b295b5168 Mon Sep 17 00:00:00 2001 From: Luke Longley <18177025+llongley@users.noreply.github.com> Date: Tue, 25 Mar 2025 17:33:44 -0700 Subject: [PATCH 2/2] Updating per feedback. --- .../UserActivityHistoryAPI.md | 115 +++++++++++------- 1 file changed, 69 insertions(+), 46 deletions(-) diff --git a/specs/UserActivityHistory/UserActivityHistoryAPI.md b/specs/UserActivityHistory/UserActivityHistoryAPI.md index 3bbefd9fa8..35f640f0ea 100644 --- a/specs/UserActivityHistory/UserActivityHistoryAPI.md +++ b/specs/UserActivityHistory/UserActivityHistoryAPI.md @@ -3,21 +3,43 @@ UserActivityHistory # Background -The UserActivity class can be used to note down and preserve a record of activities that the user +The [UserActivity](https://learn.microsoft.com/uwp/api/windows.applicationmodel.useractivities.useractivity) +class can be used to note down and preserve a record of activities that the user is currently doing on their computer - e.g., browsing a website, reading a Word document, etc. - -To record user activity, you use UserActivityChannel to retrieve a UserActivity object via the -API GetOrCreateUserActivityAsync. If a UserActivity with the given ID already exists, it will be -returned; otherwise, a new UserActivity object will be created and returned. You can then call -the API GetSession to return a UserActivitySession object that tracks how long the user is engaged -in that activity. This structure allows multiple sessions to be associated with the same activity, -representing the case where the user completes that activity a bit at a time - e.g., beginning to -watch a movie, then pausing, then watching more later. These will be treated as the same singular -user activity that spans multiple sessions. - -UserActivityHistory is a new set of APIs that allow you to query the past 28 days of the user's -activity history, which will enable you to bring back content that the user has previously been -interacting with. +This allows Windows to have insight into the application state, enabling smart experiences that are +built around the semantics of the app. For example, a document editor can give Windows information +about the document that the user is editing, so that Recall can later take the user to the document +at the same location. + +To record user activity, an app uses [UserActivityChannel](https://learn.microsoft.com/en-us/uwp/api/windows.applicationmodel.useractivities.useractivitychannel?view=winrt-26100) +to retrieve a UserActivity object via the API [GetOrCreateUserActivityAsync](https://learn.microsoft.com/en-us/uwp/api/windows.applicationmodel.useractivities.useractivitychannel.getorcreateuseractivityasync?view=winrt-26100#windows-applicationmodel-useractivities-useractivitychannel-getorcreateuseractivityasync(system-string)). +If a UserActivity with the given ID already exists, it will be returned; otherwise, a new UserActivity +object will be created and returned. You can then call the API [GetSession](https://learn.microsoft.com/en-us/uwp/api/windows.applicationmodel.useractivities.useractivity.createsession?view=winrt-26100#windows-applicationmodel-useractivities-useractivity-createsession) +to return a [UserActivitySession](https://learn.microsoft.com/en-us/uwp/api/windows.applicationmodel.useractivities.useractivitysession?view=winrt-26100) +object that tracks how long the user is engaged in that activity. This structure allows multiple +sessions to be associated with the same activity, representing the case where the user completes +that activity a bit at a time - e.g., beginning to watch a movie, then pausing, then watching more later. +These will be treated as the same singular user activity that spans multiple sessions. + +UserActivityHistory is a new set of APIs that allow you to query up to the past 28 days of the +user's activity history, which will enable you to bring back content that the user has previously +been interacting with. + +# Conceptual pages (How To) + +The intended use case of this API is to allow you to make search queries against the user's activity history +on their local computer. The string matching in the query parameters in this API is lexical in nature, +meaning that it is expected that any natural language semantic parsing of the user's input will be done by +your app prior to calling this API. + +For example, if a user types in something along the lines of, "Please find the Korean recipe I was looking at +earlier today", your app might have an agentic AI parse that input and determine that the user is looking +for a webpage that contains the keywords "Korean" and "recipe", and construct a query with those keywords, +a content type of "text/html", and an access time within the last 24 hours. + +In order for an app to make use of this API, it must be Windows logo certified, and the user must provide +their consent to allow access to their activity history. If either of these is not the case, the API will +throw an exception. # API Pages @@ -35,7 +57,7 @@ UserActivityHistoryQuery query = new(); query.Keywords = new string[] { "Korean", "recipe" }; query.LatestStartTime = DateTime.Now.AddDays(-1); -IList results = await UserActivityHistory.SearchAsync( +IList results = UserActivityHistory.Search( new UserActivityHistoryQuery[] { query }, UserActivityHistoryOrderBy.DwellTime, maxResults: 1); @@ -56,22 +78,21 @@ the criteria specified in the `queries` parameter. The results are ordered in de according by the `orderBy` parameter: either by the most recent start times, the most recent end times, or the longest time spent on the activity. -Each parameter in `queries` is ORed together in the resulting database query, whereas the contents -of a single object in the `queries` array are ANDed together. For example, if you provided two -queries, each of which contained the keywords "tax", one of which had a content type of -"application/pdf" and the other had a content type of "image/*", the resulting database query -would be something along these lines: - -```sql -SELECT * FROM UserActivityHistory WHERE - (CONTAINS (DisplayText, '"tax"')) AND - ((CONTAINS (ContentType, '"application/pdf"')) OR - (CONTAINS (ContentType, '"image/*"'))); -``` +The results will include the user activity history items that match any one of the queries passed in. +To match a given query, the item must match all of the criteria specified in that query. Any query +property that is left empty or null will be ignored. A case-insensitive lexical search will be +performed on the keywords in the query, which will match if all of the keywords are found somewhere +in the DisplayText property of the item. -## UserActivityHistory.SearchAsync method +This method will not perform any parsing of the keywords for semantic meaning or natural language - +it is expected that the app will have already performed that step and will pass the result of that +into this API. -This method is an asynchronous version of the `Search` method. +A case-insensitive lexical search will also be performed on the ContentType property of the item, +which is the [MIME type](https://docs.w3cub.com/http/basics_of_http/mime_types/complete_list_of_mime_types.html) +of the resource the user was interacting with. The search will match if the ContentType of the item matches +the ContentType property of the query. The ContentType supports using an asterisk as a wildcard +to match a range of content types - e.g., "image/*" will match "image/png", "image/jpeg", etc. ## UserActivityHistory.GetAppsWithUserActivity method @@ -80,17 +101,13 @@ history database. You can use this, for example, to show the user the list of ap queried against, so the user can understand why an app that is not recording user activity is not showing up in the results. -## UserActivityHistory.GetAppsWithUserActivityAsync method - -This method is an asynchronous version of the `GetAppsWithUserActivity` method. - ## UserActivityHistoryItem class This class represents a single item in the user's activity history. It contains properties that -describe in what app the activity occurred, what the nature of the activity was, the URI of the -resource involved in the activity (e.g., a document, a webpage, a video, etc.), the URI that -can be used to bring back the state the user left the activity in, and the times when the user -started the activity and ended the activity. +describe in what app the activity occurred, how the app described the activity, what sort of +resource was being interacted with (e.g., a document, a webpage, a video, etc.), the URI of +the resource involved in the activity, the URI that can be used to bring back the state the user +left the activity in, and the times when the user started and ended the activity. If the user performed the same activity multiple times, there will be multiple `UserActivityHistoryItem` objects returned, each with different start and end times. @@ -115,6 +132,9 @@ if the activity was reading the contents of a webpage, this property might conta This property contains the MIME type of the content being interacted with. For example, if the user was looking at a PNG image, this property would contain the string "image/png". +Note that this is a string property, not an enum, so apps that populate this property +do not necessarily have to use existing recognized common MIME types. + ## UserActivityHistoryItem.ContentUri property This property contains the URI of the content being interacted with. For example, if the user was @@ -124,7 +144,10 @@ looking at a webpage, this property would contain the URI of that webpage. This property contains the URI that can be used to bring back the state the user left the activity in. For example, if the user was looking at a webpage, this property would contain the URI of that webpage -with additional information such as what page the user was on, what their scroll position was, etc. +with additional information, such as what the scroll position was, etc. + +If this property is not populated on the UserActivity object, then we won't add it to the user activity +history database, as we won't be able to bring back that activity. ## UserActivityHistoryItem.StartTime property @@ -155,27 +178,27 @@ This property is case-insensitive. ## UserActivityHistoryQuery.EarliestStartTime property -This is a nullable DateTime property that specifies the earliest start time of the activity +This is an optional DateTime property that specifies the earliest start time of the activity you want to retrieve. Any activities with a StartTime property earlier than this will be excluded. -If this property is left as null, it will be ignored. +If this property is unspecified, it will be ignored. ## UserActivityHistoryQuery.EarliestEndTime property -This is a nullable DateTime property that specifies the earliest end time of the activity +This is an optional DateTime property that specifies the earliest end time of the activity you want to retrieve. Any activities with an EndTime property earlier than this will be excluded. -If this property is left as null, it will be ignored. +If this property is unspecified, it will be ignored. ## UserActivityHistoryQuery.LatestStartTime property -This is a nullable DateTime property that specifies the latest start time of the activity +This is an optional DateTime property that specifies the latest start time of the activity you want to retrieve. Any activities with a StartTime property later than this will be excluded. -If this property is left as null, it will be ignored. +If this property is unspecified, it will be ignored. ## UserActivityHistoryQuery.LatestEndTime property -This is a nullable DateTime property that specifies the latest end time of the activity +This is an optional DateTime property that specifies the latest end time of the activity you want to retrieve. Any activities with an EndTime property later than this will be excluded. -If this property is left as null, it will be ignored. +If this property is unspecified, it will be ignored. ## UserActivityHistoryOrderBy enum