Skip to content

Switch to TemplateSetValue with features for combining data from mult… #809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
john-hill merged 1 commit into from
Nov 23, 2025
Merged

Switch to TemplateSetValue with features for combining data from mult… #809

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions src/main/resources/templates/repo/agent/grid/grid-agent-instructions.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,29 @@
<update__json>{"updateBatch":[{"set":[{"concreteType":"org.sagebionetworks.repo.model.grid.update.LiteralSetValue","columnName":"status","value":"active"}],"filters":[{"concreteType":"org.sagebionetworks.repo.model.grid.query.CellValueFilter","columnName":"age","operator":"GREATER_THAN","value":18}]},{"set":[{"concreteType":"org.sagebionetworks.repo.model.grid.update.LiteralSetValue","columnName":"category","value":"senior"}],"filters":[{"concreteType":"org.sagebionetworks.repo.model.grid.query.CellValueFilter","columnName":"age","operator":"GREATER_THAN_OR_EQUALS","value":65}]},{"set":[{"concreteType":"org.sagebionetworks.repo.model.grid.update.LiteralSetValue","columnName":"discount","value":0.15}],"filters":[{"concreteType":"org.sagebionetworks.repo.model.grid.query.RowSelectionFilter","isSelected":true}]}]}</update__json>
</update__example>
<update__example>
<description>Extract participant ID from fileName column using regex pattern 'participant-(\d+)' and set it to the participantId column. If the pattern doesn't match, set participantId to null.</description>
<update__json>{"updateBatch":[{"set":[{"concreteType":"org.sagebionetworks.repo.model.grid.update.RegexExtractSetValue","columnName":"participantId","sourceColumnName":"fileName","pattern":"participant-(\\d+)","groupIndex":1,"onMatchFailure":"SET_NULL"}],"filters":[{"concreteType":"org.sagebionetworks.repo.model.grid.query.CellValueFilter","columnName":"fileName","operator":"IS_NOT_NULL"}]}]}</update__json>
<description>Use a template to combine firstName and lastName columns into fullName with a space separator.</description>
<update__json>{"updateBatch":[{"set":[{"concreteType":"org.sagebionetworks.repo.model.grid.update.TemplateSetValue","columnName":"fullName","sourceTemplate":"{firstName} {lastName}","onMatchFailure":"SET_NULL"}],"filters":[{"concreteType":"org.sagebionetworks.repo.model.grid.query.CellValueFilter","columnName":"firstName","operator":"IS_NOT_NULL"}]}]}</update__json>
</update__example>
<update__example>
<description>Extract domain from email using regex pattern, treating missing email values as empty strings (which won't match the pattern)</description>
<update__json>{"updateBatch":[{"set":[{"concreteType":"org.sagebionetworks.repo.model.grid.update.TemplateSetValue","columnName":"domain","sourceTemplate":"{email}","pattern":"@(.+)$","onMatchFailure":"SET_NULL","onMissingValue":"USE_EMPTY_STRING"}],"filters":[{"concreteType":"org.sagebionetworks.repo.model.grid.query.CellValueFilter","columnName":"email","operator":"IS_NOT_NULL"}]}]}</update__json>
</update__example>
<update__example>
<description>Build full file path from bucket, folder, and filename columns; skip updating rows where any source column is missing.</description>
<update__json>{"updateBatch":[{"set":[{"concreteType":"org.sagebionetworks.repo.model.grid.update.TemplateSetValue","columnName":"fullPath","sourceTemplate":"{bucket}/{folder}/{filename}","onMissingValue":"SKIP_UPDATE"}],"filters":[{"concreteType":"org.sagebionetworks.repo.model.grid.query.CellValueFilter","columnName":"fullPath","operator":"IS_NULL"}]}]}</update__json>
</update__example>
<update__example>
<description>Reformat phone numbers from '(555) 123-4567' to '555-123-4567' by removing parentheses using regex replacement. Note: This pattern assumes consistent formatting; rows with different formats will trigger onMatchFailure behavior. </description>
<update__json>{"updateBatch":[{"set":[{"concreteType":"org.sagebionetworks.repo.model.grid.update.TemplateSetValue","columnName":"phone","sourceTemplate":"{phone}","pattern":"\\((\\d{3})\\)\\s*(\\d{3}-\\d{4})","replacement":"$1-$2","onMatchFailure":"SKIP_UPDATE"}],"filters":[{"concreteType":"org.sagebionetworks.repo.model.grid.query.CellValueFilter","columnName":"phone","operator":"LIKE","value":"(%"}]}]}</update__json>
</update__example>
</examples>
</update__examples>
<template_set_value>
TemplateSetValue enables complex data transformations by:
- Interpolating values from multiple columns using {columnName} placeholders
- Applying optional regex patterns for extraction or transformation
- Providing configurable fallback behaviors for missing data or pattern mismatches
</template_set_value>
</tools>
</operational_context>

Expand Down
36 changes: 23 additions & 13 deletions src/main/resources/templates/repo/agent/grid/grid_agent_open_api.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,41 +95,51 @@
},
"description": "Results of a query against a grid session."
},
"org.sagebionetworks.repo.model.grid.update.RegexExtractSetValue": {
"org.sagebionetworks.repo.model.grid.update.TemplateSetValue": {
"type": "object",
"properties": {
"concreteType": {
"type": "string",
"description": "Required. Full name of the exact type used.",
"enum": ["org.sagebionetworks.repo.model.grid.update.RegexExtractSetValue"]
"enum": ["org.sagebionetworks.repo.model.grid.update.TemplateSetValue"]
},
"columnName": {
"type": "string",
"description": "Target column name to update."
},
"sourceColumnName": {
"sourceTemplate": {
"type": "string",
"description": "The name of the column to read the value from (e.g., 'fileName')."
"description": "Required. Template string with column placeholders in {columnName} format. Example: '{firstName} {lastName}' concatenates two columns with a space. Invalid column names in placeholders will trigger onMissingValue behavior."
},
"pattern": {
"type": "string",
"description": "The regular expression to apply. Must include at least one capture group (e.g., 'site-\\d+/(\\w+)/.*')."
"description": "Optional. A Java-compatible regex pattern to apply to the intermediate string generated by 'sourceTemplate'."
},
"groupIndex": {
"type": "integer",
"description": "The 1-based index of the capture group to extract.",
"format": "int32"
"replacement": {
"type": "string",
"description": "Optional. The replacement string using regex capture group references ($1, $2, etc.). Only used when 'pattern' is provided. Examples: '$1-$2' to join groups with a dash, '$2-group-$1' to reorder. If 'pattern' is specified without 'replacement', defaults to '$1' (extracts first capture group)."
},
"onMatchFailure": {
"type": "string",
"description": "Action to take if the regex does not match the source value. Default SET_NULL.",
"description": "Action if 'pattern' is provided but does not match. SET_NULL sets the value to NULL. SET_UNDEFINED removes the key from the JSON object. SKIP_UPDATE leaves the existing value unchanged.",
"enum": [
"SET_NULL",
"SET_UNDEFINED",
"SKIP_UPDATE"
]
},
"onMissingValue": {
"type": "string",
"description": "Action if a column referenced in 'sourceTemplate' is null/undefined. SET_NULL sets the value to NULL. SET_UNDEFINED removes the key from the JSON object. SKIP_UPDATE leaves the existing value unchanged. USE_EMPTY_STRING replaces the missing value with '' (empty string) when interpolating the template.",
"enum": [
"SET_NULL",
"SET_UNDEFINED",
"SKIP_UPDATE",
"USE_EMPTY_STRING"
]
}
},
"description": "Sets the column's value by extracting a capture group from a regex pattern applied to a source column.",
"description": "Sets a column's value by composing a string from a source template, then optionally applying a regex-replace to that intermediate string.",
"required": ["concreteType"]
},
"org.sagebionetworks.repo.model.grid.update.GridUpdateRequest": {
Expand Down Expand Up @@ -651,8 +661,8 @@
},
"description": "",
"oneOf": [
{"$ref": "#/components/schemas/org.sagebionetworks.repo.model.grid.update.RegexExtractSetValue"},
{"$ref": "#/components/schemas/org.sagebionetworks.repo.model.grid.update.LiteralSetValue"}
{"$ref": "#/components/schemas/org.sagebionetworks.repo.model.grid.update.LiteralSetValue"},
{"$ref": "#/components/schemas/org.sagebionetworks.repo.model.grid.update.TemplateSetValue"}
],
"required": ["concreteType"],
"discriminator": {"propertyName": "concreteType"}
Expand Down