Skip to content

Fix Universal Connector JSON schema reference (sourceType, ContinuationToken fields, progressWindowParams) #6954

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
diabler-ie wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Fix Universal Connector JSON schema reference (sourceType, ContinuationToken fields, progressWindowParams) #6954

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
45f4d5b
Fix Universal Connector JSON schema reference: sourceType value, Cont…
diabler-ie Jul 23, 2026
a929d0c
Address review: drop unverified PageBased, correct ContinuationToken …
diabler-ie Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 12 additions & 7 deletions ...d-collectors/cloud-to-cloud-integration-framework/universal-connector-source.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,7 +353,7 @@ Sources can be configured using UTF-8 encoded JSON files with the Collector Ma
| Parameter | Type | Value | Required | Description |
| :--------- | :---------- | :-------------------------------------------- | :------- | :------------------------------- |
| schemaRef | JSON Object | `{"type":"Universal Connector"}` | Yes | Define the specific schema type. |
| sourceType | String | `"Universal Connector"` | Yes | Type of source. |
| sourceType | String | `"Universal"` | Yes | Type of source. All Cloud-to-Cloud sources use `"Universal"` here; the connector is selected by `schemaRef`. Using `"Universal Connector"` fails with `Missing required field: 'properties'`. |
| config | JSON Object | [Configuration object](#configuration-object) | Yes | Source type specific values. |

### Configuration object
Expand Down Expand Up @@ -383,19 +383,24 @@ Sources can be configured using UTF-8 encoded JSON files with the Collector Ma
| requestHeaders | JSON Object | No | `null` | Any HTTP request headers to include. | `"requestHeaders": [{"headerName": "Accept", "headerValue": "application/json"}, {"headerName": "Content-Type", "headerValue": "application/json"}]` |
| requestParams | JSON Object | No | `null` | Any HTTP URL parameters to include. | `"requestParams": [{"paramName": "limit", "paramValue": "1000"}, {"paramName": "since", "paramValue": "{{ .WindowStartUTC \"2006-01-02T15:04:05Z07:00\" }}"}, {"paramName": "until", "paramValue": "{{ .WindowEndUTC \"2006-01-02T15:04:05Z07:00\" }}"}` |
| requestBody | String | No | `null` | The data to include in the HTTP request body if the `POST` method is used. | |
| progressType | String | Yes | `"window"` | Select the type of progression the source will use to prevent data loss and duplication. | `"progressionType": "window"` |
| progressWindowSize | String | Yes | `"5m"` | The size of the time window. | `"windowSize": "5m"` |
| progressWindowInitLookback | String | Yes | `"24h"` | How far back the source should start collecting data when created. This setting has no affect after the initial creation. | `"windowInitialLookback": "24h"` |
| progressWindowMaxLookback | String | Yes | `"31d"` | How far the window is allowed to stagnate when encountering repetitive errors. | `"windowMaxLookback": "31d"` |
| progressType | String | Yes | `"window"` | Select the type of progression the source will use to prevent data loss and duplication. | `"progressType": "window"` |
| progressWindowSize | String | Yes | `"5m"` | The size of the time window. | `"progressWindowSize": "5m"` |
| progressWindowInitLookback | String | Yes | `"24h"` | How far back the source should start collecting data when created. This setting has no effect after the initial creation. | `"progressWindowInitLookback": "24h"` |
| progressWindowMaxLookback | String | Yes | `"31d"` | How far the window is allowed to stagnate when encountering repetitive errors. | `"progressWindowMaxLookback": "31d"` |
| progressWindowParams | JSON Object | No | `start`/`end` params | The request parameters that carry the collection window timestamps. If omitted, the source defaults to appending `start` and `end` query parameters built from the window (equivalent to `[{"paramName": "start", "paramValue": "{{ .WindowStartUTC \"yyyy-MM-ddTHH:mm:ssZ\" }}"}, {"paramName": "end", "paramValue": "{{ .WindowEndUTC \"yyyy-MM-ddTHH:mm:ssZ\" }}"}]`). An API that rejects unknown query parameters will fail with a `THIRD-PARTY-CONFIG` error on the first poll (for example, `400: Unknown query parameter: 'end'`); for APIs that take no time-window parameters, set this explicitly to an empty array `[]` so the default parameters are not sent. | `"progressWindowParams": []` |
| responseLogsType | String | Yes | `"json"` | How the source should ingest logs from the response. | `"json"` |
| responseLogsJsonPaths | JSON Object | Yes | `null` | The location of logs to ingest in the JSON response and how to handle event timestamps. See [full documentation](/docs/send-data/reference-information/time-reference) for details. | `[{"logsPath": "$[*]", "logTimestampPath": "$.published", "logTimestampFormat": "2006-01-02T15:04:05.999Z", "logTimestampValueRegex": "Date\((.*)\)"}]`<br></br>**Note:** For regex, there should be at least one match group. If there is more than one match group, then only the first group will be considered. |
| paginationType | String | Yes | `"LinkHeaders"` | Pagination type `LinkHeaders`, `Offset`, or `None` | `"LinkHeaders"`, `"None"` |
| paginationType | String | Yes | `"LinkHeaders"` | Pagination type `LinkHeaders`, `ContinuationToken`, `Offset`, or `None`. The `LinkHeaders` and `ContinuationToken` types require their companion `...Type` field (see the rows below); when it is missing, validation fails with misleading errors such as `ContinuationToken is not a valid enum value` rather than naming the missing field. | `"LinkHeaders"`, `"ContinuationToken"`, `"None"` |
| paginationLinkHeadersType | String | Yes | `"headers"` | Configures if the next page URL is included in the Link HTTP response header or in the response body. | `"headers"`, `"body"` |
| paginationLinkHeadersJPath | String | No | `null` | A JSON Path to the appropriate body property |
| paginationContinuationTokenType | String | Yes | `null` | Required when `paginationType` is `ContinuationToken`. Configures if the continuation token is read from the response headers or the response body. | `"headers"`, `"body"` |
| paginationContinuationTokenJsonPath | String | No | `null` | A JSON Path to the continuation token in the response body, when `paginationContinuationTokenType` is `body`. | `"$.next_page"` |
| paginationContinuationTokenKey | String | No | `null` | The request key name that carries the continuation token on follow-up requests. | `"page"` |
| paginationContinuationTokenLocation | String | No | `null` | The location in the HTTP request to send the continuation token. | `"parameters"`, `"headers"` |
| paginationOffsetLocation | String | No | `parameters` | The location in the HTTP request to use the numeric offset pagination key/value pairs |
| paginationOffsetKey | String | No | `offset` | The key name for the offset to use in the HTTP request headers or parameters |
| paginationOffsetLimitKey | String | No | `limit` | The key name for the limit to use in the HTTP request headers or parameters |
| paginationOffsetLimitValue | Integer | No | `5000` | The limit value to use to restrict the results per page | `"$.link.next"` |
| paginationOffsetLimitValue | Integer | No | `5000` | The limit value to use to restrict the results per page | `5000` |
| clientTimeoutDuration | String | Yes | `"5m"` | How long the source allows the HTTP connection to live before closing it and setting the health to a timeout error. | `"5m"` |
| clientTimeoutRetries | Integer | Yes | `5` | The source will automatically retry without waiting for the next poll interval this many times for some errors such as 500 Internal Server. | `5` |
| clientRateLimitReqs | Integer | Yes | `1000` | The number of HTTP requests the source is allowed to make within the rate limit duration. | `1000` |
Expand Down