diff --git a/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/universal-connector-source.md b/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/universal-connector-source.md index 2e8f70fe630..852679c4efa 100644 --- a/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/universal-connector-source.md +++ b/docs/send-data/hosted-collectors/cloud-to-cloud-integration-framework/universal-connector-source.md @@ -353,7 +353,7 @@ Sources can be configured using UTF-8 encoded JSON files with the Collector Ma | Parameter | Type | Value | Required | Description | | :--------- | :---------- | :-------------------------------------------- | :------- | :------------------------------- | | schemaRef | JSON Object | `{"type":"Universal Connector"}` | Yes | Define the specific schema type. | -| sourceType | String | `"Universal Connector"` | Yes | Type of source. | +| sourceType | String | `"Universal"` | Yes | Type of source. All Cloud-to-Cloud sources use `"Universal"` here; the connector is selected by `schemaRef`. Using `"Universal Connector"` fails with `Missing required field: 'properties'`. | | config | JSON Object | [Configuration object](#configuration-object) | Yes | Source type specific values. | ### Configuration object @@ -383,19 +383,24 @@ Sources can be configured using UTF-8 encoded JSON files with the Collector Ma | requestHeaders | JSON Object | No | `null` | Any HTTP request headers to include. | `"requestHeaders": [{"headerName": "Accept", "headerValue": "application/json"}, {"headerName": "Content-Type", "headerValue": "application/json"}]` | | requestParams | JSON Object | No | `null` | Any HTTP URL parameters to include. | `"requestParams": [{"paramName": "limit", "paramValue": "1000"}, {"paramName": "since", "paramValue": "{{ .WindowStartUTC \"2006-01-02T15:04:05Z07:00\" }}"}, {"paramName": "until", "paramValue": "{{ .WindowEndUTC \"2006-01-02T15:04:05Z07:00\" }}"}` | | requestBody | String | No | `null` | The data to include in the HTTP request body if the `POST` method is used. | | -| progressType | String | Yes | `"window"` | Select the type of progression the source will use to prevent data loss and duplication. | `"progressionType": "window"` | -| progressWindowSize | String | Yes | `"5m"` | The size of the time window. | `"windowSize": "5m"` | -| progressWindowInitLookback | String | Yes | `"24h"` | How far back the source should start collecting data when created. This setting has no affect after the initial creation. | `"windowInitialLookback": "24h"` | -| progressWindowMaxLookback | String | Yes | `"31d"` | How far the window is allowed to stagnate when encountering repetitive errors. | `"windowMaxLookback": "31d"` | +| progressType | String | Yes | `"window"` | Select the type of progression the source will use to prevent data loss and duplication. | `"progressType": "window"` | +| progressWindowSize | String | Yes | `"5m"` | The size of the time window. | `"progressWindowSize": "5m"` | +| progressWindowInitLookback | String | Yes | `"24h"` | How far back the source should start collecting data when created. This setting has no effect after the initial creation. | `"progressWindowInitLookback": "24h"` | +| progressWindowMaxLookback | String | Yes | `"31d"` | How far the window is allowed to stagnate when encountering repetitive errors. | `"progressWindowMaxLookback": "31d"` | +| progressWindowParams | JSON Object | No | `start`/`end` params | The request parameters that carry the collection window timestamps. If omitted, the source defaults to appending `start` and `end` query parameters built from the window (equivalent to `[{"paramName": "start", "paramValue": "{{ .WindowStartUTC \"yyyy-MM-ddTHH:mm:ssZ\" }}"}, {"paramName": "end", "paramValue": "{{ .WindowEndUTC \"yyyy-MM-ddTHH:mm:ssZ\" }}"}]`). An API that rejects unknown query parameters will fail with a `THIRD-PARTY-CONFIG` error on the first poll (for example, `400: Unknown query parameter: 'end'`); for APIs that take no time-window parameters, set this explicitly to an empty array `[]` so the default parameters are not sent. | `"progressWindowParams": []` | | responseLogsType | String | Yes | `"json"` | How the source should ingest logs from the response. | `"json"` | | responseLogsJsonPaths | JSON Object | Yes | `null` | The location of logs to ingest in the JSON response and how to handle event timestamps. See [full documentation](/docs/send-data/reference-information/time-reference) for details. | `[{"logsPath": "$[*]", "logTimestampPath": "$.published", "logTimestampFormat": "2006-01-02T15:04:05.999Z", "logTimestampValueRegex": "Date\((.*)\)"}]`

**Note:** For regex, there should be at least one match group. If there is more than one match group, then only the first group will be considered. | -| paginationType | String | Yes | `"LinkHeaders"` | Pagination type `LinkHeaders`, `Offset`, or `None` | `"LinkHeaders"`, `"None"` | +| paginationType | String | Yes | `"LinkHeaders"` | Pagination type `LinkHeaders`, `ContinuationToken`, `Offset`, or `None`. The `LinkHeaders` and `ContinuationToken` types require their companion `...Type` field (see the rows below); when it is missing, validation fails with misleading errors such as `ContinuationToken is not a valid enum value` rather than naming the missing field. | `"LinkHeaders"`, `"ContinuationToken"`, `"None"` | | paginationLinkHeadersType | String | Yes | `"headers"` | Configures if the next page URL is included in the Link HTTP response header or in the response body. | `"headers"`, `"body"` | | paginationLinkHeadersJPath | String | No | `null` | A JSON Path to the appropriate body property | +| paginationContinuationTokenType | String | Yes | `null` | Required when `paginationType` is `ContinuationToken`. Configures if the continuation token is read from the response headers or the response body. | `"headers"`, `"body"` | +| paginationContinuationTokenJsonPath | String | No | `null` | A JSON Path to the continuation token in the response body, when `paginationContinuationTokenType` is `body`. | `"$.next_page"` | +| paginationContinuationTokenKey | String | No | `null` | The request key name that carries the continuation token on follow-up requests. | `"page"` | +| paginationContinuationTokenLocation | String | No | `null` | The location in the HTTP request to send the continuation token. | `"parameters"`, `"headers"` | | paginationOffsetLocation | String | No | `parameters` | The location in the HTTP request to use the numeric offset pagination key/value pairs | | paginationOffsetKey | String | No | `offset` | The key name for the offset to use in the HTTP request headers or parameters | | paginationOffsetLimitKey | String | No | `limit` | The key name for the limit to use in the HTTP request headers or parameters | -| paginationOffsetLimitValue | Integer | No | `5000` | The limit value to use to restrict the results per page | `"$.link.next"` | +| paginationOffsetLimitValue | Integer | No | `5000` | The limit value to use to restrict the results per page | `5000` | | clientTimeoutDuration | String | Yes | `"5m"` | How long the source allows the HTTP connection to live before closing it and setting the health to a timeout error. | `"5m"` | | clientTimeoutRetries | Integer | Yes | `5` | The source will automatically retry without waiting for the next poll interval this many times for some errors such as 500 Internal Server. | `5` | | clientRateLimitReqs | Integer | Yes | `1000` | The number of HTTP requests the source is allowed to make within the rate limit duration. | `1000` |