Issue renaming fields after splitting array using unnest!

[sunil9420]

Question

I’m processing logs that contain a top-level records array and sending them to OpenSearch through Vector.

My goal is to:

Split the records array into multiple events (one per record).

Rename or extract some fields (e.g., operationName → op_name, operationVersion → op_version, correlationId → corr_id, properties.status → status).

I’m able to successfully split the array using unnest!, but the renaming part isn’t working as expected.

Logs are successfully sent to OpenSearch when I don’t use any renaming.

Logs don’t reach OpenSearch at all when renaming is attempted after the split.

###

Vector Config

✅ If I only split without renaming, logs ingest fine:

transforms:
  split_records:
    type: remap
    inputs:
      - redpanda_provisioninglogs
    drop_on_abort: false
    source: |
      if is_array(.records) {
        . = unnest!(.records)
      }

✅ This config works (logs ingest, but renaming doesn’t happen):

transforms:
  split_records:
    type: remap
    inputs:
      - redpanda_provisioninglogs
    drop_on_abort: false
    source: |
      if is_array(.records) {
        . = unnest!(.records)
      }
      if exists(.properties) && exists(.properties.status) {
        .status = .properties.status
      }

❌ This config is not working (logs don’t reach OpenSearch at all):

transforms:
  split_records:
    type: remap
    inputs:
      - redpanda_provisioninglogs
    drop_on_abort: false
    source: |
      if is_array(.records) {
        . = unnest!(.records)
      }
      
      .status = .properties.status

Vector Logs

Sample log

{
  "records": [
    {
      "time": "2025-09-30T10:14:56.8973362Z",
      "resourceId": "/tenants/.../providers/Microsoft.aadiam",
      "operationName": "Provisioning activity",
      "operationVersion": "1.0",
      "category": "ProvisioningLogs",
      "tenantId": "bd647acc-6b4d-48bc-bfb6-7204de32003b",
      "resultType": "Skipped",
      "durationMs": 122,
      "correlationId": "9d6e1868-5f22-4d5d-adfb-cdbced03a909",
      "properties": {
        "id": "38148ed1-c9bf-b37b-c2bc-716708d63208",
        "activityDateTime": "2025-09-30T10:14:56.8973362Z",
        "action": "Create",
        "provisioningAction": "create",
        "provisioningStatusInfo": {
          "Status": "skipped",
          "errorInformation": null
        },
        "sourceIdentity": {
          "details": {
            "DisplayName": "Raquel Fernandes",
            "UserPrincipalName": "[email protected]"
          }
        }
      }
    },
    {
      "time": "2025-09-30T10:14:56.9113361Z",
      "resourceId": "/tenants/.../providers/Microsoft.aadiam",
      "operationName": "Provisioning activity",
      "operationVersion": "1.0",
      "category": "ProvisioningLogs",
      "tenantId": "bd647acc-6b4d-48bc-bfb6-7204de32003b",
      "resultType": "Skipped",
      "durationMs": 123,
      "correlationId": "9d3a6868-6ace-46fd-8397-b3dd31151c0c",
      "properties": {
        "id": "6b05faa9-c79c-bdca-93cb-c771681abbcb",
        "activityDateTime": "2025-09-30T10:14:56.9113361Z",
        "action": "Create",
        "provisioningAction": "create",
        "provisioningStatusInfo": {
          "Status": "skipped",
          "errorInformation": null
        },
        "sourceIdentity": {
          "details": {
            "DisplayName": "Ajay Pareek",
            "UserPrincipalName": "[email protected]"
          }
        }
      }
    }
  ]
}

[thomasqueirozb]

Hi @sunil9420

Logs are not getting past the transform because after you call unnest (and even before), .properties.status is not valid. Only children of records have .properties.provisioningStatusInfo.Status. You can play around with your input using the playground.

I suspect you actually want to do something like this:

if is_array(.records) {
    records = []
    for_each(array!(.records)) -> |_index, value| {
        value.status = value.properties.provisioningStatusInfo.Status
        records = append(records, [value])
    }

    .records = records
    . = unnest!(.records)
}