Using the tap-zendesk throws an exception

[melnimr]

When I pipe the output from the tap-zendesk with the bellow schema it gives me this error:
CRITICAL ('Invalid records detected above threshold: 0. See.argsfor details.', [(<ValidationError: "None is not of type 'object'">, {'type': 'RECORD', 'stream': 'tickets', 'record': {'assignee_id': 389975897113, 'brand_id': 360003165813, 'collaborator_ids': [], 'created_at': '2019-09-03T19:10:47.000000Z', 'custom_fields': [{'id': 360025763994, 'value': None}
The schema is bellow:

{"type": "SCHEMA", "stream": "tickets", "schema": {"properties": {"organization_id": {"type": ["null", "integer"]}, "requester_id": {"type": ["null", "integer"]}, "problem_id": {"type": ["null", "integer"]}, "is_public": {"type": ["null", "boolean"]}, "description": {"type": ["null", "string"]}, "follower_ids": {"items": {"type": ["null", "integer"]}, "type": ["null", "array"]}, "submitter_id": {"type": ["null", "integer"]}, "generated_timestamp": {"type": ["null", "integer"]}, "brand_id": {"type": ["null", "integer"]}, "id": {"type": ["null", "integer"]}, "group_id": {"type": ["null", "integer"]}, "type": {"type": ["null", "string"]}, "recipient": {"type": ["null", "string"]}, "collaborator_ids": {"items": {"type": ["null", "integer"]}, "type": ["null", "array"]}, "tags": {"items": {"type": ["null", "string"]}, "type": ["null", "array"]}, "has_incidents": {"type": ["null", "boolean"]}, "created_at": {"format": "date-time", "type": ["null", "string"]}, "raw_subject": {"type": ["null", "string"]}, "status": {"type": ["null", "string"]}, "updated_at": {"format": "date-time", "type": ["null", "string"]}, "custom_fields": {"items": {"properties": {"id": {"type": ["null", "integer"]}, "value": {}}, "type": ["null", "object"]}, "type": ["null", "array"]}, "url": {"type": ["null", "string"]}, "allow_channelback": {"type": ["null", "boolean"]}, "allow_attachments": {"type": ["null", "boolean"]}, "due_at": {"format": "date-time", "type": ["null", "string"]}, "followup_ids": {"items": {"type": ["null", "integer"]}, "type": ["null", "array"]}, "priority": {"type": ["null", "string"]}, "assignee_id": {"type": ["null", "integer"]}, "subject": {"type": ["null", "string"]}, "external_id": {"type": ["null", "string"]}, "via": {"properties": {"source": {"properties": {"from": {"properties": {"name": {"type": ["null", "string"]}, "ticket_id": {"type": ["null", "integer"]}, "address": {"type": ["null", "string"]}, "subject": {"type": ["null", "string"]}}, "type": ["null", "object"]}, "to": {"properties": {"address": {"type": ["null", "string"]}, "name": {"type": ["null", "string"]}}, "type": ["null", "object"]}, "rel": {"type": ["null", "string"]}}, "type": ["null", "object"]}, "channel": {"type": ["null", "string"]}}, "type": ["null", "object"]}, "ticket_form_id": {"type": ["null", "integer"]}, "satisfaction_rating": {"properties": {"id": {"type": ["null", "integer"]}, "assignee_id": {"type": ["null", "integer"]}, "group_id": {"type": ["null", "integer"]}, "reason_id": {"type": ["null", "integer"]}, "requester_id": {"type": ["null", "integer"]}, "ticket_id": {"type": ["null", "integer"]}, "updated_at": {"format": "date-time", "type": ["null", "string"]}, "created_at": {"format": "date-time", "type": ["null", "string"]}, "url": {"type": ["null", "string"]}, "score": {"type": ["null", "string"]}, "reason": {"type": ["null", "string"]}, "comment": {"type": ["null", "string"]}}, "type": ["null", "object", "string"]}, "sharing_agreement_ids": {"items": {"type": ["null", "integer"]}, "type": ["null", "array"]}, "email_cc_ids": {"items": {"type": ["null", "integer"]}, "type": ["null", "array"]}, "forum_topic_id": {"type": ["null", "integer"]}}, "type": ["null", "object"]}, "key_properties": ["id"]}

[AlexanderMann]

@melnimr, you have two options here:

1. Use some of the configuration flags offerred in target-postgres to resolve this. Namely:
   • invalid_records_detect: Could set this to false and ignore all invalid records.
     • NOTE: These will also not be persisted through to remote
   • invalid_records_threshold : Could set this to something greater than 0. This will allow some records to be invalid, but will ultimately fail on more than the threshold.
     • NOTE: These will also not be persisted through to remote
2. This is an issue with tap-zendesk passing invalid data, so you'd have to take this up with them. Your record in question:

'custom_fields': [
  {'id': 360025763994,
   'value': None}
]

Has the key value set to None.

In your schema, value is supposed to be an object, and is not nullable:

"custom_fields": {
  "items": {
    "properties": {
      "id": {"type": ["null", "integer"]},
      "value": {} <----THIS
    }

[AlexanderMann]

@melnimr, do you think the error message for this could be improved to suggest the above?

[melnimr]

@AlexanderMann so it is a problem with tap-zendesk like you mentioned, I did a PR for fixing their hard-coded schema,
singer-io/tap-zendesk#30

[melnimr]

@AlexanderMann I think the error message is fine the way it is now. Thanks.