Skip to content

Comment on pre-existing table crashes target-postgres #197

Open
@laurentS

Description

@laurentS

I tried running this target (via meltano) with postgres_schema set to an existing schema which already contained tables (with data). Some of the tables had comments on them, as created by:

COMMENT ON TABLE public.mytable IS ''my custom table with important data';

When running the target, it crashed with the following stacktrace:

target-postgres | ERROR Exception writing records
target-postgres | Traceback (most recent call last):
target-postgres |   File "~/myproject/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/target_postgres/postgres.py", line 237, in write_batch
target-postgres |     self.setup_table_mapping_cache(cur)
target-postgres |   File "~/myproject/.meltano/loaders/target-postgres/venv/lib/python3.8/site-packages/target_postgres/postgres.py", line 224, in setup_table_mapping_cache
target-postgres |     table_path = json.loads(raw_json).get('path', None)
target-postgres |   File "/home/laurent/.pyenv/versions/3.8.7/lib/python3.8/json/__init__.py", line 357, in loads
target-postgres |     return _default_decoder.decode(s)
target-postgres |   File "/home/laurent/.pyenv/versions/3.8.7/lib/python3.8/json/decoder.py", line 337, in decode
target-postgres |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
target-postgres |   File "/home/laurent/.pyenv/versions/3.8.7/lib/python3.8/json/decoder.py", line 355, in raw_decode
target-postgres |     raise JSONDecodeError("Expecting value", s, err.value) from None
target-postgres | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
target-postgres | CRITICAL ('Exception writing records', JSONDecodeError('Expecting value: line 1 column 1 (char 0)'))

With some extra logging, raw_json actually contains the value of the comment mentioned above, which clearly isn't JSON.
This is the line of code that breaks

table_path = json.loads(raw_json).get('path', None)

Is this a bug, or does target-postgres expect the schema to be empty when it starts?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions