Investigate performance issues

[edgarrmondragon]

I debugged this ELT (a log file with 13+ GB of content), and there are no traces of redundancy, every record appears only once.

The target-postgres batch_processing_time is an average of 20 s to each batch, and it sums to ~6 minutes of total COPY operation time, which is acceptable.

The tap-postgres average extraction time interval between records is ~0.001 s, what leads (roughly) to a total of 90 minutes of data reading.

If anyone has ideas, evidence (or even better, a flame graph 😉) as to what's causing this performance problem, they're more than welcome!

Related

• Discovery Performance #284
• Maybe fix: Exclude information_schema and pg_catalog from discovery #551?

Links

• Slack: https://meltano.slack.com/archives/C069A0GE129/p1739276072020269
• Linen: https://discuss.meltano.com/t/26949263/greetings-again-i-am-creating-a-tap-postgres-target-postgres