This code is both library code and code to run a process on a server. Mixing these means that it is hard to meet both demands. Case in point: #10
That change introduced a dependency on Sentry, for one thing. It also suppresses bad data in favor of logging that bad data to sentry.
A much better pattern would be that each record is returned with an error state that allows the calling code to deal with it as it wished. However, that pattern doesn't work too well with the need to checkpoint and batch load data as the design currently stands. We should clean this up.
This code is both library code and code to run a process on a server. Mixing these means that it is hard to meet both demands. Case in point: #10
That change introduced a dependency on Sentry, for one thing. It also suppresses bad data in favor of logging that bad data to sentry.
A much better pattern would be that each record is returned with an error state that allows the calling code to deal with it as it wished. However, that pattern doesn't work too well with the need to checkpoint and batch load data as the design currently stands. We should clean this up.