Refactoring how we're handling typing

[QuanMPhm]

Currently, our code is quite disorganized with how it is handling typing. A quick search of the term "pyarrow" or "decimal" shows that we're explicitly typing certain fields in miscellaneous places. In a few more places and even in our test cases, we have to typecast to fix equality checks.

Because of how disorganized things are, this has led to:

1. Debugging typing-related problems leads to even more messiness.
2. Confusion over the typing of fields, which leads to immense frustration when writing test cases

We should re-consider how we're handling typing to reduce future programming burden.

@larsks @naved001 @knikolla Happy to hear your thoughts.

[QuanMPhm]

@knikolla @larsks My proposal is to create a new processor at the start of the pipeline, which initializes all the nessecary columns and set them to the appropriate type. This will be done using a mapping between column names and their type. This would remove the need for every other processor to create columns. I’ll go forward with this if you guys think it’s fine.

[knikolla]

@knikolla @larsks My proposal is to create a new processor at the start of the pipeline, which initializes all the nessecary columns and set them to the appropriate type. This will be done using a mapping between column names and their type. This would remove the need for every other processor to create columns. I’ll go forward with this if you guys think it’s fine.

I don't agree with an initial processor precreating all the columns for the other processors. You can create functions in the base class and define standard types so that the Processors know what and how to create things consistently.

[morantara]

@QuanMPhm looks like there's still some discussion here, should it be pushed to the next sprint?

[QuanMPhm]

@morantara Since @knikolla is on PTO, I assume all of my invoicing PRs or issues will be pushed, since he is one of the main reviewers of this repo.

[QuanMPhm]

This issue will be closed by #285