Update pyarrow and microdata-tools

[pawbu]

One test for partitioned datasets read the .paquet file which returns also the partitioned column. Added a test for the usual case when just asking for "start_year=123" and not "start_year=123/04f4164ec1f247f2ad392fa9c03e71fe-0.parquet".

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[DanielElisenberg]

so reading the whole directory in as a table doesn't include the column that was partitioned upon, but reading one of the partitions directly yields this column?🤔

EDIT:

• Reading the whole dataset would yield the partitioned column status_date
• Reading the partition folder has status_data
• Reading the parquet in the partition folder does not have status_date

Would this be correct? And why do we want to test reading from a single partition, since the app never does? 👀 To document pyarrow behavior?

[pawbu]

start_year is the partitioned column in this case, so:

• Reading the whole dataset would yield the partitioned column start_year
• Reading the partition folder does not yield start_year
• Reading the (single) parquet file in the partition folder would yield start_year

And why do we want to test reading from a single partition, since the app never does?

I haven't checked the reason for the test in question. Can do that later after merging this PR since the microdata-tools is out, so we should not be runnning different version here in job-executor for too long 👍

[DanielElisenberg]

That makes sense 👍🏻 Reasonable conclusion. Let's look at whether reading the separate folders and containing parquet files is reasonable in a new PR 💯