fix: Spark Schema Evolution Fix for nested columns

[the-other-tim-brown]

Describe the issue this Pull Request addresses

Currently when we are trying to read a Parquet file through Spark and the file has a different schema than the requested schema we can have runtime errors. These errors happen specifically when a column contains a record and the query selects a subset of columns from that record. If the schema of the record for that particular data file does not have the any of the requested fields, then there will be an error.

Summary and Changelog

The Spark Parquet reader is updated to remove any fields from the selected fields if the those fields result in records with no sub-fields.

This is validated by enabling the FileGroupReader path in one of the existing tests.

Impact

Fixes a bug in the FileGroupReader path for Spark reads.

Risk Level

Low

Documentation Update

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[hudi-bot]

CI report:

• 4fe01a6 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build