Logictest: Build arrow records directly

[thorfour]

Logictests build arrow records by building a parquet record and then using the converter to convert it into an arrow record. Since we no longer support inserting by Parquet records, we should just build the arrow record directly instead.

[elliotchenzichang]

Could I take this?

[thorfour]

Could I take this?

Apologies for the slow response. Yes it would be great if you want to take this on. I'll assign the issue to you

[gernest]

Building records directly is simple. However this requires sorting arrow.Record before inserting to the table. Sorting is not implemented in arrow package yet.

I'm contemplating using normal sort.Interface to build indices and use compute.Take , I'm not so sure if it is worthy to implement this to only be used in logictest.

[asubiotto]

We actually have a lot of use for the ability to sort arrow records in a lot of places. This is (what now seems to be abandoned) work in the arrow project we've been hoping to revive at some point: apache/arrow#34719. If you're up for it and looking for this kind of challenge, it would be fantastic to have this in upstream arrow. If not, a more specific solution would already be very useful to us. Whatever you end up building for logictests would likely also be reused in other places. What you describe sounds good to me.

[asubiotto]

I also forgot that @metalmatze has a PR open to perform frostdb-specific sorting: #461. We should probably focus on merging that first, although I think that having upstream record sorting would be fantastic.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.