feat: removed all instances of deprecated virtualtable values field

[eliot1480]

Which issue does this PR close?

• Closes Remove usages of deprecated VirtualTable.values field #23608

Rationale for this change

Since the VirtualTables.values field has been deprecated and is soon to be removed in the next substrait update, we should make sure to delete all instances of it to prevent the presence of dead code.

What changes are included in this PR?

Producer now emits VirtualTable.expressions for Values and EmptyRelation, and the consumer now only reads VirtualTable.expressions. Converted 17 direct virtualTable.values entries across 10 JSON test plans to virtualTable.expressions.

Are these changes tested?

Yes they are. Tests ran:

cargo check -p datafusion-substrait
cargo test -p datafusion-substrait -- test substrait_integration roundtrip_values
cargo test -p datafusion-substrait -- test substrait_integration consumer_integration
cargo test -p datafusion-substrait -- test substrait_integration builtin_expr_semantics_tests
cargo test -p datafusion-substrait -- test substrait_integration non_nullable_lists

jq validation for JSON fixtures

Are there any user-facing changes?

No.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 78.84615% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.67%. Comparing base (f151c10) to head (b2e4cb5).
⚠️ Report is 16 commits behind head on main.

Files with missing lines	Patch %	Lines
...ubstrait/src/logical_plan/consumer/rel/read_rel.rs	79.54%	7 Missing and 2 partials ⚠️
...ubstrait/src/logical_plan/producer/rel/read_rel.rs	75.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #23672   +/-   ##
=======================================
  Coverage   80.66%   80.67%           
=======================================
  Files        1086     1086           
  Lines      366673   366740   +67     
  Branches   366673   366740   +67     
=======================================
+ Hits       295786   295853   +67     
+ Misses      53263    53261    -2     
- Partials    17624    17626    +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[vbarua]

I think we can still check and short-circuit on vt.expressions.is_empty() here.

[vbarua]

Do we actually need to special case the Literal consumer here? I would have assumed that consume_expression would be able to handle it, and more specifically the block in

if !vt.expressions.is_empty() {

had does everything we needed for the conversion here.

Put another way, it looks like you've inlined some of the convert_literal_rows code, but I don't think it's needed.

[vbarua]

Thinking about this from a migration perspective, I think we need to do ship this as 2 changes:

1. Release a change that switches the producer to generate VirtualTables using only the expressions field.
2. After that is live, release the change that fully removes the old values field.

Because the producer currently emits plans using the deprecated field, we need to give users a chance to update any stored plans, and then remove support for values.

[vbarua]

Something I'm realizing, for migration purpose we may first want to ship a change that updates this to write these values using expressions.

If we start off right of the bat removing all support for values, anyone storing plans outside of DataFusion won't be able to read them post update, because we'll only handle expressions.