Automatically try to figure out if fuzzer issue is caused by optimizers and which optimizer is the culprit

[Mytherin]

Many internal exceptions are caused by optimizers. My first step in debugging fuzzer issues is generally doing a binary search of optimizers to figure out which one is causing it (if any).

For example, this is my debugging plan for this issue duckdb/duckdb-fuzzer#3354

Simple first step, disabling optimizers and checking if the issue persists:

PRAGMA disable_optimizer;
[RUN TEST]
┌─────────────────────────────────────────┬──────┐
│                   c0                    │  c1  │
│                 uint128                 │ json │
├─────────────────────────────────────────┼──────┤
│                                       0 │ {}   │
│ 340282366920938463463374607431768211455 │ {}   │
└─────────────────────────────────────────┴──────┘

This works, so clearly the problem is an optimizer problem.

The list of optimizers can be obtained from select string_agg(name) from duckdb_optimizers();.

We can disable these optimizers:

SET disabled_optimizers='expression_rewriter,filter_pullup,filter_pushdown,empty_result_pullup,cte_filter_pusher,regex_range,in_clause,join_order,deliminator,unnest_rewriter,unused_columns,statistics_propagation,common_subexpressions,common_aggregate,column_lifetime,limit_pushdown,top_n,build_side_probe_side,compressed_materialization,duplicate_groups,reorder_filter,sampling_pushdown,join_filter_pushdown,extension,materialized_cte,sum_rewriter,late_materialization';
[RUN TEST]

This still works.

We can now try to remove optimizers one by one, until we figure out when it stops working.

set disabled_optimizers='late_materialization';
[RUN TEST]

This works, so clearly the culprit is late materialization.

This could be figured out automatically and posted in the fuzzer issue as well.