Skip to content

[Feature](lambda) Support some map lambda functions - #66968

Merged
HappenLee merged 7 commits into
apache:masterfrom
linrrzqqq:map-lambda
Sep 1, 2026
Merged

[Feature](lambda) Support some map lambda functions#66968
HappenLee merged 7 commits into
apache:masterfrom
linrrzqqq:map-lambda

Conversation

@linrrzqqq

@linrrzqqq linrrzqqq commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Related PR: #67045, #67047

Doc: apache/doris-website#4085

Problem Summary:

This PR adds MAP lambda support under the Nereids planner.

Supported higher-order functions:

  • map_filter((k, v) -> predicate, map)
  • map_exists((k, v) -> predicate, map)
  • map_all((k, v) -> predicate, map)
  • map_apply((k, v) -> struct(new_key, new_value), map)
  • transform_keys((k, v) -> new_key, map)
  • transform_values((k, v) -> new_value, map)

Example:

SELECT map_filter(
    (k, v) -> v > 10,
    map(1, 10, 2, 20)
);

SELECT transform_values(
    (k, v) -> v + 1,
    map(1, 10, 2, 20)
);

SELECT map_apply(
    (k, v) -> (k + 1, v * 2),
    map(1, 10, 2, 20)
);

Implementation

Use map_entries(m) to expand the map parameter and reuse the original array lambda for execution.

map_apply((k, v) -> (k2, v2), m)
    ↓
map_from_entries(
    array_apply(entry: struct{k, v} -> (k2, v2), map_entries(m)
);

map_filter((k, v) -> k + v > 0, m)
    ↓
%map_from_filtered_entries_unique%(
    array_map(
        entry: strct{k, v} -> if(
            e.k + e.v > 0, entry, null
        ),
    map_entries(m)
)


transform_values((k, v) -> k + v + col, m)
    ↓
%map_from_entries_unique%(
    array_map(
        entry -> (e.k, e.k + e.v + col),
        map_entries(m)
    )
)


map_exists((k, v) -> k + v = 1, m)
    ↓
array_match_any(
    array_map(
        entry -> e.k + e.v = 1,
        map_entries(m)
    )
)


map_all((k, v) -> k + v = 1, m)
    ↓
array_match_all(
    array_map(
        entry -> e.k + e.v = 1,
        map_entries(m)
    )
)

Internal helper function

This PR adds two internal Map construction functions used by the rewritten expressions:

  • %map_from_entries_unique%: rebuilds a Map from an entry array without running key deduplication. Used by transform_value
  • %map_from_filtered_entries_unique%: removes null entries input instead of throw error like map_from_entries. Used by map_filter

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@linrrzqqq linrrzqqq changed the title [tmp] [Feature](lambda) Support some map lambda functions Aug 21, 2026
@linrrzqqq
linrrzqqq force-pushed the map-lambda branch 2 times, most recently from 5c42734 to 2130e12 Compare August 21, 2026 07:14
@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review status: complete and converged after three rounds. Requesting changes for three P1 correctness issues.

Findings

  1. The FE materialization rule moves sensitive Map inputs from Join candidate-pair scope into a child-row Project, changing outer-join error behavior and volatile evaluation frequency.
  2. The same rule moves computed Maps out of selector-controlled IF/CASE branches, defeating supported short-circuit execution.
  3. The BE eight-batch direct path bypasses the variable-width lambda safety ceiling and can turn a bounded query into a multi-gigabyte ColumnString overflow.

Checkpoint conclusions

  • Goal and scope: the advertised Map lambda wrappers, constructors, tuple syntax, recursive NULL-type handling, and FE-to-BE lowerings are present. The scalar wrappers and registrations are localized; the 786-line late materialization rule and the unrelated ArrayMap batching expansion are the principal risk surfaces.
  • Planner semantics and parallel paths: Project, OneRowRelation, Filter, Having, normalized Aggregate, Generate, Join metadata, nested lambda ExprIds, marker translation, and second-pass stability were reviewed. Outside the two inline evaluation-domain failures, no additional owner or parallel path defect survived.
  • Types, nullability, compatibility, and physical symbols: lambda arity, Map-key legality, nullable Maps/predicates, nested NULL_TYPE merging, last-win semantics, and the names map_from_arrays, %map_from_arrays_unique%, map_filter, and map_from_entries align across FE and BE. No serialized format or existing function signature changes.
  • Runtime correctness and ownership: constant/nullable wrappers, offsets, selected rows, invalid entries, duplicate keys, COW detachment, and Status propagation were checked. Those paths remain sound outside the accepted lambda budget issue.
  • Performance and memory: the direct-path multiplier is a correctness failure as well as an allocation regression; no other distinct performance defect survived review.
  • Concurrency, lifecycle, configuration, persistence, transactions, writes, and observability: no new applicable mechanism is introduced.
  • Tests: the changed unit and regression sources broadly cover ordinary, null, constant, duplicate, empty, nested, aggregate, Join, Generate, and selected-row behavior, but omit the three accepted boundaries. Per the automated-review contract, I did not run builds or tests; current style, license, formatting, and secret checks pass, while build/test jobs are skipped.
  • User focus: no additional focus was supplied.

No pre-existing live review thread or comment duplicated these findings.

Comment thread be/src/exprs/lambda_function/varray_map_function.cpp
@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 17554 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2130e12f16144fc369125d34a5dda5309b3e7184, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17629	3094	3081	3081
q2	1928	255	154	154
q3	10410	954	562	562
q4	4678	264	218	218
q5	7642	580	410	410
q6	139	116	97	97
q7	510	513	399	399
q8	9242	894	1022	894
q9	3539	2462	2432	2432
q10	6528	861	720	720
q11	463	258	234	234
q12	696	388	330	330
q13	17883	1539	1200	1200
q14	155	149	144	144
q15	q16	445	402	363	363
q17	842	733	838	733
q18	3337	2268	2289	2268
q19	1104	941	838	838
q20	662	503	478	478
q21	5301	1765	1935	1765
q22	330	275	234	234
Total cold run time: 93463 ms
Total hot run time: 17554 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3444	3346	3363	3346
q2	205	223	153	153
q3	2363	2475	2245	2245
q4	1224	1203	921	921
q5	2287	2141	2145	2141
q6	184	131	89	89
q7	1073	977	892	892
q8	1659	1461	1461	1461
q9	3223	3222	3242	3222
q10	1908	1883	1669	1669
q11	363	276	262	262
q12	463	434	340	340
q13	1515	1571	1225	1225
q14	195	187	165	165
q15	q16	397	405	374	374
q17	1075	1032	1037	1032
q18	5215	4590	5136	4590
q19	898	895	887	887
q20	1106	1039	1105	1039
q21	3912	3195	3252	3195
q22	413	361	317	317
Total cold run time: 33122 ms
Total hot run time: 29565 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 84739 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2130e12f16144fc369125d34a5dda5309b3e7184, data reload: false

query5	4265	425	333	333
query6	430	162	148	148
query7	4856	460	272	272
query8	317	122	116	116
query9	8693	2890	2918	2890
query10	451	260	228	228
query11	5386	1066	924	924
query12	119	69	70	69
query13	1198	461	310	310
query14	6020	2253	2139	2139
query14_1	2009	1974	2016	1974
query15	172	122	112	112
query16	919	397	375	375
query17	795	459	370	370
query18	2338	329	239	239
query19	171	140	124	124
query20	73	75	69	69
query21	211	119	103	103
query22	5855	5395	5650	5395
query23	6869	6376	6083	6083
query23_1	6075	6220	6269	6220
query24	7296	1111	810	810
query24_1	807	790	820	790
query25	447	323	274	274
query26	1256	274	181	181
query27	2712	458	295	295
query28	4592	1498	1496	1496
query29	921	435	350	350
query30	272	182	177	177
query31	871	461	392	392
query32	101	50	51	50
query33	459	218	181	181
query34	1016	836	496	496
query35	399	398	352	352
query36	574	542	536	536
query37	116	82	69	69
query38	1016	848	835	835
query39	529	482	476	476
query39_1	501	473	450	450
query40	220	127	110	110
query41	54	52	50	50
query42	82	79	83	79
query43	243	240	214	214
query44	1012	548	573	548
query45	118	104	106	104
query46	787	839	525	525
query47	777	769	777	769
query48	327	322	239	239
query49	560	243	192	192
query50	834	329	251	251
query51	8211	8296	8337	8296
query52	73	78	65	65
query53	199	207	157	157
query54	223	172	171	171
query55	79	60	57	57
query56	231	228	204	204
query57	663	687	653	653
query58	233	192	190	190
query59	1216	1219	1095	1095
query60	258	219	188	188
query61	131	125	120	120
query62	345	201	179	179
query63	184	157	169	157
query64	2878	820	683	683
query65	1644	1619	1740	1619
query66	1917	349	262	262
query67	10387	9959	9862	9862
query68	2875	1159	794	794
query69	356	236	211	211
query70	682	622	649	622
query71	292	286	244	244
query72	2524	1761	1570	1570
query73	644	588	360	360
query74	1848	1255	1169	1169
query75	1245	1170	1017	1017
query76	2287	757	542	542
query77	263	257	214	214
query78	4011	3743	3255	3255
query79	2770	818	623	623
query80	1636	402	340	340
query81	536	203	180	180
query82	681	121	104	104
query83	319	251	234	234
query84	325	123	102	102
query85	885	444	370	370
query86	489	183	174	174
query87	1028	986	901	901
query88	2937	2126	2141	2126
query89	315	223	205	205
query90	2016	146	139	139
query91	158	141	124	124
query92	57	48	47	47
query93	1723	1163	818	818
query94	628	235	196	196
query95	641	345	361	345
query96	802	583	270	270
query97	1041	1062	1040	1040
query98	176	137	130	130
query99	416	352	309	309
Total cold run time: 181011 ms
Total hot run time: 84739 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.67 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2130e12f16144fc369125d34a5dda5309b3e7184, data reload: false

query1	0.01	0.00	0.01
query2	0.08	0.04	0.04
query3	0.24	0.10	0.10
query4	1.61	0.10	0.10
query5	0.17	0.16	0.16
query6	1.25	0.67	0.72
query7	0.03	0.01	0.00
query8	0.04	0.03	0.03
query9	0.29	0.21	0.21
query10	0.36	0.35	0.36
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.31	0.30	0.30
query14	0.46	0.44	0.45
query15	0.36	0.36	0.37
query16	0.23	0.22	0.21
query17	0.71	0.70	0.69
query18	0.18	0.17	0.17
query19	1.24	1.23	1.13
query20	0.02	0.01	0.01
query21	15.45	0.17	0.11
query22	5.06	0.05	0.04
query23	16.16	0.25	0.10
query24	2.99	0.32	0.25
query25	0.10	0.04	0.03
query26	0.74	0.17	0.12
query27	0.03	0.04	0.03
query28	3.61	0.56	0.27
query29	12.43	3.14	2.57
query30	0.26	0.12	0.12
query31	2.76	0.38	0.16
query32	3.50	0.30	0.23
query33	1.36	1.46	1.42
query34	15.41	2.19	1.79
query35	1.75	1.73	1.71
query36	0.46	0.29	0.30
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.11	0.08	0.07
query41	0.08	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 90.35 s
Total hot run time: 14.67 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 77.36% (540/698) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 62.45% (29116/46620)
Line Coverage 47.48% (304812/641967)
Region Coverage 43.15% (246043/570241)
Branch Coverage 44.68% (114474/256188)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.95% (34292/45153)
Line Coverage 60.89% (386426/634622)
Region Coverage 56.97% (323618/568077)
Branch Coverage 57.81% (147689/255463)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 68.43% (479/700) 🎉
Increment coverage report
Complete coverage report

@linrrzqqq
linrrzqqq force-pushed the map-lambda branch 2 times, most recently from 1872ace to 9651dc4 Compare August 23, 2026 18:00
@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review status: complete and converged after three rounds. Requesting changes for one new P1 correctness issue.

Finding

  • Native-table access-path pruning can lose Map values used only inside a nested Lambda and substitute default/null values; one inline comment is attached.
  • The existing P1 variable-width Lambda budget thread remains applicable and was not duplicated.

Critical checkpoint conclusions

  • Goal and proof: the PR adds Map Lambda functions, Map constructors/filtering, tuple Lambda syntax, and FE-to-BE lowering. Registrations and ordinary semantics align, and unit/regression sources cover null, empty, duplicate, constant, computed, nondeterministic, nested, aggregate, Join, and Generate cases; the accepted native stored-column boundary is not covered.
  • Scope and focus: the implementation is cohesive for the advertised feature and reuses ArrayMap/MapEntries/Map primitives. No additional user focus was supplied.
  • Concurrency: execution uses query-local expression/column state; no new shared mutable state, locks, lock ordering, or thread-safety mechanism is introduced.
  • Lifecycle/static initialization: Lambda frames use scoped guards and ColumnMap sharing uses COW; no non-intuitive release path, circular ownership, or cross-TU static initialization dependency was found.
  • Configuration: no configuration item or dynamic-update path is added.
  • Compatibility: FE wrapper names, arities, return/nullability types, and internal physical symbols match BE registration. No storage format, persisted metadata, or protocol variable changes are introduced; the new function symbols are additive.
  • Parallel paths: public Map functions, internal unique/filtered constructors, legacy two-argument map_filter, constant folding, computed inputs, nested Lambdas, and native/external scan paths were checked. The native scan divergence is the attached finding.
  • Conditional checks: const broadcasting, top-level null Maps/arrays, hidden null-row payloads, mismatched offsets, nullable predicates, and last-win deduplication checks have explicit failure/skip behavior and targeted tests. No separate conditional-check defect survived.
  • Test coverage: BE unit tests and FE/regression tests are broad, including negative cases, but need a stored native-table test where the outer Map Lambda retains the key and reads the value only inside an inner Lambda.
  • Test results: the committed .out results match the asserted ordinary semantics. Per the review-run contract, I did not run builds or tests.
  • Observability: these are scalar analysis/execution paths; existing AnalysisException/Status failures are sufficient, and no new log or metric is required.
  • Transactions/persistence: no transaction, EditLog, master-failover, or persistent-state path is changed.
  • Data writes/crashes: the feature is query-time only and does not modify stored data. COW detachment and Status propagation are sound outside the accepted read-pruning correctness issue.
  • FE-BE variables: no new session/protocol variable is transmitted. FE and BE independently computed types/names were checked and agree.
  • Performance: zero-copy/COW paths are reasonable, but the existing live byte-budget P1 remains unresolved; no distinct additional CPU/memory issue survived duplicate fencing.
  • Other issues: the scalar-subquery tuple-parser candidate was rejected because the semantic predicate only gates, rather than forces, its alternative. No other unresolved candidate remains.

@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 17112 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9651dc40a821a64e8eee1bc21d5fe3ab472bc746, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17560	3053	3010	3010
q2	1874	225	151	151
q3	10485	849	511	511
q4	4671	252	196	196
q5	7679	593	383	383
q6	137	110	99	99
q7	536	516	389	389
q8	9240	922	933	922
q9	3525	2437	2344	2344
q10	6517	898	710	710
q11	434	257	242	242
q12	684	396	328	328
q13	17855	1513	1165	1165
q14	162	148	139	139
q15	q16	431	398	364	364
q17	808	852	797	797
q18	3146	2229	2231	2229
q19	1104	855	783	783
q20	597	501	438	438
q21	5200	1685	1736	1685
q22	323	260	227	227
Total cold run time: 92968 ms
Total hot run time: 17112 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3373	3322	3286	3286
q2	206	208	154	154
q3	2211	2340	2177	2177
q4	1180	1161	887	887
q5	2182	2109	2127	2109
q6	171	122	86	86
q7	1010	905	867	867
q8	1603	1400	1385	1385
q9	3135	3058	3062	3058
q10	1874	1807	1621	1621
q11	350	268	246	246
q12	458	431	332	332
q13	1488	1522	1179	1179
q14	170	164	168	164
q15	q16	391	390	345	345
q17	1040	1032	1022	1022
q18	4887	4343	4860	4343
q19	877	855	842	842
q20	983	929	778	778
q21	3472	3301	3281	3281
q22	410	341	336	336
Total cold run time: 31471 ms
Total hot run time: 28498 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82668 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9651dc40a821a64e8eee1bc21d5fe3ab472bc746, data reload: false

query5	4259	397	344	344
query6	398	162	194	162
query7	4840	446	258	258
query8	286	119	115	115
query9	8698	2814	2849	2814
query10	399	244	240	240
query11	5377	1025	908	908
query12	119	76	71	71
query13	1201	449	333	333
query14	6174	2174	2065	2065
query14_1	1971	1945	1913	1913
query15	172	122	110	110
query16	907	376	401	376
query17	808	467	385	385
query18	2330	337	240	240
query19	167	142	113	113
query20	71	68	78	68
query21	215	117	103	103
query22	5277	5320	5170	5170
query23	6752	6310	6119	6119
query23_1	5929	6109	6006	6006
query24	7269	1112	771	771
query24_1	764	773	773	773
query25	407	288	245	245
query26	1255	264	156	156
query27	2725	443	285	285
query28	4637	1482	1486	1482
query29	915	432	344	344
query30	265	179	154	154
query31	834	418	351	351
query32	96	48	48	48
query33	451	213	162	162
query34	1010	854	484	484
query35	404	400	331	331
query36	553	536	540	536
query37	117	78	67	67
query38	993	836	825	825
query39	543	468	471	468
query39_1	477	481	510	481
query40	216	121	113	113
query41	52	50	50	50
query42	78	75	81	75
query43	235	235	208	208
query44	1008	541	565	541
query45	116	105	100	100
query46	739	851	507	507
query47	765	733	696	696
query48	318	294	228	228
query49	539	231	180	180
query50	807	324	264	264
query51	7902	7870	7931	7870
query52	78	85	71	71
query53	199	198	163	163
query54	265	175	171	171
query55	73	62	57	57
query56	273	219	222	219
query57	673	651	646	646
query58	228	209	192	192
query59	1196	1200	1111	1111
query60	254	206	216	206
query61	136	138	131	131
query62	340	212	174	174
query63	185	153	161	153
query64	2936	785	657	657
query65	1629	1572	1558	1558
query66	1979	314	248	248
query67	10288	9971	9563	9563
query68	3033	1202	718	718
query69	332	210	191	191
query70	670	583	607	583
query71	314	249	231	231
query72	2345	1750	1554	1554
query73	642	641	350	350
query74	2006	1208	1135	1135
query75	1243	1142	1013	1013
query76	2388	733	544	544
query77	262	265	208	208
query78	3844	3608	3188	3188
query79	2114	909	615	615
query80	1597	388	343	343
query81	487	195	182	182
query82	624	128	99	99
query83	322	248	231	231
query84	307	121	105	105
query85	844	438	396	396
query86	391	177	168	168
query87	991	966	898	898
query88	2778	2129	2118	2118
query89	302	221	204	204
query90	2024	146	149	146
query91	157	151	119	119
query92	52	48	44	44
query93	1503	1160	762	762
query94	634	246	224	224
query95	637	360	396	360
query96	776	561	253	253
query97	1075	1034	1011	1011
query98	145	133	132	132
query99	411	353	310	310
Total cold run time: 178242 ms
Total hot run time: 82668 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.61 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9651dc40a821a64e8eee1bc21d5fe3ab472bc746, data reload: false

query1	0.01	0.00	0.00
query2	0.07	0.03	0.03
query3	0.24	0.11	0.09
query4	1.61	0.11	0.10
query5	0.18	0.16	0.15
query6	1.26	0.67	0.67
query7	0.03	0.01	0.00
query8	0.05	0.03	0.04
query9	0.29	0.21	0.21
query10	0.35	0.34	0.36
query11	0.17	0.11	0.12
query12	0.15	0.12	0.11
query13	0.30	0.31	0.31
query14	0.44	0.45	0.46
query15	0.36	0.36	0.33
query16	0.21	0.20	0.22
query17	0.69	0.72	0.74
query18	0.17	0.16	0.17
query19	1.19	1.19	1.22
query20	0.02	0.01	0.01
query21	15.45	0.16	0.12
query22	5.10	0.04	0.05
query23	16.18	0.26	0.10
query24	2.97	0.32	0.26
query25	0.12	0.03	0.03
query26	0.79	0.16	0.13
query27	0.04	0.03	0.03
query28	3.63	0.55	0.28
query29	12.44	3.21	2.54
query30	0.27	0.11	0.12
query31	2.76	0.38	0.17
query32	3.54	0.33	0.23
query33	1.49	1.44	1.38
query34	15.40	2.19	1.75
query35	1.74	1.76	1.71
query36	0.45	0.30	0.27
query37	0.06	0.04	0.04
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.10	0.08	0.07
query41	0.08	0.03	0.02
query42	0.03	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 90.55 s
Total hot run time: 14.61 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 82.27% (283/344) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 62.47% (29128/46629)
Line Coverage 47.49% (304953/642120)
Region Coverage 43.12% (245929/570358)
Branch Coverage 44.69% (114516/256256)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.93% (34292/45162)
Line Coverage 60.91% (386640/634775)
Region Coverage 57.05% (324146/568194)
Branch Coverage 57.85% (147817/255531)

@linrrzqqq
linrrzqqq marked this pull request as ready for review August 24, 2026 01:56
linrrzqqq and others added 2 commits August 28, 2026 19:34
Issue Number: None

Related PR: apache#66968

Problem Summary: Higher-order function analysis selected map handling and array comparator behavior with hardcoded function-name checks. Move lambda parameter binding contracts into builtin function registration and use the resolved function builder metadata to analyze array and map lambdas through one flow.

None

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.nereids.rules.analysis.FunctionRegistryTest,org.apache.doris.nereids.trees.expressions.functions.scalar.MapLambdaFunctionsTest,org.apache.doris.nereids.rules.analysis.CheckExpressionLegalityTest,org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayFirstLastTest
    - mvn checkstyle:check -pl fe-core
- Behavior changed: No
- Does this need documentation: No
@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 80.11% (290/362) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 62.76% (29386/46823)
Line Coverage 47.75% (307452/643944)
Region Coverage 43.36% (248302/572604)
Branch Coverage 44.93% (115593/257272)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16843 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 21af11aee56d1953fd173513fb4b9ae8f56767a9, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17574	3017	2986	2986
q2	2080	274	227	227
q3	10226	859	531	531
q4	4668	253	201	201
q5	7674	574	392	392
q6	135	116	98	98
q7	541	514	382	382
q8	9259	883	964	883
q9	3485	2406	2390	2390
q10	6512	847	708	708
q11	389	199	182	182
q12	605	266	200	200
q13	18147	1502	1158	1158
q14	159	151	146	146
q15	q16	429	394	370	370
q17	1296	890	856	856
q18	3033	2250	2246	2246
q19	1125	885	762	762
q20	356	289	203	203
q21	4819	1693	1846	1693
q22	324	270	229	229
Total cold run time: 92836 ms
Total hot run time: 16843 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3330	3288	3282	3282
q2	493	391	375	375
q3	2239	2281	2165	2165
q4	1178	1164	884	884
q5	2179	2092	2104	2092
q6	170	118	86	86
q7	1043	928	909	909
q8	1593	1406	1412	1406
q9	3107	3093	3069	3069
q10	1868	1801	1630	1630
q11	350	268	250	250
q12	442	429	333	333
q13	1448	1553	1161	1161
q14	169	168	154	154
q15	q16	396	393	361	361
q17	3675	3342	3170	3170
q18	4814	4420	4683	4420
q19	838	804	794	794
q20	1129	996	844	844
q21	3815	3119	3287	3119
q22	405	345	312	312
Total cold run time: 34681 ms
Total hot run time: 30816 ms

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.21% (34560/45351)
Line Coverage 61.21% (389677/636587)
Region Coverage 57.42% (327544/570422)
Branch Coverage 58.24% (149417/256539)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81668 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 21af11aee56d1953fd173513fb4b9ae8f56767a9, data reload: false

query5	4289	408	347	347
query6	378	141	121	121
query7	4957	376	237	237
query8	296	125	115	115
query9	8680	2859	2869	2859
query10	391	221	191	191
query11	5387	1020	928	928
query12	116	71	67	67
query13	1185	439	318	318
query14	6050	2189	2070	2070
query14_1	1956	1960	1956	1956
query15	170	117	107	107
query16	902	363	341	341
query17	808	423	351	351
query18	2312	307	231	231
query19	171	133	109	109
query20	68	67	68	67
query21	192	100	85	85
query22	5438	5486	5282	5282
query23	6627	6305	5956	5956
query23_1	6056	6118	6098	6098
query24	7246	1078	751	751
query24_1	773	781	783	781
query25	414	287	240	240
query26	1223	229	148	148
query27	2773	428	249	249
query28	4689	1500	1497	1497
query29	910	435	342	342
query30	244	150	128	128
query31	810	400	323	323
query32	126	74	70	70
query33	457	213	172	172
query34	993	819	467	467
query35	405	395	329	329
query36	579	555	526	526
query37	116	78	69	69
query38	990	840	807	807
query39	486	484	465	465
query39_1	444	450	486	450
query40	202	92	73	73
query41	53	52	53	52
query42	78	78	77	77
query43	237	241	208	208
query44	1020	552	546	546
query45	110	108	97	97
query46	752	807	543	543
query47	756	759	694	694
query48	297	315	234	234
query49	529	224	192	192
query50	749	265	194	194
query51	7968	8042	8005	8005
query52	88	67	61	61
query53	206	202	146	146
query54	246	200	165	165
query55	81	60	56	56
query56	214	192	171	171
query57	705	651	661	651
query58	219	187	174	174
query59	1301	1261	1083	1083
query60	243	195	176	176
query61	134	129	128	128
query62	357	201	185	185
query63	174	144	151	144
query64	2837	748	599	599
query65	1621	1566	1609	1566
query66	1932	267	215	215
query67	9761	9865	9522	9522
query68	2912	1214	728	728
query69	337	213	199	199
query70	675	619	614	614
query71	258	184	164	164
query72	2328	1744	1584	1584
query73	620	583	340	340
query74	1970	1229	1140	1140
query75	1182	1098	943	943
query76	2312	732	556	556
query77	258	262	208	208
query78	4059	3676	3201	3201
query79	2789	859	577	577
query80	1577	324	299	299
query81	513	154	131	131
query82	619	122	100	100
query83	284	212	197	197
query84	294	110	89	89
query85	815	352	296	296
query86	472	177	168	168
query87	1028	970	876	876
query88	3151	2109	2109	2109
query89	286	198	173	173
query90	2170	126	125	125
query91	133	123	101	101
query92	103	72	71	71
query93	2432	1115	736	736
query94	624	240	213	213
query95	509	324	228	228
query96	803	564	271	271
query97	1042	1036	989	989
query98	176	141	129	129
query99	416	347	308	308
Total cold run time: 179532 ms
Total hot run time: 81668 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 21af11aee56d1953fd173513fb4b9ae8f56767a9, data reload: false

query1	0.00	0.01	0.00
query2	0.08	0.04	0.04
query3	0.24	0.11	0.11
query4	1.60	0.10	0.10
query5	0.18	0.15	0.16
query6	1.24	0.72	0.66
query7	0.03	0.00	0.00
query8	0.04	0.04	0.04
query9	0.29	0.22	0.21
query10	0.34	0.34	0.34
query11	0.17	0.12	0.11
query12	0.15	0.12	0.12
query13	0.30	0.31	0.31
query14	0.47	0.45	0.45
query15	0.37	0.34	0.33
query16	0.22	0.23	0.22
query17	0.67	0.70	0.71
query18	0.18	0.15	0.16
query19	1.19	1.15	1.21
query20	0.01	0.01	0.01
query21	15.44	0.16	0.11
query22	5.08	0.05	0.04
query23	16.16	0.25	0.10
query24	2.97	0.33	0.26
query25	0.11	0.04	0.04
query26	0.74	0.16	0.12
query27	0.05	0.03	0.02
query28	3.65	0.55	0.29
query29	12.43	3.15	2.54
query30	0.26	0.11	0.11
query31	2.76	0.36	0.17
query32	3.53	0.33	0.22
query33	1.36	1.40	1.43
query34	15.35	2.14	1.78
query35	1.74	1.74	1.75
query36	0.46	0.31	0.28
query37	0.06	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.02	0.03
query40	0.12	0.08	0.07
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 90.27 s
Total hot run time: 14.64 s

morrySnow
morrySnow previously approved these changes Aug 31, 2026
Comment thread fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java Outdated
@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 67.12% (298/444) 🎉
Increment coverage report
Complete coverage report

@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 62.78% (29397/46825)
Line Coverage 47.79% (307755/643968)
Region Coverage 43.35% (248250/572616)
Branch Coverage 44.94% (115615/257282)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.03% (34479/45351)
Line Coverage 61.02% (388451/636604)
Region Coverage 57.10% (325733/570430)
Branch Coverage 57.96% (148702/256545)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.86% (34405/45351)
Line Coverage 60.86% (387429/636604)
Region Coverage 56.93% (324727/570430)
Branch Coverage 57.80% (148286/256545)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 41.22% (298/723) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.86% (34405/45351)
Line Coverage 60.86% (387423/636604)
Region Coverage 56.92% (324669/570430)
Branch Coverage 57.80% (148294/256545)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 40.47% (295/729) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.87% (34406/45351)
Line Coverage 60.86% (387425/636604)
Region Coverage 56.92% (324686/570430)
Branch Coverage 57.81% (148299/256545)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16779 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba0667d57111839263495d4176819981c28ac78d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17613	3073	3056	3056
q2	2105	273	222	222
q3	10214	882	522	522
q4	4671	248	205	205
q5	7670	571	385	385
q6	135	113	92	92
q7	555	494	388	388
q8	9253	867	916	867
q9	3423	2398	2389	2389
q10	6505	869	717	717
q11	386	199	183	183
q12	613	257	199	199
q13	18134	1571	1152	1152
q14	163	149	140	140
q15	q16	442	405	370	370
q17	1374	902	767	767
q18	3167	2263	2272	2263
q19	1253	917	797	797
q20	387	283	204	204
q21	5629	1633	1816	1633
q22	336	276	228	228
Total cold run time: 94028 ms
Total hot run time: 16779 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3436	3366	3367	3366
q2	508	413	381	381
q3	2203	2586	2140	2140
q4	1202	1163	904	904
q5	2188	2142	2109	2109
q6	167	117	88	88
q7	1041	945	885	885
q8	1616	1421	1420	1420
q9	3135	3120	3120	3120
q10	1884	1825	1616	1616
q11	362	269	252	252
q12	463	437	348	348
q13	1483	1536	1170	1170
q14	166	170	159	159
q15	q16	405	393	359	359
q17	3598	3341	3257	3257
q18	4841	4384	4774	4384
q19	935	897	890	890
q20	1028	959	857	857
q21	3900	3271	3181	3181
q22	405	351	329	329
Total cold run time: 34966 ms
Total hot run time: 31215 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82451 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ba0667d57111839263495d4176819981c28ac78d, data reload: false

query5	4272	421	355	355
query6	389	141	135	135
query7	4889	398	230	230
query8	293	133	118	118
query9	8698	2920	2916	2916
query10	386	242	189	189
query11	5391	1030	920	920
query12	113	70	70	70
query13	1188	461	343	343
query14	6043	2206	2085	2085
query14_1	1991	1993	1980	1980
query15	172	127	112	112
query16	921	371	357	357
query17	786	470	360	360
query18	2336	321	230	230
query19	167	141	114	114
query20	74	72	72	72
query21	204	103	91	91
query22	5629	5471	5429	5429
query23	6735	6319	6013	6013
query23_1	6242	6106	6127	6106
query24	7323	1103	772	772
query24_1	796	813	799	799
query25	438	307	264	264
query26	1224	223	135	135
query27	2787	416	263	263
query28	4692	1543	1479	1479
query29	914	433	341	341
query30	244	160	127	127
query31	816	401	336	336
query32	127	78	72	72
query33	448	218	171	171
query34	979	843	492	492
query35	404	388	337	337
query36	569	606	530	530
query37	114	81	69	69
query38	995	856	832	832
query39	495	482	466	466
query39_1	462	433	467	433
query40	203	92	75	75
query41	55	51	52	51
query42	75	73	71	71
query43	243	242	218	218
query44	1032	552	557	552
query45	108	106	99	99
query46	762	835	521	521
query47	768	773	724	724
query48	308	312	224	224
query49	536	255	186	186
query50	789	275	201	201
query51	8277	8134	8171	8134
query52	67	68	64	64
query53	195	209	147	147
query54	231	185	173	173
query55	98	62	56	56
query56	205	159	169	159
query57	676	666	668	666
query58	193	175	169	169
query59	1227	1237	1114	1114
query60	230	204	192	192
query61	140	127	138	127
query62	347	199	173	173
query63	174	148	143	143
query64	2745	730	660	660
query65	1684	1592	1666	1592
query66	1783	261	219	219
query67	9830	9500	9718	9500
query68	3006	1206	756	756
query69	363	222	211	211
query70	677	613	602	602
query71	257	193	178	178
query72	2407	1749	1593	1593
query73	613	554	340	340
query74	2019	1222	1144	1144
query75	1204	1111	950	950
query76	2371	730	554	554
query77	264	266	201	201
query78	4115	3609	3196	3196
query79	2790	879	570	570
query80	1585	345	277	277
query81	493	155	134	134
query82	624	125	99	99
query83	290	212	194	194
query84	301	111	91	91
query85	813	351	299	299
query86	392	181	174	174
query87	1027	989	918	918
query88	2826	2136	2110	2110
query89	288	194	173	173
query90	1956	134	131	131
query91	133	121	100	100
query92	82	73	73	73
query93	1704	1105	669	669
query94	638	250	215	215
query95	527	256	233	233
query96	766	600	252	252
query97	1082	1069	1023	1023
query98	162	139	134	134
query99	419	350	317	317
Total cold run time: 179361 ms
Total hot run time: 82451 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.62 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba0667d57111839263495d4176819981c28ac78d, data reload: false

query1	0.00	0.00	0.00
query2	0.08	0.04	0.04
query3	0.24	0.09	0.10
query4	1.60	0.10	0.09
query5	0.17	0.16	0.16
query6	1.27	0.70	0.68
query7	0.03	0.01	0.00
query8	0.05	0.03	0.03
query9	0.29	0.21	0.21
query10	0.34	0.34	0.34
query11	0.17	0.11	0.12
query12	0.16	0.12	0.11
query13	0.31	0.31	0.31
query14	0.45	0.45	0.46
query15	0.36	0.35	0.34
query16	0.23	0.21	0.23
query17	0.67	0.71	0.68
query18	0.18	0.17	0.16
query19	1.23	1.08	1.21
query20	0.02	0.01	0.01
query21	15.42	0.17	0.13
query22	5.06	0.04	0.04
query23	16.23	0.27	0.10
query24	2.97	0.32	0.26
query25	0.11	0.03	0.03
query26	0.75	0.16	0.12
query27	0.05	0.03	0.04
query28	3.65	0.56	0.26
query29	12.47	3.22	2.60
query30	0.26	0.10	0.11
query31	2.75	0.38	0.17
query32	3.51	0.31	0.23
query33	1.62	1.44	1.42
query34	15.39	2.18	1.77
query35	1.77	1.73	1.73
query36	0.45	0.30	0.30
query37	0.06	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.12	0.09	0.07
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 90.71 s
Total hot run time: 14.62 s

@HappenLee HappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HappenLee
HappenLee merged commit a10c799 into apache:master Sep 1, 2026
33 of 35 checks passed
@linrrzqqq
linrrzqqq deleted the map-lambda branch September 1, 2026 06:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants