Commit 925d325
committed
[Data] Refactor eval_projection stub column handling
Improve the stub column cleanup logic in eval_projection():
- Replace hardcoded "__stub__" with _BATCH_SIZE_PRESERVING_STUB_COL_NAME constant
- Use block_accessor.select([]) instead of fill + drop pattern for cleaner logic
- Add conditional check to prevent dropping user columns that match stub name
- Reuse BlockAccessor instance to avoid redundant object creation
- Enhance comment to explain Arrow's empty table limitation
Testing:
- Add test for empty block (0 rows) handling
- Add test for stub column name collision with user projection
- Add test for single column projection
- Verify stub column is properly cleaned up in Arrow tables
- Add value assertions in stub column name collision test
Signed-off-by: slfan1989 <slfan1989@apache.org>1 parent 1ab2d07 commit 925d325
File tree
1 file changed
+26
-0
lines changed- python/ray/data/_internal/planner/plan_expression
1 file changed
+26
-0
lines changedLines changed: 26 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
748 | 748 | | |
749 | 749 | | |
750 | 750 | | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
751 | 777 | | |
752 | 778 | | |
753 | 779 | | |
| |||
0 commit comments