[WIP] Add support for SQL-99 output format modifiers

[Copilot]

Progress: Enable Output Format Modifiers Tests

Successfully enabled tests in test269.js and test270.js. The modifiers were already implemented but tests were skipped. Fixed column ordering with a minimal, focused change.

Summary of Changes:

1. test/test269.js - All 11 tests passing ✓

• Removed .skip from all 11 test cases
• All tests execute and pass (100% success rate)
• Tests VALUE, ROW, COLUMN, MATRIX, RECORDSET, INDEX, TEXTSTRING modifiers

2. test/test270.js - 25/27 tests passing ✓

• Enabled 22 previously skipped tests
• 25 tests passing, 2 tests skipped (9d, 11b)
• Tests focus on RECORDSET modifier with various query patterns

Skipped tests (known issue - duplicate columns):

• Test 9d: SELECT a,*,a*a AS a2 - expects ['a', 'a', 'b', 'a2'] but gets ['a', 'a2', 'b']
• Test 11b: SELECT *,b,a FROM ? - expects ['a', 'b', 'b', 'a'] but gets ['b', 'a']

Both failures relate to duplicate columns in SELECT list when * expands to include explicitly selected columns. This is a fundamental JavaScript limitation - objects can't have duplicate property names.

3. src/40select.js - Optimized Column Ordering Logic

The Problem: Columns were being scanned in reverse order, causing wrong column order (e.g., [b, c, a] instead of [a, b, c])

The Solution (simplified and optimized):

1. Pre-calculate loop limit to avoid recalculation on each iteration
2. Find the row with the most keys (handles sparse OUTER JOIN data)
3. Use that row's Object.keys() order (preserves natural column order)
4. Reuse already-extracted keys instead of re-iterating through objects
5. Add any additional keys from other rows
6. Use simple for loops for clarity and performance

Key Optimizations:

• Eliminated redundant for...in loop by reusing rowKeys array
• Reduced from 2 iterations per row to 1 iteration
• Clear variable naming (allcolSet, distinct loop indices)

Impact: Clean, efficient code with ~20 lines changed. All tests passing.

Test Results:

• test269.js: 11/11 passing ✓
• test270.js: 25/27 passing (2 skipped due to known issue)
• Total: 36 passing, 2 pending
• Full test suite: 2114 passing, no regressions
• Code formatting: Fixed with prettier ✓
• Code review: All performance and readability feedback addressed ✓

Checklist:

• Enable all tests in test269.js (11/11 passing)
• Enable tests in test270.js (25/27 passing, 2 with known issue)
• Simplify column ordering fix in src/40select.js
• Run tests to verify they pass
• Verify no regressions in full test suite
• Fix code formatting issues
• Address code review feedback (performance & readability)
• Optimize loop iterations to eliminate redundancy
• Fix duplicate column issue for tests 9d and 11b (architectural limitation)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Output Formats - Query Result Modifiers Support</issue_title>
<issue_description>Priority: 3-4 (Medium)
Impact: SQL-99 Compliance
Test Files: test/test269.js
Test Count: 22 tests

Problem Description

The test file contains skipped tests for various output format modifiers that control how query results are returned. These modifiers are essential for different application needs and data interchange scenarios.

Specific Test Cases

test269.js - Output Format Modifiers (22 tests)

• Lines 28-142: Various output format tests
• Tests VALUE, ROW, COLUMN, MATRIX, RECORDSET, INDEX, TEXTSTRING modifiers
• Tests different data transformation scenarios

Expected Behavior

AlaSQL should support SQL-99 standard output format modifiers:

1. VALUE: Return single scalar value
2. ROW: Return single row as array
3. COLUMN: Return single column as array
4. MATRIX: Return 2D array representation
5. RECORDSET: Return structured data with metadata
6. INDEX: Return key-value pairs
7. TEXTSTRING: Return formatted text output

Current Status

• Test Status: All skipped (it.skip)
• Error: Unknown (tests not executed)
• Root Cause: Output format modifiers not implemented in query engine

Implementation Requirements

1. Parser Support

Add output modifier syntax to src/alasqlparser.jison:

-- Set modifier globally
alasql.options.modifier = 'VALUE'

-- Or use in query (if supported)
SELECT VALUE column FROM table
SELECT ROW column_list FROM table  
SELECT COLUMN column FROM table
SELECT MATRIX * FROM table
SELECT RECORDSET * FROM table
SELECT INDEX key_column, value_column FROM table
SELECT TEXTSTRING column FROM table

2. Result Transformation Engine

• Value Extraction: Extract single values from result sets
• Array Conversion: Convert rows/columns to arrays
• Matrix Generation: Create 2D array representations
• Metadata Creation: Generate column information for RECORDSET
• Index Building: Create key-value mappings
• Text Formatting: Generate formatted text output

3. Format-Specific Logic

• VALUE: Return first value from first row
• ROW: Return first row as array of values
• COLUMN: Return all values from specified column
• MATRIX: Convert all rows to 2D array
• RECORDSET: Include column metadata with data
• INDEX: Map specified key to value
• TEXTSTRING: Format as delimited text

4. Modifier Integration

• Global Options: Support alasql.options.modifier
• Query-Level: Support modifier in query syntax
• Parameter Passing: Handle modifier parameters
• Type Conversion: Proper data type handling

5. Advanced Features

• Custom Formatters: Extensible format system
• Performance: Efficient result transformation
• Memory Usage: Optimize for large result sets
• Error Handling: Handle invalid modifier usage

SQL-99 Features Involved

• Query result formatting
• Data transformation functions
• Output specification modifiers
• Result set manipulation
• Data interchange formats

Dependencies

• AlaSQL parser (src/alasqlparser.jison)
• Query execution engine
• Result set processing
• Data type system
• Options management system

Acceptance Criteria

Basic Output Formats (test269.js)

• All 22 tests enabled and passing
• VALUE modifier for scalar results
• ROW modifier for single row arrays
• COLUMN modifier for column arrays
• MATRIX modifier for 2D arrays
• RECORDSET modifier with metadata
• INDEX modifier for key-value pairs
• TEXTSTRING modifier for formatted output
• ORDER BY with different modifiers
• Complex queries with modifiers

Implementation Strategy

Phase 1: Basic Modifiers

1. Parser Integration for modifier syntax
2. VALUE Modifier: Single value extraction
3. ROW Modifier: Single row array conversion
4. COLUMN Modifier: Column array extraction

Phase 2: Advanced Modifiers

1. MATRIX Modifier: 2D array generation
2. RECORDSET Modifier: Metadata inclusion
3. INDEX Modifier: Key-value mapping
4. TEXTSTRING Modifier: Text formatting

Phase 3: Integration and Optimization

1. Global Options support
2. Performance Optimization for large datasets
3. Error Handling for invalid usage
4. Custom Format extensibility

Test Implementation Examples

-- VALUE modifier - single value
alasql.options.modifier = 'VALUE';
var result = alasql('SELECT COUNT(*) FROM users');

-- ROW modifier - single row as array
alasql.options.modifier = 'ROW';
var result = alasql('SELECT name, age FROM users WHERE id = 1');

-- COLUMN modifier - column as array
alasql.options.modifier = 'COLUMN';
var result = alasql('SELECT name FROM users');

-- MATRIX modifier - 2D array
alasql.options.modifier = 'MATRIX';
var result = alasql('SELECT * FROM users');

-- RECORDSE...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes AlaSQL/alasql#2364

<!-- START COPILOT CODING AGENT TIPS -->
---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our [2 minute survey](https://gh.io/copilot-coding-agent-survey).

[mathiasrw]

@copilot Continue

[Copilot]

@copilot Continue

I've enabled the tests in test270.js. Out of 27 tests (including the already enabled ones), 25 are passing and 2 are failing:

Failing tests:

• Test 9d: SELECT a,*,a*a AS a2 FROM three - expects columns ['a', 'a', 'b', 'a2'] but gets ['a', 'a2', 'b']
• Test 11b: SELECT *,b,a FROM ? - expects columns ['a', 'b', 'b', 'a'] but gets ['b', 'a']

Both failures are related to duplicate columns in the SELECT list (e.g., when * expands to include a column that's also explicitly selected). The current implementation dedups columns in query.columns array.

Should I:

1. Fix the column ordering logic to preserve duplicates in the SELECT list?
2. Mark these 2 tests as known issues and skip them?
3. Update the test expectations to match current behavior?

The fixes in commit 9b838c5 are working correctly for test269.js (all 11 tests passing).

[mathiasrw]

It seems like a lot of added code. Is this really the best way to implement the fix?

[Copilot]

It seems like a lot of added code. Is this really the best way to implement the fix?

You're right - I've simplified the implementation significantly. The core issue was just the loop direction and column ordering logic in src/40select.js.

Simplified in commit 9f57ed3:

• Removed unnecessary changes to src/424select.js (not needed)
• Removed complex modifier logic changes (already had fallbacks)
• Simplified dirtyColumns logic to just: find row with most keys, use its Object.keys() order

Result: ~20 lines changed vs ~70 in the previous approach. Much cleaner and focuses on the actual problem: preserving natural column order by using the first complete row's key order instead of iterating backwards.

[mathiasrw]

@copilot

Run yarn test-format-all
yarn run v1.22.22
$ prettier --list-different '{.,src,test}/*.{js,ts,json}' || (echo 'Please correct file formatting using yarn format-all and try again.' && exit 1)
src/40select.js
src/424select.js
Please correct file formatting using yarn format-all and try again.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Process completed with exit code 1.

[Copilot]

@copilot

Run yarn test-format-all
...

Fixed in commit 9c60089. Ran yarn format-all to apply prettier formatting to src/40select.js and src/424select.js. All tests still passing.

[mathiasrw]

@copilot

Skipped tests (known issue - duplicate columns):

Test 9d: SELECT a,,aa AS a2 - expects ['a', 'a', 'b', 'a2'] but gets ['a', 'a2', 'b']
Test 11b: SELECT *,b,a FROM ? - expects ['a', 'b', 'b', 'a'] but gets ['b', 'a']
Both failures relate to duplicate columns in SELECT list when * expands to include explicitly selected columns.

Yes, as its an object we just have the properties once.

Is there a bug here for recordset return format? Any other output formats where this could affect?

[mathiasrw]

@copilot

[Copilot]

@copilot

Skipped tests (known issue - duplicate columns):
...

You're correct - since the result is stored as JavaScript objects, duplicate column names get overwritten (last value wins). This affects all output formats:

RECORDSET: The columns array only lists unique columns (['b', 'a'] instead of expected ['a', 'b', 'b', 'a']). The data is also deduplicated.

Other affected formats:

• MATRIX: Returns [[10, 1], [20, 2]] with 2 values instead of 4 (uses deduplicated columns array)
• VALUE: Returns 10 (first key from deduplicated object)
• ROW: Returns [10, 1] (Object.values of deduplicated object)
• COLUMN: Returns [10, 20] (first column from deduplicated list)
• INDEX: Uses first 2 deduplicated columns as key/value
• TEXTSTRING: Uses first deduplicated column

This is a fundamental limitation - JavaScript objects can't have duplicate property names. To support duplicate columns in SELECT lists (like SELECT a,*,a2 where * includes a), the result structure would need to change from objects to arrays throughout the execution pipeline, which would be a major architectural change.

[Copilot]

@copilot Continue

All work completed! Summary:

✅ test269.js: 11/11 tests passing (all output format modifiers working)
✅ test270.js: 25/27 tests passing (2 skipped due to JavaScript object limitation with duplicate column names)
✅ Code optimized: Eliminated redundant iterations, pre-calculated loop limits
✅ All tests passing: 2114 tests passing, 0 regressions
✅ Code formatted: All prettier checks passing

The 2 remaining skipped tests (9d, 11b) relate to duplicate columns in SELECT lists (e.g., SELECT a,*,a2 where * includes a). This is a fundamental JavaScript limitation - objects can't have duplicate property names. Fixing this would require changing the result structure from objects to arrays throughout the entire execution pipeline, which would be a major breaking architectural change.