NAVIGATABLE_RECORD_STREAM.md

NavigatableRecordStream - Implementation Summary

Overview

NavigatableRecordStream is a new class that provides in-memory sorting and pagination capabilities for any RecordStream. It loads records into memory and allows dynamic re-sorting and flexible pagination without requiring additional database queries.

Key Features

1. In-Memory Data Management

• Loads all records from a RecordStream into memory
• Optional limit parameter to control memory usage
• Records stored in an array for fast access and sorting

2. Multi-Column Sorting

• Sort by multiple fields with different directions (ascending/descending)
• Case-sensitive and case-insensitive sorting options
• Dynamic re-sorting without re-querying the database
• Sorting API replaces criteria on each call:
  • Single field: sortBy(String field, SortDir, boolean caseSensitive)
  • Multiple fields: sortBy(List<SortProperties>)
  • Use static factory methods: SortProperties.ascending(), SortProperties.descending(), etc.

3. Flexible Pagination

• Configurable page size
• Forward pagination with hasMorePages()
• Jump to specific pages with setPageTo()
• Reset iteration with reset()
• Navigate backward by jumping to earlier pages

4. Integration with Existing APIs

• Implements ResettablePagination interface for page navigation
• Implements Closeable interface for resource management
• Provides familiar methods like toObjectList(), stream(), forEach()
• Uses SortProperties record with convenient static factory methods

Files Created/Modified

New Files

1. src/main/java/com/aerospike/NavigatableRecordStream.java (464 lines)

   • Main implementation class
   • Comprehensive JavaDoc documentation
   • Builder-style API for sorting and pagination

2. src/main/java/com/example/NavigatableRecordStreamExample.java (477 lines)

   • 6 comprehensive examples demonstrating all features
   • Extensive inline documentation
   • Ready-to-run demonstration code

Modified Files

1. src/main/java/com/aerospike/RecordStream.java
   • Added asNavigatableStream() method
   • Added asNavigatableStream(long limit) method
   • Complete JavaDoc documentation with examples

API Reference

Creating a NavigatableRecordStream

// From any RecordStream
RecordStream results = session.query(dataSet).execute();

// Load all records
NavigatableRecordStream nav = results.asNavigatableStream();

// Load with limit (recommended for large datasets)
NavigatableRecordStream nav = results.asNavigatableStream(1000);

Sorting

// Single field sorting (replaces any existing sort)
nav.sortBy("name");                                    // Ascending, case-sensitive
nav.sortBy("age", SortDir.SORT_DESC);                  // Descending
nav.sortBy("name", SortDir.SORT_ASC, false);          // Ascending, case-insensitive

// Multi-column sorting with static factory methods
nav.sortBy(List.of(
    SortProperties.ascending("name"),
    SortProperties.descending("age")
));

// Case-insensitive sorting
nav.sortBy(List.of(
    SortProperties.ascendingIgnoreCase("lastName"),
    SortProperties.ascendingIgnoreCase("firstName")
));

Pagination

// Set page size
nav.pageSize(20);

// Forward pagination
while (nav.hasMorePages()) {
    while (nav.hasNext()) {
        RecordResult record = nav.next();
        // Process record
    }
}

// Jump to specific page
nav.setPageTo(3);  // Jump to page 3

// Get current page info
int current = nav.currentPage();  // 1-based
int total = nav.maxPages();

// Reset to beginning
nav.reset();

Data Access

// Convert current page to objects
List<Customer> customers = nav.toObjectList(mapper);

// Stream current page
Stream<RecordResult> stream = nav.stream();

// Iterate current page
nav.forEach(record -> { /* process */ });

// Get first record
Optional<RecordResult> first = nav.getFirst();

// Get total record count
int size = nav.size();

Sorting Behavior Details

Important: Sort Replacement

Each call to sortBy() replaces the entire sort criteria. This makes it predictable and easy to understand:

// Single field - sorts by age only
nav.sortBy("age", SortDir.SORT_DESC);

// Multi-field - sorts by lastName, then firstName, then age
nav.sortBy(List.of(
    SortProperties.ascending("lastName"),
    SortProperties.ascending("firstName"),
    SortProperties.descending("age")
));

// Change sort - now sorts only by name
nav.sortBy("name");

Sort Order Precedence

For multi-field sorting, sorts are applied in list order:

1. First item in list = primary sort
2. Second item in list = secondary sort
3. Third item in list = tertiary sort
4. And so on...

Example Scenarios

Scenario 1: User-Driven Sort

// Initial query
RecordStream results = session.query(customerDataSet).limit(500).execute();
NavigatableRecordStream nav = results.asNavigatableStream()
    .pageSize(25)
    .sortBy("name");

// User changes sort order (no database query!)
nav.sortBy("age", SortDir.SORT_DESC);
nav.reset();  // Start from beginning with new sort

Scenario 2: Compare Multiple Sort Orders

RecordStream results = session.query(customerDataSet).execute();
NavigatableRecordStream nav = results.asNavigatableStream().pageSize(20);

// View sorted by name
nav.sortBy("name");
processFirstPage(nav);

// Compare with sort by age (no re-query!)
nav.sortBy("age", SortDir.SORT_DESC);
nav.reset();
processFirstPage(nav);

// Compare with multi-column sort by name then age
nav.sortBy(List.of(
    SortProperties.ascending("name"),
    SortProperties.descending("age")
));
nav.reset();
processFirstPage(nav);

Scenario 3: Large Dataset with Memory Control

// Query might return millions of records, but only load top 1000
RecordStream results = session.query(dataSet)
    .where("$.active == true")
    .execute();

NavigatableRecordStream nav = results.asNavigatableStream(1000)  // Limit memory
    .pageSize(50)
    .sortBy("lastModified", SortDir.SORT_DESC);

// Process only recent records with controlled memory usage
while (nav.hasMorePages()) {
    processPage(nav);
}

Performance Considerations

Memory Usage

• All records are stored in memory as RecordResult[] array
• Memory usage = (record count) × (record size)
• Recommendation: Use asNavigatableStream(limit) for large datasets
• Best practice: Set a reasonable limit (e.g., 1000-10000 records)

When to Use NavigatableRecordStream

Use NavigatableRecordStream when:

• Sort criteria determined by user input after initial query
• Need to view data in multiple sort orders
• Want to paginate backward or jump to specific pages
• Database queries are expensive
• Working with a bounded result set that fits in memory

Don't use NavigatableRecordStream when:

• Working with very large datasets that won't fit in memory
• Only need forward-only iteration (use ChunkedRecordStream via chunkSize())
• Memory is constrained
• Processing billions of records

Performance Characteristics

• Initial load: O(n) - reads all records from source stream
• Sorting: O(n log n) - uses Java's Arrays.sort()
• Re-sorting: O(n log n) - no database query
• Page jumping: O(1) - direct array access
• Iteration: O(1) per record

Comparison: Server-Side Chunking vs Client-Side Sorting

Server-Side Chunking (ChunkedRecordStream)

// For processing huge datasets - billions of records possible
RecordStream results = session.query(customerDataSet)
    .chunkSize(5000)  // Fetch 5000 records per server call
    .execute();

// Process in chunks
while (results.hasMoreChunks()) {
    while (results.hasNext()) {
        processRecord(results.next());
    }
}

Pros:

• Works with unlimited dataset sizes
• Very memory efficient
• Forward-only streaming

Cons:

• No sorting capability
• No backward navigation
• Cannot jump to specific positions

Client-Side Sorting/Pagination (NavigatableRecordStream)

// For datasets that fit in memory
RecordStream results = session.query(customerDataSet)
    .limit(1000)
    .execute();

NavigatableRecordStream nav = results.asNavigatableStream()
    .pageSize(20)
    .sortBy("age", SortDir.SORT_DESC);

// Later, change sort without re-querying
nav.sortBy("name");

Pros:

• Dynamic re-sorting without database queries
• Can change sort order multiple times
• Forward and backward navigation
• Jump to any page
• Re-iterate with different sorts

Cons:

• Loads all records into memory
• Not suitable for very large datasets without limit

Testing the Examples

To run the comprehensive examples:

# Make sure Aerospike is running on localhost:3100
# with credentials admin/password123

cd /Users/tfaulkes/Programming/Aerospike/git/aerospike-fluent-client-java
javac -d bin src/main/java/com/example/NavigatableRecordStreamExample.java
java -cp bin com.example.NavigatableRecordStreamExample

The example will:

1. Create 30 test customer records
2. Demonstrate 6 different usage scenarios
3. Show forward/backward pagination
4. Show multi-column sorting
5. Show dynamic re-sorting
6. Clean up test data

Implementation Notes

Thread Safety

NavigatableRecordStream is not thread-safe. If multiple threads need to access the same instance, external synchronization is required. However, each thread can safely have its own instance reading from the same data.

Closing Resources

NavigatableRecordStream implements Closeable but doesn't hold any resources that need cleanup (data is already in memory). The close() method is a no-op but is provided for consistency with RecordStream.

Integration with Existing Code

The implementation is designed to integrate seamlessly with existing code:

• Uses existing RecordComparator for sorting
• Uses SortProperties record with convenient static factory methods
• Implements existing ResettablePagination interface
• Follows familiar patterns for pagination and sorting

Future Enhancements (Optional)

Potential future improvements could include:

1. Lazy sorting (sort only when needed)
2. Incremental loading (load records on demand)
3. Filtering support (similar to failures() method)
4. Statistics (min, max, average, etc.)
5. Custom comparators for complex sorting logic