Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,50 @@ Environment variables control implementation selection:
- Integration tests can switch between real and mock implementations
- Test files follow `*_test.go` pattern alongside implementation files

<<<<<<< HEAD
## API Features

### Date Range Filtering

The query service supports filtering resources by date ranges on fields within the `data` object.

**Query Parameters:**

- `date_field` (string, optional): Date field to filter on (automatically prefixed with `"data."`)
- `date_from` (string, optional): Start date (inclusive, gte operator)
- `date_to` (string, optional): End date (inclusive, lte operator)

**Supported Date Formats:**

1. **ISO 8601 datetime**: `2025-01-10T15:30:00Z` (time used as provided)
2. **Date-only**: `2025-01-10` (converted to start/end of day UTC)
- `date_from` → `2025-01-10T00:00:00Z` (start of day)
- `date_to` → `2025-01-10T23:59:59Z` (end of day)

**Examples:**

```bash
# Date range with date-only format
GET /query/resources?v=1&date_field=updated_at&date_from=2025-01-10&date_to=2025-01-28

# Date range with ISO 8601 format
GET /query/resources?v=1&date_field=created_at&date_from=2025-01-10T15:30:00Z&date_to=2025-01-28T18:45:00Z

# Open-ended range (only start date)
GET /query/resources?v=1&date_field=created_at&date_from=2025-01-01

# Combined with other filters
GET /query/resources?v=1&type=project&tags=active&date_field=updated_at&date_from=2025-01-01&date_to=2025-03-31
```

**Implementation Details:**

- Date parsing logic: `cmd/service/converters.go` (`parseDateFilter()` function)
- Domain model: `internal/domain/model/search_criteria.go` (DateField, DateFrom, DateTo)
- OpenSearch query: `internal/infrastructure/opensearch/template.go` (range query with gte/lte)
- API design: `design/query-svc.go` (Goa design specification)
- Test coverage: `cmd/service/converters_test.go` (17 comprehensive test cases)
=======
## CEL Filter Feature

The service supports Common Expression Language (CEL) filtering for post-query resource filtering.
Expand Down Expand Up @@ -183,3 +227,4 @@ service := service.NewResourceSearch(mockSearcher, mockAccessChecker, mockFilter
### Important Limitations

**Pagination**: CEL filters apply only to results from each OpenSearch page. If the target resource is not in the first page of OpenSearch results, it won't be found even if it matches the CEL filter. Always use specific primary search criteria (`type`, `name`, `parent`) to narrow OpenSearch results first.
>>>>>>> 3e45fc4d33aba656a5abe1c3df0d3f2bd0fd6be7
49 changes: 49 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,9 +214,17 @@ Authorization: Bearer <jwt_token>
- `name`: Resource name or alias (supports typeahead search)
- `type`: Resource type to filter by
- `parent`: Parent resource for hierarchical queries
<<<<<<< HEAD
- `tags`: Array of tags to filter by (OR logic - matches resources with any of these tags)
- `tags_all`: Array of tags to filter by (AND logic - matches resources that have all of these tags)
- `date_field`: Date field to filter on (within data object) - used with date_from and/or date_to
- `date_from`: Start date (inclusive). Format: ISO 8601 datetime or date-only (YYYY-MM-DD). Date-only uses start of day UTC
- `date_to`: End date (inclusive). Format: ISO 8601 datetime or date-only (YYYY-MM-DD). Date-only uses end of day UTC
=======
- `tags`: Array of tags to filter by (OR logic)
- `tags_all`: Array of tags where all must match (AND logic)
- `cel_filter`: CEL expression for advanced post-query filtering (see [CEL Filter](#cel-filter) section)
>>>>>>> 3e45fc4d33aba656a5abe1c3df0d3f2bd0fd6be7
- `sort`: Sort order (name_asc, name_desc, updated_asc, updated_desc)
- `page_token`: Pagination token
- `v`: API version (required)
Expand All @@ -241,6 +249,46 @@ Authorization: Bearer <jwt_token>
}
```

<<<<<<< HEAD
**Date Range Filtering Examples:**

Filter resources updated between two dates (date-only format):

```bash
GET /query/resources?v=1&date_field=updated_at&date_from=2025-01-10&date_to=2025-01-28
Authorization: Bearer <jwt_token>
```

Filter resources with precise datetime filtering (ISO 8601 format):

```bash
GET /query/resources?v=1&date_field=created_at&date_from=2025-01-10T15:30:00Z&date_to=2025-01-28T18:45:00Z
Authorization: Bearer <jwt_token>
```

Filter resources created after a specific date (open-ended range):

```bash
GET /query/resources?v=1&date_field=created_at&date_from=2025-01-01
Authorization: Bearer <jwt_token>
```

Combine date filtering with other parameters:

```bash
GET /query/resources?v=1&type=project&tags=active&date_field=updated_at&date_from=2025-01-01&date_to=2025-03-31
Authorization: Bearer <jwt_token>
```

**Date Format Notes:**

- **ISO 8601 datetime format**: `2025-01-10T15:30:00Z` (time is used as provided)
- **Date-only format**: `2025-01-10` (automatically converted to start/end of day UTC)
- For `date_from`: Converts to `2025-01-10T00:00:00Z` (start of day)
- For `date_to`: Converts to `2025-01-10T23:59:59Z` (end of day)
- All dates are inclusive (uses `gte` and `lte` operators)
- The `date_field` parameter is automatically prefixed with `"data."` to scope to the resource's data object
=======
#### CEL Filter

The `cel_filter` query parameter enables advanced filtering of search results using Common Expression Language (CEL). CEL is a non-Turing complete expression language designed for safe, fast evaluation of expressions in performance-critical applications.
Expand Down Expand Up @@ -353,6 +401,7 @@ Invalid CEL expressions return a 400 Bad Request with details:
"error": "filter expression failed: ERROR: <input>:1:6: Syntax error: mismatched input 'invalid' expecting {'[', '{', '(', '.', '-', '!', 'true', 'false', 'null', NUM_FLOAT, NUM_INT, NUM_UINT, STRING, BYTES, IDENTIFIER}"
}
```
>>>>>>> 3e45fc4d33aba656a5abe1c3df0d3f2bd0fd6be7

#### Organization Search API

Expand Down
131 changes: 131 additions & 0 deletions cmd/service/converters.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (
"context"
"fmt"
"log/slog"
"time"
"strings"

querysvc "github.com/linuxfoundation/lfx-v2-query-service/gen/query_svc"
Expand All @@ -16,6 +17,42 @@ import (
"github.com/linuxfoundation/lfx-v2-query-service/pkg/paging"
)

// parseDateFilter parses a date string in ISO 8601 datetime or date-only format
// and returns it normalized for OpenSearch range queries.
// Date-only format (YYYY-MM-DD) is converted to:
// - Start of day (00:00:00 UTC) for date_from
// - End of day (23:59:59 UTC) for date_to
func parseDateFilter(dateStr string, isEndDate bool) (string, error) {
if dateStr == "" {
return "", nil
}

// Try parsing as ISO 8601 datetime first (e.g., 2025-01-10T15:30:00Z)
t, err := time.Parse(time.RFC3339, dateStr)
if err == nil {
// Already in datetime format, return as-is
return t.Format(time.RFC3339), nil
}

// Try parsing as date-only (e.g., 2025-01-10)
t, err = time.Parse("2006-01-02", dateStr)
if err != nil {
return "", fmt.Errorf("invalid date format '%s': must be ISO 8601 datetime (2006-01-02T15:04:05Z) or date-only (2006-01-02)", dateStr)
}

// Convert date-only to datetime
if isEndDate {
// For end dates, use end of day (23:59:59 UTC)
// Note: Using 23:59:59 instead of 23:59:59.999 for simplicity and OpenSearch compatibility
t = time.Date(t.Year(), t.Month(), t.Day(), 23, 59, 59, 0, time.UTC)
} else {
// For start dates, use start of day (00:00:00 UTC)
t = time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, time.UTC)
}

return t.Format(time.RFC3339), nil
}

// parseFilters parses filter strings in "field:value" format
// All fields are automatically prefixed with "data." to filter only within the data object
func parseFilters(filters []string) ([]model.FieldFilter, error) {
Expand Down Expand Up @@ -91,6 +128,40 @@ func (s *querySvcsrvc) payloadToCriteria(ctx context.Context, p *querysvc.QueryR
)
}

// Validate date filtering parameters
if (p.DateFrom != nil || p.DateTo != nil) && p.DateField == nil {
err := fmt.Errorf("date_field is required when using date_from or date_to")
slog.ErrorContext(ctx, "invalid date filter parameters", "error", err)
return criteria, wrapError(ctx, err)
}

// Handle date filtering parameters
if p.DateField != nil {
// Auto-prefix with "data." to scope to data object
prefixedField := "data." + *p.DateField
criteria.DateField = &prefixedField

// Parse and normalize date_from
if p.DateFrom != nil {
normalizedFrom, err := parseDateFilter(*p.DateFrom, false)
if err != nil {
slog.ErrorContext(ctx, "invalid date_from format", "error", err, "date_from", *p.DateFrom)
return criteria, wrapError(ctx, err)
}
criteria.DateFrom = &normalizedFrom
}

// Parse and normalize date_to
if p.DateTo != nil {
normalizedTo, err := parseDateFilter(*p.DateTo, true)
if err != nil {
slog.ErrorContext(ctx, "invalid date_to format", "error", err, "date_to", *p.DateTo)
return criteria, wrapError(ctx, err)
}
criteria.DateTo = &normalizedTo
}
}

return criteria, nil
}

Expand Down Expand Up @@ -146,6 +217,36 @@ func (s *querySvcsrvc) payloadToCountPublicCriteria(payload *querysvc.QueryResou
criteria.ParentRef = payload.Parent
}

// Validate date filtering parameters
if (payload.DateFrom != nil || payload.DateTo != nil) && payload.DateField == nil {
return criteria, fmt.Errorf("date_field is required when using date_from or date_to")
}

// Handle date filtering parameters
if payload.DateField != nil {
// Auto-prefix with "data." to scope to data object
prefixedField := "data." + *payload.DateField
criteria.DateField = &prefixedField

// Parse and normalize date_from
if payload.DateFrom != nil {
normalizedFrom, err := parseDateFilter(*payload.DateFrom, false)
if err != nil {
return criteria, fmt.Errorf("invalid date_from: %w", err)
}
criteria.DateFrom = &normalizedFrom
}

// Parse and normalize date_to
if payload.DateTo != nil {
normalizedTo, err := parseDateFilter(*payload.DateTo, true)
if err != nil {
return criteria, fmt.Errorf("invalid date_to: %w", err)
}
criteria.DateTo = &normalizedTo
}
}

return criteria, nil
}

Expand Down Expand Up @@ -182,6 +283,36 @@ func (s *querySvcsrvc) payloadToCountAggregationCriteria(payload *querysvc.Query
criteria.ParentRef = payload.Parent
}

// Validate date filtering parameters
if (payload.DateFrom != nil || payload.DateTo != nil) && payload.DateField == nil {
return criteria, fmt.Errorf("date_field is required when using date_from or date_to")
}

// Handle date filtering parameters
if payload.DateField != nil {
// Auto-prefix with "data." to scope to data object
prefixedField := "data." + *payload.DateField
criteria.DateField = &prefixedField

// Parse and normalize date_from
if payload.DateFrom != nil {
normalizedFrom, err := parseDateFilter(*payload.DateFrom, false)
if err != nil {
return criteria, fmt.Errorf("invalid date_from: %w", err)
}
criteria.DateFrom = &normalizedFrom
}

// Parse and normalize date_to
if payload.DateTo != nil {
normalizedTo, err := parseDateFilter(*payload.DateTo, true)
if err != nil {
return criteria, fmt.Errorf("invalid date_to: %w", err)
}
criteria.DateTo = &normalizedTo
}
}

return criteria, nil
}

Expand Down
Loading