Skip to content

Conversation

Akhil-Pathivada
Copy link

Pull Request type

  • Bugfix

Changes in this PR

This PR fixes a critical bug where OpenSearch index templates were not applying field mappings correctly, resulting in indices being created with empty mappings for task_log, event, and message indices.

Root Cause

The code was using the deprecated Legacy Template API (PUT /_template/) while the template JSON files were structured in the Composable Template API format (with nested "template": {} key). The Legacy API does not recognize the nested template structure and silently ignored it, resulting in templates being created with empty mappings.

Solution

  1. Migrated to Composable Template API: Updated OpenSearchRestDAO to use /_index_template/ endpoint instead of /_template/
  2. Added template priorities: OpenSearch's Composable API requires different priorities when index patterns could potentially overlap. Added priorities (1, 2, 3) to satisfy this validation requirement
  3. Enhanced log field for full-text search: Changed log field type from keyword to text in template_task_log.json to enable full-text search capabilities on task execution logs
  4. Updated test assertions: Modified TestOpenSearchRestDAO to verify templates using the new API endpoint

Why This Approach

  • Uses recommended Composable Template API (Legacy API is deprecated)
  • Priorities satisfy OpenSearch validation for overlapping patterns
  • Text field enables full-text log search vs exact-match-only keyword type

Testing

  • Verified templates are created automatically on startup & proper field mappings are applied
  • All three templates (task_log, event, message) creating successfully with correct field types

Alternatives considered

  1. Legacy API format: Convert JSON templates to flat structure for Legacy API. Rejected because /_template/ is deprecated in OpenSearch.

  2. Specific patterns without priorities: Use patterns like *_task_log_* to avoid overlap. Rejected because OpenSearch validation still requires priorities for any potential pattern conflicts.

@Akhil-Pathivada Akhil-Pathivada changed the title fix(os-persistence): migrate index templates to Composable API with priorities fix(os-persistence): Migrate index templates to Composable API with priorities Oct 4, 2025
@Akhil-Pathivada
Copy link
Author

Akhil-Pathivada commented Oct 6, 2025

@v1r3n @manan164 can you please help in reviewing this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant