[FEATURE] Refactor/Enhance agentic memory

[ylwu-amzn]

Description

Agentic Memory was released as an experimental feature in OpenSearch 3.2. To make it production-ready, we need several key enhancements focusing on security, manageability, and scalability.

Proposed Enhancements:

1. Add Session Management

• Automatic session creation when session ID is not provided
• Session summary generation capability
• Session lifecycle management

2. Memory Storage Architecture

• Separate storage for short-term and long-term memory
• Optimized data organization for better performance and flexibility(like support multiple LLMs, multi-modal)

3. Long-term Memory Strategy

• Configurable long-term strategies in model container
• User could customize strategies

4. Security Enhancements

• Memories data will be stored in system index by default for enhanced access control
• Implement user-level access control for memory data
• Support backend role-based access control for memory containers
• Enable secure sharing of containers among multiple users

5. API Improvements

• Enhanced search capabilities for memories data
• Batch deletion support for memories
• Cascade deletion when removing memory containers
• Build APIs for get/update/delete memory data