diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/stumper-candidates.md b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/stumper-candidates.md new file mode 100644 index 0000000000..692c64a891 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/stumper-candidates.md @@ -0,0 +1,149 @@ +# Stumper Candidates: Solr→OpenSearch Migration + +20 questions designed to stump a bare LLM (no steering/reference content). +Use these to manually test bare Haiku from `/tmp`, then craft assertions +and steering/reference content for the ones that genuinely fail. + +Test with: `cd /tmp && claude --model claude-haiku-4-5-20251001 --print "QUESTION"` + +--- + +## Querqy / SMUI (niche o19s tools — highest stumper confidence) + +### 1. Querqy OpenSearch class names +**Question:** What is the exact fully-qualified Java class name for registering a CommonRules rewriter in the OpenSearch Querqy plugin REST API? Give just the class string. +**Challenge:** OpenSearch Querqy uses different package prefix and Factory suffix vs Solr. LLMs guess the Solr class name or refuse. +**Criterion A:** `icontains: "CommonRulesRewriterFactory"` +**Criterion B:** `icontains: "querqy.opensearch.rewriter"` + +### 2. Querqy infoLogging field names +**Question:** In the OpenSearch Querqy plugin, what is the exact JSON field name you add inside a querqy query clause to enable debug logging of which rules fired? And what is the exact field name in the response that contains the decoration/rule-match data? Give just the two field names. +**Challenge:** Bare LLMs refuse to guess or give wrong field names. This is barely documented. +**Criterion A:** `contains-any: ["info_logging", "infoLogging"]` +**Criterion B:** `contains-any: ["querqy_decorations", "decorations"]` + +### 3. Querqy rewriter chain order mechanism +**Question:** In Solr, Querqy rewriter chain order is defined in querqy.xml. In the OpenSearch Querqy plugin, where is the chain order defined? Is it in a config file, the cluster state, or per-query? Give the exact mechanism. +**Challenge:** The answer is per-query (in the `rewriters` array). LLMs often guess config file or cluster state. +**Criterion A:** `contains-any: ["per-query", "per query", "query time", "in the query"]` +**Criterion B:** `contains-any: ["rewriters array", "rewriters list"]` + +### 4. SMUI deployment mechanism change +**Question:** SMUI deploys Querqy rules to Solr by copying a rules text file to Solr's config directory and triggering a config reload. When targeting OpenSearch instead, what specific mechanism replaces the file-copy deployment? Name the exact API endpoint. +**Challenge:** LLMs know the concept but may not know the exact endpoint `/_plugins/_querqy/rewriter/`. +**Criterion A:** `contains-any: ["_querqy/rewriter", "_plugins/_querqy"]` +**Criterion B:** `icontains: "PUT"` + +### 5. SMUI v4.0.11 OpenSearch scope (labelling vs deployment) +**Question:** SMUI v4.0.11 added OpenSearch support according to the release notes. As a migration consultant, what specifically did v4.0.11 add — full OpenSearch deployment parity with Solr, or something narrower? What does this mean for a team planning to use SMUI as their rules-management UI on OpenSearch? +**Challenge:** Original "minimum version" framing was based on a guess (3.14) that turned out to be wrong. Audit of querqy/smui releases (2026-05-10) shows v4.0.11 (Mar 2024) is the only release mentioning OpenSearch, and PR #139 only changed UI labels — not deployment behavior. The real consultant insight is that SMUI lacks first-class OpenSearch deployment parity. +**Criterion A:** `contains-any: ["label", "labelling", "labeling", "UI", "front-end", "frontend", "button", "cosmetic"]` (must identify the change as UI-level) +**Criterion B:** `contains-any: ["not", "narrow", "limited", "no deployment", "doesn't deploy", "does not deploy", "manual", "still", "additional", "outside"]` (must flag the deployment gap) + +### 6. Querqy DOWN weight → negative_boost conversion +**Question:** In Querqy, a DOWN(500) rule penalizes matches. To translate this to OpenSearch's boosting query with negative_boost, what is the exact formula to convert the DOWN weight to a negative_boost value? Show the formula and the result for DOWN(500). +**Challenge:** LLMs guess at formulas or refuse. The conversion is `1/(1+weight)` = 0.001998. +**Criterion A:** `icontains: "negative_boost"` +**Criterion B:** `contains-any: ["0.002", "0.001", "1/(1+"]` (approximate result) + +### 7. SMUI rule export format +**Question:** What is the format of a SMUI rules export file? Is it JSON, XML, or something else? Show an example of what the export looks like for a term with SYNONYM and UP rules. +**Challenge:** LLMs fabricate JSON formats. SMUI exports Querqy Common Rules plain text with `=>` syntax. +**Criterion A:** `contains-any: ["plain text", "Common Rules", "text format"]` +**Criterion B:** `icontains: "=>"` + +## Schema and Field Migration (subtle behavioral differences) + +### 8. copyField → multi-fields vs copy_to +**Question:** In our TMDB Solr schema, we have copyField rules that copy title to title_en (English analyzer) and title_bidirect_syn (bidirectional synonym analyzer). How should we represent these in OpenSearch? Should we use copy_to or multi-fields? Explain why. +**Challenge:** LLMs often recommend copy_to, but copy_to doesn't re-analyze with a different analyzer chain. Multi-fields is correct. +**Criterion A:** `contains-any: ["Use multi-fields", "use multi-fields", "multi-fields"]` +**Criterion B:** `contains-any: ["copy_to does not re-analyze", "copy_to copies raw", "not copy_to"]` + +### 9. positionIncrementGap default difference +**Question:** Solr's default positionIncrementGap is 0. What is OpenSearch's default position_increment_gap value? A client migrating from Solr has multi-valued text fields where they INTENTIONALLY allow phrase matches across value boundaries (positionIncrementGap=0). What specific OpenSearch mapping change preserves this behavior? +**Challenge:** The default difference (Solr 0 vs OpenSearch 100) means silently broken phrase matching after migration. Most guides miss this. +**Criterion A:** `contains-any: ["100", "default is 100", "defaults to 100"]` +**Criterion B:** `icontains: "position_increment_gap"` + +### 10. Dynamic field mapping explosion limit +**Question:** Our Solr schema uses 800+ dynamic fields (*_s, *_i, *_txt, *_dt). What specific OpenSearch setting will we hit, what is its default value, and what are the risks of increasing it? +**Challenge:** LLMs may know the setting name but not the default (1000) or the risks (cluster instability, slow mapping updates). +**Criterion A:** `contains-any: ["total_fields.limit", "total_fields"]` +**Criterion B:** `contains-any: ["1000", "1,000"]` + +## Query Translation (where "close enough" breaks things) + +### 11. LocalParams replacement architecture +**Question:** We use Solr LocalParams extensively: {!boost b=recip(ms(NOW,date),3.16e-11,1,1) v=$qq} and {!type=edismax qf='title^2 body' v=$qq}. OpenSearch has no LocalParams. Give the specific architectural pattern we should use to replace this — not just "rewrite as Query DSL". +**Challenge:** LLMs give generic "rewrite" advice. The specific pattern is search templates with mustache variables + application-layer query assembly. +**Criterion A:** `contains-any: ["search template", "search_template", "mustache"]` +**Criterion B:** `contains-any: ["application layer", "application-layer", "query builder"]` + +### 12. Search Pipelines for conditional query rewrite +**Question:** Querqy's DELETE rule conditionally removes terms from a query (e.g., 'cheap laptop => DELETE: cheap'). Without the Querqy plugin in OpenSearch, name the specific OpenSearch 2.x feature that can intercept and modify queries before execution. What is it called and when was it introduced? +**Challenge:** LLMs often don't know Search Pipelines by name or give wrong introduction dates. +**Criterion A:** `contains-any: ["search pipeline", "Search Pipeline"]` +**Criterion B:** `contains-any: ["request processor", "script processor"]` + +### 13. bf additive vs boost multiplicative +**Question:** In Solr eDisMax, what is the exact difference between bf and boost in terms of how they combine with the relevance score? For each, give the exact OpenSearch function_score boost_mode value. +**Challenge:** LLMs conflate these. bf is additive (boost_mode: sum), boost is multiplicative (boost_mode: multiply). +**Criterion A:** `contains-any: ["additive", "adds to"]` +**Criterion B:** `contains-any: ["multiplicative", "multiplies", "multiply"]` + +### 14. TF-IDF to BM25 on short fields +**Question:** When migrating from Solr 6.x (ClassicSimilarity/TF-IDF) to OpenSearch (BM25), what specific ranking behavior changes on short text fields like product titles? Name the exact BM25 parameter that controls term frequency saturation and what value approximates TF-IDF behavior. +**Challenge:** The parameter is k1 (default 1.2). Setting k1 very high (~100) approximates TF-IDF's linear TF behavior. +**Criterion A:** `icontains: "k1"` +**Criterion B:** `contains-any: ["saturation", "term frequency"]` + +## Operational / AWS Managed Service + +### 15. Querqy NOT on AWS OpenSearch Service +**Question:** We use Querqy in Solr for query rewriting. We're migrating to AWS OpenSearch Service (the managed service). Can we install the Querqy plugin? If not, what must we do instead? +**Challenge:** LLMs sometimes say yes. AWS does not allow custom plugin installation. +**Criterion A:** `contains-any: ["cannot", "can't", "not available", "not supported", "No."]` +**Criterion B:** `contains-any: ["native", "bool", "should", "synonym"]` (native replacement approach) + +### 16. Collection aliases write routing +**Question:** Our Solr uses collection aliases that route writes to the latest time-partitioned collection while searching across all. OpenSearch index aliases can search multiple indices but can only write to one. How do we handle our rolling-window architecture? +**Challenge:** The answer involves ISM (Index State Management) policies with rollover action. LLMs may give generic alias advice. +**Criterion A:** `contains-any: ["ISM", "Index State Management", "rollover"]` +**Criterion B:** `icontains: "is_write_index"` + +### 17. Atomic updates without _source +**Question:** Our Solr index has stored=false on most fields but supports atomic updates via docValues. We're migrating to OpenSearch. What specific problem will we hit with partial updates? +**Challenge:** OpenSearch requires _source for partial updates. No _source = no partial updates. Must re-architect. +**Criterion A:** `icontains: "_source"` +**Criterion B:** `contains-any: ["cannot", "not possible", "requires", "must"]` + +## Consultant-Grade Judgment (from stump-the-chumps) + +### 18. Lucene-underneath fallacy +**Question:** A client says "It's all Lucene underneath, so most of it ports directly." As a migration consultant, give your response. Be specific about where this assumption breaks. +**Challenge:** LLMs give decent general answers but may not name all key divergence areas (query parsers, scoring defaults, operational model). +**Criterion A:** `contains-any: ["query parser", "parser"]` +**Criterion B:** `contains-any: ["BM25", "scoring", "similarity"]` + +### 19. Mechanical translation fantasy +**Question:** A client says their Solr-to-OpenSearch migration is "mostly config conversion work." Name 3 specific areas that require genuine redesign, not just translation. +**Challenge:** Must name specific areas (not just "relevance" but e.g., "analyzer behavior audit", "custom handler decomposition", "aggregation semantics"). +**Criterion A:** `contains-any: ["redesign", "re-design", "rethink"]` +**Criterion B:** `contains-any: ["analyzer", "handler", "scoring", "relevance"]` + +### 20. Plugin/custom handler dependency audit +**Question:** A client says "We didn't think any custom Solr components were important." What specific component types should be inventoried, and why is this claim dangerous? +**Challenge:** Must insist on systematic inventory of request handlers, update processors, search components, custom query parsers. +**Criterion A:** `contains-any: ["request handler", "RequestHandler"]` +**Criterion B:** `contains-any: ["update processor", "UpdateProcessor", "search component", "SearchComponent"]` + +--- + +## Usage + +1. Test each question bare: `cd /tmp && claude --model claude-haiku-4-5-20251001 --print "QUESTION"` +2. Mark which ones genuinely fail (LLM says "I don't know", gives wrong answer, or misses the key point) +3. For each fail, write steering + reference content that would make it pass +4. Add to `eval-guidance-impact.yaml` with the criteria above (adjust based on what you see) +5. Re-test to confirm the red→green flip diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/.gitignore b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/.gitignore new file mode 100644 index 0000000000..60f020a4eb --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/.gitignore @@ -0,0 +1,10 @@ +# Cache is addressable by hash(model + prompt) — not useful for human review. +cache/ + +# Run logs and JSON exports are regenerable from sweep.py + responses/. +*.log +*.json + +# Eval-guidance-impact run artifacts (regenerable via promptfoo). +guidance-impact* +qwen-q12-q16* diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q01-querqy-class.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q01-querqy-class.haiku.txt new file mode 100644 index 0000000000..ffda0b2c47 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q01-querqy-class.haiku.txt @@ -0,0 +1,6 @@ +I don't have reliable information about this specific class name in my training data. To get the exact fully-qualified Java class name for registering a CommonRules rewriter in the OpenSearch Querqy plugin REST API, I would need to either: + +1. Search online (requires your permission to use WebSearch) +2. Look at the OpenSearch Querqy plugin source code directly + +Do you have access to the plugin source code or documentation, or would you like me to search for this information if you grant WebSearch permission? \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q01-querqy-class.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q01-querqy-class.qwen.txt new file mode 100644 index 0000000000..baab735790 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q01-querqy-class.qwen.txt @@ -0,0 +1 @@ +`org.opensearch.queries.rewrite.commonrules.CommonRulesRewriter` \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q02-querqy-infologging.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q02-querqy-infologging.haiku.txt new file mode 100644 index 0000000000..11fd1b8b0e --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q02-querqy-infologging.haiku.txt @@ -0,0 +1,6 @@ +I don't have permission to search the web right now. Based on my training knowledge of Querqy, the fields are likely: + +1. **Query parameter**: `debug` (set to `true`) +2. **Response field**: `_querqy` + +However, I'm not entirely confident about these exact names without checking the official Querqy documentation. Could you grant WebSearch permission, or if you have access to the Querqy docs or plugin source code, I can verify the exact field names for you? \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q02-querqy-infologging.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q02-querqy-infologging.qwen.txt new file mode 100644 index 0000000000..ebf558637b --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q02-querqy-infologging.qwen.txt @@ -0,0 +1,11 @@ +To enable debug logging of which rules fired, you would add the following JSON field name inside a Querqy query clause: + +```json +"querqy.debug": true +``` + +The exact field name in the response that contains the decoration/rule-match data is: + +```json +"_querqy" +``` \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q03-querqy-chain-order.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q03-querqy-chain-order.haiku.txt new file mode 100644 index 0000000000..c5420335c5 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q03-querqy-chain-order.haiku.txt @@ -0,0 +1,14 @@ +I don't have reliable information about the exact mechanism for OpenSearch Querqy plugin's rewriter chain order configuration in my training data. This is a specialized plugin configuration detail that I can't confidently answer without current documentation. + +To get an accurate answer, I'd recommend checking: + +1. **OpenSearch Querqy Plugin Documentation** — the official plugin README or docs should specify whether chain order is defined via: + - A configuration file (and its location/format) + - Cluster state/index settings + - Per-query parameters + +2. **The plugin's GitHub repository** — often the best source for configuration details and examples + +3. **OpenSearch forums or documentation site** — they may have deployment guides + +If you have a link to the plugin or documentation, I can read it and give you the exact mechanism. Alternatively, if you're working with a specific OpenSearch Querqy plugin version, sharing that context would help. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q03-querqy-chain-order.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q03-querqy-chain-order.qwen.txt new file mode 100644 index 0000000000..9604bc272c --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q03-querqy-chain-order.qwen.txt @@ -0,0 +1,33 @@ +In the OpenSearch Querqy plugin, the chain order for the rewriter rules is typically defined in a configuration file similar to how it's done in Solr using `querqy.xml`. Specifically, this configuration is usually found in an OpenSearch plugin-specific configuration directory. + +Here’s a breakdown of where and how you can define the chain order: + +1. **Configuration File**: The rewriter chain order for Querqy in OpenSearch is often defined in a JSON or XML configuration file that is part of the Querqy plugin setup. This file might be named something like `querqy.yml` or `querqy.json`, depending on the version and setup of your OpenSearch cluster. + +2. **Location**: The exact location can vary, but it's commonly found in: + - `/opensearch/plugins/querqy/config/querqy.yml` + - `/opensearch/plugins/querqy/config/querqy.json` + +3. **Cluster State vs Per-Query**: + - **Cluster State**: The cluster state typically does not directly define the rewriter chain order for Querqy. It is more about the overall health and configuration of the cluster. + - **Per-Query**: While you can dynamically modify the rewriter chain per query using specific API calls or configurations, this is less common and usually done through predefined settings in the configuration file. + +4. **Mechanism**: + - The configuration file defines a list of rules that are applied in a specified order. + - Each rule in the chain can be configured with its own parameters and conditions. + - When a query comes in, Querqy processes it according to the defined rewriter chain, applying each rule in sequence until one of them returns a result or all rules have been processed. + +Here is an example snippet from a `querqy.yml` file that defines a rewriter chain: + +```yaml +querqy: + rewriters: + - name: "exampleRewriter1" + class: "org.apache.solr.rewrite.example.ExampleRewriteRule" + - name: "exampleRewriter2" + class: "org.apache.solr.rewrite.example.AnotherExampleRewriteRule" +``` + +In this example, `exampleRewriter1` is applied first, followed by `exampleRewriter2`. + +To summarize, the chain order for Querqy rewriters in OpenSearch is defined in a configuration file (like `querqy.yml` or `querqy.json`), not directly in the cluster state or per-query. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q04-smui-deploy.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q04-smui-deploy.haiku.txt new file mode 100644 index 0000000000..7c33124372 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q04-smui-deploy.haiku.txt @@ -0,0 +1,9 @@ +I don't have built-in knowledge of SMUI's specific OpenSearch deployment mechanism or the exact API endpoint it uses. This is detailed internal implementation knowledge that would require me to either search the SMUI documentation or examine the source code. + +I can help you find this in a few ways: + +1. **Search SMUI documentation** — if you have a link or know where it's hosted, I can fetch it +2. **Check the codebase** — if SMUI source is available locally or you can point me to the repo, I can grep for OpenSearch-related deployment logic +3. **Web search** — I can search for SMUI + OpenSearch integration documentation + +Do you have access to the SMUI source code or documentation in your current project, or should I search the web for the OpenSearch API endpoint SMUI uses for rule deployment? \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q04-smui-deploy.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q04-smui-deploy.qwen.txt new file mode 100644 index 0000000000..65fb71538f --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q04-smui-deploy.qwen.txt @@ -0,0 +1,41 @@ +When deploying Querqy rules in an environment that uses OpenSearch instead of Solr, you would typically use the OpenSearch API to manage configurations rather than copying files manually. Specifically, you can use the `PUT` request to update the configuration settings for your index. + +The exact API endpoint you would use is: + +``` +PUT //_settings +``` + +Here’s a more detailed example of how you might set this up using the OpenSearch REST API: + +1. **Prepare Your Querqy Rules**: Ensure that your Querqy rules are formatted correctly and stored in a JSON or YAML file. + +2. **Update Index Settings via API**: + You can use tools like `curl`, Postman, or any HTTP client to send the request. Here’s an example using `curl`: + + ```sh + curl -X PUT "http://localhost:9200//_settings" -H 'Content-Type: application/json' -d' + { + "analysis": { + "filter": { + "querqy_rules": { + "type": "querqy", + "rules_file": "" + } + }, + "analyzer": { + "querqy_analyzer": { + "type": "custom", + "tokenizer": "standard", + "filter": ["lowercase", "querqy_rules"] + } + } + } + }' + ``` + +3. **Replace `` and ``**: Replace these placeholders with the actual index name and path to your rules file. + +4. **Reload Configuration** (if necessary): Depending on how OpenSearch is configured, you might need to trigger a configuration reload or restart the service for changes to take effect. + +By using this API endpoint, you can programmatically manage Querqy configurations in OpenSearch without manually copying files, making your deployment process more automated and reliable. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q05-smui-min-version.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q05-smui-min-version.haiku.txt new file mode 100644 index 0000000000..078e37ec0e --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q05-smui-min-version.haiku.txt @@ -0,0 +1 @@ +I don't have that specific version information. Could you share where you found this requirement (documentation link, ticket, etc.) so I can help you locate the exact version number? Or if you have access to SMUI release notes or documentation, that would tell us the version that introduced OpenSearch support. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q05-smui-min-version.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q05-smui-min-version.qwen.txt new file mode 100644 index 0000000000..cc6612c36e --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q05-smui-min-version.qwen.txt @@ -0,0 +1 @@ +2.3.0 \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q05-smui-opensearch-scope.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q05-smui-opensearch-scope.haiku.txt new file mode 100644 index 0000000000..bddcf31f07 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q05-smui-opensearch-scope.haiku.txt @@ -0,0 +1,12 @@ +I don't have reliable details about SMUI v4.0.11's specific OpenSearch additions in my training data. To give you accurate migration guidance, I'd need to see the actual release notes or documentation you're referencing. + +Can you share: +1. **The release notes link or the specific changes listed** for v4.0.11? +2. **What context triggered this (GitHub release, official docs, vendor announcement)?** + +With that, I can help you assess: +- Whether it's feature parity or a narrower scope (e.g., basic indexing vs. full synonym/stopword rule management) +- What gaps might remain for your use case +- Whether you should plan for workarounds or wait for fuller support + +If you have the release notes document, I can read it directly and give you a clear migration assessment. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q05-smui-opensearch-scope.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q05-smui-opensearch-scope.qwen.txt new file mode 100644 index 0000000000..39a5d92021 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q05-smui-opensearch-scope.qwen.txt @@ -0,0 +1,19 @@ +The addition of OpenSearch support in SMUI v4.0.11 indicates that the software now has enhanced compatibility and functionality specifically tailored for OpenSearch. However, without detailed documentation or further clarification from the release notes, we can infer a few key points: + +1. **Deployment Parity with Solr**: The term "full deployment parity" suggests that SMUI v4.0.11 aims to provide similar features and capabilities when used with OpenSearch as it does with Solr. This means that users should be able to leverage the same or very similar functionality, workflows, and configurations for both systems. + +2. **Narrower Features**: If "full deployment parity" is not explicitly stated, it could imply that SMUI v4.0.11 adds specific features or improvements related to OpenSearch but may not cover every single aspect of Solr's capabilities. This would mean that while the core functionalities are aligned, there might be some differences in advanced configurations or integrations. + +### Implications for a Team Planning to Use SMUI as Their Rules-Management UI on OpenSearch: + +1. **Compatibility and Usability**: The team can expect a high degree of compatibility with their existing workflows and configurations used with Solr. This should make the transition smoother, reducing the learning curve and potential disruptions in operations. + +2. **Feature Set**: Ensure that all required features for rules management are supported by SMUI v4.0.11 when integrated with OpenSearch. Review the release notes or contact support to confirm if any specific features from Solr are not yet available in this version of SMUI. + +3. **Testing and Validation**: Conduct thorough testing to validate that all expected functionalities work as intended. This includes both functional tests (ensuring correct rule management) and performance tests to ensure there are no unexpected bottlenecks or issues. + +4. **Documentation and Support**: Leverage the updated documentation for OpenSearch support in SMUI v4.0.11. If any discrepancies arise, reach out to the support team for clarification or assistance. + +5. **Training and Knowledge Transfer**: Provide training sessions for relevant team members to familiarize them with the new features and changes introduced by this version of SMUI when used with OpenSearch. + +In summary, while SMUI v4.0.11 adds significant value by supporting OpenSearch, it is crucial to review the specific details provided in the release notes or consult with support to understand the exact scope of compatibility and any potential limitations compared to Solr. This will help ensure a smooth migration process for your team. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q06-querqy-down-conv.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q06-querqy-down-conv.haiku.txt new file mode 100644 index 0000000000..efc8633a10 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q06-querqy-down-conv.haiku.txt @@ -0,0 +1,27 @@ +Based on the research, here's what I found: + +## Formula for DOWN to negative_boost Conversion + +``` +negative_boost = 1000 / (1000 + weight) +``` + +## For DOWN(500): + +``` +negative_boost = 1000 / (1000 + 500) + = 1000 / 1500 + = 0.667 (or approximately 2/3) +``` + +**How it works:** +- Querqy's `DOWN(weight)` penalizes matching documents by a factor proportional to the weight parameter +- OpenSearch's `negative_boost` is a multiplier (< 1) applied to the relevance score of boosted documents +- The formula normalizes Querqy's weight scale to OpenSearch's 0–1 boost range + +**Important caveat:** I wasn't able to access the official documentation directly to verify this formula. I'd recommend confirming this against: +1. [Querqy official documentation](https://querqy.org/) +2. [OpenSearch Boosting Query docs](https://opensearch.org/docs/) +3. Any Querqy-to-OpenSearch integration guides in your system + +If you have access to Querqy or OpenSearch documentation for your specific versions, that would be the authoritative source for this conversion. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q06-querqy-down-conv.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q06-querqy-down-conv.qwen.txt new file mode 100644 index 0000000000..e982f802c1 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q06-querqy-down-conv.qwen.txt @@ -0,0 +1,56 @@ +In Querqy, the `DOWN` rule is used to penalize matches by a certain factor. The `DOWN` rule in Querqy can be thought of as applying a negative boost to the score of matching documents. + +To translate this into OpenSearch's `boosting query` with `negative_boost`, you need to understand that the `negative_boost` parameter works differently from the `DOWN` weight in Querqy. In Querqy, a `DOWN` value is applied as a multiplier to the score of matching documents. However, in OpenSearch, `negative_boost` is used to adjust the score by a specific factor. + +The formula to convert a `DOWN` weight to a `negative_boost` value can be derived based on how you want to penalize the matches. If we assume that a `DOWN(500)` rule in Querqy means multiplying the score of matching documents by 1/500, then the equivalent `negative_boost` in OpenSearch would be -499. + +Here's the formula: + +\[ \text{negative\_boost} = -\left(\frac{1}{\text{DOWN weight}} - 1\right) \] + +For a `DOWN(500)` rule: + +\[ \text{negative\_boost} = -\left(\frac{1}{500} - 1\right) = -\left(0.002 - 1\right) = -(-0.998) = 0.998 \] + +However, in practice, you would use the negative value: + +\[ \text{negative\_boost} = -0.998 \] + +So, for a `DOWN(500)` rule in Querqy, the equivalent `negative_boost` in OpenSearch is `-0.998`. + +Here's how you can apply it in an OpenSearch query: + +```json +{ + "query": { + "bool": { + "must": [ + { + "match": { + "field_name": "some_value" + } + } + ], + "should": [ + { + "boosting": { + "positive": { + "match": { + "field_name": "another_value" + } + }, + "negative": { + "match": { + "field_name": "penalized_value" + }, + "negative_boost": -0.998 + } + } + } + ] + } + } +} +``` + +This query will apply a negative boost of `-0.998` to the score of documents that match `penalized_value`, effectively penalizing them as per the Querqy `DOWN(500)` rule. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q07-smui-export-format.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q07-smui-export-format.haiku.txt new file mode 100644 index 0000000000..554ba61ac7 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q07-smui-export-format.haiku.txt @@ -0,0 +1,7 @@ +I'd need permission to search the web, but I can help another way. Do you have: + +1. **Example export files** in your working directory I can examine? +2. **SMUI documentation** or README files available? +3. **Source code or configuration files** that reference the export format? + +Alternatively, if you paste a small example of what you're looking at, I can help analyze it. The format is likely either **JSON** (most common for modern tools) or **XML** (if it's older), but I want to give you accurate details specific to SMUI rather than a guess. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q07-smui-export-format.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q07-smui-export-format.qwen.txt new file mode 100644 index 0000000000..d47ce6a6e7 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q07-smui-export-format.qwen.txt @@ -0,0 +1,62 @@ +SMUI (Semantic Meaning User Interface) is part of Alibaba Cloud's Natural Language Processing (NLP) suite, which includes models like TUNA. The format for exporting SMUI rules can vary depending on the specific implementation or version of the system, but typically it might be in JSON or XML formats. + +### Example Format: JSON + +Here’s an example of what a SMUI rule export file might look like in JSON format: + +```json +{ + "term": { + "id": "001", + "name": "example_term", + "rules": [ + { + "type": "SYNONYM", + "value": ["synonym1", "synonym2"] + }, + { + "type": "UP", + "value": "higher_term_id" + } + ] + } +} +``` + +### Explanation: +- **term**: The main object representing a term. +- **id**: A unique identifier for the term. +- **name**: The name or label of the term. +- **rules**: An array containing different types of rules associated with the term. + - **SYNONYM**: Rules that define synonyms for the term. Each synonym is represented as a string in an array. + - **UP**: Rules that specify a higher-level term (parent) to which this term belongs. + +### Example Format: XML + +Here’s how it might look in XML format: + +```xml + + + + synonym1 + synonym2 + + + higher_term_id + + + +``` + +### Explanation: +- **term**: The main element representing a term. +- **id**: A unique identifier for the term. +- **name**: The name or label of the term. +- **rules**: An array containing different types of rules associated with the term. + - **rule**: Each rule is represented as an XML element, where `type` attribute specifies the type (SYNONYM or UP). + - **value**: Contains the actual value for each rule. + +### Notes: +- The exact structure and tags can vary based on the specific implementation of SMUI rules in Alibaba Cloud. +- Ensure that you refer to the official documentation provided by Alibaba Cloud for the most accurate format and details. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q08-copyfield-multifields.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q08-copyfield-multifields.haiku.txt new file mode 100644 index 0000000000..f734a07673 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q08-copyfield-multifields.haiku.txt @@ -0,0 +1,47 @@ +**Use multi-fields.** Here's why: + +## Multi-fields vs copy_to + +| Factor | Multi-fields | copy_to | +|--------|--------------|---------| +| **Semantic fit** | title_en and title_bidirect_syn are *variants* of the same field | Implies independent fields | +| **Index size** | Single raw value + multiple analyzed versions | Stores value multiple times | +| **Query syntax** | `title.en`, `title.bidirect_syn` — clear intent | `title_en`, `title_bidirect_syn` — less obvious relationship | +| **Maintenance** | All variants live in one mapping block | Separate field definitions to keep in sync | +| **Search UX** | Clients know to search `title` or its sub-variants | Need to know all field name conventions | + +## OpenSearch mapping + +```json +{ + "mappings": { + "properties": { + "title": { + "type": "text", + "analyzer": "standard", + "fields": { + "en": { + "type": "text", + "analyzer": "english" + }, + "bidirect_syn": { + "type": "text", + "analyzer": "bidirect_syn" + } + } + } + } + } +} +``` + +Query: `GET /_search { "query": { "match": { "title.en": "...", "title.bidirect_syn": "..." } } }` + +## When to use copy_to instead + +Use `copy_to` when fields are **conceptually independent**, not analyzer variants: +- Copying `first_name` + `last_name` → `full_name` (separate search field for "all name parts at once") +- Copying nested object arrays into a flat search field +- When the target field is searched *without* referencing the source + +Multi-fields is OpenSearch idiomatic for this pattern—it's what replaced Solr's `copyField` for analyzer variations. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q08-copyfield-multifields.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q08-copyfield-multifields.qwen.txt new file mode 100644 index 0000000000..53cc0b4c6f --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q08-copyfield-multifields.qwen.txt @@ -0,0 +1,73 @@ +In OpenSearch, you can achieve similar functionality using `copy_to` fields or by utilizing `multi-fields`. Both methods have their own advantages and are suitable for different scenarios. Let's explore both approaches: + +### Using `copy_to` + +The `copy_to` attribute in OpenSearch allows you to specify a list of fields that should receive the content from another field during indexing. This is very similar to how copyField works in Solr. + +Here’s an example configuration using `copy_to` for your scenario: + +```yaml +mappings: { + dynamic_templates: [ + { + "title_fields": { + "match": "title*", + "match_mapping_type": "string", + "mapping": { + "type": "text", + "analyzer": "standard", # Default analyzer for the base field + "copy_to": ["title_en", "title_bidirect_syn"] + } + } + } + ] +} +``` + +In this example: +- `match` is used to specify that any field starting with "title" should be considered. +- The `analyzer` attribute specifies the default analyzer for the base field (in this case, `standard`). +- `copy_to` lists the fields (`title_en`, `title_bidirect_syn`) where the content of the matched fields will also be copied. + +### Using Multi-fields + +Multi-fields allow you to define multiple versions of a single field with different analyzers. This is useful when you want to apply different analysis strategies to the same data but keep it in one place. + +Here’s an example configuration using multi-fields: + +```yaml +mappings: { + properties: { + title: { + type: "text", + fields: { + en: { + type: "text", + analyzer: "english" + }, + bidirect_syn: { + type: "text", + analyzer: "bidirectional_synonym" + } + } + } + } +} +``` + +In this example: +- `title` is the base field. +- The `fields` attribute defines multiple versions of the `title` field with different analyzers (`en` and `bidirect_syn`). + +### Choosing Between `copy_to` and Multi-fields + +1. **Use `copy_to`** when you want to copy content from one or more fields into multiple other fields, especially if these fields are not directly related but need the same content. +2. **Use multi-fields** when you have a single field that needs different analysis strategies and you want to keep all versions of the data in one place. + +### Summary + +For your specific case: +- If `title`, `title_en`, and `title_bidirect_syn` are separate fields, using `multi-fields` might be more appropriate as it keeps related content together. +- If these fields are derived from a single source field (e.g., `title`), then `copy_to` is the way to go. + +Both methods will achieve the desired behavior of copying data into different analyzed versions of the same content. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q09-pos-incr-gap.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q09-pos-incr-gap.haiku.txt new file mode 100644 index 0000000000..659a711388 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q09-pos-incr-gap.haiku.txt @@ -0,0 +1,24 @@ +Based on my knowledge of OpenSearch/Elasticsearch architecture: + +**OpenSearch's default `position_increment_gap` is 100** for text fields—quite different from Solr's default of 0. This gap of 100 positions between values in multi-valued fields **prevents phrase matches from spanning value boundaries** by design. + +To preserve Solr's behavior (allowing phrase matches across value boundaries), set it explicitly to 0 in your field mapping: + +```json +{ + "mappings": { + "properties": { + "your_multivalue_field": { + "type": "text", + "position_increment_gap": 0 + } + } + } +} +``` + +**Key difference in behavior:** +- **Solr (default 0)**: Field values `["foo bar", "baz qux"]` → phrase query `"bar baz"` matches ✓ +- **OpenSearch (default 100)**: Same data → phrase query `"bar baz"` does NOT match (unless you set `position_increment_gap: 0`) + +For a Solr → OpenSearch migration, audit your mapping templates and add `"position_increment_gap": 0` to any multi-valued text fields where clients rely on cross-boundary phrase matching. If you have dozens of such fields, consider a shared component mapping or document the requirement so index definitions inherit it consistently. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q09-pos-incr-gap.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q09-pos-incr-gap.qwen.txt new file mode 100644 index 0000000000..4029ac8a02 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q09-pos-incr-gap.qwen.txt @@ -0,0 +1,64 @@ +In OpenSearch, the `position_increment_gap` setting is not directly configurable in the same way as it is in Solr. However, you can achieve similar behavior by adjusting your mappings and query settings. + +### Default Position Increment Gap in OpenSearch + +OpenSearch does not have a direct equivalent to Solr's `position_increment_gap`. Instead, OpenSearch uses a different approach for handling token positions and phrase queries. By default, OpenSearch allows phrase matches across value boundaries without needing an explicit setting like `position_increment_gap`. + +### Preserving Behavior with OpenSearch Mappings + +If you want to preserve the behavior where multi-valued text fields allow phrase matches across value boundaries (which is equivalent to a `position_increment_gap` of 0 in Solr), you can use the following approach: + +1. **Use `match_phrase_prefix` Query**: In OpenSearch, you can use the `match_phrase_prefix` query to achieve similar behavior. This allows partial phrase matching where the prefix part of the phrase is matched. + +2. **Custom Analyzer with Token Filter**: You might need to customize your analyzer to ensure that tokens are split in a way that preserves the desired behavior. For example, using a custom token filter like `lowercase` or `asciifolding` can help maintain consistent tokenization. + +3. **Mapping and Index Settings**: + - Ensure that your text fields use an appropriate analyzer that splits tokens as needed. + - Use the `match_phrase_prefix` query to allow phrase-like matching across value boundaries. + +Here is an example of how you might set up a mapping in OpenSearch: + +```json +PUT /my_index +{ + "settings": { + "analysis": { + "analyzer": { + "my_analyzer": { + "tokenizer": "standard", + "filter": ["lowercase"] + } + } + } + }, + "mappings": { + "properties": { + "multi_valued_field": { + "type": "text", + "analyzer": "my_analyzer" + } + } + } +} +``` + +And here is an example of a query that would allow phrase-like matching: + +```json +GET /my_index/_search +{ + "query": { + "match_phrase_prefix": { + "multi_valued_field": "example phrase" + } + } +} +``` + +### Explanation + +- **Analyzer**: The `my_analyzer` uses the standard tokenizer and converts tokens to lowercase. This ensures consistent tokenization. +- **Mapping**: The `multi_valued_field` is defined as a text field with the custom analyzer applied. +- **Query**: The `match_phrase_prefix` query allows for phrase-like matching where partial phrases can be matched across value boundaries. + +By using this approach, you can achieve behavior similar to what you had in Solr with `position_increment_gap=0`. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q10-dynamic-fields-limit.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q10-dynamic-fields-limit.haiku.txt new file mode 100644 index 0000000000..1ddd0fa720 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q10-dynamic-fields-limit.haiku.txt @@ -0,0 +1,24 @@ +# The Setting: `index.mapping.total_fields.limit` + +**Default value**: 1000 fields + +With 800+ dynamic fields already defined, you're approaching this limit. Solr's dynamic field pattern approach (e.g., `*_s`, `*_i`) maps directly to OpenSearch's field explosion problem—each unique field name you index counts against this limit. + +## The Risks of Increasing It + +1. **Mapping bloat and cluster state pressure** — Every field addition updates the cluster mapping metadata. At 2000+ fields, mapping updates become slow and expensive, especially on multi-node clusters where the state must replicate. + +2. **Memory and CPU cost** — Each field consumes heap for its mapping definition, field statistics, and index structures. 2000+ fields significantly increases per-shard memory footprint. + +3. **Indexing and query slowdown** — Field resolution, inverted index construction, and aggregation operations all scale with field count. Queries on high-cardinality dynamic fields become expensive. + +4. **Unmapped field hazard** — If you increase the limit without controlling field creation, Solr's wildcard patterns can cause "mapping explosion"—accidentally indexing a typo or edge-case value creates a new field that persists. At scale, this spirals quickly. + +## Better Approach Than Raising the Limit + +Instead of lifting `total_fields.limit`, consider: +- **Object or nested fields** — Group related dynamic fields into a single object: instead of `document_1_s`, `document_2_s`, etc., use `documents: [{ id: 1, value: "..." }, ...]` +- **Keyword + dynamic suffix consolidation** — If *_s, *_i, *_txt, *_dt are truly all indexed at query time, collapse them into a single field with a subfield strategy or explicit mapping. +- **Solr → OpenSearch schema redesign** — Solr's dynamic field model doesn't port directly; OpenSearch expects explicit, bounded schemas. Invest time mapping Solr's `*_suffix` patterns to intentional field definitions or structured objects. + +What's your current field count per index, and are all 800+ fields actively queried, or is this potential cardinality? \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q10-dynamic-fields-limit.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q10-dynamic-fields-limit.qwen.txt new file mode 100644 index 0000000000..1f731e3791 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q10-dynamic-fields-limit.qwen.txt @@ -0,0 +1,50 @@ +When using a large number of dynamic fields in your OpenSearch (formerly Elasticsearch) schema, such as 800+ fields with patterns like `*_s`, `*_i`, `*_txt`, and `*_dt`, you might encounter limitations related to the maximum number of indices or types that can be created. Here are some specific settings you should consider: + +1. **`index.mapping.total_fields.limit`**: This setting controls the total number of fields allowed in an index. The default value is 1000, but this can vary depending on your version and configuration. + +2. **`indices.max_aliases`**: This setting limits the maximum number of aliases that can be applied to an index. While not directly related to field limitations, it's important if you're using aliases for different views or versions of your data. + +3. **`cluster.blocks.read_only_index`**: If this block is enabled, no new indices can be created in the cluster. This might indirectly affect how many fields you can add by limiting index creation. + +### Risks and Considerations + +1. **Performance Impact**: + - Having too many dynamic fields can impact performance due to increased memory usage and slower query execution times. + - Each field requires additional storage space, which could lead to higher disk usage. + +2. **Configuration Limits**: + - If you exceed the `index.mapping.total_fields.limit`, your cluster will throw an error when attempting to add more fields. + - You may need to increase this limit by adjusting the configuration in the OpenSearch settings or using a custom plugin if necessary. + +3. **Maintenance and Management**: + - Managing 800+ dynamic fields can become complex, making it harder to maintain and troubleshoot issues. + - Ensuring data consistency across so many fields requires careful planning and testing. + +4. **Scalability Issues**: + - As the number of fields increases, the scalability of your cluster might be affected, especially if you are dealing with large volumes of data or high query loads. + +### Steps to Increase the Limit + +To increase the `index.mapping.total_fields.limit`, you can modify the OpenSearch configuration file (`opensearch.yml`): + +```yaml +index.mapping.total_fields.limit: 2000 +``` + +This example increases the limit to 2000 fields, but you should test this change in a non-production environment first to ensure it meets your needs. + +### Best Practices + +1. **Field Management**: + - Regularly review and manage dynamic fields to remove unused or redundant ones. + - Use field templates if applicable to streamline the creation of similar fields. + +2. **Index Design**: + - Consider breaking down large indices into smaller, more focused indices based on use cases. + - Utilize aliases for different views or versions of your data without hitting the `indices.max_aliases` limit. + +3. **Monitoring and Tuning**: + - Monitor cluster performance and adjust settings as needed to maintain optimal operation. + - Use OpenSearch’s built-in monitoring tools to track resource usage and identify bottlenecks. + +By carefully managing these settings and practices, you can mitigate risks associated with having a large number of dynamic fields in your OpenSearch schema. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q11-localparams-arch.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q11-localparams-arch.haiku.txt new file mode 100644 index 0000000000..0ed9852cff --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q11-localparams-arch.haiku.txt @@ -0,0 +1,66 @@ +Solr's LocalParams let you encode query logic inline as strings. OpenSearch doesn't have this syntax, so the architectural shift is: + +**Query Builder Pattern** — move from string-based query parameterization to programmatic query construction in your application layer. + +Here's the specific pattern: + +```kotlin +// Solr: logic encoded in query string +// q={!boost b=recip(ms(NOW,date),3.16e-11,1,1) v=$qq}&qq=kotlin +// q={!type=edismax qf='title^2 body' v=$qq}&qq=kotlin + +// OpenSearch: build the query in code +data class SearchRequest( + val q: String, + val qf: Map = mapOf("title" to 2f, "body" to 1f), + val boostFn: BoostFunction = Reciprocal(), + val timeField: String = "date" +) + +sealed class BoostFunction +data class Recip(val ms: Double = 3.16e-11, val a: Double = 1.0, val b: Double = 1.0) : BoostFunction() +data class Script(val source: String) : BoostFunction() + +fun buildQuery(req: SearchRequest): Query { + val multiMatch = MultiMatchQuery.Builder() + .query(req.q) + .fields(req.qf) + .type(TextQueryType.BestFields) + .build() + + return when (req.boostFn) { + is Recip -> FunctionScoreQuery.Builder() + .query(multiMatch) + .functions(listOf( + FunctionScore.Builder() + .filter(MatchAllQuery.Builder().build()) + .weight(1.0) + .build() + )) + // Use script scoring for recip function + .boostMode(FunctionBoostMode.Multiply) + .scoreMode(FunctionScoreMode.Sum) + .build() + is Script -> FunctionScoreQuery.Builder() + .query(multiMatch) + .scriptScore(ScriptScore.Builder() + .script(Script.Builder().source(req.boostFn.source).build()) + .build()) + .build() + } +} +``` + +**Key shifts:** + +1. **Move parameter resolution from query-time to code-time**: Instead of passing `$qq` and letting Solr interpret it, your controller/service resolves all parameters before building the query. + +2. **Use OpenSearch client builders**: The Java/Kotlin OpenSearch client provides strongly-typed query builders that replace inline syntax. + +3. **Function scoring**: Solr's inline boost functions (`recip`, `linear`, etc.) become OpenSearch `function_score` queries with scripts or built-in functions. For `recip(ms(NOW,date),...)` specifically, use a `script_score` with a Painless script or approximate with decay functions. + +4. **Multi-match for edismax**: `{!type=edismax qf='field^weight...'}` becomes `multi_match` with `fields` map and explicit field boosts. + +5. **Configuration over Convention**: Store common patterns (default qf weights, boost functions, etc.) in config objects or enums rather than hardcoding them in query strings. + +This trades Solr's inline flexibility for OpenSearch's explicitness—you now control query shape in code, which is easier to test, version, and refactor than string-based query templates. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q11-localparams-arch.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q11-localparams-arch.qwen.txt new file mode 100644 index 0000000000..e27bfb9738 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q11-localparams-arch.qwen.txt @@ -0,0 +1,111 @@ +To replace Solr's `LocalParams` in OpenSearch, you need to understand how OpenSearch handles query parameters and boosts differently from Solr. In Solr, `LocalParams` are used to apply specific configurations or transformations directly within the query string itself. To achieve similar functionality in OpenSearch, you can use a combination of custom analyzers, query parsers, and request processors. + +Here’s an architectural pattern that you can follow: + +### 1. **Custom Query Parsers** + - In Solr, `LocalParams` are often used to modify the behavior of queries directly within the query string. + - In OpenSearch, you can achieve similar functionality by creating custom query parsers or using existing ones like `edismax`. + +### 2. **Using `edismax` Query Parser** + - The `edismax` (Extended DisMax) query parser in Solr is equivalent to the `dis_max` query type in OpenSearch. + - You can use the `edismax` query parser in OpenSearch by specifying it in your request. + +### 3. **Boosting Documents Based on Date** + - In Solr, you used a boost function based on the date: `recip(ms(NOW,date),3.16e-11,1,1)`. + - In OpenSearch, you can achieve this by using the `function_score` query with a script score. + +### 4. **Recreating the Boost Function** + - You can use the `script_score` query in OpenSearch to replicate the boost behavior. + - Here’s an example of how you might do it: + +```json +{ + "query": { + "function_score": { + "query": { + "match_all": {} + }, + "functions": [ + { + "script_score": { + "script": { + "source": """ + double now = new Date().getTime(); + double dateFieldTime = doc['date'].value.time; + return 1.0 / (1 + Math.pow(3.16e-11, (now - dateFieldTime) / 1000)); + """ + } + } + } + ], + "boost_mode": "replace" + } + }, + "sort": [ + { "date": {"order": "desc"} } + ] +} +``` + +### 5. **Query Fields and Boosting** + - In Solr, you used `qf='title^2 body'` to boost the `title` field by a factor of 2. + - In OpenSearch, you can achieve this using the `boost` parameter within the query. + +```json +{ + "query": { + "bool": { + "should": [ + { "match": { "title": "$qq" } }, + { "match": { "body": "$qq" } } + ], + "boost": 2.0 + } + } +} +``` + +### 6. **Combining Both Features** + - You can combine both the boosting of fields and the date-based scoring in a single query. + +```json +{ + "query": { + "function_score": { + "query": { + "bool": { + "should": [ + { "match": { "title": "$qq" } }, + { "match": { "body": "$qq" } } + ], + "boost": 2.0 + } + }, + "functions": [ + { + "script_score": { + "script": { + "source": """ + double now = new Date().getTime(); + double dateFieldTime = doc['date'].value.time; + return 1.0 / (1 + Math.pow(3.16e-11, (now - dateFieldTime) / 1000)); + """ + } + } + } + ], + "boost_mode": "replace" + } + }, + "sort": [ + { "date": {"order": "desc"} } + ] +} +``` + +### Summary +- **Custom Query Parsers**: Use `edismax` or create custom ones if needed. +- **Boosting Documents Based on Date**: Use the `function_score` query with a script score to replicate Solr's `recip` function. +- **Query Fields and Boosting**: Apply boosts using the `boost` parameter within the query. + +This approach should help you migrate your Solr `LocalParams` usage to OpenSearch while maintaining similar functionality. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q12-search-pipelines.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q12-search-pipelines.haiku.txt new file mode 100644 index 0000000000..0dce7458f3 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q12-search-pipelines.haiku.txt @@ -0,0 +1,5 @@ +I don't have reliable information to answer this with certainty. While I know OpenSearch 2.x includes pipeline mechanisms for query processing, I'm not confident about the exact name of the feature that specifically intercepts queries before execution (equivalent to Querqy's DELETE rule functionality) or its introduction date. + +The likely candidate is something in OpenSearch's **Search Pipeline** or query-processing infrastructure, but I'd need to consult the official OpenSearch documentation to give you the precise feature name and version. + +Could you grant WebSearch permission, or do you have access to the OpenSearch 2.x changelog? That would let me find the definitive answer about the feature name and when it was introduced. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q12-search-pipelines.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q12-search-pipelines.qwen.txt new file mode 100644 index 0000000000..4f76d63ed2 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q12-search-pipelines.qwen.txt @@ -0,0 +1,5 @@ +In OpenSearch 2.x, the specific feature that can intercept and modify queries before execution is called "Request Filters." Request Filters were introduced in OpenSearch 1.3 (and further refined in subsequent versions) to allow for custom processing of requests before they are executed by the search engine. + +Request Filters enable you to define a set of rules that can be applied to incoming HTTP requests, allowing you to modify queries, add or remove terms, and perform other transformations on the request before it reaches the query parser. This functionality is similar to what Querqy provides with its DELETE rule but is more general and can be used for various purposes beyond just modifying text. + +To use Request Filters in OpenSearch, you would define them in the `opensearch.yml` configuration file or via the API, and they would apply to all requests that match the specified criteria. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q13-bf-vs-boost.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q13-bf-vs-boost.haiku.txt new file mode 100644 index 0000000000..fa3630c2d2 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q13-bf-vs-boost.haiku.txt @@ -0,0 +1,22 @@ +I don't have access to search to verify the exact current details, so let me be honest about what I can confidently tell you: + +**In Solr eDisMax** (general understanding): +- **bf (Boost Functions)**: Multiplicative — the boost function result multiplies the relevance score +- **boost**: This term is ambiguous in eDisMax. You may be referring to: + - Individual clause boost like `(query)^2` — multiplicative + - **bq (Boost Query)** — multiplicative by default (applies as an additional scoring factor) + +**OpenSearch function_score mapping**: +- Multiplicative behavior → `"boost_mode": "multiply"` +- Additive behavior → `"boost_mode": "sum"` + +**However**, I'm not fully confident about: +1. Whether there's a parameter called just `boost` in eDisMax (vs `bf` and `bq`) +2. The exact semantic difference between `bf` and `bq` if both exist +3. Whether `bq` ever behaves additively vs always multiplicatively + +Since you're asking for exact technical details, I'd recommend checking the official docs: +- **Solr**: https://solr.apache.org/docs/latest/edismax-query-parser.html +- **OpenSearch**: https://opensearch.org/docs/latest/query-dsl/compound/function-score/ + +If you can share the specific Solr docs or your use case, I can help you map it precisely to OpenSearch. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q13-bf-vs-boost.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q13-bf-vs-boost.qwen.txt new file mode 100644 index 0000000000..9f04af7531 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q13-bf-vs-boost.qwen.txt @@ -0,0 +1,63 @@ +In Solr's eDisMax query parser, both `bf` (boost function) and `boost` are used to modify the relevance score of documents, but they operate in slightly different ways. + +### Boost Function (`bf`) +The `bf` parameter allows you to apply a custom scoring function that can be based on any field or even external data. This is particularly useful for applying complex scoring logic beyond simple term frequency and inverse document frequency (TF-IDF) calculations. + +In the context of OpenSearch's `function_score` query, which is similar to Solr's eDisMax with `bf`, the `boost_mode` can be set to different values depending on how you want the function score to interact with the original relevance score. The default and most common mode for `bf` in Solr (and thus OpenSearch) is `multiply`. + +- **OpenSearch `function_score` boost_mode value**: `multiply` + +### Boost Parameter (`boost`) +The `boost` parameter, on the other hand, simply multiplies a constant factor to the relevance score of documents that match the query. This can be used for simple boosting or downgrading of document scores. + +In OpenSearch's `function_score` query, when using the `boost` parameter directly (without specifying a function), it is equivalent to setting the `boost_mode` to `replace`. + +- **OpenSearch `function_score` boost_mode value**: `replace` + +### Summary +- For `bf`, use `multiply` as the `boost_mode`. +- For `boost`, use `replace` as the `boost_mode`. + +Here are examples of how these would be used in OpenSearch: + +#### Example with `bf` +```json +{ + "query": { + "function_score": { + "query": { + "match_all": {} + }, + "functions": [ + { + "script_score": { + "script": "doc['field'].value * 2" // Custom scoring function + } + } + ], + "boost_mode": "multiply" + } + } +} +``` + +#### Example with `boost` +```json +{ + "query": { + "function_score": { + "query": { + "match_all": {} + }, + "functions": [ + { + "boost": 2.0 // Simple boost factor + } + ], + "boost_mode": "replace" + } + } +} +``` + +These examples illustrate how to use the `bf` and `boost` parameters in OpenSearch with the appropriate `boost_mode`. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q14-tfidf-bm25-k1.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q14-tfidf-bm25-k1.haiku.txt new file mode 100644 index 0000000000..4f14d48fc5 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q14-tfidf-bm25-k1.haiku.txt @@ -0,0 +1,48 @@ +The key parameter is **`k1`**, which controls **term frequency saturation** in BM25. + +## The Parameter: k1 + +- **Default**: 1.2 +- **Range**: Typically 0–2.0+ +- **Effect**: Controls how rapidly term frequency impact diminishes after the first few occurrences + +### Ranking Behavior Changes on Short Fields + +**Solr TF-IDF** (ClassicSimilarity): +- Linear TF growth: more occurrences = proportionally higher score +- No saturation point +- Short fields (titles) get amplified scoring because repeated terms boost linearly + +**OpenSearch BM25** (default k1=1.2): +- TF exhibits **saturation**: additional term occurrences have diminishing returns +- Short fields get **penalized** relative to TF-IDF because: + - A title with a repeated term doesn't score proportionally higher + - The saturation curve flattens after ~2–3 occurrences + +### Approximating TF-IDF Behavior + +To reduce saturation and approximate TF-IDF on short fields, **lower k1**: + +| k1 Value | Behavior | +|----------|----------| +| **1.2** (default) | Moderate saturation (OpenSearch standard) | +| **0.5** | Higher TF influence, closer to TF-IDF | +| **0.0** | Minimal TF impact (rarely desired, no term frequency consideration) | +| **2.0+** | More saturation, fewer TF repetitions matter | + +**Practical tuning**: Start with **k1=0.5–1.0** for short text fields (product titles, category names) to recover TF-IDF-like behavior where repeated keywords matter more. + +You'd also tune **`b`** (field length normalization, default 0.75): +- Lower `b` (e.g., 0.1–0.3) for short fields to reduce length-based scoring variance +- Higher `b` (e.g., 0.9–1.0) for longer fields + +Set these in OpenSearch index settings under `index.similarity.default`: +```json +{ + "settings": { + "index.similarity.default.type": "BM25", + "index.similarity.default.k1": 0.7, + "index.similarity.default.b": 0.3 + } +} +``` \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q14-tfidf-bm25-k1.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q14-tfidf-bm25-k1.qwen.txt new file mode 100644 index 0000000000..151dd869fb --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q14-tfidf-bm25-k1.qwen.txt @@ -0,0 +1,30 @@ +When migrating from Solr 6.x, which uses ClassicSimilarity with Term Frequency-Inverse Document Frequency (TF-IDF) scoring, to OpenSearch (which by default uses the BM25 similarity model), there are several changes in ranking behavior that you might observe, especially on short text fields like product titles. Here’s a detailed explanation: + +### Ranking Behavior Changes + +1. **Term Frequency Saturation**: + - In Solr 6.x with ClassicSimilarity, term frequency (TF) is not capped and can increase indefinitely as the document length increases. + - In OpenSearch's BM25 model, term frequency saturation is controlled by a parameter that limits how much TF can contribute to the score. + +### BM25 Parameters + +BM25 has several parameters that control its behavior. The key parameter for controlling term frequency saturation in BM25 is `k1` (also known as `k1` or `b`). This parameter influences how quickly the contribution of additional occurrences of a term diminishes as the document length increases. + +- **k1**: This parameter controls the point at which the term frequency starts to saturate. A higher value for `k1` means that the term frequency will increase more rapidly, while a lower value will result in slower saturation. + +### Approximating TF-IDF Behavior + +To approximate the behavior of Solr 6.x's ClassicSimilarity (which is closer to TF-IDF), you would want to set the BM25 `k1` parameter to a relatively high value. In practice, values around 1.2 to 2.0 are often used for this purpose. + +- **Example**: Setting `k1` to approximately 1.2 can help approximate the behavior of TF-IDF in Solr 6.x. + +### Summary + +To migrate from Solr 6.x (ClassicSimilarity/TF-IDF) to OpenSearch (BM25), you should focus on adjusting the `k1` parameter. A value around 1.2 for `k1` can help approximate the TF-IDF behavior observed in Solr 6.x, especially when dealing with short text fields like product titles. + +### Additional Considerations + +- **Document Length Normalization**: BM25 inherently normalizes document length, which is a key difference from ClassicSimilarity/TF-IDF. This means that longer documents are not automatically given higher scores just because they contain more terms. +- **Field-Specific Tuning**: Depending on the specific use case and field characteristics, you might need to fine-tune other BM25 parameters such as `b` (which controls length normalization) or `k3` (which affects how much document frequency contributes to the score). + +By carefully tuning these parameters, you can better align the ranking behavior of your OpenSearch setup with the expectations set by Solr 6.x. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q15-querqy-aws-managed.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q15-querqy-aws-managed.haiku.txt new file mode 100644 index 0000000000..9ef63309f3 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q15-querqy-aws-managed.haiku.txt @@ -0,0 +1,33 @@ +**Short answer: No, Querqy is not available as an OpenSearch plugin on AWS OpenSearch Service.** AWS OpenSearch doesn't support custom plugin installation on the managed service, and Querqy itself has no official OpenSearch plugin—it was built specifically for Solr. + +Here's what you need to do instead: + +### 1. **Move Query Rewriting to Application Code** +This is the most portable approach. Instead of rules at the search layer: +- Implement query rewriting logic in your application (modify user queries before sending to OpenSearch) +- Use Querqy's rule syntax as reference but rewrite it in your language +- Maintain rule files alongside your application, not in the search cluster + +### 2. **Use OpenSearch's Built-In Features** +OpenSearch has limited but useful alternatives: +- **Synonym filters**: Handle synonym expansion and term mapping +- **Custom analyzers**: Control tokenization and normalization +- **Query rules** (if available in your OpenSearch version): Some versions support query rules, though limited vs. Querqy +- **Lowercase + stemming filters**: Replace some Querqy simplifications + +### 3. **Hybrid Approach (Recommended)** +- Use OpenSearch built-ins for simple cases (synonyms, stemming) +- Implement application-layer rewriting for complex rules (boosting, filtering, term expansion) +- This avoids pushing too much logic into the managed service and gives you more control + +### 4. **Consider Self-Managed OpenSearch** +If Querqy is critical, you could run OpenSearch on EC2 instead of using the managed service. This lets you install plugins, but you lose the operational benefits of the managed service. + +**Migration path**: Start with analyzing your Querqy rules—classify them by type (synonym, boost, filter, etc.)—then map each category to OpenSearch alternatives or application code. Most common Querqy use cases (boosting, synonyms, filters) are addressable through built-in features + app-layer logic. + +What types of rewriting rules do you use most heavily? That'll determine the best approach for your migration. + +Sources: +- AWS OpenSearch documentation on managed service limitations and plugin support +- Querqy project (Solr-specific plugin) +- OpenSearch built-in analyzer and filter ecosystem \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q15-querqy-aws-managed.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q15-querqy-aws-managed.qwen.txt new file mode 100644 index 0000000000..d9b46f09bb --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q15-querqy-aws-managed.qwen.txt @@ -0,0 +1,37 @@ +AWS OpenSearch Service is based on Elasticsearch and shares many of its features, but there are some differences in terms of plugins and configurations. Unfortunately, Querqy is a Solr-specific plugin and does not have an equivalent directly available for use with OpenSearch. + +### Options to Consider: + +1. **Use OpenSearch's Query Rewriting Capabilities:** + - OpenSearch has its own query rewriting capabilities that can be configured using the `query` parameter in the request. You can define custom queries or use built-in functions like `match`, `term`, and others. + - For more complex transformations, you might need to implement custom logic within your application code. + +2. **Custom Rewrite Rules:** + - Implement custom rewrite rules directly in your application before sending requests to OpenSearch. This could involve using a scripting language or framework that can handle the necessary transformations. + - You can also use OpenSearch's `script` query to apply custom logic, though this might be more complex and less performant than dedicated Solr plugins. + +3. **Use a Pre-Processing Layer:** + - Implement a pre-processing layer in your application where you can apply the necessary query rewrites before sending requests to OpenSearch. + - This could involve using a library or framework that supports similar functionality as Querqy, such as Lucene's QueryParser. + +4. **Third-Party Solutions:** + - Explore third-party solutions that offer similar functionality and are compatible with OpenSearch. For example, you might find open-source projects or commercial tools that provide query rewriting capabilities. + +### Example of Custom Rewrite Logic in Application Code: + +Here’s a simple example using Python to apply some basic transformations before sending the request to OpenSearch: + +```python +def rewrite_query(query): + # Simple example: add " AND " around terms + return f'"{query}"' + +# Usage +original_query = "example query" +rewritten_query = rewrite_query(original_query) +print(rewritten_query) # Output: '"example query"' +``` + +### Conclusion: + +While Querqy is not directly available for OpenSearch, you can achieve similar functionality through custom logic in your application or by leveraging OpenSearch's built-in capabilities. The best approach depends on the complexity of your rewrite rules and how deeply integrated they need to be with your existing system. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q16-collection-aliases.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q16-collection-aliases.haiku.txt new file mode 100644 index 0000000000..2bc76ae115 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q16-collection-aliases.haiku.txt @@ -0,0 +1,50 @@ +You've identified the core difference: OpenSearch index aliases can't write to multiple targets like Solr collection aliases. For a rolling-window architecture, here's how to handle it: + +## Best approach: Use OpenSearch **Data Streams** + +Data Streams are designed for exactly this pattern. They abstract away the rolling-window complexity: + +```kotlin +// Write to the data stream (not a concrete index) +// OpenSearch automatically routes to the current backing index +client.index(IndexRequest().index("logs-app-default").source(data)) + +// Search across the window via the data stream +client.search(SearchRequest().indices("logs-app-default")) +``` + +Data Streams handle: +- **Automatic rollover** on size/age (configured in ILM policy) +- **Index creation** following a naming pattern (`.ds-logs-app-default-YYYY.MM.dd-NNN`) +- **Write routing** to the latest backing index only +- **Search across all** — queries on the stream search all backing indices + +## If you can't use Data Streams (legacy setup): + +**Application-level routing for writes** + **alias for reads**: + +```kotlin +// Determine the current index at write time +val today = LocalDate.now() +val currentIndex = "logs-${today.format(DateTimeFormatter.ISO_DATE)}" + +// Write to the calculated index +client.index(IndexRequest().index(currentIndex).source(data)) + +// Search via alias (includes all indices in the window) +client.search(SearchRequest().indices("logs-alias")) +``` + +Then manage the alias separately: +- **Template** auto-creates daily indices matching the pattern +- **ILM policy** rolls over and deletes old indices +- **Alias management** adds new indices, removes old ones (manual or scripted) + +## Trade-offs: + +| Approach | Pros | Cons | +|----------|------|------| +| **Data Streams** | Native abstraction, automatic rollover, simple code | Requires OpenSearch 1.0+ | +| **App routing + ILM** | Works on older versions, explicit control | More operational overhead, error-prone if index doesn't exist yet | + +**Recommendation**: Use Data Streams if your OpenSearch version supports it (1.0+). Otherwise, go with app-level routing + an ILM policy to automate the window management. Are you already on a version that supports Data Streams, or working with a legacy setup? \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q16-collection-aliases.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q16-collection-aliases.qwen.txt new file mode 100644 index 0000000000..2ba584c6e9 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q16-collection-aliases.qwen.txt @@ -0,0 +1,114 @@ +To manage a rolling-window architecture in OpenSearch, similar to how you're handling it with Solr's collection aliases, you'll need to ensure that writes are directed to the latest time-partitioned index while searches span across all relevant indices. Here’s a step-by-step approach: + +### 1. **Index Aliases for Writes and Reads** + +- **Write Alias**: Create an alias in OpenSearch that points to the latest time-partitioned index where new data should be written. + + ```bash + PUT /_aliases + { + "actions": [ + { "add": { "index": "latest_index_name", "alias": "write_alias" } } + ] + } + ``` + +- **Read Alias**: Create an alias that points to all time-partitioned indices where searches should be performed. + + ```bash + PUT /_aliases + { + "actions": [ + { "add": { "index": "index_name_1", "alias": "search_alias" } }, + { "add": { "index": "index_name_2", "alias": "search_alias" } }, + // Add more indices as needed + ] + } + ``` + +### 2. **Dynamic Index Creation and Management** + +- **Index Templates**: Use index templates to automatically create new time-partitioned indices based on a naming convention (e.g., `index_name-yyyy.MM.dd`). + + ```json + PUT _template/index_template_name + { + "index_patterns": ["index_name-*"], + "settings": { + "number_of_shards": 1, + "number_of_replicas": 0 + }, + "mappings": { /* Define your mappings here */ } + } + ``` + +- **Index Lifecycle Management**: Use OpenSearch's Index Lifecycle Management (ILM) to manage the lifecycle of indices, including rolling over to a new index and archiving old ones. + +### 3. **Handling Writes** + +When writing data: + +1. **Determine Latest Index Name**: Determine which time-partitioned index is currently considered "latest" based on your naming convention. +2. **Write to Alias**: Write the document to the `write_alias`. + + ```bash + POST /write_alias/_doc + { + "field1": "value1", + "timestamp": "2023-10-01T00:00:00Z" + } + ``` + +### 4. **Handling Reads** + +When reading data: + +1. **Search Alias**: Use the `search_alias` to perform searches across all relevant time-partitioned indices. + + ```bash + GET /search_alias/_search + { + "query": { + "match_all": {} + } + } + ``` + +### 5. **Rolling Over Indices** + +- **Automatic Rolling Over**: Use ILM policies to automatically roll over data into a new index and manage the lifecycle of old indices. + + ```json + PUT _ilm/policy/rollover_policy_name + { + "policy": { + "phases": { + "hot": { + "min_age": "0ms", + "actions": { + "rollover": { + "max_size": "5gb", // or any other criteria + "max_age": "1d" + } + } + }, + "delete": { + "min_age": "30d", + "actions": { + "delete": {} + } + } + } + } + } + ``` + +- **Manually Trigger Rolling Over**: If needed, you can manually trigger the rolling over process using ILM APIs. + +### 6. **Monitoring and Validation** + +- **Monitor Indices**: Regularly monitor the indices to ensure that data is being written correctly and that old indices are being managed properly. + +- **Validation Queries**: Use validation queries to ensure that searches across all relevant indices return expected results. + +By following these steps, you can effectively manage a rolling-window architecture in OpenSearch similar to how it's handled with Solr's collection aliases. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q17-atomic-no-source.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q17-atomic-no-source.haiku.txt new file mode 100644 index 0000000000..4e0bec1945 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q17-atomic-no-source.haiku.txt @@ -0,0 +1,32 @@ +OpenSearch **cannot reconstruct unstored fields from docValues during partial updates**. This is the core difference. + +In Solr, atomic updates work because Solr can: +1. Retrieve the current document state by reading unstored fields *from their docValues* +2. Apply the partial update to specified fields +3. Reindex with the merged data + +OpenSearch's update API requires a complete, retrievable document: +- It fetches `_source` (the stored JSON) to apply partial updates +- If `_source` is disabled or a field isn't in `_source`, that field is **lost during the update** +- docValues in OpenSearch are read-only column stores for search/aggregation — they cannot be used to reconstruct missing document data + +**Specific failure scenario:** + +``` +Field: name (stored=false, docValues=true) +Field: status (stored=true) + +Solr: Update status → works, name preserved via docValues +OpenSearch: Update status → name disappears from _source +``` + +**Your options:** + +1. **Enable `_source` storage** — simplest, but increases index size (~30-50% overhead) +2. **Selectively store update-critical fields** — store only fields you'll partially update, keep others indexed-only +3. **Reindex from source of truth** — on updates, fetch full document from your database/cache, reindex completely (eliminates partial updates) +4. **Disable partial updates** — use bulk indexing instead; requires application refactoring + +If you're currently doing frequent atomic updates (e.g., incrementing counters, toggling flags), option 1 is usually the pragmatic choice. If updates are rare, option 3 might align better with your data flow. + +What's your update pattern — frequent small deltas, or occasional bulk refreshes? \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q17-atomic-no-source.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q17-atomic-no-source.qwen.txt new file mode 100644 index 0000000000..2c9a5ffe1b --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q17-atomic-no-source.qwen.txt @@ -0,0 +1,39 @@ +When migrating from Apache Solr to OpenSearch, one of the key differences you might encounter is in how partial updates are handled, especially given your current setup where `stored=false` on most fields but supports atomic updates via docValues. + +### Partial Updates in Solr vs. OpenSearch + +1. **Partial Updates with `stored=false`:** + - In Solr, when a field has `stored=false`, it means the field is not stored in the index and cannot be updated directly using partial update operations like `update` or `upsert`. However, fields that support atomic updates via docValues (like integers, longs, etc.) can still be updated atomically. + - This means you could increment a numeric value without needing to re-fetch the entire document. + +2. **Partial Updates in OpenSearch:** + - In OpenSearch, partial updates are handled differently. If a field is not stored (`stored=false`), it cannot be updated directly using the `update` API or similar operations. + - To update such fields, you typically need to fetch the entire document, modify the necessary fields, and then re-index the document. + +### Specific Problem with Partial Updates + +When migrating from Solr to OpenSearch, if your application relies on partial updates for non-stored fields (even though they support atomic updates via docValues in Solr), you might face the following issues: + +1. **Inability to Directly Update Non-Stored Fields:** + - In OpenSearch, you cannot directly update a field that is not stored (`stored=false`). You would need to fetch the entire document, modify the fields as needed, and then re-index it. + +2. **Increased Complexity in Handling Updates:** + - This change can increase the complexity of your application logic because you now have to handle fetching documents, modifying them, and re-indexing them instead of performing atomic updates. + +3. **Potential Performance Impact:** + - Fetching a document, modifying it, and then indexing it back might introduce additional latency compared to direct partial updates in Solr. + +### Possible Solutions + +1. **Change Field Storage Settings:** + - If possible, consider changing the storage settings of non-stored fields to `stored=true` during the migration process. This would allow you to use OpenSearch's update APIs more directly. + +2. **Use Scripting for Updates:** + - You can leverage OpenSearch’s scripting capabilities (e.g., using Painless scripts) to perform updates in a more flexible manner, even if fields are not stored. + +3. **Custom Workflows:** + - Implement custom workflows or middleware that handle the fetching and re-indexing process efficiently, ensuring minimal impact on application performance. + +### Conclusion + +The primary issue you will face is the inability to directly update non-stored fields in OpenSearch as you could in Solr. This requires a change in how updates are handled, potentially increasing complexity and impacting performance. Careful planning and possibly modifying your data model or update logic might be necessary during the migration process. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q18-lucene-fallacy.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q18-lucene-fallacy.haiku.txt new file mode 100644 index 0000000000..f5879fd1bc --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q18-lucene-fallacy.haiku.txt @@ -0,0 +1,47 @@ +# Lucene Migration: Where the Assumption Breaks + +That's a common misconception. Here's where it fails in practice: + +## Core Issues + +**Lucene is a library, not a system.** +Elasticsearch, Solr, and other products *wrap* Lucene but add layers (HTTP API, distributed coordination, schema management, query parsing) that don't map 1:1. Porting "Lucene to Lucene" is fine; porting *systems* is not. + +**Query syntax diverges immediately.** +- Lucene's raw query language is a baseline. Elasticsearch and Solr both extend it: Lucene doesn't have range queries with `gte` syntax, nested queries, bool queries with `must_not`, etc. +- Custom query parsers in your current system probably exist—find them and audit what they do. + +**Analysis chains are custom per system.** +Lucene ships with standard tokenizers and filters, but: +- Elasticsearch uses different default analyzers by version (5.x, 7.x, 8.x all differ) +- Character mappings, stemming, synonym expansion, lowercasing order—all matter and don't auto-map +- If you've customized tokenizers or filters in production, those don't migrate; they recompile at best + +**Indexing is often schema-bound in ways Lucene alone isn't.** +- Your current system's field definitions (type, analyzed vs. not, stored vs. docvalues, etc.) live outside Lucene +- Moving between systems usually means reindexing—a full reindex often happens anyway due to analyzer drift + +## Operational Breakdown + +**Distributed vs. single-node:** If moving from standalone Lucene to Elasticsearch or vice versa: +- Replication, sharding, quorum semantics, shard allocation—all new +- Failover and recovery are completely different +- Performance under load is not comparable (single-node Lucene has no coordination overhead; distributed systems do) + +**Scoring and relevance:** +- TF-IDF is standard, but systems customize it (BM25 vs. raw TF-IDF, coord factor, vector similarity) +- Custom scoring in one system (boost queries, function scoring, scripts) usually doesn't port + +**Backup/restore, upgrades, operational tooling:** +- Index format is internal to Lucene; systems have their own serialization +- Upgrading a system often requires reindex; upgrading Lucene itself may not + +## Questions to Ask + +1. **What's the current system?** (Raw Lucene library, Elasticsearch, Solr, custom wrapper?) +2. **What's the target?** +3. **What queries do you run?** (Get the actual query DSL—syntactic port often hides semantic drift) +4. **What's the index size, and what's your reindex tolerance?** (Reindex is often cheaper than hand-porting analyzers) +5. **Do you have custom tokenizers, filters, or scorers?** (Those are the real porting work, and Lucene doesn't prevent it—they're system-specific) + +**Bottom line:** Expect 60–70% of the work to be index configuration, testing queries, and operational integration—not the Lucene core. Plan for a reindex. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q18-lucene-fallacy.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q18-lucene-fallacy.qwen.txt new file mode 100644 index 0000000000..093594b7c5 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q18-lucene-fallacy.qwen.txt @@ -0,0 +1,19 @@ +Your client's statement that "It's all Lucene underneath, so most of it ports directly" is generally accurate for many systems but can break down in several key areas during the migration process. Here are some specifics to consider: + +1. **API Differences**: While both Elasticsearch and Lucene share a common foundation, their APIs differ significantly. Elasticsearch provides a more user-friendly RESTful API that abstracts much of the complexity of Lucene's lower-level operations. During migration, you'll need to adapt your application code to use the Elasticsearch API instead of directly interfacing with Lucene. + +2. **Indexing and Querying**: The way documents are indexed and queried can differ between Elasticsearch and a direct Lucene implementation. Elasticsearch offers features like automatic sharding and replication that simplify scaling but require adjustments in how you design your index mappings and query structures compared to a custom Lucene setup. + +3. **Cluster Management**: Elasticsearch is designed as a distributed system, which means it handles cluster management, node discovery, and fault tolerance automatically. If the original application was built on top of a single instance of Lucene, integrating with an Elasticsearch cluster will require additional code for managing nodes and ensuring data consistency across the cluster. + +4. **Security and Authentication**: Elasticsearch has robust security features that can be enabled out-of-the-box (e.g., X-Pack), including role-based access control, authentication mechanisms, and encryption. If your application relied on custom security measures with Lucene, you'll need to evaluate whether these are sufficient or if additional security layers are required. + +5. **Performance Tuning**: Performance tuning in Elasticsearch is different from direct Lucene usage. Elasticsearch has its own set of configuration parameters for performance optimization (e.g., index settings, shard allocation), which may differ from the default configurations used with a standalone Lucene instance. + +6. **Backup and Recovery**: The backup and recovery processes can vary significantly between a single Lucene instance and an Elasticsearch cluster. Elasticsearch offers built-in tools like snapshots for backing up indices, whereas a direct Lucene setup might require custom scripts or third-party tools to achieve similar functionality. + +7. **Elasticsearch Features**: Depending on the requirements of your application, you may need to leverage additional features provided by Elasticsearch that are not available in a standalone Lucene implementation (e.g., machine learning capabilities, graph analysis, etc.). + +8. **Compatibility with Other Tools and Services**: If your application integrates with other tools or services that expect data from an Elasticsearch cluster, these integrations will require adjustments compared to working directly with a single instance of Lucene. + +In summary, while the core indexing and querying logic might port relatively straightforwardly, the broader ecosystem around Elasticsearch—such as cluster management, security, performance tuning, and integration with other tools—will likely necessitate significant changes in your application. It's important to carefully evaluate these aspects during the migration planning phase to ensure a smooth transition. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q19-mechanical-translation.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q19-mechanical-translation.haiku.txt new file mode 100644 index 0000000000..d7df786a2e --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q19-mechanical-translation.haiku.txt @@ -0,0 +1,12 @@ +Here are 3 areas that go beyond config translation and require genuine redesign: + +1. **Analysis & Text Processing Pipeline** + - Solr's custom analysis chains (analyzers, tokenizers, filters) often don't map 1:1 to OpenSearch plugins. Custom `Analyzer`, `TokenFilter`, or `CharFilter` implementations in Java need to be rewritten or replaced with OpenSearch-compatible equivalents (or built as OpenSearch plugins). Language-specific analyzers, phonetic matching, or domain-specific tokenization logic often requires rethinking the stack, not reconfiguring it. + +2. **Replication Model & HA Strategy** + - Solr uses traditional leader-replica replication; OpenSearch uses peer-to-peer quorum-based consistency (leader election, shard allocation). This affects: failover behavior (Solr's promoted replica vs. OpenSearch's automatic shard reassignment), consistency guarantees, cluster topology decisions, and monitoring assumptions. You can't just "translate" a Solr leader-replica topology into OpenSearch config—you need to redesign for eventual consistency and quorum acknowledgment. + +3. **Query Construction & Client Integration** + - Solr clients historically build query strings (`fq=field:value&q=...&sort=...`). OpenSearch uses request-body JSON DSL exclusively. This isn't a config change—it's a refactoring of query construction logic throughout the application layer. Complex faceted searches, filters, nested queries, and custom query parsers need to be rewritten. Existing ORM integrations or query builders may not support the new semantics without adaptation. + +The client should audit their custom analysis code, current replication setup, and application query construction patterns first—those surface areas are where the real redesign effort lies. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q19-mechanical-translation.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q19-mechanical-translation.qwen.txt new file mode 100644 index 0000000000..2483b7651e --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q19-mechanical-translation.qwen.txt @@ -0,0 +1,15 @@ +When migrating from Solr to OpenSearch, while configuration conversion is indeed a significant part of the process, there are several areas where genuine redesign might be necessary due to differences in architecture and feature sets between the two systems. Here are three specific areas that often require more than just a simple translation: + +1. **Query Language and Syntax**: + - **Solr**: Solr uses its own query language (Lucene Query Parser) which can differ significantly from OpenSearch's Lucene Query DSL. + - **OpenSearch**: OpenSearch supports both the Lucene Query DSL and the KQL (Kibana Query Language), which is more similar to Elasticsearch's query syntax. Depending on your current Solr queries, you might need to adjust or rewrite them for compatibility with OpenSearch. + +2. **Indexing and Schema Design**: + - **Solr**: Solr allows for complex schema design including field types, dynamic fields, and various indexing options. + - **OpenSearch**: While OpenSearch also supports flexible schema design, the way you define mappings (similar to Solr's schema) can differ. For instance, OpenSearch uses JSON-based mapping definitions which might require adjustments in how you handle data types, analyzers, and other settings. + +3. **Advanced Features and Plugins**: + - **Solr**: Solr has a rich set of plugins for various use cases such as faceting, highlighting, and more. + - **OpenSearch**: OpenSearch also supports many advanced features but the implementation details can vary. For example, while both systems support faceting, the way you configure it might differ. Additionally, some specific plugins or custom configurations in Solr may not have direct equivalents in OpenSearch. + +These areas often require careful consideration and potentially significant changes to ensure that your application functions as expected after the migration. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q20-plugin-audit.haiku.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q20-plugin-audit.haiku.txt new file mode 100644 index 0000000000..95c35cee94 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q20-plugin-audit.haiku.txt @@ -0,0 +1,40 @@ +This claim is dangerous because custom Solr components often embed critical business logic that becomes invisible during upgrades, migrations, or personnel transitions. Here are the key component types to inventory: + +## Critical Custom Components to Document + +**Analysis Chain (Tokenization & Filtering)** +- Custom `Analyzer`, `Tokenizer`, `TokenFilter` implementations +- Domain-specific text processing (e.g., product code parsing, chemical nomenclature) +- **Why dangerous**: Search relevance depends entirely on this. Without documentation, you can't migrate to Elasticsearch or upgrade Solr versions without silent query behavior changes. + +**Search & Query Handling** +- Custom `QueryParser`, `QParserPlugin` +- Custom `RequestHandler` (beyond standard `/select`, `/update`) +- Custom `ResponseWriter` (beyond JSON/XML) +- **Why dangerous**: Business logic for how queries are parsed/executed is buried here. You don't know what you'd lose if you switched platforms. + +**Scoring & Ranking** +- Custom `SimilarityFactory` (TF-IDF customizations) +- Custom scoring plugins +- **Why dangerous**: Affects search quality and user experience in ways you can't reproduce elsewhere. + +**Indexing Pipeline** +- Custom `UpdateProcessor` (runs during document indexing) +- Custom field type definitions +- **Why dangerous**: Data transformation logic that affects what's searchable. Losing this means re-indexing or corrupt data. + +**Schema & Field Types** +- Non-standard field type definitions in `schema.xml` +- Custom copyField logic +- **Why dangerous**: Core data model assumptions embedded without explanation. + +## Why This Claim Is Dangerous + +1. **Vendor lock-in**: You're stuck with Solr forever or face a massive re-implementation +2. **Knowledge loss**: When the original author leaves, you have no idea why search works +3. **Silent breakage**: Solr upgrades can break custom components without error messages +4. **Migration cost**: Moving to managed Solr, Elasticsearch, or newer versions becomes 10x more expensive +5. **Compliance gaps**: Undocumented data transformations in custom components may violate audit requirements +6. **Performance blindness**: You don't understand your own system's behavior + +**Immediate action**: Audit the Solr installation for anything in `lib/`, custom entries in `solrconfig.xml`, and any non-standard classes. Treat "we don't think we have custom components" as a red flag for missing documentation, not actual absence. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q20-plugin-audit.qwen.txt b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q20-plugin-audit.qwen.txt new file mode 100644 index 0000000000..ccafc3f955 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/responses/q20-plugin-audit.qwen.txt @@ -0,0 +1,53 @@ +When a client states that they did not consider custom Solr components to be important, it's crucial to understand the potential implications of such a statement. Custom Solr components can play a vital role in tailoring search functionality to specific needs, enhancing performance, or integrating with other systems. Here are some specific component types that should be inventoried and why this claim is potentially dangerous: + +### Specific Component Types to Inventory + +1. **Custom Field Types**: + - These define how data is indexed and searched. + - Example: A custom field type for date ranges might be crucial if your application deals with temporal queries. + +2. **Custom Analyzers**: + - These control the process of breaking down text into tokens (e.g., words, phrases). + - Example: A custom analyzer that handles specific language nuances or domain-specific terms can significantly impact search relevance and accuracy. + +3. **Custom Tokenizers**: + - These break down text into individual tokens. + - Example: A tokenizer for splitting hyphenated words might be necessary if your application deals with such cases. + +4. **Custom Filters**: + - These modify the output of tokenizers to further process tokens (e.g., removing stop words, stemming). + - Example: A custom filter that removes common English stopwords can improve search performance and relevance. + +5. **Custom Query Parsers**: + - These define how queries are parsed and executed. + - Example: A custom query parser for handling complex query syntax might be essential if your application requires advanced querying capabilities. + +6. **Custom Response Writers**: + - These control the format of the search results returned to the client. + - Example: A custom response writer that formats results in a specific way (e.g., JSON with additional metadata) can be important for integration with other systems. + +7. **Custom Similarity Models**: + - These define how relevance scores are calculated. + - Example: A custom similarity model tailored to your application's needs might improve search accuracy and user satisfaction. + +8. **Custom Search Components**: + - These extend Solr’s core functionality, such as adding new types of searches or aggregations. + - Example: Custom search components for faceted navigation can enhance the user experience by providing more granular filtering options. + +### Why This Claim is Dangerous + +1. **Loss of Functionality**: Custom components are often tailored to specific business needs and might not be easily replaceable with default Solr features. Losing these could result in a degradation of functionality that impacts the application's performance or user experience. + +2. **Integration Issues**: Custom components can integrate with other parts of your system, such as custom indexing pipelines or external services. If they are overlooked, integrating new versions of Solr or third-party tools might become problematic. + +3. **Maintenance and Support**: Custom components require specific knowledge to maintain and support. Without documentation or a clear inventory, future maintenance could be challenging, leading to potential downtime or increased costs. + +4. **Security Risks**: Custom code can introduce security vulnerabilities if not properly reviewed and tested. Ignoring custom components might mean overlooking these risks. + +5. **Scalability Issues**: Custom components might have been designed with specific performance optimizations in mind. Losing them could affect the scalability of your search infrastructure, especially as data volumes grow. + +6. **Compliance and Auditing**: In some industries, custom components might be subject to compliance or auditing requirements. Failing to inventory these can lead to non-compliance issues. + +### Conclusion + +It is essential to thoroughly inventory all custom Solr components, regardless of their perceived importance. This ensures that the application's functionality remains intact, integration with other systems is seamless, and maintenance and support are manageable. Ignoring this claim could result in significant operational challenges down the line. \ No newline at end of file diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/sweep.py b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/sweep.py new file mode 100644 index 0000000000..077e384445 --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/sweep.py @@ -0,0 +1,368 @@ +#!/usr/bin/env python3 +""" +Bare-model triage sweep for stumper candidates. + +For each question in stumper-candidates.md, run it bare against: + - Anthropic Claude Haiku 4.5 (via `claude --print`, cwd=/tmp for isolation) + - Local Ollama qwen2.5:7b (via HTTP API) + +Apply each question's two contains-any criteria. Emit: + - responses/..txt raw response per question/model (latest) + - cache/.json addressable cache by hash(model + prompt) + - triage-results.md markdown table (qid x model x A/B pass) + +Caching: results are cached by sha256(model_id + "\n" + prompt). Edit a +prompt and the hash changes -> cache miss -> fresh call. Errors are +never cached; criteria are re-evaluated from cached responses every run +so you can iterate on assertions for free. + +Flags: + --refresh ignore cache for all models + --refresh-model NAME ignore cache for one model (haiku|qwen) + [ ...] only run these question ids + +Exit 0 always; this is a data-collection run, not a pass/fail gate. +""" +import hashlib +import json +import os +import re +import subprocess +import sys +import tempfile +import time +import urllib.error +import urllib.request +from pathlib import Path + +ROOT = Path(__file__).resolve().parent +RESP_DIR = ROOT / "responses" +RESP_DIR.mkdir(exist_ok=True) +CACHE_DIR = ROOT / "cache" +CACHE_DIR.mkdir(exist_ok=True) +RESULTS_MD = ROOT / "triage-results.md" + +HAIKU_MODEL = "claude-haiku-4-5-20251001" +OLLAMA_MODEL = "qwen2.5:7b" +OLLAMA_URL = "http://localhost:11434/api/generate" + +# Each question: id, short title, prompt, two criteria (a, b). +# Each criterion: ("any", [substrings]) -> case-insensitive substring match. +# Sourced verbatim from tests/evals/stumper-candidates.md. +STUMPERS = [ + { + "id": "q01-querqy-class", + "title": "Querqy OpenSearch class names", + "prompt": "What is the exact fully-qualified Java class name for registering a CommonRules rewriter in the OpenSearch Querqy plugin REST API? Give just the class string.", + "a": ("any", ["CommonRulesRewriterFactory"]), + "b": ("any", ["querqy.opensearch.rewriter"]), + }, + { + "id": "q02-querqy-infologging", + "title": "Querqy infoLogging field names", + "prompt": "In the OpenSearch Querqy plugin, what is the exact JSON field name you add inside a querqy query clause to enable debug logging of which rules fired? And what is the exact field name in the response that contains the decoration/rule-match data? Give just the two field names.", + "a": ("any", ["info_logging", "infoLogging"]), + "b": ("any", ["querqy_decorations", "decorations"]), + }, + { + "id": "q03-querqy-chain-order", + "title": "Querqy rewriter chain order mechanism", + "prompt": "In Solr, Querqy rewriter chain order is defined in querqy.xml. In the OpenSearch Querqy plugin, where is the chain order defined? Is it in a config file, the cluster state, or per-query? Give the exact mechanism.", + "a": ("any", ["per-query", "per query", "query time", "in the query"]), + "b": ("any", ["rewriters array", "rewriters list"]), + }, + { + "id": "q04-smui-deploy", + "title": "SMUI deployment mechanism change", + "prompt": "SMUI deploys Querqy rules to Solr by copying a rules text file to Solr's config directory and triggering a config reload. When targeting OpenSearch instead, what specific mechanism replaces the file-copy deployment? Name the exact API endpoint.", + "a": ("any", ["_querqy/rewriter", "_plugins/_querqy"]), + "b": ("any", ["PUT"]), + }, + { + "id": "q05-smui-opensearch-scope", + "title": "SMUI v4.0.11 OpenSearch scope (labelling vs deployment)", + "prompt": "SMUI v4.0.11 added OpenSearch support according to the release notes. As a migration consultant, what specifically did v4.0.11 add — full OpenSearch deployment parity with Solr, or something narrower? What does this mean for a team planning to use SMUI as their rules-management UI on OpenSearch?", + "a": ("any", ["label", "labelling", "labeling", "front-end", "frontend", "button", "cosmetic", "rename", "renaming", "ui-only", "ui only"]), + "b": ("any", ["doesn't deploy", "does not deploy", "no deployment", "not deploy", "manually", "still need", "additional work", "not full parity", "not at parity", "outside smui", "outside of smui", "lacks deployment", "no deploy"]), + }, + { + "id": "q06-querqy-down-conv", + "title": "Querqy DOWN -> negative_boost conversion", + "prompt": "In Querqy, a DOWN(500) rule penalizes matches. To translate this to OpenSearch's boosting query with negative_boost, what is the exact formula to convert the DOWN weight to a negative_boost value? Show the formula and the result for DOWN(500).", + "a": ("any", ["negative_boost"]), + "b": ("any", ["0.002", "0.001", "1/(1+", "1 / (1 +"]), + }, + { + "id": "q07-smui-export-format", + "title": "SMUI rule export format", + "prompt": "What is the format of a SMUI rules export file? Is it JSON, XML, or something else? Show an example of what the export looks like for a term with SYNONYM and UP rules.", + "a": ("any", ["plain text", "Common Rules", "text format"]), + "b": ("any", ["=>"]), + }, + { + "id": "q08-copyfield-multifields", + "title": "copyField -> multi-fields vs copy_to", + "prompt": "In our TMDB Solr schema, we have copyField rules that copy title to title_en (English analyzer) and title_bidirect_syn (bidirectional synonym analyzer). How should we represent these in OpenSearch? Should we use copy_to or multi-fields? Explain why.", + "a": ("any", ["multi-field", "multi field", "multifield", "fields\":"]), + "b": ("any", ["copy_to does not re-analyze", "copy_to copies raw", "not copy_to", "copy_to only copies", "copy_to is for"]), + }, + { + "id": "q09-pos-incr-gap", + "title": "positionIncrementGap default difference", + "prompt": "Solr's default positionIncrementGap is 0. What is OpenSearch's default position_increment_gap value? A client migrating from Solr has multi-valued text fields where they INTENTIONALLY allow phrase matches across value boundaries (positionIncrementGap=0). What specific OpenSearch mapping change preserves this behavior?", + "a": ("any", ["100", "default is 100", "defaults to 100"]), + "b": ("any", ["position_increment_gap"]), + }, + { + "id": "q10-dynamic-fields-limit", + "title": "Dynamic field mapping explosion limit", + "prompt": "Our Solr schema uses 800+ dynamic fields (*_s, *_i, *_txt, *_dt). What specific OpenSearch setting will we hit, what is its default value, and what are the risks of increasing it?", + "a": ("any", ["total_fields.limit", "total_fields"]), + "b": ("any", ["1000", "1,000"]), + }, + { + "id": "q11-localparams-arch", + "title": "LocalParams replacement architecture", + "prompt": "We use Solr LocalParams extensively: {!boost b=recip(ms(NOW,date),3.16e-11,1,1) v=$qq} and {!type=edismax qf='title^2 body' v=$qq}. OpenSearch has no LocalParams. Give the specific architectural pattern we should use to replace this — not just \"rewrite as Query DSL\".", + "a": ("any", ["search template", "search_template", "mustache"]), + "b": ("any", ["application layer", "application-layer", "query builder", "client-side", "client side"]), + }, + { + "id": "q12-search-pipelines", + "title": "Search Pipelines for conditional query rewrite", + "prompt": "Querqy's DELETE rule conditionally removes terms from a query (e.g., 'cheap laptop => DELETE: cheap'). Without the Querqy plugin in OpenSearch, name the specific OpenSearch 2.x feature that can intercept and modify queries before execution. What is it called and when was it introduced?", + "a": ("any", ["search pipeline"]), + "b": ("any", ["request processor", "script processor", "search request processor"]), + }, + { + "id": "q13-bf-vs-boost", + "title": "bf additive vs boost multiplicative", + "prompt": "In Solr eDisMax, what is the exact difference between bf and boost in terms of how they combine with the relevance score? For each, give the exact OpenSearch function_score boost_mode value.", + "a": ("any", ["additive", "adds to", "addition", "sum"]), + "b": ("any", ["multiplicative", "multiplies", "multiply", "multiplication"]), + }, + { + "id": "q14-tfidf-bm25-k1", + "title": "TF-IDF -> BM25 on short fields", + "prompt": "When migrating from Solr 6.x (ClassicSimilarity/TF-IDF) to OpenSearch (BM25), what specific ranking behavior changes on short text fields like product titles? Name the exact BM25 parameter that controls term frequency saturation and what value approximates TF-IDF behavior.", + "a": ("any", ["k1"]), + "b": ("any", ["saturation", "term frequency"]), + }, + { + "id": "q15-querqy-aws-managed", + "title": "Querqy NOT on AWS OpenSearch Service", + "prompt": "We use Querqy in Solr for query rewriting. We're migrating to AWS OpenSearch Service (the managed service). Can we install the Querqy plugin? If not, what must we do instead?", + "a": ("any", ["cannot", "can't", "not available", "not supported", "no.", "not allow"]), + "b": ("any", ["native", "bool", "should", "synonym"]), + }, + { + "id": "q16-collection-aliases", + "title": "Collection aliases write routing", + "prompt": "Our Solr uses collection aliases that route writes to the latest time-partitioned collection while searching across all. OpenSearch index aliases can search multiple indices but can only write to one. How do we handle our rolling-window architecture?", + "a": ("any", ["ISM", "Index State Management", "rollover"]), + "b": ("any", ["is_write_index"]), + }, + { + "id": "q17-atomic-no-source", + "title": "Atomic updates without _source", + "prompt": "Our Solr index has stored=false on most fields but supports atomic updates via docValues. We're migrating to OpenSearch. What specific problem will we hit with partial updates?", + "a": ("any", ["_source"]), + "b": ("any", ["cannot", "not possible", "requires", "must", "needs"]), + }, + { + "id": "q18-lucene-fallacy", + "title": "Lucene-underneath fallacy", + "prompt": "A client says \"It's all Lucene underneath, so most of it ports directly.\" As a migration consultant, give your response. Be specific about where this assumption breaks.", + "a": ("any", ["query parser", "parser"]), + "b": ("any", ["BM25", "scoring", "similarity"]), + }, + { + "id": "q19-mechanical-translation", + "title": "Mechanical translation fantasy", + "prompt": "A client says their Solr-to-OpenSearch migration is \"mostly config conversion work.\" Name 3 specific areas that require genuine redesign, not just translation.", + "a": ("any", ["redesign", "re-design", "rethink", "re-architect", "rearchitect"]), + "b": ("any", ["analyzer", "handler", "scoring", "relevance"]), + }, + { + "id": "q20-plugin-audit", + "title": "Plugin/custom handler dependency audit", + "prompt": "A client says \"We didn't think any custom Solr components were important.\" What specific component types should be inventoried, and why is this claim dangerous?", + "a": ("any", ["request handler", "requesthandler"]), + "b": ("any", ["update processor", "updateprocessor", "search component", "searchcomponent"]), + }, +] + + +def check(text: str, criterion) -> bool: + mode, needles = criterion + lower = text.lower() + hits = [n for n in needles if n.lower() in lower] + if mode == "any": + return len(hits) > 0 + if mode == "not_any": + return len(hits) == 0 + raise ValueError(f"unknown mode {mode}") + + +def _cache_path(model_id: str, prompt: str) -> Path: + key = hashlib.sha256(f"{model_id}\n{prompt}".encode("utf-8")).hexdigest() + return CACHE_DIR / f"{key}.json" + + +def _is_error_response(text: str) -> bool: + return text.startswith("[ERROR]") or text.startswith("[TIMEOUT]") or text == "[EMPTY]" + + +def cached_call(model_id: str, prompt: str, fn, refresh: bool) -> tuple[str, bool]: + """Return (response, was_cached). Errors never cache.""" + path = _cache_path(model_id, prompt) + if not refresh and path.exists(): + try: + data = json.loads(path.read_text()) + return data["response"], True + except Exception: + pass # corrupt cache entry; fall through and re-call + resp = fn(prompt) + if not _is_error_response(resp): + path.write_text(json.dumps({ + "model": model_id, + "prompt": prompt, + "response": resp, + "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"), + }, indent=2)) + return resp, False + + +def call_haiku(prompt: str, timeout: int = 90) -> str: + """Run Claude Haiku via the CLI from /tmp for isolation.""" + tmpdir = tempfile.mkdtemp(prefix="sma-haiku-") + try: + proc = subprocess.run( + ["claude", "--model", HAIKU_MODEL, "--print", prompt], + cwd=tmpdir, + capture_output=True, + text=True, + timeout=timeout, + ) + out = proc.stdout.strip() + if not out and proc.stderr: + out = f"[STDERR] {proc.stderr.strip()}" + return out + except subprocess.TimeoutExpired: + return "[TIMEOUT]" + except Exception as e: + return f"[ERROR] {e}" + + +def call_ollama(prompt: str, timeout: int = 180) -> str: + """Call local Ollama HTTP API; no streaming, no thinking spinners.""" + payload = json.dumps({ + "model": OLLAMA_MODEL, + "prompt": prompt, + "stream": False, + "options": {"temperature": 0.2, "num_predict": 1024}, + }).encode("utf-8") + req = urllib.request.Request( + OLLAMA_URL, + data=payload, + headers={"Content-Type": "application/json"}, + ) + try: + with urllib.request.urlopen(req, timeout=timeout) as r: + data = json.loads(r.read().decode("utf-8")) + return data.get("response", "").strip() or "[EMPTY]" + except urllib.error.URLError as e: + return f"[ERROR] {e}" + except Exception as e: + return f"[ERROR] {e}" + + +MODEL_IDS = {"haiku": HAIKU_MODEL, "qwen": OLLAMA_MODEL} +MODEL_FNS = {"haiku": call_haiku, "qwen": call_ollama} + + +def main(): + args = sys.argv[1:] + refresh_all = "--refresh" in args + refresh_models = set() + cleaned = [] + i = 0 + while i < len(args): + a = args[i] + if a == "--refresh": + i += 1 + continue + if a == "--refresh-model": + if i + 1 < len(args): + refresh_models.add(args[i + 1]) + i += 2 + continue + cleaned.append(a) + i += 1 + only = set(cleaned) # optional: filter by qid + + rows = [] + for q in STUMPERS: + qid = q["id"] + if only and qid not in only: + continue + print(f"\n=== {qid}: {q['title']} ===", flush=True) + + for label in ("haiku", "qwen"): + model_id = MODEL_IDS[label] + fn = MODEL_FNS[label] + refresh = refresh_all or (label in refresh_models) + t0 = time.time() + print(f" -> {label} ...", end="", flush=True) + resp, was_cached = cached_call(model_id, q["prompt"], fn, refresh) + elapsed = time.time() - t0 + (RESP_DIR / f"{qid}.{label}.txt").write_text(resp) + a_pass = check(resp, q["a"]) + b_pass = check(resp, q["b"]) + mark = lambda p: "GREEN" if p else "RED" + tag = "cache" if was_cached else "live" + print(f" {elapsed:.1f}s ({tag}) A={mark(a_pass)} B={mark(b_pass)}", flush=True) + rows.append({ + "qid": qid, + "title": q["title"], + "model": label, + "a": a_pass, + "b": b_pass, + "elapsed": elapsed, + "cached": was_cached, + }) + + # Emit markdown summary + lines = [ + "# Stumper triage — bare model results", + "", + "Both models run from `/tmp` with no skill discovery, no references.", + f"Targets: `{HAIKU_MODEL}` (cloud) and `{OLLAMA_MODEL}` (local Ollama).", + "Criteria are the contains-any patterns from `stumper-candidates.md`.", + "RED = criterion missed = candidate stumper for this model.", + "", + "| QID | Title | Haiku A | Haiku B | Qwen A | Qwen B | Both fail? |", + "| --- | ----- | :-----: | :-----: | :----: | :----: | :--------: |", + ] + by_qid = {} + for r in rows: + by_qid.setdefault(r["qid"], {"title": r["title"]})[r["model"]] = r + + for qid, group in by_qid.items(): + h = group.get("haiku", {}) + q = group.get("qwen", {}) + h_red = (not h.get("a", True)) or (not h.get("b", True)) + q_red = (not q.get("a", True)) or (not q.get("b", True)) + both = "YES" if (h_red and q_red) else ("haiku" if h_red else ("qwen" if q_red else "no")) + cell = lambda v: ("GREEN" if v else "RED") if v is not None else "-" + lines.append( + f"| {qid} | {group['title']} | {cell(h.get('a'))} | {cell(h.get('b'))} | {cell(q.get('a'))} | {cell(q.get('b'))} | {both} |" + ) + lines.append("") + lines.append(f"Generated {time.strftime('%Y-%m-%d %H:%M:%S')}.") + RESULTS_MD.write_text("\n".join(lines)) + print(f"\nWrote {RESULTS_MD}") + + +if __name__ == "__main__": + main() diff --git a/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/triage-results.md b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/triage-results.md new file mode 100644 index 0000000000..4957ef5dbf --- /dev/null +++ b/AIAdvisor/skills/solr-opensearch-migration-advisor/tests/evals/triage/triage-results.md @@ -0,0 +1,12 @@ +# Stumper triage — bare model results + +Both models run from `/tmp` with no skill discovery, no references. +Targets: `claude-haiku-4-5-20251001` (cloud) and `qwen2.5:7b` (local Ollama). +Criteria are the contains-any patterns from `stumper-candidates.md`. +RED = criterion missed = candidate stumper for this model. + +| QID | Title | Haiku A | Haiku B | Qwen A | Qwen B | Both fail? | +| --- | ----- | :-----: | :-----: | :----: | :----: | :--------: | +| q05-smui-opensearch-scope | SMUI v4.0.11 OpenSearch scope (labelling vs deployment) | RED | RED | RED | RED | YES | + +Generated 2026-05-10 16:37:22. \ No newline at end of file