You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
meta_description: Learn about zscore normalization using the Neural Search plugin in OpenSearch 3.0-beta1. Discover how this new approach to hybrid search merges results from multiple query sources for improved relevance.
meta_description: Learn about z-score normalization in OpenSearch 3.0-beta1. Discover how this new approach to hybrid search merges results from multiple query sources for improved relevance.
14
15
---
15
16
16
-
In the world of search engines and machine learning, data normalization plays a crucial role in ensuring fair and accurate comparisons between different features or scores.
17
-
Hybrid query uses multiple normalization techniques for preparing final results, main two types are score based normalization and rank base combination. In score base normalization, min-max normalization(default normalization technique) doesn’t work well with outliers (Outliers are data points that significantly differ from other observations in a dataset.
18
-
In the context of normalization techniques like Min-Max scaling and Z-score (Standard Score) normalization, outliers can have a substantial impact on the results). In this blogpost we would introduce another normalization technique called as z-score which was added in OpenSearch 3.0-beta1 release.
19
-
Let's dive into what Z-score normalization is, why it's important, and how it's being used in OpenSearch.
17
+
In search engines and machine learning, **data normalization** is essential for making fair and accurate comparisons between different features or scores.
20
18
21
-
## What is Z-Score Normalization?
19
+
**Hybrid queries** use multiple normalization techniques to prepare final results. The two main types are **score-based normalization** and **rank-based combination**. In score-based normalization, the default method—**min-max normalization**—doesn't handle outliers well. *(Outliers are data points that significantly differ from the rest of a dataset. In techniques like min-max scaling and Z-score normalization, outliers can greatly impact results.)*
22
20
23
-
Z-score normalization, also known as standardization, is a method of scaling data using mean and standard deviation. The formula for calculating the Z-score is:
24
-
Z = (X - μ) / σ
25
-
Where:
21
+
In this blog post, we'll introduce **z-score normalization**, a technique added in OpenSearch 3.0-beta1. We'll explore what it is, why it matters, and how OpenSearch uses it.
22
+
23
+
## What is z-score normalization?
24
+
25
+
_Z-score normalization_, also known as _standardization_, is a method of scaling data using mean and standard deviation. The formula for calculating the Z-score is:
26
+
27
+
Z = (X - μ) / σ,
28
+
29
+
where:
26
30
27
31
* X is the original value
28
-
* μ is the mean of the population
32
+
* μ is the population mean
29
33
* σ is the standard deviation of the population
30
34
31
-
## When to use Z Score?
35
+
## When to use z-score normalization
32
36
33
-
Considering your index’s structure can help you decide which one to choose since each has advantages of its own. If your documents are more similar to one another and the top-k results of a typical query return documents that are very similar to one another and clustered together within the index, as seen in the graph below, Min-Max may be a better option.
37
+
Your index's structure can help you decide which normalization technique to use, as each has its own advantages. If your documents are highly similar and a typical query returns top-k results that are closely clustered together within the index (as shown in the following graph), min-max normalization may be the better choice.
34
38
35
-
{: .img-fluid}
39
+
{: .img-fluid}
36
40
37
-
However, Z-Score is more suited if the results are more evenly distributed and have some characteristics of a normal distribution, as shown in the example below.
41
+
However, z-score normalization is a better choice if the results are more evenly distributed and follow a pattern similar to a normal distribution, as shown in the following graph.
38
42
39
43
{: .img-fluid}
40
44
41
-
The basic flow to use between minmax and zscore looks like below:
45
+
The basic process for choosing between min-max and z-score normalization is outlined in the following diagram.
42
46
43
47
{: .img-fluid}
44
48
45
-
###How to use Z Score?
49
+
## How to use z-score normalization
46
50
47
-
To use z_score, create a search pipeline and specify `z_score` as the `technique`:
51
+
To use z-score normalization, create a search pipeline and specify `z_score` as the `technique`:
48
52
49
-
```
53
+
```json
50
54
PUT /_search/pipeline/z_score-pipeline
51
55
{
52
56
"description": "Zscore processor for hybrid search",
@@ -62,11 +66,9 @@ PUT /_search/pipeline/z_score-pipeline
62
66
}
63
67
```
64
68
65
-
Next, create a hybrid query and apply the pipeline to it
66
-
67
-
```
68
-
69
+
Next, create a hybrid query and apply the pipeline to the query:
69
70
71
+
```json
70
72
POST my_index/_search?search_pipeline=z_score-pipeline
71
73
{
72
74
"query": {
@@ -79,94 +81,89 @@ POST my_index/_search?search_pipeline=z_score-pipeline
79
81
}
80
82
```
81
83
84
+
## Benchmarking z-score normalization performance
82
85
86
+
Benchmark experiments were conducted on an OpenSearch cluster with one **r6g.8xlarge** instance as the coordinator node and three **r6g.8xlarge** instances as data nodes. To evaluate z-score normalization comprehensively, we measured two key metrics across five distinct datasets.
83
87
84
-
## Benchmarking Z Score performance
85
-
86
-
Benchmark experiments were conducted using an OpenSearch cluster consisting of a single r6g.8xlarge instance as the coordinator node, along with three r6g.8xlarge instances as data nodes. To assess Z Score’s performance comprehensively, we measured two key metrics across five distinct datasets. For information about the datasets used, see [Datasets](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/).
88
+
For more information about the datasets used, see [Datasets](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/).
87
89
88
90
### Sample queries and passages
89
91
90
92
The following table provides sample queries and passages for each dataset.
91
93
92
94
|Dataset |Sample query |Sample passage |
93
95
|:--- |:--- |:--- |
94
-
|Scidocs|CFD Analysis of Convective Heat Transfer Coefficient on External Surfaces of Buildings |`This paper provides an overview of the application of CFD in building performance simulation for the outdoor environment, focused on four topics...`|
96
+
|SciDocs|CFD Analysis of Convective Heat Transfer Coefficient on External Surfaces of Buildings |`This paper provides an overview of the application of CFD in building performance simulation for the outdoor environment, focused on four topics...`|
95
97
|FiQA |“Business day” and “due date” for bills |`I don't believe Saturday is a business day either. When I deposit a check at a bank's drive-in after 4pm Friday, the receipt tells me it will credit as if I deposited on Monday. If a business' computer doesn't adjust their billing to have a weekday due date ... `|
96
-
|nq|what is non controlling interest on balance sheet |`In accounting, minority interest (or non-controlling interest) is the portion of a subsidiary corporation's stock that is not owned by the parent corporation. The magnitude of the minority interest in the subsidiary company is generally less than 50% of outstanding shares, or the corporation would generally cease to be a subsidiary of the parent`|
98
+
|NQ|what is non controlling interest on balance sheet |`In accounting, minority interest (or non-controlling interest) is the portion of a subsidiary corporation's stock that is not owned by the parent corporation. The magnitude of the minority interest in the subsidiary company is generally less than 50% of outstanding shares, or the corporation would generally cease to be a subsidiary of the parent`|
97
99
|ArguAna |Poaching is becoming more advanced A stronger, militarised approach is needed as poaching is becoming ... |`Tougher protection of Africa\u2019s nature reserves will only result in more bloodshed. Every time the military upgrade their weaponry, tactics and logistic, the poachers improve their own methods to counter ...`|
98
100
|touche2020 |Is a college education worth it? |`The resolution used by Pro *assumes* that Australia isn't already a 'significant' country - however, in actual reality, it is. Firstly we should clarify what significance means: 1.a the state or quality of being significant1.b of consequence or..`|
99
101
100
102
101
-
Search relevance was quantified using the industry-standard Normalized Discounted Cumulative Gain at rank 10 (NDCG@10). We also tracked system performance using search latency measurements. This setup provided a strong foundation for evaluating both search quality and operational efficiency.
103
+
Search relevance was quantified using the industry-standard normalized discounted cumulative gain at rank 10 (NDCG@10). We also tracked system performance using search latency measurements. This setup provided a strong foundation for evaluating both search quality and operational efficiency.
102
104
103
105
104
106
### NDCG@10
105
107
106
-
|dataset |Hybrid (min max) |Hybrid (z score) |Percent diff |
108
+
The following table compares NDCG@10 scores for hybrid search using min-max normalization and z-score normalization, showing the percent difference in search relevance across five datasets.
The following table presents search latency measurements in milliseconds at different percentiles (p50, p90, and p99) for both the Hybrid with minmax and zscore approaches. The *Percent difference* columns show the relative performance impact between these methods.
122
+
The following table presents search latency measurements in milliseconds at different percentiles (p50, p90, and p99) for hybrid search using min-max and z-score normalization. The *Percent difference* columns indicate the relative performance impact between these methods.
Our benchmark experiments highlight the following advantages and trade-offs of using z-score normalization compared to min-max normalization in hybrid search:
124
129
125
-
Our benchmark experiments highlight the following advantages and trade-offs of Z-score normalization compared to min-max normalization in hybrid search approaches:
126
-
127
-
**Search quality (measured using NDCG@10 across four datasets)**:
128
-
129
-
* Z-score normalization shows a modest improvement in search quality, with an average increase of 2.08% in NDCG@10 scores.
130
-
* This suggests that Z-score normalization may provide slightly better relevance in search results compared to the default normalization technique min-max.
131
-
132
-
133
-
**Latency impact**:
134
-
135
-
* Z-score normalization shows a small increase in latency across different percentiles, as shown in the following table:
136
-
137
-
|Latency percentile |Percent difference |
138
-
|--- |--- |
139
-
|p50 |0.70% |
140
-
|p90 |0.50% |
141
-
|p99 |0.50% |
130
+
-**Search quality (measured using NDCG@10 across four datasets)**:
142
131
143
-
* The positive percentages indicate that Z-score normalization has slightly higher latency compared to min-max normalization, but the differences are minimal (less than 1% on average).
132
+
* Z-score normalization shows a modest improvement in search quality, with an average increase of 2.08% in NDCG@10 scores.
133
+
* This suggests that Z-score normalization may provide slightly better relevance in search results compared to the default min-max normalization.
144
134
145
-
**Trade-offs**:
146
135
147
-
* There's a slight trade-off between search quality and latency. Z-score normalization offers a improvement in search relevance (2.08% increase in NDCG@10) at the cost of a marginal increase in latency (0.50% to 0.72% across different percentiles).
136
+
-**Latency impact**:
148
137
149
-
**Overall assessment**:
138
+
* Z-score normalization shows a small increase in latency across different percentiles, as shown in the following table.
150
139
151
-
* Z-score normalization provides a modest improvement in search quality with a negligible impact on latency.
152
-
* The choice between Z-score and min-max normalization may depend on specific use cases, with Z-score potentially being preferred where improvements in search relevance are valuable and the slight latency increase is acceptable.
140
+
|Latency percentile |Percent difference |
141
+
|--- |--- |
142
+
|p50 |0.70% |
143
+
|p90 |0.50% |
144
+
|p99 |0.50% |
153
145
154
-
These findings suggest that Z-score normalization could be a viable alternative to min-max normalization in hybrid search approaches, particularly in scenarios where optimizing search relevance is a priority and the system can tolerate minimal latency increases
146
+
* The positive percentages indicate that Z-score normalization has slightly higher latency compared to min-max normalization, but the differences are minimal (less than 1% on average).
155
147
148
+
-**Trade-offs**:
156
149
150
+
* There's a slight trade-off between search quality and latency. Z-score normalization offers an improvement in search relevance (2.08% increase in NDCG@10) at the cost of a marginal increase in latency (0.50% to 0.72% across different percentiles).
157
151
158
-
## What’s next?
152
+
-**Overall assessment**:
159
153
160
-
We are also expanding OpenSearch’s hybrid search capabilities beyond z score by planning the following enhancements to our normalization framework:
154
+
* Z-score normalization provides a modest improvement in search quality with a negligible impact on latency.
155
+
* The choice between Z-score and min-max normalization may depend on your use case. If search relevance is a priority, z-score normalization may be the better option, especially if your system can tolerate a small latency increase.
161
156
162
-
**Custom normalization functions**: Enables you to define your own normalization logic and allows fine-tuning of search result rankings. For more information, see [this issue](https://github.com/opensearch-project/neural-search/issues/994).
157
+
These findings suggest that z-score normalization can be a viable alternative to min-max normalization in hybrid search, particularly when optimizing search relevance is a priority and the system can tolerate minimal latency increases.
163
158
164
-
These enhancements will provide more control over search result ranking while ensuring reliable and consistent hybrid search outcomes. Stay tuned for more information!
159
+
## What's next?
165
160
161
+
We're continuing to enhance OpenSearch's hybrid search capabilities beyond Z-score by planning to implement **custom normalization functions**. These will enable you to define your own normalization logic and allow fine-tuning of search result rankings. For more information, see [this issue](https://github.com/opensearch-project/neural-search/issues/994).
166
162
163
+
This enhancement will provide more control over search result ranking while ensuring reliable and consistent hybrid search outcomes. Stay tuned for more information!
0 commit comments