Skip to content

Commit 7d2bed7

Browse files
maziyarpanahimaziyarpanahi
authored
Merge pull request #14511 from JohnSnowLabs/release/553-release-candidate
Release/553 release candidate
2 parents d64fd69 + 7137acb commit 7d2bed7

File tree

1,773 files changed

+53092
-13379
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

1,773 files changed

+53092
-13379
lines changed

‎CHANGELOG

+14
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,17 @@
1+
========
2+
5.5.3
3+
========
4+
----------------
5+
Bug Fixes & Enhancements
6+
----------------
7+
* BGEEmbeddings: The default pretrained model for BGEEmbeddings has been changed from "bge_base" to "bge_small_en_v1.5". Users relying on the old default will need to explicitly specify "bge_base" in the pretrained method.
8+
* Added useCLSToken parameter to allow users to choose between CLS token pooling and attention-based average pooling for sentence embeddings.
9+
* BGEEmbeddings: BGEEmbeddings now supports a `useCLSToken` parameter, which defaults to True. This affects the embedding calculation strategy. Existing users should verify their usage and potentially set this parameter explicitly.
10+
* Added HasClsTokenProperties in Scala: Introduced the HasClsTokenProperties trait in Scala providing useCLSToken parameter functionality for relevant annotators.
11+
* Fixing wrong padding in attention mask in `MPNet`, `BGE`, `E5`, `Mxbai`, `Nomic`, `SnowFlake`, and `UAE`. This resulted in wrong inference results in some cases and not equal to the ONNX version in transformers/sentence-transformers.
12+
* Various Performance Optimizations: Multiple changes across different models (Albert, Bart, CLIP, CamemBert, ConvNextClassifier, DeBerta, DistilBert, E5, Instructor, MPNet, Mxbai, Nomic, RoBerta, SnowFlake, UAE, ViTClassifier, VisionEncoderDecoder, Wav2Vec2, XlmRoBertaClassification, XlmRoberta) appear to focus on performance improvements and code cleanup, especially related to OpenVINO and ONNX inference. These may lead to faster inference times.
13+
* Fixing Llama3 download issue in Python.
14+
115
========
216
5.5.2
317
========

‎README.md

+7-7
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ $ java -version
6363
$ conda create -n sparknlp python=3.7 -y
6464
$ conda activate sparknlp
6565
# spark-nlp by default is based on pyspark 3.x
66-
$ pip install spark-nlp==5.5.2 pyspark==3.3.1
66+
$ pip install spark-nlp==5.5.3 pyspark==3.3.1
6767
```
6868

6969
In Python console or Jupyter `Python3` kernel:
@@ -129,7 +129,7 @@ For a quick example of using pipelines and models take a look at our official [d
129129

130130
### Apache Spark Support
131131

132-
Spark NLP *5.5.2* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
132+
Spark NLP *5.5.3* has been built on top of Apache Spark 3.4 while fully supports Apache Spark 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x
133133

134134
| Spark NLP | Apache Spark 3.5.x | Apache Spark 3.4.x | Apache Spark 3.3.x | Apache Spark 3.2.x | Apache Spark 3.1.x | Apache Spark 3.0.x | Apache Spark 2.4.x | Apache Spark 2.3.x |
135135
|-----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|
@@ -157,7 +157,7 @@ Find out more about 4.x `SparkNLP` versions in our official [documentation](http
157157

158158
### Databricks Support
159159

160-
Spark NLP 5.5.2 has been tested and is compatible with the following runtimes:
160+
Spark NLP 5.5.3 has been tested and is compatible with the following runtimes:
161161

162162
| **CPU** | **GPU** |
163163
|--------------------|--------------------|
@@ -174,7 +174,7 @@ We are compatible with older runtimes. For a full list check databricks support
174174

175175
### EMR Support
176176

177-
Spark NLP 5.5.2 has been tested and is compatible with the following EMR releases:
177+
Spark NLP 5.5.3 has been tested and is compatible with the following EMR releases:
178178

179179
| **EMR Release** |
180180
|--------------------|
@@ -205,7 +205,7 @@ deployed to Maven central. To add any of our packages as a dependency in your ap
205205
from our official documentation.
206206

207207
If you are interested, there is a simple SBT project for Spark NLP to guide you on how to use it in your
208-
projects [Spark NLP SBT S5.5.2r](https://github.com/maziyarpanahi/spark-nlp-starter)
208+
projects [Spark NLP SBT S5.5.3r](https://github.com/maziyarpanahi/spark-nlp-starter)
209209

210210
### Python
211211

@@ -250,7 +250,7 @@ In Spark NLP we can define S3 locations to:
250250

251251
Please check [these instructions](https://sparknlp.org/docs/en/install#s3-integration) from our official documentation.
252252

253-
## Document5.5.2
253+
## Document5.5.3
254254

255255
### Examples
256256

@@ -283,7 +283,7 @@ the Spark NLP library:
283283
keywords = {Spark, Natural language processing, Deep learning, Tensorflow, Cluster},
284284
abstract = {Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant & accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100+ pretrained pipelines and models in more than 192+ languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing 9x growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the world’s most widely used NLP library in the enterprise.}
285285
}
286-
}5.5.2
286+
}5.5.3
287287
```
288288

289289
## Community support

‎build.sbt

+1-1
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ name := getPackageName(is_silicon, is_gpu, is_aarch64)
66

77
organization := "com.johnsnowlabs.nlp"
88

9-
version := "5.5.2"
9+
version := "5.5.3"
1010

1111
(ThisBuild / scalaVersion) := scalaVer
1212

‎conda/meta.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{% set name = "spark-nlp" %}
2-
{% set version = "5.5.2" %}
2+
{% set version = "5.5.3" %}
33

44
package:
55
name: {{ name|lower }}

‎docs/_layouts/landing.html

+1-1
Original file line numberDiff line numberDiff line change
@@ -201,7 +201,7 @@ <h3 class="grey h3_title">{{ _section.title }}</h3>
201201
<div class="highlight-box">
202202
{% highlight bash %}
203203
# Using PyPI
204-
$ pip install spark-nlp==5.5.2
204+
$ pip install spark-nlp==5.5.3
205205

206206
# Using Anaconda/Conda
207207
$ conda install -c johnsnowlabs spark-nlp

‎docs/api/com/index.html

+4-4
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
<head>
44
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
55
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
6-
<title>Spark NLP 5.5.1 ScalaDoc - com</title>
7-
<meta name="description" content="Spark NLP 5.5.1 ScalaDoc - com" />
8-
<meta name="keywords" content="Spark NLP 5.5.1 ScalaDoc com" />
6+
<title>Spark NLP 5.5.3 ScalaDoc - com</title>
7+
<meta name="description" content="Spark NLP 5.5.3 ScalaDoc - com" />
8+
<meta name="keywords" content="Spark NLP 5.5.3 ScalaDoc com" />
99
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
1010

1111

@@ -28,7 +28,7 @@
2828
</head>
2929
<body>
3030
<div id="search">
31-
<span id="doc-title">Spark NLP 5.5.1 ScalaDoc<span id="doc-version"></span></span>
31+
<span id="doc-title">Spark NLP 5.5.3 ScalaDoc<span id="doc-version"></span></span>
3232
<span class="close-results"><span class="left">&lt;</span> Back</span>
3333
<div id="textfilter">
3434
<span class="input">

‎docs/api/com/johnsnowlabs/client/CloudClient.html

+4-4
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
<head>
44
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
55
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
6-
<title>Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.CloudClient</title>
7-
<meta name="description" content="Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.CloudClient" />
8-
<meta name="keywords" content="Spark NLP 5.5.1 ScalaDoc com.johnsnowlabs.client.CloudClient" />
6+
<title>Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.CloudClient</title>
7+
<meta name="description" content="Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.CloudClient" />
8+
<meta name="keywords" content="Spark NLP 5.5.3 ScalaDoc com.johnsnowlabs.client.CloudClient" />
99
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
1010

1111

@@ -28,7 +28,7 @@
2828
</head>
2929
<body>
3030
<div id="search">
31-
<span id="doc-title">Spark NLP 5.5.1 ScalaDoc<span id="doc-version"></span></span>
31+
<span id="doc-title">Spark NLP 5.5.3 ScalaDoc<span id="doc-version"></span></span>
3232
<span class="close-results"><span class="left">&lt;</span> Back</span>
3333
<div id="textfilter">
3434
<span class="input">

‎docs/api/com/johnsnowlabs/client/CloudManager.html

+4-4
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
<head>
44
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
55
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
6-
<title>Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.CloudManager</title>
7-
<meta name="description" content="Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.CloudManager" />
8-
<meta name="keywords" content="Spark NLP 5.5.1 ScalaDoc com.johnsnowlabs.client.CloudManager" />
6+
<title>Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.CloudManager</title>
7+
<meta name="description" content="Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.CloudManager" />
8+
<meta name="keywords" content="Spark NLP 5.5.3 ScalaDoc com.johnsnowlabs.client.CloudManager" />
99
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
1010

1111

@@ -28,7 +28,7 @@
2828
</head>
2929
<body>
3030
<div id="search">
31-
<span id="doc-title">Spark NLP 5.5.1 ScalaDoc<span id="doc-version"></span></span>
31+
<span id="doc-title">Spark NLP 5.5.3 ScalaDoc<span id="doc-version"></span></span>
3232
<span class="close-results"><span class="left">&lt;</span> Back</span>
3333
<div id="textfilter">
3434
<span class="input">

‎docs/api/com/johnsnowlabs/client/CloudResources$.html

+4-4
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
<head>
44
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
55
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
6-
<title>Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.CloudResources</title>
7-
<meta name="description" content="Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.CloudResources" />
8-
<meta name="keywords" content="Spark NLP 5.5.1 ScalaDoc com.johnsnowlabs.client.CloudResources" />
6+
<title>Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.CloudResources</title>
7+
<meta name="description" content="Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.CloudResources" />
8+
<meta name="keywords" content="Spark NLP 5.5.3 ScalaDoc com.johnsnowlabs.client.CloudResources" />
99
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
1010

1111

@@ -28,7 +28,7 @@
2828
</head>
2929
<body>
3030
<div id="search">
31-
<span id="doc-title">Spark NLP 5.5.1 ScalaDoc<span id="doc-version"></span></span>
31+
<span id="doc-title">Spark NLP 5.5.3 ScalaDoc<span id="doc-version"></span></span>
3232
<span class="close-results"><span class="left">&lt;</span> Back</span>
3333
<div id="textfilter">
3434
<span class="input">

‎docs/api/com/johnsnowlabs/client/CloudStorage.html

+4-4
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
<head>
44
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
55
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
6-
<title>Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.CloudStorage</title>
7-
<meta name="description" content="Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.CloudStorage" />
8-
<meta name="keywords" content="Spark NLP 5.5.1 ScalaDoc com.johnsnowlabs.client.CloudStorage" />
6+
<title>Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.CloudStorage</title>
7+
<meta name="description" content="Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.CloudStorage" />
8+
<meta name="keywords" content="Spark NLP 5.5.3 ScalaDoc com.johnsnowlabs.client.CloudStorage" />
99
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
1010

1111

@@ -28,7 +28,7 @@
2828
</head>
2929
<body>
3030
<div id="search">
31-
<span id="doc-title">Spark NLP 5.5.1 ScalaDoc<span id="doc-version"></span></span>
31+
<span id="doc-title">Spark NLP 5.5.3 ScalaDoc<span id="doc-version"></span></span>
3232
<span class="close-results"><span class="left">&lt;</span> Back</span>
3333
<div id="textfilter">
3434
<span class="input">

‎docs/api/com/johnsnowlabs/client/aws/AWSAnonymousCredentials.html

+4-4
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
<head>
44
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
55
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
6-
<title>Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSAnonymousCredentials</title>
7-
<meta name="description" content="Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSAnonymousCredentials" />
8-
<meta name="keywords" content="Spark NLP 5.5.1 ScalaDoc com.johnsnowlabs.client.aws.AWSAnonymousCredentials" />
6+
<title>Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.aws.AWSAnonymousCredentials</title>
7+
<meta name="description" content="Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.aws.AWSAnonymousCredentials" />
8+
<meta name="keywords" content="Spark NLP 5.5.3 ScalaDoc com.johnsnowlabs.client.aws.AWSAnonymousCredentials" />
99
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
1010

1111

@@ -28,7 +28,7 @@
2828
</head>
2929
<body>
3030
<div id="search">
31-
<span id="doc-title">Spark NLP 5.5.1 ScalaDoc<span id="doc-version"></span></span>
31+
<span id="doc-title">Spark NLP 5.5.3 ScalaDoc<span id="doc-version"></span></span>
3232
<span class="close-results"><span class="left">&lt;</span> Back</span>
3333
<div id="textfilter">
3434
<span class="input">

‎docs/api/com/johnsnowlabs/client/aws/AWSBasicCredentials.html

+4-4
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
<head>
44
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
55
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
6-
<title>Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSBasicCredentials</title>
7-
<meta name="description" content="Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSBasicCredentials" />
8-
<meta name="keywords" content="Spark NLP 5.5.1 ScalaDoc com.johnsnowlabs.client.aws.AWSBasicCredentials" />
6+
<title>Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.aws.AWSBasicCredentials</title>
7+
<meta name="description" content="Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.aws.AWSBasicCredentials" />
8+
<meta name="keywords" content="Spark NLP 5.5.3 ScalaDoc com.johnsnowlabs.client.aws.AWSBasicCredentials" />
99
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
1010

1111

@@ -28,7 +28,7 @@
2828
</head>
2929
<body>
3030
<div id="search">
31-
<span id="doc-title">Spark NLP 5.5.1 ScalaDoc<span id="doc-version"></span></span>
31+
<span id="doc-title">Spark NLP 5.5.3 ScalaDoc<span id="doc-version"></span></span>
3232
<span class="close-results"><span class="left">&lt;</span> Back</span>
3333
<div id="textfilter">
3434
<span class="input">

‎docs/api/com/johnsnowlabs/client/aws/AWSClient.html

+4-4
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
<head>
44
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
55
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
6-
<title>Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSClient</title>
7-
<meta name="description" content="Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSClient" />
8-
<meta name="keywords" content="Spark NLP 5.5.1 ScalaDoc com.johnsnowlabs.client.aws.AWSClient" />
6+
<title>Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.aws.AWSClient</title>
7+
<meta name="description" content="Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.aws.AWSClient" />
8+
<meta name="keywords" content="Spark NLP 5.5.3 ScalaDoc com.johnsnowlabs.client.aws.AWSClient" />
99
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
1010

1111

@@ -28,7 +28,7 @@
2828
</head>
2929
<body>
3030
<div id="search">
31-
<span id="doc-title">Spark NLP 5.5.1 ScalaDoc<span id="doc-version"></span></span>
31+
<span id="doc-title">Spark NLP 5.5.3 ScalaDoc<span id="doc-version"></span></span>
3232
<span class="close-results"><span class="left">&lt;</span> Back</span>
3333
<div id="textfilter">
3434
<span class="input">

‎docs/api/com/johnsnowlabs/client/aws/AWSCredentialsProvider.html

+4-4
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
<head>
44
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
55
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
6-
<title>Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSCredentialsProvider</title>
7-
<meta name="description" content="Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSCredentialsProvider" />
8-
<meta name="keywords" content="Spark NLP 5.5.1 ScalaDoc com.johnsnowlabs.client.aws.AWSCredentialsProvider" />
6+
<title>Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.aws.AWSCredentialsProvider</title>
7+
<meta name="description" content="Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.aws.AWSCredentialsProvider" />
8+
<meta name="keywords" content="Spark NLP 5.5.3 ScalaDoc com.johnsnowlabs.client.aws.AWSCredentialsProvider" />
99
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
1010

1111

@@ -28,7 +28,7 @@
2828
</head>
2929
<body>
3030
<div id="search">
31-
<span id="doc-title">Spark NLP 5.5.1 ScalaDoc<span id="doc-version"></span></span>
31+
<span id="doc-title">Spark NLP 5.5.3 ScalaDoc<span id="doc-version"></span></span>
3232
<span class="close-results"><span class="left">&lt;</span> Back</span>
3333
<div id="textfilter">
3434
<span class="input">

‎docs/api/com/johnsnowlabs/client/aws/AWSGateway.html

+4-4
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
<head>
44
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
55
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
6-
<title>Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSGateway</title>
7-
<meta name="description" content="Spark NLP 5.5.1 ScalaDoc - com.johnsnowlabs.client.aws.AWSGateway" />
8-
<meta name="keywords" content="Spark NLP 5.5.1 ScalaDoc com.johnsnowlabs.client.aws.AWSGateway" />
6+
<title>Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.aws.AWSGateway</title>
7+
<meta name="description" content="Spark NLP 5.5.3 ScalaDoc - com.johnsnowlabs.client.aws.AWSGateway" />
8+
<meta name="keywords" content="Spark NLP 5.5.3 ScalaDoc com.johnsnowlabs.client.aws.AWSGateway" />
99
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
1010

1111

@@ -28,7 +28,7 @@
2828
</head>
2929
<body>
3030
<div id="search">
31-
<span id="doc-title">Spark NLP 5.5.1 ScalaDoc<span id="doc-version"></span></span>
31+
<span id="doc-title">Spark NLP 5.5.3 ScalaDoc<span id="doc-version"></span></span>
3232
<span class="close-results"><span class="left">&lt;</span> Back</span>
3333
<div id="textfilter">
3434
<span class="input">

0 commit comments

Comments
 (0)