Skip to content

Commit b0a3068

Browse files
suhstevesuhsteve
authored
Prep 2.0.0 release (#950)
1 parent 5f5215e commit b0a3068

File tree

5 files changed

+207
-8
lines changed

5 files changed

+207
-8
lines changed

‎README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -37,16 +37,16 @@
3737
</tr>
3838
</thead>
3939
<tbody align="center">
40-
<tr>
41-
<td >2.3</td>
42-
<td rowspan=3><a href="https://github.com/dotnet/spark/releases/tag/v1.1.1">v1.1.1</a></td>
43-
</tr>
4440
<tr>
4541
<td>2.4*</td>
42+
<td rowspan=3><a href="https://github.com/dotnet/spark/releases/tag/v2.0.0">v2.0.0</a></td>
4643
</tr>
4744
<tr>
4845
<td>3.0</td>
4946
</tr>
47+
<tr>
48+
<td>3.1</td>
49+
</tr>
5050
</tbody>
5151
</table>
5252

‎benchmark/scala/pom.xml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,13 @@
33
<modelVersion>4.0.0</modelVersion>
44
<groupId>com.microsoft.spark</groupId>
55
<artifactId>microsoft-spark-benchmark</artifactId>
6-
<version>1.1.1</version>
6+
<version>2.0.0</version>
77
<inceptionYear>2019</inceptionYear>
88
<properties>
99
<encoding>UTF-8</encoding>
1010
<scala.version>2.11.8</scala.version>
1111
<scala.binary.version>2.11</scala.binary.version>
12-
<spark.version>2.3.2</spark.version>
12+
<spark.version>2.4.4</spark.version>
1313
</properties>
1414

1515
<pluginRepositories>

‎docs/release-notes/2.0.0/release-2.0.0.md

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
# .NET for Apache Spark 2.0.0 Release Notes
2+
3+
### Deprecation for Spark 2.3
4+
5+
We have officially dropped support for Spark 2.3 in the 2.0.0 release. The last Spark 2.3 release (2.3.4) was back in September 2019, and no new release is planned for Spark 2.3. Since there have been no new features introduced for Spark 2.3 in the last few releases of .NET for Apache Spark, if you are relying on Spark 2.3, you should be able to continue using .NET for Apache Spark 1.x.
6+
7+
### New Features/Improvements
8+
9+
* Support for Spark 3.1.0 APIs ([#886](https://github.com/dotnet/spark/pull/886), [#887](https://github.com/dotnet/spark/pull/887), [#888](https://github.com/dotnet/spark/pull/888), [#889](https://github.com/dotnet/spark/pull/889), [#890](https://github.com/dotnet/spark/pull/890), [#893](https://github.com/dotnet/spark/pull/893))
10+
* Support for Spark 2.4.8, 3.1.2 ([#953](https://github.com/dotnet/spark/pull/953))
11+
* Add `Exists(string path)` to `Microsoft.Spark.Hadoop.Fs.FileSystem` ([#909](https://github.com/dotnet/spark/pull/909))
12+
* Add `Version()` to `Microsoft.Spark.SparkContext` and `Microsoft.Spark.Sql.SparkSession` ([#919](https://github.com/dotnet/spark/pull/919))
13+
* Add Avro `ToAvro`, `FromAvro` APIs ([#805](https://github.com/dotnet/spark/pull/805))
14+
* Extensions.DotNet.Interactive add ENV var to control disposal of tmp dir ([#952](https://github.com/dotnet/spark/pull/952))
15+
* Add public APIs to access `JvmBridge`, `JvmObjectReference`, `IJvmObjectReferenceProvider` ([#951](https://github.com/dotnet/spark/pull/951))
16+
> This is exposed to help users interact with the JVM. It is provided with limited support and should be used with caution.
17+
18+
### Bug Fixes
19+
20+
* None
21+
22+
### Infrastructure / Documentation / Etc.
23+
24+
* None
25+
26+
### Breaking Changes
27+
28+
* Add 'Z' to the string format in Timestamp.ToString() to indicate UTC time ([#897](https://github.com/dotnet/spark/pull/897))
29+
30+
### Known Issues
31+
32+
* Broadcast variables do not work with [dotnet-interactive](https://github.com/dotnet/interactive) ([#561](https://github.com/dotnet/spark/pull/561))
33+
* UDFs defined using class objects with closures does not work with [dotnet-interactive](https://github.com/dotnet/interactive) ([#619](https://github.com/dotnet/spark/pull/619))
34+
* In [dotnet-interactive](https://github.com/dotnet/interactive) blocking Spark methods that require external threads to unblock them does not work. ie `StreamingQuery.AwaitTermination` requires `StreamingQuery.Stop` to unblock ([#736](https://github.com/dotnet/spark/pull/736))
35+
36+
### Compatibility
37+
38+
#### Backward compatibility
39+
40+
The following table describes the oldest version of the worker that the current version is compatible with, along with new features that are incompatible with the worker.
41+
42+
<table>
43+
<thead>
44+
<tr>
45+
<th>Oldest compatible Microsoft.Spark.Worker version</th>
46+
</tr>
47+
</thead>
48+
<tbody align="center">
49+
<tr>
50+
<td>v2.0.0</td>
51+
</tr>
52+
</tbody>
53+
</table>
54+
55+
#### Forward compatibility
56+
57+
The following table describes the oldest version of .NET for Apache Spark release that the current worker is compatible with.
58+
59+
<table>
60+
<thead>
61+
<tr>
62+
<th>Oldest compatible .NET for Apache Spark release version</th>
63+
</tr>
64+
</thead>
65+
<tbody align="center">
66+
<tr>
67+
<td>v2.0.0</td>
68+
</tr>
69+
</tbody>
70+
</table>
71+
72+
### Supported Spark Versions
73+
74+
The following table outlines the supported Spark versions along with the microsoft-spark JAR to use with:
75+
76+
<table>
77+
<thead>
78+
<tr>
79+
<th>Spark Version</th>
80+
<th>microsoft-spark JAR</th>
81+
</tr>
82+
</thead>
83+
<tbody align="center">
84+
<tr>
85+
<td>2.4.0</td>
86+
<td rowspan=7>microsoft-spark-2-4_2.11-2.0.0.jar</td>
87+
</tr>
88+
<tr>
89+
<td>2.4.1</td>
90+
</tr>
91+
<tr>
92+
<td>2.4.3</td>
93+
</tr>
94+
<tr>
95+
<td>2.4.4</td>
96+
</tr>
97+
<tr>
98+
<td>2.4.5</td>
99+
</tr>
100+
<tr>
101+
<td>2.4.6</td>
102+
</tr>
103+
<tr>
104+
<td>2.4.7</td>
105+
</tr>
106+
<tr>
107+
<td>2.4.2</td>
108+
<td><a href="https://github.com/dotnet/spark/issues/60">Not supported</a></td>
109+
</tr>
110+
<tr>
111+
<td>3.0.0</td>
112+
<td rowspan=3>microsoft-spark-3-0_2.12-2.0.0.jar</td>
113+
</tr>
114+
<tr>
115+
<td>3.0.1</td>
116+
</tr>
117+
<tr>
118+
<td>3.0.2</td>
119+
</tr>
120+
<tr>
121+
<td>3.1.1</td>
122+
<td rowspan=3>microsoft-spark-3-1_2.12-2.0.0.jar</td>
123+
</tr>
124+
<tr>
125+
<td>3.1.2</td>
126+
</tr>
127+
</tbody>
128+
</table>
129+
130+
### Supported Delta Versions
131+
132+
The following table outlines the supported Delta versions along with the Microsoft.Spark.Extensions version to use with:
133+
134+
<table>
135+
<thead>
136+
<tr>
137+
<th>Delta Version</th>
138+
<th>Microsoft.Spark.Extensions.Delta</th>
139+
</tr>
140+
</thead>
141+
<tbody align="center">
142+
<tr>
143+
<td>0.1.0</td>
144+
<td rowspan=9>2.0.0</td>
145+
</tr>
146+
<tr>
147+
<td>0.2.0</td>
148+
</tr>
149+
<tr>
150+
<td>0.3.0</td>
151+
</tr>
152+
<tr>
153+
<td>0.4.0</td>
154+
</tr>
155+
<tr>
156+
<td>0.5.0</td>
157+
</tr>
158+
<tr>
159+
<td>0.6.0</td>
160+
</tr>
161+
<tr>
162+
<td>0.6.1</td>
163+
</tr>
164+
<tr>
165+
<td>0.7.0</td>
166+
</tr>
167+
<tr>
168+
<td>0.8.0</td>
169+
</tr>
170+
</tbody>
171+
</table>
172+
173+
### Supported Hyperspace Versions
174+
175+
The following table outlines the supported Hyperspace versions along with the Microsoft.Spark.Extensions version to use with:
176+
177+
<table>
178+
<thead>
179+
<tr>
180+
<th>Hyperspace Version</th>
181+
<th>Microsoft.Spark.Extensions.Hyperspace</th>
182+
</tr>
183+
</thead>
184+
<tbody align="center">
185+
<tr>
186+
<td>0.1.0</td>
187+
<td rowspan=4>2.0.0</td>
188+
</tr>
189+
<tr>
190+
<td>0.2.0</td>
191+
</tr>
192+
<tr>
193+
<td>0.3.0</td>
194+
</tr>
195+
<tr>
196+
<td>0.4.0</td>
197+
</tr>
198+
</tbody>
199+
</table>

‎eng/Versions.props

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
<?xml version="1.0" encoding="utf-8"?>
22
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
33
<PropertyGroup>
4-
<VersionPrefix>1.1.1</VersionPrefix>
4+
<VersionPrefix>2.0.0</VersionPrefix>
55
<PreReleaseVersionLabel>prerelease</PreReleaseVersionLabel>
66
</PropertyGroup>
77
</Project>

‎src/scala/pom.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
<version>${microsoft-spark.version}</version>
88
<properties>
99
<encoding>UTF-8</encoding>
10-
<microsoft-spark.version>1.1.1</microsoft-spark.version>
10+
<microsoft-spark.version>2.0.0</microsoft-spark.version>
1111
</properties>
1212

1313
<modules>

0 commit comments

Comments
 (0)