Can we breathe life back into this project?

[GoEddie]

Hi All,

It is pretty obvious that the project has come to a bit of a halt and I wondered if there was anything that we can do to get it up and running again?

I don't know the reason why it stopped in February, maybe if we knew that we could support it in some way?

I do know that I have had multiple orgs who wanted to use .NET and Spark in Databricks but in all honestly I couldn't recommend using it as it seems the project is now dead.

What would it take to bring it up to date with both .NET 7 support and support up to Apache Spark 3.4.1?

Is Microsoft willing to invest in the project or is the community able to take it on and progress it?

I'm just hoping to start a discussion really as it is a shame so much work went into this and we were so close to being able to use .NET instead of Python or Scala but it feels frustrating that we can't use .NET.

cc: @MikeRys @AFFogarty @suhsteve

@GeorgeS2019

[GeorgeS2019]

Update 30th Aug 2023

#1160

[GoEddie]

Update 30th Aug 2023

#1160

but is there any way to bring it back to life?

[luisquintanilla]

@GoEddie @GeorgeS2019 since the project is open-source and part of the .NET Foundation is this something the community would be interested in contributing to in order to help move it forward?

[GeorgeS2019]

@luisquintanilla We need to identify members here who are interested to maintain and merge the e.g. PR

For members here who are interested, please let @luisquintanilla know.

@msaltan PR: 3.3 and 3.4 support initially added

[GoEddie]

@luisquintanilla i'm definitely interested in helping to keep the project moving forward, I stopped raising/reviewing pr's as they were not getting reviewed and merged in but if there is a committer who is available to do that or if there was an opportunity for community members to become committers then I would be interested.

[luisquintanilla]

Thanks @GoEddie & @GeorgeS2019. This is definitely something for us to look into how we can unblock you and by extension the project.

[dbeavon]

@luisquintanilla I can try to help. I've encountered some minor bugs that may be low-hanging fruit.

One thing that might bog us down the most is not having the means of updating the Microsoft nuget package.
https://www.nuget.org/packages/Microsoft.Spark/

Can you please explain (or give us links that explain) how the nuget packaging works for community projects, and whether it is still possible to publish new versions of it (even after Microsoft has abandoned the community)? Will someone other than Microsoft need to start publishing a different nuget?

Also I think there are portions of this project that need to be killed as the first order of business (especially if they were done on behalf of stakeholders who have left). For example, I'm pretty eager to kill all the weird cruft related to "Microsoft.Data.Analysis". That is a very minor amount of code that never worked well, and caused a lot of confusion. For example there are critical overloaded class names like "DataFrame" which are part of both namespaces! It was a bad fit for this project. Anyone who still needs to do an integration with "Microsoft.Data.Analysis" can do their own independent work to reintroduce that mess on their own. (That other project isn't even v.1 yet, in any case.)

[dbeavon]

@GoEddie
Since you asked, one reason the activity on the project seems to have stopped is because of employee attrition at Microsoft. Certain full-time developers like Rahul Potharaju @rapoth and Terry Kim @imback82 have left Microsoft and started working at Databricks instead. And the critical PMs like @MikeRys have moved on to other areas within Microsoft. (Hi there Michael! Please come back!)

I am pretty disgruntled about the Synapse side of this story. In 2022 I had migrated all my projects from Databricks to Synapse where Microsoft was trying to monetize .Net for Spark. After spending several months working on this migration of my .Net projects, I encountered a relatively innocent bug that only affected Synapse (not OSS and not Databricks). I report the bug right around the time that the engineers were exiting the company. Only at the very end of an eight month support case did they say that they can no longer support .Net, and they are removing it from future versions of Synapse Analytics. It was a pretty painful experience, as you can probably imagine. (As a side the bug turned out to be just some stupid DNS configuration issue in their Bionic Beaver image, and wasn't specific to .Net. It would have affected the other language bindings as well.)

It doesn't stop there. After they removed .Net from their 3.3 runtime, this Synapse team started to deliberately spread misinformation to discourage people from using .Net on Spark at all.

See https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-33-runtime

They say this is a "project under the .NET Foundation that currently requires the .NET 3.1 library, which has reached the out-of-support status". Of course, the claim that Spark requires .Net Core 3.1 is categorically false. I'm able to run the project on .Net 6 without any problems. To add insult to injury they tell people "We recommend that users with existing workloads ... migrate to Python or Scala." I'm guessing that they are worried about the possibility that their .Net customers will just leave Synapse and find a better Spark offering. The PG's documentation doesn't even refer people back to this community, in order to pursue an alternative path forward with .Net. I find their communication to be pretty dishonest. And it almost seems like a deliberate attempt to sabotage this project.

I keep hoping we will see first-class support for .Net in Databricks, now that they have scalped some of the smart engineers from Synapse. (I do a regular google search, but it hasn't quite panned out yet.)

I had very high hopes for Synapse when they advertised their first-class .Net language bindings. There were even promotional community sessions where .Net for Spark was discussed, like here:

https://www.youtube.com/watch?v=-VpQheD-vE8
(start watching at 33:00 or so)

In any case, I am not that worried about the future. I am pretty sure .Net isn't going to die any time soon; and neither is Spark. The marriage of this pair is off to a rocky start. But they can't be kept apart forever!

[GoEddie]

@dbeavon Thanks for filling in the missing details!

I completely agree about getting rid of some other the other components, probably the first thing to do is to get the core Microsoft.Spark working with the newer versions of Spark and get rid of things like the extensions etc and maybe bring them back in the future.

[GeorgeS2019]

@GoEddie
@dbeavon
@luisquintanilla

Thanks for not giving up on this project.

The fact that we are having this discussion is somehow sad!

Microsoft can not advocate the latest AI, large language model, and copilot without empowering .NET users on Spark.NET => the key and perhaps one of the very few available BIG DATA ANALYTICS pipeline for .NET community to continue their support for Microsoft's latest AI leadership.

[leo-schick]

As @luisquintanilla pointed it out: The project is part of the .NET Foundation which is open source and everybody can join to be a part of the .NET success story 😃: Become a member
That Microsoft currently decided to not invest monetary ressources (e.g. paid developers( to this project should not stop this project from going forward IMO. Its just the question on how much are the community (= we/everybody) is willing to contribute and take the lead in this.
There are other projects like Mono which (I guiess) over a decade where run by the community bringin .NET to Linux before Microsoft switched to "Microsoft loves Linux" and they started actively to work together with the Mono community to merge things together. One great result of that is now .NET MAUI (I hope I get the facts right here).

But nevertheless, I think the current issue this project has is that it is missing a a maintainer who is willing to actively review/merge PRs, handle issues, publish new releases etc.

@GoEddie @dbeavon @GeorgeS2019 Can you think to be part of the success story for .NET for Apache Spark ?

[GoEddie]

@leo-schick yes but im not sure what the process would be, at the moment it seems like no maintainers are active. It would be amazing if Microsoft invested paid developers in the project but even without that we, as a community, can maintain the project but have no access to anything!

[leo-schick]

As it as pointed out earlier by @GeorgeS2019 in his post, a maintainer is missing. If you want to become one, please contact @luisquintanilla

[GoEddie]

Thanks @leo-schick have you got any contact details for yourself or @luisquintanilla? My email is ed.elliott@outlook.com

[fwaris]

I think dotnet-spark can be more liberal about which Apache Spark versions it supports. Instead of erroring out, it can just put out a warning. Apache Spark releases are more frequent, but I have found that existing dotnet-spark code works fine with newer versions of Apache Spark.

But I have to update the dotnet-spark code locally to make it work with new Spark versions (as the nuget packages don't work).

A more liberal version matching policy should reduce some of ongoing support effort required.