Decision: allow Hadoop Yarn to be phased out due to missing maintenance capacity?

[consideRatio]

I'd like to propose we stop attempting to support Hadoop Yarn in this project by:

• documenting it is being phased out unless maintenance effort beyond current capacity arrives, specifically maintenance around:
  • maintenance of the dependency skein (Protobuf stuff needs to be regenerated jcrist/skein#255)
  • making a new CI testing image not based on CentOS 7, which no longer builds (Testing images for slurm, pbs, and hadoop, are no longer buildable #837)
• allowing it to be cut away from this project over time when we encounter issues maintaining functional tests etc.

As a first step, I'd like to merge conda-forge/dask-gateway-feedstock#43 and drop the conda-forge feedstock's output dask-gateway-server-yarn that makes the entire feedstock not get upgraded to emit Python 3.13 outputs because its dependency skein isn't updated for Python 3.13.

I'm looking for a conclusion in this issue around whether it is okay to proceed with phasing out Hadoop Yarn as friction arises in maintenance while waiting for eventual maintenance capacity to arise.

/cc: @TomAugspurger @jacobtomlinson @martindurant @jcrist

[TomAugspurger]

Sounds good to me.

…

On Feb 8, 2025, at 3:52 PM, Erik Sundell ***@***.***> wrote: I'd like to propose we stop attempting to support Hadoop Yarn in this project by: documenting it is being phased out unless maintenance effort beyond current capacity arrives, specifically maintenance around: maintenance of the dependency skein <https://github.com/jcrist/skein> (jcrist/skein#255 <jcrist/skein#255>) making a new CI testing image not based on CentOS 7 which no longer builds (#837 <#837>) allowing it to be cut away from this project over time when we encounter issues maintaining functional tests etc. As a first step, I'd like to merge conda-forge/dask-gateway-feedstock#43 <conda-forge/dask-gateway-feedstock#43> and drop the conda-forge feedstock's output dask-gateway-server-yarn that makes the entire feedstock not get upgraded to emit Python 3.13 outputs because its dependency skein isn't updated for Python 3.13. /cc: @TomAugspurger <https://github.com/TomAugspurger> @jacobtomlinson <https://github.com/jacobtomlinson> @martindurant <https://github.com/martindurant> @jcrist <https://github.com/jcrist> — Reply to this email directly, view it on GitHub <#882>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIXLAGYTFFH3TZN2BTT2OZ4CLAVCNFSM6AAAAABWYC7YBGVHI2DSMVQWIX3LMV43ASLTON2WKOZSHA2DAMRZGM2DONY>. You are receiving this because you were mentioned.

[martindurant]

stop attempting to support Hadoop Yarn

It seems clear to me, that if none of us has the time or motivation to work on it, then it should not stand in the way of still-useful functionality without it.

I don't suppose we know how many people might be using legacy YARN stuff, but they could make themselves know - but I don't expect so.

okay to proceed with phasing out Hadoop Yarn

+1

[jacobtomlinson]

Totally fine with me. If users are depending on this functionality they will shout as things are removed. Then we can encourage them to either pin to an older release or step in and help with maintenance. My assumption is that there wont be much noise around this removal though.