[WIP]: Add Hackage frontend support

[Korusuke]

This can be tested now

[Korusuke]

@mojca can you please test this out a bit and see if it works ?
upt-hackage-frontend

[mojca]

I would also suggest to also ask @essandess to test (if he's interested and if someone can quickly explain what needs to be done). He contributed macports/macports-ports#4794 (and a few others), and testing hackage is definitely related to having a working Haskell available.

The ultimate test would be to get pandoc port working this way :)
@essandess used a slightly different approach, but the goal would be very similar.

[essandess]

@Korusuke @mojca I’d be happy to test, but there are caveats:

• Based on my own limited knowledge and experience, I don’t understand why it’s necessary or desirable for MacPorts to duplicate existing Haskell development tools like cabal (#4715) or stack (#4633).
• It’s trivial to build pandoc from source (#4698) using stack, and both unnecessary and difficult/practically impossible to maintain using a large number of MacPorts hs-* dependencies.
• I believe this is the reason that MacPorts ghc (#4794) and pandoc (#4698) are year/s out-of-date. There are too many moving parts and conflicting hackage versions. Just use the lts-* long-term support settings from stack, as specified by the Haskell packages themselves in their stack.yaml (or cabal.yaml) configurations, e.g. pandoc‘s stack.yaml.
• I am not aware of a useful purpose of MacPorts maintaining its hs-* ports.

[essandess]

I am not aware of a useful purpose of MacPorts maintaining its hs-* ports.

That’s too strong. These are great for ghci session load :l commands, the same way that the py-* ports are in Python.

But for building Haskell packages, using the standard development tools is the way to go.

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@7eb75a1). Click here to learn what that means.
The diff coverage is 92.85%.

@@          Coverage Diff           @@
##             master   #39   +/-   ##
======================================
  Coverage          ?   88%           
======================================
  Files             ?     1           
  Lines             ?   200           
  Branches          ?    10           
======================================
  Hits              ?   176           
  Misses            ?    24           
  Partials          ?     0

Impacted Files	Coverage Δ
upt_macports/upt_macports.py	88% <92.85%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7eb75a1...a034656. Read the comment docs.

[pmetzger]

So the issue is this: for several languages (python, ocaml, haskell, etc.) there are language-specific package systems available, but they are all-encompassing/totalizing systems that don't fit well inside the MacPorts world. Meanwhile, we have individual applications some users want to run that have nothing to do with ocaml or haskell or what have you per se; they have no idea what the program is written in, but they want a macports-style build of the package.

So it makes some sense for MacPorts to maintain packages for a limited number of such language-specific packages in order to build such apps. I've been wanting to do such a thing myself for OCaml even though opam is the better way to build ocaml apps. In fact, I'd like a opam front end developed for upt-macports for just this reason.

[reneeotten]

I've been wanting to do such a thing myself for OCaml even though opam is the better way to build ocaml apps. In fact, I'd like a opam front end developed for upt-macports for just this reason.

@pmetzger can you please open a separate issue for this and provide a little more background information? I have personally no experience with ocaml/opam, so one would need to check whether all the metadata is easily available to use with upt. Can you point us to (i) the repository where typical packages are downloaded from, (ii) your favorite port/Portfile that you manually wrote and we can use as exptected/correct output?

Again, right now @Korusuke is working on some other things but we can certainly take a look at this at a slightly later point in time.

[pmetzger]

@reneeotten It might be easier if you were to get in touch with me by email; address is on my github home page. I believe that opam files contain essentially everything one would need in order to generate Portfiles automatically, and I'd be happy to work along with you guys making this happen. There's even an ocaml package for parsing opam files that could be twisted into extracting data without needing to rewrite the parser.

[pmetzger]

I have also opened an issue... #46

[reneeotten]

back to the hackage frontend, I tested a random selection of ports (not the installation, only the Portfile generation) that are currently present in MacPorts and noticed the following:

1. the upt-hackage frontend doesn't declare its dependencies (Issue opened on https://framagit.org/upt/upt-hackage)
2. for several packages it throws an IndexError in the parse function (e.g., vector-space, json, readline)
3. parsing of the descriptions doesn't always work probably due to HTML syntax or such... (e.g., base64-bytestring, blaze-html, data-accessor-template)
4. I think all these packages actually get compiled and if that's true (please double check), then we shouldn't add the stuff about "supported_archs" in the template/Portfile
5. there should be a newline between homepage and checksums and checksums and depends_lib

[Korusuke]

1. the `upt-hackage` frontend doesn't declare its dependencies (Issue opened on https://framagit.org/upt/upt-hackage)

Sorry, forgot to add it to the commit.

2. for several packages it throws an `IndexError` in the `parse` function (e.g., `vector-space`, `json`, `readline`)

Apparently few packages doesn't declare homepage...in that case now it sets homepage to the hackage url itself.....this is already done by haskell 1.0 portgroup too, so maybe we don't need to set the homepage.

3. parsing of the descriptions doesn't always work probably due to HTML syntax or such... (e.g., `base64-bytestring`, `blaze-html`, `data-accessor-template`)

Fixed!

4. I think all these packages actually get compiled and if that's true (please double check), then we shouldn't add the stuff about "supported_archs" in the template/Portfile

I checked a few existing haskell packages and they don't declare any of this so maybe it's not needed but I will check about this more.... @mojca can help here maybe.

5. there should be a newline between `homepage` and `checksums` and `checksums` and `depends_lib`

Fixed!

[reneeotten]

Apparently few packages doesn't declare homepage...in that case now it sets homepage to the hackage url itself.....this is already done by haskell 1.0 portgroup too, so maybe we don't need to set the homepage.

I think we should set the homepage to whatever upstream gives and otherwise set it to the hackage url itself. I see indeed that this is the default in the PortGroup; so if you can leave the homepage out of the template if it's not specified upstream.

Fixed!

Good, now there are however two newlines between long_description and homepage....

[reneeotten]

Two more comments left on the template - after you changed that, please squash the commits.

I think that the required changes were merged into upt-hackage (https://framagit.org/upt/upt-hackage/) as well, correct? There is still a PR there to add tests though, that should probably still be finalized.

@Korusuke after you made the changes, please do verify that there are no errors reported anymore by port lint and check for a subset of haskell ports that they actually install correctly. Once that is done, we should be good to merge this PR and then it could be more easily tested by others as well.

[reneeotten]

also as I said earlier, please remove the noarch part from the base template for hackage just as is done for rubygems

[reneeotten]

can this become: {% if pkg.upt_pkg.frontend not in ('rubygems', 'hackage') -%}

[reneeotten]

@Korusuke after you made the changes, please do verify that there are no errors reported anymore by port lint and check for a subset of haskell ports that they actually install correctly. Once that is done, we should be good to merge this PR and then it could be more easily tested by others as well.

@Korusuke okay, so you made the changes I suggested but didn't comment on how much testing you've actually done on installing the haskell ports. There are currently 145 hs- ports in MacPorts; can you please generate Portfiles and install a random selection of ~50-75 these ports and post somewhere the information on those that installed correctly (i.e., using port -dv installed | grep "hs-") and some logs for ports that failed (if applicable)? Additionally, please verify that port lint --nitpick <name> for these ports do not give any errors/warnings (perhaps with the exception of missing dependencies if you didn't to the recursive packaging).

[Korusuke]

Yeah few days back, I had tested packaging ghc itself....I did not hope it work but atleast was expecting some error but after a whole night(10+ hours) of compiling/installing and a zillion warning and debug log, I had no more patience left so I quit it and never tried it again till now.

This time as suggested by @reneeotten, I took normal hs-* ports which are already there and tried to package those and in the way found a few issues with upt-hackage and upt.

Following are the issues faced...I will be creating a issue if not already there for each oneof the following:

• Firstly recursive ends mid way as described in recursive packaging ends mid way #50 , I have the patch for this offline will push it today and create a PR at upt.
• When packaging multiple package recursively at the same time it is possible that two packages have same dependency which results in sys.exit due to Portfile already exist error....so maybe we can replace that with just a error log message instead of exiting?
• As @Steap is implementing for upt-cpan to convert pod-name into dist-name something similar is required in upt-hackage or maybe if viable in all frontends....basically for hackage there are some dependencies like "unbuildable", "integer", etc.. which are not present in hackage and thus when trying to install port, macports gives error of dependency not found.....this can be eliminated by only adding dependency which are present in the frontend....this would save a lot of time of the maintainer.
• Last thing are hackage specifiers for some ports are bit complicated and have "&&" and "||" operators and that cause issues which I am yet not sure how to solve....more on this issue in a dedicated issue tomorrow.

After "kinda" solving or bypassing all the above issues finally a port hs-mmap is installing but I am not sure if it will install properly cause of old ghc version but yeah as of now from past 2 hours ghc is compiling/installing, so I am gonna sleep now and continue tomorrow morning.

[essandess]

@Korusuke There’s a working, bootstrapped ghc v. 8.6.5 compiler in macports/macports-ports#4794

This uses straight-up make etc. command from the upstream district without any complicating MacPorts hs-* ports.

We’ve all been waiting on the merge for a while now. Not sure what’s causing the delay.

[Korusuke]

Thanks!!! @essandess , after many many hours finally it installed.
The existing one in macports never installed even after a whole nights sleep, #4794 normally would have worked but after about 3 hours I was tired of waiting and so cancelled it installed with bootstrap variant which took more 3-4 hours...but yeah I am very happy that it finally installed.

I will be creating a issue if not already there for each oneof the following
Done!
please refer #52, #53, over at upt-hackage and the PR in upt for recusrion error.

Quickly trying to install hs-mmap seems to work after removing all it's dependency as they are the ones that #4794 ghc already installs so as mentioned in #53 we should not have them in packages again.

Trying a batch install as originally suggested by @reneeotten would be hard right now because of the manual dependency editing issue. So I would say first we will have to solve #52 #53 and then we can get back to testing this.