Server retired packages

[stschiff]

Addresses #325. Here is the Changelog entry:

• V 1.6.8.0:
  • Introduced new "retiredPackage" option in the CLI parsers to specify retired packages for serving. Retired packages are by default ignored in the /packages, /groups, /bibliography and /individuals endpoints, as well as ignored in the archive HTML page of the explorer. However, /zip_file still serves retired packages, so that they can be downloaded, and the retired packages are still available in the per-package Explorer HTML page.

[Copilot]

Pull Request Overview

This PR introduces a "retired packages" feature that allows server administrators to mark specific packages as retired. Retired packages are filtered out from most API endpoints by default but can still be accessed when explicitly requested.

• Added CLI options --retiredPackage and --retiredFile to specify retired packages for the server
• Modified API endpoints to filter out retired packages unless includeRetired parameter is provided
• Retired packages remain accessible via direct ZIP download and individual package pages

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/Poseidon/CLI/Serve.hs	Implements core retired package filtering logic and updates API endpoints
src/Poseidon/CLI/OptparseApplicativeParsers.hs	Adds CLI parsers for retired package options
src-executables/Main-trident.hs	Integrates retired package parser into serve command
src/Poseidon/EntityTypes.hs	Adds Show instance for EntityInput type
test/PoseidonGoldenTests/GoldenTestsRunCommands.hs	Updates test calls to include empty retired packages list
poseidon-hs.cabal	Bumps version to 1.6.8.0
CHANGELOG.md	Documents the new feature

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[stschiff]

One point to note: I reused the EntityInput machinery to save some code. This is a bit ugly, as retired packages should strictly be packages, but I check for this within Serve.hs and throw an error if people give groups or individuals as entities. Not ideal, but since this is supposed to be a somewhat special option only for our archive server, I think it's OK. And it's quite convenient to be able to list packages on both the CLI and as a file. The DSL is the same as for forge or fetch.

[stschiff]

OK I've now tested it and tweaked some more things, for example the explorer page now shows also the correct number of packages if retired ones are excluded. You can always add ?includeRetired to any API, both JSON and HTML, to include retired. I have tested it with the testPackages, and confirmed that indeed excluding for example Lamnidis_2018 will exclude both versions of these packages, while Lamnidis_2018-1.0.0 will exclude only this specific one.

[nevrome]

Thanks for addressing this, finally! This is a very important feature for the entire ecosystem!

Before I try it out and look at the code I have a first big question: Shouldn't we turn the CLI options retiredFile and retiredPackages to a feature of the archive configuration .yml file? So to a single new field retiredFile in ArchiveSpec?

poseidon-hs/src/Poseidon/CLI/Serve.hs

Lines 88 to 95 in 987b8d7

	data ArchiveSpec = ArchiveSpec
	{ _archSpecName :: ArchiveName
	, _archSpecPaths :: [FilePath]
	, _archSpecDescription :: Maybe String
	, _archSpecURL :: Maybe String
	, _archSpecDataURL :: Maybe String
	, _archSpecExcludePacsFromMap :: [String]
	} deriving (Show)

Filtering out retired packages is something that should imho happen on the archive level, not above.

Ultimately my idea is as follows: Every archive should have a separate list of retired packages, tracked with Git in the respective archive repository (then it's perfectly transparent what was retired when and by whom). If retiredFile is a field of ArchiveSpec, then we can simply add the respective retired-file paths to the archive config file.

[stschiff]

Excellent idea!!! I will get on with that suggestion!

[codecov]

Codecov Report

❌ Patch coverage is 57.94393% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.96%. Comparing base (66d85ff) to head (a9c792f).
⚠️ Report is 24 commits behind head on master.

Files with missing lines	Patch %	Lines
src/Poseidon/CLI/Serve.hs	60.39%	30 Missing and 10 partials ⚠️
src/Poseidon/CLI/OptparseApplicativeParsers.hs	0.00%	4 Missing ⚠️
src/Poseidon/EntityTypes.hs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #356      +/-   ##
==========================================
+ Coverage   56.83%   56.96%   +0.12%     
==========================================
  Files          33       33              
  Lines        4993     5052      +59     
  Branches      546      550       +4     
==========================================
+ Hits         2838     2878      +40     
- Misses       1609     1624      +15     
- Partials      546      550       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[stschiff]

OK, I've made this change now. I've reformatted the Serve.hs module a bit, as I was getting confused by its complexity. Not sure it helps that much. I added a Golden Test for the retired feature at the ArchiveConfig level. Looks like it works.

[stschiff]

Sorry for the reformatting of Serve.hs, I realize now that this is a bit hard to review.

[nevrome]

Code looks good to me 👍 - great! Didn't test it yet, though.

[nevrome]

I realized now that the mechanism you implemented is to read retiredPackages directly from the archive config .yml file. I fear that does not cover our specific usecase. We need a way to store retired packages IN the archive Git repositories to document transparently which packages were retired when, by whom (and potentially why).

In my opinion the .yml file should allow for optional file paths to per-archive retiredPackagesFiles. Then we can easily maintain such a file in the archives. The format for this file could also be a .yml file for consistency, and to reuse the parsing code you have already written. Bonus points for a field in said file that allows to quickly document WHY a given package was retired.

I'm also OK with other solutions, but I'm currently convinced that the retired package list should live in the archives.

[nevrome]

Beyond that things worked as expected in my test 👍. Nice!

[stschiff]

OK, so I've fixed all the other small points you've raised (all much appreciated!). Your last point I also agree with. But for simplicity sake I would then suggest to only add a retiredFile option in the YAML, and not both a direct listing and a file. So I will just replace the current option "retiredPackages" by a new option "retiredPackagesFile" which would then be a YAML list. YAML has the advantage that we can add comments, over straight text files.

[nevrome]

Sounds good 👍. As soon as this is implemented we can test it on the test server with the community-archive. The old 2019_BraceNatureEcologyEvolution package could be retired as it was merged already a long time ago with 2019_Brace_Britain.

How would you like to structure the retiredPackagesFile? Maybe a similar structure as for the chronicle file would be good, so something like this:

title: Poseidon community-archive retired package list
version: 1.58.0
lastModified: 2025-10-15
packages:
- title: 2019_BraceNatureEcologyEvolution
  version: X.X.X
  comment: Package was merged with 2019_Brace_Britain.
- ...

Not sure if we need a version number here.

The chronicle files in the archives are currently all called archive.chron. Maybe the retiredPackagesFiles could be called archive.retired or something like that. They should be added to all archives (even if empty at the beginning) and then listed in our master config file.

[stschiff]

Yes, I think a YAML format makes sense. That saves me a bit trouble in parsing entities, like I had originally, because then we can just list the version and the package title right in the YAML. I like it. And archive.retired makes sense, but I would not enforce this in the server software.

[stschiff]

OK, this is finally done. Sorry that it took me so long.

[nevrome]

Excellent! I went for a field test right away this time.

The test server at https://testserver.poseidon-adna.org is now running on this trident version. I also added an archive.retired file to the community archive here: https://github.com/poseidon-framework/community-archive/blob/master/archive.retired to retire the 2019_BraceNatureEcologyEvolution package. And I made trident aware of it with a retiredPackagesFile entry in the server_archive_config.yml file.

When I check here https://testserver.poseidon-adna.org/explorer/community-archive the package is indeed not displayed any more. I can still access it with the URL: https://testserver.poseidon-adna.org/explorer/community-archive/2019_BraceNatureEcologyEvolution/latest. That looks all fine.

On the command line I get the following expected results:

trident list --packages --remote --remoteURL https://server.poseidon-adna.org | grep "Brace"
| 2019_BraceNatureEcologyEvolution                 | 2.1.0           | True      | 2.5.0            | Originally AADR v42.4.                                       | 2023-07-04    | 14             |
| 2019_Brace_Britain                               | 2.1.0           | False     | 2.5.0            | Originally AADR v42.4.                                       | 2023-07-04    | 56             |
| 2019_Brace_Britain                               | 3.0.0           | False     | 2.7.1            | Originally AADR v42.4.                                       | 2023-07-26    | 70             |
| 2019_Brace_Britain                               | 3.1.0           | False     | 2.7.1            | Originally AADR v42.4.                                       | 2024-01-25    | 70             |
| 2019_Brace_Britain                               | 3.1.1           | True      | 2.7.1            | Originally AADR v42.4.                                       | 2024-12-17    | 70             |

trident list --packages --remote --remoteURL https://testserver.poseidon-adna.org | grep "Brace"
| 2019_Brace_Britain                               | 2.1.0           | False     | 2.5.0            | Originally AADR v42.4.                                       | 2023-07-04    | 56             |
| 2019_Brace_Britain                               | 3.0.0           | False     | 2.7.1            | Originally AADR v42.4.                                       | 2023-07-26    | 70             |
| 2019_Brace_Britain                               | 3.1.0           | False     | 2.7.1            | Originally AADR v42.4.                                       | 2024-01-25    | 70             |
| 2019_Brace_Britain                               | 3.1.1           | True      | 2.7.1            | Originally AADR v42.4.                                       | 2024-12-17    | 70             |

Should there be an option for trident list --remote to include the retired packages?