Faster way to query all CTAN package data

[muzimuzhi]

CTAN only provides API to query data for a single package: https://ctan.org/help/json/2.0/pkg.

This means to find all possible CTAN-to-TL package name mappings, the packages/data/scripts/generate-namemap.ts script has to query >2000 times in a cold run, making it slow (>10min). Warm run would get wrong results, because we don't know if any existing mappings would change.

I guess there is some non-public way to get such data as a whole. Maybe https://texlive.info/ already provides such way, but as of writing, this site is down.

---

Currently packages/data/scripts/generate-namemap.ts stores three caches under packages/data/build:

• ctan-packages.json for list of CTAN packages names, returned by a query to https://ctan.org/help/json/2.0/packages
• ctan-packages.jsonl for list of CTAN package data, by combining results of >2000 queries to https://ctan.org/help/json/2.0/pkg/<pkgname>
• tlpdb.json for tlpdb

ctan-packages.jsonl is the only bottleneck.

The subdir packages/data/build is excluded from Git, due to a .gitignore in it, created by generate-namemap.ts.

[gucci-on-fleek]

@muzimuzhi

CTAN only provides API to query data for a single package: https://ctan.org/help/json/2.0/pkg.

This means to find all possible CTAN-to-TL package name mappings, the packages/data/scripts/generate-namemap.ts script has to query >2000 times in a cold run, making it slow (>10min). Warm run would get wrong results, because we don't know if any existing mappings would change.

Does the data in the file http://maxchernoff.ca/files/network-install.files.lut.gz help at all? I'm already generating that as a part of another project, so that would be a really easy option if it works for this.

I guess there is some non-public way to get such data as a whole. Maybe https://texlive.info/ already provides such way, but as of writing, this site is down.

Ah, I have access to the internal CTAN catalogue files (example), so I can easily get that data for you. Give me a couple minutes.

[gucci-on-fleek]

Ok, here's a file that maps from CTAN package names to TL package names: https://tug.org/~mseven/ctan-to-tl.tsv. That's just a static file right now, but I can update that daily if that would be helpful.

[muzimuzhi]

Ok, here's a file that maps from CTAN package names to TL package names: https://tug.org/~mseven/ctan-to-tl.tsv. That's just a static file right now, but I can update that daily if that would be helpful.

Yes this file will definitely solve the problem here! Maybe

• packages which have the same CTAN and TL names can be filtered out (will identical maps entries be useful in some cases?)
• in JSON (easier for TypeScript but harder for Lua to parse)?

Having mapping data available from web would also allow setup-texlive-action using the latest mapping in runtime.

Not related but just the first lines in ctan-to-tl.tsv already uncovers bugs in generate-namemap.ts:

• missing entry for e.g. aaai-named beebe
• multiple copies stored in ctan-packages.json

[gucci-on-fleek]

Yes this file will definitely solve the problem here! Maybe

• packages which have the same CTAN and TL names can be filtered out (will identical maps entries be useful in some cases?)

• in JSON (easier for TypeScript but harder for Lua to parse)?

Sure, that should all be fairly doable. The current "implementation" is just a really really long shell command with like 12 different commands (who says you can't parse XML with regexes :) ), but I should hopefully have time this week to write a more maintainable implementation in Python (or Lua).