Using bkg in production for Pkgforge's needs

[Azathothas]

Hi,
I am from @pkgforge
Previously I sent a PR adding our orgs: #26

I looked at the data, it is well formatted, so thank you.
However, the data is outdated, which I understand.
I was looking into running bkg on our own infra every hr to generate data only for our org.
I couldn't find any instruction on the readme.
I looked at https://github.com/ipitio/backage/blob/master/src/bkg.sh, however it seems like some of the code contains hardcoded reference to your repository.

If you would like to make the script portable or if it is already portable, provide some basic instruction, I would highly appreciate it.
Some more context:
@pkgforge's an alternative to @Homebrew, the core difference being our packages are universal (They are self-contained, statically linked & run on both musl/glibc)
Homebrew have analytics, which we (@pkgforge) consider too much of a risk to pursue.
So our only alternative is to rely on projects like yours.
Currently, our metadata lacks rank, which we hope to calculate from the json generated by backage
You can find the struct here: https://meta.pkgforge.dev/bincache/x86_64-Linux.json
We have temporarily made the rank -1 until we have a reliable way to calculate it

[ipitio]

Hey! That's an awesome project.

Bkg is already portable, and I've just updated the readme to make that clear.

All you have to do is create a fork of just the master branch and enable Actions from its tab. Then use the Alternative URL in the readme once it updates to your fork.

Please merge the latest changes to your fork or make a new one, as I also updated the schedule to run every hour, and for your packages only in a fork.

Let me know if there are any issues! I don't believe anyone's done it exactly this way yet.

[Azathothas]

If you are open to it, I can add you as maintainer for https://github.com/pkgforge-dev/backage since this is going to be a core depedency for pkgforge now.
That would be ideal as you are the author and know it best... We had optionally also like a rank field which would be based upon the download_count..

However, if not, I will sync the changes and write here if I can't figure it out.
Thank you again, can't appreciate this project enough

[ipitio]

Sure, I'm open to that.

And by rank, you mean the index of packages within an owner, sorted by downloads?

[Azathothas]

Thank you, I have added you as Admin.

Rank needs to be for each namespaces/repositories rather than an owner

For instance
https://github.com/ipitio/backage/tree/index/pkgforge/bincache should have it's own rank sorted by downloads but only for packages belonging to this repo
Same applies for: https://github.com/ipitio/backage/tree/index/pkgforge/pkgcache

[Azathothas]

I see lots of commits.
I will let you polish things and make bkg even better.
For the time being, I have added a rather fragile approach to get the data from the pages:

grep -i -A 5 'Total downloads' | grep -oP '<h3 title="\K[0-9]+' | tr -cd '0-9' | tr -d '[:space:]'

We will switch to yours once it's ready.

Also, if you use discord, we have a server: https://docs.pkgforge.dev/contact/chat
One of our dev is trying to write your crawler in go... I am excited to see how it turns out

[ipitio]

This proved to be a valuable opportunity to fix issues with portability! I think I've got the kinks ironed out since the first batch of your packages just went through.

They will now be updated daily. At the moment the last 4 versions of each package are scanned (in case the most recent one is not tagged, aka not "latest"); if you only care about the most recent version, the frequency can be improved to twice daily. I know you wanted a full update every hour, but that might get us rate limited due to the large number of packages.

The go implementation sounds interesting!