PURLDB: Add on-demand package data collection for golang

[chinyeungli]

We want to enable the collect/ endpoint for golang to avoid failure to collect a golang PURL like pkg:golang/github.com/gorilla/context :

{
  "status": "cannot fetch Package data for pkg:golang/github.com/gorilla/context: no available handler"
}

(https://pkg.go.dev/github.com/gorilla/context)

[chinyeungli]

As most of the Go projects are hosted on github, we can redirect to use the code for "pkg:github/" for these projects.

Following are some sample projects that are not hosted on github:
https://pkg.go.dev/ben.gmbh/sourcehut-vanity
https://pkg.go.dev/gitlab.com/gitlab-org/api/client-go
https://pkg.go.dev/golang.org/x/oauth2/bitbucket
https://pkg.go.dev/bitbucket.org/lebronto_kerovol/gwerror

[chinyeungli]

@JonoYang
I am able to make it work with

pkg:golang/github.com/*
pkg:golang/gitlab.com/*
pkg:golang/bitbucket.org/*

which should cover most of the go projects. However, for the rest, it's hard to fetch the metadata unless we do web scrapping. Do you think web scrapping is the correct approach for the other cases?

[JonoYang]

@chinyeungli do you mean scraping the GitHub/gitlab/bitbucket page for metadata? There may be an api for that, otherwise I think it's worth trying

[chinyeungli]

No. I mean I can retrieve metadata from APIs for GitHub, GitLab, and Bitbucket. I was referring to scraping metadata for other sources on pkg.go.dev such as https://pkg.go.dev/ben.gmbh/sourcehut-vanity and https://pkg.go.dev/golang.org/x/oauth2/bitbucket

[JonoYang]

I think we should scrape pkg.go.dev. There may be some code for this in spats

[chinyeungli]

For gitlab, it sometimes asks for username and password in my terminal, but it runs fine if I just simply press enter without entering username and password, but if I didn't press the enter, it seems it's stuck and waiting for input.