occtet: Collect and federate raw registry/APIs metadata

[pombredanne]

• Collect and federate raw registry/APIs metadata
• For instance keep raw Maven POMs as is, including parents
• Or results of PyPI API calls
• Also Federated basic PurlDB package record, beyond just PURLs

[JonoYang]

thinking out loud:

1. create a new table to store pom content or api calls with foreign key to package its for
   • This would have a variable text field for content, file name, and date field for collection date
2. populate these tables when we visit upstream sources (in minecode pipelines or otherwise), e.g. in maven pipeline, when we process a pom for package creation, save the pom to the db
   2.1. update the minecode_pipelines to save metadata files it collects alongside the purl.yml files
3. when we federate packages to repos, also dump associated metadata files as well as purls

[JonoYang]

minecode-pipelines has been updated to collect package data from upstream package version APIs, in particular for the ecosystems of cargo, maven, npm, and pypi. In order to enable collection of api_package_version_responses, add a new entry for the api_package_version_response datakind in your data federation's aboutcode-federated-config.yml, e.g. https://github.com/aboutcode-data/aboutcode-data/blob/main/aboutcode-federated-config.yml#L106

You can then install the mining extra_requires for scancode.io and run the pipelines. The pipelines should then create api_package_version_response repos as needed. This has been added to scancode.io in #2170