[Task/Feature/Issue] OTDI Scoring Engine v0.0.1

[jolson-ibm]

Is there an existing issue for this?

• I have searched the existing issues

Task/Feature/Issue Details

OTDI Scoring Engine v0.0.1

Requirements:

1. Iterate over the entire HuggingFace (HF) data card data set
2. Score: 0 if the data set does not have a valid license, 100 if it has a valid license. 'Valid' here simply means 'license:<some string> exists in the data card.
3. Output scoring to the AI Alliance data lake in parquet format so that scoring results can be presented using Grafana and the AI Alliance analytics stack.

Additional Context

No response

[deanwampler]

I think we should reconsider how the data is used. A score is not useful to people, even if it had more fields. They will want to select for specific criteria, like does it have a license? Is it a license that's acceptable to me? Does the data cover the topics I care about? Etc. I wouldn't know how to interpret a Grafana graph of the data proposed. I would only want to use the license info as a filter, like we are effectively doing today with the static catalog.

[jolson-ibm]

Current state: In order to execute the current rev of the OTDI spec on the HuggingFace data sets, we are going to need to be able to parse the README.md, and separate it into the yaml, markdown, and raw text sections, of which there may be zero to many of each.

My first attempt to process a README.md was using regex, but that is proven to be impractical. I am currently refactoring on iterating over the text in the README.md, and that is working better.

I should have this under control by end of business today. Will update this ticket before tomorrow's standup.

This is blocking engaging with the data producers in the AI Alliance.

[jolson-ibm]

Current state of the scoring engine:

[jolson-ibm]

Reviewed the above with @AF-IBM, she made several insightful comments and sent me back to the drawing board. Refactor the code, and have another version of the above that is more concise. Will post here within the next few hours.

[jolson-ibm]

Pull request pending, once I clean up the code...

[jolson-ibm]

Job code cleaned up. I will schedule to run the job every N days (haven't figured out what N is yet) and when I do, I'll close this ticket.

[jolson-ibm]

The three other jobs: data sets, data sets detail (croissant), and data cards have be running all week as expected. Will deploy this job over the weekend, and we should be good.