Questions on Plugins, Scores, and Tolerance

[michael-hoover]

Hi, I am investigating Hipcheck for usage in a project that models SBOMs in simulating potential cyber impacts to critical infrastructure and government continuity.

• What additional plugins, beyond what is available, do you see providing the most benefit for open source software analysis?

• What guidance can you give on proper score weights for each plugin?

• How would you recommend setting your overall risk tolerance score for an organization using Hipcheck? 

[alilleybrinker]

Hi Michael, thanks for reaching out!

I'll answer questions 2 and 3 first, and come back to question 1.

Good Starting Weights for Analyses

For question 2, proper scoring weights for each plugin: the simplest model would be to equally weight all plugins, which would mean putting all analysis entries directly under the analyze block in your policy file. This gives a good starting point from which you can decide if specific analyses are more important to you for your use of open source software.

Note

You can use the hc scoring command to see a representation of how these percentages break down to debug your scoring configuration.

Good Starting Risk Tolerance

We've historically started with 0.5 as a threshold, with the policy expression (gt 0.5 $). We recommend starting with that and then adjusting up or down depending on how many false positives marked for further investigation you see in your results.

With any of these things, tuning in response to experience is going to help; the details of what analyses you run, how they're configured, and what kinds of dependencies you're assessing, will all impact what the right policies are.

What Additional Plugins Would Be Most Valuable?

The things we've usually focused on have been the practices associated with a project's development; questions like "does this project practice code review?" or "is this project actively maintained?" Currently, we have plugins that try to answer these questions (mitre/activity, mitre/review, mitre/identity), but we think there's room to continue improving, including by synthesizing the results of review and identity into one plugin (as they capture evidence for different workflows for code review) and expanding to support detecting more patterns of code review.

We also think supporting detection of concerning contributors (known-malicious identities) could be valuable. One of the processes people followed after the "xz-utils" attack last year was to see what other project Jia Tan contributed to prior to the discovery of the attack. Having automation support to flag contributions by identities known to be affiliated with supply chain attacks could be helpful.

---

Hope that helps, and happy to discuss further!

[michael-hoover]

Thanks!

[alilleybrinker]

Happy to help!