[Feature][Github] Advance Security data gathering

[cedriclecoz]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Use case

Add GitHub Advanced Security data such as Code Scanning (CodeQL), Secret Scanning, Dependabot.

Description

Adding data collection Github Advanced Security such as code scanning, secret scanning, ... would allow to get per project graphs and know which projects are active in fixing those issues or not, and track the resolution status of "leaked" passwords.
Had started looking into how to add the feature, but am a Go noob...
From what I read those Advanced Security data are only available via the Github REST API, not the GraphQL one, and in DevLake it looks like the Github plugin has been "turned off" and replaced by the github_graphql one.

I spent a few days playing with it and had quite a few problems such as:

• I didn't found how to set it up (if possible) at an organisation level so api would only be called once/page, so I had to call for it for every project, which led to the next issue
• the retries when there is no data to collect. (repo123, secret scanning not enabled or no alerts on it, devlake curl the api, gets a 404 because no data, and retries a couple of times after sleeps (I think) Run time moved from 4h to 16h+, and api credits were drying up)
• data duplication in my new _raw table

In the end I had to stop and just cobbled up a quick python script on cron schedule to curl the data and insert it in a table to Grafana to read.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[petkostas]

Thanks for raising this. In regards to Apache Devlake GitHub GraphQL API, although Devlake uses GraphQL there are some endpoints that are required to use the GitHub REST API (as those resources are not exposed via a GraphQL API). That means that cases like your proposal can be still added to Apache Devlake.