Optimize App Sync for GitLab Platform

[simon-weimann]

Which flavour of the Mend Renovate Server are you running?

Community Server (Community Edition / CE)

Which platform are you running Renovate on?

GitLab.com Self-Managed

Which version of Renovate Community / Enterprise Server are you using?

13.4.0

Please tell us more about your question or problem

We are running Renovate Community Edition in our company with a very large Organization/Repo setup.

When we start the Renovate CE Container it performs an AppSync against our GitLab Server querying all projects the renovate bot user has access to:
api/v4/projects?membership=true&per_page=100&min_access_level=30&archived=false

A single response is about 610kb and takes about 6 seconds to respond from the server. This continues for 50 times more, as we have about 5000 projects. In total the startup time is about 5 minutes due to the app sync and we are transporting about 30MB.

I wonder if the request parameters could make user of simple=true (https://docs.gitlab.com/api/projects/#list-all-projects). This reduces every response to about 110KB and 2 seconds response time.

In general it might be an idea to use the graphql API instead. It allows dedicated retrieval of only the attributes required for operation which further reduces the response size. It also opens up the possibility of filtering for other things like topics, a highly requested feature #383

Example GraphQL for all projects:

{
  projects(archived: EXCLUDE, membership: true, minAccessLevel: MAINTAINER) {
		pageInfo {
       endCursor
       startCursor
       hasNextPage
     }
    nodes {
       path
       fullPath
       namespace {
        id
        fullPath
      }
     }
  }
}

Example GraphQL for topics:

{
  projects(
    archived: EXCLUDE
    membership: true
    minAccessLevel: MAINTAINER
    topics: ["renovate"]
  ) {
    pageInfo {
      endCursor
      startCursor
      hasNextPage
    }
    nodes {
      path
      fullPath
      namespace {
        id
        fullPath
      }
    }
  }
}

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks