Repository Limit

[mcaulifn]

Moving over from renovatebot/renovate#35612 (comment)

How are you running Renovate?

Self-hosted Renovate - renovate-ce:9.9.2-full image

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitHub

Please tell us more about your question or problem

I suspect we have reached a limit with repository count. Last count, there were about 6800 repos that have Renovate enabled. During startup, Renovate will attempt to add repositories, but then fail with the SqliteError.

The information I found suggested reducing batch size to less than 999. I couldn't track down if this is done or not already.

Config

config.js

Details

const {
  ARTIFACTORY_HOST,
  ARTIFACTORY_USER,
  ARTIFACTORY_TOKEN,
  ARTYLAB_HOST,
  ARTYLAB_TOKEN,
  GITHUB_INTERNAL_TOKEN
} = process.env

const config = {
  cachePrivatePackages: true,
  onboardingConfig: {
    extends: ['config:recommended'],
    branchPrefix: 'renovate-'
  },
  maven: {
    enabled: true
  },
  hostRules: [
    {
      matchHost: ARTIFACTORY_HOST,
      insecureRegistry: false,
      username: ARTIFACTORY_USER,
      password: ARTIFACTORY_TOKEN
    },
    {
      matchHost: `https://${ARTIFACTORY_HOST}/artifactory/api/npm/public-npm-virtual/`,
      hostType: 'npm',
      token: ARTIFACTORY_TOKEN
    },
    {
      matchHost: `https://${ARTIFACTORY_HOST}/artifactory/api/npm/internal-npm-virtual/`,
      hostType: 'npm',
      token: ARTIFACTORY_TOKEN
    },
    {
      matchHost: `https://${ARTIFACTORY_HOST}/artifactory/api/npm/all-npm-virtual/`,
      hostType: 'npm',
      token: ARTIFACTORY_TOKEN
    },
    {
      matchHost: `https://${ARTYLAB_HOST}/api/npm/public-npm-virtual/`,
      hostType: 'npm',
      token: ARTYLAB_TOKEN
    },
    {
      matchHost: `https://${ARTIFACTORY_HOST}/api/pypi/all-pypi-virtual/simple`,
      hostType: 'pypi',
      username: ARTIFACTORY_USER,
      password: ARTIFACTORY_TOKEN
    },
    {
      matchHost: 'github.internal',
      token: GITHUB_INTERNAL_TOKEN
    }
  ],
  // npmrc is required to avoid Renovate defaulting to npmjs.org
  npmrc: `registry=https://${ARTIFACTORY_HOST}/artifactory/api/npm/all-npm-virtual/ \nalways-auth=true`,
  npmrcMerge: true,
  optimizeForDisabled: true,
  packageRules: [
    {
      matchCategories: ['docker'],
      enabled: true
    },
    {
      matchDatasources: ['docker'],
      registryUrls: [`https://${ARTIFACTORY_HOST}/all-docker-virtual/`]
    },
    {
      matchDatasources: ['go'],
      registryUrls: [
        `https://${ARTIFACTORY_HOST}/artifactory/api/go/all-go-virtual`
      ]
    },
    {
      matchDatasources: ['maven'],
      registryUrls: [`https://${ARTIFACTORY_HOST}/all-maven-virtual`]
    },
    {
      matchDatasources: ['helm'],
      registryUrls: [
        `https://${ARTIFACTORY_HOST}/artifactory/api/helm/all-helm-virtual`
      ]
    },
    {
      matchDatasources: ['npm'],
      registryUrls: [
        `https://${ARTIFACTORY_HOST}/artifactory/api/npm/all-npm-virtual/`
      ]
    },
    {
      matchDatasources: ['nuget'],
      registryUrls: [
        `https://${ARTIFACTORY_HOST}/artifactory/api/nuget/v3/all-nuget-virtual`
      ]
    },
    {
      matchCategories: ['python'],
      enabled: true,
      registryUrls: [
        `https://${ARTIFACTORY_HOST}/api/pypi/all-pypi-virtual/simple`
      ]
    },
    {
      matchDatasources: ['rubygems'],
      registryUrls: [
        `https://${ARTIFACTORY_HOST}/artifactory/api/gems/all-gems-virtual/`
      ]
    },
    {
      matchDatasources: ['sbt-package', 'sbt-plugin'],
      registryUrls: [
        `https://${ARTIFACTORY_HOST}/all-maven-virtual`,
        `https://${ARTIFACTORY_HOST}/artifactory/all-sbt-virtual/`
      ]
    }
  ],
  presetCachePersistence: true,
  persistRepoData: true,
  repositoryCache: 'enabled',
  transitiveRemediation: true,
  username: 'renovate[bot]'
}

module.exports = config

Environment Config

Details

- name: LOG_FORMAT
 value: json
 - name: LOG_LEVEL
 value: debug
 - name: PLATFORM
 value: github
 - name: MEND_RNV_CRON_JOB_SCHEDULER_ALL
 value: 0 0 1 1 */5
- name: MEND_RNV_ADMIN_API_ENABLED
value: 'true'
- name: MEND_RNV_ENQUEUE_JOBS_ON_STARTUP
value: disabled
- name: MEND_RNV_WORKER_EXECUTION_TIMEOUT
value: '8'
- name: RENOVATE_EXPERIMENTAL_NO_MAVEN_POM_CHECK
value: 'true'
- name: RENOVATE_REPOSITORY_CACHE_TYPE
value: s3://bucket-name
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://localhost:4318
- name: MEND_RNV_PROMETHEUS_METRICS_ENABLED
 value: 'true'

Logs (if relevant)

Logs

{"name":"mend-renovate","hostname":"renovate-prod-us-west-2-webhook-template-68f8fb79-fb85z","pid":1,"level":20,"logContext":"_lmmdn3SgzSIpeooF2GJX","repositories":["Long list of repositories here"],"msg":"Adding repositories","time":"2025-04-28T17:45:45.901Z","v":0}
{"name":"mend-renovate","hostname":"renovate-prod-us-west-2-webhook-template-68f8fb79-fb85z","pid":1,"level":50,"logContext":"_lmmdn3SgzSIpeooF2GJX","driver":"sqlite","err":{"code":"SQLITE_ERROR","message":"too many SQL variables","stack":"SqliteError: too many SQL variables\n    at Database.prepare (/usr/src/mend/node_modules/better-sqlite3/lib/methods/wrappers.js:5:21)\n    at BetterSQLiteSession.prepareQuery (file:///usr/src/mend/node_modules/drizzle-orm/better-sqlite3/session.js:20:30)\n    at BetterSQLiteSession.prepareOneTimeQuery (file:///usr/src/mend/node_modules/drizzle-orm/sqlite-core/session.js:65:17)\n    at QueryPromise._prepare (file:///usr/src/mend/node_modules/drizzle-orm/sqlite-core/query-builders/insert.js:131:81)\n    at QueryPromise.run (file:///usr/src/mend/node_modules/drizzle-orm/sqlite-core/query-builders/insert.js:142:17)\n    at QueryPromise.execute (file:///usr/src/mend/node_modules/drizzle-orm/sqlite-core/query-builders/insert.js:154:54)\n    at QueryPromise.then (file:///usr/src/mend/node_modules/drizzle-orm/query-promise.js:21:17)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)"},"msg":"Error adding repositories","time":"2025-04-28T17:45:46.190Z","v":0}
{"name":"mend-renovate","hostname":"renovate-prod-us-west-2-webhook-template-68f8fb79-fb85z","pid":1,"level":30,"logContext":"_lmmdn3SgzSIpeooF2GJX","organizationCount":1,"repositoryCount":0,"msg":"Post app sync statistics","time":"2025-04-28T17:45:46.194Z","v":0}

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

🔗 Similar Issues

Related Issues

• Renovate CE never opening PRs #610
• Bitbucket-Server Sync fails on CE >= 9.9.0 #712
• renovate-ce Docker tag to v12.1.0 - sqlite migration script failed #784
• Config.js is not picked up #707
• All PRs get "Edited/blocked notification" when using MEND_RNV_GITHUB_BOT_USER_ID with Renovate CE #741

👤 Suggested Assignees

• KarstenSiemer
• hauerhan
• 14traits
• simPod
• simonhaenisch

---

🧪 Issue enrichment is currently in early access.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

[jamietanna]

Thanks for moving this conversation over - to confirm, are you still running 9.9.2 (Renovate CLI 39.238.0 (2025-04-09))?

[nabeelsaabna]

@mcaulifn did you try using MEND_RNV_SYNC_MODE ?

MEND_RNV_SYNC_MODE: [GitHub only] Performance tuning for the App Sync operation. Set to 'batch' for orgs with very large numbers of repos.

values:

bulk (default) - will process all repos in one operation
batch - will process repos in smaller batches

[mcaulifn]

As if today, we are running 12.1.0
It looks like SYNC was added in 11? I had added a postgresql DB after I started noticing the problem. I've not tried anything else since.

[mcaulifn]

@jamietanna Resurrecting this as I think I've narrowed in on the limit. I've switched to using postgresql and have batch mode on. MEND_RNV_SYNC_MODE seems to only control the API calls, not the DB INSERT. We are running CE 14.x.

The default bulk sync fails when the GitHub App installation exceeds approximately 10,000 repositories. The single INSERT statement hits PostgreSQL's wire protocol parameter limit of 65,535 (parameter indices in the Bind message are encoded as Int16). With 6 columns per row, the ceiling is ~10,922 repos.

I would expect batch mode to also batch inserts and possibly even be configurable.