Ability for PR-Agent to continuously improve using user feedback (learning from accepted suggestions & thumbs-up responses)

[balabanluka]

Feature request

Hello Qodo team 👋

We are currently using PR-Agent in our company setup, and we’ve been exploring ways to make the agent continuously improve its review quality over time — ideally by learning from feedback provided by our developers.

I’ve reviewed your documentation under:

• Suggestion Tracking
• Extra Instructions and Best Practices

Based on that, I understand that:

• allow_thumbs_up_down = true enables collecting user reactions (👍/👎) purely for statistics, and
• wiki_page_accepted_suggestions = true can log accepted suggestions for manual review and evaluation,
  but these features don’t actually change or “teach” the model over time.

Motivation

We’d like to know whether there is (or could be) a mechanism for incremental learning or adaptive improvement — e.g.:

1. Using accepted suggestions and thumbs-up feedback to gradually refine the model’s responses.
2. Integrating a lightweight retraining loop or fine-tuning pipeline using these logged results.
3. (Alternatively) A built-in RAG-based retrieval system that references past “good” PR comments to guide future reviews.

Essentially, the idea is that over time the agent would “learn” our internal standards and communication style — beyond static extra_instructions — and progressively adapt.

Questions

• Is there any ongoing work or roadmap related to self-improvement or adaptive fine-tuning?
• Could there already be a mechanism for this (e.g., through some plugin, RAG integration, or feedback hook) that I might have missed in the documentation?
• If this direction is not currently planned, could you please share any suggestions or best practices on how a team could implement such a feedback-driven improvement layer externally?

[qodo-merge-for-open-source]

💡 Auto-generated documentation-based answer:

Short answer:

• Today, thumbs up/down and accepted-suggestion tracking are used for statistics and visibility, not for directly updating the model weights.
• However, Qodo Merge already offers “learning-style” adaptations without fine-tuning:
  1. Auto Best Practices: learns recurring patterns from accepted suggestions and generates .pr_agent_auto_best_practices in the repo wiki; these are then applied in future /improve runs.
  2. Best Practices (local + global hierarchical): you can maintain repo/org-wide best_practices.md files and have them automatically applied per repo, group, or monorepo path.
  3. Suggestion Tracking: accepted suggestions are logged to .pr_agent_accepted_suggestions (wiki) and feed the Auto Best Practices loop.
  4. Self-reflection + scoring: improves suggestion ranking/quality within a run (not cross-run training).
  5. RAG context enrichment (Enterprise): retrieves relevant code patterns from your codebase to guide analysis; not model training, but adaptive context.

Roadmap/ongoing work:

• The docs state Suggestion Tracking and the accepted-suggestions wiki “will also serve as a foundation for future AI model improvements,” but do not announce model fine-tuning or continuous retraining. Recent roadmap highlights focus on compliance defaults, smarter context, and portal UX—not adaptive fine-tuning.

What exists now to approximate “continuous improvement”:

• Enable the learning loop:
  • pr_code_suggestions.wiki_page_accepted_suggestions=true (default)
  • Ensure repository wiki is enabled and initialized.
  • Turn on Auto Best Practices:
    • [auto_best_practices] enable_auto_best_practices=true
    • [auto_best_practices] utilize_auto_best_practices=true
• Establish Best Practices sources:
  • Local best_practices.md in the repo.
  • Global hierarchical best practices in a pr-agent-settings repo + metadata.yaml mapping (groups, monorepos, subprojects).
• Combine with extra_instructions per tool for style/tone and policy.
• Optionally use RAG (Enterprise) to bring prior “good” code references into context during /review, /implement, /ask, /compliance.

About thumbs up/down:

• pr_code_suggestions.allow_thumbs_up_down=true collects user reactions only for statistics. It does not currently steer future outputs.

If you want external feedback-driven training:

• You can build an external pipeline around the wiki logs:
  • Periodically parse .pr_agent_accepted_suggestions to extract (before→after) pairs and rationales.
  • Aggregate into best_practices.md (manual curation) or your own knowledge base; optionally feed these back as extra_instructions and custom best practices.
  • For RAG setups, ingest accepted suggestions and best practices into your retrieval store so future PRs reference them.
  • If you control the model, you could fine-tune on those pairs; PR-Agent does not ship a fine-tuning pipeline.

Summary:

• No built-in fine-tuning loop today. The documented path to “learning over time” is Auto Best Practices + (global/local) Best Practices + Suggestion Tracking, with optional RAG in Enterprise. Thumbs-up/down is for analytics only.

Relevant Sources:

• https://qodo-merge-docs.qodo.ai//tools/improve#suggestion-tracking
• https://qodo-merge-docs.qodo.ai//core-abilities/auto_best_practices#auto-best-practices
• https://qodo-merge-docs.qodo.ai//tools/improve#extra-instructions-and-best-practices
• https://qodo-merge-docs.qodo.ai//tools/improve#best-practices
• https://qodo-merge-docs.qodo.ai//usage-guide/enabling_a_wiki
• https://qodo-merge-docs.qodo.ai//core-abilities/self_reflection
• https://qodo-merge-docs.qodo.ai//core-abilities/rag_context_enrichment
• https://qodo-merge-docs.qodo.ai//recent_updates/index#recent-updates-and-future-roadmap