Feat: change ingester data insertion policy

[MarceloRobert]

Description

Changes the insertion policy of the ingester from "never update" to "only update null fields". This follows the policy adopted by kcidb, including a PRIO_DB env var which tells which data to prioritize.

If PRIO_DB is set to True it will only allow updates on null data (allow null -> other value but not existing value -> other value and not existing_value -> null).
If set to False it will allow overwriting existing data as long as the new data is not null (allow existing value -> other value and allow null -> other value but not existing value -> null).

Changes

• Added a new function to get the insertion query, in SQL, for any model
• Used this function to get the query on consume_buffer instead of Django's bulk_create
• Updated unit tests

How to test

Start the ingester and insert some data to it referencing the same object, check if the policy is respected.
Example realistic data for testing can be found here, but other tests are encouraged.

Future changes

1. Having a dynamic approach to the insertion query gives us the benefit to not have to change it every time we change the models, and not need to make a separate piece of code for each model. However, it might be costly to recreate the same query all the time, so a static approach might be better. The dynamic generation could be contained in a command so that we can refresh the insertion queries when we want to;
2. Adding automated integration tests so that we can guarantee that this policy is kept regardless of changes in the ingester is also useful.

Closes #1552

[Copilot]

Pull request overview

This PR changes the ingester's data insertion policy from "never update" to "only update null fields", controlled by a new PRIO_DB environment variable. The implementation replaces Django's ORM bulk_create with raw SQL queries using PostgreSQL's ON CONFLICT clause with COALESCE operations to selectively update fields based on the priority policy.

Key changes:

• Added dynamic SQL query generation function _generate_model_insert_query that creates upsert queries with field-level update control
• Replaced bulk_create with raw SQL execution using cursor.executemany() in consume_buffer
• Introduced PRIO_DB environment variable to toggle between database-priority and data-priority update modes

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 9 comments.

File	Description
backend/kernelCI_app/constants/ingester.py	Adds PRIO_DB constant with documentation (contains typo in env var name)
backend/kernelCI_app/models.py	Adds module docstring noting explicit id column requirement for ingester
backend/kernelCI_app/management/commands/helpers/kcidbng_ingester.py	Implements dynamic SQL query generation and replaces ORM with raw SQL for data insertion
backend/kernelCI_app/tests/unitTests/commands/monitorSubmissions/kcidbng_ingester_test.py	Updates test to mock database connections instead of Django ORM methods

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The _generate_model_insert_query function is called on every consume_buffer invocation, which means the same query is regenerated repeatedly for the same table. As noted in the PR description's "Future changes" section, consider caching the generated queries per table to avoid this overhead. The query structure is static for each model and only depends on PRIO_DB, which is set at startup.

[Copilot]

The type check isinstance(value, (dict, list)) only handles JSON serialization for dicts and lists, but doesn't handle other types that might need special handling (e.g., datetime objects, Decimal, custom objects). Consider adding more comprehensive type handling or documenting the assumption that only dicts/lists need JSON serialization.

[tales-aparecida]

I think adding PRIO_DB is bringing a lot of complexity to this pull request. If we decide to change the policy again in the future, we can send another PR.
I believe we can assume you aren't going to need it, though.

[tales-aparecida]

wow, I never read this file before, it's quite high-level and scary.

Do you think you could add a test case achieving something close to:

input = json.loads(Path("fixtures/inputs/kcidb.json").read_text())
expect_data = json.loads(Path("fixtures/outputs/tables.json").read_text())

with self.assertNumQueries(5):
    ingest(input)

result_data = {
  "checkouts": Checkout.objects.all().values(),
  "builds": Build.objects.all().values(),
  "tests": Test.objects.all().values(),
  "issues": Issue.objects.all().values(),
  "incidents": Incident.objects.all().values(),
}
self.assertEqual(result_data, expected_data)