Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ By discussing issues with security team members and other maintainers, they can

## Contributing

Please see the [**Contributing Guide**](CONTRIBUTING.md) for more information on how to get started.
Please see the [**Contributing Guide**](CONTRIBUTING.md) for more information on how to get started.
2 changes: 1 addition & 1 deletion src/shared/listeners/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import shared.listeners.nix_channels # noqa
import shared.listeners.nix_evaluation # noqa
import shared.listeners.automatic_linkage # noqa
import shared.listeners.cve_derivation_matcher # noqa
import shared.listeners.cache_suggestions # noqa
import shared.listeners.notify_users # noqa
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
logger = logging.getLogger(__name__)


def produce_linkage_candidates(
def find_linkage_candidates(
container: Container,
) -> dict[NixDerivation, ProvenanceFlags]:
latest_complete_channels = (
Expand Down Expand Up @@ -93,22 +93,20 @@ def produce_linkage_candidates(
return candidates


def build_new_links(container: Container) -> bool:
def create_derivation_proposal(container: Container) -> bool:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def create_derivation_proposal(container: Container) -> bool:
def create_suggestion(container: Container) -> bool:

I'm so sorry, the naming is a mess in this code base. We're still cleaning up the prototype-era fallout. @florentc do we agree that we've firmly converged on calling the automatic matchings "suggestions"?

Copy link
Copy Markdown
Collaborator Author

@adekoder adekoder Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but isn't suggestion in the context of how the frontend treat matches ?

Copy link
Copy Markdown
Collaborator

@fricklerhandwerk fricklerhandwerk Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A(n automatically generated) suggestion links CVEs and (currently) derivations (we want that to be packages #617), we also call these links matches. We call the process "linking" or "matching". Ideally the frontend would directly visualise that data model.

The code is still all over the place, sometimes they are called "proposals" or "matches" or "links". I think we should stick with "suggestions", but "matches" is not wrong either (but would require starting to think about redirects from deprecated URL patterns).

Copy link
Copy Markdown
Member

@florentc florentc Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm biased as I spend most of my time in webview rather than shared. In webview, clearly, "suggestion" is the term that's used to refer to a CVEDerivationClusterProposal and its cached information. Stuff that is called suggestion usually has type CVEDerivationClusterProposal. It used to also refer to only the cached data. I tried to bring some consistency these last few months by always calling suggeston the CVEDerivationClusterProposal and access the cached field and the payload explicitly. There might still be some leftover.

So yeah, in webview at least, it would totally make sense to rename CVEDerivationClusterProposal to Suggestion.

In shared however I can't say with as much confidence. In the backend at the lower level, it's true that calling it "match" gives a better intuition about what it is. But this object has fields like "comment" and "status" which clearly go beyond the description of a match. Should we maybe have a Match table and and Suggestion table, each suggestion having a match (the CVE and Derivations) and adding fields describing its current state (status, comment, PackageEdits, MaintainersEdits, etc) in the suggestion workflow?

Sorry I'm not helping, just brainstorming too

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separating matches and "the thing that people edit" (I'm inclined to call that a "draft" as it was before in the UI, even at the risk of leaving the question "but draft of what?" or possibly "draft issue"). Then this draft would be part of a published issue, which tacks on even more data.

Most importantly this would be the first step towards having issues with multiple matches!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed! The draft could be linked to several matches, we could have an "add another match to the draft" UI thing, and the issue would be linked to 1 draft.

In fact, maybe issue and draft could be merged. That would clear any naming weirdness.

Match: a CVE the system matched to some derivations (and the associated packages/severity/maintainers)

Issue: A thing the user edits. Associated to 1 or several matches. Which could be in various status: untriaged, dismissed, draft, and published. When in "dismissed" or "published" the issue is frozen like suggestions are now (no edits).

Publication: What becomes attached to an issue once in "published" status. Publication would have a github link, and additional data such as open/closed, tags, and whatever we want to display (on sec tracker) is happening on GitHub about this issue.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. We can pull out cve and derivations into a Match model. We just need to think about what happens with the ignored/additional items, since right now they live on what would end up as Issue and thus apply to all of the matches. Is that what we want?

Copy link
Copy Markdown
Member

@florentc florentc Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, here is what would happen with a basic approach to the problem where Match are immutable and Issue proposes to apply ...Edits to packages and maintainers (and references).

Each Match has a set of packages (in fact derivations but viewed grouped by attribute name as packages) and their associated maintainers (and references)

An Issue may be associated to several matches.

An Issue may have 0, 1 or several PackageEdits, MaintainersEdit, and ReferencesEdit

The current state (and what's displayed) of the lists of packages/maintainers/references of an Issue would be the union of all the packages/maintainers/references of all its associated matches, on which additions and removals are applied by the overlay PackageEdit/MaintainersEdit/ReferencesEdit of that Issue.

This is, IMO, the straightforward first approach we can take.

if container.cve.triaged:
logger.info(
"Container received for '%s', but already triaged, skipping linkage.",
container.cve,
f"Container received for {container.cve}, but already triaged, skipping linkage.",
)
return False

if CVEDerivationClusterProposal.objects.filter(cve=container.cve).exists():
logger.info("Suggestion already exists for '%s', skipping", container.cve)
logger.info(f"Suggestion already exists for {container.cve}, skipping")
return False

if container.tags.filter(value="exclusively-hosted-service").exists():
logger.info(
"Container for '%s' is exclusively-hosted-service, rejecting without match.",
container.cve,
f"Container for {container.cve} is exclusively-hosted-service, rejecting without match.",
)
CVEDerivationClusterProposal.objects.create(
cve=container.cve,
Expand All @@ -117,16 +115,14 @@ def build_new_links(container: Container) -> bool:
)
return True

drvs = produce_linkage_candidates(container)
drvs = find_linkage_candidates(container)
if not drvs:
logger.info("No derivations matching '%s', ignoring", container.cve)
logger.info(f"No derivations matching {container.cve}, ignoring")
return False

if len(drvs) > settings.MAX_MATCHES:
logger.warning(
"More than '%d' derivations matching '%s', ignoring",
settings.MAX_MATCHES,
container.cve,
f"More than {settings.MAX_MATCHES} derivations matching {container.cve}, ignoring",
)
return False

Expand All @@ -144,14 +140,12 @@ def build_new_links(container: Container) -> bool:

if drvs_throughs:
logger.info(
"Matching suggestion for '%s': %d derivations found.",
container.cve,
len(drvs_throughs),
f"Matching suggestion for {container.cve}: {len(drvs_throughs)} derivations found.",
)

return True


@pgpubsub.post_insert_listener(ContainerChannel)
def build_new_links_following_new_containers(old: Container, new: Container) -> None:
build_new_links(new)
def match_derivations_on_container_insert(old: Container, new: Container) -> None:
create_derivation_proposal(new)
10 changes: 6 additions & 4 deletions src/shared/management/commands/propose_cve_links.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from django.core.management.base import BaseCommand, CommandError

from shared import models
from shared.listeners.automatic_linkage import build_new_links
from shared.listeners.cve_derivation_matcher import create_derivation_proposal

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -40,15 +40,17 @@ def handle(self, *args: Any, **kwargs: Any) -> None:
except ValueError:
raise CommandError(f"Not a valid delta format: {_delta}")

logger.info("Proposing new CVE links starting '%s'", since_date.isoformat())
logger.info(f"Proposing new CVE links starting {since_date.isoformat()}")
success = Counter()
# Collect all containers since that delta range.
containers = models.Container.objects.filter(date_public__gte=since_date)

for container in containers.iterator():
success[container.cve.cve_id] += 1 if build_new_links(container) else 0
success[container.cve.cve_id] += (
1 if create_derivation_proposal(container) else 0
)
print(".", end="", flush=True)

for cve_id, successes in success.items():
if successes == 0:
logger.warning("No derivation found for '%s', linkage failure.", cve_id)
logger.warning(f"No derivation found for {cve_id}, linkage failure.")
8 changes: 4 additions & 4 deletions src/shared/tests/test_linkage.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@

import pytest

from shared.listeners.automatic_linkage import build_new_links
from shared.listeners.cache_suggestions import cache_new_suggestions
from shared.listeners.cve_derivation_matcher import create_derivation_proposal
from shared.models.cve import Container, Tag
from shared.models.linkage import (
CVEDerivationClusterProposal,
Expand Down Expand Up @@ -67,7 +67,7 @@ def test_link_only_latest_eval(
)

container = make_container(package_name="foo", affected_version="<3.2")
match = build_new_links(container)
match = create_derivation_proposal(container)
assert match
suggestion = CVEDerivationClusterProposal.objects.first()
assert suggestion
Expand Down Expand Up @@ -113,7 +113,7 @@ def test_link_product_or_package_name(
container = make_container(package_name=package_name, product=product)
drv = make_drv(pname=drv_pname)

match = build_new_links(container)
match = create_derivation_proposal(container)

if expected_flags:
assert match
Expand All @@ -133,7 +133,7 @@ def test_exclusively_hosted_service_creates_rejected_proposal(
tag, _ = Tag.objects.get_or_create(value="exclusively-hosted-service")
container.tags.add(tag)

result = build_new_links(container)
result = create_derivation_proposal(container)

assert result is True
proposal = CVEDerivationClusterProposal.objects.get(cve=container.cve)
Expand Down