new: [trainers] Never train the CWE classifier on discouraged CWEs (ref #17)

The CWE classifier was trained to predict deep ancestor CWEs, which are
mostly Pillars and Classes whose MITRE mapping usage is Discouraged
(CWE-664, CWE-707, CWE-703, CWE-284, ...), so its suggestions were
almost always CWEs that should not be used for mapping.

The knowledge base is now refreshed from the Vulnerability-Lookup API by
tools/cwe/update_cwe_knowledge_base.py, which also stores each CWE's
Mapping_Notes usage (Allowed, Allowed-with-Review, Discouraged,
Prohibited) in cwe_usage.json. tools/cwe/build_child_to_ancestor.py then
regenerates vulntrain/data/deep_child_to_ancestor.json, mapping each CWE
to its highest ancestor with an Allowed or Allowed-with-Review usage.
The label set goes from 26 discouraged ancestors to 303 allowed CWEs;
CWEs with no allowed entry on their ancestor path (e.g. CWE-20, CWE-200)
are left out so the trainer drops those examples. The trainer itself is
unchanged apart from loading its mapping relative to the package.

Also finalize the path fixes in the CWE mapping scripts that were left
unstaged by the previous restructuring commit.

Author: cedricbonhomme

tools/cwe/README.md

@@ -7,26 +7,30 @@ regenerated.
 ## Pipeline
 
 ```
-vulnerability.circl.lu.json          # raw CWE export from https://vulnerability.circl.lu
+python tools/cwe/update_cwe_knowledge_base.py   # fetches from the Vulnerability-Lookup API
         │
-        ▼  python tools/cwe/mapping_child_parent_cwe.py
-parent_to_children_mapping.json      # parent CWE → direct children
+        ├── vulnerability.circl.lu.json      # raw CWE records
+        └── cwe_usage.json                   # per-CWE mapping usage (Allowed, Discouraged, ...)
         │
-        ▼  python tools/cwe/hierarchy.py
-cwe_hierarchy.json                   # CWEs grouped by hierarchy depth level
+        ▼  python tools/cwe/build_child_to_ancestor.py
+vulntrain/data/deep_child_to_ancestor.json   # CWE → highest allowed ancestor (trainer input)
 ```
 
+Each CWE is mapped to its highest ancestor whose MITRE mapping usage is
+*Allowed* or *Allowed-with-Review*, so the CWE classifier is never trained to
+predict a Discouraged or Prohibited CWE (issue #17). CWEs without any allowed
+entry on their ancestor path (e.g. CWE-20, CWE-200) are excluded: the trainer
+drops the corresponding examples.
+
 ## Files
 
 | File | Role |
 |------|------|
-| `vulnerability.circl.lu.json` | Raw CWE data snapshot (input). |
+| `update_cwe_knowledge_base.py` | Refreshes the knowledge base from `https://vulnerability.circl.lu/api/cwe/`. |
+| `vulnerability.circl.lu.json` | Generated: raw CWE data snapshot. |
+| `cwe_usage.json` | Generated: per-CWE name, abstraction, and mapping usage. |
+| `build_child_to_ancestor.py` | Builds `vulntrain/data/deep_child_to_ancestor.json` (shipped with the package, consumed by `vulntrain-train-cwe-classification`). |
 | `mapping_child_parent_cwe.py` | Builds `parent_to_children_mapping.json` from the raw snapshot. |
 | `parent_to_children_mapping.json` | Generated: parent → children mapping. |
 | `hierarchy.py` | Builds `cwe_hierarchy.json` (hierarchy levels) from the parent/children mapping. |
 | `cwe_hierarchy.json` | Generated: CWE IDs grouped by depth level. |
-
-The mapping consumed at training time by
-`vulntrain-train-cwe-classification` lives in
-`vulntrain/data/deep_child_to_ancestor.json` (CWE child → top ancestor); it is
-maintained by hand and shipped with the package.

tools/cwe/build_child_to_ancestor.py

@@ -0,0 +1,88 @@
+"""Build the child-to-ancestor CWE mapping used by the CWE classifier.
+
+Regenerates ``vulntrain/data/deep_child_to_ancestor.json`` from the knowledge
+base produced by ``update_cwe_knowledge_base.py``. Each CWE is mapped to its
+highest ancestor whose MITRE mapping usage is *Allowed* or
+*Allowed-with-Review*, so the classifier is never trained to predict a
+Discouraged or Prohibited CWE (e.g. the Pillars CWE-664, CWE-707...).
+See https://github.com/vulnerability-lookup/VulnTrain/issues/17
+
+CWEs with no allowed entry anywhere on their ancestor path are left out of
+the mapping: the trainer then drops the corresponding training examples.
+"""
+
+import json
+from pathlib import Path
+
+ALLOWED_USAGES = {"Allowed", "Allowed-with-Review"}
+
+HERE = Path(__file__).parent
+OUTPUT = HERE.parent.parent / "vulntrain" / "data" / "deep_child_to_ancestor.json"
+
+
+def primary_parent(record: dict) -> str | None:
+    related = (record.get("Related_Weaknesses") or {}).get("Related_Weakness")
+    if not related:
+        return None
+    if isinstance(related, dict):
+        related = [related]
+    child_of = [rel for rel in related if rel.get("@Nature") == "ChildOf"]
+    if not child_of:
+        return None
+    for rel in child_of:
+        if rel.get("@Ordinal") == "Primary":
+            return rel.get("@CWE_ID")
+    return child_of[0].get("@CWE_ID")
+
+
+def main() -> None:
+    with open(HERE / "vulnerability.circl.lu.json") as f:
+        records = json.load(f)["data"]
+    with open(HERE / "cwe_usage.json") as f:
+        usage = json.load(f)
+
+    parent = {
+        record["@ID"]: primary_parent(record)
+        for record in records
+        if primary_parent(record)
+    }
+
+    def highest_allowed_ancestor(cwe_id: str) -> str | None:
+        best = None
+        node: str | None = cwe_id
+        visited = set()
+        while node and node not in visited:
+            visited.add(node)
+            if usage.get(node, {}).get("usage") in ALLOWED_USAGES:
+                best = node
+            node = parent.get(node)
+        return best
+
+    mapping = {}
+    unmappable = []
+    for record in records:
+        cwe_id = record["@ID"]
+        ancestor = highest_allowed_ancestor(cwe_id)
+        if ancestor:
+            mapping[cwe_id] = ancestor
+        else:
+            unmappable.append(cwe_id)
+
+    with open(OUTPUT, "w") as f:
+        json.dump(dict(sorted(mapping.items(), key=lambda kv: int(kv[0]))), f, indent=2)
+
+    labels = sorted(set(mapping.values()), key=int)
+    discouraged = [
+        cwe for cwe in labels if usage.get(cwe, {}).get("usage") not in ALLOWED_USAGES
+    ]
+    print(f"{len(mapping)} CWEs mapped to {len(labels)} labels -> {OUTPUT}")
+    print(f"Labels with non-allowed usage: {discouraged or 'none'}")
+    if unmappable:
+        print(
+            f"{len(unmappable)} CWEs without any allowed ancestor "
+            f"(excluded): {sorted(unmappable, key=int)}"
+        )
+
+
+if __name__ == "__main__":
+    main()