data_model.md

Data model

Overview

The GitLab Knowledge Graph is composed of two primary sub-graphs that share a common schema foundation: the Namespace (SDLC) Graph and the Code Graph. This document details the data model for each, covering the nodes and relationships that constitute them.

The data model is designed to be intuitive and to mirror the mental model that developers and users have of the GitLab platform. By representing entities as nodes and their interactions as relationships, we can perform complex queries that would be difficult or inefficient with a traditional relational database.

The data model follows a Property Graph approach over an RDF approach, as GitLab data has strongly defined relationships between entities.

We can enable custom node and relationship expansion in the future by following the Property Graph approach and building the correct schema management capabilities.

Data Storage Location

The Knowledge Graph data is stored in ClickHouse graph tables that are separate from the raw replicated data lake tables.

• The implemented graph schema is defined in config/graph.sql.
• The implemented ontology metadata and ETL mappings are defined under config/ontology/.
• In deployed environments, operators can place the graph tables in a dedicated ClickHouse database or instance. The repository supports either a separate graph database or co-location within a broader ClickHouse deployment, depending on operational requirements.

Concepts to Know

• Unified ontology and shared graph primitives: Both the Code Graph and the SDLC Graph use the same ontology-driven entity and relationship model defined in config/ontology/ and the same ClickHouse graph schema in config/graph.sql. Edges are stored in ontology-configured edge tables (defaulting to gl_edge); each edge YAML can set a table: field to route specific relationship types to dedicated tables. This allows linking between the two graphs (e.g., a Project node from the SDLC graph can be linked to a Branch, File, or Definition from the Code Graph).
• Entity as Node: Every entity in the GitLab ecosystem (e.g., Project, Issue, File, Function Definition) is represented as a node.
• Interaction as Edge: Relationships between these entities (e.g., a User AUTHORED a Note, a File CONTAINS a Definition) are represented as directed edges.

---

The Namespace Graph Data Model

The Namespace Graph represents the software development lifecycle (SDLC) entities and their interactions within GitLab. It models how users, projects, issues, merge requests, and CI/CD components relate to one another.

Implemented Node Types

Node Type	Description	Key Properties
Group	Represents a GitLab group namespace.	id, name, full_path, visibility_level
Project	Represents a GitLab project/repository.	id, name, full_path, namespace_id
MergeRequest	Represents a GitLab merge request.	id, iid, title, state, source_branch, target_branch, project_id, merged_commit_id, denormalized metric_* snapshot columns
Pipeline	Represents a CI/CD pipeline.	id, status, source, project_id, user_id
Deployment	Represents a deployment of a commit to a CI/CD environment.	id, iid, project_id, status, ref, sha
Environment	Represents a CI/CD deployment target (production, staging, review app, etc.).	id, project_id, name, slug, state, tier, environment_type
Vulnerability	Represents a security vulnerability finding.	id, title, severity, state, project_id
User	Represents a GitLab user.	id, username, name
Note	Represents a comment or annotation on a GitLab object (issue, merge request, commit, vulnerability, etc.).	id, note, noteable_type, noteable_id, author_id
WorkItem	Represents a GitLab work item (issue, task, epic, objective, etc.).	id, iid, title, state, project_id, author_id
Milestone	Represents a milestone attached to projects or work items.	id, iid, title, state, due_date
Label	Represents a label applied to work items.	id, title, color
Branch	Represents a Git branch.	id, name, project_id, is_default
MergeRequestDiff	Represents a merge request diff version.	id, merge_request_id, project_id, state, diff_type, files_count, real_size, stored_externally
MergeRequestDiffFile	Represents a file inside a merge request diff.	id, merge_request_id, merge_request_diff_id, new_path, old_path, generated, a_mode, b_mode
Stage	Represents a CI stage.	id, name, status, position
Job	Represents a CI job (Ci::Build or Ci::Bridge).	id, name, status, ref, allow_failure, type, runner_id, timeout, timeout_source, exit_code, scheduling_type, auto_canceled_by_id
Runner	Represents a CI/CD runner (Ci::Runner). Global node, no traversal_path on the node table.	id, runner_type, name, active, locked, access_level
Finding	Represents a security_findings scan finding.	id, uuid, name, severity
SecurityScan	Represents a security scan run.	id, scan_type, status, latest
VulnerabilityOccurrence	Represents a concrete vulnerability occurrence (Vulnerabilities::Finding in Rails).	id, uuid, report_type, severity, location
Package	Represents a package published to the package registry (npm, Maven, PyPI, NuGet, and other formats).	id, name, version, package_type, status, project_id
PackageFile	Represents a file stored for a package in the package registry (the artifact itself, for example a .whl or .jar).	id, file_name, size, file_sha256, status, project_id
ContainerRepository	Represents a container image repository in a project's container registry.	id, name, status, project_id
Dependency	Represents a dependency declared by a package (npm, Maven, NuGet, Composer, and other formats).	id, name, version_pattern, project_id
VulnerabilityScanner	Represents the scanner that produced vulnerability data.	id, external_id, name, vendor
VulnerabilityIdentifier	Represents a vulnerability identifier such as CVE or GHSA.	id, external_type, external_id, name

Relationship Visualization

graph TD
    Group -- CONTAINS --> Project
    Group -- CONTAINS --> Group
    User -- CONTAINS --> Project
    Project -- CONTAINS --> Branch
    WorkItem -- CONTAINS --> WorkItem

    MergeRequest -- IN_PROJECT --> Project
    Vulnerability -- IN_PROJECT --> Project
    Pipeline -- IN_PROJECT --> Project
    Branch -- IN_PROJECT --> Project

    User -- AUTHORED --> MergeRequest
    User -- AUTHORED --> WorkItem
    User -- AUTHORED --> Note
    MergeRequest -- HAS_NOTE --> Note
    MergeRequest -- HAS_LABEL --> Label

    MergeRequest -- CLOSES --> WorkItem
    MergeRequest -- TRIGGERED --> Pipeline
    User -- TRIGGERED --> Job

    WorkItem -- IN_GROUP --> Group
    User -- APPROVED --> MergeRequest
    User -- MERGED --> MergeRequest
    User -- REVIEWER --> MergeRequest
    User -- CLOSED --> WorkItem
    User -- REOPENED --> WorkItem

    Package -- IN_PROJECT --> Project
    Package -- BUILT_BY --> Pipeline
    Package -- DECLARES_DEPENDENCY --> Dependency
    Package -- HAS_VULNERABILITY --> Vulnerability
    Package -- HAS_PACKAGE_FILE --> PackageFile
    PackageFile -- IN_PROJECT --> Project
    PackageFile -- BUILT_BY --> Pipeline
    ContainerRepository -- IN_PROJECT --> Project
    Dependency -- IN_PROJECT --> Project

Loading

Implemented Relationship Types

Relationship	From Node	To Node	Description
CONTAINS	Group, User, Project, WorkItem	Group, Project, Branch, WorkItem	A group contains a subgroup or project; a user namespace contains a project; a project contains a branch; a work item contains a child work item.
IN_PROJECT	Branch, WorkItem, Pipeline, Stage, Job, Vulnerability, Finding, VulnerabilityOccurrence, VulnerabilityIdentifier, Milestone, Label, SecurityScan, Deployment, Environment, MergeRequestDiff, Note, MergeRequest, Package, PackageFile, ContainerRepository, Dependency	Project	An entity belongs to a project. (FK on each node.)
BUILT_BY	Package, PackageFile	Pipeline	A package or package file was built by a CI/CD pipeline (sourced from the packages_build_infos and packages_package_file_build_infos join tables).
HAS_PACKAGE_FILE	Package	PackageFile	A package contains a package file (FK on the package file).
DECLARES_DEPENDENCY	Package	Dependency	A package declares a dependency (sourced from the packages_dependency_links join table).
HAS_VULNERABILITY	Package	Vulnerability	A registry package is affected by a vulnerability, matched to its SBOM component occurrence by name/version (sourced from the sbom_occurrences_vulnerabilities join and sbom_occurrences/sbom_component_versions).
IN_GROUP	WorkItem, Milestone, Label	Group	An entity belongs to a group scope.
AUTHORED	User	Note, MergeRequest, Vulnerability, WorkItem	A user authored an entity.
CLOSES	MergeRequest	WorkItem	A merge request closes a work item.
TRIGGERED	MergeRequest	Pipeline	A merge request triggered a pipeline.
TRIGGERED	User	Pipeline, Job	A user triggered a pipeline or job directly.
CLOSED	User	WorkItem, MergeRequest	A user closed a work item or merge request.
MERGED	User	MergeRequest	A user merged a merge request.
REOPENED	User	WorkItem, MergeRequest	A user reopened a work item or merge request (sourced from resource_state_events, state = 5).
APPROVED	User	MergeRequest	A user approved a merge request.
REVIEWER	User	MergeRequest	A user is a reviewer of a merge request.
CONFIRMED_BY	User	Vulnerability	A user confirmed a vulnerability.
DISMISSED_BY	User	Vulnerability	A user dismissed a vulnerability.
RESOLVED_BY	User	Vulnerability	A user resolved a vulnerability.
HAS_JOB	Stage, Pipeline	Job	A stage contains jobs (canonical Pipeline → Stage → Job traversal); also exposed directly as Pipeline → Job for the natural CI mental model.
IN_PIPELINE	Job, SecurityScan	Pipeline	A job or security scan belongs to a pipeline (one-hop replacement for Pipeline → Stage → Job).
HAS_STAGE	Pipeline	Stage	A pipeline contains stages.
AUTO_CANCELED_BY	Pipeline, Job	Pipeline, Job	Entity was auto-canceled when a newer entity of the same kind superseded it.
CHILD_OF	Pipeline	Pipeline	A pipeline is a downstream child of a parent pipeline (sourced from ci_sources_pipelines).
TRIGGERS_PIPELINE	Job	Pipeline	A bridge job (type='Ci::Bridge') triggered the downstream pipeline.
TRIGGERED_BY_PIPELINE	Job	Pipeline	A job runs in a child pipeline whose parent is the upstream pipeline (sourced from upstream_pipeline_id on builds).
RUNS_ON	Job	Runner	The runner that executed the job.
HAS_NOTE	MergeRequest, WorkItem, Vulnerability	Note	An entity has notes attached.
HAS_LABEL	WorkItem, MergeRequest	Label	An entity has labels applied.
IN_MILESTONE	WorkItem, MergeRequest	Milestone	An entity belongs to a milestone.
HAS_DIFF	MergeRequest	MergeRequestDiff	A merge request has diff versions.
HAS_LATEST_DIFF	MergeRequest	MergeRequestDiff	A merge request's most recent diff snapshot (from latest_merge_request_diff_id).
HAS_FILE	MergeRequestDiff	MergeRequestDiffFile	A diff version contains files.
SOURCE_PROJECT	MergeRequest	Project	Source project for the merge request (differs from target project for fork-based MRs).
HAS_HEAD_PIPELINE	MergeRequest	Pipeline	The pipeline running against the tip of the source branch.
UPDATED_BY	User	MergeRequest	User who most recently updated the merge request.
LAST_EDITED_BY	User	MergeRequest	User who most recently edited the merge request's content (title/description).
HAS_FINDING	SecurityScan, Vulnerability	Finding, VulnerabilityOccurrence	A security scan produced findings, or a vulnerability points at its canonical occurrence.
HAS_IDENTIFIER	Vulnerability, Finding, VulnerabilityOccurrence	VulnerabilityIdentifier	An entity is associated with vulnerability identifiers.
DETECTED_BY	Finding, VulnerabilityOccurrence	VulnerabilityScanner	Security data is associated with a scanner.
OCCURRENCE_OF	VulnerabilityOccurrence	Vulnerability	A vulnerability occurrence is linked to a vulnerability.
DEPLOYED_BY	User	Deployment	User who triggered the deployment.
DEPLOYED_TO	MergeRequest	Deployment	Merge request was included in a deployment.
IN_ENVIRONMENT	Deployment	Environment	Deployment targets a specific environment.
CREATED_FOR_MR	Environment	MergeRequest	Environment was first created by a merge request pipeline.
MEMBER_OF	User	Group, Project	A user is a member of a group or project (with access level and membership state).
CREATOR	User	Project	A user created a project.
OWNER	User	Group	A user owns a group.
ASSIGNED	User	MergeRequest, WorkItem	A user is assigned to a merge request or work item.
FIXES	MergeRequest	Vulnerability	A merge request fixes a vulnerability.
RELATED_TO	WorkItem	WorkItem	A work item is related to or blocks another work item (edge property link_type: relates_to, blocks).
MERGED_AT_COMMIT	MergeRequest	Commit	The commit a merge request was merged at. Write-once FK (immutable).
SCANS	VulnerabilityScanner	Project	A vulnerability scanner scans a project.
RAN_BY	SecurityScan	Job	A security scan was executed by a CI job.

---

The Code Graph Data Model

The Code Graph represents the structure and relationships within the source code of a repository. It models the file system hierarchy, code definitions, and the call graph.

Node Types

Node Type	Description	Key Properties
Branch	Root of the code file tree for a specific branch.	id, name, project_id, is_default
Commit	A Git commit, shared target for commit-related edges.	id, sha, project_id
Directory	Represents a directory within a repository.	relative_path, absolute_path, repository_name
File	Represents a file within a repository.	relative_path, absolute_path, language, repository_name
Definition	A code definition such as a class, function, method, or module.	fqn, name, definition_type, file_path, start_line, end_line, branch, commit_sha, virtual content
ImportedSymbol	An imported symbol or module reference within a file.	import_path, import_type, identifier_name, identifier_alias, file_path

Relationship Visualization

graph TD
    Branch -- CONTAINS --> Directory
    Branch -- CONTAINS --> File
    Branch -- IN_PROJECT --> Project
    Directory -- CONTAINS --> Directory
    Directory -- CONTAINS --> File
    File -- DEFINES --> Definition
    File -- IMPORTS --> ImportedSymbol
    File -- CALLS --> Definition
    Definition -- DEFINES --> Definition
    Definition -- CALLS --> Definition
    Definition -- CALLS --> ImportedSymbol
    Definition -- EXTENDS --> Definition
    ImportedSymbol -- IMPORTS --> Definition

Loading

Relationship Types

Relationship	From Node	To Node	Description
CONTAINS	Branch, Directory	Directory, File	A branch or directory lexically contains a directory or file.
IN_PROJECT	Branch	Project	A branch belongs to a project (links the Code Graph to the Namespace Graph).
ON_BRANCH	Directory, File, Definition, ImportedSymbol	Branch	Snapshots a code-graph node to a specific branch and commit.
DEFINES	File, Definition	Definition	File-level definition or lexical nesting between two definitions (e.g. a class containing methods). Inheritance is EXTENDS; call sites are CALLS.
IMPORTS	File, ImportedSymbol	ImportedSymbol, Definition	Module-system import edges. Covers File → ImportedSymbol for the import statement itself, and ImportedSymbol → Definition resolution (JS/TS today).
CALLS	File, Definition	Definition, ImportedSymbol	Function or method invocation. Variants: Definition → Definition (resolved call), File → Definition (top-level call outside any definition), and Definition → ImportedSymbol (call whose target is still an unresolved import).
EXTENDS	Definition	Definition	Supertype declaration. Covers class extension, interface implementation, and struct embedding (Go).

---

Cross-Graph Relationships

The Project and Branch nodes bridge the SDLC and Code graphs. A Project exists in the SDLC graph, while a Branch belongs to that project via IN_PROJECT and contains the root-level Directory and File nodes via CONTAINS. Cross-graph queries can traverse shared project, branch, and review entities even when edges live in different physical tables, because the compiler emits UNION ALL across all relevant edge tables for wildcard and multi-table relationship queries.

Namespace partitioning

The node entities in settings.partition.include_entities and the edge tables in settings.partition.include_edge_tables are PARTITION BY a hash bucket of the top-level namespace. Tables too small to hit part-count pressure (e.g. Project, Group, Milestone) are left off those lists and stay single-partition ReplacingMergeTrees, since partitioning them only adds per-partition metadata overhead. Global hubs (User, Runner) have no traversal_path and are never partitioned. The strategy is declared once in config/ontology/schema.yaml under settings.partition.strategy and rendered into config/graph.sql:

PARTITION BY (modulo(sipHash64(toUInt64OrZero(arrayElement(splitByChar('/', traversal_path), 2))), 50))

splitByChar('/', '42/100/1000/')[2] is the top-level namespace id (100); global edges write traversal_path = '0/', so they hash to bucket 0 and co-locate there. ClickHouse derives the bucket from traversal_path at insert and merge time, so there is no stored partition column and no write-side code: the indexer writes exactly the columns it always has.

The primary goal is ingest/merge isolation. Without partitioning, all ~1,100 top-level namespaces share one part budget per table, so one tenant's reindex burst can exhaust parts_to_throw_insert and dead-letter inserts for every tenant. Bucketing gives each bucket its own part budget and merge scope, so a busy tenant throttles only its own bucket. Because the partition is a deterministic function of traversal_path and every sort key leads with traversal_path, all versions of a row land in the same bucket, so ReplacingMergeTree FINAL dedup stays correct.

The hash scatters prefixes across buckets, so a startsWith(traversal_path, …) filter alone cannot prune partitions. Wherever a scan's filter already confines an alias to one or more top-level namespaces via startsWith(alias.traversal_path, '<prefix>'), the compiler ANDs a predicate on ClickHouse's _partition_id virtual column (the per-part formatted partition value, resolved from part metadata — no column read). The bucket constant is computed in ClickHouse from the prefix via the same strategy expression the DDL uses, so the two are byte-identical by construction:

-- single pinned namespace
p._partition_id = toString(modulo(sipHash64(toUInt64OrZero(arrayElement(splitByChar('/', '42/100/'), 2))), 50))

-- several pinned namespaces (a multi-path authorization OR)
p._partition_id IN tuple(
  toString(modulo(sipHash64(toUInt64OrZero(arrayElement(splitByChar('/', '42/100/'), 2))), 50)),
  toString(modulo(sipHash64(toUInt64OrZero(arrayElement(splitByChar('/', '42/200/'), 2))), 50))
)

ClickHouse folds the constants to bucket strings and prunes parts before reading data. This is a pure optimization layered on the security pass's authorization filters: emitting a superset of buckets (or nothing) is always safe, and a pinning startsWith proves the rows can only live in the matched bucket(s), so the predicate can never drop an authorized row.

A scan on an unpartitioned table never gets a _partition_id predicate: its only part is all, so a bucket-string predicate there would match nothing. Otherwise it prunes whenever the set of top-level namespaces an alias can occupy is finite and known at compile time (a scoped or multi-path authorized set). A namespaced scan with no pinning prefix — a far node reached through a cross-namespace edge — still cannot hold rows outside the caller's authorized top-level namespaces, so it falls back to _partition_id IN (<bucket of each authorized TLN>), over-scanning within the tenant instead of across all buckets. This fallback is skipped when any authorized path is org-only (pins no namespace) or the authorized set spans at least as many namespaces as there are buckets. The whole pass is a no-op for admins. Pruning never narrows below what the security pass authorizes; the bucket set is always derived from the full authorized set, a safe superset of any role-floored subset.