feat: add DAL (CM-951)

[ulemons]

Note

Medium Risk
Introduces a new Temporal ingestion path that streams large external datasets over HTTP and bulk upserts into Postgres, plus new DAL modules that will be used by other services. Risk is mainly around data correctness/performance and operational reliability (timeouts/retries, external API/bucket behavior).

Overview
Adds an Automatic Projects Discovery Temporal worker that discovers OSS repos from pluggable external sources and bulk upserts them into projectCatalog in batches, with new activities (listSources, listDatasets, processDataset) and a workflow mode switch (incremental latest-only vs full).

Introduces a source registry and two initial sources: OSSF Criticality Score (CSV snapshots from a public GCS bucket) and LF Criticality Score (paginated JSON API), including streaming + parsing/error propagation, and updates the Temporal schedule to run daily at midnight with a 2-hour workflow timeout.

Extends the data-access-layer with new project-catalog and evaluated-projects modules (CRUD, bulk insert/upsert helpers, and types) and exports them from the DAL index; also adds the csv-parse dependency and enables Postgres for the worker.

^{Written by Cursor Bugbot for commit e9852f1. This will update automatically on new commits. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

Pull request overview

Adds new Data Access Layer (DAL) modules for the project catalog and evaluated projects domain, and re-exports them from the DAL package entrypoint.

Changes:

• Introduces project-catalog DAL: types plus CRUD-ish query helpers (find/insert/bulk insert/upsert/update/delete).
• Introduces evaluated-projects DAL: types plus query helpers for evaluation lifecycle and bulk insert.
• Exports both modules via services/libs/data-access-layer/src/index.ts.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
services/libs/data-access-layer/src/project-catalog/types.ts	Defines DB-facing types for project catalog records and create/update payloads.
services/libs/data-access-layer/src/project-catalog/projectCatalog.ts	Adds SQL helpers for selecting/inserting/upserting/updating/deleting project catalog rows.
services/libs/data-access-layer/src/project-catalog/index.ts	Barrel export for the project-catalog module.
services/libs/data-access-layer/src/evaluated-projects/types.ts	Defines DB-facing types for evaluated projects and create/update payloads.
services/libs/data-access-layer/src/evaluated-projects/evaluatedProjects.ts	Adds SQL helpers for evaluated project operations (find/insert/bulk insert/update/mark evaluated/onboarded/delete).
services/libs/data-access-layer/src/evaluated-projects/index.ts	Barrel export for the evaluated-projects module.
services/libs/data-access-layer/src/index.ts	Re-exports the newly added modules from the DAL package entrypoint.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[themarolt]

@ulemons please check cursor bugbot comments - I think some are pretty valid.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

[cursor]

Ngrok development URL hardcoded as production default

High Severity

DEFAULT_API_URL is set to a temporary ngrok tunnel URL (https://hypervascular-nonduplicative-vern.ngrok-free.dev). If the LF_CRITICALITY_SCORE_API_URL environment variable is not configured, the LF criticality score source will attempt to connect to this ephemeral development endpoint. In production this will either fail outright (tunnel down) or connect to an unintended service.

 

[cursor]

HTTP responses not consumed on error or redirect

Low Severity

Both httpsGet and getHttpsStream reject or follow redirects without calling res.resume() on the response. The LF source's fetchPage correctly calls res.resume() on error (line 54), but these functions don't. Unconsumed HTTP responses prevent the underlying TCP socket from being released back to the agent's connection pool, which can lead to socket exhaustion during repeated bucket listing or when retries encounter errors.

Additional Locations (1)

• services/apps/automatic_projects_discovery_worker/src/sources/ossf-criticality-score/bucketClient.ts#L86-L89