[internal] Add submit log flag to forward logs to DD

[yoannmoinet]

What and why?

This PR enhances our internal telemetry capabilities by implementing log forwarding to Datadog for better observability of build processes.

The changes enable us to monitor build plugin performance and issues in production environments while ensuring these operations don't impact build performance.

The list of changes can be daunting, but there is a lot of code moving but not a lot of actual changes/additions.

Moves:

• logger code has been split out.
• sendLog (to Datadog) has been extracted and moved to core helpers.
• GlobalContext now use simpler and more contained arguments data and stores.

Changes:

• Add more mocks to the tests (data, stores, etc...) and factorise others (bundler, logger, etc...).
• Internal telemetry on our own repository builds.
• Fix yarn cli create-plugin command.

Additions:

• async-queue-plugin.
• { forward: boolean } option to logger.

Key improvements:

• Internal telemetry: Only enable telemetry for our main build (removed from types generation and sub-builds). This will reduce the amount of metrics we send.
• Non-blocking operations: New async queue system ensures we can do async actions without delaying the build process, while still waiting for them to finish before quitting the build.
• Architecture improvements:
  • Better isolation of data and stores in the global context system.
  • Refactored how we log things following the data/stores management changes.
  • Changed how we mock loggers in tests to avoid circular dependencies in some mocking situations.

How?

The implementation is organized around three main themes:

1. Log Forwarding Infrastructure

• Added submitLog helper in packages/core/src/helpers/log.ts to send logs to Datadog intake API.
• Refactored logger (packages/factory/src/helpers/logger.ts) to support conditional log forwarding.
  • Simply use log.error('New error happened.', { forward: true }); to forward any log.
• Refactored context.sendLog following the previous changes.

2. Async Queue Plugin

• Created new @dd/async-queue-plugin to handle non-blocking operations during the build
  • Using it when sending and forwarding logs to DD.
• Integrated queue into factory context as context.queue(), allowing plugins to defer work without blocking builds.

3. Architecture Improvements

• Fixed circular dependencies in test mocks by refactoring timeLoggers mocking approach.
• Created GlobalStores and GlobalData types to strictly define what the global context is using.
• GlobalContext is now built from data and stores passed as arguments.

Hopefully this will offer new ways of understanding usages of the plugins while simplifying the architecture of the whole system.

[yoannmoinet]

This is just a fix for the yarn cli create-plugin command that would produce a faulty package.json breaking any yarn run afterward.

[cy-moi]

It seems like we don't give access to context.sendLog anymore, is this a breaking change?

[yoannmoinet]

Good catch, we're still give access to the function, but it's not defined from there anymore.

I forgot to document it from the context now.
Will do.

[sethfowler-datadog]

Nice; this will be a very handy feature to have! LGTM.

[sethfowler-datadog]

Nit: perhaps call getSendLog() once outside of this closure and capture the result?

[yoannmoinet]

No strong feeling about it, but curious as to why?

[sethfowler-datadog]

I wonder if it's worth setting forward: true here, so that these errors could have a chance to get reported to the server?

It'd admittedly be a bit of a pain, since you have to deal with the fact that the network request will get pushed onto stores.queue, so you'd have to handle that and do a second round of awaiting. (Presumably giving up on forward: true if there are still errors.) So, no strong feelings, but I figured it was worth considering at least.

[yoannmoinet]

For now it would probably be correct, but the more we go, the more I'd be worried to get all kind of failures unrelated to "us".

Like it could fail from custom plugins for instance.