Skip to content

Add system architecture documentation#7249

Open
pingsutw wants to merge 15 commits intoflyteorg:v2from
pingsutw:add-architecture-doc
Open

Add system architecture documentation#7249
pingsutw wants to merge 15 commits intoflyteorg:v2from
pingsutw:add-architecture-doc

Conversation

@pingsutw
Copy link
Copy Markdown
Member

@pingsutw pingsutw commented Apr 21, 2026

Tracking issue

N/A

Why are the changes needed?

The Flyte 2 backend currently lacks a centralized architecture document. This makes it harder for new contributors and users to understand how the system components fit together, how data flows through the platform, and what role each service plays.

What changes were proposed in this pull request?

Add docs/architecture.md covering:

  • High-level overview with ASCII architecture diagrams (control plane + data plane)
  • End-to-end data flow from CreateRun through pod execution to streaming results
  • Component reference for all services (Runs, Executor, Actions, DataProxy, Cache, Events, Secret, App, Copilot, Manager)
  • Shared libraries (flytestdlib, flyteplugins, flyteidl2)
  • Plugin system architecture with lifecycle and built-in plugin catalog
  • Infrastructure dependencies (Kubernetes, PostgreSQL, Object Storage)
  • Full API surface — user-facing and internal gRPC services
  • Deployment modes — unified, split, multi-cluster
  • Security layers
  • Key source locations quick-reference table

How was this patch tested?

Documentation only — no code changes.

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Document the Flyte 2 backend architecture including component
diagrams, data flow, plugin system, API surface, deployment modes,
and infrastructure dependencies.

Signed-off-by: Kevin Su <pingsutw@apache.org>
@pingsutw pingsutw added the added Merged changes that add new functionality label Apr 21, 2026
@pingsutw pingsutw self-assigned this Apr 21, 2026
@pingsutw pingsutw modified the milestone: V2 GA Apr 21, 2026
@pingsutw pingsutw force-pushed the add-architecture-doc branch from a745017 to cd13a3a Compare April 22, 2026 19:42
pingsutw added 10 commits April 22, 2026 12:44
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
- InternalRunService is called by the Actions Service (CRD watcher) and
  Events Service, not by the Executor or Copilot.
- Copilot sidecar is only injected for raw-container tasks; regular
  Python tasks read/write object storage directly from user code.

Signed-off-by: Kevin Su <pingsutw@apache.org>
…ient)

Signed-off-by: Kevin Su <pingsutw@apache.org>
…/WatchForUpdates/Abort)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
; branch 'v2' of github.com:flyteorg/flyte into add-architecture-doc
Signed-off-by: Kevin Su <pingsutw@apache.org>
@pingsutw pingsutw marked this pull request as ready for review April 27, 2026 23:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

added Merged changes that add new functionality flyte2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant