Skip to content

AGENTS.md

Latest commit

 

History

History
285 lines (231 loc) · 12.9 KB

AGENTS.md

File metadata and controls

285 lines (231 loc) · 12.9 KB
Raw

AGENTS.md

Architecture Map

Apache Texera: Scala/sbt backend services + the Amber workflow execution engine, an Angular UI, and the agent service. JVM modules wired in build.sbt.

Area Path Detail
Workflow execution engine (Amber) amber/ amber/README.md
Backend services config-service/, access-control-service/, file-service/, computing-unit-managing-service/, workflow-compiling-service/, notebook-migration-service/ build.sbt
Shared Scala libs common/ (auth, config, dao, util, workflow-core, workflow-operator, pybuilder) build.sbt
Frontend (Angular) frontend/ frontend/README.md
Agent service (Bun/TS, LLM agents) agent-service/ agent-service/package.json
Pyright language service pyright-language-service/ pyright-language-service/README.md
Deploy scripts / Dockerfiles bin/ README / k8s / single-node
DDL, sbt plugins sql/, project/ files therein

Amber breakdown

Path Role
amber/src/main/scala Pekko actors, scheduler, reconfiguration, fault tolerance, gRPC/proto
amber/src/main/python/pyamber Python engine (pyamber) — bridge to the Scala engine
amber/src/main/python/pytexera Python operator SDK exposed to UDFs

Where Things Live

Topic Source of truth
Contribution / PR / lint / format / testing / license header CONTRIBUTING.md
Reporting security issues SECURITY.md
PR template .github/PULL_REQUEST_TEMPLATE
Issue templates bug / task / feature
License-header coverage; vendored workflow-operator .licenserc.yaml; project/AddMetaInfLicenseFiles.scala
Run the local dev stack (infra in Docker; backend/frontend/agent-service native) bin/local-dev.sh
Single-node / k8s deploy single-node, k8s

If a topic is above, read that file instead of asking here.

Agent-Specific Rules

Scope and safety

  • Narrowly scoped changes. No unrelated rewrites or cross-service moves.
  • git status --short before editing; don't revert unrelated dirty files.
  • Never commit secrets / local config / build output / caches / binaries (python_udf.conf, .env, target/, dist/, .pytest_cache/, .ruff_cache/, logs).

Develop in a worktree

Leave texera/ on main. One worktree per PR, branched off a freshly fetched upstream/main.

texera/                      # stays on main, never dirty
texera-worktrees/<branch>/   # one worktree per PR

Reset to upstream/main at start; git log upstream/main..HEAD should contain only this PR's commits before pushing; remove the worktree after merge.

Prefer bin/local-dev.sh to run the stack while developing. Its native services bind fixed ports and share one PID/state dir, so only one worktree's stack runs at a time: bin/local-dev.sh down in the old worktree before switching, then up in the new one. Use the non-interactive CLI subcommands (up / down / status / logs); the interactive TUI (-i) is for humans, not agents.

Environment

Component Version
Java JDK 17
Scala 2.13
Python 3.12
Node 24

One Python venv shared across worktrees, sibling of the texera checkout:

<workspace>/
├── texera/                   # main checkout
├── texera-worktrees/<br>/    # per-PR worktrees
└── venv312/                  # shared Python 3.12 venv

python3.12 -m venv ../venv312 && source ../venv312/bin/activate
pip install -r amber/requirements.txt -r amber/operator-requirements.txt
# For pytest or running bin/python-proto-gen.sh, also install dev deps:
pip install -r amber/dev-requirements.txt

Tests that spawn Python workers need an interpreter path. Edit python.path in udf.conf or export UDF_PYTHON_PATH="$(pwd)/../venv312/bin/python" (env var overrides). Without it, sbt Python-integration tests fail to launch a worker.

.jvmopts holds every --add-opens flag Texera needs for JDK 17+, with each group annotated by its upstream source (Kryo, Apache Arrow, Apache Pekko). sbt's launcher and the .run/ configs read it automatically; for raw java launches, pass it as an argfile: java @.jvmopts -jar …. If a future library version or a new code path triggers an InaccessibleObjectException, add the open to .jvmopts. project/JdkOptions.scala will propagates the changed options to forked test JVMs, sbt-native-packager dist launchers, and IntelliJ.

Branch and commit naming

Short, Conventional Commits, same shape for branch and commit subject.

Kind Branch Commit
Feature feat/agent-workflow-edit feat(agent-service): enable workflow edit
Bug fix fix/marker-replay fix(amber): marker replay during reconfiguration
Tests test/pyamber-handlers test(pyamber): add handler unit tests
Chore chore/angular-21 chore(deps, frontend): upgrade to Angular 21
CI ci/merge-queue-stacking ci: stack merge-queue builds by module

Both ≤ ~60 chars. For code changes, if you use a scope, use the module name (amber, pyamber, frontend, agent-service, file-service, …) — not amber-python. No Co-authored-by: trailer for the repo owner.

Choosing the type turns on what happens to the behavior, not on how big the diff is:

The change Type
Worked before, broken now fix
Support never existed; adding it feat
Support exists; removing it feat
Reworked so user-facing behavior intentionally changes feat
User-facing behavior unchanged refactor

Behavior is what the code does, not what a doc or an old PR description claims it does: implementing something that was never actually there is a feat.

refactor claims the user-facing behavior is identical. Tests that pin a user-facing API must pass untouched — editing one of those assertions means the behavior moved, so it is a feat or a fix. Tests that pin internals (a private helper's signature, call order between collaborators, the shape of an intermediate value) mirror the implementation, so rewriting them alongside the code they mirror is still a refactor.

Tests. A test-only PR is test(<module>): .... Repairing a broken or flaky test is a bug fix in test code: fix(test, <module>): ....

Dependencies. fix only when the bump carries a security fix:

Bump Commit
Patches a CVE fix(deps, <module>): ...
Everything else chore(deps, <module>): ...
GitHub Actions chore(deps, ci): ...

Omit the module for cross-module bumps (sbt). GitHub Actions bumps take ci as their module — that is what .github/renovate.json5 opens them with; a bare ci: ... is for hand-written CI and workflow changes.

Backports. A PR targeting release/vX.Y appends the version as the last scope component — fix(deps, frontend, v1.2): .... Version tags belong only on release-branch PRs, never on one targeting main.

Issues and PRs

Issue-first; both stay short.

issue (template + Type)  ->  /take  ->  PR (Closes #N, template)  ->  review  ->  merge

  • Every change starts as an issue (minor typo / docs excepted). File against apache/texera, never a fork.

  • Pick the right template and set the GitHub Issue Type explicitly (Bug / Task / Feature); the template's type: frontmatter doesn't always apply on creation.

  • Claim before coding. The assignee is the claim; a local branch is not. An issue is taken if anyone but you is assigned, or a linked PR is OPEN and someone else wrote it (an outside PR opener isn't always auto-assigned):

    gh api user --jq .login                                  # you
gh issue view N --repo apache/texera --json assignees,closedByPullRequestsReferences
gh pr view P --repo apache/texera --json author,state    # each linked PR

    Not taken: /take it, even one you just filed (comment commands; /untake if you drop it). Taken: it's theirs. Don't start; tell the human.

  • Reference the issue: Closes #N (or Fixes / Resolves, or "related to"). Right before opening the PR, re-run the check for every issue your diff fixes, closed or only related, including each site of a bundled PR. If one is taken, leave that issue and its code change out and tell the human: merging resets each closed issue's assignees to the PR's authors (pr-assignment.yml).

  • Issue titles are plain prose; never use the Conventional Commits format (type(scope): ...) — that prefix is for commit and PR titles only.

  • Task issues match task-template.yaml exactly.

  • Prefer tables and small ASCII diagrams over long bullets. Don't restate the diff or the template.

  • For bugs, lead with root cause and a before -> after sketch:

    Before:  reconfiguration -> replay marker -> worker hangs
After:   reconfiguration -> replay marker -> resume from checkpoint

  • Frontend PRs: any visible UI change requires screenshots / GIF, before / after side by side. For purely visual fixes that's the primary verification under "How was this PR tested?"; interactive flows also list manual steps (click path, browser, viewport).

Tests come first

TDD. Write the test before the source change.

write/adjust test (red)  ->  edit source (green)  ->  refactor
Situation Order
New feature / behavior change Failing test, then implement.
Bug fix Regression test reproducing the bug, then fix.
Code with no tests Characterization tests pin current behavior first; only then change source.
Refactor (no user-facing behavior change) Tests stay green throughout. User-facing API assertions stay untouched; tests that mirror internals may be rewritten with the code.

Every test must cover:

  • Both directions: positive (valid → expected) and negative (invalid / error → specific failure mode).
  • Edge cases: empty / null / zero / max / boundary, unicode, concurrency/order, missing or malformed config.
  • Don't assume valid. External input (user / API / file / message) must be tested with bad input.

Don't claim "tested" without commands. Paste the exact sbt testOnly / pytest / yarn test:ci / bun test invocation under "How was this PR tested?".

CI labels & gating

CI runs are selected by PR labels, not by file diff.

diff -> pr-labeler -> labels on PR -> required-checks maps labels to stacks -> CI runs
  • Path → label rules: .github/labeler.yml
  • Label → stacks (LABEL_STACKS, source of truth): .github/workflows/required-checks.yml. Read it directly; don't duplicate the mapping here.
  • Need extra coverage the diff doesn't imply (e.g. a common/ change you suspect breaks the frontend)? Add the relevant label manually.
  • Empty stack union (docs-only / dev-only / dependencies / feature / fix / refactor / release/* only) skips every build stack on purpose.
  • release/* labels nominate backport targets. A nominated target is backported only once that branch's release manager — listed in .github/release-branches.yml — approves the PR, and the required Backport Approvals check blocks the merge until every release/* label on the PR is approved. A manager declines by removing their label, so the labels on a merged PR are exactly the branches it was backported to.