docs: complete MAINTENANCE.md and clarify maintainers responsibility (#1314)

**Motivation:**

complete MAINTENANCE.md and clarify maintainers responsibility

**Modifications:**

complete MAINTENANCE.md

**Result:**

clarify maintainers responsibility

Author: bowenli86

MAINTENANCE.md

@@ -1,72 +1,245 @@
-# Maintenance
+# Maintenance Guide
 
-This document outlines basic procedures for maintenance of PRs, and releases that maintainers of the repository should follow.
+This document outlines essential procedures for repository maintenance, including PR management, branching strategies, and release processes that maintainers should follow to ensure code quality and stability.
 
+## Code Review Process
 
-## Enrivonment Network
+Code reviewers are responsible for ensuring that PRs meet the following criteria before merging:
 
-Before getting into maintenance details, let's define environment network first.
+- **Properly labeled** according to our labeling conventions
+- **Well-formatted** code adhering to our style guidelines
+- **Comprehensive test coverage** including unit and integration tests
+- **Documentation updates** where appropriate
+- **Security considerations** addressed and potential vulnerabilities mitigated
+
+For first-time external contributors, a maintainer must:
+1. Review the PR with additional scrutiny
+2. Verify the contributor has signed the CLA (Contributor License Agreement)
+3. Manually approve CI workflows to run
+4. Provide detailed feedback to help onboard the contributor
+
+
+## Branching and Merge Strategy for Development
+
+### Core Branches
+
+- **`main` (default)**: 
+  - The canonical branch where all approved code changes are merged
+  - Must remain stable and suitable for internal/external testing
+  - All CI workflows must pass before and after every commit
+  - Security audits are performed against specific commits on this branch
+
+- **`release-dev/xxx`**:
+  - Used for large features requiring multiple commits that introduce breaking changes
+  - Examples: `release-dev/slashing-v1`, `release-dev/rewards-v2`
+  - Branch from `main` for isolated development
+  - Must be regularly rebased against `main` to reduce merge conflicts
+  - Merged back into `main` as a single logical unit when complete
+  - No security audits are performed directly on these branches
+
+### Merge Strategies
+
+We employ different merge strategies based on PR type:
+
+#### 1. Regular PRs
+Regular PRs include bug fixes, small features, or incremental updates where each PR represents an atomic change.
+
+**Merge Strategy**: Always use **Squash and Merge**
+- Combines all commits into a single, atomic unit
+- Maintains a clean, linear history on the target branch
+- Preserves PR number in commit message for traceability
+- Example: `PR #123 -> [squash] -> main`
+
+#### 2. Release-Dev Merges
+When merging a complete `release-dev` branch into `main` after a major feature is completed.
+
+**Merge Strategy**: Always use **Merge Commit (no squash)**
+- Preserves the complete commit history of the feature branch
+- Creates a merge commit that serves as a feature completion marker
+- Simplifies cherry-picking specific changes for hotfixes
+- Example: `release-dev/feature -> [merge commit] -> main`
+
+
+### Best Practices
+
+- **Branch Hygiene**: Delete branches after merging to avoid clutter
+- **CI Validation**: Never merge code that fails CI checks
+- **Documentation**: Update documentation alongside code changes
+- **Breaking Changes**: Clearly document any breaking changes in PR descriptions
+
+
+
+## Release Management
+
+
+### Environment Networks
+
+Before diving into maintenance details, it's important to understand our environment network structure.
 
 | Environment | Environment Network | Definition and Practices |
 |-------------|---------------------|--------------------------|
-| mainnet | mainnet-ethereum | - mainnet production env on Ethereum<br>- only one single mainnet in foreseeable future<br>- external facing |
-| testnet | testnet-holesky<br>testnet-sepolia<br>testnet-hoodi | - testnet envs on Ethereum testnets like holesky or sepolia<br>- can be 1 or more testnet env networks for EigenLayer<br>- external facing, long lived for both internal and external developers to test on |
-| preprod | preprod-holesky<br>preprod-sepolia<br>preprod-hoodi | - preprod envs on Ethereum testnets like holesky or sepolia<br>- can be 1 or more preprod env networks for EigenLayer<br>- long lived for internal development and testing, before moving to testnets |
+| **mainnet** | mainnet-ethereum | - Production environment on Ethereum mainnet<br>- Single canonical mainnet instance in foreseeable future<br>- External facing with highest security requirements |
+| **testnet** | testnet-holesky<br>testnet-sepolia<br>testnet-hoodi | - Test environments on Ethereum testnets (holesky, sepolia, etc.)<br>- Multiple testnet environments may exist simultaneously<br>- External facing, long-lived environments for both internal and external developers<br>- Used for integration testing, AVS onboarding, and community engagement |
+| **preprod** | preprod-holesky<br>preprod-sepolia<br>preprod-hoodi | - Pre-production environments on Ethereum testnets<br>- Multiple preprod environments may exist simultaneously<br>- Internal facing for development team use<br>- Used for feature verification before promoting to testnet<br>- Provides safe environment for testing breaking changes |
+
+For more details on our testing infrastructure strategy, see our blog post: [The Future of EigenLayer Testing: New & Improved Testnets & Tooling Coming Soon](https://www.blog.eigenlayer.xyz/the-future-of-eigenlayer-testing-new-and-improved-testnets-tooling-coming-soon/)
+
+
+### Release Tags and Semver
+
+This section defines release tag semver where `<semver>` = `<major>.<minor>.<patch>`
+
+E.g.
+- v1.1.0
+- v1.1.1
+
+`<semver>` should follow [best practices of semver definition](https://semver.org/)
+
+
+In observing semver, we define the following;
+
+1. A **major** upgrade
+    1. An ideologically breaking upgrade, which may require re-learning parts of the system. 
+    2. “Upgrade Required” — Major upgrades are **assumed to break** all tooling, integrations, and downstream consumers unless otherwise stated.
+    3. Major upgrades are announced ahead of time via blog posts, the ELIP process, and onchain via the Eigen timelock.
+    4. example: slashing, which changes most of our codebase
+2. A **minor** upgrade
+    1. A conceptually breaking upgrade, which require adapting break changes and APIs, upgrading tools, CLIs, or integrations with EigenLayer.
+    2. “Upgrade carefully” — Minor upgrades are **assumed not to break** any tooling, CLIs, or interactions with EigenLayer unless otherwise stated in release notes.
+3. A **patch** upgrade
+    1. A “silent” upgrade, which addresses a bug or inconsistency in the system, but does not change any interfaces.
+        1. i.e. fixes reveal the need for a rounding fix in a contract. Notably **abi** and **tooling** remains stable across patches.
+    2. “Upgrade suggested” — Patch upgrades may come with upgrades to tooling, CLIs, or other Eigen infrastructure. You are encouraged to read the patch notes and upgrade.
+
+
+### Release Base and Cherry Pick
+
+All releases will be cut from `main` the base branch, to ensure global consistency, clean legibility, and high code quality with no env-network specific code
+
+Each env_network deployment will pick one release above.
+
+
+During release, release manager can cherry pick to exclude certain commits, especially in case of a non-sequential feature deployment
+
+
+#### Non Sequential Feature Release
+
+In smart contracts, there is a common scenario for non sequential feature deployment due to Ethereum hardforks, whose time we cannot control.
+
+E.g. say we have features A, B, and Ethereum hardfork X,  where B is an upgrade to adapt to X. The deployment sequence for testnet is A, B, X. While that of mainnet has to be B, X, A, because 1/ we don’t control the time of X, 2/ B must happen before X, and 3/ A is not ready for mainnet yet which can’t be deployed before B
+
+Concrete example just happened is A = slashing V1, B = prooftra, X = pectra
+
+So our release branch and model must be flexible enough to solve it via cherry picking
+
+
+### Release and Deployment Scope
+
+- a release would upgrade versions for all code, even though there's no change to a file from last release
+- a deployment would adopt a release as a whole, so all contracts would upgrade to that release, instead of "some file on release v1.3 and some on release v1.4"
+    - there's no phrasing of, for example, "all non-EigenPod contracts be on `v1.3.0` and  EigenPods contract on `v1.4.0`" - they would all be on v1.4.0, just the non-eigenpod contract are not changed at all from v1.3.0
+    - as implementation detail, we can skip upgrade the contracts that have not changed, but the codebase and releases are on new release version
+- all changes have to be deployed to testnet first, before mainnet. No skipping from preprod to mainnet directly
+
 
-See blog [The Future of EigenLayer Testing: New & Improved Testnets & Tooling Coming Soon](https://www.blog.eigenlayer.xyz/the-future-of-eigenlayer-testing-new-and-improved-testnets-tooling-coming-soon/) for detailed motivation
+### Release Branches
 
+Once release are created, corresponding release branches will live forever for future reference and NEVER be deleted. 
 
+Together with “**Fork-based PR**” defined in CONTRIBUTING.md, it means there are only 3 categories of branches in the repository:
 
-##  Development
+1. `main`: live forever, never delete 
+2. release-dev branches `release-dev/*`e: long living branch for a period of time but will be deleted once their commits are merged back to `main`
+3. release branches `vx.y.z`: live forever, never delete after a release is officially create
 
-### Code Review
 
-Code reviewers take responsibility of making sure PRs are rigorously labeled, well formatted, well tested before merging them. 
+### Changelog, Release Note, Release Announcement Blog
 
-For first-time external contributors, a maintainer should review their PR carefully and then manually approve CI to run.
+Each release in testnet and mainnet must have corresponding release note, changelog, and announcement blog
 
+- Changelog
+    - lives in the repo `/CHANGELOG` dir
+    - exact commit history diff from last release
+    - See examples in Kubernetes https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG
+- Release note:
+    - lives in the repo as part of release description under “[Releases](https://github.com/Layr-Labs/eigenlayer-contracts/releases)”
+    - high level summary of key changes users need to be aware of
+    - See template below
+    - Owner: release manager and PM
+- Release announcement blog
+    - published to [blog.eigenlayer.xyz](http://blog.eigenlayer.xyz)
+    - an announcement with polished language to introduce the release to users. It will include content in both release notes and changelog
+    - See examples in Golang https://tip.golang.org/doc/go1.24
+    - Owner: release manager and PM
 
-### Branching Model
+Release Note must follow template:
 
-#### Development Branches
+```jsx
+<release-version>
 
-- `main (default)`: 
-    - The primary development and canonical branch where all new code are merged
-    - It should remain stable enough for internal and external testing, ensuring continuous integration (CI) runs smoothly on every commit
-    - Security audit will always happen on main by a given commit
-- `release-dev/xxx`:
-    - For large features with many commits that will be breaking changes and we want to all-or-none in `main`, maintainer should create a `release-dev/xxx` branch to develop on
-        - eg. release-dev/slashing-v1, release-dev/rewards-v2
-    - These branches branch off from `main` for development, should be constantly rebased to `main` to be compatible with canonical branch, and will be merged back into `main` when the development is done and ready to deploy
-    - release-dev branches should be merged into main for auditing, no auditing will be on release-dev branches
+🚀 New Features – Highlight major new functionality.
+	- ...
+	- ...
 
+⛔ Breaking Changes – Call out backward-incompatible changes.
+	- ...
+	- ...
 
-### Merge PRs
+📌 Deprecations – Mention features that are being phased out.
+	- ...
+	- ...
 
-We classify PRs into two types:
+🛠️ Security Fixes – Specify patched vulnerabilities.
+	- ...
+	- ...
 
-1. **Regular PRs** – Used for bug fixes, small changes, or incremental updates. Each PR is atomic itself.
-2. **Release-dev Merges** – Merging a release-dev branch into `main` after completing a major release. The release-dev branch and such a merge contains many regular PRs
+🔧 Improvements – Enhancements to existing features.
+	- ...
+	- ...
 
-We use different merge rules for each PR type.
+🐛 Bug Fixes – List resolved issues.
+	- ...
+	- ...
+```
 
-- For regular PRs:  
-    - Always **rebase, squash, and merge**. Since multiple commits in a PR represent a single unit of work, merging them separately can be unnecessary or even break the code. Squashing ensures atomicity and keeps the commit history clean.
-    - demonstration: PR -> squash -> `main` or `release-dev`
+When writing **release notes** and **changelog entries**, follow these best practices to ensure clarity, usefulness, and professionalism.
 
+**Required Best Practices**
 
-- For release-dev merges:
-    - Always **rebase and create a new commit to merge, never squash**
-    - All commit history should be preserved - a release-dev branch contains multiple independent units of work, and squashing would create a massive, monolithic commit that erases history, making it difficult to track and understand individual changes.
-    - The additional commit will act as a marker in `main` as end of a release-dev merge. With each release-dev merge having its own end marker, it will significantly simplify the cherry-picking process in release creation
-    - demonstration: `release-dev` -> merge -> `main`
+1. **Be Clear and Concise** – Use simple, direct language. Avoid jargon unless necessary.
+2. **Categorize Changes** – Use standard sectioned template above.
+3. **Provide Context** – Briefly explain why the change was made, not just what changed.
+4. **Use a Consistent Format** – Follow a structured template for every release.
+5. **Include Relevant Links** – Link to issue trackers, PRs, or documentation for more details.
+6. **Highlight User Impact** – Mention how the update affects users and any required actions.
+7. **Keep It Versioned** – Use semantic versioning (e.g., `v2.3.1`) and list versions in descending order.
+8. **Avoid Internal-Only Details** – Don't include internal project discussions that aren’t relevant to users.
+9. **Maintain a Single Source of Truth** – Store changelogs in a `CHANGELOG.md` file or an equivalent documentation system.
 
 
+## Release Manager Responsibilities
 
-## Releases
+We introduce a new critical role called release manager (RM)
 
+Each release should have a primary RM and a secondary RM. RM is on rotating base, not necessarily always be the tech lead of the project, so we can amortize the cost and onboard all developers with same standards
 
+Primary Release Manager is responsible for
 
-## Release Manager
+- Managing the release-dev branch during development phase
+    - create it from `main` and eventually merge back to `main`
+    - constantly rebasing to `main` to resolve conflict early, without required efforts from other developers working on the release-dev branch
+    - holding high bar for quality:
+        - code quality
+        - PR templating
+        - test quality (e.g. no flaky or failed tests, keep test runtime under control)
+- Manage release branches and deployment
+    - create and manage corresponding release branches `Vx.y.z`
+    - author and publish release notes and changelog
+    - author and publish release blog with PM and marketing team
+    - lead communication with marketing and BD team to inform stakerholders for preparation
+    - lead deployment to env network
+- Mentoring and knowledge sharing
+    - mentor and train secondary release manager, who will be the primary manager for next release
 
 
+By following these maintenance procedures, we ensure a stable, secure, and well-documented codebase that meets the high standards required for EigenLayer's infrastructure.