Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ yarn-error.log
dist
./*.js
.DS_Store
logfile.log
logfile.log
config.json
96 changes: 88 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,28 +2,108 @@

## Overview

This is a monorepo containing a collection of scripts designed for monitoring various aspects of Arbitrum chains. Each package focuses on one particular monitoring task, like monitoring retryables, batch posting, etc.
This monitoring suite helps you track the health and performance of your Arbitrum chains through three specialized monitors:

1. [**Retryable Monitor**](./packages/retryable-monitor/README.md) - Tracks ParentChain->ChildChain message execution and retryable ticket lifecycle
2. [**Batch Poster Monitor**](./packages/batch-poster-monitor/README.md) - Monitors batch posting and data availability
3. [**Assertion Monitor**](./packages/assertion-monitor/README.md) - Monitor assertion creation and validation on Arbitrum chains

Each monitor has its own detailed documentation with technical specifics and implementation details.

## Prerequisites

- Node.js v18 or greater
- Yarn package manager
- Access to Arbitrum chain RPC endpoints
- Access to parent chain RPC endpoints
- Slack workspace for alerts (optional)

## Installation

Ensure you are using Node v18 or greater.
1. Clone and install dependencies:

```bash
git clone https://github.com/OffchainLabs/arbitrum-monitoring.git
cd arbitrum-monitoring
yarn install
```

## Running Monitoring Scripts
## Configuration

### Chain Configuration

1. Copy and edit the config file:

```bash
cp config.example.json config.json
```

2. Configure your chains in `config.json`:

```json
{
"childChains": [
{
"name": "Your Chain Name",
"chainId": 421614,
"parentChainId": 11155111,
"confirmPeriodBlocks": 45818,
"parentRpcUrl": "https://your-parent-chain-rpc",
"orbitRpcUrl": "https://your-chain-rpc",
"ethBridge": {
"bridge": "0x...",
"inbox": "0x...",
"outbox": "0x...",
"rollup": "0x...",
"sequencerInbox": "0x..."
}
}
]
}
```

To run a monitoring tool, navigate to the relevant package folder inside `packages`, and run the desired scripts. Detailed instructions for each script can be found within their respective folders.
### Alert Configuration

### Example
1. Copy and configure the environment file:

```bash
yarn workspace retryable-monitor dev [--options]
cp .env.sample .env
```

This can also be accessed simply from the root-level in this repository by:
2. Set up Slack alerts in `.env` (optional):

```bash
yarn retryable-monitor [--options]
NODE_ENV=CI
RETRYABLE_MONITORING_SLACK_TOKEN=your-slack-token
RETRYABLE_MONITORING_SLACK_CHANNEL=your-slack-channel
BATCH_POSTER_MONITORING_SLACK_TOKEN=your-slack-token
BATCH_POSTER_MONITORING_SLACK_CHANNEL=your-slack-channel
ASSERTION_MONITORING_SLACK_TOKEN=your-slack-token
ASSERTION_MONITORING_SLACK_CHANNEL=your-slack-channel
```

## Usage

All monitors support these base options:

- `--configPath`: Path to configuration file (default: "config.json")
- `--enableAlerting`: Enable Slack alerts (default: false)

### Quick Start Commands

```bash
# Monitor retryable tickets
yarn retryable-monitor [options]

# Monitor batch posting
yarn batch-poster-monitor [options]

# Monitor chain assertions
yarn assertion-monitor [options]
```

See individual monitor READMEs for specific options and features:

- [Retryable Monitor Details](./packages/retryable-monitor/README.md)
- [Batch Poster Monitor Details](./packages/batch-poster-monitor/README.md)
- [Assertion Monitor Details](./packages/assertion-monitor/README.md)
55 changes: 55 additions & 0 deletions config.example.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
{
"childChains": [
{
"name": "Arbitrum Sepolia",
"chainId": 421614,
"parentChainId": 11155111,
"confirmPeriodBlocks": 45818,
"parentRpcUrl": "https://sepolia.drpc.org",
"orbitRpcUrl": "https://sepolia-rollup.arbitrum.io/rpc",
"explorerUrl": "https://sepolia.arbiscan.io",
"parentExplorerUrl": "https://sepolia.etherscan.io",
"ethBridge": {
"bridge": "0x38f918D0E9F1b721EDaA41302E399fa1B79333a9",
"inbox": "0xaAe29B0366299461418F5324a79Afc425BE5ae21",
"outbox": "0x65f07C7D521164a4d5DaC6eB8Fac8DA067A3B78F",
"rollup": "0x042b2e6c5e99d4c521bd49beed5e99651d9b0cf4",
"sequencerInbox": "0x6c97864CE4bE1C2C8bB6aFe3A115E6D7Dca82E71"
}
},
{
"name": "Xai Testnet",
"chainId": 37714555429,
"parentChainId": 421614,
"confirmPeriodBlocks": 150,
"parentRpcUrl": "https://sepolia-rollup.arbitrum.io/rpc",
"orbitRpcUrl": "https://testnet-v2.xai-chain.net/rpc",
"explorerUrl": "https://testnet-explorer-v2.xai-chain.net/",
"parentExplorerUrl": "https://sepolia.arbiscan.io/",
"ethBridge": {
"bridge": "0x6c7FAC4edC72E86B3388B48979eF37Ecca5027e6",
"inbox": "0x6396825803B720bc6A43c63caa1DcD7B31EB4dd0",
"outbox": "0xc7491a559b416540427f9f112C5c98b1412c5d51",
"rollup": "0xeedE9367Df91913ab149e828BDd6bE336df2c892",
"sequencerInbox": "0x529a2061A1973be80D315770bA9469F3Da40D938"
},
"nativeToken": "0x4e6f41acbfa8eb4a3b25e151834d9a14b49b69d2",
"tokenBridge": {
"parentCustomGateway": "0x04e14E04949D49ae9c551ca8Cc3192310Ce65D88",
"parentErc20Gateway": "0xCcB451C4Df22addCFe1447c58bC6b2f264Bb1256",
"parentGatewayRouter": "0x185b868DBBF41554465fcb99C6FAb9383E15f47A",
"parentMultiCall": "0xA115146782b7143fAdB3065D86eACB54c169d092",
"parentProxyAdmin": "0x022c515aEAb29aaFf82e86A10950cE14eA89C9c5",
"parentWeth": "0x0000000000000000000000000000000000000000",
"parentWethGateway": "0x0000000000000000000000000000000000000000",
"childCustomGateway": "0xea1ce1CC75C948488515A3058E10aa82da40cE8F",
"childErc20Gateway": "0xD840761a09609394FaFA3404bEEAb312059AC558",
"childGatewayRouter": "0x3B8ba769a43f34cdD67a20aF60d08D54C9C8f1AD",
"childMultiCall": "0x5CBd60Ae5Af80A42FA8b0F20ADF95A8879844984",
"childProxyAdmin": "0x7C1BA251d812fb34aF5C2566040C3C30585aFed9",
"childWeth": "0x0000000000000000000000000000000000000000",
"childWethGateway": "0x0000000000000000000000000000000000000000"
}
}
]
}
118 changes: 46 additions & 72 deletions packages/assertion-monitor/README.md
Original file line number Diff line number Diff line change
@@ -1,97 +1,71 @@
# Assertion Monitor

This tool is designed to monitor assertions on Arbitrum chains. It performs comprehensive monitoring of both BOLD (Bounded Liquidity Delay) and Classic rollup chains, with the following checks:
> For installation and general configuration, see the [main README](../../README.md).

## Chain Activity Monitoring
1. Tracks chain activity by monitoring block progression
2. Verifies the latest confirmed block number from the child chain
3. Detects periods of inactivity or stalled chain progress
## Overview

## Assertion Creation Monitoring
1. No assertions created in the last 4 hours when there is chain activity
2. Tracks creation events and validates their frequency
3. Alerts when chain activity exists without new assertions
The Assertion Monitor monitors the lifecycle of assertions in both BoLD (Bounded Liquidity Delay) and pre-BoLD rollup chains. [Learn more](https://docs.arbitrum.io/how-arbitrum-works/inside-arbitrum-nitro#arbitrum-rollup-protocol).

## Assertion Confirmation Monitoring
1. No assertions confirmed within the chain's confirmation period
2. Tracks unconfirmed assertions and their age
3. Monitors confirmation delays against the chain's configured period
4. Validates the time between creation and confirmation events
## Command-Line Interface

## Alert Types
The monitor generates alerts for the following conditions:

1. Creation Issues:
- No assertion creation events in the last 7 days
- Chain activity detected but no new assertions created in the last 4 hours

2. Confirmation Issues:
- Parent chain confirmation issues - assertions not confirming
- Unconfirmed assertions present
- Assertion age exceeds confirmation period
- Confirmation delay exceeds the configured period

3. Chain State Issues:
- Chain stalled or inactive
- Block synchronization issues
- Validator whitelist status changes

Read more about assertions [here](https://docs.arbitrum.io/how-arbitrum-works/inside-arbitrum-nitro#arbitrum-rollup-protocol).

## Prerequisites

Before using this tool, make sure you have the following installed:

- [Node.js](https://nodejs.org/en)
- [Yarn](https://classic.yarnpkg.com/lang/en/docs/install/#mac-stable)
```bash
yarn assertion-monitor [options]

Additionally, ensure that you have added your Arbitrum network configuration to the `config.json` file in the `lib` directory;
Monitor assertion creation and validation on Arbitrum chains

## Installation
Options:
--help Show help [boolean]
--version Show version number [boolean]
--configPath Path to config file [string] [default: "config.json"]
--enableAlerting Enable Slack alerts [boolean] [default: false]

From the root directory of the project, run the following command to install dependencies:
Examples:
yarn assertion-monitor Run with default config
yarn assertion-monitor --enableAlerting Enable Slack notifications
yarn assertion-monitor --configPath=custom.json Use custom config file

```bash
yarn install
Environment Variables:
ASSERTION_MONITORING_SLACK_TOKEN Slack API token for alerts
ASSERTION_MONITORING_SLACK_CHANNEL Slack channel for alerts
```

## Execution
## Monitor Details

### One-off Check
The Assertion Monitor tracks assertions through their lifecycle, implementing distinct strategies for BoLD and pre-BoLD rollup chains.

To find assertion events and display their status for a specific block range, execute the following command:

```bash
yarn dev [--configPath=<CONFIG_PATH>]
```
### Critical Events Monitored

- If `--configPath` is not provided, it defaults to `config.json`.
- This command will identify all assertion events (both creation and confirmation) from the parent chain to your Orbit chain within the specified block range.
The monitor tracks five categories of blockchain events:

### Error Generation and Reporting
- **Creation Events**: Records when new assertions and nodes are created on the chain to verify transaction execution
- **Confirmation Events**: Identifies when assertions are confirmed on the parent chain after challenge periods end
- **Validator Events**: Tracks validator participation metrics including stakes, challenges, and whitelist status
- **Block Events**: Monitors block production rates, finalization timing, and synchronization between chains
- **Chain State**: Analyzes the overall consistency between on-chain state and expected protocol behavior

To enable reporting, use `--enableAlerting` flag.
### Alert Scenarios

This will enable alerts for all the conditions listed above in the Alert Types section.
The monitor triggers alerts when these conditions are detected:

Additionally, you might also want to log these errors to Slack, for which you will need to configure, in the `.env` file:
#### Creation Issues

- `NODE_ENV=CI`
- `ASSERTION_MONITORING_SLACK_TOKEN=<your-slack-token>`
- `ASSERTION_MONITORING_SLACK_CHANNEL=<your-slack-channel-key>`
- No assertion creation events within configured time window
- Chain activity without corresponding recent assertions
- Extended node creation gaps on non-BoLD chains
- Validator participation below required thresholds

Check [Slack integration documentation](https://api.slack.com/quickstart) for more information about getting these auth tokens.
#### Confirmation Issues

## Chain Support
- Parent chain block threshold exceeded
- Assertions stuck in challenge period
- Data inconsistencies between confirmation events and confirmed blocks
- Confirmation events missing despite available creation events

The monitor automatically detects and adapts to different chain types:
#### Other Issues

1. BOLD (Bounded Liquidity Delay) Chains:
- Uses BOLD-specific assertion formats and validation
- Tracks BOLD-specific confirmation processes
- Monitors BOLD genesis assertion hash
- Validator whitelist disabled on pre-BoLD chains
- Base stake below 1 ETH threshold on BoLD chains
- Parent-child chain synchronization anomalies
- State inconsistencies between expected and observed chain state

2. Classic Rollup Chains:
- Uses Classic node creation and confirmation events
- Monitors Classic-specific block validation
- Tracks node confirmation processes
For implementation details and thresholds, see `alerts.ts` and `monitoring.ts`.
Loading