How to run Nitro Testnode in Production?

[air3ijai]

Hello,

We are checking how to run Nitro Testnode for internal use and we have just two guides for that

1. Docs - How to run a local full chain simulation
2. Repository - README.md

However, in the docs, we have a warning

It would be good to provide more information, or even simplify, how to run Nitro Testnode in Production or in a secure way by replacing all the addresses

Make it more ready for Production

• List of addresses and private keys to be replaced
• How to manage node?
• L2 traffic
• Successive run is broken with a non-default folder name
• Init script help is very messy
• Docker compose
• Expose to the Internet
• Hardware for Nitro Testnode

List of addresses and private keys to be replaced

1. Dev account (prefunded with ETH in all networks)

   • docker-compose.yaml - geth.unlock and geth.miner.etherbase
   • test-node.bash - devprivkey
   • scripts/config.ts - gethConfig
   • scripts/index.ts - l2owner

2. L1 genesis addresses

   • scripts/config.ts - gethConfig

3. Rest of the addresses (Sequencer, Validator, L2 rollup owner, L3 rollup owner, L3 sequencer)

   • scripts/consts.ts - l1mnemonic (l1passphrase is hard-coded in other place and does not affect security)

4. Name based user_ accounts

   • scripts/accounts.ts - (name) --> (name + "<random string>")

5. Geth jwt secrets - scripts/config.ts - jwt and val_jwt

6. Batch poster

   • scripts/config.ts - baseConfig.batch-poster.redis-signer.signing-key

[air3ijai]

How to manage node?

I also have a question how to manage that Nitro Testnode instance?

1. How to start it - it is more or lees clear - /test-node.bash --init --blockscout --detach
2. How to stop it - docker compose down?
3. How to start it back again - /test-node.bash --blockscout --detach?

These are basic questions, but probably they are important, because we have to know how to manage our instance and perform an upgrade or reboot.

First run

Command /test-node.bash --init --blockscout --detach works well and run "something" automatically and we can see blocks in explorer.

Stop

Using a command docker compose down stop all the services, but is it a proper way?

Start it back

We assume that for a subsequent run we have to use /test-node.bash --blockscout --detach but after that command we have just 4 services running and Arbitrum blocks are stuck and does not increase - check next comments for explanation

blockscout
nitro-testnode-sequencer-1
postgres
nitro-testnode-geth-1

[air3ijai]

Some information from the docs, which might explain lack of the chain activity - Arbitrum block numbers

But, we have a CLI arg --no-l2-traffic which is not used by default and "someone" should produce L2 activity and blocks should be created

nitro-testnode/test-node.bash

Line 321 in 34f0e1c

echo --no-l2-traffic disables L2 spam transaction traffic \(default: enabled\)

[air3ijai]

L2 traffic

L2 traffic which is required for the network to work independently is generated only inside a if $force_init; then loop and it will be run at the initial init, but not on the subsequent runs

nitro-testnode/test-node.bash

Lines 611 to 615 in 947cb19

	if $l2_traffic; then
	echo == create l2 traffic
	run_script send-l2 --ethamount 100 --to user_traffic_generator --wait
	run_script send-l2 --ethamount 0.0001 --from user_traffic_generator --to user_traffic_generator --wait --delay 500 --times 1000000 > /dev/null &
	fi

Run it manually

In case we have to generate it we should consider to run it manually all the time

# Bridge funds from L1 to L2
./test-node.bash script bridge-funds --from funnel --ethamount 1000000000

# Send large amount of ETH to user_traffic_generator
./test-node.bash script send-l2 --ethamount 1000000000 --to user_traffic_generator --wait

# Start traffic generator in a background
nohup bash -c "while true; do
  ./test-node.bash script send-l2 --ethamount 0.0001 --from user_traffic_generator --to user_traffic_generator --wait --delay 500 --times 10000000 > /dev/null
done" &

# Stop traffic generator
pkill -f user_traffic_generator
docker kill `docker ps --filter "name=nitro-testnode-scripts-run" -q`

[air3ijai]

We can use cast from a Foundry toolchain to generate a mnemonic and derived key pairs to update keys related data

New random mnemonic with 40 accounts

cast wallet new-mnemonic -a 40

Generating mnemonic from provided entropy...
Successfully generated a new mnemonic.
Phrase:
museum subject bid royal enrich bargain copy student heart rent grace street

Accounts:
- Account 0:
Address:     0x49E64E36c7C87b8Ba24513719925e55d9e237f5a
Private key: 0xf719473116813fd09fe31d93856773c1458f861c1cd49a2c75e6bed1ec9d23e2

- Account 1:
Address:     0x2a24423b30F97d46183d38c04909745c7E452265
Private key: 0xe9abb8051dd624298d68e2c98d76b22f5a6cb448a353722cd9c762da1707d8a7

...

[air3ijai]

Successive run is broken with a non-default folder name

It was observed, that when we clone a repository in a non-default directory, init script will not detect existing volumes and on a successive runs will ⚠️ clear all chain data and redeploy! ⚠️

# Clone
git clone -b release --recurse-submodules https://github.com/OffchainLabs/nitro-testnode.git arbitrum-nitro-testnode
cd arbitrum-nitro-testnode

# First run
./test-node.bash --init --tokenbridge --blockscout --detach

# Stop
docker compose down --remove-orphans

# Successive run
./test-node.bash --blockscout --detach

That happens, because a hard-coded folder name, which is a part of the Docker Compose volume name

nitro-testnode/test-node.bash

Line 36 in 947cb19

num_volumes=`docker volume ls --filter label=com.docker.compose.project=nitro-testnode -q | wc -l`

And a fix might be the following

+ num_volumes=`docker volume ls --filter label=com.docker.compose.project=$(basename $PWD) -q | wc -l`  
- num_volumes=`docker volume ls --filter label=com.docker.compose.project=nitro-testnode -q | wc -l` 

[air3ijai]

Init script help is very messy

A --help in test-node.bash is not so easy to read and that can be improved. We also should consider to exclude volumes enumeration when help is invoked, because it does not require Docker and it might not be installed or run.

Also, it would be good to check capitalization and punctuation, just to have some consistency.

test-node.bash.patch

diff --git a/test-node.bash b/test-node.bash
index e3221b5..82bdb85 100755
--- a/test-node.bash
+++ b/test-node.bash
@@ -297,43 +297,43 @@ while [[ $# -gt 0 ]]; do
             shift
             ;;
         *)
-            echo Usage: $0 \[OPTIONS..]
-            echo        $0 script [SCRIPT-ARGS]
+            echo "Usage: $0 [OPTIONS..]"
+            echo "       $0 script [SCRIPT-ARGS]"
             echo
-            echo OPTIONS:
-            echo --build           rebuild docker images
-            echo --no-build        don\'t rebuild docker images
-            echo --dev             build nitro and blockscout dockers from source instead of pulling them. Disables simple mode
-            echo --dev-contracts   build scripts with local development version of contracts
-            echo --init            remove all data, rebuild, deploy new rollup
-            echo --pos             l1 is a proof-of-stake chain \(using prysm for consensus\)
-            echo --validate        heavy computation, validating all blocks in WASM
-            echo --l3node          deploys an L3 node on top of the L2
-            echo --l3-fee-token    L3 chain is set up to use custom fee token. Only valid if also '--l3node' is provided
-            echo --l3-fee-token-decimals Number of decimals to use for custom fee token. Only valid if also '--l3-fee-token' is provided
-            echo --l3-token-bridge Deploy L2-L3 token bridge. Only valid if also '--l3node' is provided
-            echo --l2-anytrust     run the L2 as an AnyTrust chain
-            echo --l2-timeboost    run the L2 with Timeboost enabled, including auctioneer and bid validator
-            echo --batchposters    batch posters [0-3]
-            echo --redundantsequencers redundant sequencers [0-3]
-            echo --detach          detach from nodes after running them
-            echo --blockscout      build or launch blockscout
-            echo --simple          run a simple configuration. one node as sequencer/batch-poster/staker \(default unless using --dev\)
-            echo --tokenbridge     deploy L1-L2 token bridge.
-            echo --no-tokenbridge  don\'t build or launch tokenbridge
-            echo --no-run          does not launch nodes \(useful with build or init\)
-            echo --no-l2-traffic   disables L2 spam transaction traffic \(default: enabled\)
-            echo --no-l3-traffic   disables L3 spam transaction traffic \(default: enabled\)
-            echo --no-simple       run a full configuration with separate sequencer/batch-poster/validator/relayer
-            echo --build-dev-nitro     rebuild dev nitro docker image
-            echo --no-build-dev-nitro  don\'t rebuild dev nitro docker image
-            echo --build-dev-blockscout     rebuild dev blockscout docker image
-            echo --no-build-dev-blockscout  don\'t rebuild dev blockscout docker image
-            echo --build-utils         rebuild scripts, rollupcreator, token bridge docker images
-            echo --no-build-utils      don\'t rebuild scripts, rollupcreator, token bridge docker images
-            echo --force-build-utils   force rebuilding utils, useful if NITRO_CONTRACTS_ or TOKEN_BRIDGE_BRANCH changes
+            echo "OPTIONS:"
+            echo "  --build                    rebuild docker images"
+            echo "  --no-build                 don't rebuild docker images"
+            echo "  --dev                      build nitro and blockscout dockers from source instead of pulling them. Disables simple mode"
+            echo "  --dev-contracts            build scripts with local development version of contracts"
+            echo "  --init                     remove all data, rebuild, deploy new rollup"
+            echo "  --pos                      l1 is a proof-of-stake chain (using prysm for consensus)"
+            echo "  --validate                 heavy computation, validating all blocks in WASM"
+            echo "  --l3node                   deploys an L3 node on top of the L2"
+            echo "  --l3-fee-token             L3 chain is set up to use custom fee token. Only valid if also '--l3node' is provided"
+            echo "  --l3-fee-token-decimals    Number of decimals to use for custom fee token. Only valid if also '--l3-fee-token' is provided"
+            echo "  --l3-token-bridge          Deploy L2-L3 token bridge. Only valid if also '--l3node' is provided"
+            echo "  --l2-anytrust              run the L2 as an AnyTrust chain"
+            echo "  --l2-timeboost             run the L2 with Timeboost enabled, including auctioneer and bid validator"
+            echo "  --batchposters             batch posters [0-3]"
+            echo "  --redundantsequencers      redundant sequencers [0-3]"
+            echo "  --detach                   detach from nodes after running them"
+            echo "  --blockscout               build or launch blockscout"
+            echo "  --simple                   run a simple configuration. one node as sequencer/batch-poster/staker (default unless using --dev)"
+            echo "  --tokenbridge              deploy L1-L2 token bridge"
+            echo "  --no-tokenbridge           don't build or launch tokenbridge"
+            echo "  --no-run                   does not launch nodes (useful with build or init)"
+            echo "  --no-l2-traffic            disables L2 spam transaction traffic (default: enabled)"
+            echo "  --no-l3-traffic            disables L3 spam transaction traffic (default: enabled)"
+            echo "  --no-simple                run a full configuration with separate sequencer/batch-poster/validator/relayer"
+            echo "  --build-dev-nitro          rebuild dev nitro docker image"
+            echo "  --no-build-dev-nitro       don't rebuild dev nitro docker image"
+            echo "  --build-dev-blockscout     rebuild dev blockscout docker image"
+            echo "  --no-build-dev-blockscout  don't rebuild dev blockscout docker image"
+            echo "  --build-utils              rebuild scripts, rollupcreator, token bridge docker images"
+            echo "  --no-build-utils           don't rebuild scripts, rollupcreator, token bridge docker images"
+            echo "  --force-build-utils        force rebuilding utils, useful if NITRO_CONTRACTS_ or TOKEN_BRIDGE_BRANCH changes"
             echo
-            echo script runs inside a separate docker. For SCRIPT-ARGS, run $0 script --help
+            echo "  script runs inside a separate docker. For SCRIPT-ARGS, run $0 script --help"
             exit 0
     esac
 done

Before

After

[air3ijai]

Docker compose stop

It is a widely known that, at this point, Arbitrum DB is the most fragile part and can be broken by a simple Docker container stop - Unlcean shutdown corrupts the database entirely #3334.

In a public network we have a redundancy and data is stored by other participants and we can start a new local node. In case of a private network, DB corruption will lead to the whole chain data lost.

In the documentation How to run a full node for an Arbitrum or Arbitrum chain there is a mentioning of a workaround

When shutting down the Docker image, it is important to allow a graceful shutdown to save the current state to disk. Here is an example of how to do a graceful shutdown of all Docker images currently running

docker stop --time=1800 $(docker ps -aq)

And for Docker compose we can use stop_grace_period: 1800s, however it is not implemented in the provided docker-compose.yml file and we have to add it to every Arbitrum Nitro Service

  sequencer:
    pid: host # allow debugging
    image: nitro-node-dev-testnode
    ...
    stop_grace_period: 1800s
    depends_on:
      - geth

Note: Check Hardware for Nitro Testnode fro a fix for postgres service.

Docker compose logs

Based on the Configure logging drivers

Use the local logging driver to prevent disk-exhaustion. By default, no log-rotation is performed. As a result, log-files stored by the default json-file logging driver logging driver can cause a significant amount of disk space to be used for containers that generate much output, which can lead to disk space exhaustion.

We may not know who is produces a lot of logs and have to adjust all services to use a log-driver and adjust a rotation if required

  sequencer:
    pid: host # allow debugging
    image: nitro-node-dev-testnode
    ...
    stop_grace_period: 1800s
    # logging:
    #   driver: local
    logging:
      driver: json-file
      options:
        max-size: 10m
        max-file: 10
    depends_on:
      - geth

Simplify

We also could use a Extensions to simplify compose file adjustments

x-common: &common
  stop_grace_period: 1800s
  logging:
    driver: local

services:
  sequencer:
    <<: *common
    pid: host # allow debugging
    image: nitro-node-dev-testnode
    ...
    stop_grace_period: 1800s
    depends_on:
      - geth

# Override stop_grace_period for some services
  blockscout:
    <<: *common
    stop_grace_period: 10s

  postgres:
    <<: *common
    stop_grace_period: 30s

  scripts:
    <<: *common
    stop_grace_period: 10s

[air3ijai]

Expose to the Internet

One of the way to make node publicly available would be to using Nginx as a reverse proxy and exporting via HTTPS

• Arbitrum Nitro RPC: rpc.domain.tld
• Ethereum RPC: rpc-ethereum.domain.tld
• Explorer: explorer.domain.tld

---

1. Install Nginx and Certbot

   apt update
   apt install -y nginx certbot python3-certbot-nginx

2. Generate self-signed certificate

   openssl \
     req -x509 \
     -newkey rsa:4096 \
     -out /etc/nginx/self.crt \
     -sha256 -days 3650 -nodes \
     -keyout /etc/nginx/self.key \
     -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=rpc.domain.tld" \
     -addext "subjectAltName=DNS:rpc-ethereum.domain.tld,DNS:explorer.domain.tld"

3. Copy Nginx config and temporary point ssl_certificate_key and ssl_certificate to the self-signed files

   nginx.conf

   user                www-data;
   worker_processes    auto;
   
   pid          /run/nginx.pid;
   error_log    /var/log/nginx/error.log   notice;
   
   events {
       worker_connections    1024;
       accept_mutex          off;
       use                   epoll;
   }
   
   http {
       include         /etc/nginx/mime.types;
       default_type    application/octet-stream;
   
       map "$time_local:$msec" $time_local_ms { ~(^\S+)(\s+\S+):\d+\.(\d+)$ $1.$3$2; }
   
       map $http_upgrade $connection_upgrade {
       default    upgrade;
       ''         close;
       }
   
       log_format main escape=json
           '$remote_addr - $request_time - [$time_local_ms] - $msec - '
           '$request_id - $connection-$connection_requests - '
           '$scheme - $host - $server_port - '
           '$request_method - $request_uri - $server_protocol - $status - $request_completion - '
           '$bytes_sent - $request_length - "$http_referer" - "$http_user_agent" - '
           '$proxy_host - "$upstream_addr" - "$upstream_status" - "$upstream_connect_time" - "$upstream_response_time" - \'$request_body\' ';
   
       access_log    /var/log/nginx/access.log  main;
   
       sendfile             on;
       keepalive_timeout    65;
   
       ssl_protocols                TLSv1.2 TLSv1.3;
       ssl_ciphers                  HIGH:!aNULL:!MD5;
       ssl_prefer_server_ciphers    on;
   
       server_names_hash_bucket_size    512;
   
       upstream ethereum-http {
           server    localhost:8545  fail_timeout=0;
       }
   
       upstream arbitrum-http {
           server    localhost:8547  fail_timeout=0;
       }
   
       upstream blockscout {
           server    localhost:4000  fail_timeout=0;
       }
   
       proxy_buffer_size    16k;
       proxy_buffers        16  8k;
   
       send_timeout             600s;
       proxy_connect_timeout    60s;
       proxy_read_timeout       600s;
       proxy_send_timeout       600s;
   
       server {
           listen    80   default_server;
           return    301  https://$host$request_uri;
       }
   
       server {
           listen    443  ssl  http2  default_server;
           return    444;
   
           ssl_protocols                TLSv1.2 TLSv1.3;
           ssl_ciphers                  HIGH:!aNULL:!MD5;
           ssl_prefer_server_ciphers    on;
   
           ssl_certificate_key    /etc/nginx/self.key;
           ssl_certificate        /etc/nginx/self.crt;
       }
   
       server {
           listen         127.0.0.1:80;
           server_name    stub-status;
   
           stub_status;
       }
   
       server {
           listen         443  ssl  http2;
           server_name    rpc-ethereum.domain.tld;
   
           # ssl_certificate_key    /etc/letsencrypt/live/rpc.domain.tld/privkey.pem;   # managed by Certbot
           # ssl_certificate        /etc/letsencrypt/live/rpc.domain.tld/fullchain.pem; # managed by Certbot
           ssl_certificate_key    /etc/nginx/self.key;
           ssl_certificate        /etc/nginx/self.crt;
   
           access_log    /var/log/nginx/ethereum-access.log  main;
           error_log     /var/log/nginx/ethereum-error.log   notice;
   
           location / {
               proxy_set_header    X-Forwarded-For      $proxy_add_x_forwarded_for;
               proxy_set_header    X-Forwarded-Proto    $scheme;
               proxy_set_header    X-Forwarded-Host     $server_name;
               proxy_set_header    X-Real-IP            $remote_addr;
               proxy_set_header    Host                 $host;
               proxy_redirect      off;
               proxy_pass          http://ethereum-http;
           }
       }
   
       server {
           listen         443  ssl  http2;
           server_name    rpc.domain.tld;
   
           # ssl_certificate_key    /etc/letsencrypt/live/rpc.domain.tld/privkey.pem;   # managed by Certbot
           # ssl_certificate        /etc/letsencrypt/live/rpc.domain.tld/fullchain.pem; # managed by Certbot
           ssl_certificate_key    /etc/nginx/self.key;
           ssl_certificate        /etc/nginx/self.crt;
   
   
           access_log    /var/log/nginx/arbitrum-access.log  main;
           error_log     /var/log/nginx/arbitrum-error.log   notice;
   
           location / {
               proxy_set_header    X-Real-IP            $remote_addr;
               proxy_set_header    X-Forwarded-For      $proxy_add_x_forwarded_for;
               proxy_set_header    X-Forwarded-Proto    $scheme;
               proxy_set_header    X-Forwarded-Host     $server_name;
               proxy_set_header    Host                 $host;
               proxy_redirect      off;
               proxy_pass          http://arbitrum-http;
           }
       }
   
       server {
           listen         443  ssl  http2;
           server_name    explorer.domain.tld;
   
           # ssl_certificate_key    /etc/letsencrypt/live/rpc.domain.tld/privkey.pem;   # managed by Certbot
           # ssl_certificate        /etc/letsencrypt/live/rpc.domain.tld/fullchain.pem; # managed by Certbot
           ssl_certificate_key    /etc/nginx/self.key;
           ssl_certificate        /etc/nginx/self.crt;
   
   
           access_log    /var/log/nginx/explorer-access.log  main;
           error_log     /var/log/nginx/explorer-error.log   notice;
   
           location / {
               proxy_set_header      X-Real-IP            $remote_addr;
               proxy_set_header      X-Forwarded-For      $proxy_add_x_forwarded_for;
               proxy_set_header      X-Forwarded-Proto    $scheme;
               proxy_set_header      X-Forwarded-Host     $server_name;
               proxy_set_header      Host                 $host;
               proxy_set_header      Upgrade              $http_upgrade;
               proxy_set_header      Connection           $connection_upgrade;
               proxy_redirect        off;
               proxy_http_version    1.1;
               proxy_cache_bypass    $http_upgrade;
   
               proxy_pass            http://blockscout;
           }
       }
   }

   # Backup
   cp /etc/nginx/nginx.conf{,.bak}
   
   # Copy
   cp nginx.conf /etc/nginx/nginx.conf
   
   # Edit
   ssl_certificate_key    /etc/nginx/self.key;
   ssl_certificate        /etc/nginx/self.crt;
   
   # Apply
   nginx config -t
   systemctl config reload

4. Run Certbot

   certbot --nginx -d rpc.domain.tld,rpc-ethereum.domain.tld,explorer.domain.tld

[air3ijai]

Hardware for Nitro Testnode

Some experiments were performed on Hetzner in Nuremberg region.

./test-node.bash --init --tokenbridge --blockscout --detach

Server	Deploy	~ Load	vCPU	RAM	Disk	Traffic	Price
CX23	14m38.892s	100% CPU / 2 GB	2	4 GB	40 GB	20 TB	3.49 $/m
CX33	13m5.248s	91% CPU / 2 GB	4	8 GB	80 GB	20 TB	5.99 $/m
CX43	8m37.557s	75% CPU / 2 GB	8	16 GB	160 GB	20 TB	9.99 $/m
CX53	8m45.586s	49% CPU / 2 GB	16	32 GB	320 GB	20 TB	18.99$/m
CAX11	13m15.521s	99% CPU / 2 GB	2	4 GB	40 GB	20 TB	3.99 $/m
CAX21	11m9.272s	82% CPU / 2 GB	4	8 GB	80 GB	20 TB	6.99 $/m
CAX31	10m48.732s	65% CPU / 2 GB	8	16 GB	160 GB	20 TB	13.49 $/m
CAX41	11m2.930s	37% CPU / 2 GB	16	32 GB	320 GB	20 TB	26.99 $/m
CPX22	9m44.132s	100% CPU / 2 GB	2	4 GB	80 GB	20 TB	6.99 $/m
CPX32	9m17.915s	90% CPU / 2 GB	4	8 GB	160 GB	20 TB	11.99 $/m
CPX42	9m15.247s	80% CPU / 2 GB	8	16 GB	320 GB	20 TB	21.99 $/m
CPX52	9m4.764s	65% CPU / 2 GB	12	24 GB	480 GB	20 TB	31.49 $/m
CPX62	9m6.836s	53% CPU / 2 GB	16	32 GB	640 GB	20 TB	42.99 $/m
CCX13	9m49.135s	100% CPU / 2 GB	2	8 GB	80 GB	20 TB	13.49 $/m
CCX23	9m3.777s	96% CPU / 2 GB	4	16 GB	160 GB	20 TB	26.49 $/m
CCX33	8m38.946s	82% CPU / 2 GB	8	32 GB	240 GB	20 TB	53.49 $/m

Disk space: OS + Data ~ 15 GB

Note: Disk and memory usage will increase over time.

Most of the load is from blockscout service. Also, during the run on CX53 it was observed that blockscout started to crash due to the connection limit to the postgres database and it can be fixed in the following way

  postgres:
    <<: *common
    stop_grace_period: 30s
    image: postgres:13.6
    restart: always
    container_name: 'postgres'
+   command: -c 'max_connections=1000'
    environment:
        POSTGRES_PASSWORD: ''
        POSTGRES_USER: 'postgres'
        POSTGRES_HOST_AUTH_METHOD: 'trust'
    volumes:
      - "postgres-data:/var/lib/postgresql/data"
    ports:
      - "127.0.0.1:7432:5432"