Migrate signalling server away from Heroku free tier

[ErnWong]

Consider either upgrade or move to another PaaS.
Going to move to railway.

Plan:

• Deploy signalling server instance Railway
• Update code to point to new signalling server instance
• Shutdown Heroku signalling server instance.

Places to update in the code:

• The RustConfig file (used by Heroku to know which crate we want to build)

  dango-tribute/RustConfig

  Line 1 in c0fd1ee

  RUST_CARGO_BUILD_FLAGS="--release --package signalling-server"

• Heroku endpoint URL hardcoded to the client now needs to be updated to the Railway endpoint

  dango-tribute/client/src/main.rs

  Line 149 in c0fd1ee

  let endpoint_url = format!("http://dango-daikazoku.herokuapp.com/join/{}", host_id);

• Heroku endpoint URL hardcoded to the server also needs to be updated to the Railway endpoint

  dango-tribute/server/src/main.rs

  Line 88 in 456307e

  let endpoint_url = "ws://dango-daikazoku.herokuapp.com/host".to_string();

To check:

• Cross origin policies
• SSL

[ErnWong]

[Edited: moved comment contents into issue description]

[ErnWong]

Railway's Nixpacks automatically-deduced build plan did not work:

==============
Using Nixpacks
==============
Nixpacks build failed
 
Error: Error reading crates/bevy_prototype_frameshader/src/Cargo.toml
 
Caused by:
No such file or directory (os error 2)

Going to look at setting up the build plan manually or help prod the Nixpack in the right direction

[ErnWong]

Why is Nixpacks trying to read a Cargo.toml in a src directory? Is it because Nixpacks is interpreting the wildcard differently for the Cargo workspace?

dango-tribute/Cargo.toml

Line 2 in c0fd1ee

members = ["crates/*", "client", "server", "shared", "signalling-server"]

[ErnWong]

Looks like the corresponding Nixpacks configuration for RustConfig's --package setting is the NIXPACKS_CARGO_WORKSPACE environment variable: https://nixpacks.com/docs/providers/rust#workspaces

[ErnWong]

Environment variable NIXPACKS_CARGO_WORKSPACE=signalling-server has been manually set in Railway.

[ErnWong]

Now we're further ahead in the deployment. The next error is:

 error: opening file '/app/.nixpacks/rust-toolchain': No such file or directory

Full logs:

==============
Using Nixpacks
==============
 
╔══════════════════════════════ Nixpacks v0.14.0 ══════════════════════════════╗
║ setup      │ binutils, gcc, (rust-bin.fromRustupToolchainFile ./rust-        ║
║            │ toolchain)                                                      ║
║──────────────────────────────────────────────────────────────────────────────║
║ build      │ mkdir -p bin                                                    ║
║            │ cargo build --release --package signalling-server               ║
║            │ cp target/release/signalling-server bin                         ║
║──────────────────────────────────────────────────────────────────────────────║
║ start      │ ./target/release/signalling-server                              ║
╚══════════════════════════════════════════════════════════════════════════════╝
 
 
#1 [internal] load build definition from Dockerfile
#1 sha256:e52de91af0a36b6086dbfb6b10dcd89c9962c2b33c6ba6508a13ba705d2dae49
#1 transferring dockerfile: 2.34kB done
 
#1 DONE 0.1s
 
#2 [internal] load .dockerignore
#2 sha256:9c5447916487526753159a1c919227b4851a6fbd092dbed641d905eb2bd766d0
#2 transferring context: 2B done
 
#2 DONE 0.1s
 
 
#3 [internal] load metadata for ghcr.io/railwayapp/nixpacks:debian-1667866002
#3 sha256:ff4546c29b1eb585dd7df95164933728a568cb8dc35398f9f51f11afd8b9a1cb
 
#3 DONE 0.2s
 
#4 [stage-0  1/10] FROM ghcr.io/railwayapp/nixpacks:debian-1667866002@sha256:96bc6d9c576900c9a99db8f7c8d6ef86c9e8dd9e54f46a119ff8499b9f5fe3b0
#4 sha256:a6aa9f4410ef5777e49afa17be8a1e260d09acb8a74de2dab233288bf3c56052
#4 DONE 0.0s
 
#5 [stage-0  2/10] WORKDIR /app/
#5 sha256:f1ff5906bd390875dc0cc6c2f06540658835071f9bb0611cd6ce76c903f39537
#5 CACHED
 
#6 [internal] load build context
#6 sha256:a73ec9408899d8af217c0d8d4817336bfad68eb49806caaa281d3068fe2bcd04
 
#6 transferring context: 19.60MB 0.2s done
 
#6 DONE 0.2s
 
#7 [stage-0 3/9] COPY .nixpacks/nixpkgs-a0b7e70db7a55088d3de0cc370a59f9fbcc906c3.nix .nixpacks/nixpkgs-a0b7e70db7a55088d3de0cc370a59f9fbcc906c3.nix
#7 sha256:40ef9b0a8bf94dcb42856d09167eabb4e7f550255081d5b473fae7c8257f55cb
 
#7 DONE 0.3s
 
#8 [stage-0 4/9] RUN nix-env -if .nixpacks/nixpkgs-a0b7e70db7a55088d3de0cc370a59f9fbcc906c3.nix && nix-collect-garbage -d
#8 sha256:47c6e2f8c88efbc361d03db9b11d2fe3a0279f3fed0751d9b239c2fac7f6029e
 
#8 27.72 installing 'a0b7e70db7a55088d3de0cc370a59f9fbcc906c3-env'
 
#8 28.13 error: opening file '/app/.nixpacks/rust-toolchain': No such file or directory
#8 28.13 (use '--show-trace' to show detailed location information)
 
#8 ERROR: executor failed running [/bin/bash -ol pipefail -c nix-env -if .nixpacks/nixpkgs-a0b7e70db7a55088d3de0cc370a59f9fbcc906c3.nix && nix-collect-garbage -d]: exit code: 1
-----
> [stage-0 4/9] RUN nix-env -if .nixpacks/nixpkgs-a0b7e70db7a55088d3de0cc370a59f9fbcc906c3.nix && nix-collect-garbage -d:
-----
executor failed running [/bin/bash -ol pipefail -c nix-env -if .nixpacks/nixpkgs-a0b7e70db7a55088d3de0cc370a59f9fbcc906c3.nix && nix-collect-garbage -d]: exit code: 1
 
Error: Docker build failed

[ErnWong]

I speculate the error is coming from the rust nix overlay due to the fromRustupToolchainFile option, but either it's looking in the wrong place or the toolchain file didn't get copied across to the right place. Could be wrong. What is the .nixpacks directory?

[ErnWong]

.nixpacks comes from here: https://github.com/railwayapp/nixpacks/blob/1a37a464e187781d5670d8b0abf9d3a823574663/src/nixpacks/builder/docker/dockerfile_generation.rs#L23

[ErnWong]

Tried installing railway cli to my NixOS but my nixpkgs might be too old. Trying to install it with the -I argument specifying the latest nixpkgs didn't work either.

Tried installing nixpacks through nix flakes, but failing with:

error: builder for '/nix/store/1zfqc00vcbw5v7xnf2vqyngmqyxnafm2-nixpacks-v0.3.0.drv' failed with exit code 1;
       last 10 log lines:
       > error[E0670]: `async fn` is not permitted in Rust 2015
       >  --> /build/dp51xbncj6pxg9fnbqgjhjrzp5f6hghs-source/src/nixpacks/builder/mod.rs:9:5
       >   |
       > 9 |     async fn create_image(
       >   |     ^^^^^ to use `async fn`, switch to Rust 2018 or later
       >   |
       >   = help: set `edition = "2021"` in `Cargo.toml`
       >   = note: for more on editions, read https://doc.rust-lang.org/edition-guide
       >
       > Error writing files: failed to resolve mod `builder`: cannot parse /build/dp51xbncj6pxg9fnbqgjhjrzp5f6hghs-source/src/nixpacks/builder/mod.rs
       For full logs, run 'nix log /nix/store/1zfqc00vcbw5v7xnf2vqyngmqyxnafm2-nixpacks-v0.3.0.drv'.
error: 1 dependencies of derivation '/nix/store/bx4rxg3j9rh0vvc4d03xxvm7dyp2hkwx-nix-shell-env.drv' failed to build

I suspect it's because in the check phase, it's using rustfmt outside of cargo, and therefore isn't aware of the edition set by Cargo.toml. I worked around it by disabling checks:

{
  description = "fix-my-dango";
  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    nixpacks.url = "github:railwayapp/nixpacks?rev=a8bf45a551e6be451633da04ba1d763c997ccc32";
  };
  outputs = { self, nixpkgs, flake-utils, nixpacks }:
    flake-utils.lib.eachDefaultSystem (system:  {
      devShell = pkgs.mkShell {
        nativeBuildInputs = with pkgs; [
          (nixpacks.packages.${system}.nixpacks.overrideAttrs (finalAttrs: previousAttrs: { doCheck = false; }))
        ];
      };
    });
}

Strangely, running nixpacks locally gives me error: opening file '/app/.nixpacks/rust-toolchain': No such file or directory even when NIXPACKS_CARGO_WORKSPACE=signalling-server is specified. My workaround is to remove ./crates/* from the cargo worksace in Cargo.toml.

Next up, I need to install docker onto my system.

[ErnWong]

Running NIXPACKS_CARGO_WORKSPACE=signalling-server nixpacks build ../dango-tribute/ --verbose

[ErnWong]

Fails locally

[nixos@NixOS:~/repos/fix-dango]$ NIXPACKS_CARGO_WORKSPACE=signalling-server nixpacks build ../dango-tribute/ --verbose

╔══════════════════════════════ Nixpacks v0.14.0 ══════════════════════════════╗
║ setup      │ binutils, gcc, (rust-bin.fromRustupToolchainFile ./rust-        ║
║            │ toolchain)                                                      ║
║──────────────────────────────────────────────────────────────────────────────║
║ build      │ mkdir -p bin                                                    ║
║            │ cargo build --release --package client                          ║
║            │ cp target/release/client bin                                    ║
║──────────────────────────────────────────────────────────────────────────────║
║ start      │ ./target/release/signalling-server                              ║
╚══════════════════════════════════════════════════════════════════════════════╝

#1 [internal] load build definition from Dockerfile
#1 sha256:ae5e3284bbc2039d6bb646ea2399e953c644ca00b76cd66ce98f68293102222b
#1 transferring dockerfile: 1.18kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:05e9b9f25881d05b34aaa4255e49918643784478b8b7509970938d742487ed6c
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for ghcr.io/railwayapp/nixpacks:debian-1667866002
#3 sha256:ff4546c29b1eb585dd7df95164933728a568cb8dc35398f9f51f11afd8b9a1cb
#3 DONE 2.9s

#4 [stage-0 1/9] FROM ghcr.io/railwayapp/nixpacks:debian-1667866002@sha256:96bc6d9c576900c9a99db8f7c8d6ef86c9e8dd9e54f46a119ff8499b9f5fe3b0
#4 sha256:a6aa9f4410ef5777e49afa17be8a1e260d09acb8a74de2dab233288bf3c56052
#4 resolve ghcr.io/railwayapp/nixpacks:debian-1667866002@sha256:96bc6d9c576900c9a99db8f7c8d6ef86c9e8dd9e54f46a119ff8499b9f5fe3b0 0.0s done
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 0B / 31.37MB 0.1s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 0B / 43.97MB 0.1s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 0B / 31.32MB 0.1s
#4 sha256:96bc6d9c576900c9a99db8f7c8d6ef86c9e8dd9e54f46a119ff8499b9f5fe3b0 1.05kB / 1.05kB done
#4 sha256:6f23d6fa754b831d051be24378aa643f53b753f57269fb9c5678a19ab673cf06 953B / 953B done
#4 sha256:af4b12c8eaa23e13d394a5847e0771fd9cbcecc43fbb534ffb360a1e8f319702 3.83kB / 3.83kB done
#4 ...

#6 [internal] load build context
#6 sha256:e0a7a3f33e7efa1312dc8cd253e37a35efd8782581beb6da5699f0c0a7520d83
#6 transferring context: 77.86MB 1.4s done
#6 DONE 1.5s

#4 [stage-0 1/9] FROM ghcr.io/railwayapp/nixpacks:debian-1667866002@sha256:96bc6d9c576900c9a99db8f7c8d6ef86c9e8dd9e54f46a119ff8499b9f5fe3b0
#4 sha256:a6aa9f4410ef5777e49afa17be8a1e260d09acb8a74de2dab233288bf3c56052
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 2.10MB / 31.32MB 1.7s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 2.10MB / 31.37MB 2.1s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 4.19MB / 31.32MB 2.4s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 6.29MB / 31.32MB 2.9s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 4.19MB / 31.37MB 3.3s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 3.15MB / 43.97MB 3.3s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 8.39MB / 31.32MB 3.5s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 10.49MB / 31.32MB 5.3s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 6.29MB / 31.37MB 5.5s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 12.58MB / 31.32MB 6.8s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 5.24MB / 43.97MB 8.4s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 14.68MB / 31.32MB 9.7s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 7.34MB / 31.37MB 10.6s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 6.29MB / 43.97MB 13.5s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 16.78MB / 31.32MB 14.4s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 9.44MB / 31.37MB 14.5s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 18.87MB / 31.32MB 15.0s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 11.53MB / 31.37MB 15.7s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 9.44MB / 43.97MB 15.7s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 20.97MB / 31.32MB 15.7s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 23.07MB / 31.32MB 16.3s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 13.63MB / 31.37MB 16.8s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 25.17MB / 31.32MB 17.0s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 12.58MB / 43.97MB 17.3s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 15.73MB / 31.37MB 17.8s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 27.26MB / 31.32MB 17.8s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 29.36MB / 31.32MB 18.4s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 17.83MB / 31.37MB 18.9s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 15.73MB / 43.97MB 19.1s
#4 sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 31.32MB / 31.32MB 19.2s done
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 19.92MB / 31.37MB 19.7s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 18.87MB / 43.97MB 20.0s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 22.02MB / 31.37MB 20.3s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 24.12MB / 31.37MB 21.0s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 22.02MB / 43.97MB 21.0s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 26.21MB / 31.37MB 21.6s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 25.17MB / 43.97MB 21.9s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 28.31MB / 31.37MB 22.4s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 30.41MB / 31.37MB 22.9s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 28.31MB / 43.97MB 22.9s
#4 sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 31.37MB / 31.37MB 23.1s done
#4 extracting sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 31.46MB / 43.97MB 23.4s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 34.60MB / 43.97MB 23.9s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 37.75MB / 43.97MB 24.3s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 40.89MB / 43.97MB 24.8s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 43.97MB / 43.97MB 25.3s
#4 sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 43.97MB / 43.97MB 25.3s done
#4 extracting sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7 2.8s done
#4 extracting sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 0.1s
#4 extracting sha256:178aa227edbde8b2fdd551b4ae9f6926fbe04923f910779888cf58588fefd3a9 4.0s done
#4 extracting sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930
#4 extracting sha256:2f73ae600eeca70e016decc7577a9be5f4daff0d315a19173dbd97ce21e14930 2.3s done
#4 DONE 32.9s

#5 [stage-0 2/9] WORKDIR /app/
#5 sha256:f1ff5906bd390875dc0cc6c2f06540658835071f9bb0611cd6ce76c903f39537
#5 DONE 0.7s

#7 [stage-0 3/9] COPY .nixpacks/nixpkgs-a0b7e70db7a55088d3de0cc370a59f9fbcc906c3.nix .nixpacks/nixpkgs-a0b7e70db7a55088d3de0cc370a59f9fbcc906c3.nix
#7 sha256:34aa8b39f06b0d6eba231927cc021fc32d75113f7981774572c2f7229ad57c61
#7 DONE 0.0s

#8 [stage-0 4/9] RUN nix-env -if .nixpacks/nixpkgs-a0b7e70db7a55088d3de0cc370a59f9fbcc906c3.nix && nix-collect-garbage -d
#8 sha256:6eca83260e93852bf497edcb86ca656f85a0474fd6801cf8e69a123007b924c3
#8 0.318 error: Operation not permitted
#8 ERROR: executor failed running [/bin/bash -ol pipefail -c nix-env -if .nixpacks/nixpkgs-a0b7e70db7a55088d3de0cc370a59f9fbcc906c3.nix && nix-collect-garbage -d]: exit code: 1
------
 > [stage-0 4/9] RUN nix-env -if .nixpacks/nixpkgs-a0b7e70db7a55088d3de0cc370a59f9fbcc906c3.nix && nix-collect-garbage -d:
------
executor failed running [/bin/bash -ol pipefail -c nix-env -if .nixpacks/nixpkgs-a0b7e70db7a55088d3de0cc370a59f9fbcc906c3.nix && nix-collect-garbage -d]: exit code: 1
Error: Docker build failed

[ErnWong]

Setting security.unprivilegedUsernsClone in my nixos configuration to true didn't help. It would be weird if it's a misconfiguration of the debian image used as the base of the docker image, since it works fine on the railway website. Looks very similar to the error here: https://github.com/railwayapp/nixpacks/issues/441. I might try updating my nixos.

[ErnWong]

After updating nixos, I now get pass the Operation not permitted error, and instead get the same error as in Railway about rust-toolchain not found.

[ErnWong]

Indeed if I follow https://github.com/railwayapp/nixpacks/blob/1a37a464e187781d5670d8b0abf9d3a823574663/CONTRIBUTING.md#debugging and run NIXPACKS_CARGO_WORKSPACE=signalling-server nixpacks build ../dango-tribute/ --out out, and inspect the ./out directory, I don't see any out/.nixpacks/rust-toolchain file in there: just the Dockerfile and the nix file.