baseline http vs conductor overhead benchmark #622

YassinEldeeb · 2024-11-27T12:16:38Z

introduces a benchmark between the purest form of an http layer in Rust (Actix Web) against Conductor's implementation, comparing their performance (requests per second and P95 latency) under load, then results are posted as a comment on PRs.

github-actions · 2024-11-27T12:17:16Z

🚨 Rust Panic Audit: 342 Potential Panic Points Detected 🚨

Crate: ``

📊 Total Usages: 239

🔎 expect usages: 41
🎁 unwrap usages: 159
🔢 array_index usages: 31
🚨 panic usages: 8

Crate: `federation_query_planner`

📊 Total Usages: 50

🚨 panic usages: 3
🔢 array_index usages: 10
🔎 expect usages: 8
🎁 unwrap usages: 29

Crate: `conductor`

📊 Total Usages: 11

🎁 unwrap usages: 2
🚨 panic usages: 2
🔎 expect usages: 7

Crate: `common`

📊 Total Usages: 11

🎁 unwrap usages: 10
🔢 array_index usages: 1

Crate: `cloudflare_worker`

📊 Total Usages: 8

🎁 unwrap usages: 5
🚨 panic usages: 1
🔎 expect usages: 2

Crate: `telemetry`

📊 Total Usages: 7

🔢 array_index usages: 4
🎁 unwrap usages: 3

Crate: `engine`

📊 Total Usages: 7

🎁 unwrap usages: 6
🔎 expect usages: 1

Crate: `tracing`

📊 Total Usages: 6

🎁 unwrap usages: 5
🔎 expect usages: 1

Crate: `config`

📊 Total Usages: 3

🎁 unwrap usages: 2
🚨 panic usages: 1

📌 Expected Annotations

Crate: `common`

📊 Total Expected Usages: 1

expand details

Reason: "we're parsing a statically defined constant, we know it works ;)"

Code: .unwrap()
Location: ./libs/common/src/graphql.rs:31

Crate: `conductor`

📊 Total Expected Usages: 2

expand details

Reason: "we need to exit the process, if the logger can't be correctly set."

Code: let _guard = tracing::subscriber::set_default(subscriber);
Location: ./bin/conductor/src/lib.rs:64

Reason: "we need to exit the process, if the provided configuration file is incorrect."

Code: panic!("Failed to initialize gateway: {:?}", e);
Location: ./bin/conductor/src/lib.rs:103

Crate: `napi`

📊 Total Expected Usages: 1

expand details

Reason: "we need this"

Code: panic!("Exited process!")
Location: ./libs/napi/src/lib.rs:18

Crate: `jwt_auth`

📊 Total Expected Usages: 1

expand details

Reason: "if initiating an http client fails, then we have to exit."

Code: let client = wasm_polyfills::create_http_client().build().unwrap();
Location: ./plugins/jwt_auth/src/jwks_provider.rs:49

Crate: `vrl`

📊 Total Expected Usages: 2

expand details

Reason: "if the provided VRL code in the config file can't compile, we have to exit."

Code: panic!("failed to compile vrl program");
Location: ./plugins/vrl/src/plugin.rs:129

Reason: "states is a non-user provided variable"

Code: .expect("can't merge states when states is an empty vector!")
Location: ./plugins/vrl/src/plugin.rs:146

Crate: `config`

📊 Total Expected Usages: 9

expand details

Reason: "part of development docgen CLI"

Code: .expect("Failed to serialize json schema for config file!");
Location: ./libs/config/src/generate-json-schema.rs:50

Reason: "part of development docgen CLI"

Code: .expect("Failed to write the json schema to the file system!");
Location: ./libs/config/src/generate-json-schema.rs:54

Reason: "👇"

Code: let raw_contents = read_to_string(file_path)
Location: ./libs/config/src/lib.rs:815

Reason: "👇"

Code: panic!("Failed to interpolate config file, please resolve the above errors");
Location: ./libs/config/src/lib.rs:847

Reason: "👇"

Code: parse_config_from_json(&config_string).expect("Failed to parse JSON config file")
Location: ./libs/config/src/lib.rs:854

Reason: "👇"

Code: parse_config_from_yaml(&config_string).expect("Failed to parse YAML config file")
Location: ./libs/config/src/lib.rs:858

Reason: "👇"

Code: _ => panic!("Unsupported config file extension"),
Location: ./libs/config/src/lib.rs:875

Reason: "👇"

Code: None => panic!("Config file has no extension"),
Location: ./libs/config/src/lib.rs:878

Reason: "statically defined regex pattern, we know it works ;)"

Code: .unwrap();
Location: ./libs/config/src/interpolate.rs:18

Crate: ``

📊 Total Expected Usages: 22

expand details

Reason: "if we are unable to construct the endpoints and attach them onto the gateway's http server, we have to exit"

Code: Err(e) => panic!("failed to construct endpoint: {:?}", e),
Location: ./libs/engine/src/gateway.rs:158

Reason: "we can safely index here, it's inside a test with constant defined fixtures."

Code: ConductorGateway::execute(request, &gw.routes[0].route_data).await
Location: ./libs/engine/src/gateway.rs:190

Reason: "we're parsing a statically defined constant, we know it works ;)"

Code: .unwrap()
Location: ./libs/common/src/graphql.rs:31

Reason: "we need this"

Code: panic!("Exited process!")
Location: ./libs/napi/src/lib.rs:18

Reason: "part of development docgen CLI"

Code: .expect("Failed to serialize json schema for config file!");
Location: ./libs/config/src/generate-json-schema.rs:50

Reason: "part of development docgen CLI"

Code: .expect("Failed to write the json schema to the file system!");
Location: ./libs/config/src/generate-json-schema.rs:54

Reason: "👇"

Code: let raw_contents = read_to_string(file_path)
Location: ./libs/config/src/lib.rs:815

Reason: "👇"

Code: panic!("Failed to interpolate config file, please resolve the above errors");
Location: ./libs/config/src/lib.rs:847

Reason: "👇"

Code: parse_config_from_json(&config_string).expect("Failed to parse JSON config file")
Location: ./libs/config/src/lib.rs:854

Reason: "👇"

Code: parse_config_from_yaml(&config_string).expect("Failed to parse YAML config file")
Location: ./libs/config/src/lib.rs:858

Reason: "👇"

Code: _ => panic!("Unsupported config file extension"),
Location: ./libs/config/src/lib.rs:875

Reason: "👇"

Code: None => panic!("Config file has no extension"),
Location: ./libs/config/src/lib.rs:878

Reason: "statically defined regex pattern, we know it works ;)"

Code: .unwrap();
Location: ./libs/config/src/interpolate.rs:18

Reason: "we need to exit the process, if the logger can't be correctly set."

Code: let _guard = tracing::subscriber::set_default(subscriber);
Location: ./bin/conductor/src/lib.rs:64

Reason: "we need to exit the process, if the provided configuration file is incorrect."

Code: panic!("Failed to initialize gateway: {:?}", e);
Location: ./bin/conductor/src/lib.rs:103

Reason: "it panics only if the header name is not valid, and we know it is."

Code: .unwrap()
Location: ./bin/cloudflare_worker/src/http_tracing.rs:20

Reason: "it panics only if the URL source is not valid, and it's already validated before."

Code: let url = req.url().unwrap();
Location: ./bin/cloudflare_worker/src/http_tracing.rs:23

Reason: "it only panics if we are not running in a CF context, should be safe."

Code: let cf_info = req.cf().unwrap();
Location: ./bin/cloudflare_worker/src/http_tracing.rs:27

Reason: "unwraps only in special cases where "data:text" is used."

Code: let http_host = url.host().unwrap().to_string();
Location: ./bin/cloudflare_worker/src/http_tracing.rs:36

Reason: "if initiating an http client fails, then we have to exit."

Code: let client = wasm_polyfills::create_http_client().build().unwrap();
Location: ./plugins/jwt_auth/src/jwks_provider.rs:49

Reason: "if the provided VRL code in the config file can't compile, we have to exit."

Code: panic!("failed to compile vrl program");
Location: ./plugins/vrl/src/plugin.rs:129

Reason: "states is a non-user provided variable"

Code: .expect("can't merge states when states is an empty vector!")
Location: ./plugins/vrl/src/plugin.rs:146

Crate: `engine`

📊 Total Expected Usages: 2

expand details

Reason: "if we are unable to construct the endpoints and attach them onto the gateway's http server, we have to exit"

Code: Err(e) => panic!("failed to construct endpoint: {:?}", e),
Location: ./libs/engine/src/gateway.rs:158

Reason: "we can safely index here, it's inside a test with constant defined fixtures."

Code: ConductorGateway::execute(request, &gw.routes[0].route_data).await
Location: ./libs/engine/src/gateway.rs:190

Crate: `cloudflare_worker`

📊 Total Expected Usages: 4

expand details

Reason: "it panics only if the header name is not valid, and we know it is."

Code: .unwrap()
Location: ./bin/cloudflare_worker/src/http_tracing.rs:20

Reason: "it panics only if the URL source is not valid, and it's already validated before."

Code: let url = req.url().unwrap();
Location: ./bin/cloudflare_worker/src/http_tracing.rs:23

Reason: "it only panics if we are not running in a CF context, should be safe."

Code: let cf_info = req.cf().unwrap();
Location: ./bin/cloudflare_worker/src/http_tracing.rs:27

Reason: "unwraps only in special cases where "data:text" is used."

Code: let http_host = url.host().unwrap().to_string();
Location: ./bin/cloudflare_worker/src/http_tracing.rs:36

github-actions · 2024-11-27T12:24:39Z

🐋 This PR was built and pushed to the following Docker images:

Docker Bake metadata

{
"conductor": {
  "buildx.build.provenance": {
    "buildType": "https://mobyproject.org/buildkit@v1",
    "materials": [
      {
        "uri": "pkg:docker/[email protected]?platform=linux%2Famd64",
        "digest": {
          "sha256": "a92ed51e0996d8e9de041ca05ce623d2c491444df6a535a566dabd5cb8336946"
        }
      },
      {
        "uri": "pkg:docker/[email protected]?platform=linux%2Famd64",
        "digest": {
          "sha256": "83101f6985c93e1e6501b3375de188ee3d2cbb89968bcc91611591f9f447bd42"
        }
      }
    ],
    "invocation": {
      "configSource": {
        "entryPoint": "Dockerfile"
      },
      "parameters": {
        "frontend": "dockerfile.v0",
        "args": {
          "label:org.opencontainers.image.authors": "The Guild <[email protected]>",
          "label:org.opencontainers.image.description": "Conductor is a robust GraphQL Gateway.",
          "label:org.opencontainers.image.docs": "https://the-guild.dev/graphql/gateway",
          "label:org.opencontainers.image.licenses": "MIT",
          "label:org.opencontainers.image.revision": "c845e2ed9514caa425bc72ff5e1fc6ef4d5bbfbf",
          "label:org.opencontainers.image.source": "https://github.com/the-guild-org/conductor",
          "label:org.opencontainers.image.title": "Conductor",
          "label:org.opencontainers.image.url": "https://the-guild.dev/graphql/gateway",
          "label:org.opencontainers.image.vendor": "The Guild",
          "label:org.opencontainers.image.version": ""
        },
        "locals": [
          {
            "name": "context"
          },
          {
            "name": "dockerfile"
          }
        ]
      },
      "environment": {
        "platform": "linux/amd64"
      }
    }
  },
  "buildx.build.ref": "builder-af8e143d-af81-4a40-8ad8-88f365234828/builder-af8e143d-af81-4a40-8ad8-88f3652348280/fpt4tjpnbhxpsp9qt1yeua86u",
  "containerimage.config.digest": "sha256:1ce7dd1b3f8b008d04218e281e037bf989c2e84ca1815be3147ea62b58bbc8f8",
  "containerimage.descriptor": {
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "digest": "sha256:d3071b2137b6c88ce39fe0905b8feb320b296d59e98df95105fc81bc44f3040d",
    "size": 902,
    "platform": {
      "architecture": "amd64",
      "os": "linux"
    }
  },
  "containerimage.digest": "sha256:d3071b2137b6c88ce39fe0905b8feb320b296d59e98df95105fc81bc44f3040d",
  "image.name": "ghcr.io/the-guild-org/conductor/conductor:c845e2ed9514caa425bc72ff5e1fc6ef4d5bbfbf"
}
}

github-actions · 2024-11-27T12:25:43Z

✅ Benchmark Results

     data_received..................: 13 MB   221 kB/s
     data_sent......................: 22 MB   363 kB/s
     http_req_blocked...............: min=701ns    avg=2.69µs   med=2.02µs   max=717.62µs p(95)=2.99µs   p(99)=11.49µs 
     http_req_connecting............: min=0s       avg=353ns    med=0s       max=670.6µs  p(95)=0s       p(99)=0s      
     http_req_duration..............: min=311.19µs avg=405.54µs med=384.03µs max=12.76ms  p(95)=476.85µs p(99)=561.07µs
       { expected_response:true }...: min=311.19µs avg=405.54µs med=384.03µs max=12.76ms  p(95)=476.85µs p(99)=561.07µs
     ✓ { scenario:rps_1000 }........: min=311.19µs avg=405.54µs med=384.03µs max=12.76ms  p(95)=476.85µs p(99)=561.07µs
     http_req_failed................: 0.00%   ✓ 0          ✗ 60001
     ✓ { scenario:rps_1000 }........: 0.00%   ✓ 0          ✗ 60001
     http_req_receiving.............: min=10.13µs  avg=26.16µs  med=25.68µs  max=2.53ms   p(95)=33.43µs  p(99)=39.6µs  
     http_req_sending...............: min=6.2µs    avg=14.9µs   med=13.81µs  max=1.4ms    p(95)=22.8µs   p(99)=32.18µs 
     http_req_tls_handshaking.......: min=0s       avg=0s       med=0s       max=0s       p(95)=0s       p(99)=0s      
     http_req_waiting...............: min=271.65µs avg=364.48µs med=343.25µs max=12.73ms  p(95)=436.25µs p(99)=514.79µs
     http_reqs......................: 60001   999.997335/s
     ✓ { scenario:rps_1000 }........: 60001   999.997335/s
     iteration_duration.............: min=387.98µs avg=492.44µs med=469.85µs max=12.87ms  p(95)=568.16µs p(99)=721.07µs
     iterations.....................: 60001   999.997335/s
     ✓ { scenario:rps_1000 }........: 60001   999.997335/s
     valid_graphql_response.........: 100.00% ✓ 60001      ✗ 0    
     ✓ { scenario:rps_1000 }........: 100.00% ✓ 60001      ✗ 0    
     valid_http_code................: 100.00% ✓ 60001      ✗ 0    
     ✓ { scenario:rps_1000 }........: 100.00% ✓ 60001      ✗ 0    
     vus............................: 1       min=0        max=2  
     vus_max........................: 200     min=200      max=200

github-actions · 2024-11-27T13:49:49Z

Pure HTTP vs Conductor Overhead Benchmark Results

Implementation	Requests/sec	P95 Latency (ms)
Pure HTTP (Actix Web)	15483	198
Conductor	13816	236

baseline http vs conductor overhead benchmark

48eeafe

YassinEldeeb added 6 commits November 27, 2024 12:38

ci

311f687

fix k6 installation

4e9d045

fix k6 paths

d8f69bb

fix k6 paths

33a6d17

shorten benches

48995e1

increase timeout

79228ea

YassinEldeeb added 4 commits November 27, 2024 14:26

fix

91e8d94

fix

7407890

fix

d22750c

fix

7925839

YassinEldeeb requested a review from dotansimha November 27, 2024 14:59

YassinEldeeb added 2 commits November 27, 2024 15:19

update title

ebc3312

update

c845e2e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

baseline http vs conductor overhead benchmark #622

baseline http vs conductor overhead benchmark #622

Uh oh!

YassinEldeeb commented Nov 27, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Nov 27, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Nov 27, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Nov 27, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Nov 27, 2024 •

edited

Loading

Uh oh!

Uh oh!

baseline http vs conductor overhead benchmark #622

Are you sure you want to change the base?

baseline http vs conductor overhead benchmark #622

Uh oh!

Conversation

YassinEldeeb commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚨 Rust Panic Audit: 342 Potential Panic Points Detected 🚨

Crate: ``

Crate: federation_query_planner

Crate: conductor

Crate: common

Crate: cloudflare_worker

Crate: telemetry

Crate: engine

Crate: tracing

Crate: config

📌 Expected Annotations

Crate: common

Crate: conductor

Crate: napi

Crate: jwt_auth

Crate: vrl

Crate: config

Crate: ``

Crate: engine

Crate: cloudflare_worker

Uh oh!

github-actions bot commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Benchmark Results

Uh oh!

github-actions bot commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pure HTTP vs Conductor Overhead Benchmark Results

Uh oh!

Uh oh!

YassinEldeeb commented Nov 27, 2024 •

edited

Loading

github-actions bot commented Nov 27, 2024 •

edited

Loading

Crate: `federation_query_planner`

Crate: `conductor`

Crate: `common`

Crate: `cloudflare_worker`

Crate: `telemetry`

Crate: `engine`

Crate: `tracing`

Crate: `config`

Crate: `common`

Crate: `conductor`

Crate: `napi`

Crate: `jwt_auth`

Crate: `vrl`

Crate: `config`

Crate: `engine`

Crate: `cloudflare_worker`

github-actions bot commented Nov 27, 2024 •

edited

Loading

github-actions bot commented Nov 27, 2024 •

edited

Loading

github-actions bot commented Nov 27, 2024 •

edited

Loading