How to capture the Chromium's requests and responses traffic using thirtyfour

[Codekies]

I want to use Rust with thirtyfour to open chromium and capture my HTTPS web app traffic. It is for the security tools development usage.

[stevepryde]

It's technically possible using CDP but you'll need to wire it up yourself.

CDP example here
https://github.com/stevepryde/thirtyfour/blob/main/thirtyfour/examples/chrome_devtools.rs

Chrome devtools protocol docs here
https://chromedevtools.github.io/devtools-protocol/

[Codekies]

Hi Steve,
Thanks a lot for the help. I have written some code to access google.com and capture the HTTP requests, response

use thirtyfour::extensions::cdp::ChromeDevTools;
use thirtyfour::prelude::*;

#[tokio::main]
async fn main() -> WebDriverResult<()> {
    let caps = DesiredCapabilities::chrome();
    let driver = WebDriver::new("http://localhost:9515", caps).await?;

    // Use Chrome Devtools Protocol (CDP).
    let dev_tools = ChromeDevTools::new(driver.handle.clone());

    // Enable network tracking.
    dev_tools.execute_cdp("Network.enable").await?;

    // Navigate to https://www.google.com.
    driver.goto("https://www.google.com").await?;

    // Wait for a short duration to allow requests and responses to be captured.
    tokio::time::sleep(std::time::Duration::from_secs(5)).await;

    // Get the captured network events.
    let request_events = dev_tools.execute_cdp("Network.Request").await?;
    let response_events = dev_tools.execute_cdp("Network.Response").await?;

    // Print the captured network events.
    println!("Requests:");
    println!("{:?}", request_events);

    println!("Responses:");
    println!("{:?}", response_events);

    // Disable network tracking.
    dev_tools.execute_cdp("Network.disable").await?;

    // Always explicitly close the browser. There are no async destructors.
    driver.quit().await?;

    Ok(())
}

but when I ran the executable, it showed me this.
Error: UnknownCommand(WebDriverErrorInfo { status: 404, error: "", value: WebDriverErrorValue { message: "unknown command: 'Network.Request' wasn't found\n (Session info: chrome=123.0.6312.123)", error: Some("unknown command"), stacktrace: Some("\tGetHandleVerifier [0x00007FF7F97A7072+63090]\n\t(No symbol) [0x00007FF7F9712CC2]\n\t(No symbol) [0x00007FF7F95AEC65]\n\t(No symbol) [0x00007FF7F959C96F]\n\t(No symbol) [0x00007FF7F959B100]\n\t(No symbol) [0x00007FF7F959BA8F]\n\t(No symbol) [0x00007FF7F959B9C0]\n\t(No symbol) [0x00007FF7F95B1973]\n\t(No symbol) [0x00007FF7F963FEBA]\n\t(No symbol) [0x00007FF7F9616FDA]\n\t(No symbol) [0x00007FF7F9633412]\n\t(No symbol) [0x00007FF7F9616D83]\n\t(No symbol) [0x00007FF7F95E83A8]\n\t(No symbol) [0x00007FF7F95E9441]\n\tGetHandleVerifier [0x00007FF7F9BA25CD+4238285]\n\tGetHandleVerifier [0x00007FF7F9BDF72D+4488493]\n\tGetHandleVerifier [0x00007FF7F9BD7A0F+4456463]\n\tGetHandleVerifier [0x00007FF7F98805B6+953270]\n\t(No symbol) [0x00007FF7F971E58F]\n\t(No symbol) [0x00007FF7F9719264]\n\t(No symbol) [0x00007FF7F971939B]\n\t(No symbol) [0x00007FF7F9709BD4]\n\tBaseThreadInitThunk [0x00007FFD482A257D+29]\n\tRtlUserThreadStart [0x00007FFD48EAAA48+40]\n"), data: None } })
Would you mind helping me to write an example code if you have time? Thanks a lot.

[stevepryde]

Hmm, looking at the docs here: https://chromedevtools.github.io/devtools-protocol/tot/Network/

It looks like Network.Request is a type, not a command.

I've seen some docs pointing at setRequestInterception: https://chromedevtools.github.io/devtools-protocol/tot/Network/#method-setRequestInterception but that shows as deprecated. It says to try the Fetch domain but I couldn't see anything obvious there.

There is some info here that might be useful: https://www.selenium.dev/documentation/webdriver/bidirectional/chrome_devtools/

I'm going to take a look at the selenium bindings and see how they do it there.

[stevepryde]

There's more info in #55 that might be helpful. Also check out https://github.com/mattsse/chromiumoxide which uses CDP directly without the webdriver side (more like puppeteer).

[stevepryde]

Also https://github.com/btbaggin/cdp-rs

[A1Gard]

I have same problem when or how, we can use the commands like: requestWillBeSent , responseReceived and etc?

this my code ( edited main sample ):

use thirtyfour::prelude::*;
use thirtyfour::extensions::cdp::ChromeDevTools;
use std::process::Command;
use tokio::time::{sleep, Duration};


#[tokio::main]
async fn main() -> WebDriverResult<()> {
    // Start ChromeDriver on port 2020
    let mut child = Command::new("chromedriver")
        .arg("--port=2020")
        .spawn()
        .expect("Failed to start chromedriver");

    // Wait for a moment to ensure ChromeDriver is up and running
    sleep(Duration::from_secs(2)).await;

    // Set up desired capabilities for Chrome
    let caps = DesiredCapabilities::chrome();

    // Connect to the ChromeDriver
    let driver = WebDriver::new("http://127.0.0.1:2020", caps).await?;

    // Enable the Chrome DevTools Protocol for network monitoring
    let dev_tools = ChromeDevTools::new(driver.handle.clone());
    dev_tools.execute_cdp("Network.enable").await?;


    // Set up a listener for network requests



    // Navigate to https://wikipedia.org
    driver.goto("https://wikipedia.org").await?;
    let elem_form = driver.find(By::Id("search-form")).await?;

    // Find the search input element
    let elem_text = elem_form.find(By::Id("searchInput")).await?;

    // Type in the search terms
    elem_text.send_keys("selenium").await?;

    // Click the search button
    let elem_button = elem_form.find(By::Css("button[type='submit']")).await?;
    elem_button.click().await?;

    // Look for header to implicitly wait for the page to load
    driver.find(By::ClassName("firstHeading")).await?;
    assert_eq!(driver.title().await?, "Selenium - Wikipedia");


    let response_interceptor = dev_tools.execute_cdp("Network.responseReceived").await?;
    println!("responses:");
    println!("{:?}", response_interceptor);

    
    // let request_interceptor = dev_tools.execute_cdp("Network.requestWillBeSent").await?;
    // println!("Requests:");
    // println!("{:?}", request_interceptor);


    let version_info = dev_tools.execute_cdp("Browser.getVersion").await?;
    println!("{:#?}", version_info);
    dev_tools.execute_cdp("Network.disable").await?;
    // Always explicitly close the browser
    driver.quit().await?;

    // Ensure the ChromeDriver process is terminated
    let _ = child.kill();

    Ok(())
}

error:

Error: WebDriverError(UnknownCommand(WebDriverErrorInfo { status: 404, error: "", value: WebDriverErrorValue { message: "unknown command: 'Network.responseReceived' wasn't found\n (Session info: chrome=131.0.6778.139)", error: Some("unknown command"), stacktrace: Some("#0 0x560244f3dd4e \n#1 0x560244a5fecb \n#2 0x560244a4b796 \n#3 0x560244a4a232 \n#4 0x560244a4a4e9 \n#5 0x560244a4a434 \n#6 0x560244a63e27 \n#7 0x560244af68c6 \n#8 0x560244acea62 \n#9 0x560244aeb5de \n#10 0x560244ace7e3 \n#11 0x560244a9b427 \n#12 0x560244a9c1c2 \n#13 0x560244f1141a \n#14 0x560244f14825 \n#15 0x560244f14268 \n#16 0x560244f14cc5 \n#17 0x560244f01bd2 \n#18 0x560244f15031 \n#19 0x560244eeaa09 \n#20 0x560244f2dd45 \n#21 0x560244f2df2b \n#22 0x560244f3cea5 \n#23 0x7f466dea81c4 \n"), data: None } }))
Process finished with exit code 1

there is not way to listen on events otherwise i study this issue too: #55