[BUG] Network (request/response) intercept doesn't work with location.reload(true) on Firefox

[barbolo]

System info

• Playwright Version: 1.32.2
• Operating System: macOS 13.3.1
• Browser: Firefox
• Other info:

Source code

const { firefox } = require('playwright');

(async () => {
  const browser = await firefox.launch();
  const page = await browser.newPage();

  page.on('request', request => console.log(`page.on("request") - ${request.url()}`))
  page.on('response', response => console.log(`page.on("response") - ${response.url()}`))
  page.on('framenavigated', frame => { console.log(`framenavigated - ${frame.url()}`) });
  page.context().on('request', (request) => console.log(`page.context.on('request') - ${request.url()}`));
  await page.route('**/*', (route, request) => {
    console.log(`page.route('**/*') - ${request.url()}`)
    route.continue();
  });

  await page.goto('https://cdn.infosimples.com/reload.html');

  // keep page reloading every 5 seconds
})();

Steps

• Run the source code

Expected

The expected output would be something like this (interceptions work for page reload).

page.context.on('request') - https://cdn.infosimples.com/reload.html
page.on("request") - https://cdn.infosimples.com/reload.html
page.route('**/*') - https://cdn.infosimples.com/reload.html
page.on("response") - https://cdn.infosimples.com/reload.html
framenavigated - https://cdn.infosimples.com/reload.html

page.context.on('request') - https://cdn.infosimples.com/reload.html
page.on("request") - https://cdn.infosimples.com/reload.html
page.route('**/*') - https://cdn.infosimples.com/reload.html
page.on("response") - https://cdn.infosimples.com/reload.html
framenavigated - https://cdn.infosimples.com/reload.html

...

Actual

Only framenavigated event is being intercepted after the page is first loaded:

page.context.on('request') - https://cdn.infosimples.com/reload.html
page.on("request") - https://cdn.infosimples.com/reload.html
page.route('**/*') - https://cdn.infosimples.com/reload.html
page.on("response") - https://cdn.infosimples.com/reload.html
framenavigated - https://cdn.infosimples.com/reload.html
framenavigated - https://cdn.infosimples.com/reload.html
framenavigated - https://cdn.infosimples.com/reload.html
framenavigated - https://cdn.infosimples.com/reload.html
framenavigated - https://cdn.infosimples.com/reload.html
framenavigated - https://cdn.infosimples.com/reload.html
framenavigated - https://cdn.infosimples.com/reload.html
framenavigated - https://cdn.infosimples.com/reload.html
....

[barbolo]

According to MDN docs, location.reload accepts forceGet boolean parameter in Firefox:

Pass true to force a reload bypassing the cache. Defaults to false. Only supported in Firefox.

Looking at firefox Location::Reload code:
https://searchfox.org/mozilla-central/rev/e9b8c73771d0038214c69f17c5934ba8ae049799/dom/base/Location.cpp#544

I believe a patch that would make firefox act like other browsers regarding location.reload would be to comment the following lines:

  if (aForceget) {
    reloadFlags = nsIWebNavigation::LOAD_FLAGS_BYPASS_CACHE |
                  nsIWebNavigation::LOAD_FLAGS_BYPASS_PROXY;
  }

I've spent some time looking at playwright code, but I couldn't find out why this reload is not intercepted.

[barbolo]

This looks like a problematic flag that might make playwright incapable of intercepting requests:

  /**
   * This flag specifies that any intermediate proxy caches should be bypassed
   * (i.e., that the content should be loaded from the origin server).
   */
  const unsigned long LOAD_FLAGS_BYPASS_PROXY    = 0x0200;

[barbolo]

Hi, @aslushnikov and @pavelfeldman.

Sorry for bothering you.

I need to fix this issue in the project I'm working on. If you decide to apply a patch in Firefox to fix this, how long it usually takes for firefox patched binaries to be available?

I'm going to patch and build Firefox on my own while you consider this issue, but it would be helpful to know how you deal with issues like this.

If you need any help, please let me know. Thanks.

[aslushnikov]

Hey @barbolo,

As a rule of thumb, we don't change how Web APIs work in browser; we only add instrumentation. So changing the behavior for the Location::Reload would go against this principle.

A proper solution would be figuring out why current network instrumentation is not triggered, and fix this.

[barbolo]

Hi, @aslushnikov. I'm taking a look at playwright code trying to understand how network instrumentation is implemented.

Do you have documentation or advice on this? For example, is playwright acting like an HTTP/MITM proxy when firefox starts? Or maybe you have patched firefox's network code to intercept request/response there?

[aslushnikov]

@barbolo yes, this is a part of Juggler instrumentation. See here: https://github.com/microsoft/playwright/blob/main/browser_patches/firefox/juggler/NetworkObserver.js

[barbolo]

Is the script roll_from_upstream.sh intended to be used with a private repo playwright-browsers?

I'm trying to build and debug firefox the same way you do so I can investigate this issue further.

[barbolo]

If anyone else wants to build firefox by their own, what I'm currently doing to understand how it might be done is to look at old scripts that are not in this repository anymore:
https://github.com/microsoft/playwright/tree/release-1.25/browser_patches

[dacloutier-logmein]

I'm facing a similar issue where request / response intercept doesn't work on Firefox, but we don't have any reload involved in the process.

All we're trying to do is block requests routed to a certain script (Pendo integration)

const blockPendo = async (route: Route): Promise<requestStatus> => {
  const req = route.request()
  if (req.url().includes('pendo.js') || req.url().includes('/pendo-')) {
    logger.trace(`Blocked request to ${req.url()}`)
    await route.abort()
    return requestStatus.BLOCKED
  }
  return requestStatus.VALID
}

This code works just fine with Chrome and Edge, but doesn't seem to have any effect on firefox.

using 1.51.0