Feature : Assertion Support

[mastrzyz]

A proper test follows:

• Arrange
• Act
• Assert

Right now, MCP does Arrange/Act but we don't have great support to do assertions:

browser_assert_visible

Description: Asserts an element is visible
Parameters:
element (string): Human-readable element description used to obtain permission to interact with the element
ref (string): Exact target element reference from the page snapshot

browser_assert_checked

[pavelfeldman]

It is unclear why with the presence of the LLM you want to use assertions. LLM is more than capable of doing it itself. Sounds like what you are really after is emitting source code with that assertion for a test. Could you provide an example of the workflow?

[mastrzyz]

In the cases where we are using accessibility snapshot, the LMM can't really detect if an element isEmpty or containsText in all situations.

Yes my end goal is to output all of what the LLM did to achieve a scenario into an actual Playwright test file.

Previously we had a function given to LLM with schema :

const validAssertions = [
  "toBeAttached",
  "toBeChecked",
  "toBeDisabled",
  "toBeEditable",
  "toBeEmpty",
  "toBeEnabled",
  "toBeFocused",
  "toBeHidden",
  "toBeInViewport",
  "toBeVisible",
  "toContainText",
  "toHaveAccessibleDescription",
  "toHaveAccessibleName",
  "toHaveAttribute",
  "toHaveClass",
  "toHaveCount",
  "toHaveCSS",
  "toHaveId",
  "toHaveJSProperty",
  "toHaveRole",
  "toHaveText",
  "toHaveValue",
  "toHaveValues",
  "pagetoHaveTitle",
  "pagetoHaveURL",
  "responseToBeOK",
];
  {
    name: "assert",
    description:
      "Perform various Playwright expect(...) checks on an element, page, or response",
    parameters: {
      type: "object",
      properties: {
        ...intentProperty,
        ...elementIdProperty,
        assertionType: {
          type: "string",
          enum: validAssertions,
          description: "Which Playwright assertion to run",
        },
        assertionValue: {
          type: "string",
          description: "Value to check against",
        },
      },
      required: ["intent", "assertionType", "assertionValue"],
    },
  }

And we told the LLM specifically to assert :

• Organizing each test step into three distinct phases: Arrange, Act, and Assert.

We then programmatically invoked the assertion and if all was good, we output the code

  if (toolName === "assert") {
    const assertionType = params.assertionType as string;
    if (validAssertions.indexOf(assertionType) === -1) {
      throw new Error(
        `AI provided invalid assertion type: ${assertionType}, it doesn't exist. Please provide from the list of valid assertions: ${validAssertions}`
      );
    }

    const assertionValue = params.assertionValue as string;
    if (assertionType === "toHaveText") {
      code.push(

        `await expect(page.${locatorString}).${assertionType}(\`${assertionValue}\`)`
      );
      await expect(locator)[assertionType](assertionValue);
      // else use a generic assertion if provided
    } else {
      console.log(
        `Creating Assertion with type ${assertionType} and value ${assertionValue}`
      );
      if (
        // doesn't need a value
        assertionType === "toBeAttached" ||
        assertionType === "toBeVisible" ||
        assertionType === "toBeHidden" ||
        assertionType === "toBeFocused" ||
        assertionType === "toBeEnabled" ||
        assertionType === "toBeDisabled" ||
        assertionType === "toBeEmpty" ||
        assertionValue === "" ||
        assertionValue === "undefined"
      ) {
        code.push(`await expect(page.${locatorString}).${assertionType}()`);
        await expect(locator)[assertionType]();
      } else {
        code.push(
          // Make sure to escape the string, the product has " in the UI, so use ` instead of "
          `await expect(page.${locatorString}).${assertionType}(\`${assertionValue}\`)`
        );
        await expect(locator)[assertionType](assertionValue);
      }
    }
    return;
  }

[mastrzyz]

@pavelfeldman not sure if this helps