Some YAMLs only work with JavaScript

[crdunwel]

Perhaps this issue has been raised before but here it goes.

Example: https://github.com/unitedstates/contact-congress/blob/master/members/P000593.yaml

Problematic URL in Form submission process: https://forms.house.gov/htbin/formproc_za/perlmutter/webforms/issue_subscribe_parm.txt&form=/perlmutter/webforms/issue_subscribe_verify.html

<script language="javascript">
//document.all.Submit.click();
document.forms[0].Submit.click();
</script>

As one can see, there is a final step at the end of this form submission process where JavaScript is submitting a form. If one uses a headless browser like Selenium then the JavaScript will execute and the form will submit. Otherwise, this is the last page that the steps defined by the YAML will reach (and thus not submitting the form).

It would be helpful to identify and fix the YAMLs where this issue occurs.

[j-ro]

What exactly is the fix? A find command before success?

[crdunwel]

I'll submit a pull request soon with my fix for this example. All the YAMLs experiencing a javascript problem have not been identified and so they may vary in how to fix. In the example from this issue, there needs to be additional steps for finding and clicking on the submit button since the javascript may not fire. So we can add to the end of the steps:

- find:
  - selector: "#sbutton"
- click_on:
  - selector: "#sbutton input[type='submit'][value='Submit']"

Which should fix the the issue in this example.

Additionally, the success page's body indeed does contain "Thank you for contacting me through my website" but it also contains the following in the head:

<META HTTP-EQUIV="refresh" CONTENT="10;url=http://perlmutter.house.gov">

Some drivers by default will honor this meta tag and treat the refresh as the final destination of the URL (ultimately causing failures because the final body text, in this case, is the homepage of the Rep.). This can be solved in a number of ways, but one is to set a "matches" attribute instead of "contains" in the success portion of the YAML with a regex for either the initial page or the page after refresh since either may be considered a success. Alternatively, we can ignore it and let the developer compensate by altering options of their driver (probably best for now).

[j-ro]

cool, I'll be on the lookout for finds. And with refreshes, as you've said, I've made it work either by matching the first page or the one after refresh.