Javascript crawler Architecture and Implementation plan
The goal of this document is to define how the JavaScript crawler feature is going to work, and the steps required to implement it.
The proof of concept step allows us to confirm that the architecture meets all the requirements needed to have a working javascript crawler.
The proof of concept step will only send the URL to the browser and wait for it to complete loading. During this load phase the site might redirect the user to another location using JS, send XMLHTTPRequests, etc. All of those HTTP requests will be captured and were invisible to w3af without the browser engine.
The crawler is now extended to click on links, buttons, etc.
This step gives more value to the users, since now the javascript crawler is actually interacting with the site.
The crawler is now extended to fill forms and submit them.
This is the final step in the implementation and will give users great value. This final step is making w3af interact with the site like any real-life user would.