Nick is a special module made by the Phantombuster team. It provides an easy navigation/web automation/scraping system. It's based on CasperJS and is written in CoffeeScript.
CasperJS is a powerful library with many methods. But only a few are essential. Nick limits the number of methods and replaces the step-by-step CasperJS paradigm with the async/callback paradigm of Node.
Before Nick methods can be used, you need to require() a Nick class and instantiate it.
Your agents must be launched with the CasperJS command when using this module. Nick is not compatible with Node or PhantomJS.
The module is named Nick. Use require('lib-Nick-beta') and create an instance to use it. A typical Nick-based script starts like this:
'use strict';
'phantombuster command: casperjs';
'phantombuster package: 2';
'phantombuster dependencies: lib-Nick-beta.coffee'
var Nick = require('lib-Nick-beta');
var nick = new Nick({
printNavigation: true
});
An instance of Nick is similar to a browser tab. If you want to open multiple pages/tabs simultaneously, instantiate multiple Nicks.
Following the philosophy of Node, most of Nick's methods are asynchronous. You have to use the callback function to know when (and if) a call finished successfully.
For example, this is bad:
nick.screenshot('image.jpg', function() {
console.log('Screenshot taken');
});
nick.exit(); // BAD! nick.screenshot() will not have finished here!
This is better:
nick.screenshot('image.jpg', function(err) {
if (err) {
console.log('Error when capturing the screenshot: ' + err);
nick.exit(1);
} else {
nick.exit(0);
}
});
nick.open(url [, options], callback);
Opens the webpage at url. You can forge GET, POST, PUT, DELETE and HEAD requests.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#open
url(String)- URL of the page to visit.
options(PlainObject)Optional request headers.
{ method: 'post', data: { 'title': 'Plop', 'body': 'Wow.' }, headers: { 'Accept-Language': 'fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3' } }callback(Function())- Function called when finished. No arguments are returned. To know if the page opened successfully, use waitUntilVisible() or similar.
Example:
nick.open("https://phantombuster.com/cloud-services", function() {
console.log("The page is loaded (or not!) — use waitUntilVisible() or similar to be sure");
nick.exit();
});
nick.wait(duration, callback);
Waits for a duration time before calling callback.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#wait
duration(Number)- Milliseconds to wait before calling
callbackfunction. callback(Function())- Function called when finished. This function never fails, no arguments will be passed.
Example:
nick.open("https://phantombuster.com/cloud-services", function() {
console.log('Hello')
nick.wait(1000, function() {
console.log('world!');
nick.exit();
})
});
nick.waitUntilPresent(selectors, timeout [, condition = "and"], callback);
Waits until a DOM element, matching the provided selector, is present. If the method has to wait more than timeout milliseconds, callback is called with a "timeout" error. By default, condition is set to "and".
It is considered good practice to always use a wait*() method after a page load and before any action on selectors.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#waitforselector
selectors(Array or String)- An array of CSS3 selectors describing the path to DOM elements.
timeout(Number)- Milliseconds to wait before calling
callbackfunction with an error. condition(String)- When
selectorsis an array, this argument lets you choose how to wait for the elements. Ifconditionis"and", the method will wait for the presence of allselectorsin the DOM. Otherwise ifconditionis"or", the method will wait until anyselectorof the array is present in the DOM. callback(Function(String err, String sel))Function called when finished. When there is no error,
erris null.- In case of success (
erris null): - if
conditionis"and"thenselis null because all selectors are present - if
conditionis"or"thenselis one of the present selectors of the given array
- if
- In case of success (
- In case of failure (
erris"timeout") - if
conditionis"and"thenselis one of the absent selectors of the given array - if
conditionis"or"thenselis null because no selectors were found
- if
- In case of failure (
Example with selector argument as a string:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitUntilPresent('html', 2000, function(err) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("'html' selector is present");
nick.exit(0);
});
});
This example succeeds if all selectors are present in the DOM:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitUntilPresent(['p', 'span', 'h2.title'], 2000, 'and', function(err, selector) {
if (err) {
console.log(err);
console.log("One of the missing selectors is:", selector);
nick.exit(1);
}
console.log("'html', 'foo', 'bar' selectors are present");
nick.exit(0);
});
});
This example succeeds if one or more selector is present in the DOM:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitUntilPresent(['p', 'span', 'h2.title'], 2000, 'or', function(err, selector) {
if (err) {
console.log(err);
console.log("'p', 'span', 'h2.title' selectors are missing");
nick.exit(1);
}
console.log("First matching selector:", selector);
nick.exit(0);
});
});
nick.waitWhilePresent(selectors, timeout [, condition = "and"], callback);
Waits while a DOM element, matching the provided selector, is present. If the method has to wait more than timeout milliseconds, callback is called with a "timeout" error. By default, condition is set to "and".
It is considered good practice to always use a wait*() method after a page load and before any action on selectors.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#waitwhileselector
selectors(Array or String)- An array of CSS3 selectors describing the path to DOM elements.
timeout(Number)- Milliseconds to wait before calling
callbackfunction with an error. condition(String)- When
selectorsis an array, this argument lets you choose how to wait for the elements. Ifconditionis"and", the method will wait for the presence of allselectorsin the DOM. Otherwise ifconditionis"or", the method will wait until anyselectorof the array is present in the DOM. callback(Function(String err, String sel))Function called when finished. When there is no error,
erris null.- In case of success (
erris null): - if
conditionis"and"thenselis null because all selectors are present - if
conditionis"or"thenselis one of the present selectors of the given array
- if
- In case of success (
- In case of failure (
erris"timeout") - if
conditionis"and"thenselis one of the absent selectors of the given array - if
conditionis"or"thenselis null because no selectors were found
- if
- In case of failure (
Example with selector argument as a string:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitWhilePresent('html', 2000, function(err) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("'html' selector is not present anymore");
nick.exit(0);
});
});
This example succeeds if all selectors is not present in the DOM:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitWhilePresent(['p', 'span', 'h2.title'], 2000, 'and', function(err, selector) {
if (err) {
console.log(err);
console.log("One of the missing selectors is:", selector);
nick.exit(1);
}
console.log("'html', 'foo', 'bar' selectors are present");
nick.exit(0);
});
});
This example succeeds if one or more selector is not present in the DOM:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitWhilePresent(['p', 'span', 'h2.title'], 2000, 'or', function(err, selector) {
if (err) {
console.log(err);
console.log("'p', 'span', 'h2.title' selectors are missing");
nick.exit(1);
}
console.log("First matching selector:", selector);
nick.exit(0);
});
});
nick.waitUntilVisible(selectors, timeout [, condition = "and"], callback);
Waits until a DOM element, matching the provided selector, is visible. If the method has to wait more than timeout milliseconds, callback is called with a "timeout" error. By default, condition is set to "and".
It is considered good practice to always use a wait*() method after a page load and before any action on selectors.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#waituntilvisible
selectors(Array or String)- An array of CSS3 selectors describing the path to DOM elements.
timeout(Number)- Milliseconds to wait before calling
callbackfunction with an error. condition(String)- When
selectorsis an array, this argument lets you choose how to wait for the elements. Ifconditionis"and", the method will wait for the presence of allselectorsin the DOM. Otherwise ifconditionis"or", the method will wait until anyselectorof the array is present in the DOM. callback(Function(String err, String sel))Function called when finished. When there is no error,
erris null.- In case of success (
erris null): - if
conditionis"and"thenselis null because all selectors are present - if
conditionis"or"thenselis one of the present selectors of the given array
- if
- In case of success (
- In case of failure (
erris"timeout") - if
conditionis"and"thenselis one of the absent selectors of the given array - if
conditionis"or"thenselis null because no selectors were found
- if
- In case of failure (
Example with selector argument as a string:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitUntilVisible('html', 2000, function(err) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("'html' selector is not present anymore");
nick.exit(0);
});
});
This example succeeds if all selectors is visible in the DOM:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitUntilVisible(['p', 'span', 'h2.title'], 2000, 'and', function(err, selector) {
if (err) {
console.log(err);
console.log("One of the missing selectors is:", selector);
nick.exit(1);
}
console.log("'html', 'foo', 'bar' selectors are present");
nick.exit(0);
});
});
This example succeeds if one or more selector is visible in the DOM:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitUntilVisible(['p', 'span', 'h2.title'], 2000, 'or', function(err, selector) {
if (err) {
console.log(err);
console.log("'p', 'span', 'h2.title' selectors are missing");
nick.exit(1);
}
console.log("First matching selector:", selector);
nick.exit(0);
});
});
nick.waitWhileVisible(selectors, timeout [, condition = "and"], callback);
Waits while a DOM element, matching the provided selector, is visible. If the method has to wait more than timeout milliseconds, callback is called with a "timeout" error. By default, condition is set to "and".
It is considered good practice to always use a wait*() method after a page load and before any action on selectors.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#waitwhilevisible
selectors(Array or String)- An array of CSS3 selectors describing the path to DOM elements.
timeout(Number)- Milliseconds to wait before calling
callbackfunction with an error. condition(String)- When
selectorsis an array, this argument lets you choose how to wait for the elements. Ifconditionis"and", the method will wait for the presence of allselectorsin the DOM. Otherwise ifconditionis"or", the method will wait until anyselectorof the array is present in the DOM. callback(Function(String err, String sel))Function called when finished. When there is no error,
erris null.- In case of success (
erris null): - if
conditionis"and"thenselis null because all selectors are present - if
conditionis"or"thenselis one of the present selectors of the given array
- if
- In case of success (
- In case of failure (
erris"timeout") - if
conditionis"and"thenselis one of the absent selectors of the given array - if
conditionis"or"thenselis null because no selectors were found
- if
- In case of failure (
Example with selector argument as a string:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitWhileVisible('html', 2000, function(err) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("'html' selector is not present anymore");
nick.exit(0);
});
});
This example succeeds if all selectors are not visible:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitWhileVisible(['p', 'span', 'h2.title'], 2000, 'and', function(err, selector) {
if (err) {
console.log(err);
console.log("One of the missing selectors is:", selector);
nick.exit(1);
}
console.log("'html', 'foo', 'bar' selectors are present");
nick.exit(0);
});
});
This example succeeds if one or more selector is not visible:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitWhileVisible(['p', 'span', 'h2.title'], 2000, 'or', function(err, selector) {
if (err) {
console.log(err);
console.log("'p', 'span', 'h2.title' selectors are missing");
nick.exit(1);
}
console.log("First matching selector:", selector);
nick.exit(0);
});
});
Exit the process.
Exit the process.
nick.evaluate(sandboxedFunction [, argumentObject], callback);
Evaluates the function in the current page DOM context. The execution is sandboxed, the web page has no access to the Nick context. Data can be given through argumentObject.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#evaluate
sandboxedFunction(Function([Object argumentObject]))- The function evaluated in the DOM context.
argumentObjectis a copy of the object given in the second optional argument. argumentObject(PlainObject)- Object to copy to the DOM context and given to the
sandboxedFunctionoptional argument. callback(Function(String err[, Object ret]))- Function called when finished. When there is no error,
erris null andretis a copy of the object returned by sandboxedFunction call in DOM context.
Example:
var num = 21;
nick.evaluate(function(arg) {
return arg.n * 2;
}, {
'n': num
}, function(err, ret) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("Evaluation succeeded. Return value is", ret); // "Evaluation succeeded. Return value is 42"
nick.exit(0);
});
nick.evaluateAsync(sandboxedFunction [, argumentObject], callback);
Evaluates the function in the current page DOM context. The execution is sandboxed and asynchronous, the web page has no access to the Nick context. Data can be given through argumentObject. Because sandboxedFunction is asynchronous the function done must be called.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#evaluate
sandboxedFunction(Function([Object argumentObject], done))- The function evaluated in the DOM context.
argumentObjectis a copy of the object given in the second optional argument.donemust be called before the function ends with the same arguments ascallback. argumentObject(PlainObject)- Object to copy to the DOM context and given to the
sandboxedFunctionoptional argument. callback(Function(String err[, Object ret]))- Function called when finished. When there is no error,
erris null andretis a copy of the object returned by sandboxedFunction call in DOM context.
Example:
var num = 21;
nick.evaluateAsync(function(arg, done) {
return done(null, arg.n * 2;)
}, {
'n': num
}, function(err, ret) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("Evaluation succeeded. Return value is", ret); // "Evaluation succeeded. Return value is 42"
nick.exit(0);
});
nick.inject(url, callback);
Inject a script in the current DOM page context. The script can be hosted locally on the agent's disk or on a remote server.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
url(object)- Path to a script hosted locally or remotely.
callback(Function(String err))- Function called when finished. When there is no error,
erris null.
Example:
nick.inject("https://code.jquery.com/jquery-2.1.4.min.js", function(err) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("Jquery script inserted!");
nick.exit(0);
});
nick.click(selector, callback);
Performs a click on the element matching the provided selector expression.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#click
selector(string)- A CSS3 or XPath expression that describe the path to DOM elements.
callback(Function(String err))- Function called when finished. When there is no error,
erris null and object is a valid object (which may be empty but never null).
Example:
var selector = "a.btn-warning";
nick.open("https://phantombuster.com/cloud-services", function() {
nick.waitUntilVisible(selector, 2000, function(err) {
if (err) {
console.log(err)
nick.exit(1);
}
nick.click(selector, function(err) {
if (err) {
console.log(err)
nick.exit(1);
}
console.log("Click on 'TRY FREE' button done.");
nick.exit(0);
});
});
});
nick.getCurrentUrl(callback)
Retrieves current page URL and calls the callback function with the URL in second argument. Note that the url will be url-decoded.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#getcurrenturl
callback(Function(String err, String decodedUrl))- Function called when finished. When there is no error,
erris null anddecodedUrlis a url-decoded string.
Example:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.getCurrentUrl(function(err, url) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("Current Url: ", url);
nick.exit(0);
});
});
nick.getCurrentUrlOrNull()
This method is synchronous and returns null if it fails otherwise it returns a the current URL as a string. Note that the url will be url-decoded.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#getcurrenturl
This function takes no arguments.
Example:
nick.open("https://phantombuster.com/cloud-services", function() {
console.log(nick.getCurrentUrlOrNull());
nick.exit();
});
nick.getHtml(callback)
Retrieves current page HTML and calls the callback function with the HTML as a string in second argument.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#gethtml
callback(Function(String err, String html))- Function called when finished. When there is no error,
erris null andhtmlis the HTML string.
Example:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.getHtml(function(err, html) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("HTML: ", html);
nick.exit(0);
});
});
nick.getHtmlOrNull()
This method is synchronous and returns null if it fails otherwise it returns the page HTML as a string.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#gethtml
This function takes no arguments.
Example:
nick.open("https://phantombuster.com/cloud-services", function() {
console.log(nick.getHtmlOrNull());
nick.exit();
});
nick.getContent(callback)
Retrieves current page content and call the callback function with the page content as a string in the second argument.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#getpagecontent
callback(Function(String err, String html))- Function called when finished. When there is no error,
erris null andhtmlis the HTML string.
Example:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.getPageContent(function(err, content) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("Page content: ", content);
nick.exit(0);
});
});
nick.getContentOrNull()
This method is synchronous and returns null if it fails otherwise it returns the page content (string).
More info: http://docs.casperjs.org/en/latest/modules/casper.html#getpagecontent
This function takes no arguments.
Example:
nick.open("https://phantombuster.com/cloud-services", function() {
var content = nick.getPageContentOrNull();
if (content == null) {
console.log("content is null");
nick.exit(1);
}
console.log("Content: ", content);
nick.exit(0);
});
nick.getTitle(callback)
Retrieves current page title and call the callback function with the title in second argument.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#gettitle
callback(Function(String err, String title))- Function called when finished. When there is no error,
erris null andtitleis the current page title string.
Example:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.getTitle(function(err, title) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("Page title: ", title);
nick.exit(0);
});
});
nick.getTitleOrNull()
This method is synchronous and returns null if it fails otherwise it returns a the current page title string.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#gettitle
This function takes no arguments.
Example:
nick.open("https://phantombuster.com/cloud-services", function() {
var title = nick.getTitleOrNull();
if (title == null) {
console.log("title is null");
nick.exit(1);
}
console.log("Title: ", title);
nick.exit(0);
});
nick.fill(selector, inputs [, submit], callback);
Fills form inputs with the given values and optionally submits it. Inputs are referenced by their name attribute.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#gettitle
selector(String)- A CSS3 or XPath expression that describe the path to DOM elements.
inputs(PlainObject)- An object composed by name:value, with name, the input name and value, the value to set.
submit(Boolean)- If
truethe form will be automatically sent. callback(Function(String err))- Function called when finished. When there is no error,
erris null.
Example with simple HTML form:
<form action="/contact" id="contact-form" enctype="multipart/form-data">
<input type="text" name="subject"/>
<textearea name="content"></textearea>
<input type="radio" name="civility" value="Mr"/> Mr
<input type="radio" name="civility" value="Mrs"/> Mrs
<input type="text" name="name"/>
<input type="email" name="email"/>
<input type="file" name="attachment"/>
<input type="checkbox" name="cc"/> Receive a copy
<input type="submit"/>
</form>
A Nick script filling the form and sending it:
nick.open("https://example.com", function() {
nick.fill('form#contact-form', {
'subject': 'I am watching you',
'content': 'So be careful.',
'civility': 'Mr',
'name': 'Chuck Norris',
'email': 'chuck@norris.com',
'cc': true,
'attachment': '/Users/chuck/roundhousekick.doc'
}, true, function(err) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("Form sent!");
nick.exit(0);
});
});
nick.screenshot(filename [, clipRect, imgOptions], callback)
Take a screenshot of the current page. Without optional arguments, this method take a screenshot of the entire page.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#capture
path(String)- The path of the screenshot. The format is defined by the file extention. 'image.jpg' will create a JPEG image in the current folder.
clipRect(PlainObject)This optional argument set the position and the size of the screenshot square.
Example:
clipRect = { top: 100, left: 100, width: 500, height: 400 }imgOptions(PlainObject)This optional argument set the two avalaible image options. Such as the format and the quality of the screenshot image.
Example:
imgOptions = { format: 'jpg', quality: 50 }callback(Function(String err))- Function called when finished. When there is no error,
erris null.
Example:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.screenshot('./image.jpg', function(err) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("Screenshot saved!")
nick.exit(0);
});
});
Example with options:
var buster = require('phantombuster').create()
nick.open("https://phantombuster.com/cloud-services", function() {
nick.screenshot('./image.jpg'
, {
top: 90,
left: 190,
width: 900,
height: 360
}
, {
format: 'png',
quality: 100
}
, function(err) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("Screenshot saved!")
buster.saveFolder(function(err) {
if (err) {
console.log(err);
nick.exit(1);
}
nick.exit(0);
});
});
});
nick.sendKeys(selector, keys [, options], callback)
Write keys in an <input>, <textarea> or any DOM element with contenteditable="true" in the current page.
This method is asynchronous and returns nothing. Use the callback to know when it has finished.
More info: http://docs.casperjs.org/en/latest/modules/casper.html#sendkeys
selector(String)- A CSS3 or XPath expression that describes the path to DOM elements.
keys(String)- Keys to send to the editable DOM element.
options(PlainObject)- The three options avalable are:
reset(Boolean): remove the content of the targeted element before sending key presses.keepFocus(Boolean): keep the focus in the editable DOM element after keys have been sent.modifiers(PlainObject): modifier string concatenated with a+(available modifiers arectrl,alt,shift,metaandkeypad).
callback(Function(String err))- Function called when finished. When there is no error,
erris null.
Example:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.sendKeys('#message', "Boo!", function(err) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("Keys sent!")
nick.exit(0);
});
});
Example with optional argument:
nick.open("https://phantombuster.com/cloud-services", function() {
nick.sendKeys('#message', "s", {
reset: false,
keepFocus: true,
modifiers: "ctrl+alt+shift"
}, function(err) {
if (err) {
console.log(err);
nick.exit(1);
}
console.log("Keys sent!")
nick.exit(0);
});
});