Import

Wiki ▸ API Reference ▸ Import

Data Import

# dl.load(options[, callback])

Load data from a URL or file. This method does not perform any parsing, it simply returns the loaded data as either a Buffer or String instance, depending on the execution environment. To also parse the data, see the read method or any of the format-specific data loading helpers.

The options hash may contain the following entries:

• url: The url to load. On the server-side, local files can be specified using the file:// protocol prefix.
• baseURL: A base URL prefix to append to the provided url value. This is useful to include in applications that load multiple data sets from the same domain.
• domainWhiteList: An array of allowed domains from which to load data. If provided, requests to any domains not included in the white list will be denied. This is useful for enforcing security restrictions.
• headers: An object of key-values indicating custom request headers, used only when loading via HTTP. These headers will be combined with (and, in the case of shared keys, overwrite) the headers specified in dl.load.headers.
• file: (server-side only) If url is omitted, specifies a local file to load.

If callback is provided, loading will be done asynchronously. Results or errors will be returned by invoking the callback. If callback is not provided, loading will be performed synchronously, and this method will directly return the loaded data.

Synchronous execution is provided to support rapid exploration and console experimentation. However, it should not be used in production deployments, as blocking the UI while waiting for a file to load can severely damage performance and the user experience.

// load synchronously
var data = dl.load({url: 'data.json'});

// load asynchronously
dl.load({url: 'data.json'}, function(err, data) {
  if (err) {
    // handle error...
  } else {
    // do something with data!
  }
});

# dl.load.headers

An object containing custom request headers to include with all HTTP requests. Defaults to an empty object. One can either set properties on this object directly or replace it with another object.

dl.load.headers['User-Agent'] = 'datalib'; // add header
dl.load.headers = {'User-Agent': 'datalib'}; // replace headers

# dl.load.useXHR

Boolean flag indicating if XMLHttpRequest (XHR) should be used for file loading. By default, XHR is used in the browser environment, otherwise standard server-side modules for file and url loading are used. This flag is automatically set at load time by looking for the presence of XMLHttpRequest in the global scope. The flag can be directly set as well (e.g., to force use of a shim XHR module for testing purposes), but this should not be done unless you know what you are doing!

# dl.load.sanitizeURL

Checks and potentially modifies a URL based on the given load configuration. This method is used internally by dl.load to ensure the URL is valid and to add additional protocol and hostname information, if needed. This method accepts as input the same parameter object accepted by dl.load and returns either a URL string or null if the URL is invalid or disallowed. If desired, clients can overwrite this method to perform their own URL sanitization procedure.

# dl.read(data[, format])

Parse loaded data according to the given format. The data argument should be either a String or Buffer instance (for example, the result of calling dl.load).

The format hash contents may depend on the data format (see below). Common options include:

• type: The data format type, for example json, csv, tsv, topojson or treejson.
• property: For JSON types, specifies a property of the loaded JSON to reference. This is useful if a loaded JSON file contains multiple data sets and one would like to parse data under a specific property.
• parse: When set to 'auto' (the default), the method will perform type inference (using dl.read.types) to guess the data types of each variable. Alternatively, users can provide specific parsing rules through an input hash of variable name, data type pairs (for example: {'timestamp': 'date', 'price': 'number'}). The valid data type options are 'boolean', 'integer', 'number', 'date', and 'string'.

The date data type takes an optional format ('date:fmt'). It uses d3-time-format's time formatting utilities and accepts a valid D3 time format pattern. Note that the format string must be quoted (e.g., '%A').

// read loaded csv data, automatically infer value types
var csv_data = dl.load({url: 'data/stocks.csv'});
var data = dl.read(csv_data, {type: 'csv', parse: 'auto'});

// read loaded csv data, using provided value types
var csv_data = dl.load({url: 'data/stocks.csv'});
var data = dl.read(csv_data, {type: 'csv', parse: {'date': 'date', 'price': 'number'}});

// read loaded topojson data, extract mesh of countries
var topojson = dl.load({url: 'data/world-110m.json'});
var data = dl.read(topojson, {type: 'topojson', mesh: 'countries'});

# dl.type(values[, accessor])

Perform a type lookup for a set of input values. This method will look for the first non-null element in the input array and return the type ('boolean', 'number', 'date', or 'string') of that value. This method does not perform type inference: it will not attempt to parse string values (see dl.type.infer instead).

dl.type([null, 'a']); // 'string'
dl.type([1, 2, 3]);  // 'number'

# dl.type.all(data[, fields])

Perform type lookups for an input data table. Returns an object whose keys are property names and whose values are the observed types. An optional fields array may be specified, indicating which properties should be subject to type lookup; if unspecified, lookup will be performed for all property names found on the first object in the input data array. Internally, this method calls dl.type for each field.

var table = [{a:true, b:1.2, c:"x"}, {a:false, b:3.7, c:"y"}];
var types = dl.type.all(table); // {a: 'boolean', b: 'number', c: 'string'}

# dl.type.infer(values[, accessor, ignore])

Perform type inference for a set of input values, recognizing both primitive JavaScript data types and string-encoded data. An optional accessor function may be specified, which is equivalent to calling values.map(accessor) before performing type inference. The optional ignore argument takes a RegExp or object with a test method: if test returns true for a value, that value will be ignored by the inference procedure. Returns a string-valued type name of (in precedence order) 'boolean', 'integer', 'number', 'date', or 'string'. Type inference is performed by testing the type of each value, ignoring null, undefined or NaN values. The most specific type that passes all type checks is then returned. The integer type assumes 32-bit integers, larger integer values will be treated as more general numbers. String values of numbers that include decimal points ('3.0') will parse as integers if the numerals after the decimal point are always zero. The date type applies to date-formatted strings; this method will consider UNIX timestamps to be number-typed values.

dl.type.infer([1, 2, 3]); // 'integer'
dl.type.infer(['1', '2', '3']); // 'integer'
dl.type.infer([1.0, 2.3, 3.2]); // 'number'
dl.type.infer(['a', 'b', null, 'c']); // 'string'
dl.type.infer(['true', 'false']); // 'boolean'
dl.type.infer(['1/1/2010', '5/5/2015']); // 'date'
dl.type.infer([12, true, 'hello world']); // 'string'

# dl.type.inferAll(data[, fields, ignore])

Perform type inference for an input data table. Returns an object whose keys are property names and whose values are the inferred types. An optional fields array may be specified, indicating which properties should be subject to type inference; if unspecified, type inference will be performed for all property names found on the first object in the input data array. The optional ignore argument takes a RegExp or object with a test method: if test returns true for a value, that value will be ignored by the inference procedure. Internally, this method calls dl.type.infer for each field.

var table = [{a:1, b:1.2, c:"x"}, {a:4, b:3.7, c:"y"}];
var types = dl.type.inferAll(table); // {a: 'integer', b: 'number', c: 'string'}

# dl.json(options[, format, callback])

Convenience method to load and parse a JSON data file. The options argument can be either a URL string or an options hash, as accepted by dl.load. If provided, the optional format argument is passed as a parameter to dl.read. If callback is provided, loading will be done asynchronously. Results or errors will be returned by invoking the callback. If callback is not provided, loading will be performed synchronously, and this method will directly return the loaded and parsed data.

// import json data from relative URL
var data = dl.json('data/table.json');

# dl.csv(options[, format, callback])

Convenience method to load and parse a CSV (comma-separated values) data file. The options argument can be either a URL string or an options hash, as accepted by dl.load. If provided, the optional format argument is passed as a parameter to dl.read. If callback is provided, loading will be done asynchronously. Results or errors will be returned by invoking the callback. If callback is not provided, loading will be performed synchronously, and this method will directly return the loaded and parsed data.

// import csv data, automatically infer value types
var data = dl.csv('data/stocks.csv');

// import csv data, using provided value types
var data = dl.csv('data/stocks.csv', {parse: {'date': 'date', 'price': 'number'}});

# dl.tsv(options[, format, callback])

Convenience method to load and parse a TSV (tab-separated values) data file. The options argument can be either a URL string or an options hash, as accepted by dl.load. If provided, the optional format argument is passed as a parameter to dl.read. If callback is provided, loading will be done asynchronously. Results or errors will be returned by invoking the callback. If callback is not provided, loading will be performed synchronously, and this method will directly return the loaded and parsed data.

// import tsv data, automatically infer value types
var data = dl.tsv('data/stocks.tsv');

// import tsv data, using provided value types
var data = dl.tsv('data/stocks.tsv', {parse: {'date': 'date', 'price': 'number'}});

# dl.dsv(options, format[, callback])

Convenience method to load and parse delimited text data with a custom delimiter. The options argument can be either a URL string or an options hash, as accepted by dl.load. The format argument is passed as a parameter to dl.read, here it is required and must have a property delimiter with a string value indicating the field separator in the text. If callback is provided, loading will be done asynchronously. Results or errors will be returned by invoking the callback. If callback is not provided, loading will be performed synchronously, and this method will directly return the loaded and parsed data.

// import pipe-delimited data, automatically infer value types
var data = dl.dsv('data/stocks.txt', {delimiter: '|'});

// import pipe-delimited data, using provided value types
var data = dl.dsv('data/stocks.txt', {
  delimiter: '|',
  parse: {'date': 'date', 'price': 'number'}
});

# dl.topojson(options[, format, callback])

Convenience method to load and parse a TopoJSON data file. The options argument can be either a URL string or an options hash, as accepted by dl.load. If provided, the optional format argument is passed as a parameter to dl.read. If callback is provided, loading will be done asynchronously. Results or errors will be returned by invoking the callback. If callback is not provided, loading will be performed synchronously, and this method will directly return the loaded and parsed data.

Additional format options for TopoJSON data:

• feature: The name of the feature set to extract from the TopoJSON file.
• mesh: The name of the mesh to extract from the TopoJSON file. One (and only one) of either feature or mesh are required for successful TopoJSON parsing. Both feature and mesh names must defined within the top-level TopoJSON objects property.

// import topojson data, extract mesh of countries
var data = dl.topojson('data/world-110m.json', {mesh: 'countries'});

# dl.treejson(options[, format, callback])

Convenience method to load and parse a JSON file of hierarchical (tree) data. The options argument can be either a URL string or an options hash, as accepted by dl.load. If provided, the optional format argument is passed as a parameter to dl.read. If callback is provided, loading will be done asynchronously. Results or errors will be returned by invoking the callback. If callback is not provided, loading will be performed synchronously, and this method will directly return the loaded and parsed data.

Additional format options for tree JSON data:

• children: For each object (node), the name of the property containing an array of child objects (nodes). If not specified, the property name 'children' is used by default.

// import treejson data, with explicit indication of children property
var tree = dl.treejson('data/flare.json', {children: 'children'});