-
Notifications
You must be signed in to change notification settings - Fork 1
Prince API Reference
Method to call at the beginning of all programs. Initializer function, that have to be called as early as possible in the main section of the calling program. It ensures that all accesses to mapper and reducer functions are intercepted.
task_return = run(mapper, reducer, inputs, output, files=None, parameters=None, inputformat='auto', outputformat='auto'):
Run a MapReduce task using Hadoop Streaming.
mapper : method
Mapper method. The prototype has to be map(key, value), and key and value will be filled with the data read from the specified input files. key and value are strings.
reducer : method
Reducer method. The prototype has to be reduce(key, values), and key and values will be filled with the data read from the mapper task. key is a string and values is a list of strings.
inputs : string or list of strings
Paths to the files for the mapper read from on the DFS.
output : string
Name of the file for the reducer to write on the DFS.
files : list of strings (optional)
Names of the files to be included in the path of the mapper and reducer methods. All file used by the program, and imported Python files must be specified here.
parameters : dictionary (optional)
Dictionary of options to be passed to the mapper and reducer methods. For each item in the dictionary, the key is the name of the parameter, and the value is value of the parameter.
inputformat : string (optional)
Input format of the input files. Can be either ‘text’ or ‘auto’, default is ‘auto’.
outputformat : string (optional)
Output format of the output file. Can be either ‘text’ or ‘auto’, default is ‘auto’.
task_return : string
Return of the Hadoop task called.
Return the parameters passed to the mapper and reducer tasks through the run() method. When used, it has to be called in the mapper and reducer methods.
'param1'[, 'param2', ...] : strings
Variable number of strings, each of them being the name of a parameter to get. See the Examples section below.
(param1_value[, param2_value, ...]) : list of strings
The values of the parameters of which the names have been given in parameter. A simple string is returned in case only one name is given, and a tuple of strings if multiple name are given. In case a parameter is not found, the value None is returned.
param1_value = get_parameters('param1')
(param1_value, param2_value) = get_parameters('param1', 'param2')
Read the content of files on the DFS. Multiple files can be specified, and it is possible to read only n lines at the beginning or at the end of the file. first and last being exclusive parameters, if both of them are used then only first is used.
filenames : string or list of strings
Files to read from on the DFS.
first : int (optional)
Number of lines to read at the beginning of the file.
last : int (optional)
Number of lines to read at the end of the file
lines : list of strings
Lines of the file(s) on the DFS.
Write text to a file on the DFS
filename : string
File name where to write the text on the DFS.
content : string or list of two-item tuples
If it is a string, the text is just written as it is. If it is a tuple or a list of tuples, each tuple is written as a MapReduce entry (key, value), separated by the default separator.
lines : list of strings
Lines of the file(s) on the DFS.
dfs.write('foo', 'String of text')
dfs.write('foo', (0, 0))
dfs.write('foo', [(0, 1), (1, 1)])
Test if a path exists on the DFS.
NOTE: The current implementation is based on ‘dfs -ls’ and is therefore very slow. This is due to the fact that the implementation of ‘dfs -test -e’ in the current Hadoop version (0.20.1) is buggy and cannot be used properly.
path : string
Path of which the existence on the DFS has to be tested.
exists : boolean
True if the file exists, False otherwise.
key : string
Numerical key provided by Prince. Prince ensures that this key is unique to each mapper process by incrementing its own internal key every time an item is returned or yielded by the mapper method. It is given as a string to be coherent with the rest of the API. Therefore, to get the integer value it contains, a conversion with int() is necessary: num_key = int(key).
value : string
Value of the item as it appear in the input data. If input data is a file, then value is a line of this file.
Mappers need to either return a tuple (key, value) or a sequence of tuples <(key, value), …>, or to yield tuples of the form (key, value). The most common usage is to use yield, but depending on the situation, return can be useful too.
key : string
Key read from the mapper.
values : generator of strings
Produces all the values that map to the given key from the preceding map task. Beware here that the type of values is a generator producing strings. This means that if you need to know how many values there are, len(values) won’t work, as you need to read the content of the generator first: nb_items = len([v for v in values]).
Reducers need to either return or yield a tuple (key, value). The most common usage is to use yield, but depending on the situation, return can be useful too.