-
Notifications
You must be signed in to change notification settings - Fork 1
Prince API Reference
Method to call at the beginning of all programs. Initializer function, that have to be called as early as possible in the main section of the calling program. It ensures that all accesses to mapper and reducer functions are intercepted.
task_return = def run(mapper, reducer, inputs, output, files=None, parameters=None, inputformat='auto', outputformat='auto'):
Run a MapReduce task using Hadoop Streaming.
mapper : method
Mapper method. The prototype has to be map(key, value), and key and value will be filled with the data read from the specified input files. key and value are strings.
reducer : method
Reducer method. The prototype has to be reduce(key, values), and key and values will be filled with the data read from the mapper task. key is a string and values is a list of strings.
inputs : string or list of strings
Paths to the files for the mapper read from on the DFS.
output : string
Name of the file for the reducer to write on the DFS.
files : list of strings (optional)
Names of the files to be included in the path of the mapper and reducer methods. All file used by the program, and imported Python files must be specified here.
parameters : dictionary (optional)
Dictionary of options to be passed to the mapper and reducer methods. For each item in the dictionary, the key is the name of the parameter, and the value is value of the parameter.
inputformat : string (optional)
Input format of the input files. Can be either ‘text’ or ‘auto’, default is ‘auto’.
outputformat : string (optional)
Output format of the output file. Can be either ‘text’ or ‘auto’, default is ‘auto’.
task_return : string
Return of the Hadoop task called.
Method callable from the mapper and reducer methods to get parameters passed by the run() method.
Reads the content of a file on the DFS.
Read the content of files on the DFS. Multiple files can be specified, and it is possible to read only n lines at the beginning or at the end of the file. first and last being exclusive parameters, if both of them are used then only first is used.
filenames : string or list of strings
Files to read from on the DFS.
first : int (optional)
Number of lines to read at the beginning of the file.
last : int (optional)
Number of lines to read at the end of the file
lines : list of strings
Lines of the file(s) on the DFS.
Write text to a file on the DFS
filename : string
File name where to write the text on the DFS.
content : string or list of two-item tuples
If it is a string, the text is just written as it is. If it is a tuple or a list of tuples, each tuple is written as a MapReduce entry (key, value), separated by the default separator.
dfs_write('foo', 'String of text')
dfs_write('foo', (0, 0))
dfs_write('foo', [(0, 1), (1, 1)])
lines : list of strings
Lines of the file(s) on the DFS.
Test if a path exists on the DFS.
NOTE: The current implementation is based on ‘dfs -ls’ and is therefore very slow. This is due to the fact that the implementation of ‘dfs -test -e’ in the current Hadoop version (0.20.1) is buggy and cannot be used properly.
path : string
File name for the file of which the existence on the DFS has to be tested.
exists : boolean
True if the file exists, False otherwise.