Work generators

A **work generator ** is a C++ program that stages input files and submits jobs. Here are some examples.

Submitting one job

This program submits one job (error-checking is omitted for clarity):

#include "boinc_db.h"
#include "backend_lib.h"

int main() {
    DB_APP app;
    DB_WORKUNIT wu;
    char* wu_template;
    const char* infiles[] = {"infile"};
    char path[1024];

    SCHED_CONFIG config;
    config.parse_file();

    boinc_db.open(config.db_name, config.db_host, config.db_user, config.db_passwd);
    app.lookup("where name='myappname'");

    // write input file in the download directory
    //
    config.download_path("infile", path);
    FILE* f = fopen(path, "w");
    fwrite(f, "random stuff");
    fclose(f);

    read_file_malloc("templates/input_template.xml", wu_template);
    wu.clear();     // zeroes all fields
    strcpy(wu.name, "test_name");
    wu.appid = app.id;
    wu.min_quorum = 2;
    wu.target_nresults = 2;
    wu.max_error_results = 5;
    wu.max_total_results = 5;
    wu.max_success_results = 5;
    wu.rsc_fpops_est = 1e10;
    wu.rsc_fpops_bound = 1e11;
    wu.rsc_memory_bound = 1e8;
    wu.rsc_disk_bound = 1e8;
    wu.delay_bound = 7*86400;
    create_work(
        wu,
        wu_template,
        "templates/output_template.xml",
        "templates/output_template.xml",
        infiles,
        1,
        config
    );
}

The program must be run in the project directory.

Making lots of jobs

If you're making lots of jobs (e.g. to do the various parts of a parallel computation) you'll want the jobs to differ in their input files and/or their command-line arguments.

For example, let's say you want to run a program on ten input files 'file0', 'file1', ..., 'file9'. You might modify the above program with the following code:

    char filename[256];
    char* infiles[1];
    infiles[0] = filename;
    ...
    for (i=0; i<10; i++) {
        sprintf(wu.name, "wu_appName_%d", i);
        sprintf(filename, "file%d", i);
        create_work(
            wu,
            wu_template,
            "templates/output_template.xml",
            "templates/output_template.xml",
            infiles,
            1,
            config
        );
    }

Note that you only need one workunit template file and one result template file.

Now suppose you want to run a program against a single input file, but with ten different command lines, '-flag 0', '-flag 1', ..., '-flag 9'. You might modify the above program as follows:

    char command_line[256];
    ...
    for (i=0; i<10; i++) {
        sprintf(wu.name, "wu_appName_%d", i);
        sprintf(command_line, "-flag %d", i);
        create_work(
            wu,
            wu_template,
            "templates/output_template.xml",
            "templates/output_template.xml",
            infiles,
            1,
            config,
            command_line
        );
    }

Again, you only need one input template file and one output template file.

Sample work generators

Two work generator programs are supplied with BOINC.

Flow-controlled work generator

Some projects have an essentially infinite supply of work. This can use a 'flow-controlled work generator' that tries to maintain a constant number of unsent jobs (typically a few hundred or thousand). It does this by periodically querying the BOINC database to find the number of unsent jobs. An example of this is sched/sample_work_generator.cpp. You can modify this program to suit your application.

make_work

During the testing phase of a project, you may want to copy a given workunit as needed to maintain a constant supply of work. The daemon program

make_work --wu_name name [--wu_name name2 ... ] --cushion N

does this; it creates copies of the given workunits (cyclically) as needed to maintain a supply of at least N unsent results. Other command-line arguments:

--max_wus N

Exit after creating N workunits.

--one_pass

Check if workunits are needed, and if so create them. Then exit.

Note: if you run the file_deleter and/or db_purge, the master workunit or its input files may be deleted (which will cause make_work to fail). To avoid this, give the master workunit a name that contains 'nodelete' as a substring. This causes the file_deleter and db_purge to skip over it.

It may be convenient to have a script that recreates the master workunit. For example:

cp test_workunits/12ja04aa `bin/dir_hier_path 12ja04aa`
bin/create_work --appname setiathome --wu_name sah_nodelete --wu_template templates/setiathome_wu_0 --result_template templates/setiathome_result_0 12ja04aa