Skip to content

generating_udf_files.md

Latest commit

 

History

History
107 lines (70 loc) · 4.11 KB

generating_udf_files.md

File metadata and controls

107 lines (70 loc) · 4.11 KB

FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the blog post

Generating UDF code configs for the server

The server starts with a simple pass-through implementation at public/udf/constants.h.

UDF configurations can be updated as the server is running using delta/snapshot files as per the data loading guide.

  • To override an existing UDF, the delta/snapshot file must have a DataRecord with a UserDefinedFunctionsConfig.

  • Similar to a KeyValueMutationRecord, the UserDefinedFunctionsConfig has a logical_commit_time. The UDF will only be updated for configs with a higher logical_commit_time than the existing one. The minimum logical_commit_time is 1.

Note that there is a maximum code snippet size limit of 500MB.

Please read through the UDF explainer for more requirements and APIs.

Steps for including the UDF delta file

1. Write your UDF

Option A. Write a custom UDF

Write the UDF according to the API in the UDF explainer.

Note that the UDF should be in JavaScript (and optionally JavaScript + inline WASM).

Option B. Use the reference UDF

We provide a simple reference implementation:

  • The implementation ignores part of the request, e.g. the metadata field.
  • For each keyGroup in the request, it calls getValues(keyGroup) to retrieve the keys from the internal cache and returns the key-value pairs in its response.

2. Generate a UDF delta file

Option 1. Using provided UDF tools

Tools to generate UDF delta files and test them are in the tools/udf directory.

  1. Build the tools binaries docker image:

    -$ builders/tools/bazel-debian run //production/packaging/tools:copy_to_dist --config=local_instance --config=local_platform

  2. Load the tools binaries docker image:

    -$ docker load -i dist/tools_binaries_docker_image.tar

  3. Generate a UDF delta file using the udf_delta_file_generator executable.

    Flags:

    • --udf_handler_name — UDF handler name/entry point
    • --output_dir — output directory for the generated delta file
    • --udf_file_path — path to the UDF JavaScript file
    • --logical_commit_time — logical commit time of the UDF config
    • --code_snippet_version — UDF version. For telemetry, should be > 1.
    • --wasm_binary_file_path — WASM binary to load as a module. This is the recommended way to load WASM alongside a Javscript UDF.
    • --data_loading_file_format — Data loading file format. Either riegeli or avro.

    Example:

    -$ export DATA_DIR=<data_dir>
docker run -it --rm \
  --volume=$DATA_DIR:$DATA_DIR \
  --user $(id -u ${USER}):$(id -g ${USER}) \
  --entrypoint=/tools/udf/udf_delta_file_generator \
  bazel/production/packaging/tools:tools_binaries_docker_image \
  --output_dir="$DATA_DIR" \
  --udf_file_path="$DATA_DIR/udf.js"

Option 2. Generating your own delta file

You can use other options to generate delta files, e.g. using the data_cli tool.

The delta file must have a DataRecord with a UserDefinedFunctionsConfig as its record.

Option 3. Using sample UDF configurations

A sample UDF JavaScript file is located under the tools/udf/sample_udf directory.

3. Test the UDF delta file

Test the generated UDF delta file using the UDF Delta file tester.

4. Provide a UDF to the server

Generally, the delta/snapshot file just needs to be included in delta storage/bucket. Follow the different deployment guides on how to configure your delta file storage.

The UDF will be executed for the V2 API.