Question about building fcp infrastructure.

[ko0tae]

Hi this is YoungTae Ko, ML Researcher.

I'm try to build infrastructure for testing federated learning of Android.

Using terraform Im deploying infrastructure following guide,

regarding this "Coordinator Configuration" there's no description. would you give more guides for this?

"Fields under Coordinator Configuration should be updated to the values provided by the Coordinator operators."

Anyway I deployed servers without changing upper setting. and get stucked in end_to_end_test like this. (i have added --encrypt false option)

# bazel run //java/src/it/java/com/google/ondevicepersonalization/federatedcompute/endtoendtests:end_to_end_test -- --task_management_server https://### --operation CREATE_AND_COMPLETE_TRAINING_TASK --server https://### --encrypt false
INFO: Analyzed target //java/src/it/java/com/google/ondevicepersonalization/federatedcompute/endtoendtests:end_to_end_test (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //java/src/it/java/com/google/ondevicepersonalization/federatedcompute/endtoendtests:end_to_end_test up-to-date:
  bazel-bin/java/src/it/java/com/google/ondevicepersonalization/federatedcompute/endtoendtests/end_to_end_test
  bazel-bin/java/src/it/java/com/google/ondevicepersonalization/federatedcompute/endtoendtests/end_to_end_test.jar
INFO: Elapsed time: 0.307s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/java/src/it/java/com/google/ondevicepersonalization/federatedcompute/endtoendtests/end_to_end_test --task_management_server https://### --operation CREATE_AND_COMPLETE_TRAINING_TASK --server https://### --encrypt false
Creating task at server with basic task:
population_name: "e2e-test-2025-02-18T08:41:36.295518129Z"
total_iteration: 1
min_aggregation_size: 2
max_aggregation_size: 3
max_parallel: 1
min_client_version: "341912000"
max_client_version: "999999999"
info {
  traffic_weight: 1
  training_info {
  }
}

Created task at server, population name: e2e-test-2025-02-18T08:41:36.295518129Z, task id: 1
Uploading metadata for the task...
upload byte[] source to target: gs://fcp-test-m-0/e2e-test-2025-02-18T08:41:36.295518129Z/1/0/s/0/client_only_plan
upload byte[] source to target: gs://fcp-test-m-0/e2e-test-2025-02-18T08:41:36.295518129Z/1/0/s/0/server_phase
upload byte[] source to target: gs://fcp-test-m-0/e2e-test-2025-02-18T08:41:36.295518129Z/1/0/s/0/checkpoint
Task status is CREATED, wait 5 seconds and retry...
After waiting, verify the task status is OPEN...
Processing population e2e-test-2025-02-18T08:41:36.295518129Z, task id 1, iteration 1 ...
Devices start to create task assignment...
Created task assignment, task id 1, assignment id fe88fa0d-0d7c-4339-98df-1ba56d76aabe
reportLocalFailed
Created task assignment, task id 1, assignment id f795dc80-1a29-42d7-8965-b2fac000b698
downloadModel: https://cdn.test-aip-cloud.com/e2e-test-2025-02-18T08:41:36.295518129Z/1/0/d/0/client_checkpoint?Expires=1739869033&KeyName=test-signed-url-key-a&Signature=LUacBieM8xlUZ8Ar-bQKIVfBAiU=
submitResult: /population/e2e-test-2025-02-18T08:41:36.295518129Z/task/1/aggregation/1/task-assignment/f795dc80-1a29-42d7-8965-b2fac000b698
Need 2 contributor(s), 1 contributor(s) are submitted.
Created task assignment, task id 1, assignment id caa43845-7acc-4300-beac-fb2d4312959e
reportLocalFailed
Created task assignment, task id 1, assignment id 1ddff272-6c78-41ba-84e4-b023a043531d
downloadModel: https://cdn.test-aip-cloud.com/e2e-test-2025-02-18T08:41:36.295518129Z/1/0/d/0/client_checkpoint?Expires=1739869037&KeyName=test-signed-url-key-a&Signature=xli2RhVa99N4PHMz1MCmpw5g-NA=
submitResult: /population/e2e-test-2025-02-18T08:41:36.295518129Z/task/1/aggregation/1/task-assignment/1ddff272-6c78-41ba-84e4-b023a043531d
Need 2 contributor(s), 2 contributor(s) are submitted.
Processed population e2e-test-2025-02-18T08:41:36.295518129Z, task id 1, iteration 1 ...
Waiting for task to be complete...
After waiting, the task:
 population_name: "e2e-test-2025-02-18T08:41:36.295518129Z"
task_id: 1
total_iteration: 1
min_aggregation_size: 2
max_aggregation_size: 3
status: OPEN
created_time {
  seconds: 1739868098
  nanos: 192114129
}
max_parallel: 1
min_client_version: "341912000"
max_client_version: "999999999"
client_only_plan_url: "gs://fcp-test-m-0/e2e-test-2025-02-18T08:41:36.295518129Z/1/0/s/0/client_only_plan"
server_phase_url: "gs://fcp-test-m-0/e2e-test-2025-02-18T08:41:36.295518129Z/1/0/s/0/server_phase"
init_checkpoint_url: "gs://fcp-test-m-0/e2e-test-2025-02-18T08:41:36.295518129Z/1/0/s/0/checkpoint"
metrics_url: "gs://fcp-test-m-0/e2e-test-2025-02-18T08:41:36.295518129Z/1/0/s/0/metrics"
info {
  traffic_weight: 1
  training_info {
  }
}

Exception in thread "main" expected: COMPLETED
but was : OPEN
        at com.google.ondevicepersonalization.federatedcompute.endtoendtests.EndToEndTest.completeTask(EndToEndTest.java:397)
        at com.google.ondevicepersonalization.federatedcompute.endtoendtests.EndToEndTest.main(EndToEndTest.java:136)

[ko0tae]

I found stucked log in task scheduler

I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

[andrewkvuong]

odp-federatedcompute is designed to work with trusted coordinators(https://github.com/privacysandbox/coordinator-services-and-shared-libraries) in production. Currently the --encrypt flag on the test client will not pass a full end-to-end run without the coordinators configured as there is another layer of encryption currently enforced between the aggregators and the model updater. We're looking into enabling local testing without coordinators by optionally disabling that layer of encryption, but this would be exclusive to server-side testing as the Android device will always encrypt the output. This should be available in the next release. The output log you provided is a tensorflow notice/warning and should not affect the end to end flow of the server. Could you let us know what you're trying to do and we can figure out how best to unblock you?

[ko0tae]

odp-federatedcompute is designed to work with trusted coordinators(https://github.com/privacysandbox/coordinator-services-and-shared-libraries) in production. Currently the --encrypt flag on the test client will not pass a full end-to-end run without the coordinators configured as there is another layer of encryption currently enforced between the aggregators and the model updater. We're looking into enabling local testing without coordinators by optionally disabling that layer of encryption, but this would be exclusive to server-side testing as the Android device will always encrypt the output. This should be available in the next release. The output log you provided is a tensorflow notice/warning and should not affect the end to end flow of the server. Could you let us know what you're trying to do and we can figure out how best to unblock you?

hi Andrew

I understood that cannot complete end to end testing without coordinators as aggregator and updater has enforced encryption.
And the upper log is just warning of tensorflow and not blocking issue. Thanks for quick reply.

I have interest an operation flow in fcp, managing populations and how to check huge devices's output efficiently in iterations.
So I had tried to skip cordinators and encryptions. actually not understand fully in key services.
I will check more in link from you about that.

Terraform code was helpful to understand and build infra in handy.
Some web console would be helpful for novice. (using “looker” was a bit hurdle.)

• checking each module's status and exec sample end to end testing for understanding.
• create task and browsing task and artifacts (status, plan and checkpoints)

[andrewkvuong]

We've released a new v0.7.0. This adds support to optionally disable encryption for testing and debugging which will allow the end-to-end test client with --encrypt false flag to be able to finish a full round of training. This can be enabled adding the flag --should_encrypt_aggregator_output 'false' to the input params of the aggregator. The updated line should look like env = {"ENCRYPTION_OPTS": "--public_key_service_base_url 'https://publickeyservice.odp.gcp.privacysandboxservices.com/.well-known/odp/v1/public-keys' --should_encrypt_aggregator_output 'false'"},. Then rebuilding and deploying the image.

Thanks for the feedback. We'll look into improving our metrics & dashboards (with more Cloud/GCP native solutions) and reducing the reliance on Looker for easily viewing the system's status.

[ko0tae]

We've released a new v0.7.0. This adds support to optionally disable encryption for testing and debugging which will allow the end-to-end test client with --encrypt false flag to be able to finish a full round of training. This can be enabled adding the flag --should_encrypt_aggregator_output 'false' to the input params of the aggregator. The updated line should look like env = {"ENCRYPTION_OPTS": "--public_key_service_base_url 'https://publickeyservice.odp.gcp.privacysandboxservices.com/.well-known/odp/v1/public-keys' --should_encrypt_aggregator_output 'false'"},. Then rebuilding and deploying the image.

Thanks for the feedback. We'll look into improving our metrics & dashboards (with more Cloud/GCP native solutions) and reducing the reliance on Looker for easily viewing the system's status.

Thanks for response. I will try v0.7.0
If I succeed e2e client test with v0.7.0 then How about on Android Device?

I knew that this project currently preview stage. And I checked almost documents in site about odp fl.
It's not enough to build total integration mobile app and server.
In Android Devices currently having some model(emnist) and sample applications are in for test?
Would you let me know the current status of this project, especially integration with real Android device.
I could not find How and What should be developed and modified by 3rd party side.

I Understood that it's early stage and hope to support detail guide in future.

[andrewkvuong]

Sorry for the delays... We've released some sample apps for Android that can be used for testing with OnDevicePersonalization at https://github.com/privacysandbox/ondevicepersonalization.

We have included a tutorial to run through a federated learning task with the server from this repository and the sample apps in the ondevicepersonalization repository with encryption disabled to avoid the dependency on coordinators. The latest release of this repository (v0.7.1) added a sample Keras model which is used for the tutorial.

[piotrrsawicki]

@andrewkvuong

I'm trying to run the sample Android applications mentioned in the tutorial, but I'm stuck setting up the necessary Google Cloud services for the odp-federatedcompute services.​

Could you share information about the minimum requirements? Specifically, which Google Cloud services need to be enabled (and purchased) and the resources they require (e.g. amount of storage space). I'm asking because even the locally deployed services (https://github.com/privacysandbox/odp-federatedcompute/blob/main/scripts/docker/README.md) still require access to Google Cloud services like Spanner database. This information would help me estimate the costs.