v0.5.28

[SanjayVas]

See the corresponding milestone for this release.

Highlights

• RequisitionFetcher cache EventGroups to prevent multiple grpc calls. See [Issue RequisitionFetcher cache EventGroups to prevent multiple grpc calls #3136]
• Support listing EventGroups with data availability which contains a given range. See [Issue Support listing EventGroups with data availability which contains a given range #3078]
• Monitoring: Data Availability Sync. See [Issue Monitoring: Data Availability Sync #3054]
• Monitoring: Event Group Sync. See [Issue Monitoring: Event Group Sync #3053]
• Monitoring: Requisition Fetcher. See [Issue Monitoring: Requisition Fetcher #3052]
• Monitoring: Results Fulfiller. See [Issue Monitoring: Results Fulfiller #3051]
• Monitoring: OpenTelemetry SDK Module. See [Issue Monitoring: OpenTelemetry SDK Module #3049]
• Add model_line to BasicReport. See [Issue Add model_line to BasicReport #3044]
• Update Test Event and US/UK Events to use Version Annotations. See [Issue Update Test Event and US/UK Events to use Version Annotations #2903]
• Set DataProvider data availability per ModelLine from EDP simulator. See [Issue Set DataProvider data availability per ModelLine from EDP simulator #2723]
• EDPA Deployment Guide. See [Issue EDPA Deployment Guide #2528]
• Monitoring: Data Watcher. See [Issue Monitoring: Data Watcher #2526]
• Add Dead letter queue subscriber. See [Issue Add Dead letter queue subscriber #2396]

Issues Addressed

• add-report-results-missing-columns.sql schema update fails on Cloud Spanner. See [Issue add-report-results-missing-columns.sql schema update fails on Cloud Spanner #3196]
• Noise corrector can return solutions with negative frequency values. See [Issue Noise corrector can return solutions with negative frequency values #3185]
• Erroneous large corrections due to rounding error. See [Issue Erroneous large corrections due to rounding error #3159]
• GcsStorageClient incorrectly implements retry behavior. See [Issue GcsStorageClient incorrectly implements retry behavior #3154]
• Population Count needs to always be requested by BasicReport. See [Issue Population Count needs to always be requested by BasicReport #3152]
• ResultsFulfiller process one Pub/Sub message at the time. See [Issue ResultsFulfiller process one Pub/Sub message at the time #3116]
• RequisitionFetcher correctly filters out existing requisitions in the Requisition Metadata. See [Issue RequisitionFetcher correctly filters out existing requisitions in the Requisition Metadata #3135]
• SetMeasurementResults statement running long. See [Issue SetMeasurementResults statement running long #2790]
• Discrepancy in incremental reach figures. See [Issue Discrepancy in incremental reach figures #2880]
• Noise correction cannot find a value for union_measurement.sigma. See [Issue Noise correction cannot find a value for union_measurement.sigma #3126]
• Image pull failures for busybox on GKE 1.33+. See [Issue Image pull failures for busybox on GKE 1.33+ #3067]

Potentially Requiring Action

• The data-watcher Cloud Run Function now supports exporting metrics and traces to Google Cloud Monitoring and Google Cloud Trace, respectively. This can be enabled by including the following in the extra_env_vars field of the appropriate entry in the cloud_function_configs variable of the edp-aggregator Terraform module: OTEL_SERVICE_NAME=edpa.data_watcher,OTEL_METRICS_EXPORTER=google_cloud_monitoring,OTEL_TRACES_EXPORTER=google_cloud_trace,OTEL_LOGS_EXPORTER=logging,OTEL_METRIC_EXPORT_INTERVAL=60000. See [PR feat: enable telemetry for data watcher #3184]
• Halo-managed environments have reduced the frequency of the Requisition Fetcher component EDPA to every 15 minutes to avoid problems from having more than one instance running concurrently. Operators may wish to do the same. This can be configured using the requisition_fetcher_scheduler_config variable of the edp-aggregator Terraform module. See [PR fix(edpa): Reduce fetcher frequency to every 15 minutes to avoid race conditions #3175]
• Operators may wish to increase the max heap size and container memory of the Duchy Requisition fulfillment server to handle a larger number of simultaneous fulfillments. See [PR fix: increase duchy memory and nodes #3177]
• add metadata for resultsfulfiller TEE

  cross-media-measurement/src/main/terraform/gcloud/modules/mig/main.tf

  Lines 19 to 37 in 8d3f8b7

  	metadata_map = merge(
  	{
  	"tee-signed-image-repos" = var.edpa_tee_signed_image_repo
  	"tee-image-reference" = var.docker_image
  	"tee-cmd" = jsonencode(var.tee_cmd),
  	"tee-env-OTEL_SERVICE_NAME" = "edpa.results_fulfiller",
  	"tee-env-OTEL_METRICS_EXPORTER" = "google_cloud_monitoring",
  	"tee-env-OTEL_TRACES_EXPORTER" = "google_cloud_trace",
  	"tee-env-OTEL_LOGS_EXPORTER" = "logging",
  	"tee-env-OTEL_SERVICE_NAME" = "edpa.results_fulfiller",
  	"tee-env-OTEL_EXPORTER_GOOGLE_CLOUD_PROJECT_ID" = data.google_project.project.project_id
  	"tee-env-OTEL_METRIC_EXPORT_INTERVAL" = "60000"
  	},
  	var.config_storage_bucket == null ? {} : {
  	"tee-env-EDPA_CONFIG_STORAGE_BUCKET" = "gs://${var.config_storage_bucket}"
  	},
  	var.java_tool_options == "" ? {} : {
  	"tee-env-JAVA_TOOL_OPTIONS" = var.java_tool_options
  	}

  . See [PR fix!: add missing telemetry deps and variables #3167]
• requires environment vars OTEL_SERVICE_NAME=edpa.requisition_fetcher,OTEL_METRICS_EXPORTER=google_cloud_monitoring,OTEL_TRACES_EXPORTER=google_cloud_trace,OTEL_LOGS_EXPORTER=logging,OTEL_METRIC_EXPORT_INTERVAL=60000. See [Issue Monitoring: Requisition Fetcher #3052]
• requires additional metadata for the TEE https://github.com/world-federation-of-advertisers/cross-media-measurement/blob/main/src/main/terraform/gcloud/modules/mig/main.tf. See [PR feat: resultsfulfiller telemetry #3081]
• requires env vars OTEL_SERVICE_NAME=edpa.event_group_sync,OTEL_METRICS_EXPORTER=google_cloud_monitoring,OTEL_TRACES_EXPORTER=google_cloud_trace,OTEL_LOGS_EXPORTER=logging,OTEL_METRIC_EXPORT_INTERVAL=60000. See [PR feat!: event group sync telemetry #3021]
• OTEL_SERVICE_NAME=edpa.data_availability,OTEL_METRICS_EXPORTER=google_cloud_monitoring,OTEL_TRACES_EXPORTER=google_cloud_trace,OTEL_LOGS_EXPORTER=logging,OTEL_METRIC_EXPORT_INTERVAL=60000. See [PR feat!: data availability telemetry #3020]

What's Changed

• fix: Reference busybox:1.27 for containerd 2.0 compatibility by @SanjayVas in fix: Reference busybox:1.27 for containerd 2.0 compatibility #3068
• feat: Generate OpenAPI documents for Reporting API services by @SanjayVas in feat: Generate OpenAPI documents for Reporting API services #1788
• ci: Add release draft automation by @laureanobrs in ci: Add release draft automation #2853
• ci: Add option to disable cache when running coverage by @laureanobrs in ci: Add option to disable cache when running coverage #3012
• feat: Enforce that DataProvider availability is within ModelLine active interval by @SanjayVas in feat: Enforce that DataProvider availability is within ModelLine active interval #3032
• feat: Support enumerating valid ModelLines across providers/suites by @SanjayVas in feat: Support enumerating valid ModelLines across providers/suites #3033
• test: Add Basic Report creation to correctness test by @tristanvuong2021 in test: Add Basic Report creation to correctness test #3007
• feat: Support active interval filter and wildcards for ListModelLines by @SanjayVas in feat: Support active interval filter and wildcards for ListModelLines #3034
• perf: Increase container memory for Reporting internal API server by @SanjayVas in perf: Increase container memory for Reporting internal API server #3082
• ci: Use rsync to speed up large directory deletion by @SanjayVas in ci: Use rsync to speed up large directory deletion #3083
• build: Add py_image macro for Python container images by @SanjayVas in build: Add py_image macro for Python container images #3063
• feat: Update Kingdom to support TrusTEE Measurement creation. by @renjiezh in feat: Update Kingdom to support TrusTEE Measurement creation. #3072
• feat: Add model_line to BasicReport by @tristanvuong2021 in feat: Add model_line to BasicReport #3045
• feat: Replace DataProvider availability intervals in EDP simulator by @SanjayVas in feat: Replace DataProvider availability intervals in EDP simulator #3085
• feat: Support listing EventGroups by range contained by data availability by @SanjayVas in feat: Support listing EventGroups by range contained by data availability #3093
• feat: add roles to cloud functions for opentelemetry by @Marco-Premier in feat: add roles to cloud functions for opentelemetry #3094
• fix: add grpc shutdown hook in the RequisitionFetcher by @Marco-Premier in fix: add grpc shutdown hook in the RequisitionFetcher #3097
• allow dynamic event group paths by @stevenwarejones in allow dynamic event group paths #3091
• feat: Update ReportResults service to have clearer semantics by @SanjayVas in feat: Update ReportResults service to have clearer semantics #3077
• ci: add OpenSSF Scorecard checks by @TNATALI in ci: add OpenSSF Scorecard checks #3098
• feat: Transform ReportResults into BasicReport ResultGroups by @tristanvuong2021 in feat: Transform ReportResults into BasicReport ResultGroups #3070
• fix(report-processor): handle report with non-video EDP. by @ple13 in fix(report-processor): handle report with non-video EDP. #3117
• feat: Implement ImpressionMetadata batch delete by @roaminggypsy in feat: Implement ImpressionMetadata batch delete #3076
• perf: Refactor Reporting Queries to have Values clauses in Materialized CTEs by @tristanvuong2021 in perf: Refactor Reporting Queries to have Values clauses in Materialized CTEs #3120
• feat: Implement impression metadata batch create by @roaminggypsy in feat: Implement impression metadata batch create #3087
• feat: terraform for dead letter queue by @georgi in feat: terraform for dead letter queue #3092
• Fix: bug in the RequisitionFetcher where existing requisition metadata with state QUEUED or PROCESSING where treated as new by @Marco-Premier in Fix: bug in the RequisitionFetcher where existing requisition metadata with state QUEUED or PROCESSING where treated as new #3121
• refactor: requisition grouper to cache event groups by @Marco-Premier in refactor: requisition grouper to cache event groups #3125
• refactor: update logic to signal requisition metadata by @Marco-Premier in refactor: update logic to signal requisition metadata  #3134
• feat: Use integers for histogram bins and add fingerprint fields by @SanjayVas in feat: Use integers for histogram bins and add fingerprint fields #3129
• ci: Restrict write permissions in workflows by @laureanobrs in ci: Restrict write permissions in workflows #3142
• feat: implement internal RequisitionMetadata batch create by @roaminggypsy in feat: implement internal RequisitionMetadata batch create #3132
• perf: Increase heap size of Requisition Fulfillment server by @SanjayVas in perf: Increase heap size of Requisition Fulfillment server #2981
• build: update dependencies by @georgi in build: update dependencies #3141
• Allow additional env vars for Resultsfulfiller by @georgi in Allow additional env vars for Resultsfulfiller #3137
• fix: limit fulfillment parallism to 15 by @georgi in fix: limit fulfillment parallism to 15 #3144
• fix: reduce memory usage by @georgi in fix: reduce memory usage #3140
• feat: set workers based on cpu count by @georgi in feat: set workers based on cpu count #3139
• feat: add telemetry module by @georgi in feat: add telemetry module #3066
• refactor: use InProcessServerMethods from common-jvm by @georgi in refactor: use InProcessServerMethods from common-jvm #2488
• feat: Add dead letter queue listener for failed work items by @georgi in feat: Add dead letter queue listener for failed work items #2351
• feat: data watcher telemetry by @georgi in feat: data watcher telemetry #3019
• feat!: data availability telemetry by @georgi in feat!: data availability telemetry #3020
• feat!: event group sync telemetry by @georgi in feat!: event group sync telemetry #3021
• feat!: resultsfulfiller telemetry by @georgi in feat: resultsfulfiller telemetry #3081
• feat: BasicReport always requests Population Count Metric by @tristanvuong2021 in feat: BasicReport always requests Population Count Metric #3151
• build: Update common-jvm dep for changes to GcsStorageClient by @SanjayVas in build: Update common-jvm dep for changes to GcsStorageClient #3155
• feat: Upgrade Google Terraform provider for edp-aggregator module to 6.50.0 by @SanjayVas in feat: Upgrade Google Terraform provider for edp-aggregator module to 6.50.0 #3153
• Add 'needs triage' label to Issue templates by @laureanobrs in Add 'needs triage' label to Issue templates #3145
• feat!: requisition fetcher telemetry by @georgi in feat!: requisition fetcher telemetry #3069
• perf: Increase reporting public server heap size to 64M. by @renjiezh in perf: Increase reporting public server heap size to 64M. #3157
• feat: Implement requisition metadata batch create public API by @roaminggypsy in feat: Implement requisition metadata batch create public API #3148
• refactor: synthetic data generation by @Marco-Premier in refactor: synthetic data generation #3074
• fix: Fix Herald timeout error in release-test by @renjiezh in fix: Fix Herald timeout error in release-test #3163
• feat(report-processor): convert report results to report summary v2. by @ple13 in feat(report-processor): convert report results to report summary v2. #3103
• fix: edpa - calculate correct frequency histogram when reach is zero by @stevenwarejones in fix: edpa - calculate correct frequency histogram when reach is zero #3166
• feat: Update Duchy to support TrusTEE requisition fulfillment by @renjiezh in feat: Update Duchy to support TrusTEE requisition fulfillment #3086
• fix!: add missing telemetry deps and variables by @georgi in fix!: add missing telemetry deps and variables #3167
• fix: add missing terraform var by @georgi in fix: add missing terraform var #3174
• fix: increase duchy memory and nodes by @georgi in fix: increase duchy memory and nodes #3177
• fix(edpa): Reduce fetcher frequency to every 15 minutes to avoid race conditions by @georgi in fix(edpa): Reduce fetcher frequency to every 15 minutes to avoid race conditions #3175
• ci: Sign artifacts attached to releases by @laureanobrs in ci: Sign artifacts attached to releases #3176
• feat: dont fail for requisitions already in a terminal state by @stevenwarejones in feat: dont fail for requisitions already in a terminal state #3161
• feat: add withdrawn state to requisition metadata store by @stevenwarejones in feat: add withdrawn state to requisition metadata store #3182
• feat: EDPA only process unfulfilled requistions by @stevenwarejones in feat: EDPA only process unfulfilled requistions #3165
• fix(report-processor): use double type for frequency histogram value. by @ple13 in fix(report-processor): use double type for frequency histogram value. #3180
• docs: add edpa deployment guide by @Marco-Premier in docs: add edpa deployment guide #3115
• fix: use hyperdisk to enable c4d instance types by @georgi in fix: use hyperdisk to enable c4d instance types #3181
• feat: enable telemetry for data watcher by @georgi in feat: enable telemetry for data watcher #3184
• fix: use pull subscriber to extend ack deadline reliably by @georgi in fix: use pull subscriber to extend ack deadline reliably #3183
• refactor: Extract ReportingSetResult into its own entity with an ID by @SanjayVas in refactor: Extract ReportingSetResult into its own entity with an ID #3173
• feat: Add fingerprinting functions for ReportResult fields by @SanjayVas in feat: Add fingerprinting functions for ReportResult fields #3102
• fix(report-processor): invalidate solution with negative results. by @ple13 in fix(report-processor): invalidate solution with negative results. #3191
• feat: cloud logging for resultsfulfiller by @georgi in feat: cloud logging for resultsfulfiller #3187
• fix: Update proto bundle for ReportResults schema by @SanjayVas in fix: Update proto bundle for ReportResults schema #3195

New Contributors

• @TNATALI made their first contribution in ci: add OpenSSF Scorecard checks #3098

Full Changelog: v0.5.27...v0.5.28

---

This discussion was created from the release v0.5.28.