v0.5.30

See the corresponding milestone for this release.

Highlights

• The CreateBasicReport method is now available for use on the Reporting public API via gRPC and HTTP JSON. (MC API Phase II)
  • To enable, make the following changes in Kubernetes:
    • Ensure that the Population Requisition Fulfiller is running in the Kingdom cluster. This has been available since v0.5.26. See the deployment guide.
      • This is necessary for the new required --pdp-name option on the Reporting public API server.
    • Ensure that the MeasurementConsumerConfig in the mc-config Secret has offline_principal specified for each MeasurementConsumer.
      • This is used as the auth Principal for BasicReport processing and must have at least the reporting.reports.get and reporting.metrics.get permissions.
    • Specify the --basic-reports-enabled option on the Reporting internal API server. This will make the following options required. See the deployment guide for more information on configuration files.
      • --impression-qualification-filter-config-file. Note that this must contain entries with external IDs ami and mrc.
      • --event-message-descriptor-set
      • --event-message-type-url
      • --spanner-project
      • --spanner-instance
      • --spanner-database
    • Add a CronJob to run the BasicReports results processor and report results post-processor. NetworkPolicy resources may need to be updated to allow this to call the Kingdom and the Reporting internal API server. See report-result-post-processor-cronjob in the dev configuration for an example.
    • Configure which ImpressionQualificationFilters (IQFs) are always applied using the --base-impression-qualification-filter option on the Reporting public API server. For example, --base-impression-qualification-filter=impressionQualificationFilters/ami --base-impression-qualification-filter=impressionQualificationFilters/mrc
• Support uncapped impressions in EDPA when frequency_cap_per_user is -1. See [Issue #3368]
• Run ResultsFulfiller on c4d machine in Halo-managed environments. See [Issue #3366]
• Validate edp impression path in DataAvailabilitySync. See [Issue #3360]
• Allow multiple MCs to share the same EventGroupReferenceId in EventGroupSync.. See [Issue #3321]
• Basic Report Custom Filters Should Not Overlap Named Filters. See [Issue #3285]
• Remove services from gRPC gateway that need not be exposed for MC API Phase II. See [Issue #3278]
• Restrict usage of DEV ModelLine type in new Reports. See [Issue #3221]
• Support configuration for default IQFs for CreateBasicReport. See [Issue #3216]
• Complete End-to-End InProcess test for Public BasicReport Creation. See [Issue #3209]
• Complete End-to-End Correctness Test for Public BasicReport Creation. See [Issue #3208]

Issues Addressed

• NOT Null cannot be applied to ModelLineSystemSpecified when NULL values exist. See [Issue #3389]
• ReportResultPostProcessor CronJob Network Policy is setup incorrectly. See [Issue #3376]
• ResourceSetup does not set offline_principal in MeasurementConsumerConfig. See [Issue #3377]
• Retrieving a ReportingSetResult fails when event filter is empty. See [Issue #3379]
• ReportingWindowResult cannot be retrieved when nonCumulative Start doesn't exist. See [Issue #3380]
• BasicReport ResultGroups kPlusReach and percentKPlusReach have more values than requested in ResultGroupSpecs. See [Issue #3381]
• BasicReportsReportsJob incorrectly sets VennDiagramRegionType to PRIMITIVE instead of UNION. See [Issue #3382]
• CreateBasicReport does not persist ImpressionQualificationFilterSpecs used at time of creation. See [Issue #3383]
• CreateBasicReport creates MetricCalculationSpec Groupings using the unset Enum value. See [Issue #3384]
• ModelLineSystemSpecified column in BasicReports table is allowed to be null. See [Issue #3385]
• BasicReport Cumulative Weekly only supports reach and percent_reach. See [Issue #3386]
• DeleteRole Fails when PolicyBindings associated with the Role exists. See [Issue #3373]
• ImpressionQualificationFilter caching in BasicReports Service may cause caching of wrong ImpressionQualificationFilters. See [Issue #3325]
• EDPA Cloud Run Functions may not export metrics before completion. See [Issue #3367]
• Logs not exported for EDPA processes in TEE. See [Issue #3357]
• ImpressionQualificationFilterConfig allows specs with no filters. See [Issue #3315]
• Frequent Reporting Postgres DB locking from GetReport. See [Issue #3337]
• Fix code scanning alert - libpng: LIBPNG buffer overflow / CVE-2025-64720. See [Issue #3236]
• Fix code scanning alert - libpng: LIBPNG heap buffer overflow / CVE-2025-65018. See [Issue #3237]
• Fix code scanning alert - libpng: LIBPNG out-of-bounds read in png_image_read_composite / CVE-2025-66293. See [Issue #3333]
• Incompatible protobuf gencode/runtime versions in report result processor. See [Issue #3327]
• ImpressionQualificationFilterConfig is not Validated against the Event Descriptor. See [Issue #3276]
• CreateBasicReport does not request Population correctly. See [Issue #3302]
• Population Metric does not create a Population Measurement. See [Issue #3290]
• update-edp-aggregator-services workflow sometimes fails with deployments.apps "default-collector" not found. See [Issue #3296]

Potentially Requiring Action

• BasicMetricSet only supports reach and percent_reach for cumulative weekly. See [Issue #3363]
• Operators may want to use a larger machine type such as c4d-standard-32 for the Results fulfiller, increasing the maximum Java heap size to 96 GiB. These can be configured using variables in the edp-aggregator Terraform module. See [Issue #3366]
• The post processor requires that ImpressionQualificationFilters named "mrc" and "ami" exist. This Issue will be corrected in the future so that the specific names are flexible. See [PR #3344]
• Creating a Metric (and therefore a Report or BasicReport) using a ModelLine of type DEV requires the Principal to have the reporting.metrics.createWithDevModelLine permission. See [Issue #3221]
• Creating a BasicReport using a ModelLine of type DEV requires the Principal to have the reporting.basicReports.createWithDevModelLine permission. See [Issue #3221]
• The Reporting public API server has a new --base-impression-qualification-filter option to specify the resource name of an ImpressionQualificationFilter resource which is always included in BasicReport creation. It can be specified multiple times. See [Issue #3216]
• The Reporting public API server has a new required --pdp-name option. This is the resource name of the Population Data Provider (PDP) corresponding to the running Population Requisition Fulfiller instance. See [Issue #3290]

What's Changed

• ci: Fix Update-cmms dependency graph by @laureanobrs in #3297
• chore: Update urllib3 by @laureanobrs in #3298
• fix: set max frequency below the ring modulus by @stevenwarejones in #3292
• feat(report-processor): process MC API phase 2 report result. by @ple13 in #3248
• fix!: Metrics Service now correctly creates a Population Measurement for a Population Metric by @tristanvuong2021 in #3291
• feat: Implement public EventGroup batch create by @roaminggypsy in #3234
• feat: Add Event Group Activities table by @roaminggypsy in #3288
• fix: CreateBasicReport does not include IQF filters in MetricCalculationSpecs that have a Population MetricSpec by @tristanvuong2021 in #3303
• feat(report-processor): add post process report result job. by @ple13 in #3275
• fix: Validate IQF config against Event Descriptor by @tristanvuong2021 in #3277
• feat: Update gRPC Gateway for BasicReport by @tristanvuong2021 in #3279
• feat: Add base ImpressionQualificationFilters flags by @tristanvuong2021 in #3268
• docs: Update dev standards by @laureanobrs in #3287
• ci: Run nightly build twice a day, everyday by @laureanobrs in #3309
• fix(report-processor): fix large correction test for unnoised edps. by @ple13 in #3318
• refactor: use join key as request id when creating an event group by @Marco-Premier in #3305
• testing: Test event group reference ids that map to multiple MCs by @stevenwarejones in #3301
• ci: Tempor...

v0.5.29

See the corresponding milestone for this release.

Highlights

• Allow Results Fulfiller to override frequency cap for direct impression measurements. See [Issue #3246]
• Allow multiple consumers to use the same event group reference id. See [Issue #3245]
• Allow event groups to be deleted in CMMS. See [Issue #3244]
• Increase page size to make event group loading more efficient. See [Issue #3243]
• Prevent duplicate event groups from being created. See [Issue #3242]
• Build ResultGroups for SUCCEEDED BasicReports. See [Issue #3222]
• Allow ModelLine type to be changed. See [Issue #3213]
• Select or Validate BasicReport ModelLine using EnumerateValidModelLines. See [Issue #3131]
• Add overload for WithEnvelopeEncryption with custom Tink keyset parsing. See [Issue #2967]

Issues Addressed

• EDPA DataAvailability update can result in no more available connections. See [Issue #3241]
• Address expat vulnerabilities in v0.5.24 - CVE-2024-8176 / CVE-2023-52425. See [Issue #2876]

Potentially Requiring Action

• The Reporting internal API server has new options --event-message-descriptor-set and --event-message-type-url. These are required when the the --basic-reports-enabled option is true. See [Issue #3096]

What's Changed

• feat: Add GroupingDimensions for fingerprint to grouping mapping by @SanjayVas in #3124
• build: Update some python packages by @laureanobrs in #3199
• feat: Implement ReportResults service methods for noisy values by @SanjayVas in #3158
• feat: Implement ReportResults/AddDenoisedResultValues by @SanjayVas in #3190
• chore: Delete experimental ui by @laureanobrs in #3197
• ci: Pin workflow dependencies by @laureanobrs in #3202
• feat: Transition BasicReport to NOISY_RESULTS_READY when noisy results are added by @SanjayVas in #3192
• Create many benchmark measurements at once by @stevenwarejones in #2989
• feat: BasicReportsReportsJob updates BasicReport State by @tristanvuong2021 in #3205
• refactor: ModelLine creation is moved to ResourceSetup by @tristanvuong2021 in #3179
• feat: Enable structured logging for EDPA Deployments on GKE by @renjiezh in #3212
• feat: Transition BasicReport to SUCCEEDED when processed results are added by @SanjayVas in #3200
• feat: Select or Validate BasicReport ModelLine using EnumerateValidModelLines by @tristanvuong2021 in #3130
• feat!: Wire up internal ReportResults service in servers by @SanjayVas in #3203
• perf: Switch to cheaper C3D machine family for Mills in Halo GCloud projects by @SanjayVas in #3219
• refactor: Refer to unprocessed/processed instead of noisy/de-noised by @SanjayVas in #3206
• docs: Add security policy by @laureanobrs in #3220
• fix: use batching and global throttle in data availability by @georgi in #3207
• build: Update common-jvm by @laureanobrs in #3227
• feat: Implement Event Group internal batch update by @roaminggypsy in #3210
• fix: use max page size for fetching event groups by @georgi in #3228
• refactor: set request id on event group creation by @Marco-Premier in #3247
• feat: SpannerBasicReportsService builds ResultGroups for SUCCEEDED BasicReports by @tristanvuong2021 in #3223
• feat: Implement ModelLines/SetModelLineType by @SanjayVas in #3217
• refactor: accept max frequency for impression count by @Marco-Premier in #3235
• refactor: update event groups based on both reference id and measurement consumer by @Marco-Premier in #3239
• build: Pin dependencies in Dockerfile by @laureanobrs in #3204
• chore: Update code style configurations for Kotlin and Python by @SanjayVas in #3249
• chore: Scorecard workflow cleanup by @laureanobrs in #3251
• chore: update MODULE.bazel.lock file by @laureanobrs in #3267
• ci: Attach reporting API specs to releases by @laureanobrs in #3193
• ci: Add free-disk-space step to sign-images workflow by @laureanobrs in #3272
• ci: Rename workflows by @laureanobrs in #3273
• build: Update deps for googleapis by @SanjayVas in #3270
• ci: Fix update-cmms workflow dependency graph by @laureanobrs in #3282
• feat: Update terraform and cue for network between internal api and TrusTEE mill by @renjiezh in #3215
• refactor: support event group deletion by @Marco-Premier in #3240
• fix: share grpc channel across data availability requests by @Marco-Premier in #3231
• feat: Update kingdom system api for TrusTEE by @renjiezh in #3252
• feat: Implement internal EventGroup batch create by @roaminggypsy in #3233
• feat: Implement event group public batch update by @roaminggypsy in #3211
• fix: Fix duchy ci issue by adding trustee_mill_subnetwork_cidr_range by @renjiezh in #3294
• fix: missing frequency cap value passed to the fulfiller selector by @Marco-Premier in #3280
• chore: Add OSSF Scorecard badge by @laureanobrs in #3293

Full Changelog: v0.5.28...v0.5.29

v0.5.28

See the corresponding milestone for this release.

Highlights

• RequisitionFetcher cache EventGroups to prevent multiple grpc calls. See [Issue #3136]
• Support listing EventGroups with data availability which contains a given range. See [Issue #3078]
• Monitoring: Data Availability Sync. See [Issue #3054]
• Monitoring: Event Group Sync. See [Issue #3053]
• Monitoring: Requisition Fetcher. See [Issue #3052]
• Monitoring: Results Fulfiller. See [Issue #3051]
• Monitoring: OpenTelemetry SDK Module. See [Issue #3049]
• Add model_line to BasicReport. See [Issue #3044]
• Update Test Event and US/UK Events to use Version Annotations. See [Issue #2903]
• Set DataProvider data availability per ModelLine from EDP simulator. See [Issue #2723]
• EDPA Deployment Guide. See [Issue #2528]
• Monitoring: Data Watcher. See [Issue #2526]
• Add Dead letter queue subscriber. See [Issue #2396]

Issues Addressed

• add-report-results-missing-columns.sql schema update fails on Cloud Spanner. See [Issue #3196]
• Noise corrector can return solutions with negative frequency values. See [Issue #3185]
• Erroneous large corrections due to rounding error. See [Issue #3159]
• GcsStorageClient incorrectly implements retry behavior. See [Issue #3154]
• Population Count needs to always be requested by BasicReport. See [Issue #3152]
• ResultsFulfiller process one Pub/Sub message at the time. See [Issue #3116]
• RequisitionFetcher correctly filters out existing requisitions in the Requisition Metadata. See [Issue #3135]
• SetMeasurementResults statement running long. See [Issue #2790]
• Discrepancy in incremental reach figures. See [Issue #2880]
• Noise correction cannot find a value for union_measurement.sigma. See [Issue #3126]
• Image pull failures for busybox on GKE 1.33+. See [Issue #3067]

Potentially Requiring Action

• The data-watcher Cloud Run Function now supports exporting metrics and traces to Google Cloud Monitoring and Google Cloud Trace, respectively. This should be enabled by including the following in the extra_env_vars field of the appropriate entry in the cloud_function_configs variable of the edp-aggregator Terraform module: OTEL_SERVICE_NAME=edpa.data_watcher,OTEL_METRICS_EXPORTER=google_cloud_monitoring,OTEL_TRACES_EXPORTER=google_cloud_trace,OTEL_LOGS_EXPORTER=logging,OTEL_METRIC_EXPORT_INTERVAL=60000. See [PR #3184]
• Halo-managed environments have reduced the frequency of the Requisition Fetcher component EDPA to every 15 minutes to avoid problems from having more than one instance running concurrently. Operators may wish to do the same. This can be configured using the requisition_fetcher_scheduler_config variable of the edp-aggregator Terraform module. See [PR #3175]
• Operators may wish to increase the max heap size and container memory of the Duchy Requisition fulfillment server to handle a larger number of simultaneous fulfillments. See [PR #3177]
• now supports exporting metrics and traces to Google Cloud Monitoring and Google Cloud Trace, respectively. Provide open telemetry environment variables for ResultsFulfiller TEE to enable it. https://github.com/world-federation-of-advertisers/cross-media-measurement/blob/8d3f8b7a07215306520b6eb703dd8b6b3a462de8/src/main/terraform/gcloud/modules/mig/main.tf#L19-L37. See [PR #3167]
• The requisition-fetcher Cloud Run Function now supports exporting metrics and traces to Google Cloud Monitoring and Google Cloud Trace, respectively. This should be enabled by including the following in the extra_env_vars field of the appropriate entry in the cloud_function_configs variable of the edp-aggregator Terraform module: OTEL_SERVICE_NAME=edpa.requisition_fetcher,OTEL_METRICS_EXPORTER=google_cloud_monitoring,OTEL_TRACES_EXPORTER=google_cloud_trace,OTEL_LOGS_EXPORTER=logging,OTEL_METRIC_EXPORT_INTERVAL=60000. See [Issue #3052]
• The event-group-sync Cloud Run Function now supports exporting metrics and traces to Google Cloud Monitoring and Google Cloud Trace, respectively. This should be enabled by including the following in the extra_env_vars field of the appropriate entry in the cloud_function_configs variable of the edp-aggregator Terraform module: OTEL_SERVICE_NAME=edpa.event_group_sync,OTEL_METRICS_EXPORTER=google_cloud_monitoring,OTEL_TRACES_EXPORTER=google_cloud_trace,OTEL_LOGS_EXPORTER=logging,OTEL_METRIC_EXPORT_INTERVAL=60000. See [PR #3021]
• The data-availability Cloud Run Function now supports exporting metrics and traces to Google Cloud Monitoring and Google Cloud Trace, respectively. This should be enabled by including the following in the extra_env_vars field of the appropriate entry in the cloud_function_configs variable of the edp-aggregator Terraform module: OTEL_SERVICE_NAME=edpa.data_availability,OTEL_METRICS_EXPORTER=google_cloud_monitoring,OTEL_TRACES_EXPORTER=google_cloud_trace,OTEL_LOGS_EXPORTER=logging,OTEL_METRIC_EXPORT_INTERVAL=60000. See [PR #3020]

What's Changed

• fix: Reference busybox:1.27 for containerd 2.0 compatibility by @SanjayVas in #3068
• feat: Generate OpenAPI documents for Reporting API services by @SanjayVas in #1788
• ci: Add release draft automation by @laureanobrs in #2853
• ci: Add option to disable cache when running coverage by @laureanobrs in #3012
• feat: Enforce that DataProvider availability is within ModelLine active interval by @SanjayVas in #3032
• feat: Support enumerating valid ModelLines across providers/suites by @SanjayVas in #3033
• test: Add Basic Report creation to correctness test by @tristanvuong2021 in #3007
• feat: Support active interval filter and wildcards for ListModelLines by @SanjayVas in #3034
• perf: Increase container memory for Reporting internal API server by @SanjayVas in #3082
• ci: Use rsync to speed up large directory deletion by @SanjayVas in #3083
• build: Add py_image macro for Python container images by @SanjayVas in #3063
• feat: Update Kingdom to support TrusTEE Measurement creation. by @renjiezh in #3072
• feat: Add model_line to BasicReport by @tristanvuong2021 in #3045
• feat: Replace DataProvider availability intervals in EDP simulator by @SanjayVas in #3085
• feat: Support listing EventGroups by range contained by data availability by @SanjayVas in #3093
• feat: add roles to cloud functions for opentelemetry by @Marco-Premier in #3094
• fix: add grpc shutdown hook in the RequisitionFetcher by @Marco-Premier in #3097
• allow dynamic event group paths by @stevenwarejones in #3091
• feat: Update ReportResults service to have clearer semantics by @SanjayVas in #3077
• ci: add OpenSSF Scorecard checks by @TNATALI in #3098
• feat: Transform ReportResults into BasicReport ResultGroups by @tristanvuong2021 in #3070
• fix(report-processor): handle report with non-video EDP. by @ple13 in #3117
• feat: Implement ImpressionMetadata batch delete by @roaminggypsy in #3076
• perf: Refactor Reporting Queries to have Values clauses in Materialized CTEs by @tristanvuong2021 in #3120
• feat: Implement impression metadata batch create by @roaminggypsy in #3087
• feat: terraform for dead letter queue by @georgi in #3092
• Fix: bug in the RequisitionFetcher where existing requisition metadata with state QUEUED or PROCESSING where treated as new by @Marco-Premier in #3121
• refactor: requisition grouper to cache event groups by @Marco-Premier in #3125
• refactor: update logic to signal requisition metadata by @Marco-Premier in #3134
• feat: Use integers for histogram bins and add fingerprint fields by @SanjayVas in https://github.com/world-federation-of-a...

v0.5.28-rc2

See the corresponding milestone for this release.

Highlights

• RequisitionFetcher cache EventGroups to prevent multiple grpc calls. See [Issue #3136]
• Support listing EventGroups with data availability which contains a given range. See [Issue #3078]
• Monitoring: Data Availability Sync. See [Issue #3054]
• Monitoring: Event Group Sync. See [Issue #3053]
• Monitoring: Requisition Fetcher. See [Issue #3052]
• Monitoring: Results Fulfiller. See [Issue #3051]
• Monitoring: OpenTelemetry SDK Module. See [Issue #3049]
• Add model_line to BasicReport. See [Issue #3044]
• Update Test Event and US/UK Events to use Version Annotations. See [Issue #2903]
• Set DataProvider data availability per ModelLine from EDP simulator. See [Issue #2723]
• EDPA Deployment Guide. See [Issue #2528]
• Monitoring: Data Watcher. See [Issue #2526]
• Add Dead letter queue subscriber. See [Issue #2396]

Issues Addressed

• Erroneous large corrections due to rounding error. See [Issue #3159]
• GcsStorageClient incorrectly implements retry behavior. See [Issue #3154]
• Population Count needs to always be requested by BasicReport. See [Issue #3152]
• ResultsFulfiller process one Pub/Sub message at the time. See [Issue #3116]
• RequisitionFetcher correctly filters out existing requisitions in the Requisition Metadata. See [Issue #3135]
• SetMeasurementResults statement running long. See [Issue #2790]
• Discrepancy in incremental reach figures. See [Issue #2880]
• Noise correction cannot find a value for union_measurement.sigma. See [Issue #3126]
• Image pull failures for busybox on GKE 1.33+. See [Issue #3067]

Potentially Requiring Action

What's Changed

• fix: Reference busybox:1.27 for containerd 2.0 compatibility by @SanjayVas in #3068
• feat: Generate OpenAPI documents for Reporting API services by @SanjayVas in #1788
• ci: Add release draft automation by @laureanobrs in #2853
• ci: Add option to disable cache when running coverage by @laureanobrs in #3012
• feat: Enforce that DataProvider availability is within ModelLine active interval by @SanjayVas in #3032
• feat: Support enumerating valid ModelLines across providers/suites by @SanjayVas in #3033
• test: Add Basic Report creation to correctness test by @tristanvuong2021 in #3007
• feat: Support active interval filter and wildcards for ListModelLines by @SanjayVas in #3034
• perf: Increase container memory for Reporting internal API server by @SanjayVas in #3082
• ci: Use rsync to speed up large directory deletion by @SanjayVas in #3083
• build: Add py_image macro for Python container images by @SanjayVas in #3063
• feat: Update Kingdom to support TrusTEE Measurement creation. by @renjiezh in #3072
• feat: Add model_line to BasicReport by @tristanvuong2021 in #3045
• feat: Replace DataProvider availability intervals in EDP simulator by @SanjayVas in #3085
• feat: Support listing EventGroups by range contained by data availability by @SanjayVas in #3093
• feat: add roles to cloud functions for opentelemetry by @Marco-Premier in #3094
• fix: add grpc shutdown hook in the RequisitionFetcher by @Marco-Premier in #3097
• allow dynamic event group paths by @stevenwarejones in #3091
• feat: Update ReportResults service to have clearer semantics by @SanjayVas in #3077
• ci: add OpenSSF Scorecard checks by @TNATALI in #3098
• feat: Transform ReportResults into BasicReport ResultGroups by @tristanvuong2021 in #3070
• fix(report-processor): handle report with non-video EDP. by @ple13 in #3117
• feat: Implement ImpressionMetadata batch delete by @roaminggypsy in #3076
• perf: Refactor Reporting Queries to have Values clauses in Materialized CTEs by @tristanvuong2021 in #3120
• feat: Implement impression metadata batch create by @roaminggypsy in #3087
• feat: terraform for dead letter queue by @georgi in #3092
• Fix: bug in the RequisitionFetcher where existing requisition metadata with state QUEUED or PROCESSING where treated as new by @Marco-Premier in #3121
• refactor: requisition grouper to cache event groups by @Marco-Premier in #3125
• refactor: update logic to signal requisition metadata by @Marco-Premier in #3134
• feat: Use integers for histogram bins and add fingerprint fields by @SanjayVas in #3129
• ci: Restrict write permissions in workflows by @laureanobrs in #3142
• feat: implement internal RequisitionMetadata batch create by @roaminggypsy in #3132
• perf: Increase heap size of Requisition Fulfillment server by @SanjayVas in #2981
• build: update dependencies by @georgi in #3141
• Allow additional env vars for Resultsfulfiller by @georgi in #3137
• fix: limit fulfillment parallism to 15 by @georgi in #3144
• fix: reduce memory usage by @georgi in #3140
• feat: set workers based on cpu count by @georgi in #3139
• feat: add telemetry module by @georgi in #3066
• refactor: use InProcessServerMethods from common-jvm by @georgi in #2488
• feat: Add dead letter queue listener for failed work items by @georgi in #2351
• feat: data watcher telemetry by @georgi in #3019
• feat: data availability telemetry by @georgi in #3020
• feat: event group sync telemetry by @georgi in #3021
• feat: resultsfulfiller telemetry by @georgi in #3081
• feat: BasicReport always requests Population Count Metric by @tristanvuong2021 in #3151
• build: Update common-jvm dep for changes to GcsStorageClient by @SanjayVas in #3155
• feat: Upgrade Google Terraform provider for edp-aggregator module to 6.50.0 by @SanjayVas in #3153
• Add 'needs triage' label to Issue templates by @laureanobrs in #3145
• feat: requisition fetcher telemetry by @georgi in #3069
• perf: Increase reporting public server heap size to 64M. by @renjiezh in #3157
• feat: Implement requisition metadata batch create public API by @roaminggypsy in https://github.com/world-federation-of-advertisers/cross-media-measu...

v0.5.28-rc1

See the corresponding milestone for this release.

Highlights

• Support listing EventGroups with data availability which contains a given range. See [Issue #3078]
• Add model_line to BasicReport. See [Issue #3044]
• Set DataProvider data availability per ModelLine from EDP simulator. See [Issue #2723]
• Integration Guide. See [Issue #2529]

Issues Addressed

• Discrepancy in incremental reach figures. See [Issue #2880]
• fix(report-processor): handle report with non-video EDP. by @ple13 in #3117
• Image pull failures for busybox on GKE 1.33+. See [Issue #3067]

Potentially Requiring Action

What's Changed

• fix: Reference busybox:1.27 for containerd 2.0 compatibility by @SanjayVas in #3068
• feat: Generate OpenAPI documents for Reporting API services by @SanjayVas in #1788
• ci: Add release draft automation by @laureanobrs in #2853
• ci: Add option to disable cache when running coverage by @laureanobrs in #3012
• feat: Enforce that DataProvider availability is within ModelLine active interval by @SanjayVas in #3032
• feat: Support enumerating valid ModelLines across providers/suites by @SanjayVas in #3033
• test: Add Basic Report creation to correctness test by @tristanvuong2021 in #3007
• feat: Support active interval filter and wildcards for ListModelLines by @SanjayVas in #3034
• perf: Increase container memory for Reporting internal API server by @SanjayVas in #3082
• ci: Use rsync to speed up large directory deletion by @SanjayVas in #3083
• build: Add py_image macro for Python container images by @SanjayVas in #3063
• feat: Update Kingdom to support TrusTEE Measurement creation. by @renjiezh in #3072
• feat: Add model_line to BasicReport by @tristanvuong2021 in #3045
• feat: Replace DataProvider availability intervals in EDP simulator by @SanjayVas in #3085
• feat: Support listing EventGroups by range contained by data availability by @SanjayVas in #3093
• feat: add roles to cloud functions for opentelemetry by @Marco-Premier in #3094
• fix: add grpc shutdown hook in the RequisitionFetcher by @Marco-Premier in #3097
• allow dynamic event group paths by @stevenwarejones in #3091
• feat: Update ReportResults service to have clearer semantics by @SanjayVas in #3077
• ci: add OpenSSF Scorecard checks by @TNATALI in #3098
• feat: Transform ReportResults into BasicReport ResultGroups by @tristanvuong2021 in #3070
• fix(report-processor): handle report with non-video EDP. by @ple13 in #3117

New Contributors

• @TNATALI made their first contribution in #3098

Full Changelog: v0.5.27...v0.5.28-rc1

v0.5.27

See the corresponding milestone for this release.

Highlights

• EDPA now uses metadata storage.
• The Population Requisition Fulfiller implementation is now covered by cloud tests. See #2831

Issues Addressed

• Single DEADLINE_EXCEEDED on CreateComputationLogEntry can result in Computation failure
• Stack overflow when calling CreateEventGroupMetadataDescriptor
• DEADLINE_EXCEEDED errors writing to GCS
• The report processor does not compute the IndependenceCheckStatus.

Potentially Requiring Action

• EDPA deployment requires a new Terraform module called "cloud-scheduler" for automatically launching the RequisitionFetcher Cloud Run Function. See #2447
• A new Cloud Function is now deployed: DataAvailabilitySync. Terraform variables must be updated accordingly. See #2934
• The RequisitionFetcherFunction has a new required environment variable: METADATA_STORAGE_TARGET. See #2996
• The requisition fetcher config file has a new field: requisition_metadata_storage_connection. See #2996
• ResultsFulfiller has new required flags to support the RequisitionMetadataStorage: --edpa-aggregator-public-api-target. See #2985
• ResultsFulfiller has a new required terraform variable: edp_aggregator_public_api_target. See #2985

What's Changed

• feat!: add Cloud Scheduler for requisition fetcher with 1-minute interval by @georgi in #2447
• feat!: data availability sync terraform deployment by @Marco-Premier in #2934
• feat: Implement system API for RequisitionMetadata service by @renjiezh in #2788
• feat: Allow retroactive creation of ModelLines and ModelRollouts by @SanjayVas in #2921
• build: Update common-jvm dep for grpc-java by @SanjayVas in #2947
• feat!: add private network by @georgi in #2391
• feat: Support listing ModelReleases by Population by @SanjayVas in #2932
• perf: Reduce heap size of Requisition Fulfillment server by @SanjayVas in #2951
• feat: Add --default-service-config option to Duchy internal API server by @SanjayVas in #2954
• feat: integrate event pipeline into results fulfiller by @georgi in #2836
• feat: enable buildbuddy on edpa workflows by @Marco-Premier in #2953
• feat: add event group sync json support by @Marco-Premier in #2949
• ci: Fix nightly build tag creation by @laureanobrs in #2968
• build: Add code coverage report for kotlin packages by @laureanobrs in #2102
• Revert "feat!: add private network " by @SanjayVas in #2957
• fix: Partially revert "feat: Collect API change metrics for CMMS public API" by @SanjayVas in #2976
• feat: implement ImpressionMetadata system API (get, create, delete) by @roaminggypsy in #2941
• fix: Modify conversion of IQF to CEL string by @tristanvuong2021 in #2963
• feat: Support listing ModelRollouts by ModelRelease by @SanjayVas in #2939
• feat: Add deployment for RequisitionMetadata service by @renjiezh in #2952
• feat: Implement List ImpressionMetadata, change delete from hard to soft by @roaminggypsy in #2970
• feat: Add ImpressionMetadataService into deployment by @renjiezh in #2982
• feat: Implement internal computeModelLineBounds by @renjiezh in #2969
• feat(report-processor): support weekly non cumulative measurements. by @ple13 in #2938
• refactor: Make Reporting event message flags reusable across different classes by @tristanvuong2021 in #2980
• test: Exercise Population Requisition Fulfiller in K8s test by @SanjayVas in #2942
• feat: Implement system computeModelLineBounds by @roaminggypsy in #2977
• fix: Remove edpa tf settings that should be reverted for private network by @renjiezh in #2990
• feat!: add private network by @Marco-Premier in #2978
• feat: add internal api for listRequisitionMetadata by @Marco-Premier in #2994
• fix: wrong replacement trigger to mig module by @Marco-Premier in #3002
• test: Add public BasicReport creation to InProcess Reporting test by @tristanvuong2021 in #3000
• fix: report_result and corresponding service by @tristanvuong2021 in #2973
• test: Don't allow run ReportProcessorTest to run remotely by @SanjayVas in #3014
• fix: Use http for GCS to avoid connection issue by @renjiezh in #3011
• docs: Fix misleading screenshot for commit description by @SanjayVas in #3030
• feat: Transform Report Results into format applicable to BasicReport by @tristanvuong2021 in #2962
• feat: Collect API change metrics for CMMS public API by @SanjayVas in #2988
• fix: Allow network access to EDPA API servers via NetworkPolicy by @Marco-Premier in #3024
• fix: wrong certificates content by @Marco-Premier in #3026
• fix: bugs in the DataAvailability Sync function by @Marco-Premier in #3025
• feat(report-processor): get report from report summary v2. by @ple13 in #2959
• fix(report-processor): add threshold for large correction. by @ple13 in #3017
• feat: add address resource for edp aggregator system api by @Marco-Premier in #3023
• fix(report-processor): enable the computation of IndependenceCheckStatus for union. by @ple13 in #3004
• feat: add report to the listRequisitionMetadata service by @Marco-Premier in #3035
• feat: add support to base64 dek by @Marco-Premier in #2905
• Add EDP Aggregator service into Update CMMS workflow by @renjiezh in #3041
• feat: add support for single json file without recordio by @stevenwarejones in #3039
• feat: edpa: add support for aes gcm json parsing by @stevenwarejones in #3013
• feat: Efficient VidIndexMap by @georgi in #2974
• feat(report-processor): add postprocessing_successful field to the log. by...

v0.5.27-rc3

Issues Addressed

• Single DEADLINE_EXCEEDED on CreateComputationLogEntry can result in Computation failure
• Stack overflow when calling CreateEventGroupMetadataDescriptor
• DEADLINE_EXCEEDED errors writing to GCS
• The report processor does not compute the IndependenceCheckStatus.

Potentially Requiring Action

• EDPA deployment requires a new Terraform module called "cloud-scheduler" for automatically launching the RequisitionFetcher Cloud Run Function. See #2447
• A new Cloud Function is now deployed: DataAvailabilitySync. Terraform variables must be updated accordingly. See #2934

What's Changed

• feat!: add Cloud Scheduler for requisition fetcher with 1-minute interval by @georgi in #2447
• feat!: data availability sync terraform deployment by @Marco-Premier in #2934
• feat: Implement system API for RequisitionMetadata service by @renjiezh in #2788
• feat: Allow retroactive creation of ModelLines and ModelRollouts by @SanjayVas in #2921
• build: Update common-jvm dep for grpc-java by @SanjayVas in #2947
• feat!: add private network by @georgi in #2391
• feat: Support listing ModelReleases by Population by @SanjayVas in #2932
• perf: Reduce heap size of Requisition Fulfillment server by @SanjayVas in #2951
• feat: Add --default-service-config option to Duchy internal API server by @SanjayVas in #2954
• feat: integrate event pipeline into results fulfiller by @georgi in #2836
• feat: enable buildbuddy on edpa workflows by @Marco-Premier in #2953
• feat: add event group sync json support by @Marco-Premier in #2949
• ci: Fix nightly build tag creation by @laureanobrs in #2968
• build: Add code coverage report for kotlin packages by @laureanobrs in #2102
• Revert "feat!: add private network " by @SanjayVas in #2957
• fix: Partially revert "feat: Collect API change metrics for CMMS public API" by @SanjayVas in #2976
• feat: implement ImpressionMetadata system API (get, create, delete) by @roaminggypsy in #2941
• fix: Modify conversion of IQF to CEL string by @tristanvuong2021 in #2963
• feat: Support listing ModelRollouts by ModelRelease by @SanjayVas in #2939
• feat: Add deployment for RequisitionMetadata service by @renjiezh in #2952
• feat: Implement List ImpressionMetadata, change delete from hard to soft by @roaminggypsy in #2970
• feat: Add ImpressionMetadataService into deployment by @renjiezh in #2982
• feat: Implement internal computeModelLineBounds by @renjiezh in #2969
• feat(report-processor): support weekly non cumulative measurements. by @ple13 in #2938
• refactor: Make Reporting event message flags reusable across different classes by @tristanvuong2021 in #2980
• test: Exercise Population Requisition Fulfiller in K8s test by @SanjayVas in #2942
• feat: Implement system computeModelLineBounds by @roaminggypsy in #2977
• fix: Remove edpa tf settings that should be reverted for private network by @renjiezh in #2990
• feat!: add private network by @Marco-Premier in #2978
• feat: add internal api for listRequisitionMetadata by @Marco-Premier in #2994
• fix: wrong replacement trigger to mig module by @Marco-Premier in #3002
• test: Add public BasicReport creation to InProcess Reporting test by @tristanvuong2021 in #3000
• fix: report_result and corresponding service by @tristanvuong2021 in #2973
• test: Don't allow run ReportProcessorTest to run remotely by @SanjayVas in #3014
• fix: Use http for GCS to avoid connection issue by @renjiezh in #3011
• docs: Fix misleading screenshot for commit description by @SanjayVas in #3030
• feat: Transform Report Results into format applicable to BasicReport by @tristanvuong2021 in #2962
• feat: Collect API change metrics for CMMS public API by @SanjayVas in #2988
• fix: Allow network access to EDPA API servers via NetworkPolicy by @Marco-Premier in #3024
• fix: wrong certificates content by @Marco-Premier in #3026
• fix: bugs in the DataAvailability Sync function by @Marco-Premier in #3025
• feat(report-processor): get report from report summary v2. by @ple13 in #2959
• fix(report-processor): add threshold for large correction. by @ple13 in #3017
• feat: add address resource for edp aggregator system api by @Marco-Premier in #3023
• fix(report-processor): enable the computation of IndependenceCheckStatus for union. by @ple13 in #3004

Full Changelog: v0.5.26...v0.5.27-rc3

v0.5.27-rc2

Issues Addressed

• Single DEADLINE_EXCEEDED on CreateComputationLogEntry can result in Computation failure
• Stack overflow when calling CreateEventGroupMetadataDescriptor
• DEADLINE_EXCEEDED errors writing to GCS

Potentially Requiring Action

• EDPA deployment requires a new Terraform module called "cloud-scheduler" for automatically launching the RequisitionFetcher Cloud Run Function. See #2447
• A new Cloud Function is now deployed: DataAvailabilitySync. Terraform variables must be updated accordingly. See #2934

What's Changed

• feat!: add Cloud Scheduler for requisition fetcher with 1-minute interval by @georgi in #2447
• feat!: data availability sync terraform deployment by @Marco-Premier in #2934
• feat: Implement system API for RequisitionMetadata service by @renjiezh in #2788
• feat: Allow retroactive creation of ModelLines and ModelRollouts by @SanjayVas in #2921
• build: Update common-jvm dep for grpc-java by @SanjayVas in #2947
• feat!: add private network by @georgi in #2391
• feat: Support listing ModelReleases by Population by @SanjayVas in #2932
• perf: Reduce heap size of Requisition Fulfillment server by @SanjayVas in #2951
• feat: Add --default-service-config option to Duchy internal API server by @SanjayVas in #2954
• feat: integrate event pipeline into results fulfiller by @georgi in #2836
• feat: enable buildbuddy on edpa workflows by @Marco-Premier in #2953
• feat: add event group sync json support by @Marco-Premier in #2949
• ci: Fix nightly build tag creation by @laureanobrs in #2968
• build: Add code coverage report for kotlin packages by @laureanobrs in #2102
• Revert "feat!: add private network " by @SanjayVas in #2957
• fix: Partially revert "feat: Collect API change metrics for CMMS public API" by @SanjayVas in #2976
• feat: implement ImpressionMetadata system API (get, create, delete) by @roaminggypsy in #2941
• fix: Modify conversion of IQF to CEL string by @tristanvuong2021 in #2963
• feat: Support listing ModelRollouts by ModelRelease by @SanjayVas in #2939
• feat: Add deployment for RequisitionMetadata service by @renjiezh in #2952
• feat: Implement List ImpressionMetadata, change delete from hard to soft by @roaminggypsy in #2970
• feat: Add ImpressionMetadataService into deployment by @renjiezh in #2982
• feat: Implement internal computeModelLineBounds by @renjiezh in #2969
• feat(report-processor): support weekly non cumulative measurements. by @ple13 in #2938
• refactor: Make Reporting event message flags reusable across different classes by @tristanvuong2021 in #2980
• test: Exercise Population Requisition Fulfiller in K8s test by @SanjayVas in #2942
• feat: Implement system computeModelLineBounds by @roaminggypsy in #2977
• fix: Remove edpa tf settings that should be reverted for private network by @renjiezh in #2990
• feat!: add private network by @Marco-Premier in #2978
• feat: add internal api for listRequisitionMetadata by @Marco-Premier in #2994
• fix: wrong replacement trigger to mig module by @Marco-Premier in #3002
• test: Add public BasicReport creation to InProcess Reporting test by @tristanvuong2021 in #3000
• fix: report_result and corresponding service by @tristanvuong2021 in #2973
• test: Don't allow run ReportProcessorTest to run remotely by @SanjayVas in #3014
• fix: Use http for GCS to avoid connection issue by @renjiezh in #3011

Full Changelog: v0.5.26...v0.5.27-rc2

v0.5.27-rc1

Issues Addressed

• Single DEADLINE_EXCEEDED on CreateComputationLogEntry can result in Computation failure
• Stack overflow when calling CreateEventGroupMetadataDescriptor

Potentially Requiring Action

• EDPA deployment requires a new Terraform module called "cloud-scheduler" for automatically launching the RequisitionFetcher Cloud Run Function. See #2447
• A new Cloud Function is now deployed: DataAvailabilitySync. Terraform variables must be updated accordingly. See #2934

What's Changed

• feat!: add Cloud Scheduler for requisition fetcher with 1-minute interval by @georgi in #2447
• feat!: data availability sync terraform deployment by @Marco-Premier in #2934
• feat: Implement system API for RequisitionMetadata service by @renjiezh in #2788
• feat: Allow retroactive creation of ModelLines and ModelRollouts by @SanjayVas in #2921
• build: Update common-jvm dep for grpc-java by @SanjayVas in #2947
• feat!: add private network by @georgi in #2391
• feat: Support listing ModelReleases by Population by @SanjayVas in #2932
• perf: Reduce heap size of Requisition Fulfillment server by @SanjayVas in #2951
• feat: Add --default-service-config option to Duchy internal API server by @SanjayVas in #2954
• feat: integrate event pipeline into results fulfiller by @georgi in #2836
• feat: enable buildbuddy on edpa workflows by @Marco-Premier in #2953
• feat: add event group sync json support by @Marco-Premier in #2949
• ci: Fix nightly build tag creation by @laureanobrs in #2968
• build: Add code coverage report for kotlin packages by @laureanobrs in #2102
• Revert "feat!: add private network " by @SanjayVas in #2957
• fix: Partially revert "feat: Collect API change metrics for CMMS public API" by @SanjayVas in #2976
• feat: implement ImpressionMetadata system API (get, create, delete) by @roaminggypsy in #2941

Full Changelog: v0.5.26...v0.5.27-rc1

v0.5.26

See the corresponding milestone for this release.

Highlights

• The Population Requisition Fulfiller can now be deployed in the same cluster as the Kingdom. See the deployment guide.
  • Note that this is not yet tested. See #2907

Issues Addressed

• HMSS computation fails due to behind peer duchy stage
• ReportPostProcessor does not write the log to the storage bucket when an exception is thrown.

What's Changed

• feat: adapt EventReader to batch processing by @georgi in #2680
• fix(report-processor): always write report post processor log to bucket. by @ple13 in #2885
• refactor: Restrict visibility of requisitions common library by @SanjayVas in #2886
• feat: Storage Event Source by @georgi in #2709
• feat: filter processing by @georgi in #2683
• feat: add event processing pipeline by @georgi in #2684
• feat: event processing orchestrator by @georgi in #2707
• style: change proto field for event group metadata blob uri by @Marco-Premier in #2879
• feat: Add State Filter to Internal ListBasicReports by @tristanvuong2021 in #2897
• feat: Add Internal BasicReports method to Fail BasicReport by @tristanvuong2021 in #2895
• build(container): Use disk cache rather than mounting host output user root by @SanjayVas in #2906
• feat: Output certificate name when creating DataProvider by @SanjayVas in #2908
• fix: change data_availability_interval field description by @Marco-Premier in #2878
• feat: Implement create impression metadata schema sql by @roaminggypsy in #2868
• feat: Add version annotations to TestEvent by @kungfucraig in #2901
• feat: Add results schema for Measurement Consumer API by @kungfucraig in #2763
• feat: Create Job to check Reports backing BasicReports by @tristanvuong2021 in #2881
• feat: Implement internal RequisitionMetadata service. by @renjiezh in #2869
• refactor: Update InMemoryVidIndexMap to use an Int keyed map by @kungfucraig in #2919
• refactor: update generate synthetic data to support model line by @Marco-Premier in #2910
• feat: Create Report for BasicReport by @tristanvuong2021 in #2917
• feat: Fulfill population Requisitions using PopulationSpec from resource by @SanjayVas in #2858
• ci: Add local K8s configuration targets to build-test workflow. by @SanjayVas in #2899
• feat: Add cloud K8s configuration for Population Requisition Fulfiller by @SanjayVas in #2892
• feat: Update population requisition fulfiller in Halo-managed environments by @SanjayVas in #2904
• feat: get model line availability by @Marco-Premier in #2913
• feat: add support for signed builds on edpa results fulfiller by @Marco-Premier in #2918
• feat: optimize vid index map by @georgi in #2928
• feat: read event group reference id from blob details by @georgi in #2931
• feat: add DataAvailability to sync impressions availability for data providers with the kingdom by @Marco-Premier in #2893
• refactor: Dynamically convert SyntheticPopulationSpec to PopulationSpec by @SanjayVas in #2900
• refactor: allow data watcher to process empty blobs if are called 'done' by @Marco-Premier in #2935
• feat: add data availability function by @Marco-Premier in #2894
• Fix: Throw transient error from HMSS mill for a behind peer duchy stage by @renjiezh in #2943
• perf: Increase memory to Population requisition fulfiller to avoid OOMKilled by @SanjayVas in #2937
• fix: Handle multiple connections for a single PortForwarder by @SanjayVas in #2940
• feat: impression metadata internal API (create, get, delete) by @roaminggypsy in #2909
• feat(report-processor): Log when a Measurement has a large correction by @ple13 in #2946
• feat: Add Event DescriptorSet flags to Reporting Server for BasicReports by @tristanvuong2021 in #2923

Full Changelog: v0.5.25...v0.5.26