v0.5.27

[SanjayVas]

See the corresponding milestone for this release.

Highlights

• EDPA now uses metadata storage.
• The Population Requisition Fulfiller implementation is now covered by cloud tests. See Support running Population Data Provider (PDP) requisition fulfiller in cloud #2831

Issues Addressed

• Single DEADLINE_EXCEEDED on CreateComputationLogEntry can result in Computation failure
• Stack overflow when calling CreateEventGroupMetadataDescriptor
• DEADLINE_EXCEEDED errors writing to GCS
• The report processor does not compute the IndependenceCheckStatus.

Potentially Requiring Action

• EDPA deployment requires a new Terraform module called "cloud-scheduler" for automatically launching the RequisitionFetcher Cloud Run Function. See feat!: add Cloud Scheduler for requisition fetcher with 1-minute interval #2447
• A new Cloud Function is now deployed: DataAvailabilitySync. Terraform variables must be updated accordingly. See feat!: data availability sync terraform deployment #2934
• The RequisitionFetcherFunction has a new required environment variable: METADATA_STORAGE_TARGET. See feat!: requisition fetcher use metadata storage #2996
• The requisition fetcher config file has a new field: requisition_metadata_storage_connection. See feat!: requisition fetcher use metadata storage #2996
• ResultsFulfiller has new required flags to support the RequisitionMetadataStorage: --edpa-aggregator-public-api-target. See feat!: integrate requisition metadata storage in results fulfiller #2985
• ResultsFulfiller has a new required terraform variable: edp_aggregator_public_api_target. See feat!: integrate requisition metadata storage in results fulfiller #2985

What's Changed

• feat!: add Cloud Scheduler for requisition fetcher with 1-minute interval by @georgi in feat!: add Cloud Scheduler for requisition fetcher with 1-minute interval #2447
• feat!: data availability sync terraform deployment by @Marco-Premier in feat!: data availability sync terraform deployment #2934
• feat: Implement system API for RequisitionMetadata service by @renjiezh in feat: Implement system API for RequisitionMetadata service  #2788
• feat: Allow retroactive creation of ModelLines and ModelRollouts by @SanjayVas in feat: Allow retroactive creation of ModelLines and ModelRollouts #2921
• build: Update common-jvm dep for grpc-java by @SanjayVas in build: Update common-jvm dep for grpc-java #2947
• feat!: add private network by @georgi in feat!: add private network  #2391
• feat: Support listing ModelReleases by Population by @SanjayVas in feat: Support listing ModelReleases by Population #2932
• perf: Reduce heap size of Requisition Fulfillment server by @SanjayVas in perf: Reduce heap size of Requisition Fulfillment server #2951
• feat: Add --default-service-config option to Duchy internal API server by @SanjayVas in feat: Add --default-service-config option to Duchy internal API server #2954
• feat: integrate event pipeline into results fulfiller by @georgi in feat: integrate event pipeline into results fulfiller  #2836
• feat: enable buildbuddy on edpa workflows by @Marco-Premier in feat: enable buildbuddy on edpa workflows #2953
• feat: add event group sync json support by @Marco-Premier in feat: add event group sync json support #2949
• ci: Fix nightly build tag creation by @laureanobrs in ci: Fix nightly build tag creation #2968
• build: Add code coverage report for kotlin packages by @laureanobrs in build: Add code coverage report for kotlin packages #2102
• Revert "feat!: add private network " by @SanjayVas in Revert "feat!: add private network " #2957
• fix: Partially revert "feat: Collect API change metrics for CMMS public API" by @SanjayVas in fix: Partially revert "feat: Collect API change metrics for CMMS public API" #2976
• feat: implement ImpressionMetadata system API (get, create, delete) by @roaminggypsy in feat: implement ImpressionMetadata system API (get, create, delete) #2941
• fix: Modify conversion of IQF to CEL string by @tristanvuong2021 in fix: Modify conversion of IQF to CEL string #2963
• feat: Support listing ModelRollouts by ModelRelease by @SanjayVas in feat: Support listing ModelRollouts by ModelRelease #2939
• feat: Add deployment for RequisitionMetadata service by @renjiezh in feat: Add deployment for RequisitionMetadata service #2952
• feat: Implement List ImpressionMetadata, change delete from hard to soft by @roaminggypsy in feat: Implement List ImpressionMetadata, change delete from hard to soft #2970
• feat: Add ImpressionMetadataService into deployment by @renjiezh in feat: Add ImpressionMetadataService into deployment #2982
• feat: Implement internal computeModelLineBounds by @renjiezh in feat: Implement internal computeModelLineBounds #2969
• feat(report-processor): support weekly non cumulative measurements. by @ple13 in feat(report-processor): support weekly non cumulative measurements. #2938
• refactor: Make Reporting event message flags reusable across different classes by @tristanvuong2021 in refactor: Make Reporting event message flags reusable across different classes #2980
• test: Exercise Population Requisition Fulfiller in K8s test by @SanjayVas in test: Exercise Population Requisition Fulfiller in K8s test #2942
• feat: Implement system computeModelLineBounds by @roaminggypsy in feat: Implement system computeModelLineBounds #2977
• fix: Remove edpa tf settings that should be reverted for private network by @renjiezh in fix: Remove edpa tf settings that should be reverted for private network #2990
• feat!: add private network by @Marco-Premier in feat!: add private network #2978
• feat: add internal api for listRequisitionMetadata by @Marco-Premier in feat: add internal api for listRequisitionMetadata #2994
• fix: wrong replacement trigger to mig module by @Marco-Premier in fix: wrong replacement trigger to mig module #3002
• test: Add public BasicReport creation to InProcess Reporting test by @tristanvuong2021 in test: Add public BasicReport creation to InProcess Reporting test #3000
• fix: report_result and corresponding service by @tristanvuong2021 in fix: report_result and corresponding service #2973
• test: Don't allow run ReportProcessorTest to run remotely by @SanjayVas in test: Don't allow run ReportProcessorTest to run remotely #3014
• fix: Use http for GCS to avoid connection issue by @renjiezh in fix: Use http for GCS to avoid connection issue #3011
• docs: Fix misleading screenshot for commit description by @SanjayVas in docs: Fix misleading screenshot for commit description #3030
• feat: Transform Report Results into format applicable to BasicReport by @tristanvuong2021 in feat: Transform Report Results into format applicable to BasicReport #2962
• feat: Collect API change metrics for CMMS public API by @SanjayVas in feat: Collect API change metrics for CMMS public API #2988
• fix: Allow network access to EDPA API servers via NetworkPolicy by @Marco-Premier in fix: Allow network access to EDPA API servers via NetworkPolicy #3024
• fix: wrong certificates content by @Marco-Premier in fix: wrong certificates content #3026
• fix: bugs in the DataAvailability Sync function by @Marco-Premier in fix: bugs in the DataAvailability Sync function #3025
• feat(report-processor): get report from report summary v2. by @ple13 in feat(report-processor): get report from report summary v2. #2959
• fix(report-processor): add threshold for large correction. by @ple13 in fix(report-processor): add threshold for large correction. #3017
• feat: add address resource for edp aggregator system api by @Marco-Premier in feat: add address resource for edp aggregator system api #3023
• fix(report-processor): enable the computation of IndependenceCheckStatus for union. by @ple13 in fix(report-processor): enable the computation of IndependenceCheckStatus for union. #3004
• feat: add report to the listRequisitionMetadata service by @Marco-Premier in feat: add report to the listRequisitionMetadata service #3035
• feat: add support to base64 dek by @Marco-Premier in feat: add support to base64 dek #2905
• Add EDP Aggregator service into Update CMMS workflow by @renjiezh in Add EDP Aggregator service into Update CMMS workflow #3041
• feat: add support for single json file without recordio by @stevenwarejones in feat: add support for single json file without recordio #3039
• feat: edpa: add support for aes gcm json parsing by @stevenwarejones in feat: edpa: add support for aes gcm json parsing #3013
• feat: Efficient VidIndexMap by @georgi in feat: Efficient VidIndexMap #2974
• feat(report-processor): add postprocessing_successful field to the log. by @ple13 in feat(report-processor): add postprocessing_successful field to the log. #3043
• feat: enable requisition fetcher secrets needed for the metadata storage by @Marco-Premier in feat: enable requisition fetcher secrets needed for the metadata storage #3036
• feat: add terrafrom changes for the metadata storage and results fulfiller by @Marco-Premier in feat: add terrafrom changes for the metadata storage and results fulfiller #3037
• feat!: requisition fetcher use metadata storage by @Marco-Premier in feat!: requisition fetcher use metadata storage #2996
• feat!: integrate requisition metadata storage in results fulfiller by @Marco-Premier in feat!: integrate requisition metadata storage in results fulfiller #2985
• fix: terraform deployment for data availability function by @Marco-Premier in fix: terraform deployment for data availability function #3027
• test: Add tests for Computing Metric Results when Measurement Results are 0 by @tristanvuong2021 in test: Add tests for Computing Metric Results when Measurement Results are 0 #2929
• feat: integrate impression metadata storage by @Marco-Premier in feat: integrate impression metadata storage #3001
• feat: enable data availability on the cloud test by @Marco-Premier in feat: enable data availability on the cloud test #3028

Full Changelog: v0.5.26...v0.5.27

---

This discussion was created from the release v0.5.27.