207255_Celery_tasks_refactor

[MarekBiczysko]

AB#207255

Added django_celery_boost to:

• hope/apps/account/celery_tasks.py:
  invalidate_permissions_cache_for_user_if_expired_role

• hope/apps/accountability/celery_tasks.py:
  export_survey_sample_task, send_survey_to_users

• hope/apps/core/celery_tasks.py:
  upload_new_kobo_template_and_update_flex_fields_task_with_retry, upload_new_kobo_template_and_update_flex_fields_task, async_job_task, async_retry_job_task

• hope/apps/dashboard/celery_tasks.py:
  update_dashboard_figures, update_recent_dashboard_figures, generate_dash_report_task

• hope/apps/generic_import/celery_tasks.py:
  process_generic_import_task

• hope/apps/geo/celery_tasks.py:
  import_areas_from_csv_task

• hope/apps/grievance/celery_tasks.py:
  deduplicate_and_check_against_sanctions_list_task_single_individual, periodic_grievances_notifications

• hope/apps/household/celery_tasks.py:
  recalculate_population_fields_chunk_task, recalculate_population_fields_task, interval_recalculate_population_fields_task, calculate_children_fields_for_not_collected_individual_data, revalidate_phone_number_task, enroll_households_to_program_task, mass_withdraw_households_from_list_task, cleanup_indexes_in_inactive_programs_task

• hope/apps/payment/celery_tasks.py:
  get_sync_run_rapid_pro_task, create_payment_verification_plan_xlsx, remove_old_cash_plan_payment_verification_xlsx, create_payment_plan_payment_list_xlsx,
  create_payment_plan_payment_list_xlsx_per_fsp, send_payment_plan_payment_list_xlsx_per_fsp_password, import_payment_plan_payment_list_from_xlsx, payment_plan_set_entitlement_flat_amount,
  payment_plan_apply_custom_exchange_rate, import_payment_plan_payment_list_per_fsp_from_xlsx, payment_plan_apply_engine_rule, update_exchange_rate_on_release_payments, remove_old_payment_plan_payment_list_xlsx,
  prepare_payment_plan_task, prepare_follow_up_payment_plan_task, payment_plan_exclude_beneficiaries, export_pdf_payment_plan_summary, periodic_sync_payment_gateway_fsp,
  periodic_sync_payment_gateway_account_types, send_to_payment_gateway, periodic_sync_payment_gateway_records, send_payment_notification_emails, periodic_sync_payment_gateway_delivery_mechanisms,
  payment_plan_apply_steficon_hh_selection, payment_plan_rebuild_stats, payment_plan_full_rebuild, periodic_sync_payment_plan_invoices_western_union_ftp, send_qcf_report_email_notifications,
  periodic_send_payment_plan_reconciliation_overdue_emails, send_payment_plan_reconciliation_overdue_email

• hope/apps/periodic_data_update/celery_tasks.py:
  import_periodic_data_update, export_periodic_data_update_export_template_service, generate_pdu_online_edit_data_task, merge_pdu_online_edit_task,
  remove_old_pdu_template_files_task, send_pdu_online_edit_notification_emails

• hope/apps/program/celery_tasks.py:
  copy_program_task, adjust_program_size_task, populate_pdu_new_rounds_with_null_values_task

• hope/apps/registration_data/celery_tasks.py:
  registration_xlsx_import_task, registration_program_population_import_task, registration_kobo_import_task, registration_kobo_import_hourly_task,
  registration_xlsx_import_hourly_task, merge_registration_data_import_task, rdi_deduplication_task, pull_kobo_submissions_task, validate_xlsx_import_task, deduplicate_documents, check_rdi_import_periodic_task,
  deduplication_engine_process, fetch_biometric_deduplication_results_and_process

• hope/apps/sanction_list/celery_tasks.py:
  sync_sanction_list_task, check_against_sanction_list_task

• hope/apps/targeting/celery_tasks.py:
  create_tp_from_list

• hope/apps/universal_update_script/celery_tasks.py:
  run_universal_individual_update, generate_universal_individual_update_template

• hope/apps/utils/celery_tasks.py:
  send_email_task

• hope/contrib/aurora/celery_tasks.py:
  process_flex_records_task, extract_records_task, fresh_extract_records_task, automate_rdi_creation_task, clean_old_record_files_task

Refactored:

• hope/apps/accountability/celery_tasks.py

• Issue: send_survey_to_users_action did a redundant BusinessArea.objects.get().
  Fix: removed the extra query and used survey.business_area.slug directly.

hope/apps/dashboard/celery_tasks.py

• Issue: update_recent_dashboard_figures eagerly materialized all active business areas with list(...) before iterating.
  Fix: iterates the queryset directly.

hope/apps/grievance/celery_tasks.py

• Issue: grievance notification task accessed ticket.business_area in loops without eager loading, causing N+1 queries.
  Fix: added select_related("business_area").
• Issue: notification timestamp updates used broader saves than needed.
  Fix: saves only last_notification_sent with update_fields=["last_notification_sent"].

hope/apps/household/celery_tasks.py

• Issue: recalculate_population_fields_chunk_task_action loaded households with too little related data and then accessed hh.business_area.name, causing extra queries.
  Fix: added select_related("business_area") and limited fields to the values actually used.

hope/apps/payment/celery_tasks.py

• Issue: payment_plan_apply_custom_exchange_rate_action/update_exchange_rate_on_release_payments_action
  accumulated all eligible payments in memory before bulk_update.
  Fix: switched to chunked iteration and chunked bulk_update.
• Issue: payment_plan_exclude_beneficiaries_action used per-ID .exists() checks, creating an N+1 query pattern.
  Fix: replaced validation with set-based queries and set comparisons.
• Issue: payment_plan_apply_steficon_hh_selection_action accessed payment.household in a loop without eager loading and accumulated large update lists.
  Fix: added select_related("household"), iterated in chunks, and bulk-updated in chunks.
• Issue: remove_old_cash_plan_payment_verification_xlsx_action/remove_old_payment_plan_payment_list_xlsx_action
  used queryset truthiness and logged .count() after deleting rows.
  Fix: captures removed_count before deletion, returns early when empty, and streams deletions with iterator

hope/apps/periodic_data_update/celery_tasks.py

• Issue: remove_old_pdu_template_files_task_action used queryset truthiness, then deleted rows before calling .count(), and iterated deletions without streaming.
  Fix: captures removed_count up front, returns early when empty, streams file deletions with .iterator(chunk_size=1000).

hope/apps/registration_data/celery_tasks.py

• Issue: registration_kobo_import_hourly_task_action/registration_xlsx_import_hourly_task_action
  re-fetched related models after already loading the pending RDI.
  Fix: uses select_related("business_area", "program", "import_data") and reuses the fetched relations.
• Issue: deduplication_engine_process_action fetched the same Program twice.
  Fix: removed the duplicate lookup.
• Issue: check_and_set_taxid iterated with .all() and called save() per row without limiting updated fields.
  Fix: iterates with .iterator(chunk_size=1000) and saves only unique_field via update_fields=["unique_field"].

[codecov]

Codecov Report

❌ Patch coverage is 99.75590% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.27%. Comparing base (f8e7ee1) to head (57072e9).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
src/hope/models/universal_update_script.py	80.00%	1 Missing and 1 partial ⚠️
src/hope/apps/targeting/steficon.py	0.00%	1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (99.75%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #5815      +/-   ##
===========================================
+ Coverage    91.79%   92.27%   +0.47%     
===========================================
  Files          496      494       -2     
  Lines        34343    34480     +137     
  Branches      3569     3597      +28     
===========================================
+ Hits         31525    31816     +291     
+ Misses        2047     1914     -133     
+ Partials       771      750      -21     

Flag	Coverage Δ
e2e	52.87% <37.83%> (+0.09%)	⬆️
unit	91.96% <99.75%> (+0.49%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.