Skip to content

Commit 9270edf

Browse files
henrihaapalasiiligithub-code-quality[bot]
henrihaapalasiili
and
github-code-quality[bot]
authored
KAAV-3492 Validoinnin nopeutus ja Seuraavat elementit eivät siirry eteenpäin, kun elementtiä siirretään menneisyydestä (#362)
* KAAV-3492 Next elements not validated correctly when moving item from past to after current date fix * Potential fix for pull request finding 'Modification of parameter with default' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * Potential fix for pull request finding 'Variable defined multiple times' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * Potential fix for pull request finding 'Unused local variable' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * Potential fix for pull request finding 'Explicit returns mixed with implicit (fall through) returns' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * Potential fix for pull request finding 'Unused local variable' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * Potential fix for pull request finding 'Unused local variable' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * speed up project validation preview: cache attribute metadata per validation request and feed it through update_attribute_data so fake PATCH flows stop re-querying Attributes - add timing metrics around validation + deadline calculations to keep measuring latency regressions - memoize preview deadline calculations so initial/update passes reuse results instead of recomputing the same dependency graph * Restore on-hold timestamp and repair migration pytest kept crashing because the serializer/admin still toggle Project.onhold_at but the column no longer existed. Migration 0184 adds the field back while the model/admin/serializer changes keep behaviour consistent. That exposed migration 0166 importing the live Project model, which now included onhold_at and caused undefined-column errors on fresh DBs. Switching the data migration to use apps.get_model ties it to the historical schema so migrations (and tests) run cleanly. * Potential fix for pull request finding 'Syntax error' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * KAAV-3492 Return all date corrections in single validation pass - Skip deadline validator for ?fake=true requests (section.py) - Merge preview values into response for fake requests (project.py) - Filter date errors for fake requests, let backend auto-correct - Real saves unchanged: still validate and reject invalid dates Eliminates cascading validation calls that caused 30s+ per round. * KAAV-3492 fix onhold_at verbose_name for localisation * Potential fix for pull request finding 'Explicit returns mixed with implicit (fall through) returns' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * KAAV-3492: Optimize fake validation for timeline dates - Add fast-path in views.py for ?fake=true PATCH requests that bypasses the full serializer and calls get_preview_deadlines() directly - Skip expensive external API calls (kaavoitus_api, AD) for fake requests in project serializer - Add payload_keys filter in section serializer to only validate attributes present in the request - Fix syntax error in section.py (stray .select_related line) - Add debug logging for fake validation flow (RECEIVED/CORRECTIONS/RETURNING) Performance improvement: ~30s -> ~2s for timeline validation * KAAV-3492: Optimize timetable validation for fake requests - Only filter section attributes by payload keys when is_fake_request=True (preserves original behavior for normal requests and tests) - Remove debug logging from project serializer and views - Fast-path for fake validation returns corrected dates via get_preview_deadlines() * get_preview_deadlines() now passes updated_attribute_data (all existing values + changes) instead of just updated_attributes (only changes) Deadlines are preserved and only moved if they violate minimum distances * fix: prevent saving during preview/validation mode CRITICAL BUG FIX: _set_calculated_deadlines was calling self.save() even when preview=True (validation mode), causing unintended data saves. Now only saves when not in preview mode. * fix: cascade deadline updates from ehdotus to tarkistettu_ehdotus Removed line that was overwriting calculated values with old values from preview_attribute_data, blocking the cascade chain. Also removed debug logging. * KAAV-3517: Fix confirmed fields not being protected during deadline recalculation (#365) Backend changes: - Fixed _set_calculated_deadline() to return the original confirmed value instead of the calculated value when a field is in confirmed_fields - This prevents confirmed/locked dates from being overwritten during save The bug was that when a field was in confirmed_fields, the code was returning the newly calculated date instead of the original saved date, causing confirmed dates to be recalculated on every save. * KAAV-3492: Fix lautakunta deadline when esilläolo OFF - Add _get_esillaolo_off_distance_override() for special case handling - Calculate from phase start instead of esilläolo_paattyy when esilläolo is OFF - Read distance values from DB (DeadlineDistance) instead of hardcoding - Supports both periaatteet and luonnos phases * KAAV-3492: Skip distance rules when conditions not met in validation - Check distance.check_conditions() before applying min distance rules - Prevents invalid distance enforcement when esilläolo groups are OFF * KAAV-3428: Fix lautakunta dates not pushed when adding 3rd+ esillaolo Added _get_latest_esillaolo_date() to find latest esillaolo variant. Special handling in _enforce_distance_requirements() for P6/P7. Fixed visibility boolean init in get_preview_deadlines(). * KAAV-3517: Fix selective deadline distance enforcement - Only enforce distance requirements on deadlines that actually changed (compare incoming values vs database, not just what frontend sends) - Only apply enforcement when minimum distance is actually violated - Remove hacky filter condition that incorrectly excluded deadlines - Prevents cascade effects where moving one deadline shifts unrelated dates - Add comprehensive tests for the new selective enforcement logic * fixed test * removed non used stuff * KAAV-3492: Fix phase bar expansion when deleting deadline groups - Respect explicitly sent visibility booleans from frontend (False for deletion) - Don't auto-enable secondary slots (_2, _3, _4) when primary (_1) is disabled - Check stored visibility before auto-enabling existing deadlines - Skip deadline validation for disabled deadline groups - Refactor: rename 'date' to 'enforced_date' for clarity, use != instead of 'not ==' - Refactor: extract is_phase_boundary variable in deadline calculation * KAAV-3492: Fix timeline cascade and lautakunta date snapping - Add forward cascade in get_preview_deadlines() using distances_to_next - Detect visibility bool False→True to trigger distance enforcement on re-add - Snap dates to valid date_type in _enforce_distance_requirements() (ensures lautakunta dates always land on valid Tuesdays) - Add test files for deadline lifecycle and data completeness * KAAV-3492: Always snap deadlines to valid date_type in preview Per AT2.5.1: Deadline must be a work day. Now all deadlines with date_type are snapped via get_closest_valid_date(), not just those needing distance enforcement. Fixes July dates not correcting to August. * KAAV-3492 Iterates through all deadlines after phase boundaries are recalculated Skips deadlines already processed or in the original cascade Checks if any distances_to_previous rules are now violated Enforces the minimum distance if violated (with log message [KAAV-3492 POST-CALC]) * KAAV-3492 Refactoring code to clear cache and include attribute enforcement in the convergence loop. Removing isolated post-calc loop before integration. Replacing convergence loop with integrated enforcement logic. * KAAV-3492 adding logic fixes * KAAV-3492 refactored code * KAAV-3942 fix confirmed fields saving * Potential fix for pull request finding 'Unused local variable' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * removed unused variable * KAAV-3492 reverted files not needed and modified gitignore * removed non needed files * removed settings.json from branch * test for deadline lifecycle * KAAV-3492 fix backend silently not updating correct changes from updated_attribute_data to response and possibly messing up some validations * removed some faulty log infos * Clean up unused imports and fix iteration variable bug * modofied tests * KAAV-3492 Fix in _set_calculated_deadline: Checks preview_attribute_data before using calculated values, preventing the convergence loop from overwriting enforced or user-provided values. Missing preview_attribute_data parameter: Added the parameter to the first _set_calculated_deadlines call, ensuring preview data flows through the entire calculation chain. * Fix auto-calculated deadlines using stale preview values Skip preview values for deadlines with edit_privilege=None. Ensures periaatteetvaihe_paattyy_pvm recalculates to match esillaolo_paattyy. * KAAV-3492: Fix timeline cascade using minimum distances instead of initial distances - Use distance_from_previous (minimum) instead of initial_distance for cascade calculations - Prevent cross-phase cascade for non-boundary deadlines - Fix phase prefix detection order (tarkistettu_ehdotus before ehdotus) - Remove debug logging statements * prevent backward date snap when addin Change condition from > to < in get_preview_deadlines() so minimum distances only enforce forward movement, not backward snapping. * Initial positioning to be used when adding new elements and continuing with minium distance validation * temp logging * KAAV-3492 audit and cleanup command for possible stale data on project * (KAAV-3492): Prevent stale deadline dates from being created - Root cause: _set_calculated_deadline() wrote dates to attribute_data regardless of visibility bool value, causing stale data in all project sizes (not just XL) - Added visibility bool check before writing dates to prevent creation of stale data when vis_bool=False - Added clean_stale_deadline_fields() call in update_initial_data() to cleanup orphaned dates when XL project toggles are disabled - Created shared deadline_utils.py module with DEADLINE_GROUP_DATE_FIELDS constant and utility functions for detection/cleanup - Reduced code duplication in audit/cleanup management commands * unused import remove * - Add user_changed_fields param to _set_calculated_deadlines to skip recalculation for explicitly changed deadlines - Exclude user-changed fields from update_dls_to_calc - Update deadline_utils and serializer for fix support * fixes * Test to be more reliable * test fixed for github * removed useless tests * fixes * fix: exclude auto-calculated deadlines from actually_changed set Phase boundaries (edit_privilege=None) now always recalculate when inner deadlines cascade forward, per AT1.2.1/AT1.2.3. * modified comments to be cleaner for the chanhes * fix: phase boundaries recalculate after cascade; remove debug logging - Exclude auto-calculated deadlines (edit_privilege=None) from actually_changed set so E9/T5 phase boundaries always recalculate when inner deadlines cascade forward (per AT1.2.1/AT1.2.3) - Cache calculate_initial/calculate_updated querysets on Deadline instance to avoid repeated DB queries in convergence loop - Remove all debug logging from project.py, serializers/project.py, section.py, and views.py * sonar cloud suggestions * Add timeline_save mode with raw preview validation - Add timeline_save query param to skip cascade during validation - New get_raw_deadline_preview() validates user's exact input - Fix deadline distance lookup to check conditions (not just .first()) - Skip deadline recalculation in update_deadlines during save - Process ALL phases during timeline_save validation * tests added * recalculate RE-ENABLED deadlines from predecessor When a deadline group's visibility bool changes False→True, existing deadlines now recalculate from their predecessor using distances_to_previous instead of keeping stale dates. This fixes the lautakunta ehdotus 3 bug where E8.3 kept old date (2030-11-19) causing E4 to jump 300 days forward. * re-enabling phases condition fix so they actually can be moved after re-enabling * removed unsused code * prevent date recalculation during timeline_save - Add timeline_save parameter to update_deadlines() to skip _set_calculated_deadlines() during save - Accept ALL deadline attribute fields (including phase boundaries) in valid_attributes for timeline_save - Fixes dates shifting unexpectedly when saving timeline Per validation.md: Save Must NOT Recalculate Dates * U1 date to sync with K1 --------- Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
1 parent 4ebe30d commit 9270edf

27 files changed

Lines changed: 6731 additions & 252 deletions

‎.gitignore‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
11
# Backups
22
*~
33

4+
# Copilot instructions
5+
.github/copilot-instructions.md
6+
#vscode
7+
.vscode/
48
# Byte-compiled / optimized / DLL files
59
__pycache__/
610
*.py[cod]
@@ -116,3 +120,15 @@ media/
116120

117121
# MacOS
118122
.DS_Store
123+
124+
# Locally removed files
125+
010925_projektitiedot.xlsx
126+
1.0_aikataulutiedot.xlsx
127+
1.0_projektitiedot.xlsx
128+
121125_projektitiedot.xlsx
129+
200225_aikataulutiedot.xlsx
130+
230925_projektitiedot.xlsx
131+
media.zip
132+
test_251125_aikataulutiedot.xlsx
133+
test_261125_aikataulutiedot.xlsx
134+
~$040425_projektitiedot.xlsx

‎conftest.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
import os
2+
import django
3+
4+
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "kaavapino.settings")
5+
django.setup()

‎projects/deadline_utils.py‎

Lines changed: 254 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,254 @@
1+
"""
2+
Shared utilities for deadline management.
3+
4+
This module contains constants and functions used across deadline-related
5+
commands and features, particularly for KAAV-3492 stale deadline detection/cleanup.
6+
"""
7+
from projects.serializers.utils import VIS_BOOL_MAP
8+
9+
10+
# Map each deadline group to its associated date fields
11+
# These are the fields that should be cleared when the group is deleted
12+
DEADLINE_GROUP_DATE_FIELDS = {
13+
# Periaatteet esilläolo
14+
'periaatteet_esillaolokerta_1': [
15+
'milloin_periaatteet_esillaolo_alkaa',
16+
'milloin_periaatteet_esillaolo_paattyy',
17+
'periaatteet_esillaolo_aineiston_maaraaika',
18+
'viimeistaan_mielipiteet_periaatteista',
19+
],
20+
'periaatteet_esillaolokerta_2': [
21+
'milloin_periaatteet_esillaolo_alkaa_2',
22+
'milloin_periaatteet_esillaolo_paattyy_2',
23+
'periaatteet_esillaolo_aineiston_maaraaika_2',
24+
'viimeistaan_mielipiteet_periaatteista_2',
25+
],
26+
'periaatteet_esillaolokerta_3': [
27+
'milloin_periaatteet_esillaolo_alkaa_3',
28+
'milloin_periaatteet_esillaolo_paattyy_3',
29+
'periaatteet_esillaolo_aineiston_maaraaika_3',
30+
'viimeistaan_mielipiteet_periaatteista_3',
31+
],
32+
# Periaatteet lautakunta
33+
'periaatteet_lautakuntakerta_1': [
34+
'milloin_periaatteet_lautakunnassa',
35+
'periaatteet_lautakunta_aineiston_maaraaika',
36+
],
37+
'periaatteet_lautakuntakerta_2': [
38+
'milloin_periaatteet_lautakunnassa_2',
39+
'periaatteet_lautakunta_aineiston_maaraaika_2',
40+
],
41+
'periaatteet_lautakuntakerta_3': [
42+
'milloin_periaatteet_lautakunnassa_3',
43+
'periaatteet_lautakunta_aineiston_maaraaika_3',
44+
],
45+
'periaatteet_lautakuntakerta_4': [
46+
'milloin_periaatteet_lautakunnassa_4',
47+
'periaatteet_lautakunta_aineiston_maaraaika_4',
48+
],
49+
# OAS esilläolo
50+
'oas_esillaolokerta_1': [
51+
'milloin_oas_esillaolo_alkaa',
52+
'milloin_oas_esillaolo_paattyy',
53+
'oas_esillaolo_aineiston_maaraaika',
54+
'viimeistaan_mielipiteet_oas',
55+
],
56+
'oas_esillaolokerta_2': [
57+
'milloin_oas_esillaolo_alkaa_2',
58+
'milloin_oas_esillaolo_paattyy_2',
59+
'oas_esillaolo_aineiston_maaraaika_2',
60+
'viimeistaan_mielipiteet_oas_2',
61+
],
62+
'oas_esillaolokerta_3': [
63+
'milloin_oas_esillaolo_alkaa_3',
64+
'milloin_oas_esillaolo_paattyy_3',
65+
'oas_esillaolo_aineiston_maaraaika_3',
66+
'viimeistaan_mielipiteet_oas_3',
67+
],
68+
# Luonnos esilläolo
69+
'luonnos_esillaolokerta_1': [
70+
'milloin_luonnos_esillaolo_alkaa',
71+
'milloin_luonnos_esillaolo_paattyy',
72+
'kaavaluonnos_esillaolo_aineiston_maaraaika',
73+
'viimeistaan_mielipiteet_luonnos',
74+
],
75+
'luonnos_esillaolokerta_2': [
76+
'milloin_luonnos_esillaolo_alkaa_2',
77+
'milloin_luonnos_esillaolo_paattyy_2',
78+
'kaavaluonnos_esillaolo_aineiston_maaraaika_2',
79+
'viimeistaan_mielipiteet_luonnos_2',
80+
],
81+
'luonnos_esillaolokerta_3': [
82+
'milloin_luonnos_esillaolo_alkaa_3',
83+
'milloin_luonnos_esillaolo_paattyy_3',
84+
'kaavaluonnos_esillaolo_aineiston_maaraaika_3',
85+
'viimeistaan_mielipiteet_luonnos_3',
86+
],
87+
# Luonnos lautakunta
88+
'luonnos_lautakuntakerta_1': [
89+
'milloin_kaavaluonnos_lautakunnassa',
90+
'kaavaluonnos_kylk_aineiston_maaraaika',
91+
],
92+
'luonnos_lautakuntakerta_2': [
93+
'milloin_kaavaluonnos_lautakunnassa_2',
94+
'kaavaluonnos_kylk_aineiston_maaraaika_2',
95+
],
96+
'luonnos_lautakuntakerta_3': [
97+
'milloin_kaavaluonnos_lautakunnassa_3',
98+
'kaavaluonnos_kylk_aineiston_maaraaika_3',
99+
],
100+
'luonnos_lautakuntakerta_4': [
101+
'milloin_kaavaluonnos_lautakunnassa_4',
102+
'kaavaluonnos_kylk_aineiston_maaraaika_4',
103+
],
104+
# Ehdotus nähtävilläolo
105+
'ehdotus_nahtavillaolokerta_1': [
106+
'milloin_ehdotuksen_nahtavilla_alkaa_pieni',
107+
'milloin_ehdotuksen_nahtavilla_alkaa_iso',
108+
'milloin_ehdotuksen_nahtavilla_paattyy',
109+
'ehdotus_nahtaville_aineiston_maaraaika',
110+
'viimeistaan_lausunnot_ehdotuksesta',
111+
],
112+
'ehdotus_nahtavillaolokerta_2': [
113+
'milloin_ehdotuksen_nahtavilla_alkaa_pieni_2',
114+
'milloin_ehdotuksen_nahtavilla_alkaa_iso_2',
115+
'milloin_ehdotuksen_nahtavilla_paattyy_2',
116+
'ehdotus_nahtaville_aineiston_maaraaika_2',
117+
'viimeistaan_lausunnot_ehdotuksesta_2',
118+
],
119+
'ehdotus_nahtavillaolokerta_3': [
120+
'milloin_ehdotuksen_nahtavilla_alkaa_pieni_3',
121+
'milloin_ehdotuksen_nahtavilla_alkaa_iso_3',
122+
'milloin_ehdotuksen_nahtavilla_paattyy_3',
123+
'ehdotus_nahtaville_aineiston_maaraaika_3',
124+
'viimeistaan_lausunnot_ehdotuksesta_3',
125+
],
126+
'ehdotus_nahtavillaolokerta_4': [
127+
'milloin_ehdotuksen_nahtavilla_alkaa_pieni_4',
128+
'milloin_ehdotuksen_nahtavilla_alkaa_iso_4',
129+
'milloin_ehdotuksen_nahtavilla_paattyy_4',
130+
'ehdotus_nahtaville_aineiston_maaraaika_4',
131+
'viimeistaan_lausunnot_ehdotuksesta_4',
132+
],
133+
# Ehdotus lautakunta
134+
'ehdotus_lautakuntakerta_1': [
135+
'milloin_kaavaehdotus_lautakunnassa',
136+
'ehdotus_kylk_aineiston_maaraaika',
137+
],
138+
'ehdotus_lautakuntakerta_2': [
139+
'milloin_kaavaehdotus_lautakunnassa_2',
140+
],
141+
'ehdotus_lautakuntakerta_3': [
142+
'milloin_kaavaehdotus_lautakunnassa_3',
143+
],
144+
'ehdotus_lautakuntakerta_4': [
145+
'milloin_kaavaehdotus_lautakunnassa_4',
146+
],
147+
# Tarkistettu ehdotus lautakunta
148+
'tarkistettu_ehdotus_lautakuntakerta_1': [
149+
'milloin_tarkistettu_ehdotus_lautakunnassa',
150+
'tarkistettu_ehdotus_kylk_aineiston_maaraaika',
151+
],
152+
'tarkistettu_ehdotus_lautakuntakerta_2': [
153+
'milloin_tarkistettu_ehdotus_lautakunnassa_2',
154+
'tarkistettu_ehdotus_kylk_aineiston_maaraaika_2',
155+
],
156+
'tarkistettu_ehdotus_lautakuntakerta_3': [
157+
'milloin_tarkistettu_ehdotus_lautakunnassa_3',
158+
'tarkistettu_ehdotus_kylk_aineiston_maaraaika_3',
159+
],
160+
'tarkistettu_ehdotus_lautakuntakerta_4': [
161+
'milloin_tarkistettu_ehdotus_lautakunnassa_4',
162+
'tarkistettu_ehdotus_kylk_aineiston_maaraaika_4',
163+
],
164+
}
165+
166+
167+
def find_stale_deadline_fields(attribute_data):
168+
"""
169+
Find stale deadline date fields in project attribute data.
170+
171+
A date field is considered stale when:
172+
1. The associated deadline group's visibility bool is False
173+
2. But the date field still has a value
174+
175+
Args:
176+
attribute_data (dict): The project's attribute_data dictionary
177+
178+
Returns:
179+
list: List of tuples (deadline_group, vis_bool_name, stale_fields_list)
180+
where stale_fields_list is a list of dicts with 'field' and 'value' keys
181+
"""
182+
stale_data = []
183+
attr_data = attribute_data or {}
184+
185+
for deadline_group, vis_bool_name in VIS_BOOL_MAP.items():
186+
# Skip groups without visibility bools (kaynnistys, hyvaksyminen, voimaantulo)
187+
if vis_bool_name is None:
188+
continue
189+
190+
# Get the visibility bool value
191+
vis_bool_value = attr_data.get(vis_bool_name)
192+
193+
# Only check if vis_bool is explicitly False
194+
if vis_bool_value is not False:
195+
continue
196+
197+
# Get date fields for this group
198+
date_fields = DEADLINE_GROUP_DATE_FIELDS.get(deadline_group, [])
199+
200+
# Check for stale dates
201+
stale_fields = []
202+
for field in date_fields:
203+
value = attr_data.get(field)
204+
if value is not None:
205+
stale_fields.append({
206+
'field': field,
207+
'value': value,
208+
})
209+
210+
if stale_fields:
211+
stale_data.append((deadline_group, vis_bool_name, stale_fields))
212+
213+
return stale_data
214+
215+
216+
def clean_stale_deadline_fields(attribute_data):
217+
"""
218+
Clear deadline date fields when their visibility bool is False.
219+
220+
When a visibility bool is explicitly False, sets associated date fields to None
221+
to prevent stale data from leaking through during dict merge.
222+
223+
Args:
224+
attribute_data (dict): The project's attribute_data (modified in-place)
225+
226+
Returns:
227+
int: Number of fields that were cleared
228+
"""
229+
if not isinstance(attribute_data, dict):
230+
return 0
231+
232+
cleared_count = 0
233+
234+
for deadline_group, vis_bool_name in VIS_BOOL_MAP.items():
235+
# Skip groups without visibility bools
236+
if vis_bool_name is None:
237+
continue
238+
239+
# Only clean if vis_bool is explicitly False
240+
vis_bool_value = attribute_data.get(vis_bool_name)
241+
if vis_bool_value is not False:
242+
continue
243+
244+
# Get date fields for this group
245+
date_fields = DEADLINE_GROUP_DATE_FIELDS.get(deadline_group, [])
246+
247+
# Set None to override DB values during merge: {**db_data, **request_data}
248+
for field in date_fields:
249+
# Only count as cleared if value was non-None (for idempotency)
250+
if attribute_data.get(field) is not None:
251+
cleared_count += 1
252+
attribute_data[field] = None
253+
254+
return cleared_count

0 commit comments

Comments
 (0)