Skip to content

CLAUDE.md

Latest commit

 

History

History
598 lines (539 loc) · 38.1 KB

CLAUDE.md

File metadata and controls

598 lines (539 loc) · 38.1 KB
Raw

Repo-specific context for Claude

This file is committed so any machine's Claude Code session starts from the same ground truth. Session-local memory (~/.claude/.../memory/) does not travel between machines — this file does. Keep it updated when a session establishes something a future session, on any machine, would need.

Standing rules (user, 2026-09-18) -- read first

  • Terminal SOC is always a SOFT constraint, in every algorithm (FilterDDP, centralized Ipopt, tADMM, run_bf): objective + gamma * sum_j (B_j^T - B_j^0)^2. gamma is fixed per system and never varies with the horizon T. Single source of truth: ddp/examples/power_system/terminal_soc_penalty.jl (ieee123 1.75e4, ieee2522 3.29e4, large10k 420). gamma is not C_B. Currently wired into the FilterDDP driver and centralized_ipopt_matched.jl behind TERMINAL_SOC_SOFT=1; tADMM and run_bf still need it before their next run.

  • C_B is fixed per system, one number each (user, 2026-09-26), sized so each battery's price-driven swing is 3x the system's MEDIAN battery rating: C_B = (c_max - cbar) / (2 * 3 * median P_rated) = ieee123 1.0e-3, med2522 1.9e-3, large10k 2.4e-5. Same single source file as gamma (CB_BY_SYSTEM), selected in both FilterDDP and Ipopt by REDUCED_CB=system. The earlier uniform C_B = 1e-3 left large10k's (417 kW median) batteries mostly idle (5% power use, 30% of the energy window); with these values every system uses 100% of its energy window. Derivation and checks: ddp/notes/IPOPT_WITHOUT_CB.md. C_B never varies with T. The diagonal Hessian relies on C_B > 0: at C_B = 0 diagonal-Hessian FilterDDP fails large10k T=6 (line search, iteration 2) and med2522 T=96, while the exact Hessian converges (large10k T=6, 108 iterations, no regularization). Never pair FILTERDDP_DIAG_HESSIAN=1 with C_B = 0. (Those failures were measured before the 2026-10-05 configuration. In it, both cases reach near-optimality with the diagonal Hessian at C_B = 0: large10k T=6 in 17 iterations, med2522 T=96 in 55. The rule stands until the user lifts it.)

  • Solver timing comparisons only on a provably identical problem: same exported instance, C_B, gamma and profile, with objective agreement checked before any timing is quoted. The paper's older centralized Ipopt sweep (ddp/results/centralized_ipopt/) is on a DIFFERENT instance family (C_B ~ 8.8e-8, old price sampling, free terminal SOC) -- do not race against it.

  • Near-optimality primal infeasibility thresholds are per-system (user, 2026-09-23). FilterDDP "time to near-optimality" (Table V in the TPEC paper) is the first iteration with objective within 0.5% of Ipopt AND primal_inf below the system threshold: ieee123 1e-6, ieee2522 1e-5, large10k 1e-5. Single source of truth: NEAR_OPT_PRIMAL_BY_SYSTEM in ddp/examples/power_system/terminal_soc_penalty.jl. The race script (run_matched_ipopt_race.sh) reads matching values. Do not use a uniform threshold across systems. large10k was 1e-4 until 2026-10-11. The user moved it to 1e-5 after seeing the clean batch re-read at both ("1e-5 is the sweet spot"; 1e-4 a bit too loose, 5e-5 too arbitrary): Ipopt's own infeasibility falls below 1e-5 only near the end of its solve, the diagonal FilterDDP iterate at the 0.5% point is already below it from T=24 up. Every large10k near-optimality figure dated before that, in the notes and in the TPEC paper (results table, screening-ablation column), is at 1e-4. Runs that stopped at near-optimality cannot be re-read at 1e-5; runs to tolerance (the clean batch) can (clean_hp_table.py).

  • Every timed FilterDDP run must enable full per-iteration logging (user, 2026-09-23). Set FILTERDDP_TIMING_DIAGNOSTIC=1 and FILTERDDP_FEASIBILITY_DIAGNOSTIC=1. This emits per-iteration: objective, primal_inf, dual_inf, complementarity (cs_inf), barrier parameter (mu), regularisation, step size, backtracks, and per-phase wall time (backward_s, forward_s, total_s). The race script extracts a trace CSV (extract_filterddp_feasibility_trace.jl) and the NO-point summary (near_opt_from_logs.jl) post-hoc from these logs. Without these env vars the run is unreportable — you cannot reconstruct the NO crossing point.

  • Report solve time, not compile time (user, 2026-10-05). Julia compiles each method on first use: 15-30 s, all inside FilterDDP's first iteration (14.7 of the 17.9 s once reported for ieee123 T=6). Timed FilterDDP runs set FILTERDDP_WARMUP=full on the Table II cells: the whole solve once, silently, then the timed solve from the same start, same iterates. (A few warm-up iterations are not enough: they leave about 1.7 s of compilation in code reached later in the solve.) At horizons of 192 or more a full extra solve is too long; use FILTERDDP_WARMUP=8 there. Every FilterDDP time measured before 2026-10-05 includes compilation.

  • Screened model and power-flow start, for both solvers (2026-10-05, see ddp/notes/CONSTRAINT_SCREENING.md). The substation voltage is fixed at exactly its upper limit, which stopped FilterDDP from ever taking a full step; every iteration count measured before 2026-10-05 carries that throttle. Current configuration: FILTERDDP_SCREEN=substation,vupper,ell,psubs, FILTERDDP_LOADFLOW_START=1, FILTERDDP_AFFINE_LINESEARCH=1. The reductions are exact (same optimum), but they and the start also speed Ipopt up (53 to 12 iterations on large10k T=6), so any comparison gives Ipopt the same: IPOPT_SCREEN=<same list>, IPOPT_LOADFLOW_START=1.

  • large10k had no PV real power until 2026-10-05. envs/tadmm/parse_opendss.jl read each PV rating as PVsystems.kW(), the unit's present output. large10k's units carry Daily=LoadShapePVDefault, so all 1,022 read as 0 kW (350 kW each, 357.7 MW in total); ieee123 and ieee2522 have no daily shape and were unaffected. Found by the OpenDSS check (ddp/examples/power_system/opendss_check.jl), fixed to read Pmpp. The large10k instances were re-exported; the old ones are kept locally as network_data_large10kC_1ph_T*_periodic_nopv.jls. Every large10k result dated before this, for either solver, is on the PV-less system (still a matched comparison); logs on the corrected instance carry the tag pv, and Ipopt references must come from it. The MSOPF parsers in envs/multi_poi still use the same call. Standing practice (user): check a dispatch against OpenDSS before trusting an instance or a result.

  • Logs over 100 MB cannot be pushed to GitHub. Per-iteration logs at horizons of several hundred steps reach 75-140 MB: commit them gzipped, and check after every push that the branch is not still ahead of the remote.

  • Exact rewrites of 2026-10-06, and the parallel-in-time count (ddp/notes/PARALLEL_IN_TIME.md, Section 9). FILTERDDP_LEAN_VALUE=1 (identical iterates) and FILTERDDP_HALF_BLOCK=1 (battery block at half size; iterates agree to rounding, check with compare_runs.py) make the sequential run 43-44% faster on large10k and 24-27% on med2522 than the 2026-10-05 configuration. They are in the TPEC paper since 2026-10-06 (its results table and Section "Cost of One Iteration") but not yet in the standard run scripts other than run_parsim.sh; for ieee123 (UMFPACK, no tree solver) pass FILTERDDP_LEAN_VALUE=1 to run_blocked_solve_fullrun.sh. FILTERDDP_PARSIM=1 times each stage as what needs nothing from t+1 / what needs the value function / what follows it, and parsim_from_log.py counts one worker per period (the slowest period for the independent parts). The user accepts that simulated count ("like my tADMM Ipopt solve"); always state that it assumes one core per period, no communication, and that Ipopt is a single process.

  • NO_COLOR in the environment defeats the warm-up (found 2026-10-06). Julia then wraps stdout in an IOContext and every print is compiled again in the timed solve (0.7 s; 1.5-2.3 s with the PARSIM lines). Processes started from PowerShell carry it. run_parsim.sh and run_blocked_solve_fullrun.sh unset it; the driver prints WARMUP_STREAM_MISMATCH when it happens. Any new timed launcher must unset NO_COLOR and FORCE_COLOR too. Do not edit solver sources while a queued batch is running: each run loads them afresh.

  • Scope of interest (user, 2026-10-06). Methods must help every test system: no speedup that exploits the topology of one system and does nothing for another (large10k is, to 1e-7, 102 independent feeders; that is a caveat about the test system, not a method to build). Horizons of interest are the practical ones, T=6 to T=96 (up to a few hundred at most): med2522 T=1152 and T=1536 were cancelled and are not to be requeued. A horizon must mean something (user, 2026-10-11): one day at a standard step. Beyond T=96 (15 minutes, the standing goal-post) the points are T=144 (10 minutes) and T=288 (5 minutes), not T=192. Several days is a possible later step, once a solve is cheap enough. Naming them is not a request to run them: the user stopped queued T=288 runs the same day ("I only want 144", for med2522, large10k optional). Ask before any T=288 run.

  • The OpenDSS check is part of every run (user, 2026-10-11). Stop the timer, keep the dispatch, hand it to OpenDSS, log the discrepancy, then start the next run; there is no timing conflict in that order. run_parsim.sh does it by default (OPENDSS_CHECK=1); any new launcher must, the centralized ones included ("everything goes into OpenDSS"). Never set FILTERDDP_SKIP_SOLUTION_WRITE=1 in a run whose result may be reported.

  • Stopping rules (user, 2026-10-10). Near-optimality always means the objective criterion AND the primal-infeasibility criterion together. FilterDDP runs go to a tolerance of 1e-5 and centralized runs to 1e-6, with the objective and the infeasibility of every iteration logged (IPOPT_TRACE=1 for Ipopt), and the reported criterion is chosen afterwards, the same for both solvers. Two primal infeasibilities exist and differ a great deal: the internal one (an inequality counted through its slack: FilterDDP's primal_inf, Ipopt's callback) and the as-posed one (Ipopt's iteration table, FILTERDDP_ORIGINAL_PRIMAL). Never compare one solver's internal figure with the other's as-posed one.

  • Power plan. The 309 PC under High performance is 14-24% faster per iteration than under Balanced (same runs, 2026-10-05/06 against 10-09; indicative). Every time in the TPEC table up to 2026-10-10 was measured under Balanced, for both solvers; the clean batch of 2026-10-11 (run_clean_hp_batch.sh, logs tagged hp) is under High performance. Never mix the two in one comparison; record the plan with the run.

  • Exact Hessian and the pairwise sweep (2026-10-10/11, ddp/notes/PARALLEL_IN_TIME.md, Section 10). FILTERDDP_LEAN_EXACT=1 (with FILTERDDP_HALF_BLOCK=1, no FILTERDDP_DIAG_HESSIAN) builds the halved battery block before the sweep: same iterates as the plain exact path, and its step B costs about what the diagonal path's does. pint_scan.jl (FILTERDDP_PINT_SCAN=1) redoes the value recursion by pairwise combination of the periods (Sarkka and Garcia-Fernandez, 2023): exact, slower at T=6, 1.7-2.0 times faster at T=24 and 4.6-5.1 times at T=96 on step B, as a count. Since the night of 2026-10-11 a solve uses it (paired_pass.jl, bullet below), and the TPEC paper's counted column is the exact Hessian with the pairwise sweep; the diagonal Hessian stays the faster sequential arrangement on the two larger feeders.

  • Gurobi works on the 309 PC (Gurobi 13, licence to 2027-09-22) and is run for reference (user, 2026-10-06 and 2026-10-11). It needs v and ell nonnegative to recognise the cone, so its reduced model keeps those two and drops the other screened limits; its barrier method takes no starting point for a conic problem (Gurobi documentation, "Warm Start Options"), so the power-flow start has no effect on it.

  • Night of 2026-10-11 (ddp/notes/PARALLEL_IN_TIME.md, Section 11; branch ddp-oct10-new-tests). Three things a later session should know:

    • FILTERDDP_REUSE_BARRIER=1: the backward pass that solve! repeats after lowering the barrier parameter keeps every stage's factor (no stage matrix depends on that parameter). Identical iterates, checked on all three systems with both Hessians. Use it in timed runs from now on; logs carry the tag reuse.
    • Barrier schedule and hybrid Hessian, 132 untimed runs: Ipopt's schedule (FILTERDDP_MU_INIT=0.1, _THETA_MU=1.5) cuts the iterations to tolerance (252 to 187 over six cases with the diagonal Hessian) but is mixed at the 0.5% rule and costs iterations on large10k; the hybrid (FILTERDDP_HYBRID_MU) gains nothing. The default schedule stays until the user decides otherwise.
    • FILTERDDP_THREADED=1 (threaded_pass.jl): the backward and forward passes with one task per period, same iterates as the sequential run (bit for bit backward, to rounding forward). Correctness only: the user does not want parallel timings from the 309 PC. It is the code for a machine with a core per period; a Kamiak account was requested on 2026-10-11 (ticket KSD-7186, PI Dubey).

  • The pairwise sweep in the solve, and the collector in the count (2026-10-11 night; ddp/notes/PARALLEL_IN_TIME.md, Section 12).

    • paired_pass.jl does the battery sweep by pairs inside the solve: same iterations and objective as the ordinary exact solve on all twelve cases. Launcher: bash ddp/results/paired_pass/run_paired.sh <system> <T> count <tag> (OpenDSS check included; PAIRED_ROUNDS=0 runs the ordinary sweep in the same pass for a count on the same timing; PAIRED_WARMUP=full, 8 on large10k T=96). The driver has no switch of its own for it yet.
    • The count charges garbage collection per worker (user, 2026-10-11: "our default assumption", and the paper must say so). Julia's collector takes 17-22% of a run; its pauses used to land inside one stage's time and make it the slowest. Counting runs now time each stage without the pauses in it and print the collector's time (FILTERDDP_PARSIM_GCNET=1, set by the launcher), and the scripts give each worker 1/T of it; clean_hp_table.py --gc=sequential gives the other reading. Logs before this (the clean batch, tags count_hp_tol1e5 and inplace_hp_tol1e5) are on the old timing: never put their counts beside the new ones as like for like. Always state the three assumptions: one core per period, no time for passing data, a memory per worker; and that Ipopt is one process. A real parallel run should be one process per period, not threads in one program.
    • Result at the 0.5% rule (same at 0.1%), Ipopt / DDP counted, seconds: med2522 13.3 / 4.2 (T=24), 67.8 / 8.4 (T=96), 110.2 / 14.4 (T=144); large10k 21.9 / 12.4, 44.4 / 15.8 (T=48), 101.4 / 22.6 (T=96). At T=6: level on med2522, half Ipopt's speed on large10k. With every collection in sequence only large10k T=48 and T=96 stay ahead. One process runs the pairwise arrangement slower than the ordinary sweep.
    • Do not retry "run the collector less often": held off for a whole pass and run once it costs the same (its cost follows the volume allocated), and the allocation has no single source.
    • Longer horizons (user's go, 2026-10-12): med2522 T=288 counts 26.4 s against 255.2 s for Ipopt (9.7 times). large10k T=144 counts 80.6 s against 138.9 s, but the PC (31.7 GB) runs short of memory there and the count carries the stalls (about 45 s without them, an estimate): say so wherever that figure appears (an asterisk in the paper's table, explained in the text). A clean rerun needs the other applications closed; the user parked it on 2026-10-12 until there is access to a parallel machine. large10k T=288 cannot run on this PC. Warm-up 12 for med2522 T=288, 8 for large10k T=144.
    • ieee123 T=24 needs tolerance 1e-6 (PAIRED_TOL) to meet its 1e-6 infeasibility threshold.

  • The paper targets IEEE Transactions on Power Systems (user, 2026-10-12; it was drafted as an Industry Applications paper). Its GitHub repository was renamed on 2026-10-13 to Realife-Brahmin/TPWRS-2027-Sparse-Parallel-DDP-for-MPOPF, and the local folder to documents_general/TPWRS-2027-Sparse-Parallel-DDP-for-MPOPF (it was TPEC-2027-DDP-MPOPF: "the TPEC repo" and "the TPEC paper" in older notes mean this one). Its two linked worktrees under .codex/worktrees/ were repaired after the move. PES rules, from the PES authors' page: 10 pages at submission (returned without review above that), 12 before mandatory overlength charges; abstract of about 150-200 words in one paragraph with no abbreviations, references or equations; three or four keywords; AI-generated content must be disclosed in the Acknowledgments (system, sections, level of use). As of that date the abstract (111 words, with abbreviations, written before the results), the keywords and the acknowledgment were still to do, and are the user's to word. The algorithm's name is Sparse Parallel DDP (user, 2026-10-13), in the title and defined in the introduction; "DDP" stays the shorthand in the body and tables. The user's wishes for the front matter: an abstract in the conventional order (problem, structure, "this paper presents", the strategies in a sentence each, tests, comparison), horizons given as a day at a resolution (288 periods = 5 minutes, 144 = 10 minutes), the method not presented as resting on the network being radial (say tree-like or sparse; the elimination needs only that loops stay in the kept block, which is not implemented), and index terms that are specific (elimination tree, parallel prefix scan, Riccati recursion, Schur complement), in alphabetical order. Two claims the user floated are not supported by the runs and must stay out unless measured: that the method uses less memory than the centralized solver (on one PC it used more: 17-20 GB against 13.6 GB at large10k T=144), and that it solves cases the centralized solver cannot (Ipopt with MA57 solved every case run).

Working branch — the name below is a snapshot, not the truth

Any branch named in this file may already be stale. It records what was current the last time someone edited this line, not what is current now. As of 2026-09-17 this section said ddp-understanding-sep14 while the remote's newest branch was ddp-understanding-sep15 — assume that kind of drift by default.

Always resolve the branch from the remote before starting work:

git fetch origin --prune
git for-each-ref --sort=-committerdate refs/remotes/origin \
    --format='%(committerdate:short)  %(refname:short)'

Then pick the branch whose name matches the task you were given, taking the latest date among those — not simply the newest branch in the list. The date suffix breaks ties within a workstream; it does not make an unrelated workstream's branch the right base.

Workstreams run on parallel dated branches and are expected to coexist. The three use cases in this repo (tADMM, DDP, and MSOPF — the environment still on disk as envs/multi_poi/, being renamed MultiSourceOPF) share the same underlying optimization but are separate lines of work with separate branches. DDP work belongs on its own dated branch (ddp-understanding-<date>, ddp-hessian-rewrites-<date>, ...); MSOPF work belongs on its own, e.g. msopf-initial-testing-sep17. Do not retarget MSOPF work onto the current DDP branch (or vice versa) merely because that branch is newest.

These branches are not expected to conflict: MSOPF's task is building a working OpenDSS-backed workflow, not running heavy optimization, so it touches largely different files. If you do find yourself resolving conflicts between two workstream branches, stop and confirm the base branch was chosen correctly before continuing.

For genuinely new work, cut a new dated branch rather than reusing an existing one, and name it for the workstream it serves.

MSOPF quick facts (2026-09-28)

  • Setup, drivers and decks: envs/multi_poi/README.md. The full-angle drivers run in envs/tadmm (tracked Manifest; checked from a fresh clone), not envs/multi_poi.
  • The paper is the separate PESGM2027_Multi-Source-Multi-Period-OPF repo. Its Section II documents solve_full_angle in envs/multi_poi/full_angle_pf.jl, so keep the two in sync. Do not confuse it with the rejected PESGM2026 repo.
  • Decks ieee123_5poi_1ph and small2poi_1ph carry a realistic source impedance, Zs = 0.032 + j0.799 ohm (cited in the deck comments). ideal_sources = true gives the old stiff sources.
  • Battery model: one variable P_B per battery, lossless, with an optional C_B * P_B^2 cycling cost (the user's standard). Feasibility means the network equations, no-backflow (P_Subs >= 0, the key constraint) and the 0.95-1.05 pu band on buses not set by a substation.

The solver is DDP4OPF, committed at ddp/DDP4OPF.jl

Since 2026-09-15 the DDP solver is our own MIT fork of FilterDDP.jl, committed as source, named DDP4OPF with a fresh UUID (005a218a-...). using DDP4OPF everywhere; FilterDDP is no longer resolvable in envs/ddp2026. Mingda Xu's MIT notice is retained in its LICENSE, and NOTICE.md records the upstream repo, the fork point 513a104, and what changed. Credit the original papers.

Why it was forked rather than kept as a clone plus patches: the working solver existed only as uncommitted edits in a gitignored clone, and the documented patch recipe no longer rebuilt it (3 of 9 patches failed on a clean 513a104; the result differed in 4 source files). The fork was verified bit-identical to that clone -- 0 of 2526 full-space and 0 of 459 reduced-space solution entries differed on ieee123 T=3.

  • A new name and UUID were required, not cosmetic: FilterDDP is registered in the Julia General registry, and Julia identifies packages by UUID.
  • ddp/patches/ is history, not a build recipe. Do not try to rebuild from it.
  • Environment variables still use the FILTERDDP_ prefix on purpose: sweep and queue scripts set them. Renaming them is a separate decision.
  • ddp/external/FilterDDP.jl (gitignored) is now only needed to reproduce the authors' own shipped example (Stage 3).
  • Historical notes in ddp/notes/ still cite ddp/external/FilterDDP.jl paths and line numbers. They record the clone as it was at the time; leave them.

Centralized IPOPT timing sweep (complete)

That sweep is finished across all three systems and all matrix horizons, with the TPEC table/caption/PDF updated. The procedure below is kept for reference. Reconstruct the IAS-style centralized C (s) column with fresh JuMP--IPOPT runs by following ddp/notes/CENTRALIZED_IPOPT_TIMING_SWEEP.md. Use scripts/run_centralized_ipopt_case.ps1, one case at a time. A case is not finished until its validated row and raw log are pushed here and the matching TPEC table/PDF update is pushed to the TPEC repository. Resume from the first incomplete row; never replace missing data with an older Gurobi timing.

Two DDP codebases here — both Differential Dynamic Programming

Naming: the user's method is DIFFERENTIAL Dynamic Programming. It is never "Distributed." Corrected throughout on 2026-08-14, including the two source files that originally carried the wrong expansion (envs/ddp/tex/ddp_copperplate_formulation.tex, envs/ddp/root_level/ddp_copperplate.jl). An earlier version of this file recorded "Distributed" as a deliberate word choice; that was wrong. Do not reintroduce it anywhere, and do not describe the two codebases as different algorithms that merely share an initialism — they are both DDP, differing in order. (Unrelated: "Distributed Energy Resources" in envs/multi_poi/ and envs/tadmm/ is correct and must be left alone; and the repo name's "Dist" means distribution networks.)

  • ddp/ (repo root) is the FilterDDP evaluation: a reproducibility study of the authors' solver (github.com/mingu6/FilterDDP.jl, arXiv 2504.08278 / 2606.01487), done 2026-08-03 to present. Full status, findings, and the MPOPF-fit conclusion are in ddp/README_FILTERDDP_EXPERIMENT.md. The paper is at ddp/resources/Xu_2026_FilterDDP.pdf (and a duplicate at envs/ddp/resources/Xu_2026_FilterDDP.pdf, see manifests below).

  • envs/ddp/ is the user's own, independently-derived first-order DDP scheme for copper-plate MPOPF (committed 2025-11-12, predates any Claude involvement — verified by git blame, not AI-authored). A clean minimal restatement of it lives at ddp/examples/power_system/user_ddp.jl.

The difference is the order of the cost-to-go model, not the family:

envs/ddp/ (user's own) ddp/ (FilterDDP, the paper)
Backward information Passes μ[t], the dynamics-constraint dual, backward one stage per outer forward sweep (envs/ddp/root_level/ddp_copperplate.jl:539-550, mu_prev/mu_coupling) Backward pass sweeps t=N→1 within one iteration, building both V_x and V_xx (Riccati recursion, backward_pass.jl in the FilterDDP clone)
Order of approximation First-order only: the coupling term μ[t+1]·(B[t+1]−B[t]+Δt·P_B[t+1]) is linear in B[t] — no curvature crosses a stage boundary Second-order: full local quadratic model of the cost-to-go
Per-stage solve Calls an external solver (Gurobi/Ipopt) per stage per sweep In-house sparse nu×nu KKT solve (UMFPACK) inside the backward pass, with filter line-search globalization. An earlier version of this row said "dense"; measured at large10k a dense factorisation would need ~540 s and 22 GB per stage against 2.25 s actual
Propagation speed Information from stage T reaches stage 1 after roughly T outer iterations (one stage per sweep) Full-horizon propagation in one backward sweep per Newton-type iteration

Important nuance established 2026-08-05: μ[t] in the user's method is legitimately the same object as V_x[t] (the costate / value-gradient) — this is not a naive approach, it's a genuine first-order DDP. The gap is the missing curvature (V_xx) and the cross-iteration staleness, not the absence of backward-looking information altogether.

Measured 2026-08-14 on the shared T = 6 instance, via user_ddp.jl: the first-order scheme's fixed point is correct — fed the true μ* and B*, one sweep reproduces the centralized optimum to 1.7e-08 — but it does not converge to it from a cold start, settling into a period-two limit cycle 2.1e-03 short in objective and 121 kW out in dispatch. Damping shrinks the cycle without closing it (a = 0.1 still 3.2e-05 short after 2000 sweeps), because started at the optimum the sweep returns μ off by 9.2e-04: the optimum is not exactly a fixed point of the linearised map. FilterDDP reaches 2.2e-10 on the same instance in 16 iterations. That is the V_xx gap, quantified.

Julia environment: use envs/ddp2026 for everything DDP

Since 2026-08-07 there is one environment for all DDP work — FilterDDP, the centralized JuMP/Ipopt reference, and any further formulation: envs/ddp2026 (README). FilterDDP and JuMP coexist without conflict. Stages 5, 6 and 8 all reproduce under it.

  • envs/ddp/Project.toml (2025-11) is superseded. Its Plots/Crayons/ Revise deps existed because verification then meant a human reading formatted output. Treat envs/ddp/ as a read-only reference for what the user's own algorithm did — not expected to be run again.
  • ddp/env (the FilterDDP-only env) was removed 2026-08-07. Its Manifest was never tracked, so it preserved nothing envs/ddp2026/Project.toml doesn't, and Stages 5/6 reproduce identically under the new env. It's in git history if ever needed.
  • Pkg strips all comments from Project.toml on every add/resolve, so put rationale in a README beside it, never in the file.
  • Julia via the Bash tool, not PowerShell: PowerShell mangles quotes in julia -e '...' and has silently corrupted Project.toml this way.

Current copper-plate formulation

P_Subs^t has no upper bound — only P_Subs^t ≥ 0 (no export upstream). This is the latest formulation, confirmed by the user 2026-08-07. Older notes referring to an active Psub[2] ≤ 1.35 or ≤ 1.45 are stale; the scripts and logs were already correct and the experiment README has been fixed to match.

The ddp/paper/ write-up (including copper_plate_model.tex) was deleted 2026-09-08 — it encoded a superseded formulation. The authoritative formulation now lives in the TPEC repo, not here. The modeling rationale that section used to carry (why no η, why the terminal target is a penalty, why C_B = 0.5 here vs. ≈10⁻⁶·min c^t in the tADMM paper) survives only in ddp/README_FILTERDDP_EXPERIMENT.md.

Instance data is now the tADMM profiles (changed 2026-08-07 at the user's request). Demand and price come from envs/tadmm/root_level/config.jl, shared via ddp/examples/power_system/tadmm_profiles.jl. Three things to know:

  • They resample with T — tadmm_cost(3) is NOT tadmm_cost(6)[1:3]. Never slice a fixed vector.
  • Committed results are T = 6 only (T = 3 removed 2026-08-07 at the user's request; regenerating another T when analysis needs it is expected). But inspect the price before adopting a new T: at T = 3 the samples land where sin vanishes (0, π, 2π), so the price comes out constant and the instance carries no arbitrage signal at all. That, plus binding bounds only a few kW wide, made T = 3 both the weakest benchmark and the worst behaved numerically (26 iterations, trajectory agreement only 1.9e-05).
  • This closed the earlier collinearity concern: r went from 0.9968 to 0.644 at T = 6, thanks to the −0.8 rad phase offset on the load.

C_B = 0.05 since 2026-08-07 (was 0.5). This is the COPPER-PLATE value and does not apply to the network cases: the exported network_data_*.jls files carry C_B = 1.4e-07 with dt = 8 (the tADMM regime). Established 2026-09-15 -- the difference is not cosmetic, see the reduced-space section below. C_B sets how far the battery moves: P_B^k = (c^k - 2wΔt·s)/(2C_B), so the swing scales as 1/C_B — the bounds are not the knob. At 0.05 the T = 6 battery swings ±570 kW against a 1623-1998 kW load and cycles B from 2000 down to 1070 kWh: 46% depth of discharge, matching the ieee123C_1ph battery/load ratio of 44%. cond(Q) = 601, so the closed-form and active-set references stay exact to ~1e-14.

Still not tADMM's own C_B = 1e-6·min(c) ≈ 8e-8: there cond(Q) ≈ 3.6e8 and the battery goes purely bound-limited (±340 GW absent bounds) — the tADMM regime, but useless as a precision reference. An earlier note that this makes Q rank-1 was an overstatement: Q = 2C_B Δt I + 2w Δt² 11ᵀ is PD for any C_B > 0, rank-1 only in the limit.

Plot battery quantities in kW and kWh, never p.u. (user preference, 2026-08-07). Base is tADMM's kVA_B = 1000: 1 p.u. = 1000 kW, 1 p.u.h = 1000 kWh. The model and Table I stay in p.u.; the figures convert. Reference asset sizes, for sanity checks: ads10A_1ph 87 kW load / 4.7 kW batt; ieee123C_1ph 1163 kW / 507 kW / 2027 kWh; ieee123_5poi_1ph 1163 kW / 1318 kW / 5273 kWh. SOC runs 30%-95% with B_0 at 62.5%.

Figures are generated, never hand-written was the rule for the (now-deleted) ddp/paper/figures/ — make_figure_data.jl read the verified centralized reference and emitted balance.csv, schedule_interval.csv, schedule_soc.csv for the .tex files to read. Keep the same discipline (generated data, no inlined coordinates) if/when figures are rebuilt against the TPEC repo's formulation.

Reduced-space MPOPF (2026-09-14/15): works, and C_B governs whether it is usable

Full write-up and raw data: ddp/notes/REDUCED_SPACE_INNER_OPF_FEASIBILITY.md, ddp/results/reduced_space/. The reduced-space work did not modify the solver (now ddp/DDP4OPF.jl) -- the OCP is assembled from hand-written closures, which is forced anyway since an Ipopt solve is not automatically differentiable.

The network can be eliminated exactly: FilterDDP optimises battery quantities only while an inner single-period OPF recovers every network variable, with the real-power balance duals supplying dPhi/dP_B for free. Verified against the stored full-space FilterDDP solutions to 6.0e-09 (ieee123) and 1.3e-08 (ieee2522) relative objective.

RETRACTED 2026-09-15 -- read before trusting the table below. Every T = 3 instance has exactly zero price spread: the profile generator sampled sin at 0, pi, 2pi. With no arbitrage signal the batteries barely move (5% of rating at large10k), so the table measures an idle-battery problem. It is a valid timing on that instance and not evidence the decomposition works on a real scheduling problem. On the first instance with a real price signal (ieee2522 T = 12, 147% spread, batteries up to 97% of rating) the reduced method fails to converge while full space converges. Why is still open. Use PROFILE_PERIODIC=1 exports (118% spread at T = 3); the exporter now warns on flat profiles. Full account: ddp/notes/FINDINGS_2026-09-14_15.md.

Measured crossover at T = 3, C_B = 1e-3, both formulations re-run at the same C_B (degenerate instances -- see retraction above):

full-space reduced (low-rank) outcome
ieee123 (nu 791 -> 102) 14.4 s / 46 it 21.7 s / 26 it 1.5x slower
ieee2522 (nu 13358 -> 500) 107.3 s / 56 it 107.1 s / 33 it parity
large10k (nu 54665 -> 2040) 1658.2 s / 100 it 802.5 s / 13 it 2.07x faster

That was read as "the decomposition is a LARGE-system technique"; given the retraction, that conclusion is unsupported until it is re-measured on non-degenerate instances. Outer iteration counts run the other way with size (full-space 46/56/100, reduced 26/33/13), and at large10k inner solves are only 25% of wall -- the bottleneck has moved to FilterDDP's own outer cost.

C_B decides whether any of this works. It enters the stage Hessian only as a perfectly-conditioned diagonal 2*C_B*S^2*dt, and on ieee2522 d2Phi spans -0.148 .. 21194. At the exported C_B = 1.4e-07 that diagonal is 2.24, so there is effectively no damping (cond ~10131) and EVERY cheap curvature model fails -- battery-term-only, low-rank Nystrom, and floored Nystrom all stall. Only an exact per-stage Hessian converges, and it is too expensive to win. At C_B = 1e-3 the diagonal is 16000 (cond 2.32) and the low-rank Hessian converges, which is what produces the table above. Size C_B from measurement: 2*C_B*S^2*dt > -lambda_min(d2Phi) is mandatory for positive definiteness, and ~ lambda_max/kappa sets conditioning; lambda_max costs ~10 inner solves by power iteration since H*d is one solve.

Two further results worth not re-deriving: the battery-power box is not recourse-feasible in general (true on ieee123, false on ieee2522 -- 8 of 111 dispatches rejected, all undervoltage), and F_t is horizon-independent, so T only selects which load/PV snapshots get tested.

What is and isn't verified (as of 2026-08-07)

FilterDDP is cross-checked against a centralized JuMP/Ipopt solve of eq:cp_all on all six T = 6 cases (Stage 8 of the experiment README): exact on the base instance, worst objective gap 2.2e-10 and worst trajectory gap 6.9e-09 with bounds, and Ipopt independently certifies the 6g bound set infeasible. The base instance carries three mutually independent references (closed form, dense KKT, Ipopt). Stage 9 records the per-iteration trace — 17 iterations, regularisation never firing. Do not redo any of this.

Not verified, so don't overclaim: anything with a network (no LinDistFlow, no BFM); any horizon other than T = 6; and the user's own DDP algorithm, which has not been compared at all — that is the pending task below.

Pending task (do not start until asked)

Blocked as of 2026-09-08: this task's shared reference point, ddp/paper/sections/copper_plate_model.tex, was deleted along with the rest of ddp/paper/ — that formulation is superseded and the authoritative one now lives in the TPEC repo. Before restarting this task, get the equivalent problem statement from the TPEC repo and re-approve it as the shared reference point; do not resurrect the deleted file from git history as a substitute.

Write a side-by-side workflow comparison — the user's exact DDP algorithm vs. FilterDDP's algorithm — both grounded in the exact problem statement of the "dummy paper" (previously ddp/paper/sections/copper_plate_model.tex, see blocker above). Requirements:

  • Use the user's own notation throughout: P_Subs^t, P_B^t, B^t, c^t, C_B, p_L^t, w, B_0, P_B^{min/max}, B^{min/max} (from copper_plate_model.tex), plus μ[t], λ_Bmin[t], λ_Bmax[t] (from envs/ddp/tex/ddp_copperplate_formulation.tex). No unqualified generic control-theory notation (L, l, Q, V_x, V_xx unless explicitly mapped to one of the user's own symbols first).
  • Every dual/auxiliary variable that plays the same role in both workflows should be given the same symbol in both descriptions, with the correspondence stated explicitly (e.g. μ[t] ↔ V_x[t], the box-constraint duals λ_Bmin[t]/λ_Bmax[t] vs. FilterDDP's interior-point bound multipliers).
  • Where a concept exists in FilterDDP but has no analog in the user's method (e.g. V_xx/curvature), say so explicitly rather than inventing a correspondence.

Reference-paper bookkeeping

Every resources/ folder (ddp/resources/, envs/ddp/resources/, envs/multi_poi/resources/) has a MANIFEST.txt (filename | url | description). Run bash scripts/fetch_resources.sh from repo root to fetch anything missing — safe to re-run, present files are left alone. Blank url fields mean the source isn't tracked down yet; fill in when found.