Implement a faster and more memory-efficient version of the flow decomposition algorithm.

docs/flow_decomposition/algorithm-description.md

@@ -104,6 +104,9 @@ The following matrices are calculated using [sensitivity analysis](inv:powsyblco
 - $\mathrm{PTDF}$ is the matrix of the sensitivity of the network element flow to each network injection shift,
 - $\mathrm{PSDF}$ is the matrix of the sensitivity of the network element flow to each phase shift transformer tap angle change,
 
+> **_NOTE:_** When *DIRECT_SENSITIVITY_BASED* flow partitioner is used, those matrices aren't calculated explicitly, and both sensitivity analysis and flow partitioning is done "at once".
+> As a result, PTDF and PSDF matrices cannot be reported anymore.
+
 ## Flow partitioning
 
 Based on previously calculated elements, flow partitioning can now be calculated as follows:

docs/flow_decomposition/configuration.md

@@ -2,19 +2,20 @@
 
 ## Dedicated parameters
 
-| Name                                      | Type    | Default value | Description                                                                                                                                                                                                                                                                                                                                                                  |
-|-------------------------------------------|---------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| enable-losses-compensation                | boolean | false         | When set to true, adds losses compensation step of the algorithm. Otherwise, all losses will be compensated using chosen power flow compensation strategy.                                                                                                                                                                                                                   |
-| losses-compensation-epsilon               | double  | 1e-5          | Threshold used in losses compensation step of the algorihm. If actual losses are below the given threshold on a branch, no injection is created in the network to compensate these losses. Used to avoid creating too many injections in the network. May have an impact in overall algorithm performance and memory usage.                                                  |
-| sensitivity-epsilon                       | double  | 1e-5          | Threshold used when filling PTDF and PSDF matrices. If a sensitivity is below the given threshold, it is set to zero. Used to keep sparse matrices in the algorithm. May have an impact in overall algorithm performance and memory usage.                                                                                                                                   |
-| rescale-mode                              | enum    | NONE          | Use NONE if you don't want to rescale flow decomposition results. Use ACER_METHODOLOGY for the ACER methodology rescaling strategy. Use PROPORTIONAL for a proportional rescaling. Use MAX_CURRENT_OVERLOAD for a rescaling based on AC current overloads. See [Flow parts rescaling](../flow_decomposition/algorithm-description.md#flow-parts-rescaling) for more details. |
-| proportional-rescaler-min-flow-tolerance  | double  | 1e-6          | Option used from rescale modes PROPORTIONAL and MAX_CURRENT_OVERLOAD. Defines the minimum DC flow required in MW for the rescaling to happen.                                                                                                                                                                                                                                |    
-| dc-fallback-enabled-after-ac-divergence   | boolean | true          | Defines the fallback behavior after an AC divergence Use True to run DC loadflow if an AC loadflow diverges (default). Use False to throw an exception if an AC loadflow diverges.                                                                                                                                                                                           |
-| sensitivity-variable-batch-size           | int     | 15000         | When set to a lower value, this parameter will reduce memory usage, but it might increase computation time                                                                                                                                                                                                                                                                   |
+| Name                                     | Type    | Default value | Description                                                                                                                                                                                                                                                                                                                                                                  |
+|------------------------------------------|---------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| enable-losses-compensation               | boolean | false         | When set to true, adds losses compensation step of the algorithm. Otherwise, all losses will be compensated using chosen power flow compensation strategy.                                                                                                                                                                                                                   |
+| losses-compensation-epsilon              | double  | 1e-5          | Threshold used in losses compensation step of the algorihm. If actual losses are below the given threshold on a branch, no injection is created in the network to compensate these losses. Used to avoid creating too many injections in the network. May have an impact in overall algorithm performance and memory usage.                                                  |
+| sensitivity-epsilon                      | double  | 1e-5          | Threshold used when filling PTDF and PSDF matrices. If a sensitivity is below the given threshold, it is set to zero. Used to keep sparse matrices in the algorithm. May have an impact in overall algorithm performance and memory usage.                                                                                                                                   |
+| rescale-mode                             | enum    | NONE          | Use NONE if you don't want to rescale flow decomposition results. Use ACER_METHODOLOGY for the ACER methodology rescaling strategy. Use PROPORTIONAL for a proportional rescaling. Use MAX_CURRENT_OVERLOAD for a rescaling based on AC current overloads. See [Flow parts rescaling](../flow_decomposition/algorithm-description.md#flow-parts-rescaling) for more details. |
+| proportional-rescaler-min-flow-tolerance | double  | 1e-6          | Option used from rescale modes PROPORTIONAL and MAX_CURRENT_OVERLOAD. Defines the minimum DC flow required in MW for the rescaling to happen.                                                                                                                                                                                                                                |    
+| dc-fallback-enabled-after-ac-divergence  | boolean | true          | Defines the fallback behavior after an AC divergence Use True to run DC loadflow if an AC loadflow diverges (default). Use False to throw an exception if an AC loadflow diverges.                                                                                                                                                                                           |
+| sensitivity-variable-batch-size          | int     | 15000         | When set to a lower value, this parameter will reduce memory usage, but it might increase computation time.                                                                                                                                                                                                                                                                  |
+| flow-partitioner                         | enum    | MATRIX_BASED  | Use DIRECT_SENSITIVITY_BASED for better performance. However, nodal PTDF aren't explicitely calculated anymore and won't be reported. Use MATRIX_BASED if all detailed node PTDF needs to be reported.                                                                                                                                                                       |
 
 ## Impact of existing parameters
 
 Any implementation of load flow provider and sensitivity analysis provider can be used, as the entire algorithm only
 relies on common loadflow API and sensitivity analysis API.
 
-Thus, flow decomposition algorithm relies on [load flow parameters](inv:powsyblcore:*:*#loadflow-generic-parameters) and [sensitivity analysis parameters](inv:powsyblcore:*:*#sensitivity-generic-parameter).
+Thus, the flow decomposition algorithm relies on [load flow parameters](inv:powsyblcore:*:*#loadflow-generic-parameters) and [sensitivity analysis parameters](inv:powsyblcore:*:*#sensitivity-generic-parameter).

flow-decomposition/pom.xml

@@ -76,11 +76,6 @@
             <artifactId>jimfs</artifactId>
             <scope>test</scope>
         </dependency>
-        <dependency>
-            <groupId>com.powsybl</groupId>
-            <artifactId>powsybl-config-test</artifactId>
-            <scope>test</scope>
-        </dependency>
         <dependency>
             <groupId>com.powsybl</groupId>
             <artifactId>powsybl-cgmes-conformity</artifactId>

flow-decomposition/src/main/java/com/powsybl/flow_decomposition/AbstractSensitivityAnalyser.java

@@ -14,7 +14,6 @@
 import com.powsybl.iidm.network.Network;
 import com.powsybl.loadflow.LoadFlowParameters;
 import com.powsybl.sensitivity.*;
-import org.jgrapht.alg.util.Pair;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -25,15 +24,15 @@
 /**
  * @author Hugo Schindler {@literal <hugo.schindler at rte-france.com>}
  */
-abstract class AbstractSensitivityAnalyser {
+public abstract class AbstractSensitivityAnalyser {
     public static final SensitivityFunctionType SENSITIVITY_FUNCTION_TYPE = SensitivityFunctionType.BRANCH_ACTIVE_POWER_1;
     public static final List<Contingency> CONTINGENCIES = Collections.emptyList();
     private static final Logger LOGGER = LoggerFactory.getLogger(AbstractSensitivityAnalyser.class);
     private static final boolean DC_LOAD_FLOW = true;
     protected final SensitivityAnalysisParameters sensitivityAnalysisParameters;
     protected final SensitivityAnalysis.Runner runner;
 
-    AbstractSensitivityAnalyser(LoadFlowParameters loadFlowParameters, SensitivityAnalysis.Runner runner) {
+    protected AbstractSensitivityAnalyser(LoadFlowParameters loadFlowParameters, SensitivityAnalysis.Runner runner) {
         this.sensitivityAnalysisParameters = initSensitivityAnalysisParameters(loadFlowParameters);
         this.runner = runner;
     }
@@ -51,21 +50,21 @@ private static LoadFlowParameters enforceDcLoadFlowCalculation(LoadFlowParameter
         return dcEnforcedParameters;
     }
 
-    protected List<Pair<String, String>> getFunctionVariableFactors(List<String> variableList,
+    protected List<FunctionVariableFactor> getFunctionVariableFactors(List<String> variableList,
                                                                     List<Branch> functionList) {
-        List<Pair<String, String>> factors = new ArrayList<>();
+        List<FunctionVariableFactor> factors = new ArrayList<>();
         variableList.forEach(
             variable -> functionList.forEach(
-                function -> factors.add(Pair.of(function.getId(), variable))));
+                function -> factors.add(new FunctionVariableFactor(function.getId(), variable))));
         return factors;
     }
 
-    protected static SensitivityFactorReader getSensitivityFactorReader(List<Pair<String, String>> factors, SensitivityVariableType sensitivityVariableType, boolean sensitivityVariableSet) {
+    protected static SensitivityFactorReader getSensitivityFactorReader(List<FunctionVariableFactor> factors, SensitivityVariableType sensitivityVariableType, boolean sensitivityVariableSet) {
         return handler -> factors.forEach(
             pair -> handler.onFactor(SENSITIVITY_FUNCTION_TYPE,
-                pair.getFirst(),
+                pair.functionId(),
                 sensitivityVariableType,
-                pair.getSecond(),
+                pair.variableId(),
                 sensitivityVariableSet,
                 ContingencyContext.none()));
     }

flow-decomposition/src/main/java/com/powsybl/flow_decomposition/CsvExporter.java

@@ -6,6 +6,7 @@
  */
 package com.powsybl.flow_decomposition;
 
+import com.powsybl.iidm.network.Country;
 import org.apache.commons.csv.CSVFormat;
 import org.apache.commons.csv.CSVPrinter;
 import org.slf4j.Logger;
@@ -51,16 +52,16 @@ void export(Path dirPath, String basename, Map<String, DecomposedFlow> decompose
             BufferedWriter writer = Files.newBufferedWriter(path, CHARSET);
             CSVPrinter printer = new CSVPrinter(writer, FORMAT)
         ) {
-            Set<String> allLoopFlowKeys = aggregateAllLoopFlowKeys(decomposedFlowMap);
+            Set<Country> allLoopFlowKeys = aggregateAllLoopFlowKeys(decomposedFlowMap);
             printHeaderRow(allLoopFlowKeys, printer);
             printContentRows(decomposedFlowMap, allLoopFlowKeys, printer);
         } catch (IOException e) {
             throw new UncheckedIOException(e);
         }
     }
 
-    private Set<String> aggregateAllLoopFlowKeys(Map<String, DecomposedFlow> decomposedFlowMap) {
-        return decomposedFlowMap.values().stream().flatMap(decomposedFlow -> decomposedFlow.getLoopFlows().keySet().stream()).collect(Collectors.toSet());
+    private Set<Country> aggregateAllLoopFlowKeys(Map<String, DecomposedFlow> decomposedFlowMap) {
+        return decomposedFlowMap.values().stream().flatMap(decomposedFlow -> decomposedFlow.getFlowPartition().loopFlowPerCountry().keySet().stream()).collect(Collectors.toSet());
     }
 
     private void failSilentlyPrint(CSVPrinter printer, Object valueToPrint) {
@@ -79,28 +80,28 @@ private void failSilentlyPrintLn(CSVPrinter printer) {
         }
     }
 
-    private void printHeaderRow(Set<String> loopFlowKeys, CSVPrinter printer) {
+    private void printHeaderRow(Set<Country> loopFlowKeys, CSVPrinter printer) {
         failSilentlyPrint(printer, EMPTY_CELL_VALUE);
         failSilentlyPrint(printer, DecomposedFlow.ALLOCATED_COLUMN_NAME);
         failSilentlyPrint(printer, DecomposedFlow.INTERNAL_COLUMN_NAME);
         failSilentlyPrint(printer, DecomposedFlow.PST_COLUMN_NAME);
-        loopFlowKeys.stream().sorted().forEach(loopFlowKey -> failSilentlyPrint(printer, loopFlowKey));
+        loopFlowKeys.stream().sorted().forEach(loopFlowKey -> failSilentlyPrint(printer, NetworkUtil.getLoopFlowIdFromCountry(loopFlowKey)));
         failSilentlyPrint(printer, DecomposedFlow.AC_REFERENCE_FLOW_1_COLUMN_NAME);
         failSilentlyPrint(printer, DecomposedFlow.AC_REFERENCE_FLOW_2_COLUMN_NAME);
         failSilentlyPrint(printer, DecomposedFlow.DC_REFERENCE_FLOW_COLUMN_NAME);
         failSilentlyPrintLn(printer);
     }
 
-    private void printContentRows(Map<String, DecomposedFlow> decomposedFlowMap, Set<String> allLoopFlowKeys, CSVPrinter printer) {
+    private void printContentRows(Map<String, DecomposedFlow> decomposedFlowMap, Set<Country> allLoopFlowKeys, CSVPrinter printer) {
         decomposedFlowMap.forEach((xnecId, decomposedFlow) -> printContentRow(xnecId, decomposedFlow, allLoopFlowKeys, printer));
     }
 
-    private void printContentRow(String xnecId, DecomposedFlow decomposedFlow, Set<String> allLoopFlowKeys, CSVPrinter printer) {
+    private void printContentRow(String xnecId, DecomposedFlow decomposedFlow, Set<Country> allLoopFlowKeys, CSVPrinter printer) {
         failSilentlyPrint(printer, xnecId);
         failSilentlyPrint(printer, decomposedFlow.getAllocatedFlow());
         failSilentlyPrint(printer, decomposedFlow.getInternalFlow());
         failSilentlyPrint(printer, decomposedFlow.getPstFlow());
-        allLoopFlowKeys.stream().sorted().forEach(loopFlowKey -> failSilentlyPrint(printer, decomposedFlow.getLoopFlows().getOrDefault(loopFlowKey, NO_FLOW)));
+        allLoopFlowKeys.stream().sorted().forEach(loopFlowKey -> failSilentlyPrint(printer, decomposedFlow.getFlowPartition().loopFlowPerCountry().getOrDefault(loopFlowKey, NO_FLOW)));
         failSilentlyPrint(printer, decomposedFlow.getAcTerminal1ReferenceFlow());
         failSilentlyPrint(printer, decomposedFlow.getAcTerminal2ReferenceFlow());
         failSilentlyPrint(printer, decomposedFlow.getDcReferenceFlow());