Merge pull request #63 from IBM/v0.5.7.0b

[0.5.7.0]- 2024-11-26

Changed

• BACKWARD INCOMPATIBILITY - ScenarioDbManager.init: changed default values for db_type=DatabaseType.SQLite. For other uses (DB2 or PostgreSQL, always specify the db_type.
• BACKWARD INCOMPATIBILITY - ScenarioDbManager.init: changed default values for enable_scenario_seq=True, future=True. This reflects the current best practices.
• BACKWARD INCOMPATIBILITY - Removed ScenarioDbManager from dse_do_utils.__init__.py. This avoids the dependency on sqlalchemy with use of dse_do_utils where the ScenarioDbManager is not used.
  Introduces a slight backward incompatibility. Need to import as: from dse_do_utils.scenariodbmanager import ScenarioDbManager
• Removed (deprecated) module_reload() from dse_do_utils.__init__.py. In notebooks autoreload works well,
• Generics in ScenarioRunner
• Removed deprecated optional argument dtypes from Core01DataManager.prepare_output_data_frames()
• Fixed mutable default arguments in scenariodbmanager module

Added

• CplexDot in core01_optimization_engine: generic function class to use groupby aggregation and the mdl.dot() function.
• PlotlyManager - self.ref_dm, self.ms_inputs, self.ms_outputs property declarations and documentation
• PlotlyManager.plotly_kpi_compare_bar_charts and .get_multi_scenario_table for scenario compare
• Core01DataManager and Core01OptimizationEngine: added support for parameter mipGap. Sets the mdl.parameters.mip.tolerances.mipgap if value > 0
• DataManager.extract_solution adds option allow_mixed_type_columns. If True allows dvar/expr in column to be a regular Python value and not have the solution_value attribute

v0.5.6.0

[0.5.6.0]- 2023-05-13

Changed

• Bumping version to 0.5.6.0 given the amount of changes and additions

[0.5.5.2b6] - 2024-05-12

Added

• DeployedDOModel.monitor_execution_v1 efficiency improvement: only polls for the 'status'
• DeployedDOModel.solve refactoring
• DeployedDOModel retrieves the full log file
• DeployedDOModel increased default hardware spec nodes
• DOModelDeployerLocal to deploy DO model in WML from a local (non-CP4D/wx.ai) machine

[0.5.5.2b5] - 2024-05-07

Added

• ScenarioDbManager.duplicate_scenario_in_db returns scenario_name

Fixed

• ScenarioRunner.get_parameters can handle case where inputs do not (yet) have a 'Parameter' or 'Parameters' tab
• DataManager.extract_solution_v1 if argument round_decimals = 0 then convert column type to int

[0.5.5.2b4] - 2024-04-10

Added

• DataManager.extract_solution_v1 argument round_decimals to allow rounding of solution value
• DataManager.extract_solution_v1 argument extract_dvar_names can also be a Dict[str, str] where the value if the solumn column name.
• DataManager.extract_solution_v1 argument solution_column_name_post_fix to allow custom post fix (instead of hard-coded 'Sol')
• ScenarioRunner logs Excel input file name
• utils.df_itertuples_with_index_names()
• Core02ScenarioGenerator and Core02ScenarioConfig: configure LexOptiLevels and LexOptiGoals
• Core01DataManager.remove_zero_quantity_output to filter rows from a df where a solution column is less than the threshold
• DataManager.get_parameter_value improved handling of datetime when reading directly from Excel

Fixed

• Small error in ScenarioRunner.insert_outputs_in_db

[0.5.5.2b3]- 2023-07-05

Added

• Core01EnvironmentManager.find_project_root_directory
• MapManager.add_bar_chart_in_map
• DataManager.extract_solution_v1 adds option to drop small values to zero

Changed

• Core01DataManager.prepare_df now merges local dtypes with self.dtypes, where overlapping local dtypes take priority

[0.5.5.2b2]- 2023-06-08

Added

• Core01OptimizationEngine, Core02OptimizationEngine, PlotlyManager - added support for DataManager generics
• Core01EnvironmentManager can define logging scope

Changed

• Core01DataManager logger now uses __name__
• Core01DataManager - removed deprecated logger methods

[0.5.5.2b1]- 2023-05-23

Changed

• Support enable_refine_conflict in Core02OptimizationEngine and ScenarioRunner
• Requires Pandas 2.0 which fixes an issue with SQLAlchemy 2.0 Future

[0.5.5.2b0]- 2023-05-02

Added

• DataManager class generics to OptimizationEngine, Core01OptimizationEngine

Changed

• core01_environment_manager.HostEnvironment is now an enum.IntEnum to allow comparing integer values
• OptimizationEngine.integer_var_series_s etc. define the dtype='object' to avoid FutureWarning
• ScenarioManager.write_data_to_excel switch from pd.ExcelWriter.save() to .close() to avoid FutureWarning

v0.5.5.1

[0.5.5.1]- 2023-03-06

Changed

• README documentation updates.
• Removing deprecated modules deployeddmmodelcpd21.py, scenariodbmanager2.py

Fixed

• setup.py to include the dse_do_utils.core subpackage

v0.5.4.5

[0.5.4.5] - 2022-12-08

Changed

• DeployedDOModel.get_solve_details_objective uses PROGRESS_CURRENT_OBJECTIVE instead of PROGRESS_BEST_OBJECTIVE and adds exception handling
• setup.py avoids import of dse_do_utils to get version

Added

• RunConfig.export_sav option

[0.5.4.5b4] - 2022-11-15

Added

• OptimizationEngine.create_do_model() to instantiate as CPOptimizer model
• OptimizationEngine CPOptimizer methods to create dvar as DataFrame columns
• OptimizationEngine.semicontinuous_var_series
• OptimizationEngine.semiinteger_var_series

Changed

• ScenarioDbManager: before any DB insert (bulk and row-by-row), replace NaN with None to avoid FK problems in DB2

[0.5.4.5b3] - 2022-10-17

Added

• Support for scenarioSeq in ScenarioDbManager

[0.5.4.5b2] - 2022-10-02

Changed

• ScenarioRunner and ScenarioGenerator refactoring

[0.5.4.5b1] - 2022-09-22

Added

• ScenarioRunner and ScenarioGenerator

[0.5.4.5b0] - 2022-09-06

Changed

• ScenarioDbManager.resolve_metadata_column_conflicts: resolve conflicts where a ScenarioDbTable subclass redefines a column.
• ScenarioDbManager._insert_table_in_db_by_row automatically inserts None for columns in defined schema, but missing in DataFrame.

v0.5.4.4

[0.5.4.4] - 2022-04-28

Added

• ScenarioManager.load_data_from_zip_csv_s: loading input data from a zipped archive of csv files
• ScenarioManager.write_data_to_zip_csv_s: writing input/output data as a zipped archive of csv files

Removed

• ScenarioManager.write_data_to_csv_s no longer adds files as data assets

v0.5.4.3

[0.5.4.3] - 2022-04-21

Changed

• Fixed indentation bug in domodeldeployer
• Update of CONTRIBUTING.md
• Format thousands for num rows in db-insert progress print message

Added

• DOModelDeployer support for gz/zip package files
• ScenarioManager.load_data_from_parquet and .write_data_to_parquet
• ScenarioDbManager.init added parameter enable_debug_print to print connection string

[Unreleased]## [0.5.4.3b0]

Changed

• Fixed bug in ScenarioDbManager._check_schema_name if schema is None (e.g. when using SQLite)
• ScenarioDbTable.insert_table_in_db_bulk selects columns present in both the df and the schema. Avoids error when column is defined in DB but not in df.

Added

• Added 'group' as reserved table name
• Added local_relative_data_path in ScenarioManager.init() to allow more flexibility in defining local paths
• ScenarioDbManager.init added parameter enable_astype: bool = True: If True, force data-type of DataFrame to match schema before (bulk) insert.

v0.5.4.2

[0.5.4.2] - 2022-02-24

Changed

• ScenarioDManager.read_scenario_input_tables_from_db now only returns Inputs (vs the tuple (Inputs,Outputs)). This could cause some backward incompatibility.
• OptimizationEngine.get_kpi_output_table now returns Dataframe with capitalized column names: 'NAME' and 'VALUE' to make consistent with DO Experiment kpis.

Added

• MapManager.get_tooltip_table to get a tooltip formatted as table
• ScenarioDbManager.read_multi_scenario_tables_from_db reads multiple scenarios in same DataFrames (for scenario compare feature)
• ScenarioDbTable.get_sa_column works also on reflected Table, which supports use of cell-updates with a AutoScenarioDbTable
• ScenarioDbManager._check_schema_name: warns if schema name is mixed case (which is causing problems).

v0.5.4.1

[0.5.4.1] - 2022-01-24

Changed

• Fixed bug in ScenarioDbManager._read_scenario_tables_from_db
• Fixed bug in ScenarioDbManager.update_scenario_output_tables_in_db
• Fixed bug in ScenarioManager.add_file_as_data_asset when using CPDaaS
• Fixed bug in ScenarioManager.load_data_from_excel when using CPDaaS
• DOModelDeployer working in CPD4.0.3

Added

• Added DataManager.set_parameters()
• Added print of num rows and columns inserting in ScenarioDbManager.update_scenario_output_tables_in_db
• ScenarioDbManager.insert_scenarios_from_zip
• ScenarioManager.insert_scenarios_from_zip
• AutoScenarioDbTable reflection to get SQLAlchemy table metadata from DB. Fixes bug using AutoScenarioDbTable.
• ScenarioManager.load_data_from_excel now accepts excel_file_name with and without .xlsx extension
• ScenarioManager.write_data_to_excel now accepts excel_file_name with and without .xlsx extension

v0.5.4.0

[0.5.4.0] - 2022-01-11

Changed

• ScenarioDbManager - Converted text SQL operations to SQLAlchemy operations to support any column-name (i.e. lower, upper, mixed, reserved words)
• Updated ScenarioDbManager.read_scenario_tables_from_db to selectively read tables from a scenario

Added

• ScenarioDbManager - Edit cells in tables
• ScenarioDbManager - Duplicate, Rename and Delete scenario
• ScenarioDbManager.read_scenario_input_tables_from_db main API to read input for solve
• ScenarioDbManager.update_scenario_output_tables_in_db main API to store solve output

v0.5.3.1

[0.5.3.1] - 2021-12-30

Changed

• (critical) ScenarioDbManager - Replaced OrderedDict with Dict as type. Was causing a syntax error.