Memory usage accumulation in CBC solver with python-mip wrapper

[quanhoang-pm]

The code below solves an IP relating to an $n\times n$ chessboard using CBC solver with python-mip wrapper (version 1.15.0). It solves the problem repeatedly with a time limit of 60 seconds. Within the given time limit, the program only produces a feasible solution. The IP itself does not matter, it only illustrates my question in the end.

import itertools

import mip
import numpy as np


def getAllCells(n):
    return tuple(itertools.product(range(n), repeat=2))

def sameMainDiagonal(cellA, cellB):
    return cellA[0] - cellA[1] == cellB[0] - cellB[1]

def sameAntiDiagonal(cellA, cellB):
    return cellA[0] + cellA[1] == cellB[0] + cellB[1]

def getAllNeighborCells(n, cell):
    allCells = getAllCells(n)
    sameRowCells = [x for x in allCells if x[0] == cell[0] and x != cell]
    sameColumnCells = [x for x in allCells if x[1] == cell[1] and x != cell]
    sameMajorDiagonalCells = [x for x in allCells if sameMainDiagonal(x, cell) and x != cell]
    sameAntiDiagonalCells = [x for x in allCells if sameAntiDiagonal(x, cell) and x != cell]
    allNeighbors = sameRowCells + sameColumnCells + sameAntiDiagonalCells + sameMajorDiagonalCells
    assert len(set(allNeighbors)) == len(allNeighbors)
    return allNeighbors

n = 12

for iterationIndex in itertools.count(start=0):
    print(f'\n{iterationIndex = }')
    model = mip.Model(name='perQueenAttackAtMostOneQueen', solver_name='CBC')
    model.max_seconds = 60

    varsArr = np.empty(shape=(n, n), dtype=object)
    for i in range(n):
        for j in range(n):
            varsArr[i, j] = model.add_var(name=f'x_{i}_{j}', var_type=mip.BINARY)
    cellAndNeighborsDict = {cell: getAllNeighborCells(n, cell) for cell in getAllCells(n)}
    for cell in getAllCells(n):
        arr = np.array(cellAndNeighborsDict[cell])
        nNeighbors = arr.shape[0]
        allNeighborVars = varsArr[
            arr[:, 0],
            arr[:, 1],
        ]
        x, y = cell
        model.add_constr((nNeighbors - 1) * varsArr[x, y] + allNeighborVars.sum() <= nNeighbors)

    model.objective = mip.maximize(varsArr.sum())
    model.optimize()
    resultArr = np.empty(shape=(n, n), dtype=int)
    for i in range(n):
        for j in range(n):
            resultArr[i, j] = varsArr[i, j].x
    print(resultArr)
    del model

During the execution on my laptop (Ubuntu 22.04LTS), I notice that the memory usage slowly increases after each iteration. Below are the statistics

iterationIndex = 2 --> 195 MB
iterationIndex = 6 --> 250 MB
iterationIndex = 14 --> 380 MB
iterationIndex = 30 --> 700 MB
iterationIndex = 35 --> 780 MB

Intuitively, the program's memory usage should be reset to 0 after each iteration (since the model variable is re-created). I've tried del model at the end of the iteration, but it does not resolve the problem.

I wonder what is the cause of the accumulation. Right now, I suspect that the search tree stored in CBC solver still exists even after it reaches the time limit.

My questions:

• Is there something happening under the hood in CBC solver that causes the accumulation of memory usage? Does it have anything to do with python-mip wrapper?
• How to completely remove the (memory usage of) existing models in python-mip?

Side note. This question has been asked on stackoverflow, but it has received no answer until now.

[rma-rripken]

I don't see anything in your code sample that would track the memory usage - how are you measuring that? Is this running in pypy or cpython?
I did the following:

conda create -c conda-forge -n "solvertest" pypy python=3.9.18 pip numpy
conda activate solvertest
pip install mip
pypy3 -m pip install mip
python solvertest.py

I let your example code run. Its 20 iterations in and so far the process memory usage according to Windows Task Manager is a consistently between 120 and 140MB.

[rma-rripken]

I left it running over-night. Got to iteration 710 and memory was still less than 150Mb. This was with mip 1.15.0, python 3.9.18, pypy 7.3.13.

[s-c-e]

I had the same issue, and posted my workaround in the python-mip discussion here: coin-or/python-mip#353 (comment)

Hope it helps!