MPE bug

[Sleantt]

Hi,

As stated in #128 i am currently trying to implement a ProbLog classifier to solve the Kaggle challenge "Titanic : Machine learning from a disaster". While using the mpe modality of ProbLog to classify a set of passengers and test the classifiers accuracy, i encountered a bug.

problog mpe titanic_problog/last_run/samples.pl
Traceback (most recent call last):
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/tasks/mpe.py", line 164, in main_mpe_maxsat
    prob, output_facts = mpe_maxsat(
                         ~~~~~~~~~~^
        dag, verbose=args.verbose, solver=args.solver, minpe=args.minpe
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/tasks/mpe.py", line 199, in mpe_maxsat
    result = frozenset(solver.evaluate(cnf, invert_weights=minpe))
                       ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/maxsat.py", line 63, in evaluate
    output = self.call_process(inputf)
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/maxsat.py", line 57, in call_process
    return subprocess_check_output(self.command + [filename])
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/util.py", line 166, in subprocess_check_output
    raise subprocess.CalledProcessError(retcode, cmd, output=err)
subprocess.CalledProcessError: Command '['/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/bin/linux/maxsatz', '/tmp/tmpqk6nhkmi.cnf']' returned non-zero exit status 1.

An unexpected error has occurred.
(False, CalledProcessError(1, ['/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/bin/linux/maxsatz', '/tmp/tmpqk6nhkmi.cnf']))

The classification is based on this model :

% age(X) : X is a supported age group. This predicates serves as a mean to
% divide each passengers in age groups.
t(_)::age(0);t(_)::age(1);t(_)::age(2);t(_)::age(3).
t(_)::sex(0);t(_)::sex(1).
t(_)::pclass(1);t(_)::pclass(2);t(_)::pclass(3).

% person(ID, Sex, PassengerClass, Age) characterizes a titanic passenger
% with their passenger id, sex, passenger class and age
person(_ID, Sex, PassengerClass,Age) :- 
  sex(Sex), 
  pclass(PassengerClass), 
  age(Age).

% survived(PassengerId, Survived) : the passenger with the given Id survived if the Survived atom is
% equal to '1'.
t(_, Sex, PassengerClass, Age)::survived(PassengerId, 1); t(_, Sex, PassengerClass, Age)::survived(PassengerId, 0) :- 
  person(PassengerId, Sex, PassengerClass, Age).

The samples file used here is around 1800 lines long and has the following format :

:-consult('/path/to/learned/model').
person(242,1,3, 1).
query(survived(242, 1)).
person(795,0,3, 1).
query(survived(795, 1)).
[...]

And here is the learned model if that helps :

0.183673469387755::age(0); 0.63265306122449::age(1); 0.154195011337868::age(2); 0.029478458049887::age(3).
0.649659863945578::sex(0); 0.350340136054422::sex(1).
0.241496598639456::pclass(1); 0.205215419501134::pclass(2); 0.55328798185941::pclass(3).
person(_ID,Sex,PassengerClass,Age) :- sex(Sex), pclass(PassengerClass), age(Age).
0.534090909090909::survived(PassengerId,1); 0.465909090909091::survived(PassengerId,0) :- person(PassengerId,1,3,1).
0.533333333333333::survived(PassengerId,1); 0.466666666666667::survived(PassengerId,0) :- person(PassengerId,1,3,0).
1.0::survived(PassengerId,1); 0.0::survived(PassengerId,0) :- person(PassengerId,1,3,3).
0.980392156862745::survived(PassengerId,1); 0.019607843137255::survived(PassengerId,0) :- person(PassengerId,1,1,1).
0.928571428571429::survived(PassengerId,1); 0.071428571428571::survived(PassengerId,0) :- person(PassengerId,1,1,0).
1.0::survived(PassengerId,1); 0.0::survived(PassengerId,0) :- person(PassengerId,1,1,3).
0.0::survived(PassengerId,1); 1.0::survived(PassengerId,0) :- person(PassengerId,1,3,2).
0.958333333333333::survived(PassengerId,1); 0.041666666666667::survived(PassengerId,0) :- person(PassengerId,1,1,2).
0.909090909090909::survived(PassengerId,1); 0.090909090909091::survived(PassengerId,0) :- person(PassengerId,1,2,1).
1.0::survived(PassengerId,1); 0.0::survived(PassengerId,0) :- person(PassengerId,1,2,0).
0.0::survived(PassengerId,1); 0.0::survived(PassengerId,0) :- person(PassengerId,1,2,3).
0.875::survived(PassengerId,1); 0.125::survived(PassengerId,0) :- person(PassengerId,1,2,2).
0.133603238866397::survived(PassengerId,1); 0.866396761133603::survived(PassengerId,0) :- person(PassengerId,0,3,1).
0.426229508196721::survived(PassengerId,1); 0.573770491803279::survived(PassengerId,0) :- person(PassengerId,0,1,1).
0.19047619047619::survived(PassengerId,1); 0.80952380952381::survived(PassengerId,0) :- person(PassengerId,0,3,0).
0.571428571428571::survived(PassengerId,1); 0.428571428571429::survived(PassengerId,0) :- person(PassengerId,0,1,0).
0.0::survived(PassengerId,1); 1.0::survived(PassengerId,0) :- person(PassengerId,0,3,3).
0.142857142857143::survived(PassengerId,1); 0.857142857142857::survived(PassengerId,0) :- person(PassengerId,0,1,3).
0.064516129032258::survived(PassengerId,1); 0.935483870967742::survived(PassengerId,0) :- person(PassengerId,0,3,2).
0.333333333333333::survived(PassengerId,1); 0.666666666666667::survived(PassengerId,0) :- person(PassengerId,0,1,2).
0.074626865671642::survived(PassengerId,1); 0.925373134328358::survived(PassengerId,0) :- person(PassengerId,0,2,1).
0.526315789473684::survived(PassengerId,1); 0.473684210526316::survived(PassengerId,0) :- person(PassengerId,0,2,0).
0.25::survived(PassengerId,1); 0.75::survived(PassengerId,0) :- person(PassengerId,0,2,3).
0.058823529411765::survived(PassengerId,1); 0.941176470588235::survived(PassengerId,0) :- person(PassengerId,0,2,2).

At first i couldn't make sense of the error but after fiddling with the library i found out that it was an out of memory error :

def subprocess_check_output(*popenargs, **kwargs):
    [...]
    process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
    try:
        output, err = process.communicate()
        retcode = process.poll()
        print(f"{output}\r\n{err}\r\n{retcode}\r\n") # <-- This is my alteration
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
            raise subprocess.CalledProcessError(retcode, cmd, output=err)
        return output.decode()
    except KeyboardInterrupt:
        kill_proc_tree(process)
        raise
    except SystemExit:
        kill_proc_tree(process)
        raise
    [...]

Output :

problog mpe titanic_problog/last_run/samples.pl           
b'c ----------------------------\nc - Weighted Partial MaxSATZ -\nc ----------------------------\nc Instance info: p wcnf 82941 284149 529846361505\nERROR: Out of memory.\n'
b''
1

Traceback (most recent call last):
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/tasks/mpe.py", line 164, in main_mpe_maxsat
    prob, output_facts = mpe_maxsat(
                         ~~~~~~~~~~^
        dag, verbose=args.verbose, solver=args.solver, minpe=args.minpe
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/tasks/mpe.py", line 199, in mpe_maxsat
    result = frozenset(solver.evaluate(cnf, invert_weights=minpe))
                       ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/maxsat.py", line 63, in evaluate
    output = self.call_process(inputf)
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/maxsat.py", line 57, in call_process
    return subprocess_check_output(self.command + [filename])
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/util.py", line 166, in subprocess_check_output
    raise subprocess.CalledProcessError(retcode, cmd, output=err)
subprocess.CalledProcessError: Command '['/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/bin/linux/maxsatz', '/tmp/tmpt7iwkxy7.cnf']' returned non-zero exit status 1.

An unexpected error has occurred.

Apparently when the file used with ProbLogs MPE modality is too long, the program runs out of memory.

I also tried using different solvers to see if the error would only occur with the 'maxsatz' solver :

problog mpe --solver sat4j titanic_problog/last_run/samples.pl
b''
b'Error: Unable to access jarfile /home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/bin/java/sat4j-maxsat.jar\n'
1

problog mpe --solver scip titanic_problog/last_run/samples.pl
Traceback (most recent call last):
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/tasks/mpe.py", line 164, in main_mpe_maxsat
    prob, output_facts = mpe_maxsat(
                         ~~~~~~~~~~^
        dag, verbose=args.verbose, solver=args.solver, minpe=args.minpe
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/tasks/mpe.py", line 199, in mpe_maxsat
    result = frozenset(solver.evaluate(cnf, invert_weights=minpe))
                       ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/maxsat.py", line 61, in evaluate
    inputf = self.prepare_input(formula, **kwargs)
  File "/home/lucam/Projects/PS6/.venv/lib/python3.13/site-packages/problog/maxsat.py", line 78, in prepare_input
    return formula.to_lp(**kwargs)
           ~~~~~~~~~~~~~^^^^^^^^^^
TypeError: CNF.to_lp() got an unexpected keyword argument 'invert_weights'

An unexpected error has occurred.
(False, TypeError("CNF.to_lp() got an unexpected keyword argument 'invert_weights'"))

But as you can see i am encountering other errors.

I am unsure whether i am misusing ProbLogs MPE modality or if the error is unexpected.

Thank you for your time.

[rmanhaeve]

Hi.

Thanks for already looking in-depth into this issue yourself. The model is indeed quite large, and I am unsure of what sizes we have tested ourselves. How much memory does the machine have you're running this on?

[Sleantt]

Hi,

Thank you for your reply.

Here are the specifications for my computer :

OS : Arch Linux x86_64
CPU : 12th Gen Intel(R) Core(TM) i7-12700H (20) @ 4.70 GHz
Memory (RAM) : 32GiB

While running ProbLog i did not notice any physical limitations being reached, ProbLog uses one core of my CPU and around 150 M of memory.

I also noticed while measuring this that maxsatz crashes nearly as soon as it starts, so no memory or CPU issue on this side either.

Tell me if you need more informations.

[rmanhaeve]

Ok, that seems like a reasonable amount of ram. Still the program might be too big. In any case, it is happening in an external tool (maxsatz). The only thing I can think of for now is looking at the program that is given as input here (the temporary file), and trying to isolate the call to the tool. We have (probably) dropped support for the other solvers.

[Sleantt]

Hi, while trying to work around the issue i think that i encountered another bug.

My workaround was to split the file in segments of around 200 lines, to allow maxsatz to function normally. However, i encountered an issue where problog would run indefinitely. After trying to narrow down the issue, i found out that my last file was the problem. The first 8 files are exactly 201 lines and problog can handle them fine, but the last one is shorter and problog gets stuck while processing it.

Weirdly, removing the last two lines seems to fix the issue and i have no idea why.

Here are the last four lines of the file :

[...]
person(236,1,3, 1).
query(survived(236, 1)).
person(38,0,3, 1).
query(survived(38, 1)).

Removing the third and fourth lines above allows the command :

problog mpe -v samples.txt -o  test.pl

To run perfectly fine. The same goes if the first and second lines are removed instead. Adding two lines instead of removing them makes the issue appear again however. I checked for duplicate ids and such but there aren't any. I cannot provide much more insight into this issue, i really don't understand where it could come from (and no errors are generated).

Here is the sample file that i'm working with if that helps.

[rmanhaeve]

I don't think the file you've provided is the correct one. Is it ProbLog that hangs, or the maxsatz subprocess?