What is a performant way to load a million+ facts?

[jeshan]

I have a lot of static data which I store in a file as simple Prolog terms. I'd then assert them with assertz/1.

Loading it seems never ending while I have been able to load it in a few seconds with Sicstus, Eclipse and SWI (latter with read_file_to_terms/3 which uses read_term/3 internally).

For Scryer, I tried the following minimal example:

read_file_to_terms(Path) :-
    open(Path, read, Stream, []),
    repeat,
    read_term(Stream, _Term, []),
    false.

So, what's the recommended way to load a lot of Prolog terms?

Note that I have liberty on how to write the term file.

I don't know if/how library pio could be made to work more efficiently than the above.

[pmoura]

As you have the liberty on how to write the term file, why not write a clause file and skip assert all together by simply consulting the file? Some post-processing not possible at the time the file is written?

P.S. Also note that e.g. SWI-Prolog use a foreign library to accelerate predicates such as read_file_to_terms/3; the Prolog version of the predicate is just a fallback in case the foreign library is not available.

[jeshan]

Can you clarify what you mean by a clause file? If you meant like a normal Prolog source file, it loads but it's not any faster as I understood that consult does a lot more work than the simpler read_term/3.

SWI-Prolog use a foreign library to accelerate predicates

That's good to know and is another reason why we should look to make it fast for Scryer too :)

[pmoura]

A normal Prolog file. If you're not doing any post-processing on the read terms, then you can consult the terms as clauses for static predicates. Depending on the Prolog system, consulting vs read terms & asserting may indeed be faster or slower; worth experimenting with, something you have already done for comparing the times?

[jeshan]

If you're not doing any post-processing on the read terms

Indeed, I'm not doing that.

consulting vs read terms & asserting may indeed be faster or slower

I have not tested every combination as I gave up, expecting it to complete within a few seconds. I know that for SWI consult wasn't faster. Scryer crashed similar to the other experiment that I mentioned in a recent comment below.

can consult the terms as clauses for static predicates

Can you explain with an example? e.g how would this be consulted (if not consult/1 directly) ?

couple(alice, bob).

language(prolog, [logic, declarative]).

father_son(bob, carl).

[UWN]

Could you provide concrete data?

ulrich@p0:/opt/gupu/scryer-prolog$ target/release/scryer-prolog -f
?- open('/dev/null',read,Stream), repeat, read_term(Stream, Term, []).
   Stream = '$stream'(0x7f4eb5c3e6b8), Term = end_of_file
;  Stream = '$stream'(0x7f4eb5c3e6b8), Term = end_of_file
;  Stream = '$stream'(0x7f4eb5c3e6b8), Term = end_of_file
;  Stream = '$stream'(0x7f4eb5c3e6b8), Term = end_of_file
;  ... .

So at least for your test case, it is not surprising that Scryer did not terminate.

[jeshan]

You're right: I had a bug in my less-than-ideal example but that's not the only reason it wasn't terminating. With the fixed code:

% elegance definitely not required here!
read_file_to_terms(Path) :-
    open(Path, read, Stream),
    repeat,
    read_term(Stream, Term, []),
    write(Term),
    (Term=end_of_file -> throw('the end'); true),
    false.

We can see that it is in fact taking a lot of time going through the million facts.

[jeshan]

Could you provide concrete data?

The data file could just be as simple as the following but with a million such examples

couple(alice, bob).

language(prolog, [logic, declarative]).

father_son(bob, carla).

[UWN]

(It helps a lot to have exactly one such example, maybe just a little program generating these facts.)

[jeshan]

I was about to post this and then saw your last message!

I didn't want to post this at first because I thought it would distract from the conversation

write_demo_file :-
    open('out/demo_file.pl', write, Stream),
    ((between(1, 1000000, N),
    write_term(Stream, fact(N), []),
    write(Stream, '.\n'),
    false) -> false; close(Stream)).

[jeshan]

On another note: I'm experimenting with writing the term file as a single fact with all the data as a single list in the fact's argument. It crashes with Scryer but works well with SWI:

thread 'main' has overflowed its stack
fatal runtime error: stack overflow
Aborted (core dumped)

[UWN]

What I have tired. A million facts f(N).. with

?- open(fiche_1604,write,S),(between(1,1000000,N),portray_clause(S,f(N)),false;close(S)).
   S = '$dropped_value'.

In Scryer that is ~64s vs ~35s in SWI.

ulrich@TU-Wien:~/scryer$ time /opt/gupu/scryer-prolog/target/release/scryer-prolog -f
?- 

real	0m1,022s
user	0m0,236s
sys	0m0,008s
ulrich@TU-Wien:~/scryer$ time /opt/gupu/scryer-prolog/target/release/scryer-prolog -f fiche_1604 
?- 

real	1m7,738s
user	0m59,047s
sys	0m5,149s
ulrich@TU-Wien:~/scryer$ time swi-prolog.swipl -f none
Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.15)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- ^D
% halt

real	0m1,766s
user	0m0,276s
sys	0m0,084s
ulrich@TU-Wien:~/scryer$ 
ulrich@TU-Wien:~/scryer$ time swi-prolog.swipl -f none fiche_1604 
Welcome to SWI-Prolog (threaded, 64 bits, version 8.5.15)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- ^D
% halt

real	0m35,918s
user	0m34,682s
sys	0m0,219s

[jeshan]

Check my last message. Were you able to load it back quickly?

[UWN]

Not clear why you just put between(1, 10, N) into write_demo_file/0 which just works. But essentially that's the same as I have shown above. I replaced it with between(1, 1000000,N). Not much of a difference to my test case above.

[jeshan]

yes, the 10 was to check with a smaller file first. I then tried with a million and will update the comment

[UWN]

And just for the record, SICStus is here as fast as SWI.

[UWN]

To summarize: Scryer is currently about a factor of two slower than SWI for consulting facts.

[jeshan]

If fast loading isn't available yet, should I close this discussion and open a tracking issue instead?

[UWN]

What exactly is the problem? Being a factor two slower than a systems that is more than ten times as old?

[jeshan]

Being a factor two slower

Consulting may be twice as slow but I'm not describing the same thing.

What exactly is the problem?

The question in the title still stands: how to quickly load many facts? (whether via consult, read_term or any other way)

slower than a systems that is more than ten times as old

Well, we'd have to get to it some day, even if it's lower priority.

[triska]

Personally, I would phrase it like this:

1. If you have all these facts in a file, simply consult it directly, as already suggested by @pmoura. For example, if you know the file before starting your program, simply specify the file on the command line:

   $ scryer-prolog facts.pl

   Use this if you have many facts, and also if you have few facts: This is a normal way to consult any Prolog program.
2. If you do not know the file before starting the program, or you want to make the loading dependent on other factors that only become apparent at run-time (such as input from the user etc.), or you do not want to restart the Prolog system etc., you can use consult/1 to consult the facts (and also rules, and in fact any Prolog program that is stored in a file). You can also use consult/1 as an alternative to (1) at any time.
3. If you do not want to assert the facts, but rather reason about them as Prolog terms, use read/1 or any of the related predicates that start with read.... In this way, the facts do not become part of the clause database, but are available as Prolog terms for reasoning within your program. Use this approach for example if you want to rewrite these terms etc. without asserting them as facts. This is conceptually different from (1) and (2).