Can Soufflé be used incrementally?

[andrewcropper]

We are trying to determine whether Soufflé is suitable for our application.

We have a fixed Datalog program P, which can be very large. We want to query P many (250k+) times.

For instance, P might be the simple set of facts:

rich(alice).
rich(bob).
tall(bob).
tall(charlie).
italian(charlie).

And our queries might be:

f1(A):-rich(A).
f2(A):-rich(A), tall(A).
f3(A):-rich(A), tall(A), italian(A).

Importantly, for each query/rule f_i, we do not need a Datalog engine to remember the f_i facts as they will never appear in the body of a rule.

From the documentation, it looks like Soufflé does not support incremental solving like the above.

Can anyone provide any insight into whether Soufflé might be suitable?

Thanks,

Andrew

[psubotic]

Hi, I’m not sure I fully understand your use case but if you simply need incrementalism I.e., facts can be added or removed on a constant rule set, please have a look at the following: https://souffle-lang.github.io/ppdp21.html

…

On Fri, 3 May 2024 at 12:18 PM, Andrew Cropper ***@***.***> wrote: We are trying to determine whether Soufflé is suitable for our application. We have a fixed Datalog program P, which can be very large. We want to query P many (250k+) times. For instance, P might be the simple set of facts: rich(alice). rich(bob). tall(bob). tall(charlie). italian(charlie). And our queries might be: f1(A):-rich(A). f2(A):-rich(A), tall(A). f3(A):-rich(A), tall(A), italian(A). Importantly, for each query/rule f_i, we do not need a Datalog engine to remember the f_i facts as they will never appear in the body of a rule. From the documentation, it looks like Soufflé does not support incremental solving like the above. Can anyone provide any insight into whether Soufflé might be suitable? Thanks, Andrew — Reply to this email directly, view it on GitHub <#2487>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC3OOPH4QJUD7EAEVKIQ6DZANP7LAVCNFSM6AAAAABHFIHZRWVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZWGYYDAMJUGQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[andrewcropper]

Hi,

That paper is vaguely related. Maybe I can try to make our use case clearer.

Suppose I have this Soufflé program:

.decl rich(x:number)
.input rich

.decl tall(x:number)
.input tall

.decl happy1(x:number)
.output happy1

happy1(a):- rich(a).

I can run it and find the happy1 logical consequences.

Now suppose I have this Soufflé program:

.decl rich(x:number)
.input rich

.decl tall(x:number)
.input tall

.decl happy2(x:number)
.output happy2

happy2(a):- rich(a), tall(a).

I can again run it and find the happy2 logical consequences.

We can use Soufflé like the above.

However, when we have many input facts, Soufflé takes a long time (20+ minutes), just to read and process them. Therefore, we do not want to keep reloading/parsing the input relations.

Note that we cannot batch the output relations, i.e. put them in a single program.

Is our use case clearer?

[psubotic]

Hi , Sorry if I’m not understanding you, but let my try see if I understand now: you want to varying the rulesets but materialise the IDB relations so that when you add/remove rules from the rulesets you don’t have to compute everything from scratch, just what needs to be recomputed? I came across the need for this in a use case a few years ago but never got around to doing anything. So soufflé as far as I’m aware does not have direct support for this, but I don’t think an extension like this would be too hard to do though. If you want more details feel free to ping me ***@***.*** Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Andrew Cropper ***@***.***> Sent: Friday, May 3, 2024 3:56:33 PM To: souffle-lang/souffle ***@***.***> Cc: Pavle Subotic ***@***.***>; Comment ***@***.***> Subject: Re: [souffle-lang/souffle] Can Soufflé be used incrementally? (Discussion #2487) Hi, That paper is vaguely related. Maybe I can try to make our use case clearer. Suppose I have this Soufflé program: .decl rich(x:number) .input rich .decl tall(x:number) .input tall .decl happy1(x:number) .output happy1 happy1(a):- rich(a). I can run it and find the happy1 logical consequences. Now suppose I have this Soufflé program: .decl rich(x:number) .input rich .decl tall(x:number) .input tall .decl happy2(x:number) .output happy2 happy2(a):- rich(a), tall(a). I can again run it and find the happy2 logical consequences. We can use Soufflé like the above. However, when we have many input facts, Soufflé takes a long time (20+ minutes), just to read and process them. Therefore, we do not want to keep reloading/parsing the input relations. Note that we cannot batch the output relations, i.e. put them in a single program. Is our use case clearer? — Reply to this email directly, view it on GitHub<#2487 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAC3OONHOHHDTA5TY2KPQK3ZAOJRDAVCNFSM6AAAAABHFIHZRWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGMBWGEYDE>. You are receiving this because you commented.Message ID: ***@***.***>

[andrewcropper]

Yes, precisely. I will email you. Thanks!

[ecleipteon]

Hi @psubotic,

I was doing some search as we have a use case that would really benefit of the Elastic Incremental Datalog. I have just came across this thread. Could you please clarify if [1] was eventually merged into the main branch and, in case, share the pull request id?

Many thanks

[1] https://souffle-lang.github.io/ppdp21.html

[psubotic]

Hi Luke, I don’t believe it was ever merged into main from David’s repo ***@***.***). David perhaps you *can you give you more info?

…

On Wed, 21 Aug 2024 at 4:20 PM, Luke ***@***.***> wrote: Hi @psubotic <https://github.com/psubotic>, I was doing some search as we have a use case that would really benefit of the Elastic Incremental Datalog. I have just came across this thread. Could you please clarify if [1] was eventually merged into the main branch and, in case, share the pull request id? Many thanks [1] https://souffle-lang.github.io/ppdp21.html — Reply to this email directly, view it on GitHub <#2487 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC3OOJC2EPYFXNCDO2JS2LZSSO2JAVCNFSM6AAAAABHFIHZRWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANBQHAZTKOA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ecleipteon]

Thanks for the super quick reply! :)

[psubotic]

Heres the branch its in https://github.com/davidwzhao/souffle/tree/incremental-with-provenance-eager-diffs

…

On Wed, 21 Aug 2024 at 4:35 PM, P Subotic ***@***.***> wrote: Hi Luke, I don’t believe it was ever merged into main from David’s repo ***@***.***). David perhaps you *can you give you more info? On Wed, 21 Aug 2024 at 4:20 PM, Luke ***@***.***> wrote: > Hi @psubotic <https://github.com/psubotic>, > > I was doing some search as we have a use case that would really benefit > of the Elastic Incremental Datalog. I have just came across this thread. > Could you please clarify if [1] was eventually merged into the main branch > and, in case, share the pull request id? > > Many thanks > > [1] https://souffle-lang.github.io/ppdp21.html > > — > Reply to this email directly, view it on GitHub > <#2487 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAC3OOJC2EPYFXNCDO2JS2LZSSO2JAVCNFSM6AAAAABHFIHZRWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANBQHAZTKOA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[ecleipteon]

Thanks!