Is LLM worth trying for Flowdroid?

[yokotayokota]

Dear Flowdroid developers

You must already know about static analysis using LLM like this paper.
Is LLM worth trying for Flowdroid?
Or do you already plan to use it in Flowdroid?

Ex.) "LLM-Assisted Static Analysis for Detecting Security Vulnerabilities"
https://arxiv.org/html/2405.17238v1

Three years ago I tried Flowdroid ICC analysis, but it didn't work with apps using Android X, so I gave it up.
I haven't tried it since, but I hope that Flowdroid can be used with the latest version of Android apps.

[MoranLian]

@yokotayokota
Hello yokotayokota

It is exciting to find the same idea to facilitate Flowdroid via LLM, and I am also really interested on ICC analysis. My email is morangeous@gmail.com, can we discuss some possible research direction on this topic?

Hoping to receive your email~

[StevenArzt]

FlowDroid has integrated ICCTA for ICC analysis. However, we need an ICC model (basically a graph of which ICC method invoked which activity) as an input. FlowDroid currently doesn't generate such an ICC model. In the research papers, we used IC3 as an external tool for the task.

In our commercial vulnerability scanner, we have integrated ValDroid (https://dl.acm.org/doi/pdf/10.1145/3649591) to resolve the values of Intents. We then build the ICC links from the reconstructed values and feed them into FlowDroid / ICCTA.

If you plan to build an LLM-based tool for generating ICC models to feed into FlowDroid, I am more than happy to help you with an evaluation against the combination of FlowDroid with ValDroid (i.e., run the experiment with our commercial tool).

With these LLM-based approaches, keep in mind that the models have been trained using public data. That means your model might just repeat already existing information ratehr than actually reasoning about the test case. To properly evaluate the approach, I suggest that you build private challenges that the model definitely hasn't seen before.

[MoranLian]

@StevenArzt
Hello Steven,

I’m really glad to receive your message!

I have some ideas and plans regarding the facilitation of ICC analysis in FlowDroid. Would it be possible for me to discuss this topic (i.e., ICC analysis of FlowDroid) with you via email? I’m currently in the process of revising another paper, so I may take some time to get back to you, but I would love to continue the conversation once that’s done.

[StevenArzt]

Sure, send me an email to steven.arzt@sit.fraunhofer.de