Merge pull request #3138 from intelowlproject/develop

v6.5.0

Author: mlodic

.github/CHANGELOG.md

@@ -2,6 +2,34 @@
 
 [**Upgrade Guide**](https://intelowlproject.github.io/docs/IntelOwl/installation/#update-to-the-most-recent-version)
 
+## [v6.5.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.5.0)
+Happy new year! :sparkler: And Happy Birthday IntelOwl! :tada: 
+
+We are celebrating the 6th IntelOwl Birthday! :sunglasses: WOW! Such a Milestone!
+
+And we reached almost 4.5k stars! :star: Thank you for your support!
+
+This release merges all the developments performed by our Google Summer of Code contributors for this year. You can read the related blogs for more info about:
+- [Akshit Maheshwary](https://x.com/Akshit20437406): [IntelOwl Improvements: Analyzers and Integrations](https://intelowlproject.github.io/blogs/gsoc_25_new_analyzers_and_integrations)
+- [Pranjal Gupta](https://github.com/pranjalg1331): [IntelOwl improvements: refactor analyzer tests](https://intelowlproject.github.io/blogs/gsoc25_refactor_analyzer_tests)
+
+A special thanks to the new maintainer and GSoC mentor for the 2025: [Federico Gibertoni](https://github.com/fgibertoni).
+
+The UI now supports a new page for the so called "Artifacts" or "Analyzables". They are a representation of an observable or a sample.
+Thanks to this new section, you can now store your evaluations for each observables/samples and make them count in your analyses results!
+Please take time to explore this new section in the GUI and provide feedback!
+[Docs reference](https://intelowlproject.github.io/docs/IntelOwl/usage/#analyzables-artifacts)
+
+As usual, we add new plugins. This release brings the following new ones:
+* [Hunting Abuse.ch](https://hunting.abuse.ch/api/): new central API for Abuse.ch
+* [YaraX](https://virustotal.github.io/yara-x/docs/intro/getting-started/) integration: you can now run your Yara rules with the new engine written in Rust
+* Now [Floss](https://github.com/mandiant/flare-floss) and [Capa](https://github.com/mandiant/capa) are integrated directly in the main container so you don't need anymore to run the optional container `malware_tools_analyzers" for them.
+* [Phunter](https://github.com/N0rz3/Phunter) which requires the execution of a new optional container with `--phunter`.
+* [JoeSandbox](https://www.joesandbox.com/), a malware analysis tool.
+* "ExpandURL" which takes a shortened URL and provides us the actual expanded URL, along with full redirection chain.
+
+We don't mention here all the other adjustments, fixes and dependencies upgrades. Please check the full changelog for that.
+
 ## [v6.4.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.4.0)
 This release mostly provides important changes in the backend part that will be supported in the UI in the next releases.
 * Analyzable: Representation of an observable or a sample: every job is linked to the scanned analyzable.

.github/pull_request_template.md

@@ -26,8 +26,8 @@ Please delete options that are not relevant.
     - [ ] Check if it could make sense to add that analyzer/connector to other [freely available playbooks](https://intelowlproject.github.io/docs/IntelOwl/usage/#list-of-pre-built-playbooks).
     - [ ] I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
     - [ ] If the plugin interacts with an external service, I have created an attribute called precisely `url` that contains this information. This is required for Health Checks (HEAD HTTP requests). 
-    - [ ] If the plugin requires mocked testing, `_monkeypatch()` was used in its class to apply the necessary decorators.
-    - [ ] I have added that raw JSON sample to the `MockUpResponse` of the `_monkeypatch()` method. This serves us to provide a valid sample for testing.
+    - [ ] If a new analyzer has beed added, I have created a unittest for it in the appropriate dir. I have also mocked all the external calls, so that no real calls are being made while testing.
+    - [ ] I have added that raw JSON sample to the `get_mocker_response()` method of the unittest class. This serves us to provide a valid sample for testing. 
     - [ ] I have created the corresponding `DataModel` for the new analyzer following the [documentation](https://intelowlproject.github.io/docs/IntelOwl/contribute/#how-to-create-a-datamodel)
 - [ ] I have inserted the copyright banner at the start of the file: ```# This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl # See the file 'LICENSE' for copying permission.```
 - [ ] Please avoid adding new libraries as requirements whenever it is possible. Use new libraries only if strictly needed to solve the issue you are working for. In case of doubt, ask a maintainer permission to use a specific library.

.github/workflows/codeql-analysis.yml

@@ -65,7 +65,7 @@ jobs:
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v3.28.0
+      uses: github/codeql-action/init@v4.31.0
       with:
         languages: python
         # Override the default behavior so that the action doesn't attempt
@@ -93,4 +93,4 @@ jobs:
     #   make release
 
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v3.28.0
+      uses: github/codeql-action/analyze@v4.31.0

.github/workflows/pull_request_automation.yml

@@ -115,6 +115,9 @@ jobs:
       - name: Run test
         run: |
           docker exec intelowl_uwsgi coverage run manage.py test --keepdb tests
+      - name: Run async tests
+        run: |
+          docker exec intelowl_uwsgi coverage run manage.py test --keepdb async_tests
 
   frontend-tests:
     runs-on: ubuntu-latest
@@ -128,7 +131,7 @@ jobs:
         with:
           node-version: 18
       - name: Cache node modules
-        uses: actions/cache@v4
+        uses: actions/cache@v5
         with:
           path: ~/.npm
           key: npm-build-${{ hashFiles('frontend/package-lock.json') }}

.github/workflows/scorecard.yml

@@ -59,7 +59,7 @@ jobs:
       # Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF
       # format to the repository Actions tab.
       - name: "Upload artifact"
-        uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0
         with:
           name: SARIF file
           path: results.sarif
@@ -68,6 +68,6 @@ jobs:
 
       # Upload the results to GitHub's code scanning dashboard.
       - name: "Upload to code-scanning"
-        uses: github/codeql-action/upload-sarif@5b6e617dc0241b2d60c2bccea90c56b67eceb797 # v2.22.11
+        uses: github/codeql-action/upload-sarif@8d77149e0c9e2199ac9cfc90c9e15116f5c69c48 # v2.22.11
         with:
           sarif_file: results.sarif

README.md

@@ -64,21 +64,19 @@ You can see the full list of all available analyzers in the [documentation](http
 
 As open source project maintainers, we strongly rely on external support to get the resources and time to work on keeping the project alive, with a constant release of new features, bug fixes and general improvements.
 
-Because of this, we joined [Open Collective](https://opencollective.com/intelowl-project) to obtain non-profit equal level status which allows the organization to receive and manage donations transparently. Please support IntelOwl and all the community by choosing a plan (BRONZE, SILVER, etc).
+Because of this, we joined [Open Collective](https://opencollective.com/intelowl-project) to obtain US and EU non-profit equal level status which allows the organization to receive and manage donations transparently and with tax exemption. Please support IntelOwl and all the community by choosing a plan (BRONZE, SILVER, etc).
 
 <a href="https://opencollective.com/intelowl-project/donate" target="_blank">
   <img src="https://opencollective.com/intelowl-project/donate/button@2x.png?color=blue" width=200 />
 </a>
 
-### 🥇 GOLD
-
 #### Certego
 
 <a href="https://certego.net/?utm_source=intelowl"> <img style="margin-right: 2px" width=250 height=71 src="static/Certego.png" alt="Certego Logo"/></a>
 
 [Certego](https://certego.net/?utm_source=intelowl) is a MDR (Managed Detection and Response) and Threat Intelligence Provider based in Italy.
 
-IntelOwl was born out of Certego's Threat intelligence R&D division and is constantly maintained and updated thanks to them.
+IntelOwl was born out of Certego's Threat intelligence R&D division and is mostly maintained and updated thanks to them.
 
 #### The Honeynet Project
 
@@ -96,16 +94,6 @@ Since its birth this project has been participating in the [Google Summer of Cod
 If you are interested in participating in the next Google Summer of Code, check all the info available in the [dedicated repository](https://github.com/intelowlproject/gsoc)!
 
 
-### 🥈 SILVER
-
-#### ThreatHunter.ai
-
-<a href="https://threathunter.ai?utm_source=intelowl"> <img style="border: 0.2px solid black" width=194 height=80 src="static/threathunter_logo.png" alt="ThreatHunter.ai logo"> </a>
-
-[ThreatHunter.ai®](https://threathunter.ai?utm_source=intelowl), is a 100% Service-Disabled Veteran-Owned Small Business started in 2007 under the name Milton Security Group. ThreatHunter.ai is the global leader in Dynamic Threat Hunting. Operating a true 24x7x365 Security Operation Center with AI/ML-enhanced human Threat Hunters, ThreatHunter.ai has changed the industry in how threats are found, and mitigated in real time. For over 15 years, our teams of Threat Hunters have stopped hundreds of thousands of threats and assisted organizations in defending against threat actors around the clock.
-
-### 🥉 BRONZE
-
 #### Docker
 
 In 2021 IntelOwl joined the official [Docker Open Source Program](https://www.docker.com/blog/expanded-support-for-open-source-software-projects/). This allows IntelOwl developers to easily manage Docker images and focus on writing the code. You may find the official IntelOwl Docker images [here](https://hub.docker.com/search?q=intelowlproject).

api_app/analyzables_manager/filters.py

@@ -0,0 +1,18 @@
+import rest_framework_filters as filters
+from django_filters.widgets import QueryArrayWidget
+
+from api_app.analyzables_manager.models import Analyzable
+
+
+class CharInFilter(filters.BaseInFilter, filters.CharFilter):
+    pass
+
+
+class AnalyzableFilter(filters.FilterSet):
+    name = CharInFilter(widget=QueryArrayWidget)
+
+    class Meta:
+        model = Analyzable
+        fields = {
+            "discovery_date": ["lte", "gte"],
+        }

api_app/analyzables_manager/models.py

@@ -1,3 +1,4 @@
+import logging
 from typing import Type, Union
 
 from django.core.exceptions import ValidationError
@@ -7,17 +8,14 @@
 
 from api_app.analyzables_manager.queryset import AnalyzableQuerySet
 from api_app.choices import Classification
-from api_app.data_model_manager.models import (
-    BaseDataModel,
-    DomainDataModel,
-    FileDataModel,
-    IPDataModel,
-)
+from api_app.data_model_manager.models import BaseDataModel
 from api_app.data_model_manager.queryset import BaseDataModelQuerySet
 from api_app.defaults import file_directory_path
 from api_app.helpers import calculate_md5, calculate_sha1, calculate_sha256
 from certego_saas.models import User
 
+logger = logging.getLogger(__name__)
+
 
 class Analyzable(models.Model):
     name = models.CharField(max_length=255)
@@ -82,23 +80,11 @@ def get_all_user_events_data_model(
                     ).values_list("data_model__pk", flat=True)
                 )
             query |= query2
+        logger.debug(f"{query=}")
         return self.get_data_model_class().objects.filter(query)
 
     def get_data_model_class(self) -> Type[BaseDataModel]:
-        if self.classification == Classification.IP.value:
-            return IPDataModel
-        elif self.classification in [
-            Classification.URL.value,
-            Classification.DOMAIN.value,
-        ]:
-            return DomainDataModel
-        elif self.classification in [
-            Classification.HASH.value,
-            Classification.FILE.value,
-        ]:
-            return FileDataModel
-        else:
-            raise NotImplementedError()
+        return self.CLASSIFICATIONS.get_data_model_class(self.classification)
 
     def _set_hashes(self, value: Union[str, bytes]):
         if isinstance(value, str):
@@ -111,6 +97,10 @@ def _set_hashes(self, value: Union[str, bytes]):
             self.sha1 = calculate_sha1(value)
 
     def clean(self):
+        if self.file:
+            self.classification = Classification.FILE.value
+        else:
+            self.classification = Classification.calculate_observable(self.name)
         if self.classification == Classification.FILE.value:
             from api_app.analyzers_manager.models import MimeTypes
 

api_app/analyzables_manager/queryset.py

@@ -10,14 +10,21 @@ class AnalyzableQuerySet(QuerySet):
     def visible_for_user(self, user):
 
         from api_app.models import Job
+        from api_app.user_events_manager.models import UserAnalyzableEvent
 
-        analyzables = (
+        analyzables_job = (
             Job.objects.visible_for_user(user)
             .values("analyzable")
             .distinct()
             .values_list("analyzable__pk", flat=True)
         )
-        return self.filter(pk__in=analyzables)
+        analyzables_ue = (
+            UserAnalyzableEvent.objects.visible_for_user(user)
+            .values("analyzable")
+            .distinct()
+            .values_list("analyzable__pk", flat=True)
+        )
+        return self.filter(pk__in=analyzables_job) | self.filter(pk__in=analyzables_ue)
 
     def create(self, *args, **kwargs):
         obj = self.model(**kwargs)

api_app/analyzables_manager/serializers.py

@@ -1,12 +1,74 @@
-from rest_framework.serializers import ModelSerializer
+import logging
+
+from rest_framework import serializers as rfs
 
 from api_app.analyzables_manager.models import Analyzable
+from api_app.choices import Classification
+from api_app.models import Job
 from api_app.serializers.job import JobRelatedField
 
+logger = logging.getLogger(__name__)
+
 
-class AnalyzableSerializer(ModelSerializer):
+class AnalyzableSerializer(rfs.ModelSerializer):
     jobs = JobRelatedField(many=True, read_only=True)
 
     class Meta:
         model = Analyzable
         fields = "__all__"
+        read_only_fields = [
+            "jobs",
+            "discovery_date",
+            "md5",
+            "classification",
+            "sha256",
+            "sha1",
+            "mimetype",
+        ]
+
+    def to_representation(self, instance):
+        logger.debug(f"{instance=}")
+        analyzable = super().to_representation(instance)
+        job = (
+            Job.objects.filter(id__in=analyzable["jobs"])
+            .order_by("-finished_analysis_time")
+            .first()
+        )
+        try:
+            user_event_data_model = (
+                instance.get_all_user_events_data_model().order_by("-date").first()
+            )
+        except NotImplementedError:
+            user_event_data_model = None
+        logger.debug(f"{job=}, {user_event_data_model=}")
+        if not job and not user_event_data_model:
+            analyzable["last_data_model"] = None
+            logger.debug(f"no data {analyzable=}")
+            return analyzable
+        elif (job and job.data_model) or user_event_data_model:
+            if not job or not job.data_model:
+                last_data_model = user_event_data_model
+            elif not user_event_data_model:
+                last_data_model = job.data_model
+            else:
+                if (job and job.data_model) and (
+                    job.data_model.date > user_event_data_model.date
+                ):
+                    last_data_model = job.data_model
+                else:
+                    last_data_model = user_event_data_model
+
+            try:
+                serializer_class = Classification.get_data_model_class(
+                    classification=analyzable["classification"],
+                ).get_serializer()
+            except NotImplementedError:
+                pass
+            else:
+                analyzable["last_data_model"] = serializer_class(last_data_model).data
+        logger.debug(f"before return {analyzable=}")
+        return analyzable
+
+    def create(self, validated_data):
+        instance, _ = self.Meta.model.objects.get_or_create(**validated_data)
+        return instance