Speed up unit tests. Closes #2958

[gqvz]

Description

Speed up unit tests. Closes #2958

Type of change

• Bug fix (non-breaking change which fixes an issue).

Checklist

• I have read and understood the rules about how to Contribute to this project
• The pull request is for the branch develop
• No new plugin was added.
• I have inserted the copyright banner at the start of the file: # This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl # See the file 'LICENSE' for copying permission.
• No new libraries were added.
• Linters (Ruff) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
• I have added tests for the feature/bug I solved (see tests folder). All the tests (new and old ones) gave 0 errors.
• No GUI edits
• After you had submitted the PR, if DeepSource, Django Doctors or other third-party linters have triggered any alerts during the CI checks, I have solved those alerts.

[gqvz]

on my machine: 16m->4m
most of the time was creating jobs for playbooks, it was running the analyzers for the job, but not checking the output of the analyzers (not what the test is for), analyzers are tested in their own unit tests, i'll upload the new test times soon

[gqvz]

FAIL: test_analyzer_on_supported_filetypes (tests.api_app.analyzers_manager.unit_tests.file_analyzers.test_capa_info.TestCapaInfoAnalyzer.test_analyzer_on_supported_filetypes) (mimetype='application/x-sharedlib')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/deploy/intel_owl/api_app/analyzers_manager/file_analyzers/capa_info.py", line 68, in _download_signatures
    filename = signature["name"]
               ~~~~~~~~~^^^^^^^^
TypeError: string indices must be integers, not 'str'

IntelOwl/api_app/analyzers_manager/file_analyzers/capa_info.py

Lines 52 to 81 in 8419ace

	def _download_signatures(cls) -> None:
	logger.info(f"Downloading signatures at {SIGNATURE_LOCATION} now")
	if os.path.exists(SIGNATURE_LOCATION):
	logger.info(f"Removing existing signatures at {SIGNATURE_LOCATION}")
	shutil.rmtree(SIGNATURE_LOCATION)
	os.makedirs(SIGNATURE_LOCATION)
	logger.info(f"Created fresh signatures directory at {SIGNATURE_LOCATION}")
	signatures_url = "https://api.github.com/repos/mandiant/capa/contents/sigs"
	try:
	response = requests.get(signatures_url)
	signatures_list = response.json()
	for signature in signatures_list:
	filename = signature["name"]
	download_url = signature["download_url"]
	signature_file_path = os.path.join(SIGNATURE_LOCATION, filename)
	sig_content = requests.get(download_url, stream=True)
	with open(signature_file_path, mode="wb") as file:
	for chunk in sig_content.iter_content(chunk_size=10 * 1024):
	file.write(chunk)
	except Exception as e:
	logger.error(f"Failed to download signature: {e}")
	raise AnalyzerRunException("Failed to update signatures")
	logger.info("Successfully updated signatures")

Shouldnt this method be patched out, my commit has nothing to do with this, but its still failing

[gqvz]

https://gist.github.com/gqvz/2bb8e9386b47dc792a874cc9724cea4b updated times

[gqvz]

IntelOwl/tests/api_app/analyzers_manager/test_views.py

Lines 32 to 39 in 8556c53

	response = self.client.post(f"{self.URL}/{analyzer}/pull")
	self.assertEqual(response.status_code, 200)
	self.client.force_authenticate(self.superuser)
	with patch.object(YaraScan, "update", return_value=True):
	response = self.client.post(f"{self.URL}/{analyzer}/pull")
	self.assertEqual(response.status_code, 200, response.json())

what is this test even doing? i dont think its supposed to call /pull twice, i'll just bandage this for now

[gqvz]

https://github.com/intelowlproject/IntelOwl/blob/develop/tests/test_crons.py#L245-L247
i think we should if_mock_connection -> skip this one, it uses git for updates, needs connection

[gqvz]

the test step of the ci now takes ~ half time

[Copilot]

Pull request overview

This PR focuses on reducing CI/unit-test runtime by preventing slow external operations (Celery task scheduling, polling sleeps, and updater downloads/git activity) from running during tests.

Changes:

• Skip or mock slow updater behavior (e.g., Yara updater in cron tests; Capa signatures download).
• Patch Celery pipeline enqueue (job_pipeline.apply_async) during API tests to avoid running the full job pipeline.
• Patch time.sleep in analyzer unit tests to eliminate real polling delays.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tests/test_crons.py	Skips the Yara updater test under MOCK_CONNECTIONS to avoid unmocked git operations.
tests/api_app/test_api.py	Patches job_pipeline.apply_async for several endpoints/rescan tests to avoid slow pipeline execution.
tests/api_app/analyzers_manager/unit_tests/observable_analyzers/test_pulsedive.py	Patches time.sleep to remove polling delay.
tests/api_app/analyzers_manager/unit_tests/file_analyzers/test_virushee.py	Patches time.sleep to remove polling delay.
tests/api_app/analyzers_manager/unit_tests/file_analyzers/test_capa_info.py	Mocks signature download to avoid filesystem/network work during tests.
tests/api_app/analyzers_manager/test_views.py	Wraps analyzer “pull” calls with a mocked YaraScan.update to avoid slow updater execution.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Typo in comment: "dont" → "don't".

Suggested change

	# dont actually run the analyzers, they are tested in unit tests
	# don't actually run the analyzers, they are tested in unit tests

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[mlodic]

I applied copilot suggestions

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

These mocks only create directories, but YaraScan.update() calls repo.compile() for each configured repository. With no .yar/.yara/.rule files created by the mocks, YaraRepo.compile() will attempt to compile an empty ruleset and is likely to error. To keep this fast and deterministic, either mock out YaraRepo.compile/yara.compile for this test, or have the mock extractall/clone_from create at least one minimal valid YARA rule file in the repo directory.

Suggested change

	mock_repo.clone_from.side_effect = lambda url, path, **kwargs: os.makedirs(path, exist_ok=True)
	# When settings.MOCK_CONNECTIONS is False, @if_mock_connections is a no-op
	# and no mocks are injected, so mock_zipfile will be None. In that case,
	# skip this test to avoid AttributeError in CI.
	if mock_zipfile is None:
	self.skipTest("MOCK_CONNECTIONS is disabled; yara updater test requires mocked connections.")
	mock_zipfile.return_value.extractall.side_effect = lambda path: os.makedirs(path, exist_ok=True)
	def _create_dummy_rule(path: str) -> None:
	os.makedirs(path, exist_ok=True)
	rule_path = os.path.join(path, "dummy.yar")
	# Minimal valid YARA rule to avoid empty ruleset compilation errors
	with open(rule_path, "w", encoding="utf-8") as f:
	f.write("rule dummy { condition: true }")
	mock_repo.clone_from.side_effect = (
	lambda url, path, **kwargs: _create_dummy_rule(path)
	)
	# When settings.MOCK_CONNECTIONS is False, @if_mock_connections is a no-op
	# and no mocks are injected, so mock_zipfile will be None. In that case,
	# skip this test to avoid AttributeError in CI.
	if mock_zipfile is None:
	self.skipTest("MOCK_CONNECTIONS is disabled; yara updater test requires mocked connections.")
	mock_zipfile.return_value.extractall.side_effect = (
	lambda path: _create_dummy_rule(path)
	)

[gqvz]

Is there anything else you'd like me to do?
If not then I'd like to try and speedup other steps of the CI pipeline, I'll open an issue on that soon

[mlodic]

if you can please address the remaining copilot issues

[gqvz]

I don't think the second change is required, compile method would've failed the test if that was the behaviour, but you can commit the suggestion if youd like

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.