Skip to content

Add generic JSON import module (generic_json_import)#781

Draft
adulau wants to merge 2 commits into
mainfrom
codex/create-generic-json-import-module
Draft

Add generic JSON import module (generic_json_import)#781
adulau wants to merge 2 commits into
mainfrom
codex/create-generic-json-import-module

Conversation

@adulau

@adulau adulau commented Jun 3, 2026

Copy link
Copy Markdown
Member

Motivation

  • Provide a generic import module that can fetch a JSON file from an HTTP(S) URL and convert records into MISP objects or attributes.
  • Allow flexible mapping by analysing JSON keys/values against the MISP object templates bundled with PyMISP to produce misp_standard output.

Description

  • Add a new import module at misp_modules/modules/import_mod/generic_json_import.py that fetches remote JSON, validates URLs, streams with a 20 MB guard, and parses the content.
  • Implement template discovery by loading PyMISP object definitions from MISPObject(...).misp_objects_path and heuristically matching flattened JSON records to templates using alias and inferred value-type scoring.
  • Emit matched records as MISPObject instances appended to a MISPEvent and optionally create standalone attributes for recognised indicator values that were not mapped to an object.
  • Add configurable userConfig options: timeout, max_records, and include_unmapped_attributes, and include unit tests in tests/test_generic_json_import.py covering object mapping and fallback attribute import.

Testing

  • python -m compileall -q misp_modules/modules/import_mod/generic_json_import.py tests/test_generic_json_import.py succeeded.
  • python -m black --check misp_modules/modules/import_mod/generic_json_import.py tests/test_generic_json_import.py succeeded after formatting.
  • pytest -q tests/test_generic_json_import.py passed (new module tests).
  • pytest -q tests/test.py::TestModules::test_introspection_module_structure failed due to an existing test-suite issue (an unrelated module misp_modules.modules.export_mod lacks an expected __all__ attribute), not caused by the new module.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant