Skip to content

file_info_format_reference.md

Latest commit

 

History

History
423 lines (343 loc) · 17.1 KB

file_info_format_reference.md

File metadata and controls

423 lines (343 loc) · 17.1 KB
Raw

Analysis Metadata and Reports

This document describes the format and content of the analysis reports generated by DeepExtractIDA. These reports are produced only when C++/report generation is enabled (e.g., --generate-cpp) and at least one function has usable decompiled output (the same filter used for C++ file generation). When disabled or when no valid decompilation output exists, analysis output is limited to the SQLite database.

  1. file_info.md: A human-readable Markdown report summarizing the binary's characteristics.
  2. file_info.json: A machine-readable JSON file containing the structured data for automation.

File Locations

The reports are generated inside a per-module subdirectory (only when the decompilation filter produces at least one function):

  • {cpp_output_dir}/{module_name}/file_info.md
  • {cpp_output_dir}/{module_name}/file_info.json

module_name is derived from the input file name as {stem}_{extension} (extension without the dot), then sanitized with CppGenerator.sanitize_filename() (:: -> _, non [a-zA-Z0-9_.-] replaced with _, truncated to 100 chars). For example: kernel32.dll -> kernel32_dll.

Output directory resolution:

  • main.py default: if --cpp-output-dir is not provided, output goes to {sqlite_db_dir}/extracted_raw_code/{module_name}/.
  • main.py with --cpp-output-dir: output goes to {cpp_output_dir}/{module_name}/.
  • headless_batch_extractor.ps1: passes --cpp-output-dir "{StorageDir}/extracted_code", so reports are written to {StorageDir}/extracted_code/{module_name}/.

C++ File Naming

When C++ generation is enabled, functions are grouped into combined files of about 450--500 lines each to reduce file count:

  • Class methods: Grouped by class into {module_name}_{class}_group_1.cpp, {module_name}_{class}_group_2.cpp, etc. Methods are packed in alphabetical order.
  • Standalone functions: Grouped into {module_name}_standalone_group_1.cpp, {module_name}_standalone_group_2.cpp, etc. Functions are packed in alphabetical order.

Class and function names are sanitized: :: is replaced with _, invalid filename characters are removed, and components are truncated to 100 characters. The generator also caps total filename length to avoid OS path limits, appending a hash if needed. If a name collision occurs, a numeric suffix (e.g., _2) is appended.

Each function in a grouped file is preceded by a comment block with its name, mangled name, signature, and optional library tag. The Function Signature (Extended) line is included only when the extended signature is present and different from the base signature.

Markdown Report (file_info.md)

The Markdown report provides a structured overview of the analyzed binary, including technical metadata and extracted function summaries. JSON sections are parsed and pretty-printed; if a JSON field fails to parse, the raw string is emitted instead. Empty lists/dicts are rendered as “No … data available.”

Sections

  1. Basic File Information: File path, size, MD5/SHA256 hashes, and analysis timestamp.
  2. PE Version Information: Metadata from the binary's version resource (Company, Product, Version, etc.).
  3. Key PE Metadata: Compilation timestamp, PDB path, .NET assembly status, and IDA DB cache path.
  4. Detailed Technical Sections: Embedded as formatted JSON blocks. When a section is empty or missing, the report prints “No … data available.”
    • Entry Points: Detected entry points with detection method and confidence.
    • Imports: Imported modules and functions. Delay‑load imports are merged and labeled with is_delay_loaded.
    • Exports: Exported symbols and forwarder information.
    • PE Sections: Section names, virtual addresses, sizes, and flags.
    • Security Features: Status of ASLR/DEP/CFG/SEH, plus SafeSEH/CFG metadata when available.
    • DLL Characteristics: Raw and decoded DllCharacteristics flags.
    • Rich Header: Compiler/linker toolchain metadata when present.
    • TLS Callbacks: Thread Local Storage callbacks with security-analysis metadata.
    • Load Configuration: SEH/CFG guard table pointers and related fields.
    • Exception Information: Exception directory presence and size.
    • CLR Metadata (.NET only): CLR header fields when the binary is a .NET assembly.
  5. Function Summary:
    • Total Functions Extracted equals the number of functions with decompiled output that is not null, not "Decompiler not available", and not "Decompilation failed: …" (the same filter used to generate .cpp files).
    • Class Methods and Standalone Functions are derived from the full functions table and grouped by :: naming, so they can include functions that did not produce .cpp output.
    • Markdown output is truncated for readability: top 20 classes, 50 methods per class, and 200 standalone functions (with “... more” indicators).

JSON Format (file_info.json)

The file_info.json file contains the same data points as the Markdown report, but in a structured JSON form.

JSON Schema

{
  "module_name": "string",
  "basic_file_info": {
    "file_path": "string",
    "base_dir": "string",
    "file_name": "string",
    "extension": "string",
    "size_bytes": number,
    "md5_hash": "string",
    "sha256_hash": "string",
    "analysis_timestamp": "string | null"
  },
  "pe_version_info": {
    "file_version": "string | null",
    "product_version": "string | null",
    "company_name": "string | null",
    "product_name": "string | null",
    "file_description": "string | null",
    "internal_name": "string | null",
    "original_filename": "string | null",
    "legal_copyright": "string | null"
  },
  "pe_metadata": {
    "compilation_timestamp": "string | null",
    "file_modified_date": "string | null",
    "pdb_path": "string | null",
    "is_net_assembly": "boolean | null",
    "idb_cache_path": "string | null"
  },
  "entry_points": [
    {
      "address": "string",
      "address_int": number,
      "detection_method": "string",
      "method_details": "string",
      "confidence": number,
      "is_primary": boolean,
      "entry_name": "string",
      "mangled_name": "string",
      "function_name": "string",
      "function_signature_extended": "string",
      "ordinal": number,
      "index": number,
      "is_validated": boolean,
      "detection_success": boolean,
      "detection_errors": ["string"]
    }
  ] | null,
  "imports": [
    {
      "module_name": "string",
      "raw_module_name": "string",
      "is_api_set": boolean,
      "resolved_module": "string | null",
      "functions": [
        {
          "address": "string | null",
          "mangled_name": "string",
          "raw_name": "string | null",
          "raw_name_mangled": "string | null",
          "function_name": "string",
          "function_signature_extended": "string",
          "ordinal": number,
          "is_delay_loaded": boolean
        }
      ]
    }
  ] | null,
  "exports": [
    {
      "index": number,
      "address": "string",
      "mangled_name": "string",
      "raw_name": "string",
      "raw_name_mangled": "string",
      "function_name": "string",
      "function_signature_extended": "string",
      "ordinal": number,
      "is_forwarded": boolean,
      "forwarded_to": "string | null"
    }
  ] | null,
  "sections": [
    {
      "name": "string",
      "virtual_address": "string",
      "virtual_size": number,
      "raw_size": number,
      "characteristics": "string",
      "readable": boolean,
      "writable": boolean,
      "executable": boolean
    }
  ] | null,
  "security_features": {
    "aslr_enabled": boolean,
    "dep_enabled": boolean,
    "cfg_enabled": boolean,
    "seh_enabled": boolean,
    "code_integrity": boolean,
    "isolated": boolean,
    "safeseh_present": boolean,
    "safeseh_handlers": number,
    "cfg_check_function_present": boolean
  } | null,
  "dll_characteristics": {
    "raw_value": "string",
    "DYNAMIC_BASE": boolean,
    "FORCE_INTEGRITY": boolean,
    "NX_COMPAT": boolean,
    "NO_ISOLATION": boolean,
    "NO_SEH": boolean,
    "NO_BIND": boolean,
    "APPCONTAINER": boolean,
    "WDM_DRIVER": boolean,
    "GUARD_CF": boolean,
    "TERMINAL_SERVER_AWARE": boolean
  } | null,
  "rich_header": {
    "present": boolean,
    "valid": boolean,
    "checksum": "string | number | null",
    "raw_data_hex": "string | null",
    "tools": [
      {
        "product_id": number,
        "build_number": number,
        "object_count": number,
        "tool_name": "string",
        "combined_id": number
      }
    ],
    "total_objects": number,
    "unique_tools": number,
    "extraction_method": "string",
    "errors": ["string"],
    "warnings": ["string"]
  } | null,
  "tls_callbacks": [
    {
      "index": number,
      "address": "string",
      "rva": number,
      "function_name": "string",
      "mangled_name": "string",
      "demangled_name": "string",
      "is_function": boolean,
      "function_size": number,
      "segment_name": "string",
      "segment_permissions": {
        "read": boolean,
        "write": boolean,
        "execute": boolean
      },
      "has_name": boolean,
      "extraction_errors": ["string"],
      "ida_flags": number,
      "is_code": boolean,
      "has_xrefs": boolean,
      "has_strings": boolean,
      "has_crypto_constants": boolean,
      "has_api_calls": boolean,
      "has_anti_debug": boolean,
      "suspicious_patterns": ["string"],
      "string_count": number,
      "api_call_count": number,
      "anti_debug_api_count": number,
      "timing_check_count": number,
      "anti_debug_apis_found": ["string"],
      "analysis_notes": ["string"],
      "threat_score": number,
      "threat_level": "MINIMAL | LOW | MEDIUM | HIGH"
    }
  ],
  "load_config": {
    "present": boolean,
    "size": number,
    "se_handler_table": "string",
    "se_handler_count": number,
    "guard_cf_check_function": "string",
    "guard_cf_dispatch": "string",
    "guard_flags": "string"
  } | null,
  "exception_info": {
    "present": boolean,
    "virtual_address": "string",
    "size": number
  } | null,
  "clr_metadata": "null | { major_runtime_version, minor_runtime_version, flags: string, entry_point_token: string }",
  "function_summary": "null | { total_functions, class_methods[], standalone_functions[] }"
}

Field Descriptions

module_name

The identifier for the analyzed module. It is derived from {stem}_{extension} and sanitized for filenames (e.g., kernel32.dll -> kernel32_dll). :: is replaced with _, invalid characters are replaced with _, and the result is truncated to 100 characters.

basic_file_info

Filesystem-level metadata. extension includes the leading dot as stored in the database (e.g., .dll). analysis_timestamp is stored as YYYY-MM-DD HH:MM:SS at the time the file_info row is inserted.

pe_version_info

Key-value pairs extracted from the PE StringFileInfo structure. Empty strings or null are possible when a field is missing.

pe_metadata

Technical PE headers summary. compilation_timestamp and file_modified_date use YYYY-MM-DD HH:MM:SS when available. is_net_assembly may be null if runtime info extraction is disabled or unavailable. idb_cache_path is stored as a relative path (forward slashes) from the SQLite DB directory (or batch storage directory).

entry_points

Array of identified entry points produced by multi-method detection (ida_entry_api, pe_header_direct). Results are sorted with primary entries first, then by descending confidence, then by address_int. The list is capped at 100 entries in database storage. If stored JSON is missing or invalid, this can be null or a raw string.

imports/exports

Structured lists of imported and exported symbols.

  • imports merges delay-load imports into the same list. Each function entry includes is_delay_loaded (true for delay-load). Regular import entries include raw_name and raw_name_mangled; delay-load entries omit those fields and use address: null.
  • module_name is API-set resolved (e.g., api-ms-win-core-file-l1-1-0.dll -> kernel32.dll); the original name is preserved as raw_module_name. resolved_module is present only when is_api_set is true. Imports are capped at 1000 modules in database storage.
  • exports include raw_name and raw_name_mangled. forwarded_to is present only when is_forwarded is true. Exports are capped at 5000 entries in database storage.
  • Legacy note: If imports is missing but combined_imports exists in the database, the generator will fall back to combined_imports (the current pipeline does not write combined_imports).
  • If stored JSON is missing or invalid (e.g., older DB), imports/exports can be null or a raw string value.

sections

PE section table entries with names, virtual addresses, sizes, and permissions. This section can be null when PE metadata extraction is disabled.

dll_characteristics

Decoded DllCharacteristics flags plus raw_value (hex string). This section can be null when PE metadata extraction is disabled.

security_features

Boolean flags for exploit-mitigation technologies derived from DllCharacteristics, the exception directory, and load-config metadata. Additional keys (safeseh_present, safeseh_handlers, cfg_check_function_present) appear only when that data is present. This section can be null if PE metadata extraction is disabled.

rich_header

Visual Studio-specific metadata containing compiler/linker versioning information. When not present, this is usually an empty object; null is possible if stored JSON is missing or invalid. warnings is optional and appears when manual parsing adds notes.

tls_callbacks

Detailed TLS callback metadata with security-analysis fields (string usage, API calls, anti-debug indicators, threat scoring). The list is capped at 256 entries in database storage and is empty when no TLS directory is present.

load_config

Load configuration fields (SEH, CFG guard pointers, and guard flags). When the load-config directory is absent, this is null.

exception_info

Exception directory presence and size metadata. When unavailable, this may be an empty object or null (if PE metadata extraction is disabled).

clr_metadata

CLR header fields for .NET assemblies. flags and entry_point_token are hex strings. null for native binaries or when runtime info extraction is disabled.

function_summary

Categorized view of processed functions. total_functions reflects the decompiled function count used for reporting, while class/standalone lists are derived from the full functions table and may be larger. The JSON output is not truncated, and class names are sorted with methods listed in database (function name) order. Structure:

{
  "total_functions": number,
  "class_methods": [
    {
      "class_name": "string",
      "method_count": number,
      "methods": [
        { "name": "string", "signature": "string" }
      ]
    }
  ],
  "standalone_functions": [
    { "name": "string", "signature": "string" }
  ]
}

Serialization Notes

file_info.json is built by parsing JSON strings stored in SQLite via json.loads and re-serializing them with json.dump (no additional json_safety pass). Any truncation metadata (e.g., _truncated, _truncated_keys, _serialization_metadata) is inherited from the values stored in the database. If a stored JSON field is malformed or not valid JSON, _build_file_info_dict() falls back to the raw string value in the output.

Usage Examples

Python

import json

# Load the JSON file
with open('file_info.json', 'r', encoding='utf-8') as f:
    data = json.load(f)

# Access basic metadata
print(f"Module: {data['module_name']}")
print(f"MD5: {data['basic_file_info']['md5_hash']}")

# Process imports
for module in data.get('imports', []):
    print(f"Importing from: {module['module_name']}")
    for func in module['functions']:
        print(f"  - {func['function_name']}")

PowerShell

# Load the JSON file
$data = Get-Content -Path "file_info.json" -Raw | ConvertFrom-Json

# Access file information
Write-Host "File: $($data.basic_file_info.file_name)"
Write-Host "SHA256: $($data.basic_file_info.sha256_hash)"

# Function statistics
if ($data.function_summary) {
    Write-Host "Total functions: $($data.function_summary.total_functions)"
}

JavaScript (Node.js)

const fs = require("fs");

const data = JSON.parse(fs.readFileSync("file_info.json", "utf8"));

console.log(`Analyzing: ${data.basic_file_info.file_name}`);
console.log(`Security Features:`, data.security_features);

if (data.function_summary) {
  console.log(
    `Classes identified: ${data.function_summary.class_methods.length}`,
  );
}

Implementation Reference

The generation logic resides in deep_extract/cpp_generator.py:

  • generate_file_info_markdown(): Entry point for Markdown generation.
  • generate_file_info_json(): Entry point for JSON generation.
  • _build_file_info_dict(): Core logic for mapping database records to the structured report schema.

The system ensures that file_info.json and file_info.md remain synchronized by deriving both from the same database record.