Skip to content

README.md

Latest commit

 

History

History
230 lines (158 loc) · 8.85 KB

README.md

File metadata and controls

230 lines (158 loc) · 8.85 KB

HomeNetSec

HomeNetSec is a dockerized home network security pipeline.

Continuous-operation references:

It produces a continuous dashboard and persistent alert/device state by combining:

  • PCAP-based network telemetry (offline Zeek analysis)
  • TLS fingerprint telemetry (offline Suricata → EVE JSON)
  • DNS telemetry (AdGuard Home stats)
  • Behavioral enrichment (RITA beacon/long-connection analysis)
  • Baselines & anomaly candidates (local SQLite state)

The design is intentionally modular so you can swap out how PCAPs arrive (OPNsense, tcpdump, SPAN port, etc.) without rewriting analysis/reporting.

Current design

HomeNetSec runs on the continuous pipeline:

  1. Continuous ingest: merge eligible PCAPs, run Suricata + Zeek, queue merged artifacts
  2. Continuous analysis: update rolling baselines, detect alerts, update device inventory, run incremental RITA
  3. Live dashboard/API: serve current alerts, devices, pipeline health, and review actions

A) Ingest + processing pipeline (ingest cadence aligned with your cron)

Entry point:

  • scripts/continuous_ingest.sh

Goal: keep PCAP processing incremental, flatten artifacts into the continuous layout, and enqueue work for analysis.

What it does:

  1. Reads eligible new PCAPs using output/state/ingest_state.json.
  2. Applies a safety lag and skips incomplete files.
  3. Merges eligible PCAPs and writes manifest metadata.
  4. Runs Suricata + Zeek on merged PCAPs.
  5. Flattens artifacts into the continuous layout.
  6. Queues merged artifacts for analysis.

Retention defaults (optimized for analysis quality vs disk usage):

  • merged PCAPs: 3 days
  • merged source PCAPs in PCAP_SOURCE_DIR: 48 hours after successful merge (SOURCE_PCAP_DELETE_DELAY_HOURS=48)
  • Suricata EVE outputs + Zeek logs: 30 days minimum (needed for RITA's rolling window + baseline analysis)
    • Recommended: 60 days for better baseline/anomaly detection
    • Logs are tiny (~50-100 MB/day) compared to PCAPs (~5-20 GB/day)

B) Analysis + dashboard pipeline (triggered whenever you refresh reports)

This pipeline does not pull PCAPs and does not rerun Zeek/Suricata. It consumes the queue and artifact outputs from continuous ingest.

Typical command sequence:

  • ./scripts/continuous_analysis.sh --process-queue
  • ./scripts/generate_dashboard.sh

C) Full pipeline (ingest + analysis + dashboard)

If you want everything updated together, run:

  1. scripts/run_ingest_and_analysis.sh

What this does (high level)

  • Ingest/processing: PCAP ingest → merge → Suricata + Zeek
  • Analysis/dashboard: rolling baselines + continuous detection + RITA + live dashboard

Repo layout

  • scripts/continuous_ingest.sh — queue-backed continuous ingest entrypoint
  • scripts/continuous_analysis.sh — queue-backed continuous analysis entrypoint
  • scripts/run_ingest_and_analysis.sh — unified continuous wrapper
  • assets/docker-compose.yml — Zeek + RITA + Mongo + Suricata stack
  • assets/rita-config.yaml.example — example RITA config
  • references/ — supporting docs

Runtime outputs are not committed (PCAPs/logs/reports/DB).

How DNS + PCAP are wired (in my setup)

This pipeline assumes two independent inputs:

PCAP input (network traffic)

In my setup, PCAPs are generated by a firewall/router and periodically pulled onto the machine running HomeNetSec. The pipeline then processes those PCAPs offline.

Important: the rest of the pipeline does not care where PCAPs come from, only that they appear on disk in the expected folder structure.

Expected source path:

  • PCAP_SOURCE_DIR/lan-YYYY-MM-DD_*.pcap*

Continuous ingest copies and merges from PCAP_SOURCE_DIR into $HOMENETSEC_WORKDIR/output/pcaps/.

DNS input (resolver telemetry)

DNS telemetry is collected from AdGuard Home via its HTTP API.

Expected configuration is provided via environment variables (or a sourced env file):

  • ADGUARD_URL
  • ADGUARD_USER
  • ADGUARD_PASS

This is used only for summary stats in the report (e.g., top clients, top blocked domains). It is not required to run PCAP/Zeek/RITA.

Configuration

Environment file (.env)

All scripts automatically source .env from the repo root if present. Copy the example and customize:

cp .env.example .env
# Edit .env with your paths

The .env file is gitignored — keep your local configuration there.

Core environment variables

  • HOMENETSEC_WORKDIR — where PCAPs/logs/reports live (default: ./output)

Dashboard environment variables

  • HOMENETSEC_LAN_BIND_IP — LAN IP to bind dashboard (required for dashboard)
  • HOMENETSEC_TS_BIND_IP — Tailscale IP to bind dashboard (required for dashboard)
  • HOMENETSEC_DASHBOARD_PORT — dashboard port (default: 8088)
  • HOMENETSEC_DASHBOARD_BASE_URL — base URL for dashboard links (optional)
  • HOMENETSEC_DASHBOARD_MODE — keep as live (legacy mode removed)
  • HOMENETSEC_MIN_ALERT_CONFIDENCE — filter alerts in /api/alerts below this confidence (0.0 to 1.0, default 0)
  • HOMENETSEC_ALERT_SEVERITIES — comma-separated alert severities shown in /api/alerts (critical,high,med,low,info by default)

Running the dashboard:

cd HomeNetSec
docker compose --env-file .env -f assets/dashboard-compose.yml up -d

PCAP source configuration

HomeNetSec copies PCAPs from a local source directory for processing.

Required env var:

  • PCAP_SOURCE_DIR — local directory where PCAPs are collected (REQUIRED - must be configured in .env)
    • PCAPs are merged into $HOMENETSEC_WORKDIR/output/pcaps/
    • A .manifest.json sidecar is written for each merged PCAP
    • Source PCAPs are deleted from PCAP_SOURCE_DIR after the configured delay, not immediately
    • Merged PCAPs are retained for 3 days

Ingest controls

  • SAFETY_LAG_SECONDS — ignore files newer than now - lag (default: 120)
  • MERGE_RETRIES — retries for merge+verify (default: 1)
  • VERIFY_MERGE — verify merged packet counts (default: 1)
  • SOURCE_PCAP_DELETE_DELAY_HOURS — delay before deleting successfully merged source PCAPs from PCAP_SOURCE_DIR (default: 48)
  • RECENT_MERGE_INDEX_BACKFILL_HOURS — backfill window for reconstructing missing merge manifests from recent merged PCAPs (default: 36)
  • Retention:
    • MERGED_PCAP_RETENTION_DAYS (default: 3 - PCAPs are large; logs contain extracted intelligence)
    • HOURLY_ARTIFACT_RETENTION_DAYS (default: 30 - recommend 60 for better baseline analysis)
    • RUN_RETENTION_CLEANUP (default: 1)

Pipeline toggles

  • RUN_RITA — set to 0 to skip continuous RITA (default: 1 for full environments)
  • HOMENETSEC_RITA_STAGING_DIR — host path for flattened recent Zeek logs consumed by RITA (default: $HOMENETSEC_WORKDIR/state/zeek-flat-staging)
  • SKIP_INGEST — set to 1 to skip the ingest phase in run_ingest_and_analysis.sh
  • SKIP_ANALYSIS — set to 1 to skip the analysis phase in run_ingest_and_analysis.sh
  • SKIP_DASHBOARD — set to 1 to skip dashboard regeneration in run_ingest_and_analysis.sh
  • INGEST_EXPECT_NEXT_RUN_SECONDS — optional status hint for cron-driven ingest cadence

AdGuard Home (optional)

  • ADGUARD_ENV — path to an env file with ADGUARD_URL/USER/PASS
    • default: ~/.config/homenetsec/adguard.env
  • HOMENETSEC_ADGUARD_QUERYLOG_LIMIT — max querylog records triage checks for source-IP/domain correlation (default: 500)

Device visibility filters (optional)

  • HOMENETSEC_VISIBLE_DEVICE_CIDRS — comma-separated CIDRs considered relevant for device views
  • HOMENETSEC_HIDDEN_DEVICE_CIDRS — comma-separated CIDRs suppressed as infrastructure ranges (for example Docker bridge networks)

Running

Automated Tests

cd HomeNetSec
python3 -m unittest discover -s tests -v

Ingest pipeline (continuous)

cd HomeNetSec
export HOMENETSEC_WORKDIR="$PWD/output"
./scripts/continuous_ingest.sh --once

Analysis pipeline (continuous)

cd HomeNetSec
export HOMENETSEC_WORKDIR="$PWD/output"
./scripts/continuous_analysis.sh --process-queue

Full pipeline

cd HomeNetSec
export HOMENETSEC_WORKDIR="$PWD/output"
./scripts/run_ingest_and_analysis.sh

Modularity notes (how to adapt this to your network)

The cleanest way to adapt HomeNetSec is to treat it as layers:

  1. Ingest layer: get PCAPs into PCAP_SOURCE_DIR
  2. Processing layer: merge + Suricata + Zeek
  3. Analysis layer: rolling baselines + detection + device inventory + RITA
  4. Dashboard layer: live API + static dashboard

You can replace the ingest layer without changing the rest.

Security & privacy

  • Do not commit PCAPs, Zeek logs, reports, or credential files.
  • Keep secrets in environment variables or external env files.
  • PCAPs may contain sensitive metadata/content; handle and share with care.