HomeNetSec

HomeNetSec is a dockerized home network security pipeline.

Continuous-operation references:

• references/CONTINUOUS_OPERATIONS.md
• TRIAGE_INTEGRATION_PLAN.md

It produces a continuous dashboard and persistent alert/device state by combining:

• PCAP-based network telemetry (offline Zeek analysis)
• TLS fingerprint telemetry (offline Suricata → EVE JSON)
• DNS telemetry (AdGuard Home stats)
• Behavioral enrichment (RITA beacon/long-connection analysis)
• Baselines & anomaly candidates (local SQLite state)

The design is intentionally modular so you can swap out how PCAPs arrive (OPNsense, tcpdump, SPAN port, etc.) without rewriting analysis/reporting.

Current design

HomeNetSec runs on the continuous pipeline:

1. Continuous ingest: merge eligible PCAPs, run Suricata + Zeek, queue merged artifacts
2. Continuous analysis: update rolling baselines, detect alerts, update device inventory, run incremental RITA
3. Live dashboard/API: serve current alerts, devices, pipeline health, and review actions

A) Ingest + processing pipeline (ingest cadence aligned with your cron)

Entry point:

• scripts/continuous_ingest.sh

Goal: keep PCAP processing incremental, flatten artifacts into the continuous layout, and enqueue work for analysis.

What it does:

1. Reads eligible new PCAPs using output/state/ingest_state.json.
2. Applies a safety lag and skips incomplete files.
3. Merges eligible PCAPs and writes manifest metadata.
4. Runs Suricata + Zeek on merged PCAPs.
5. Flattens artifacts into the continuous layout.
6. Queues merged artifacts for analysis.

Retention defaults (optimized for analysis quality vs disk usage):

• merged PCAPs: 3 days
• merged source PCAPs in PCAP_SOURCE_DIR: 48 hours after successful merge (SOURCE_PCAP_DELETE_DELAY_HOURS=48)
• Suricata EVE outputs + Zeek logs: 30 days minimum (needed for RITA's rolling window + baseline analysis)
  • Recommended: 60 days for better baseline/anomaly detection
  • Logs are tiny (~50-100 MB/day) compared to PCAPs (~5-20 GB/day)

B) Analysis + dashboard pipeline (triggered whenever you refresh reports)

This pipeline does not pull PCAPs and does not rerun Zeek/Suricata. It consumes the queue and artifact outputs from continuous ingest.

Typical command sequence:

• ./scripts/continuous_analysis.sh --process-queue
• ./scripts/generate_dashboard.sh

C) Full pipeline (ingest + analysis + dashboard)

If you want everything updated together, run:

1. scripts/run_ingest_and_analysis.sh

What this does (high level)

• Ingest/processing: PCAP ingest → merge → Suricata + Zeek
• Analysis/dashboard: rolling baselines + continuous detection + RITA + live dashboard

Repo layout

• scripts/continuous_ingest.sh — queue-backed continuous ingest entrypoint
• scripts/continuous_analysis.sh — queue-backed continuous analysis entrypoint
• scripts/run_ingest_and_analysis.sh — unified continuous wrapper
• assets/docker-compose.yml — Zeek + RITA + Mongo + Suricata stack
• assets/rita-config.yaml.example — example RITA config
• references/ — supporting docs

Runtime outputs are not committed (PCAPs/logs/reports/DB).

How DNS + PCAP are wired (in my setup)

This pipeline assumes two independent inputs:

PCAP input (network traffic)

In my setup, PCAPs are generated by a firewall/router and periodically pulled onto the machine running HomeNetSec. The pipeline then processes those PCAPs offline.

Important: the rest of the pipeline does not care where PCAPs come from, only that they appear on disk in the expected folder structure.

Expected source path:

• PCAP_SOURCE_DIR/lan-YYYY-MM-DD_*.pcap*

Continuous ingest copies and merges from PCAP_SOURCE_DIR into $HOMENETSEC_WORKDIR/output/pcaps/.

DNS input (resolver telemetry)

DNS telemetry is collected from AdGuard Home via its HTTP API.

Expected configuration is provided via environment variables (or a sourced env file):

• ADGUARD_URL
• ADGUARD_USER
• ADGUARD_PASS

This is used only for summary stats in the report (e.g., top clients, top blocked domains). It is not required to run PCAP/Zeek/RITA.

Configuration

Environment file (.env)

All scripts automatically source .env from the repo root if present. Copy the example and customize:

cp .env.example .env
# Edit .env with your paths

The .env file is gitignored — keep your local configuration there.

Core environment variables

• HOMENETSEC_WORKDIR — where PCAPs/logs/reports live (default: ./output)

Dashboard environment variables

• HOMENETSEC_LAN_BIND_IP — LAN IP to bind dashboard (required for dashboard)
• HOMENETSEC_TS_BIND_IP — Tailscale IP to bind dashboard (required for dashboard)
• HOMENETSEC_DASHBOARD_PORT — dashboard port (default: 8088)
• HOMENETSEC_DASHBOARD_BASE_URL — base URL for dashboard links (optional)
• HOMENETSEC_DASHBOARD_MODE — keep as live (legacy mode removed)
• HOMENETSEC_MIN_ALERT_CONFIDENCE — filter alerts in /api/alerts below this confidence (0.0 to 1.0, default 0)
• HOMENETSEC_ALERT_SEVERITIES — comma-separated alert severities shown in /api/alerts (critical,high,med,low,info by default)

Running the dashboard:

cd HomeNetSec
docker compose --env-file .env -f assets/dashboard-compose.yml up -d

PCAP source configuration

HomeNetSec copies PCAPs from a local source directory for processing.

Required env var:

• PCAP_SOURCE_DIR — local directory where PCAPs are collected (REQUIRED - must be configured in .env)
  • PCAPs are merged into $HOMENETSEC_WORKDIR/output/pcaps/
  • A .manifest.json sidecar is written for each merged PCAP
  • Source PCAPs are deleted from PCAP_SOURCE_DIR after the configured delay, not immediately
  • Merged PCAPs are retained for 3 days

Ingest controls

• SAFETY_LAG_SECONDS — ignore files newer than now - lag (default: 120)
• MERGE_RETRIES — retries for merge+verify (default: 1)
• VERIFY_MERGE — verify merged packet counts (default: 1)
• SOURCE_PCAP_DELETE_DELAY_HOURS — delay before deleting successfully merged source PCAPs from PCAP_SOURCE_DIR (default: 48)
• RECENT_MERGE_INDEX_BACKFILL_HOURS — backfill window for reconstructing missing merge manifests from recent merged PCAPs (default: 36)
• Retention:
  • MERGED_PCAP_RETENTION_DAYS (default: 3 - PCAPs are large; logs contain extracted intelligence)
  • HOURLY_ARTIFACT_RETENTION_DAYS (default: 30 - recommend 60 for better baseline analysis)
  • RUN_RETENTION_CLEANUP (default: 1)

Pipeline toggles

• RUN_RITA — set to 0 to skip continuous RITA (default: 1 for full environments)
• HOMENETSEC_RITA_STAGING_DIR — host path for flattened recent Zeek logs consumed by RITA (default: $HOMENETSEC_WORKDIR/state/zeek-flat-staging)
• SKIP_INGEST — set to 1 to skip the ingest phase in run_ingest_and_analysis.sh
• SKIP_ANALYSIS — set to 1 to skip the analysis phase in run_ingest_and_analysis.sh
• SKIP_DASHBOARD — set to 1 to skip dashboard regeneration in run_ingest_and_analysis.sh
• INGEST_EXPECT_NEXT_RUN_SECONDS — optional status hint for cron-driven ingest cadence

AdGuard Home (optional)

• ADGUARD_ENV — path to an env file with ADGUARD_URL/USER/PASS
  • default: ~/.config/homenetsec/adguard.env
• HOMENETSEC_ADGUARD_QUERYLOG_LIMIT — max querylog records triage checks for source-IP/domain correlation (default: 500)

Device visibility filters (optional)

• HOMENETSEC_VISIBLE_DEVICE_CIDRS — comma-separated CIDRs considered relevant for device views
• HOMENETSEC_HIDDEN_DEVICE_CIDRS — comma-separated CIDRs suppressed as infrastructure ranges (for example Docker bridge networks)

Running

Automated Tests

cd HomeNetSec
python3 -m unittest discover -s tests -v

Ingest pipeline (continuous)

cd HomeNetSec
export HOMENETSEC_WORKDIR="$PWD/output"
./scripts/continuous_ingest.sh --once

Analysis pipeline (continuous)

cd HomeNetSec
export HOMENETSEC_WORKDIR="$PWD/output"
./scripts/continuous_analysis.sh --process-queue

Full pipeline

cd HomeNetSec
export HOMENETSEC_WORKDIR="$PWD/output"
./scripts/run_ingest_and_analysis.sh

Modularity notes (how to adapt this to your network)

The cleanest way to adapt HomeNetSec is to treat it as layers:

1. Ingest layer: get PCAPs into PCAP_SOURCE_DIR
2. Processing layer: merge + Suricata + Zeek
3. Analysis layer: rolling baselines + detection + device inventory + RITA
4. Dashboard layer: live API + static dashboard

You can replace the ingest layer without changing the rest.

Security & privacy

• Do not commit PCAPs, Zeek logs, reports, or credential files.
• Keep secrets in environment variables or external env files.
• PCAPs may contain sensitive metadata/content; handle and share with care.