### Fixed (#55)

- **MITRE ATT&CK**: Updated for detection strategies Issue #53
- Fixed hardcoding issues

#### New Data Sources

- **MITRE ATLAS**
  - Parsing script: `download_threat_information/parsing_scripts/parse_atlas.py`
  - Build script: `offense/build_atlas.py`

- **Campaigns**: Support for threat campaign data
  - Build script: `offense/build_campaigns.py`

- **CAR (Cyber Analytics Repository)**: Integration of MITRE CAR analytics
  - Analytics processing: `mitigations/ecar_analytics.py`

- **KEV (Known Exploited Vulnerabilities)**: CISA's KEV catalog integration
  - KEV schema and edge collections linking KEV entries to CVEs

#### Schema Enhancements

- Added schemas for new data types:

#### Infrastructure Improvements

- Updated Docker and docker-compose configuration
- Enhanced bootstrap script with improved CVE processing (year-by-year to avoid crashes)

Author: hembergerik

Dockerfile

@@ -1,8 +1,8 @@
 # This stage is used to get pre-compiled arangodb client binaries.
-FROM arangodb:3.8.1 AS deps
+FROM arangodb:3.12.4 AS deps
 
 # This stage installs dependencies to build and load BRON into arangodb. 
-FROM python:3.8-slim AS runtime
+FROM python:3.12-slim AS runtime
 
 RUN apt-get update && \
     apt-get upgrade -y && \
@@ -14,6 +14,8 @@ RUN pip3 install -r requirements.txt
 
 COPY --from=deps /usr/bin/arangoimport /usr/bin/.
 COPY --from=deps /etc/arangodb3/arangoimport.conf /etc/arangodb3/arangoimport.conf
+COPY --from=deps /usr/bin/arangoexport /usr/bin/.
+COPY --from=deps /etc/arangodb3/arangoexport.conf /etc/arangodb3/arangoexport.conf
 COPY . .
 
 ENV DATA_DIR=$dir

README.md

@@ -1,8 +1,12 @@
+[![BRON](docs/figures/bron-logo.png)](docs/figures/brom-logo.png)
+
 # BRON - Link and evaluate public threat and mitigation data for Cyber Hunting
 
+**TODO** update figures
+
 [![BRON February 2023](docs/figures/BRON_drawing.png)](docs/figures/BRON_drawing.png)
 
-Threat data from [MITRE ATT&CK](https://attack.mitre.org/), [CAPEC](https://capec.mitre.org/), [CWE](https://cwe.mitre.org/) , [CVE](https://nvd.nist.gov), [MITRE Engage](https://engage.mitre.org/), [MITRE D3FEND](https://d3fend.mitre.org/), [MITRE CAR](https://car.mitre.org/)  and , [exploitdb](https://exploit-db.com/) data sources are linked together in a graph called BRON. The data types are linked with bidirectional edges. Orange nodes in figure have "offensive" information. Blue nodes in figure are "defensive" information.
+Threat data from [MITRE ATT&CK](https://attack.mitre.org/), [CAPEC](https://capec.mitre.org/), [CWE](https://cwe.mitre.org/) , [CVE](https://nvd.nist.gov), [MITRE Engage](https://engage.mitre.org/), [MITRE D3FEND](https://d3fend.mitre.org/), [MITRE CAR](https://car.mitre.org/) , [exploitdb](https://exploit-db.com/) , [MITRE ATLAS](https://atlas.mitre.org/) data sources are linked together in a graph called BRON. The data types are linked with bidirectional edges. Orange nodes in figure have "offensive" information. Blue nodes in figure are "defensive" information.
 
 ## Deployment
 See [graph_db](graph_db) for a public instance of graph data base implementaion [bron.alfa.csail.mit.edu](http://bron.alfa.csail.mit.edu:8529)
@@ -17,7 +21,7 @@ sudo apt-key add - < Release.key
 echo 'deb https://download.arangodb.com/arangodb310/DEBIAN/ /' | sudo tee /etc/apt/sources.list.d/arangodb.list
 sudo apt-get install apt-transport-https
 sudo apt-get update
-sudo apt-get install arangodb3=3.10.2-1
+sudo apt-get install arangodb3=3.12.4
 
 # Python venv
 python3 -m venv ~/.venvs/BRON-dev
@@ -86,31 +90,33 @@ tail -n 1 build_bron.log
 This should produce a `build_bron.log` file that ends with `END building BRON`.
 
 ## Tutorials
-Tutorials are available in the `tutorials` folder on the following topics:
-- Using BRON in Arangodb, `tutorials/using_bron_graphdb.py`
+Tutorials are available in the `tutorials` folder 
 
 
 ## Usage
 ```
-usage: build_bron.py [-h] --username USERNAME --password PASSWORD --ip IP [--clean] [--clean_local_files] [--delete_mitigations] [--no_download] [--no_parsing] [--no_building] [--no_arangodb]
-                     [--no_mitigations] [--no_validation]
+usage: build_bron.py [-h] --username USERNAME --password PASSWORD --ip IP [--clean] [--clean_local_files] [--no_download] [--no_parsing] [--no_building] [--no_atlas] [--only_recent] [--no_arangodb] [--no_mitigations] [--no_validation] [--start_year START_YEAR] [--end_year END_YEAR]
 
 Build BRON in Arango DB
 
-optional arguments:
+options:
   -h, --help            show this help message and exit
   --username USERNAME   DB username
   --password PASSWORD   DB password
   --ip IP               DB IP address
   --clean               Clean all files and db
   --clean_local_files   Clean all local files
-  --delete_mitigations  Clean all mitigation collections
   --no_download         Do not download data
   --no_parsing          Do not parse data
   --no_building         Do not build BRON
+  --no_atlas            Do not add ATLAS
+  --only_recent         Only recent CVEs
   --no_arangodb         Do not create and import to Arangodb
   --no_mitigations      Do not create and import mitigations
   --no_validation       Do not validate entries imported to the ArangoDb
+  --start_year START_YEAR
+                        Start year
+  --end_year END_YEAR   End year
 ```
 
 ## Structure of BRON
@@ -130,3 +136,17 @@ arXiv report: [https://arxiv.org/abs/2010.00533](https://arxiv.org/abs/2010.0053
       primaryClass={cs.CR}
 }
 ```
+
+DTRAP paper: [https://dl.acm.org/doi/full/10.1145/3615668](https://dl.acm.org/doi/full/10.1145/3615668)
+```
+@article{hemberg2024enhancements,
+  title={Enhancements to threat, vulnerability, and mitigation knowledge for cyber analytics, hunting, and simulations},
+  author={Hemberg, Erik and Turner, Matthew J and Rutar, Nick and O’reilly, Una-May},
+  journal={Digital Threats: Research and Practice},
+  volume={5},
+  number={1},
+  pages={1--33},
+  year={2024},
+  publisher={ACM New York, NY}
+}
+```

bootstrap.sh

@@ -3,5 +3,21 @@
 # parse environment
 ARANGO_ROOT_PASSWORD=$(cat $ARANGO_ROOT_PASSWORD_FILE)
 
+# TODO build BRON a year at a time (Docker can run out of memory)
 # build BRON
-python3 tutorials/build_bron.py --username=root --password=${ARANGO_ROOT_PASSWORD} --ip=brondb
+
+YEARS=(2024 2023 2022 2021 2020 2019 2018 2017 2016 2015 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 2004 2003 2002)
+INITIAL_YEAR=2025
+END_YEAR=$((INITIAL_YEAR + 1))
+echo "++++ Building BRON for ${INITIAL_YEAR} to ${END_YEAR}"
+python3 tutorials/build_bron.py --username=root --password=${ARANGO_ROOT_PASSWORD} --ip=brondb --start_year=${INITIAL_YEAR} --end_year=${END_YEAR} 
+
+for YEAR in ${YEARS[@]}; do
+    END_YEAR=$((YEAR + 1))
+    echo "++++ Building BRON for ${YEAR} to ${END_YEAR}"
+    python3 tutorials/build_bron.py --username=root --password=${ARANGO_ROOT_PASSWORD} --ip=brondb --start_year=${YEAR} --end_year=${END_YEAR} --no_atlas --no_mitigations
+done
+
+echo "++++ Building BRON for final mitigations"
+END_YEAR=$((INITIAL_YEAR + 1))
+python3 tutorials/build_bron.py --username=root --password=${ARANGO_ROOT_PASSWORD} --ip=brondb --no_arangodb --no_atlas --start_year=$INITIAL_YEAR --end_year=$END_YEAR

docker-compose.yml

@@ -1,8 +1,8 @@
-version: "3.5"
 services:
   brondb:
+    platform: linux/arm64
     container_name: brondb
-    image: arangodb:3.8.1
+    image: arangodb:3.12.4
     environment:
       ARANGO_ROOT_PASSWORD_FILE: /run/secrets/arango_root_password
     ports:
@@ -13,6 +13,7 @@ services:
       - brondb_data_container:/var/lib/arangodb3
       - brondb_apps_data_container:/var/lib/arangodb3-apps
   bootstrap:
+    platform: linux/arm64
     container_name: bootstrap
     build: .
     image: bronbootstrap
@@ -24,6 +25,7 @@ services:
     secrets:
       - arango_root_password
     volumes:
+      - .:/usr/local/bron
       - bootstrap_data_container:/data
     links:
       - brondb

docs/figures/bron-logo.png

@@ -9,7 +9,8 @@
 import logging
 
 import requests
-
+import yaml
+from tqdm import tqdm
 
 # TODO how to check if there are updates, or different URLs
 # TODO add some verbosity
@@ -21,10 +22,11 @@
 CAPEC_XML_URL = "http://capec.mitre.org/data/xml/capec_latest.xml"
 # TODO are there better CAPEC views?
 CWE_XML_URL = "http://cwe.mitre.org/data/xml/cwec_latest.xml.zip"
-CVE_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/1.1"
+CVE_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/2.0"
+ATLAS_DATA_URL = "https://raw.githubusercontent.com/mitre-atlas/atlas-data/refs/heads/main/dist/ATLAS.yaml"
 LAST_YEAR = datetime.datetime.now().year + 1
 FIRST_YEAR = 2002
-RECENT_OFFSET = 1
+RECENT_OFFSET = 10
 CVE_ALL_YEARS = list(map(str, range(FIRST_YEAR, LAST_YEAR)))
 CVE_RECENT_YEARS = list(map(str, range(LAST_YEAR - RECENT_OFFSET, LAST_YEAR)))
 
@@ -36,6 +38,7 @@
     "CWE": "raw_CWE.zip",
     "CWE_XML": "raw_CWE_xml.zip",
     "CVE": "raw_CVE.json.gz",
+    "ATLAS": "raw_atlas.yaml",
 }
 BRON_META_DATA_PATH = "bron_meta_data.json"
 
@@ -73,11 +76,26 @@ def md5(file_path: str) -> str:
     return hash_md5.hexdigest()
 
 
+
+def _download_atlas():
+    response = requests.get(ATLAS_DATA_URL)
+    file_path = os.path.join(OUTPUT_FOLDER, THREAT_DATA_TYPES["ATLAS"])
+    data = response.text
+    yaml_data = yaml.safe_load(data)
+    with open(file_path, "w") as fd:
+        yaml.safe_dump(yaml_data, fd)
+
+    assert os.path.exists(file_path)
+    _write_meta_data(threat_data_type="ATLAS", file_path=file_path)
+    logging.info(f"Downloaded ATLAS from {ATLAS_DATA_URL} to {file_path}")
+
+
 def _download_attack():
     response = requests.get(ENTERPRISE_ATTACK_URL)
     file_path = os.path.join(OUTPUT_FOLDER, THREAT_DATA_TYPES["ATTACK"])
     with open(file_path, "w") as fd:
-        json.dump(response.json(), fd)
+        json_data = response.json()
+        json.dump(json_data, fd)
 
     # TODO make sure to remove file before for this assert to be meaningful (or assert timestamps)
     assert os.path.exists(file_path)
@@ -110,10 +128,11 @@ def _download_cwe_xml():
 
 
 def _download_cve(cve_years):
-    combined_cve = {"CVE_Items": []}
-    for year in cve_years:
+    logging.info(f"Begin download CVEs from {CVE_BASE_URL} for {cve_years}")
+    combined_cve = {"vulnerabilities": []}
+    for year in tqdm(cve_years, desc="Downloading CVEs"):
         # Download CVE data for each year
-        cve_url = os.path.join(CVE_BASE_URL, f"nvdcve-1.1-{year}.json.gz")
+        cve_url = os.path.join(CVE_BASE_URL, f"nvdcve-2.0-{year}.json.gz")
         response = requests.get(cve_url, stream=True)
         year_file_path = os.path.join(OUTPUT_FOLDER, f"raw_CVE_{year}.json.gz")
         with open(year_file_path, "wb") as fd:
@@ -124,7 +143,7 @@ def _download_cve(cve_years):
         cve_path = os.path.join(OUTPUT_FOLDER, f"raw_CVE_{year}.json.gz")
         with gzip.open(cve_path, "rt", encoding="utf-8") as f:
             cve_data = json.load(f)
-        combined_cve["CVE_Items"].extend(cve_data["CVE_Items"])
+        combined_cve["vulnerabilities"].extend(cve_data["vulnerabilities"])
     json_string = json.dumps(combined_cve)
     encoded = json_string.encode("utf-8")
     file_path = os.path.join(OUTPUT_FOLDER, THREAT_DATA_TYPES["CVE"])
@@ -141,6 +160,7 @@ def main(cve_years) -> None:
         os.makedirs(OUTPUT_FOLDER)
         logging.info(f"Created {OUTPUT_FOLDER}")
 
+    _download_atlas()
     _download_attack()
     _download_capec_xml()
     _download_cwe_xml()
@@ -196,6 +216,8 @@ def clean():
             _download_cwe_xml()
         elif args.threat_data_type == "CVE":
             _download_cve(cve_years_)
+        elif args.threat_data_type == "ATLAS":
+            _download_atlas()            
         else:
             raise Exception(f"Bad threat data type: {args.threat_data_type}")
     else: