renamed jira secret scanner to be more consistent

Author: cutler-scott-newrelic

scripts/README.md

@@ -4,7 +4,7 @@ This is a collection of scripts that act as wrappers around the Rusty Hog binari
 They provide additional functionality that the New Relic security team uses to monitor
 and perform wider scans.
 
-## GHE Secret Monitor
+## ghe_secret_monitor.py
 
 This is a Python script, re-written based on Douglas Day's work, that performs a scan
 of the last 24 hours of commits for an entire GitHub Enterprise instance. It outputs
@@ -28,13 +28,12 @@ sudo systemctl enable ghe_secret_monitor
 You can then perform a one-time execution of secret_scanner with the command 
 `sudo service ghe_secret_monitor start` and examine the results in /var/log/messages
 
-## JIRA Secret Scanner
+## jira_secret_monitor.py
 
-This is a Python script, also re-written based on Douglas Day's work, that performs a
-scan of any Google Docs that were linked in JIRA over the last 24 hours. Essentially 
-it runs a JQL query for all tickets modified in the last 24 hours, collects all GDrive 
-links from the text and comments of each JIRA ticket, and runs ankamali_hog against each
-document. It then collects the results and outputs them to New Relic Insights. You can use 
+This is a Python script, also re-written based on Douglas Day's work, that performs 2 scans: 
+1) It scans all JIRA tickets modified in the last 24 hours for secrets using gottingen_hog.
+2) Within those JIRA tickets it looks for GDrive links and scans those docs for secrets using anakmali_hog.
+It then collects the results and outputs them to New Relic Insights. You can use 
 the same installation method as above, substituting jira_secret_scanner for secret_scanner
 in each step.
 
@@ -89,6 +88,8 @@ INSIGHTS_INSERT_KEY - the New Relic Insights Insert API key for results
 INSIGHTS_ACCT_ID - the New Relic Insights account number for results
 DUROC_HOG_PATH - the path to the duroc hog binary (relative or absolute)
 
+You will also need to install the third party python library htmllistparse
+
 ## s3weblisting_secret_monitor.py
 
 This is a python script meant to perform a Rusty Hog scan for all binaries on a web server that uses the generic
@@ -100,4 +101,6 @@ You will need to supply 4 environment variables to it:
 DOWNLOAD_CONFIG_PATH - the path to the JSON config file, e.g. scripts/s3weblisting_secret_monitor.json
 INSIGHTS_INSERT_KEY - the New Relic Insights Insert API key for results
 INSIGHTS_ACCT_ID - the New Relic Insights account number for results
-DUROC_HOG_PATH - the path to the duroc hog binary (relative or absolute)
+DUROC_HOG_PATH - the path to the duroc hog binary (relative or absolute)
+
+You will also need to install the third party python library htmllistparse

scripts/ghe_secret_monitor.py

@@ -1,3 +1,25 @@
+# This is a Python script, re-written based on Douglas Day's work, that performs a scan
+# of the last 24 hours of commits for an entire GitHub Enterprise instance. It outputs
+# the results to the New Relic Insights product which allows you to create alerts and
+# visualizations based on the scanning results. It was designed to be run on an Amazon Linux
+# EC2 instance on a nightly basis. You can install it using the following commands:
+#
+# ```shell script
+# cd ~
+# mkdir rusty-hog
+# wget https://github.com/newrelic/rusty-hog/releases/download/v1.0.4/rustyhogs-musl_darwin_1.0.4.zip
+# unzip rustyhogs-musl_darwin_1.0.4.zip -d rusty_hog_1.0.4
+# cd rusty_hog_1.0.4
+# sudo cp ghe_secret_monitor.service /etc/systemd/system
+# sudo vi ghe_secret_monitor.timer # modify each <redacted for git> field
+# sudo cp ghe_secret_monitor.timer /etc/systemd/system
+# sudo systemctl daemon-reload
+# sudo systemctl enable ghe_secret_monitor
+# ```
+#
+# You can then perform a one-time execution of secret_scanner with the command
+# `sudo service ghe_secret_monitor start` and examine the results in /var/log/messages
+
 from datetime import datetime, timedelta
 from github import Github, GithubException
 from multiprocessing import Pool

scripts/htmldirlisting_secret_monitor.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 
-# This is a simple script meant to retrieve the latest pypi package (provided through environment variables) and perform a
-# Rusty Hog scan on the contents of the download. It will then post the results to Insights.
+# This is a simple script meant to retrieve all files from a web server with an HTML directory listing and scan
+# the files for secrets using duroc_hog. It will then post the results to Insights.
 
 import os
 import gzip

scripts/jira_secret_scanner.py scripts/jira_secret_monitor.pyscripts/jira_secret_scanner.py renamed to scripts/jira_secret_monitor.py

@@ -1,10 +1,10 @@
 [Unit]
 Description=Runs JIRA + GDrive Secret Scanner scan
-Wants=jira_secret_scanner.timer
+Wants=jira_secret_monitor.timer
 
 [Service]
 Type=oneshot
-ExecStart=/home/ec2-user/anaconda3/bin/python jira_secret_scanner.py
+ExecStart=/home/ec2-user/anaconda3/bin/python jira_secret_monitor.py
 WorkingDirectory=/home/ec2-user/rusty_hog_1.0.4
 User=ec2-user
 Group=ec2-user

scripts/jira_secret_scanner.service scripts/jira_secret_monitor.servicescripts/jira_secret_scanner.service renamed to scripts/jira_secret_monitor.service

@@ -1,6 +1,6 @@
 [Unit]
 Description=Run secret_scanner every day at 11:30pm pacific
-Requires=jira_secret_scanner.service
+Requires=jira_secret_monitor.service
 
 [Timer]
 OnCalendar=*-*-* 7:30:00

scripts/jira_secret_scanner.timer scripts/jira_secret_monitor.timerscripts/jira_secret_scanner.timer renamed to scripts/jira_secret_monitor.timer

@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 
-# This is a simple script meant to retrieve the latest pypi package (provided through environment variables) and perform a
-# Rusty Hog scan on the contents of the download. It will then post the results to Insights.
+# This is a simple script meant to retrieve all files from a web server with an HTML-fronted S3 bucket and scan
+# the files for secrets using duroc_hog. It will then post the results to Insights.
 
 import os
 import gzip