Skip to content

Commit 1a34778

Browse files
cutler-scott-newreliccutler-scott-newrelic
committed
initial commit
1 parent 7e8c52a commit 1a34778

15 files changed

+2792
-0
lines changed

‎.gitignore‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
/target
2+
**/*.rs.bk
3+
Cargo.lock
4+
.idea
5+
*.iml
6+
*.ipr
7+
*.iws
8+
.gradle
9+
out/
10+
build/
11+
gen/
12+
deps/
13+
exampleProject
14+
testData
15+
.DS_Store
16+
*.json
17+
bootstrap
18+
*.zip
19+
temp_token

‎CODE_OF_CONDUCT.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Contributor Covenant Code of Conduct
2+
3+
## Our Pledge
4+
5+
In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
6+
7+
## Our Standards
8+
9+
Examples of behavior that contributes to creating a positive environment include:
10+
11+
* Using welcoming and inclusive language
12+
* Being respectful of differing viewpoints and experiences
13+
* Gracefully accepting constructive criticism
14+
* Focusing on what is best for the community
15+
* Showing empathy towards other community members
16+
17+
Examples of unacceptable behavior by participants include:
18+
19+
* The use of sexualized language or imagery and unwelcome sexual attention or advances
20+
* Trolling, insulting/derogatory comments, and personal or political attacks
21+
* Public or private harassment
22+
* Publishing others' private information, such as a physical or electronic address, without explicit permission
23+
* Other conduct which could reasonably be considered inappropriate in a professional setting
24+
25+
## Our Responsibilities
26+
27+
Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
28+
29+
Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
30+
31+
## Scope
32+
33+
This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
34+
35+
## Enforcement
36+
37+
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at opensource@newrelic.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
38+
39+
Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
40+
41+
## Attribution
42+
43+
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
44+
45+
[homepage]: http://contributor-covenant.org
46+
[version]: http://contributor-covenant.org/version/1/4/

‎CONTRIBUTING.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Contributing
2+
3+
Contributions are always welcome. Before contributing please read the
4+
[code of conduct](./CODE_OF_CONDUCT.md) and [search the issue tracker](issues); your issue may have already been discussed or fixed in `master`. To contribute,
5+
[fork](https://help.github.com/articles/fork-a-repo/) this repository, commit your changes, and [send a Pull Request](https://help.github.com/articles/using-pull-requests/).
6+
7+
Note that our [code of conduct](./CODE_OF_CONDUCT.md) applies to all platforms and venues related to this project; please follow it in all your interactions with the project and its participants.
8+
9+
## Feature Requests
10+
11+
Feature requests should be submitted in the [Issue tracker](../../issues), with a description of the expected behavior & use case, where they’ll remain closed until sufficient interest, [e.g. :+1: reactions](https://help.github.com/articles/about-discussions-in-issues-and-pull-requests/), has been [shown by the community](../../issues?q=label%3A%22votes+needed%22+sort%3Areactions-%2B1-desc).
12+
Before submitting an Issue, please search for similar ones in the
13+
[closed issues](../../issues?q=is%3Aissue+is%3Aclosed+label%3Aenhancement).
14+
15+
## Pull Requests
16+
17+
1. Ensure any install or build dependencies are removed before the end of the layer when doing a build.
18+
2. Increase the version numbers in any examples files and the README.md to the new version that this Pull Request would represent. The versioning scheme we use is [SemVer](http://semver.org/).
19+
3. You may merge the Pull Request in once you have the sign-off of two other developers, or if you do not have permission to do that, you may request the second reviewer to merge it for you.
20+
21+
## Contributor License Agreement
22+
23+
Keep in mind that when you submit your Pull Request, you'll need to sign the CLA via the click-through using CLA-Assistant. If you'd like to execute our corporate CLA, or if you have any questions, please drop us an email at opensource@newrelic.com.
24+
25+
For more information about CLAs, please check out Alex Russell’s excellent post,
26+
[“Why Do I Need to Sign This?”](https://infrequently.org/2008/06/why-do-i-need-to-sign-this/).
27+
28+
## Slack
29+
30+
For contributors and maintainers of open source projects hosted by New Relic, we host a public Slack with a channel dedicated to this project. If you are contributing to this project, you're welcome to request access to that community space.

‎Cargo.toml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
[package]
2+
name = "rusty_hogs"
3+
version = "0.4.4"
4+
authors = ["Scott Cutler <scutler@newrelic.com>"]
5+
edition = "2018"
6+
7+
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
8+
[lib]
9+
name = "secret_scanning"
10+
path = "src/lib.rs"
11+
12+
[dependencies]
13+
git2 = "0.10"
14+
serde = { version = "1.0", features = ["derive"] }
15+
serde_json = "1.0"
16+
serde_derive = "^1"
17+
clap = "2"
18+
regex = "1"
19+
url = "2"
20+
tempdir = "0.3"
21+
base64 = "0.11.0"
22+
log = "^0.4"
23+
simple_logger = "^1"
24+
simple-error = "0.2"
25+
chrono = "0.4"
26+
encoding = "0.2"
27+
hex = "0.4"
28+
lambda_runtime = "0.2"
29+
rust-s3 = "0.18.3"
30+
google-drive3 = "*"
31+
# This project intentionally uses an old version of Hyper. See
32+
# https://github.com/Byron/google-apis-rs/issues/173 for more
33+
# information.
34+
hyper = "^0.10"
35+
hyper-rustls = "^0.6"
36+
yup-oauth2 = "^1.0"

‎README.md‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
# Rusty Hogs
2+
A suite of secret scanners built in Rust for performance. Based on [TruffleHog](https://github.com/dxa4481/truffleHog)
3+
which is written in Python.
4+
5+
Ankamali Hog: Scan for secrets in a Google Doc
6+
7+
Berkshire Hog: Scan for secrets in an S3 bucket
8+
9+
Choctaw Hog: Scan for secrets in a Git repository
10+
11+
* [Rusty Hogs](#rusty-hogs)
12+
* [How to run](#how-to-run)
13+
* [How to build](#how-to-build)
14+
* [Anakamali Hog Usage](#anakamali-hog-usage)
15+
* [Berkshire Hog (CLI) Usage](#berkshire-hog-cli-usage)
16+
* [Berkshire Hog (Lambda) Usage](#berkshire-hog-lambda-usage)
17+
* [Choctaw Hog Usage](#choctaw-hog-usage)
18+
* [Open Source License](#open-source-license)
19+
* [Support](#support)
20+
* [Community](#community)
21+
* [Issues / Enhancement Requests](#issues--enhancement-requests)
22+
* [Contributing](#contributing)
23+
* [Feature Roadmap](#feature-roadmap)
24+
* [Performance comparison](#performance-comparison)
25+
* [What does the name mean?](#what-does-the-name-mean)
26+
27+
## How to run
28+
Download and unzip the [latest ZIP](https://source.datanerd.us/security/rusty_hogs/releases/download/0.4.1/release.zip)
29+
on the releases tab, then you can run each binary with `-h` to see the usage.
30+
31+
```shell script
32+
wget https://source.datanerd.us/security/rusty_hogs/releases/download/0.4.1/release.zip
33+
unzip release.zip
34+
cd darwin_releases
35+
./choctaw_hog -h
36+
```
37+
38+
## How to build
39+
Ensure you have [Rust](https://www.rust-lang.org/learn/get-started) installed and on your path.
40+
41+
Perform a git clone, then run `cargo build --release`. The binaries will be located in `target/release`
42+
43+
To cross-compile Berkshire Hog for the AWS Lambda environment, first install
44+
[cross](https://github.com/rust-embedded/cross). Then run the following commands and upload berkshire_lambda.zip:
45+
```shell script
46+
cross build --release --target x86_64-unknown-linux-musl
47+
cp target/x86_64-unknown-linux-musl/release/berkshire_hog bootstrap
48+
zip -j berkshire_lambda.zip bootstrap
49+
```
50+
51+
## Anakamali Hog Usage
52+
```
53+
USAGE:
54+
ankamali_hog [FLAGS] [OPTIONS] <GDRIVEID>
55+
56+
FLAGS:
57+
--caseinsensitive Sets the case insensitive flag for all regexes
58+
--entropy Enables entropy scanning
59+
--prettyprint Output the JSON in human readable format
60+
-v, --verbose Sets the level of debugging information
61+
-h, --help Prints help information
62+
-V, --version Prints version information
63+
64+
OPTIONS:
65+
-o, --outputfile <OUTPUT> Sets the path to write the scanner results to (stdout by default)
66+
--regex <REGEX> Sets a custom regex JSON file
67+
68+
ARGS:
69+
<GDRIVEID> The ID of the google drive file you want to scan
70+
```
71+
72+
## Berkshire Hog (CLI) Usage
73+
```
74+
USAGE:
75+
berkshire_hog [FLAGS] [OPTIONS] <S3URI> <S3REGION>
76+
77+
FLAGS:
78+
--caseinsensitive Sets the case insensitive flag for all regexes
79+
--entropy Enables entropy scanning
80+
--prettyprint Output the JSON in human readable format
81+
-r, --recursive Will recursively scan files under the prefix.
82+
-v, --verbose Sets the level of debugging information
83+
-h, --help Prints help information
84+
-V, --version Prints version information
85+
86+
OPTIONS:
87+
-o, --outputfile <OUTPUT> Sets the path to write the scanner results to (stdout by default)
88+
--profile <PROFILE> When using a configuration file, use a non-default profile
89+
--regex <REGEX> Sets a custom regex JSON file
90+
91+
ARGS:
92+
<S3URI> The location of a S3 bucket and optional prefix or filename to scan. This must be written in the form
93+
s3://mybucket[/prefix_or_file]
94+
<S3REGION> Sets the region of the S3 bucket to scan.
95+
```
96+
97+
98+
## Berkshire Hog (Lambda) Usage
99+
Berkshire Hog is currently designed to be used as a Lambda function. It was written with this overall data-flow
100+
in mind:
101+
<pre>
102+
┌───────────┐ ┌───────┐ ┌────────────────┐ ┌────────────┐
103+
│ S3 Bucket │ ┌────────┐ │ │ │ Berkshire Hog │ │ S3 Bucket │
104+
│ (input) ─┼─┤S3 Event├──▶│ SQS │────▶│ (Lambda) │────▶│ (output) │
105+
│ │ └────────┘ │ │ │ │ │ │
106+
└───────────┘ └───────┘ └────────────────┘ └────────────┘
107+
</pre>
108+
109+
In order to run this you will need to setup the following things:
110+
1) The input bucket must be configured to send an "event" to SQS for each PUSH/PUT event
111+
2) The SQS topic must be setup to accept events from S3, including IAM permissions.
112+
3) Berkshire hog must be running with IAM access to SQS and S3.
113+
114+
## Choctaw Hog Usage
115+
```
116+
USAGE:
117+
choctaw_hog [FLAGS] [OPTIONS] <GITPATH>
118+
119+
FLAGS:
120+
--caseinsensitive Sets the case insensitive flag for all regexes
121+
--entropy Enables entropy scanning
122+
--prettyprint Output the JSON in human readable format
123+
-v, --verbose Sets the level of debugging information
124+
-h, --help Prints help information
125+
-V, --version Prints version information
126+
127+
OPTIONS:
128+
-o, --outputfile <OUTPUT> Sets the path to write the scanner results to (stdout by default)
129+
--regex <REGEX> Sets a custom regex JSON file, defaults to ./trufflehog_rules.json
130+
--since_commit <SINCECOMMIT> Filters commits based on date committed (branch agnostic)
131+
--sshkeypath <SSHKEYPATH> Takes a path to a private SSH key for git authentication, defaults to ssh-agent
132+
--sshkeyphrase <SSHKEYPHRASE> Takes a passphrase to a private SSH key for git authentication, defaults to
133+
none
134+
135+
ARGS:
136+
<GITPATH> Sets the path (or URL) of the Git repo to scan. SSH links must include username (git@)
137+
```
138+
139+
## Open Source License
140+
141+
This project is distributed under the [Apache 2 license](LICENSE).
142+
143+
## Support
144+
145+
New Relic has open-sourced this project. This project is provided AS-IS WITHOUT WARRANTY OR SUPPORT, although you can report issues and contribute to the project here on GitHub.
146+
147+
_Please do not report issues with this software to New Relic Global Technical Support._
148+
149+
## Community
150+
151+
New Relic hosts and moderates an online forum where customers can interact with New Relic employees as well as other customers to get help and share best practices. Like all official New Relic open source projects, there's a related Community topic in the New Relic Explorer's Hub. You can find this project's topic/threads here:
152+
153+
TODO: Create topic in discuss.newrelic.com and put the link here.
154+
155+
## Issues / Enhancement Requests
156+
157+
Issues and enhancement requests can be submitted in the [Issues tab of this repository](../../issues). Please search for and review the existing open issues before submitting a new issue.
158+
159+
## Contributing
160+
161+
Contributions are welcome (and if you submit a Enhancement Request, expect to be invited to contribute it yourself :grin:). Please review our [Contributors Guide](CONTRIBUTING.md).
162+
163+
Keep in mind that when you submit your pull request, you'll need to sign the CLA via the click-through using CLA-Assistant. If you'd like to execute our corporate CLA, or if you have any questions, please drop us an email at opensource@newrelic.com.
164+
165+
166+
## Feature Roadmap
167+
- 1.0: Initial open-source release
168+
- [x] Refactor git-agnostic code into a reusable library
169+
- [x] Implement logging correctly
170+
- [x] Prep for New Relic Homebrew release
171+
- [x] Prep for New Relic GitHub release
172+
- [x] Implement licensing
173+
- [x] Clear with New Relic open source committee
174+
- [x] Finish initial implementation of Ankamali Hog and Berkshire Hog CLI
175+
- [ ] Finish New Relic Open Source checklist
176+
- [ ] Unit tests
177+
- [ ] Prep for crates.io release
178+
- [ ] Flatten original Git repo
179+
180+
- 1.1: Enterprise features
181+
- [ ] Support config files (instead of command line args)
182+
- [ ] Save state between scans, remember and filter "false positives"
183+
- [ ] Multi-threading
184+
- [ ] Better context detection and false positive filtering (GitHound, machine learning)
185+
- [ ] Support for other modes of use for Berkshire Hog (CLI, lambda without SQS)
186+
- [ ] Use Rusoto instead of s3-rust
187+
188+
- 1.2: Integration with larger scripts and UIs
189+
- [ ] Support Github API for larger org management
190+
- [ ] Scan all repos for a list of users
191+
- [ ] Scan all repos in an org
192+
- [ ] Generate a web-report or web-interface. Support "save state" generation from UI.
193+
- [ ] Agent/manager model
194+
- [ ] Scheduler process (blocked by save state support)
195+
196+
197+
## Performance comparison
198+
Using this repo as a test: `git clone git@github.com:NathanRomike/dictionary-builder.git`
199+
200+
I ran trufflehog 50 times and saw it take 81 seconds...
201+
```
202+
time ( repeat 50 { trufflehog --rules trufflehog_rules.json --regex --entropy=False ../dictionary-builder/ })
203+
204+
37.67s user 40.56s system 95% cpu 1:21.88 total
205+
```
206+
207+
Then I ran Choctaw Hog 50 times and saw it take 49 seconds...
208+
```
209+
time ( repeat 50 { target/release/choctaw_hog ../dictionary-builder })
210+
211+
46.28s user 1.94s system 98% cpu 48.749 total
212+
```
213+
214+
## What does the name mean?
215+
TruffleHog is considered the de-facto standard / original secret scanner. I have been
216+
building a suite of secret scanning tools for various platforms based on TruffleHog
217+
and needed a naming scheme, so I started at the top of Wikipedia's
218+
[list of pig breeds](https://en.wikipedia.org/wiki/List_of_pig_breeds).
219+
Thus each tool name is a breed of pig starting at A and working up.
220+

0 commit comments

Comments
 (0)