New documentation relying on mkdocs-material and a github action for deploying doc changes

llavarello · llavarello · commit 5e1088f9d079 · 2024-08-06T12:20:25.000-03:00
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -0,0 +1,21 @@
+name: ci
+on:
+  push:
+    branches:
+      - main
+permissions:
+  contents: write
+jobs:
+  docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.x
+      - uses: actions/cache@v2
+        with:
+          key: ${{ github.ref }}
+          path: .cache
+      - run: pip install mkdocs-material
+      - run: mkdocs gh-deploy --force
diff --git a/docs/code_structure.md b/docs/code_structure.md
@@ -0,0 +1,27 @@
+# Code structure
+
+A few pointers on how the Gitxray codebase is structured:
+
+* `gitxray.py` - The main script, creates a gx_context, gx_output and calls X-Ray modules.
+
+The include directory has files split in two naming conventions:
+
+* Suffix: `gh_` - Files that handle GitHub API responses or talk to the GitHub API
+* Suffix: `gx_` - Files with more Gitxray-specific logic
+
+Some of the supporting files in the include directory:
+
+* `gx_context.py` - Holds a context of data that is shared across different X-Ray modules and through out execution.
+* `gx_output.py` - Takes care of any console printing, as well as text and json output.
+
+For parsing SSH and PGP signatures, we wrote our own code and placed it in:
+
+* `gx_ugly_openpgp_parser.py` - Parses Armors and BLOBs based on RFC4880
+* `gx_ugly_ssh_parser.py` - Parses (if you can call that Parsing) SSH Armors and BLOBs
+
+Finally, last but not least important, the X-Rays under the xrays directory:
+
+* `contributors_xray.py` - Handles all Contributor-related data and decides what to log.
+* `repository_xray.py` - Handles all Repository-related data and decides what to log.
+* `associations_xray.py` - Analyzes and reports all associations carried from prior X-Ray modules.
+
diff --git a/docs/forensics_spotting.md b/docs/forensics_spotting.md
@@ -0,0 +1,52 @@
+# Forensics and Spotting Associations
+
+Open source projects are under attack, with malicious actors hiding in plain sight. Attackers are normally well-funded and take precautions to avoid detection. Enter some of `gitxray`'s features: If you've woken up today wearing your Batman t-shirt and/or you're carefully looking at evidence while protecting a repository, try leveraging some of `gitxray`'s features.
+
+## Important - Please read
+
+Associations MUST NOT be directly and blindly used to report fake or shadow accounts. They are automatic observations from a piece of well-intended code. Do NOT treat association results as findings directly. 
+
+Remember, there is more good than evil out there. We must protect open-source projects by first and foremost respecting open-source developers. Ensure that any actions taken are thoughtful and based on solid evidence, not just automated associations. Collaboration and trust are the foundations of the open-source community, and preserving these values is crucial for its continued success.
+## Run a full verbose X-Ray on a repository
+``` bash
+gitxray -r https://github.com/SampleOrg/Repo1 -v
+```
+
+## Focus on associations
+Associations are currently supported at a _Repository level only_. 
+
+Spotted associations can include:
+
+* Sharing of PGP Keys or Emails
+* Accounts that were created and/or updated precisely in the same day
+* Accounts which relied on GitHub's Web Editor to sign commits
+* Accounts which used the same Algorithms for SSH/PGP signatures
+* Armored Keys which displayed the same Comment or Version fields
+
+``` bash
+gitxray -r https://github.com/SampleOrg/repo1 -f association
+```
+
+## Spot users who are in the Top 3 of Rejected PRs
+``` bash
+gitxray -r https://github.com/SampleOrg/repo1 -f contributors
+```
+
+## Releases and Assets
+When reviewing repository Releases and Assets, Gitxray will:
+
+* Print out which users have historically created releases and/or uploaded assets, as well as the % vs. the total amount of releases or assets; so you can flag potential suspicious activity.
+
+* Report on any Assets that were uploaded at least a day AFTER their initial release, which might lead to suggest they've been infected and/or tampered with.
+
+``` bash
+gitxray -r https://github.com/SampleOrg/repo1 -f user_input,pass,private
+```
+
+## Observing user Recent behaviour 
+When Verbose mode is enabled, recent events from the past 90 days for each user account will be processed and printed by Gitxray, among which you may spot:
+* Accounts which are automatically following other users. You'll likely notice a large amount of consecutive follow events.
+* Accounts which are automatically starring repositories. You'll likely notice a large amount of consecutive watching activity.
+* Accounts which are forking a large amount of repositories; potentially for the sake of tricking victims into using their tampered fork.
+
+More use-cases for gitxray can be found in the [OSINT and Pentesting](osint_pentest.md) and [Securing your Team and Repositories](securing_repos.md) sections.
diff --git a/docs/ideas_pending.md b/docs/ideas_pending.md
@@ -0,0 +1,7 @@
+# Ideas pending to be implemented
+
+* Use repository stats, eg the punch_card endpoint to identify commits made at a time of day which break patterns
+* SSH Public Keys: Improve BLOB parsing and collect Key strength bits which could be used to link to other User keys.
+* Try to identify starring attacks (fake stars in repositories or accounts following eachother) by using a non-expensive path (eg few requests)
+* Inspect Issues to identify potential malicious behavior by using a non-expensive path (e.g. a call to events per issue would lead to a lot of data, but it requires 1 request per [open] issue)
+* Analyze "reactions" and see if we can identify mood in some way. 
diff --git a/docs/images/console_gitxray.png b/docs/images/console_gitxray.png
diff --git a/docs/images/logo_gitxray.png b/docs/images/logo_gitxray.png
diff --git a/docs/index.md b/docs/index.md
@@ -0,0 +1,28 @@
+# Welcome to Gitxray
+
+Gitxray (short for Git X-Ray) is a multifaceted security tool designed for use on GitHub repositories. It serves various use cases, including OSINT, forensics, and security teams, as well as developers looking to secure their repositories, organizations, and related contributors. Gitxray leverages public GitHub REST APIs to gather information that would otherwise be very time-consuming to obtain manually. Additionally, it seeks out information in unconventional places.
+
+## Use-cases when using Gitxray
+
+* [OSINT and Pentesting](osint_pentest.md)
+* [Forensics and Spotting Associations](forensics_spotting.md)
+* [Securing your Team and Repositories](securing_repos.md)
+
+## Rate Limits and the GitHub API
+
+Gitxray can work out of the box without a GitHub API key, _but_ you'll likely hit RateLimits pretty fast. This is detailed by GitHub in their [documentation here](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#primary-rate-limit-for-unauthenticated-users). A simple read-only token created for PUBLIC repositories will however help you increase those restrictions considerably. If you're not in a hurry or can leave Gitxray running you'll be able to use Gitxray in its full capacity, as it pauses execution while waiting for the limits to lift.
+
+## Reasons to &hearts; Gitxray 
+
+A few reasons why we think Gitxray is special include:
+
+* It parses Signature BLOBs in order to capture Personal Information, Algorithm configurations, Creation Time subpackets, and more.
+* It displays user-supplied data obtained from weird places, such as Key names and copy-paste mistakes appended before/after armored Keys.
+* It cross-references data from different users, displaying what we call "*associations*", likely revealing _potential_ shared or fake accounts.
+* It inspects Repository releases and identifies when assets were updated *after* being released; _potentially_ implying infected or tampered assets.
+* And a lot more for you to discover.
+
+
+## Ready to get started?
+
+Jump to [Installing and running Gitxray](installing.md) in order to get started.
diff --git a/docs/installing.md b/docs/installing.md
@@ -0,0 +1,74 @@
+# Installing and running Gitxray
+
+gitxray was written with no use of external package dependencies other than the `requests` library.
+
+## PyPI (PIP) Way
+
+`gitxray` is on PyPI and can be installed with:
+
+```bash
+pip install gitxray
+```
+
+Once installed, simply run gitxray from your command line by typing:
+```bash
+gitxray -h
+```
+
+## Installing from source
+
+You may also run `gitxray` directly by cloning or downloading its GitHub [repository](https://github.com/kulkansecurity/gitxray/) and running:
+
+```bash
+python3 -m pip install -r requirements.txt
+cd src/
+python3 -m gitxray.gitxray
+```
+
+## Command Line Arguments
+
+### Required Arguments
+
+One of the following must be specified:
+
+* `-r, --repository [URL]` - Specify a single repository URL to check. The URL must begin with `https://`. **Example**: `--repository https://github.com/example/repo`
+
+* `-rf, --repositories-file [FILEPATH]` - Provide a file path containing a list of repositories, each on a new line. The file must exist. **Example**: `--repositories-file ./list_of_repos.txt`
+
+* `-o, --organization [URL]` - Specify an organization URL to check all repositories under that organization. The URL must begin with `https://`. **Example**: `--organization https://github.com/exampleOrg`
+
+### Optional Arguments
+
+You'll find these optional but very handy in common gitxray usage.
+
+- `-l, --list` - List contributors if a repository is specified or list repositories if an organization is specified. Useful for further focusing on specific entities. **Example**: `--list`
+
+- `-c, --contributor [USERNAMES]` - A comma-separated list of GitHub usernames to focus on within the specified repository or organization. **Example**: `--contributor user1,user2`
+
+- `-f, --filters [KEYWORDS]` - Comma-separated keywords to filter the results by, such as 'user_input', 'association', or 'mac'. **Example**: `--filters user_input,association,mac`
+
+#### Verbose and Debug
+- `-v, --verbose` - Enable verbose output which, for example, provides a detailed list of public events instead of a summary. **Example**: `--verbose`
+
+- `--debug` - Enable Debug mode for a detailed and extensive output. **Example**: `--debug`
+
+#### Output and Formats
+
+- `-out, --outfile [FILEPATH]` - Specify the file path for the output log. Cannot be a directory. **Example**: `--outfile ./output.log`
+
+- `-outformat, --output-format [FORMAT]` - Set the format for the log file. Supported formats are `text` and `json`. Default is `text`. **Example**: `--output-format json`
+
+### Usage Examples
+Refer to [Use cases](index.md#use-cases-when-using-gitxray) for combinations of commands aligned with specific use-cases.
+
+``` bash
+# Analyze a single repository with verbose output
+gitxray --repository https://github.com/example/repo --verbose
+
+# List all repository names under a given Organization
+gitxray --organization https://github.com/exampleOrg --list
+
+# Analyze all repositories under an Organization with filters capturing user_input and associations
+gitxray -org https://github.com/exampleOrg -f user_input,association
+```
+
diff --git a/docs/osint_pentest.md b/docs/osint_pentest.md
@@ -0,0 +1,46 @@
+# OSINT and Pentesting
+
+A common use-case for using `gitxray` is to gather as much information as possible for a target Organization, Repository and/or Contributors. Here are a few suggestions on commands you may want to run. You'll soon discover your own favorite combo.
+
+## Listing repositories for a given Organization
+``` bash
+gitxray -o https://github.com/SampleOrg -l
+```
+
+## Listing contributors for a given Repository
+``` bash
+gitxray -r https://github.com/SampleOrg/repo1 -l
+```
+
+## Examining all contributors in a repository with Verbose enabled
+``` bash
+gitxray -r https://github.com/SampleOrg/repo1 -v
+```
+
+## Filtering for potential disclosures in user_input
+``` bash
+gitxray -r https://github.com/SampleOrg/repo1 -f user_input,pass
+```
+
+## Filtering for collected e-mail addresses
+``` bash
+gitxray -r https://github.com/SampleOrg/repo1 -f emails
+```
+
+## Focusing on personal fields 
+``` bash
+gitxray -r https://github.com/SampleOrg/repo1 -f personal
+```
+
+## Searching for key-related information across an entire Organization
+``` bash
+gitxray -o https://github.com/SampleOrg -f keys
+```
+
+## Filtering in Verbose mode for users who recently moved repositories from Private to Public
+``` bash
+gitxray -o https://github.com/SampleOrg -v -f private
+```
+
+More use-cases for gitxray can be found in the [Forensics and Spotting Associations](forensics_spotting.md) and [Securing your Team and Repositories](securing_repos.md) sections.
+
diff --git a/docs/securing_repos.md b/docs/securing_repos.md
@@ -0,0 +1,41 @@
+# Securing your Team and Repositories
+
+If you have one or more repositories to safeguard, then you may be able to leverage `gitxray` to improve the security stance of your repository and/or your development team.
+
+## Checking contributors for information disclosure
+
+You may be concerned about what your team members are currently sharing in their profile to the world.
+
+Filtering results across all Organization repositories focused on personal information and emails
+``` bash
+gitxray -o https://github.com/SampleOrg -f personal,emails
+```
+
+We've discovered A LOT of internal information being disclosed by accident through Key names. Try filtering for user_input to find those cases
+``` bash
+gitxray -o https://github.com/SampleOrg -f user_input
+```
+
+## Checking for contributors who have not signed their commits
+Those who have not signed ANY of their commits may need your help in understanding how signing is done and the dangers of impersonation.
+
+``` bash
+gitxray -r https://github.com/SampleOrg/repo1 -f "not signed any"
+```
+
+## Checking for contributors who have a mix of signed and unsigned commits
+These contributors likely need to be reminded of the importance of signing, AND their unsigned commits could require an audit.
+
+``` bash
+gitxray -r https://github.com/SampleOrg/repo1 -f "has a mix of"
+```
+
+## Checking for broken signatures
+These are attempts at signing that have likely failed for multiple reasons. You'll likely want to check the reason behind these failures; as it sometimes is as simple as the contributor not having uploaded their public key to their profile.
+
+``` bash
+gitxray -r https://github.com/SampleOrg/repo1 -f signatures
+```
+
+More use-cases for gitxray can be found in the [Forensics and Spotting Associations](forensics_spotting.md) and [Securing your Team and Repositories](securing_repos.md) sections.
+
diff --git a/docs/show_love.md b/docs/show_love.md
@@ -0,0 +1,10 @@
+# Show your love &hearts;
+
+If and only If `gitxray` has been helpful to you, your support in spreading the `gitxray` word is appreciated. 
+
+* <a href="https://twitter.com/intent/tweet?text=I'm%20using%20Gitxray%20(https://github.com/kulkansecurity/gitxray/)%20developed%20by%20%40kulkansecurity%20and%20I'm%20loving%20it!" target='_blank'>Share on Twitter</a>
+* <a href="https://www.linkedin.com/shareArticle?mini=true&title=Gitxray&summary=Alalala&url=https%3A//github.com/kulkansecurity/gitxray/" target='_blank'>Share on LinkedIn - Write about your Gitxray experience</a>
+
+And if you know of anyone looking for a Penetration Testing vendor, send them our way:
+
+* <a href="https://www.kulkan.com/" target='_blank'>Kulkan Security</a>
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -0,0 +1,29 @@
+site_name: Gitxray
+site_url: https://www.gitxray.com/
+repo_url: https://github.com/kulkansecurity/gitxray
+theme:
+  name: readthedocs
+  language: en
+  features:
+    - navigation.expand
+
+nav:
+  - 'index.md'
+  - 'installing.md'
+  - 'osint_pentest.md'
+  - 'forensics_spotting.md'
+  - 'securing_repos.md'
+  - 'ideas_pending.md'
+  - 'code_structure.md'
+  - 'show_love.md'
+
+markdown_extensions:
+  - toc:
+      permalink: 
+  - attr_list
+  - md_in_html
+  - pymdownx.arithmatex:
+      generic: true
+
+copyright: |
+  Made with &hearts; by <a href="https://www.kulkan.com" target="_blank" rel="noopener">Kulkan Security</a>
