Skip to content
This repository was archived by the owner on Nov 3, 2021. It is now read-only.

Commit 9d3e8c5

Browse files
author
Tristan Weir
authored
Merge pull request #1210 from gene1wood/fix-docker-tag
Fix docker tagging and add CI/CD docs
2 parents cfeafb7 + 43951fa commit 9d3e8c5

4 files changed

Lines changed: 213 additions & 6 deletions

File tree

‎CHANGELOG‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
55

66
## [Unreleased]
77

8+
## [v1.38.4] - 2019-04-08
9+
### Fixed
10+
- Docker image tagging for git version tag builds
11+
- Correctly propagate the source ip address to the details.sourceipaddress in Duo logpull
12+
- Invalid literal in squidFixup.py destionationport field
13+
- Lowercase TAGS in squidFixup.py
14+
- Adding check for None type object in date fields to address GuardDuty null date
15+
16+
### Added
17+
- Documentation on the CI/CD process
18+
- A summary to squidFixup.py
19+
- Tags assertions to tests
20+
821
## [v1.38.3] - 2019-04-01
922
### Fixed
1023
- AWS CodeBuild tag semver regex
@@ -51,7 +64,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
5164
- Added checks on sending SQS messages to only accept intra-account messages
5265
- Improved docker performance and disk space requirements
5366

54-
[Unreleased]: https://github.com/mozilla/MozDef/compare/v1.38.3...HEAD
67+
[Unreleased]: https://github.com/mozilla/MozDef/compare/v1.38.4...HEAD
68+
[v1.38.4]: https://github.com/mozilla/MozDef/compare/v1.38.3...v1.38.4
5569
[v1.38.3]: https://github.com/mozilla/MozDef/compare/v1.38.2...v1.38.3
5670
[v1.38.2]: https://github.com/mozilla/MozDef/compare/v1.38.1...v1.38.2
5771
[v1.38.1]: https://github.com/mozilla/MozDef/compare/v1.38...v1.38.1

‎cloudy_mozdef/ci/docker_tag_or_push‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,13 @@ for name in mozdef_meteor mozdef_base mozdef_tester mozdef_mq_worker mozdef_kiba
88
mozdef_bootstrap mozdef_alerts mozdef_nginx mozdef_alertactions mozdef_rabbitmq \
99
mozdef_rest mozdef_base ; do
1010
if [ "${action}" == "tag" ]; then
11-
if [ "${branch}" == "master" ]; then
12-
docker tag mozdef/${name}:latest mozdef/${name}:${branch}
13-
else
14-
docker tag mozdef/${name}:${branch}
15-
fi
11+
echo "Tagging local image mozdef/${name}:latest as mozdef/${name}:${branch}"
12+
docker tag mozdef/${name}:latest mozdef/${name}:${branch}
1613
elif [ "${action}" == "push" ]; then
14+
echo "Pushing local image mozdef/${name}:${branch} to DockerHub"
1715
docker push mozdef/${name}:${branch}
1816
if [ "${branch}" == "master" ]; then
17+
echo "Pushing local image mozdef/${name}:latest to DockerHub"
1918
docker push mozdef/${name}:latest
2019
fi
2120
fi

‎docs/source/cicd.rst‎

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
Continuous Integration and Continuous Deployment
2+
================================================
3+
4+
Overview
5+
--------
6+
7+
Each git commit to the `master` branch in GitHub triggers both the TravisCI
8+
automated tests as well as the AWS CodeBuild building. Each git tag applied to a
9+
git commit triggers a CodeBuild build.
10+
11+
Travis CI
12+
---------
13+
14+
Travis CI runs tests on the MozDef code base with each commit to `master`. The
15+
results can be seen on the
16+
`Travis CI MozDef dashboard <https://travis-ci.org/mozilla/MozDef/>`_
17+
18+
The Test Sequence
19+
_________________
20+
21+
* Travis CI creates webhooks when first setup which allow commits to the MozDef
22+
GitHub repo to trigger Travis
23+
* When a commit is made to MozDef, Travis CI follows the instructions in the
24+
`.travis.yml <https://github.com/mozilla/MozDef/blob/master/.travis.yml>`_
25+
file
26+
* `.travis.yml` installs `docker-compose` in the `before_install` phase
27+
* in the `install` phase, Travis runs the
28+
`build-tests <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L88-L89>`_
29+
make target which calls `docker-compose build` on the
30+
`docker/compose/docker-compose-tests.yml`_ file which builds a few docker
31+
containers to use for testing
32+
* in the `script` phase, Travis runs the
33+
`tests <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L52>`_
34+
make target which
35+
36+
* calls the `build-tests` make target which again runs `docker-compose build`
37+
on the `docker/compose/docker-compose-tests.yml`_ file
38+
* calls the
39+
`run-tests <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L67-L69>`_
40+
make target which
41+
42+
* calls the
43+
`run-tests-resources <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L60-L61>`_
44+
make target which starts the docker
45+
containers listed in `docker/compose/docker-compose-tests.yml`_
46+
* runs `flake8` with the
47+
`.flake8 <https://github.com/mozilla/MozDef/blob/master/.flake8>`_
48+
config file to check code style
49+
* runs `py.test tests` which runs all the test cases
50+
51+
AWS CodeBuild
52+
-------------
53+
54+
Enabling GitHub AWS CodeBuild Integration
55+
_________________________________________
56+
57+
* Request that a github.com/mozilla GitHub Organization owner temporarily
58+
`approve / whitelist
59+
<https://help.github.com/en/articles/approving-oauth-apps-for-your-organization>`_
60+
the `AWS CodeBuild integration <https://bugzilla.mozilla.org/show_bug.cgi?id=1506740>`_
61+
in the github.com/mozilla GitHub Organization
62+
* Manually configure the GitHub integration in AWS CodeBuild which will create
63+
the GitHub webhooks needed using the dedicated, AWS account specific, GitHub
64+
service user. A service user is needed as AWS CodeBuild can only integrate
65+
with GitHub from one AWS account in one region with a single GitHub user.
66+
Technically we could use different users for each region in a single AWS
67+
account, but for simplicity we're limiting to only one GitHub user per AWS
68+
account (instead of one GitHub user per AWS account per region)
69+
70+
* For the `infosec-prod` AWS account use the `infosec-prod-371522382791-codebuild`
71+
GitHub user
72+
* For the `infosec-dev` AWS account use the `infosec-dev-656532927350-codebuild`
73+
GitHub user
74+
75+
* Request that a GitHub Organization owner, re-deny the integration for
76+
github.com/mozilla
77+
78+
The Build Sequence
79+
__________________
80+
81+
* A branch is merged into `master` in the GitHub repo
82+
* GitHub emits a webhook event to AWS CodeBuild indicating that a commit was
83+
pushed to `master`
84+
* AWS CodeBuild reads the
85+
`buildspec.yml <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/buildspec.yml>`_
86+
file to know what to do
87+
* The `install` phase of the `buildspec.yml` fetches
88+
`packer <https://www.packer.io/>`_ and unzips it
89+
90+
* `packer` is a tool that spawns an ec2 instance, provisions it, and renders
91+
an AWS Machine Image (AMI) from it.
92+
93+
* The `build` phase of the `buildspec.yml` runs the
94+
`cloudy_mozdef/ci/deploy <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/deploy>`_
95+
script in the AWS CodeBuild Ubuntu 14.04 environment
96+
* The `deploy` script calls the
97+
`build-from-cwd <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L78-L79>`_
98+
target of the `Makefile` which calls `docker-compose build` on the
99+
`docker-compose.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose.yml>`_
100+
file, building the docker images in the AWS CodeBuild environment. These are
101+
built so they can be uploaded later to DockerHub for use by developers and
102+
the community.
103+
* The `deploy` script next calls the
104+
`packer-build-github <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/Makefile#L34-L36>`_
105+
make target in the
106+
`cloudy_mozdef/Makefile <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/Makefile>`_
107+
which calls the
108+
`ci/pack_and_copy <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/pack_and_copy>`_
109+
script which does the following steps
110+
111+
* Calls packer which launches an ec2 instance, executing a bunch of steps and
112+
and producing an AMI
113+
* Shares the resulting AMI with the AWS Marketplace account
114+
* Copies the resulting AMI to a list of additional AWS regions
115+
* Copies the tags from the original AMI to these copied AMIs in other regions
116+
* Shares the AMIs in these other regions with the AWS Marketplace account
117+
* Creates a blob of YAML which contains the AMI IDs. This blob will be used
118+
in the CloudFormation templates
119+
120+
* When `ci/pack_and_copy` calls packer, packer launches an ec2 instance based on
121+
the configuration in
122+
`cloudy_mozdef/packer/packer.json <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/packer/packer.json>`_
123+
124+
* Within this ec2 instance, packer `clones the MozDef GitHub repo and checks
125+
out the branch that triggered this build
126+
<https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/packer/packer.json#L59-L60>`_
127+
* packer calls the `set-version-and-fetch-docker-container <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L148-L149>`_
128+
target of the `Makefile` which
129+
130+
* Calls the
131+
`build-from-cwd <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L78-L79>`_
132+
target of the `Makefile` which calls `docker-compose build` on the
133+
`docker-compose.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose.yml>`_
134+
file, building the docker images in the packer ec2 environment
135+
* Calls the tag-images_
136+
make target which calls the `cloudy_mozdef/ci/docker_tag_or_push tag <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_
137+
script.
138+
139+
* This applies a docker image tag to the local image that was just built
140+
by packer. This tag will be referenced by the
141+
`docker-compose-cloudy-mozdef.yml` file when the MozDef ec2 instance is
142+
launched from the AMI.
143+
144+
* Replaces all instances of the word `latest` in the
145+
`docker-compose-cloudy-mozdef.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-cloudy-mozdef.yml>`_
146+
file with either the branch `master` or the version tag (e.g. `v1.2.3`)
147+
148+
* packer runs `docker-compose pull` on the
149+
`docker-compose-cloudy-mozdef.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-cloudy-mozdef.yml>`_
150+
file to pull down any remaining non MozDef container images that weren't
151+
just built in preceding packer steps
152+
153+
* After packer completes executing the steps laid out in `packer.json` inside
154+
the ec2 instance, it generates an AMI from that instance and continues with
155+
the copying, tagging and sharing steps described above
156+
* Now back in the AWS CodeBuild environment, the `deploy` script continues by
157+
calling the
158+
`publish-versioned-templates <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/Makefile#L85-L87>`_
159+
make target which runs the
160+
`ci/publish_versioned_templates <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/publish_versioned_templates>`_
161+
script which
162+
163+
* injects the AMI map yaml blob produced
164+
earlier into the
165+
`mozdef-parent.yml <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/cloudformation/mozdef-parent.yml#L86-L87>`_
166+
CloudFormation template so that the template knows the AMI IDs of that
167+
specific branch of code.
168+
* uploads the CloudFormation templates to S3 in a directory either called
169+
`master` or the tag version that was built (e.g. `v1.2.3`)
170+
171+
* `deploy` then calls the
172+
`docker-push-tagged <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L113>`_
173+
make target which calls
174+
175+
* the tag-images_
176+
make target which calls the
177+
`cloudy_mozdef/ci/docker_tag_or_push tag <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_
178+
script which applies a docker image tag to the local image that was just
179+
built by AWS CodeBuild.
180+
* the
181+
`hub-tagged <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L116-L117>`_
182+
make target which calls the
183+
`cloudy_mozdef/ci/docker_tag_or_push push <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_
184+
script which
185+
186+
* Uploads the local image that was just built by AWS CodeBuild to DockerHub.
187+
If the branch being built is `master` then the image is uploaded both with
188+
a tag of `master` as well as with a tag of `latest`
189+
* If the branch being built is from a version tag (e.g. `v1.2.3`) then the
190+
image is uploaded with only that version tag applied
191+
192+
.. _docker/compose/docker-compose-tests.yml: https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-tests.yml
193+
.. _tag-images: https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L109-L110

‎docs/source/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ Table of Contents
2121
advanced_settings
2222
code
2323
benchmarking
24+
cicd
2425
contributors
2526

2627

0 commit comments

Comments
 (0)