Skip to content
This repository was archived by the owner on Nov 3, 2021. It is now read-only.

Commit 33a770b

Browse files
pwnbuspwnbus
authored
Merge pull request #1214 from gene1wood/change-cicd-sequence
Change CI/CD sequence for v1.38.5
2 parents d32ed51 + 8c4ad42 commit 33a770b

6 files changed

Lines changed: 109 additions & 119 deletions

File tree

‎CHANGELOG‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
55

66
## [Unreleased]
77

8+
## [v1.38.5] - 2019-04-09
9+
### Added
10+
- Support for CSS themes
11+
12+
### Changed
13+
- The CI/CD order to now build docker images in CodeBuild, upload them
14+
to DockerHub and then pull them down in the packer instance. Updated docs.
15+
- Assert TravisCI Python version in advance of change of Travis default to 3.6
16+
17+
### Fixed
18+
- Dashboard error on docker spinup
19+
20+
821
## [v1.38.4] - 2019-04-08
922
### Fixed
1023
- Docker image tagging for git version tag builds
@@ -64,7 +77,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
6477
- Added checks on sending SQS messages to only accept intra-account messages
6578
- Improved docker performance and disk space requirements
6679

67-
[Unreleased]: https://github.com/mozilla/MozDef/compare/v1.38.4...HEAD
80+
[Unreleased]: https://github.com/mozilla/MozDef/compare/v1.38.5...HEAD
81+
[v1.38.5]: https://github.com/mozilla/MozDef/compare/v1.38.4...v1.38.5
6882
[v1.38.4]: https://github.com/mozilla/MozDef/compare/v1.38.3...v1.38.4
6983
[v1.38.3]: https://github.com/mozilla/MozDef/compare/v1.38.2...v1.38.3
7084
[v1.38.2]: https://github.com/mozilla/MozDef/compare/v1.38.1...v1.38.2

‎Makefile‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -143,7 +143,3 @@ rebuild: clean build-from-cwd
143143
.PHONY: new-alert
144144
new-alert: ## Create an example alert and working alert unit test
145145
python tests/alert_templater.py
146-
147-
.PHONY: set-version-and-fetch-docker-container
148-
set-version-and-fetch-docker-container: build-from-cwd tag-images # Lock the release of MozDef by pulling the docker containers on AMI build and caching replace all instances of latest in the compose override with the BRANCH
149-
sed -i s/latest/$(BRANCH)/g docker/compose/docker-compose-cloudy-mozdef.yml

‎cloudy_mozdef/ci/deploy‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,20 +15,20 @@ echo "It's dangerous to go alone. Take one of these: <%%%%|==========>"
1515
# Then again we probably do not need to run the test suite here because it has been run three times to get the code here.
1616
# echo "Tests complete.
1717

18-
echo "Processing webhook event for ${CODEBUILD_WEBHOOK_TRIGGER}."
18+
echo "Processing webhook event for '${CODEBUILD_WEBHOOK_TRIGGER}'."
1919

2020
if [[ "branch/master" == "$CODEBUILD_WEBHOOK_TRIGGER" \
2121
|| "$CODEBUILD_WEBHOOK_TRIGGER" =~ ^tag\/v[0-9]+\.[0-9]+\.[0-9]+(\-(prod|pre|testing))?$ ]]; then
2222
echo "Building a release"
2323
echo "C|_| This may take a bit. Might as well grab a coffee."
24+
BRANCH="`echo $CODEBUILD_WEBHOOK_TRIGGER | cut -d '/' -f2`"
2425
make build-from-cwd
26+
make hub-login
27+
make BRANCH=${BRANCH} docker-push-tagged
2528
cd cloudy_mozdef
26-
BRANCH="`echo $CODEBUILD_WEBHOOK_TRIGGER | cut -d '/' -f2`"
2729
make BRANCH=${BRANCH} packer-build-github
2830
make BRANCH=${BRANCH} publish-versioned-templates
2931
cd ..
30-
make hub-login
31-
make BRANCH=${BRANCH} docker-push-tagged
3232
fi
3333

3434
echo "End build of the MozDef codebase."

‎cloudy_mozdef/packer/packer.json‎

Lines changed: 57 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -1,72 +1,69 @@
11
{
22
"variables": {
3-
"aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
4-
"aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
5-
"aws_security_token": "{{env `AWS_SESSION_TOKEN`}}"
3+
"aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
4+
"aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
5+
"aws_security_token": "{{env `AWS_SESSION_TOKEN`}}"
66
},
7-
"builders": [{
8-
"type": "amazon-ebs",
9-
"region": "us-west-2",
10-
"access_key": "{{user `aws_access_key`}}",
11-
"secret_key": "{{user `aws_secret_key`}}",
12-
"token": "{{user `aws_security_token`}}",
13-
"source_ami": "ami-0d1000aff9a9bad89",
14-
"instance_type": "t2.large",
15-
"ssh_pty" : "true",
16-
"ssh_username": "ec2-user",
17-
"ami_name": "mozdef_{{timestamp}}",
18-
"launch_block_device_mappings": [
19-
{
20-
"delete_on_termination": true,
21-
"device_name": "/dev/xvda",
22-
"volume_size": 14
7+
"builders": [
8+
{
9+
"type": "amazon-ebs",
10+
"region": "us-west-2",
11+
"access_key": "{{user `aws_access_key`}}",
12+
"secret_key": "{{user `aws_secret_key`}}",
13+
"token": "{{user `aws_security_token`}}",
14+
"source_ami": "ami-0d1000aff9a9bad89",
15+
"instance_type": "t2.large",
16+
"ssh_pty": "true",
17+
"ssh_username": "ec2-user",
18+
"ami_name": "mozdef_{{timestamp}}",
19+
"launch_block_device_mappings": [
20+
{
21+
"delete_on_termination": true,
22+
"device_name": "/dev/xvda",
23+
"volume_size": 14
24+
}
25+
],
26+
"ami_description": "An automated build of MozDef triggered via the makefile.",
27+
"ami_groups": [
28+
"all"
29+
],
30+
"run_tags": {
31+
"app": "packer-builder-mozdef"
32+
},
33+
"run_volume_tags": {
34+
"app": "packer-builder-mozdef"
35+
},
36+
"snapshot_tags": {
37+
"app": "packer-builder-mozdef"
38+
},
39+
"tags": {
40+
"github:Branch": "{{ user `github_branch`}}",
41+
"buildTimestamp": "{{timestamp}}",
42+
"app": "mozdef"
2343
}
24-
],
25-
"ami_description": "An automated build of MozDef triggered via the makefile.",
26-
"ami_groups": [
27-
"all"
28-
],
29-
"run_tags": {
30-
"app": "packer-builder-mozdef"
31-
},
32-
"run_volume_tags": {
33-
"app": "packer-builder-mozdef"
34-
},
35-
"snapshot_tags": {
36-
"app": "packer-builder-mozdef"
37-
},
38-
"tags": {
39-
"github:Branch": "{{ user `github_branch`}}",
40-
"buildTimestamp": "{{timestamp}}",
41-
"app": "mozdef"
4244
}
43-
}],
44-
"provisioners": [
45-
{ "type": "shell",
45+
],
46+
"provisioners": [
47+
{
48+
"type": "shell",
4649
"inline": [
50+
"set -e",
4751
"sudo yum update -y",
4852
"sudo yum makecache fast",
49-
"sudo yum install -y glibc-devel gcc libstdc++ libffi-devel zlib-devel make ",
50-
"sudo yum install -y mysql-devel python python-devel python-pip",
51-
"sudo yum install -y git",
52-
"sudo yum install -y docker",
53-
"sudo yum install -y python3",
54-
"sudo pip install virtualenv ",
55-
"sudo pip install docker-compose",
53+
"sudo yum install -y glibc-devel gcc libstdc++ libffi-devel zlib-devel make mysql-devel python python-devel python-pip git docker python3",
54+
"sudo pip install virtualenv docker-compose",
5655
"sudo systemctl enable docker",
5756
"sudo systemctl start docker",
58-
"sudo mkdir -p /opt/mozdef/",
57+
"sudo mkdir --verbose --parents /opt/mozdef/",
5958
"sudo git clone https://github.com/mozilla/MozDef /opt/mozdef",
60-
"cd /opt/mozdef && sudo git checkout {{ user `github_branch`}}",
61-
"cd /opt/mozdef && sudo git rev-parse HEAD",
62-
"cd /opt/mozdef && sudo touch docker/compose/cloudy_mozdef.env docker/compose/rabbitmq.env docker/compose/cloudy_mozdef_mq_cloudtrail.env docker/compose/cloudy_mozdef_mq_sns_sqs.env docker/compose/cloudy_mozdef_kibana.env",
63-
"cd /opt/mozdef && sudo make BRANCH={{ user `github_branch`}} set-version-and-fetch-docker-container",
64-
"cd /opt/mozdef && sudo docker-compose -f docker/compose/docker-compose-cloudy-mozdef.yml -p mozdef pull",
65-
"rm -rf /home/ec2-user/.ssh/authorized_keys",
66-
"rm -rf /home/ec2-user/.ssh/known_hosts",
67-
"sudo rm -rf /tmp/*",
68-
"sudo rm -rf /home/ec2-user/.bash_history",
69-
"sudo rm -rf /root/.ssh"
70-
]}
71-
]
59+
"cd /opt/mozdef",
60+
"sudo git checkout {{ user `github_branch`}}",
61+
"sudo git rev-parse HEAD",
62+
"sudo touch docker/compose/cloudy_mozdef.env docker/compose/rabbitmq.env docker/compose/cloudy_mozdef_mq_cloudtrail.env docker/compose/cloudy_mozdef_mq_sns_sqs.env docker/compose/cloudy_mozdef_kibana.env",
63+
"sudo sed --in-place s/latest/{{ user `github_branch`}}/g docker/compose/docker-compose-cloudy-mozdef.yml",
64+
"sudo docker-compose --file docker/compose/docker-compose-cloudy-mozdef.yml --project-name mozdef pull",
65+
"sudo rm --recursive --force --verbose /tmp/* /home/ec2-user/.bash_history /root/.ssh /home/ec2-user/.ssh/known_hosts /home/ec2-user/.ssh/authorized_keys"
66+
]
67+
}
68+
]
7269
}

‎docker/compose/mozdef_meteor/Dockerfile‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,8 @@ RUN mkdir -p /opt/mozdef/envs/meteor/mozdef
4646
RUN if [ "${METEOR_BUILD}" = "YES" ]; then \
4747
cd /opt/mozdef/envs/mozdef/meteor && \
4848
meteor npm install && \
49-
meteor build --server localhost:3002 --directory /opt/mozdef/envs/meteor/mozdef && \
49+
echo "Starting meteor build" && \
50+
time meteor build --server localhost:3002 --directory /opt/mozdef/envs/meteor/mozdef && \
5051
cp -r /opt/mozdef/envs/mozdef/meteor/node_modules /opt/mozdef/envs/meteor/mozdef/node_modules &&\
5152
cd /opt/mozdef/envs/meteor/mozdef/bundle/programs/server && \
5253
npm install ;\

‎docs/source/cicd.rst‎

Lines changed: 31 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -78,9 +78,9 @@ _________________________________________
7878
The Build Sequence
7979
__________________
8080

81-
* A branch is merged into `master` in the GitHub repo
82-
* GitHub emits a webhook event to AWS CodeBuild indicating that a commit was
83-
pushed to `master`
81+
* A branch is merged into `master` in the GitHub repo or a version git tag is
82+
applied to a commit
83+
* GitHub emits a webhook event to AWS CodeBuild indicating this
8484
* AWS CodeBuild reads the
8585
`buildspec.yml <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/buildspec.yml>`_
8686
file to know what to do
@@ -98,8 +98,28 @@ __________________
9898
target of the `Makefile` which calls `docker-compose build` on the
9999
`docker-compose.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose.yml>`_
100100
file, building the docker images in the AWS CodeBuild environment. These are
101-
built so they can be uploaded later to DockerHub for use by developers and
102-
the community.
101+
built both so they can be consumed later in the build by packer and also
102+
for use by developers and the community.
103+
* `deploy` then calls the
104+
`docker-push-tagged <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L113>`_
105+
make target which calls
106+
107+
* the tag-images_
108+
make target which calls the
109+
`cloudy_mozdef/ci/docker_tag_or_push tag <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_
110+
script which applies a docker image tag to the local image that was just
111+
built by AWS CodeBuild.
112+
* the
113+
`hub-tagged <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L116-L117>`_
114+
make target which calls the
115+
`cloudy_mozdef/ci/docker_tag_or_push push <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_
116+
script which
117+
118+
* Uploads the local image that was just built by AWS CodeBuild to DockerHub.
119+
If the branch being built is `master` then the image is uploaded both with
120+
a tag of `master` as well as with a tag of `latest`
121+
* If the branch being built is from a version tag (e.g. `v1.2.3`) then the
122+
image is uploaded with only that version tag applied
103123
* The `deploy` script next calls the
104124
`packer-build-github <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/Makefile#L34-L36>`_
105125
make target in the
@@ -124,31 +144,14 @@ __________________
124144
* Within this ec2 instance, packer `clones the MozDef GitHub repo and checks
125145
out the branch that triggered this build
126146
<https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/cloudy_mozdef/packer/packer.json#L59-L60>`_
127-
* packer calls the `set-version-and-fetch-docker-container <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L148-L149>`_
128-
target of the `Makefile` which
129-
130-
* Calls the
131-
`build-from-cwd <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L78-L79>`_
132-
target of the `Makefile` which calls `docker-compose build` on the
133-
`docker-compose.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose.yml>`_
134-
file, building the docker images in the packer ec2 environment
135-
* Calls the tag-images_
136-
make target which calls the `cloudy_mozdef/ci/docker_tag_or_push tag <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_
137-
script.
138-
139-
* This applies a docker image tag to the local image that was just built
140-
by packer. This tag will be referenced by the
141-
`docker-compose-cloudy-mozdef.yml` file when the MozDef ec2 instance is
142-
launched from the AMI.
143-
144-
* Replaces all instances of the word `latest` in the
145-
`docker-compose-cloudy-mozdef.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-cloudy-mozdef.yml>`_
146-
file with either the branch `master` or the version tag (e.g. `v1.2.3`)
147-
147+
* packer replaces all instances of the word `latest` in the
148+
`docker-compose-cloudy-mozdef.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-cloudy-mozdef.yml>`_
149+
file with either the branch `master` or the version tag (e.g. `v1.2.3`)
148150
* packer runs `docker-compose pull` on the
149151
`docker-compose-cloudy-mozdef.yml <https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-cloudy-mozdef.yml>`_
150-
file to pull down any remaining non MozDef container images that weren't
151-
just built in preceding packer steps
152+
file to pull down both the docker images that were just built by AWS
153+
CodeBuild and uploaded to Dockerhub as well as other non MozDef docker
154+
images
152155

153156
* After packer completes executing the steps laid out in `packer.json` inside
154157
the ec2 instance, it generates an AMI from that instance and continues with
@@ -168,26 +171,5 @@ __________________
168171
* uploads the CloudFormation templates to S3 in a directory either called
169172
`master` or the tag version that was built (e.g. `v1.2.3`)
170173

171-
* `deploy` then calls the
172-
`docker-push-tagged <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L113>`_
173-
make target which calls
174-
175-
* the tag-images_
176-
make target which calls the
177-
`cloudy_mozdef/ci/docker_tag_or_push tag <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_
178-
script which applies a docker image tag to the local image that was just
179-
built by AWS CodeBuild.
180-
* the
181-
`hub-tagged <https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L116-L117>`_
182-
make target which calls the
183-
`cloudy_mozdef/ci/docker_tag_or_push push <https://github.com/mozilla/MozDef/blob/master/cloudy_mozdef/ci/docker_tag_or_push>`_
184-
script which
185-
186-
* Uploads the local image that was just built by AWS CodeBuild to DockerHub.
187-
If the branch being built is `master` then the image is uploaded both with
188-
a tag of `master` as well as with a tag of `latest`
189-
* If the branch being built is from a version tag (e.g. `v1.2.3`) then the
190-
image is uploaded with only that version tag applied
191-
192174
.. _docker/compose/docker-compose-tests.yml: https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-tests.yml
193175
.. _tag-images: https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L109-L110

0 commit comments

Comments
 (0)