working 3 member sensu cluster install playbook and cleanup playbook #153
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Added a loadblancer vm into the scenario. Provisioned via simple inline script in the Vagrant file. I didn't want to overly complicate the Sensu focused playbook with the loadbalancer provisioning. |
tadeboro
force-pushed
the
scenario-docs
branch
from
1203dc4 to
ad59cb1
Compare
tadeboro
force-pushed
the
scenario-docs
branch
from
ad59cb1 to
fe4b4b4
Compare
added 10 commits
January 31, 2020 13:56
Ubuntu Cosmic is not maintained anymore and since the repositories for this release were removed, it is almost impossible to do anything with it. This is why we decided to remove it from our test suite.
All our info modules falsely advertised that they always contain an object field while in reality, they only contain it if the API call to the backend did not fail in one way or another.
Right now, parent's run method only receives a tmp parameter, which is not what we intended. Passing variables as a keyword argument sets things straight.
Metadata now contains all supported platforms that we actually test against using our integration tests.
Galaxy performs some tests when importing roles. One of them is checking for a validity of the supported platforms. And because no existing tool does this for us, we created a simple python script that queries the galaxy web API and ensures that we have a valid set of platforms listed in role metadata.
When asked about the asset information, the Sensu backend returns builds: null field since 5.16. This confused our comparator because we always expected the builds parameter to be a list if present.
One of the image updates broke our CI. And because the error message we get on failure is almost completely useless, it will take some time to get to the bottom of this issue. And while we cannot completely bypass this problem (we must update our test images from time to time), we can reduce its impact on the development team by pinning the images down to the SHA256 hash. This way, we can perform the image updates ,under our terms and conditions, but more importantly, we remove one moving part from the CI pipeline.
We removed the tests for the unsupported versions of the Sensu Go. The only exception to this rule is the login functionality. We still make sure that the login failure is still properly reported on Sensu Go 5.15.0 and older.
added 5 commits
June 22, 2020 21:57
Splitting the role tasks into separate task files that perform a certain part of the installation makes it much easier to reuse the needed parts in the playbooks.
Up until now, we kept the docker definitions in a separate repository, but that made it quite hard to keep the things in sync. Changes in this commit move the docker image definitions into the collection. In order to prevent any name clashes, we also prefix all image names with the `sensu-go-tests-`, explicitly stating what those images are used for.
In order to keep the result conversion as dry as possible, we created a new helper method that provides a compatibility layer between the v1 and v2 versions of Sensu Go API.
Datastore API changed a bit somewhere between 5.15.0 and 5.18.0. In order to keep up with the changes, we updated the module code in order to handle the change. Note that we did try to make the change backwards-compatible, but since the versions of the Sensu Go that contained an old API are not supported anymore by the upstream, we do not test for this compatibility.
added 19 commits
July 1, 2020 10:14
Because the Sensu Go backend does not keep its API backwards-compatible between the minor version bumps, we need a way to get the backend version number into modules so they can adapt to API changes.
Sensu Go 5.21.0 broke the backwards-compatibility of the user API. Changes in this commit gloss over those changes and make sure playbook authors can still use the same API as before. Note that because the backend now accepts an encrypted password in the payload, the collection gained another dependency that needs to be installed on the node that module executes on: bcrypt.
Because the user module requires bcrypt dependency, we also updated the docker container definition that now contains the required python packages.
In order to inform users about the system dependencies modules might have, we added a requirements section to the documentation template. Note that this section is only rendered if the module docs contain requirements specification.
Payload for creating a check is large and contains quite a few lists that serve as a set of things. All this makes comparing the payloads a bit tougher that for the rest of the resources. Another source of problems is the backend's tendency to treat null and empty list as equivalent, adding yet another place where equality checks can go wrong. Changes in this commit try to fix all of the idempotency issues we could think of, but there is a slight possibility that we still missed something. But things should be in a much better shape now.
Testing our modules using python 3.8 revealed two minor issues in our unit tests. This commit fixes those issues and prepares our collection for python 3.8 testing.
Since python 3.8 has been marked stable for quite some time now, we added it to our test suite to make sure our code works well on recent distributions.
Up until now, the HTTP client that we use to interact with the Sensu Go backend always verified the backend certificate using the same CAs as the python itself. In this commit, we added two new parameters to the authentication part of the module. The verify parameter allows users to disable the verification altogether and should only be used in development environments. In production, users should either use certificates that are trusted by default (via systems root CAs) or use a ca_path parameter that allows users to specify what CA bundle should be used to validate the backend certificate. Both new parameters can also be set via the environment variables. We added this fallback option to keep the auth module parameter optional as it is now.
The documentation about this parameter was severely lacking because we deviate a bit from the usptream format. To rectify the situation, we added an example that demonstrates the remaping in practice. We also added link to the upstrea docs for this specific parameter.
The documentation fragment now clearly states that the name parameter serves as a unique identifier for each resource.
Our collection does not support Python 2.6 because that version is long obsolete and almost impossible to test against. Unfortunately for us, the Automation Hub import process still runs the compile tests using that version. To avoid getting errors during the import process, we added an ignore file that instructs ansible-test to skip the compilation test on certain files that use constructs introduces in Python 2.7.
When testing, we need to use the right version of pytest package
becase:
1. pytest >= 5 does not support python 2.7 anymore, and
2. pytest >= 6 has some problems with relative imports in
collections.
The apt_key module is not really properly equipped for the key downloads because it can only handle the happy path scenarios. In order to get full control over the download, we instead opted to use the uri module to perform the download. This will allow us to expand the download part in the future when we add support for packagercloud's read tokens.
Up until now, it was not possible to install Sensu Go from a private repositories on packagecloud. To overcome that deficiency, we added another variable to the install role that should contain read token. Note that we currently do not have any tests for this functionality because we do not have access to any of the private repositories. Hopefully, this will get fixed in the future.
There are two main reasons for this change: make it easier to debug the assert statements (because we have the result printed) and make it easier to find the failing task (because we have the line numbers in the output).
Sensu Go 6 prepopulates the backend with some additional resources such as role bindings that were breaking with our asserts. Apart from that, Sensu Go Ansible Collection works great with the latest Sensu Go release.
|
I'm going to move this off into a new sub-directory under examples instead of trying to have it be a documented scenario. We can perhaps point to it as and advanced reference example |
|
Closing this PR out, as its replaced by #213 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sensu backend cluster scenario.
I'll be continuing to add to this.
Here's what is working right now.
playbook.yml will spin up a working cluster
cleanup_playbook.yml will disable the cluster and empty out each member's etcd store making it possible to quickly re-provision a new cluster from scratch (skipping the package download step)