-
Notifications
You must be signed in to change notification settings - Fork 15
Added cleanup meachnism for tendrl setup #129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,6 +15,12 @@ there](https://github.com/Tendrl/documentation/wiki/Tendrl-release-latest) to | |
| have basic understanding of various machine roles in Tendrl cluster before | ||
| using tendrl-ansible. | ||
|
|
||
| tendrl-ansible also provides a playbook to cleanup the existing tendrl setup. It | ||
| performs the below steps | ||
| * Stops all the tendrl specific services on storage nodes | ||
| * Stops all the tendrl specific services on tendrl server node | ||
| * Cleans etcd, grafana and carbon data files | ||
|
|
||
|
|
||
| ## How to get tendrl-ansible? | ||
|
|
||
|
|
@@ -61,6 +67,9 @@ Ansible roles for Tendrl: | |
| (where Tendrl api, web and etcd are running) | ||
| * `tendrl-ansible.tendrl-storage-node`: installation of **Tendrl Storage Node** | ||
| machines (Gluster servers, which you would like to monitor by Tendrl) | ||
| * `tendrl-ansible.tendrl-server-cleanup`: cleanup tendrl setup from server node | ||
| * `tendrl-ansible.tendrl-storage-node-cleanup`: cleanup tendrl setup from | ||
| storage nodes | ||
|
|
||
| Roles installing yum repositories of Tendrl dependencies: | ||
|
|
||
|
|
@@ -79,6 +88,8 @@ Playbook files: | |
| (see comments inside the playbook file for references) | ||
| * `site.yml`: main playbook of tendrl-ansible, which one will use to install | ||
| Tendrl | ||
| * `cleanup.yml`: playbook for cleaning up the tendrl setup from storage nodes | ||
| and tendrl server node | ||
|
|
||
|
|
||
| ## Where are the roles and playbooks if I use rpm package? | ||
|
|
@@ -317,6 +328,63 @@ tendrl-ansible: | |
| on [TEN-257](https://tendrl.atlassian.net/browse/TEN-257)). | ||
|
|
||
|
|
||
| ## How do I cleanup Tendrl setup using tendrl-ansible? | ||
|
|
||
| 1) If you use tendrl-ansible from rpm package, copy `cleanup.yml` playbook into | ||
| working directory (where you already store the inventory file): | ||
|
|
||
| ``` | ||
| $ cp /usr/share/doc/tendrl-ansible-VERSION/cleanup.yml . | ||
| ``` | ||
|
|
||
| 2) Check that ssh can connect to all machines from the inventory file without | ||
| asking for password or validation of public key by running: | ||
|
|
||
| ``` | ||
| $ ansible -i inventory_file -m ping all | ||
| ``` | ||
|
|
||
| You should see ansible to show `"pong"` message for all machines. | ||
| In case of any problems, you need to fix it before going on. If you are not | ||
| sure what's wrong, consult documentation of ansible and/or ssh. | ||
|
|
||
| The following example shows how to use [ansible become | ||
| feature](https://docs.ansible.com/ansible/latest/become.html) **when direct | ||
| ssh login of root user is not allowed** and you are connecting via non-root | ||
| `cloud-user` account, which can leverage `sudo` to run any command as root | ||
| without any password: | ||
|
|
||
| ``` | ||
| $ ansible --become -u cloud-user -i inventory_file -m ping all | ||
| ``` | ||
|
|
||
| If this is your case, you may consider converting command line arguments | ||
| related to *Ansbile become feature* into [behavioral inventory | ||
| parameters](https://docs.ansible.com/ansible/latest/intro_inventory.html#list-of-behavioral-inventory-parameters) | ||
| and adding them into the inventory file. This way, you don't need to | ||
| specify these arguments again for every ansible command. Example of this | ||
| update which matches previous command line example follows (it should be | ||
| appended to the `[all:vars]` section): | ||
|
|
||
| ``` | ||
| ansible_become=yes | ||
| ansible_user=cloud-user | ||
| ``` | ||
|
|
||
| After this edit, you can re run the ping example without become command | ||
| line arguments: | ||
|
|
||
| ``` | ||
| $ ansible -i inventory_file -m ping all | ||
| ``` | ||
|
|
||
| 3) Then we are ready to run ansible to cleanup Tendrl setup: | ||
|
|
||
| ``` | ||
| $ ansible-playbook -i inventory_file cleanup.yml | ||
| ``` | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe I would not repeat all the details already mentioned in the installation section to avoid duplication a bit, and make this little more succinct (after all, one needs to install first to do the cleanup). We could point out that the same ideas as for installation apply. |
||
|
|
||
|
|
||
| ## How do I expand cluster with tendrl-ansible? | ||
|
|
||
| See [Tendrl wiki](https://github.com/Tendrl/documentation/wiki) for full | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| --- | ||
| # This Ansible playbook is to cleanup Tendrl setup | ||
| # It just stops all the running services related to Tendrl | ||
| # on storage nodes as well as server. Also it cleans the | ||
| # existing etcd, grafana and carbon data (if any) for existing | ||
| # setup | ||
|
|
||
| - hosts: gluster_servers | ||
| user: root | ||
|
|
||
| roles: | ||
| - tendrl-storage-node-cleanup | ||
|
|
||
| - hosts: tendrl_server | ||
| user: root | ||
| vars: | ||
| # etcd db file to be deleted | ||
| etcd_db_path: "/var/lib/etcd/default.etcd" | ||
|
|
||
| # grafana db file to be deleted | ||
| grafana_db_file: "/var/lib/grafana/grafana.db" | ||
|
|
||
| # carbon data to be deleted | ||
| carbon_data_path: "/var/lib/carbon/whisper/tendrl/" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would rather moved these default values into
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ack |
||
|
|
||
| roles: | ||
| - tendrl-server-cleanup | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| --- | ||
| # Task file for storage nodes | ||
|
|
||
| - name: Stop tendrl services | ||
| service: | ||
| name: tendrl-node-agent | ||
| state: stopped | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why we are not stopping tendrl-api?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ack. Will add tendrl-api as well. |
||
|
|
||
| - name: Stop etcd service | ||
| service: | ||
| name: etcd | ||
| state: stopped | ||
|
|
||
| - name: Stop grafana-server service | ||
| service: | ||
| name: grafana-server | ||
| state: stopped | ||
|
|
||
| - name: Stop carbon-cache service | ||
| service: | ||
| name: carbon-cache | ||
| state: stopped | ||
|
|
||
| - name: Stop httpd service | ||
| service: | ||
| name: httpd | ||
| state: stopped | ||
|
|
||
| - name: Cleanup etcd data | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. isn't it destructive to remove the ETCD? User may have different namespace running for different things can we just delete the data? Doing it for the dev purpose makes sense to me but doing this same on production server of a client, I am a bit skeptical about it. thoughts? @shtripat @nthomas-redhat @anmolsachan @GowthamShanmugam
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. if we are sure then the customer doesn't have anything else running on the tendrl server machine then we can take this path.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We save tendrl data in default profile. That itself is not very correct I feel.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct I agree shubhendu. IF some other data in default profile that will also get flushed.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There could be another option to delete all tendrl created dirs under etcd like |
||
| file: | ||
| path: "{{etcd_db_path}}/" | ||
| state: absent | ||
|
|
||
| - name: Cleanup grafana data | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same as above
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Here as well, I feel we are using default location to save tendrl data.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what if someone is having other dashboards on grafana which are not related to tendrl. By cleaning grafana I think those will be lost too and then there will be no recovery for it. I think grafanadb file will be common for all
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Here as well we can try to remove only tendrl specific dashboards and keep grafana-server running. Need to verify though. |
||
| file: | ||
| path: "{{grafana_db_file}}" | ||
| state: absent | ||
|
|
||
| - name: Cleanup carbon data | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same as above
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In case of carbon we will delete only tendrl specific directory and nothing else.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct. I missed that. |
||
| file: | ||
| path: "{{carbon_data_path}}/" | ||
| state: absent | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
|
|
||
| --- | ||
| # Task file for storage nodes | ||
|
|
||
| - name: Stop tendrl services | ||
| service: | ||
| name: tendrl-gluster-integration | ||
| state: stopped | ||
|
|
||
| - name: Stop tendrl services | ||
| service: | ||
| name: tendrl-node-agent | ||
| state: stopped | ||
|
|
||
| - name: Stop collectd | ||
| service: | ||
| name: collectd | ||
| state: stopped |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Main information we are missing here is what does this cleanup mean and why would one want to do one. This helps to convey context of this feature.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also it seems that the cleanup deletes all tendrl internal data (etcd, graphite, ...). It seems to me that this needs to be highlighted here again.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ack