Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ there](https://github.com/Tendrl/documentation/wiki/Tendrl-release-latest) to
have basic understanding of various machine roles in Tendrl cluster before
using tendrl-ansible.

tendrl-ansible also provides a playbook to cleanup the existing tendrl setup. It
performs the below steps
* Stops all the tendrl specific services on storage nodes
* Stops all the tendrl specific services on tendrl server node
* Cleans etcd, grafana and carbon data files


## How to get tendrl-ansible?

Expand Down Expand Up @@ -61,6 +67,9 @@ Ansible roles for Tendrl:
(where Tendrl api, web and etcd are running)
* `tendrl-ansible.tendrl-storage-node`: installation of **Tendrl Storage Node**
machines (Gluster servers, which you would like to monitor by Tendrl)
* `tendrl-ansible.tendrl-server-cleanup`: cleanup tendrl setup from server node
* `tendrl-ansible.tendrl-storage-node-cleanup`: cleanup tendrl setup from
storage nodes

Roles installing yum repositories of Tendrl dependencies:

Expand All @@ -79,6 +88,8 @@ Playbook files:
(see comments inside the playbook file for references)
* `site.yml`: main playbook of tendrl-ansible, which one will use to install
Tendrl
* `cleanup.yml`: playbook for cleaning up the tendrl setup from storage nodes
and tendrl server node


## Where are the roles and playbooks if I use rpm package?
Expand Down Expand Up @@ -317,6 +328,63 @@ tendrl-ansible:
on [TEN-257](https://tendrl.atlassian.net/browse/TEN-257)).


## How do I cleanup Tendrl setup using tendrl-ansible?

1) If you use tendrl-ansible from rpm package, copy `cleanup.yml` playbook into
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main information we are missing here is what does this cleanup mean and why would one want to do one. This helps to convey context of this feature.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also it seems that the cleanup deletes all tendrl internal data (etcd, graphite, ...). It seems to me that this needs to be highlighted here again.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack

working directory (where you already store the inventory file):

```
$ cp /usr/share/doc/tendrl-ansible-VERSION/cleanup.yml .
```

2) Check that ssh can connect to all machines from the inventory file without
asking for password or validation of public key by running:

```
$ ansible -i inventory_file -m ping all
```

You should see ansible to show `"pong"` message for all machines.
In case of any problems, you need to fix it before going on. If you are not
sure what's wrong, consult documentation of ansible and/or ssh.

The following example shows how to use [ansible become
feature](https://docs.ansible.com/ansible/latest/become.html) **when direct
ssh login of root user is not allowed** and you are connecting via non-root
`cloud-user` account, which can leverage `sudo` to run any command as root
without any password:

```
$ ansible --become -u cloud-user -i inventory_file -m ping all
```

If this is your case, you may consider converting command line arguments
related to *Ansbile become feature* into [behavioral inventory
parameters](https://docs.ansible.com/ansible/latest/intro_inventory.html#list-of-behavioral-inventory-parameters)
and adding them into the inventory file. This way, you don't need to
specify these arguments again for every ansible command. Example of this
update which matches previous command line example follows (it should be
appended to the `[all:vars]` section):

```
ansible_become=yes
ansible_user=cloud-user
```

After this edit, you can re run the ping example without become command
line arguments:

```
$ ansible -i inventory_file -m ping all
```

3) Then we are ready to run ansible to cleanup Tendrl setup:

```
$ ansible-playbook -i inventory_file cleanup.yml
```
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I would not repeat all the details already mentioned in the installation section to avoid duplication a bit, and make this little more succinct (after all, one needs to install first to do the cleanup). We could point out that the same ideas as for installation apply.



## How do I expand cluster with tendrl-ansible?

See [Tendrl wiki](https://github.com/Tendrl/documentation/wiki) for full
Expand Down
29 changes: 29 additions & 0 deletions cleanup.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
# This Ansible playbook is to cleanup Tendrl setup
# It just stops all the running services related to Tendrl
# on storage nodes as well as server. Also it cleans the
# existing etcd, grafana and carbon data (if any) for existing
# setup

- hosts: gluster_servers
user: root

roles:
- tendrl-storage-node-cleanup

- hosts: tendrl_server
user: root
vars:
# etcd db file to be deleted
etcd_db_path: "/var/lib/etcd/default.etcd"

# grafana db file to be deleted
grafana_db_file: "/var/lib/grafana/grafana.db"

# carbon data to be deleted
carbon_data_path: "/var/lib/carbon/whisper/tendrl/"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather moved these default values into roles/tendrl-ansible.tendrl-server-cleanup/defaults/main.yml.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack


roles:
- tendrl-server-cleanup


42 changes: 42 additions & 0 deletions roles/tendrl-ansible.tendrl-server-cleanup/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
# Task file for storage nodes

- name: Stop tendrl services
service:
name: tendrl-node-agent
state: stopped
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we are not stopping tendrl-api?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. Will add tendrl-api as well.


- name: Stop etcd service
service:
name: etcd
state: stopped

- name: Stop grafana-server service
service:
name: grafana-server
state: stopped

- name: Stop carbon-cache service
service:
name: carbon-cache
state: stopped

- name: Stop httpd service
service:
name: httpd
state: stopped

- name: Cleanup etcd data
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't it destructive to remove the ETCD? User may have different namespace running for different things can we just delete the data?

Doing it for the dev purpose makes sense to me but doing this same on production server of a client, I am a bit skeptical about it.

thoughts? @shtripat @nthomas-redhat @anmolsachan @GowthamShanmugam

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we are sure then the customer doesn't have anything else running on the tendrl server machine then we can take this path.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We save tendrl data in default profile. That itself is not very correct I feel.
We can have a disclaimer in document regarding the same, that it will flush all the etcd data.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct I agree shubhendu. IF some other data in default profile that will also get flushed.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There could be another option to delete all tendrl created dirs under etcd like /clusters, /nodes and tags etc. After this keep etcd running as usual and tendrl-ansible shouldnt cause issues. We will need to test the scenario though.

file:
path: "{{etcd_db_path}}/"
state: absent

- name: Cleanup grafana data
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here as well, I feel we are using default location to save tendrl data.

Copy link
Copy Markdown
Member

@cloudbehl cloudbehl Nov 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if someone is having other dashboards on grafana which are not related to tendrl. By cleaning grafana I think those will be lost too and then there will be no recovery for it.

I think grafanadb file will be common for all

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here as well we can try to remove only tendrl specific dashboards and keep grafana-server running. Need to verify though.

file:
path: "{{grafana_db_file}}"
state: absent

- name: Cleanup carbon data
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case of carbon we will delete only tendrl specific directory and nothing else.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. I missed that.

file:
path: "{{carbon_data_path}}/"
state: absent
18 changes: 18 additions & 0 deletions roles/tendrl-ansible.tendrl-storage-node-cleanup/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@

---
# Task file for storage nodes

- name: Stop tendrl services
service:
name: tendrl-gluster-integration
state: stopped

- name: Stop tendrl services
service:
name: tendrl-node-agent
state: stopped

- name: Stop collectd
service:
name: collectd
state: stopped