Skip to content

Update upgrade-guide-from-monitoring-3.x-to-monitoring-4.y.rst #1773

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

Update upgrade-guide-from-monitoring-3.x-to-monitoring-4.y.rst #1773

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
e8f90c8
Update upgrade-guide-from-monitoring-3.x-to-monitoring-4.y.rst
ljgonzalez37 Jul 25, 2022
da6173e
Update docs/source/upgrade/upgrade-guide-from-monitoring-3.x-to-monit…
ljgonzalez37 Aug 4, 2022
31dd3f6
Update docs/source/upgrade/upgrade-guide-from-monitoring-3.x-to-monit…
ljgonzalez37 Aug 4, 2022
edc0375
Update docs/source/upgrade/upgrade-guide-from-monitoring-3.x-to-monit…
ljgonzalez37 Aug 4, 2022
1dfb43f
Update docs/source/upgrade/upgrade-guide-from-monitoring-3.x-to-monit…
ljgonzalez37 Aug 4, 2022
3957be2
Update docs/source/upgrade/upgrade-guide-from-monitoring-3.x-to-monit…
ljgonzalez37 Aug 4, 2022
1c29903
Update docs/source/upgrade/upgrade-guide-from-monitoring-3.x-to-monit…
ljgonzalez37 Aug 4, 2022
542896c
Update upgrade-guide-from-monitoring-3.x-to-monitoring-4.y.rst
ljgonzalez37 Aug 4, 2022
9dbf7bc
Update upgrade-guide-from-monitoring-3.x-to-monitoring-4.y.rst
ljgonzalez37 Aug 4, 2022
f7f352f
Update upgrade-guide-from-monitoring-3.x-to-monitoring-4.y.rst
ljgonzalez37 Aug 4, 2022
9586f0c
Update upgrade-guide-from-monitoring-3.x-to-monitoring-4.y.rst
ljgonzalez37 Aug 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 12 additions & 19 deletions docs/source/upgrade/upgrade-guide-from-monitoring-3.x-to-monitoring-4.y.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,8 +99,7 @@ We assume that you are using external volume to store the metrics data.
Backup
^^^^^^

We suggest to copy the Prometheus external directory first and use the copy as the data directory for the new monitoring stack.
Newer Monitoring stack uses newer Promethues versions, and keeping a backup of the prometheus dir would allow you to rollback.
A copy of the Prometheus external directory should be made first and used as the data directory for the new monitoring stack. The new monitoring stack uses newer versions of Prometheus and keeping a backup would enable you to rollback to a previous version of Prometheus.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why move to passive voice? I find the original version clearer.
You switch to singular, but we are talking about versions (i.e. 3.x and 4.y)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid the passive voice, we can use the following paragraph:

We suggest to make a copy of Prometheus' external directory and use it as the data directory for the new monitoring stack. The new monitoring stack uses newer versions of Prometheus and keeping a backup would enable you to rollback to a previous version of Prometheus.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A copy of the Prometheus external directory should be made first and used as the data directory for the new monitoring stack. The new monitoring stack uses newer versions of Prometheus and keeping a backup would enable you to rollback to a previous version of Prometheus.
We suggest making a copy of Prometheus's external directory to use as the data directory for the new version of Monitoring Stack. The new version of Monitoring Stack uses the new version of Prometheus. If you keep a backup of Prometheus's external directory, you can roll back to the previous Prometheus version.


Kill all containers
^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -210,10 +209,11 @@ The point in time that the graphs start will be your back-filling end time. Chec

Backfilling Process
-------------------
backup
Backup
^^^^^^
If you have a long retention period you are using an external directory that holds the Prometheus data, back it up, in case
If you have a long retention period, you are using an external directory that holds the Prometheus data back it up; if something goes wrong in the process, you can revert the process.
Backup any external directory containing Prometheus data; if something goes wrong, you can revert the changes.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not any, it the external data directory

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The modified sentence would be:

Backup the external directory containing Prometheus data; if something goes wrong, you can revert the changes.


The monitoring stack will need to be restarted at least once. This process cannot be completed without an external directory when using the -d command-line option.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is confusing, see the original description (droping the duplication).
To complete this process, you need to have the data in an exteranl directory, the way you use an external directory is with the -d command line option.


To complete the process you will need to restart the monitoring stack at least once. If you are not using an external directory (The ``-d``
command-line option) You cannot complete it.
Expand All @@ -226,13 +226,9 @@ You need to stop the monitoring stack and run the ``stat-all.sh`` command with a

Create the data files
^^^^^^^^^^^^^^^^^^^^^^^^^
We will use the Promtool utility; it's already installed for you if you are using the docker container.
You will need the start time and end time for the process, in our example the start time is 360 days ago and the end time is 90 days ago.
We will create the data files using the Promtool utility, which has been installed in the docker container. In order to run the utility, the start time and end times must be passed in epoch format. For example, suppose the start and end times are 360 and 90 days back, then the following commands can be used to transform those numbers to epoch: ``echo $((`date +%s` - 3600*24*90))`` and ``echo $((`date +%s` - 3600*24*360))``
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is true only if you are using the docker container.
I prefer the original description that breaks the explenation into stages, you need a start and end time.
you need them in epoch,
there are 100 different ways of transling time to epoch, this is just one example of how to do it from the command line

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new paragraph would be:

We will create the data files using the Promtool utility, which has been installed in the docker container. In order to run the utility, the start time and end times must be passed in epoch format. For example, suppose the start and end times are 360 and 90 days back, then the following commands can be used to transform those numbers to epoch: echo $((`date +%s` - 3600*24*90)) and echo $((`date +%s` - 3600*24*360)) It is important to note that the previous example is just one of many ways to convert time to epoch.


The start and end times are in epoc, so you will need to translate the times to epoc. There are many ways to do this - for example, from the command line.
Run the following command to get the epoc time for 90 days ago: : ``echo $((`date +%s` - 3600*24*90))``

Log in to your docker container and run the following (``start`` and ``end`` should be the start and end in epoc time):
Log in to your docker container and run the following (``start`` and ``end`` should be in epoch format):

.. code-block:: bash

Expand All @@ -244,24 +240,21 @@ Log in to your docker container and run the following (``start`` and ``end`` sho
--url http://localhost:9090 \
/etc/prometheus/prom_rules/back_fill/3.8/rules.1.yml

It will create a ``data`` directory in the directory where you run it.
The reason to run it under the ``/prometheus/data/`` is you can be sure Prometheus has write privileges there.
A ``data`` directory will be created in the directory where you run the previous commands. The reason to run it under the ``/prometheus/data/`` is you can be sure Prometheus has write privileges there.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why switch to passive voice? I find the the longer explenation confusing.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A ``data`` directory will be created in the directory where you run the previous commands. The reason to run it under the ``/prometheus/data/`` is you can be sure Prometheus has write privileges there.
It will create a ``data`` directory in the directory where you run it. The reason to run it under the ``/prometheus/data/`` is to ensure that Prometheus has write privileges to the directory.


.. note::
Depending on the time range and the number of cores, the process can take a long time. During testing it took an hour for every week of data,
for a cluster with a total of 100 cores. Make sure that the creation process is not inerupt. You can split the time range to smaller durations
(e.g. instead of an entire year, do it a weeks at a time).
.. note:
This process may take a long time depending on the time range and number of cores. For instance, for a cluster with 100 cores, the process took an hour for every week of data during testing. Hence, please be patient and make sure that the creation process is not interrupted. Please note the time range can be split into smaller intervals (e.g., instead of an entire year, break it down into weeks).


Copy the data files
^^^^^^^^^^^^^^^^^^^
Make sure that the process is completed successfully - don't start this section before you complete the previous sections.
You should not start this section until all the previous sections have been completed. To copy the data files from the Docker Host to the Prometheus directory in the Docker container, run the following command:

Copy the data files to the Prometheus directory:

.. code-block:: bash

cp data/* .
docker cp {options} SRC_PATH aprom:/prometheus/data

The rules will be evaluated next time Prometheus will perform compaction. You can force it by restarting the server using ``docker restart aprom``

Expand Down