|
1 | 1 | Setting up a Rucio development environment |
2 | 2 | ========================================== |
3 | 3 |
|
4 | | -Prerequisites |
5 | | --------------- |
6 | | - |
7 | | -We provide a containerised version of the Rucio development environment for a quick start. Our containers are ready-made for Docker, which means you need to have a working Docker installation. To install Docker for your platform, please refer to the `Docker installation guide <https://docs.docker.com/install/>`_, for example, for Debian/Ubuntu `follow these instructions for the Docker Community Edition <https://docs.docker.com/install/linux/docker-ce/debian/>`_. |
8 | | - |
9 | | -Start the Docker daemon with `sudo systemctl start docker`. You can confirm that Docker is running properly by executing (might need `sudo`):: |
10 | | - |
11 | | - docker run hello-world |
12 | | - |
13 | | -If successful, this will print an informational message telling you that you are ready to go. Now, also install the `docker-compose` helper tool, e.g., with `sudo apt install docker-compose`. You are now ready to install the Rucio development environment. |
14 | | - |
15 | | -This container can be found on Dockerhub as `rucio/rucio-dev`, and the corresponding `Dockerfile <https://github.com/rucio/containers/tree/master/dev>`_ is also available. It provides a Rucio environment which allows you to mount your local code in the containers `bin`, `lib`, and `tools` directory. The container is set up to run against a PostgreSQL database with fsync and most durability features for the WAL disabled to improve testing IO throughput. Tests and checks can be run against the development code without having to rebuild the container. |
16 | | - |
17 | | -Preparing the environment |
18 | | -------------------------- |
19 | | - |
20 | | -The first step is to fork the `main Rucio repository on GitHub <https://github.com/rucio/rucio>`_ by clicking the yellow Fork Star button, and then clone your private forked Rucio repository to your `~/dev/rucio`. Afterwards add the main upstream repository as an additional remote to be able to submit pull requests later:: |
21 | | - |
22 | | - cd ~/dev |
23 | | - git clone git@github.com:<your_username>/rucio.git |
24 | | - cd rucio |
25 | | - git remote add upstream git@github.com:rucio/rucio.git |
26 | | - git fetch --all |
27 | | - |
28 | | -Now, ensure that the `.git/config` is proper, i.e., mentioning your full name and email address, and that the `.githubtoken` is correctly set. Optionally, you can also replace the `~/dev/rucio/tools/pre-commit` hook with the one provided `here <https://raw.githubusercontent.com/rucio/containers/master/dev/pre-commit>`_ so that `pylint` run in the container rather then in the local system. |
29 | | - |
30 | | -Next, setup and configure the Rucio development environment (again might need `sudo`):: |
31 | | - |
32 | | - docker-compose --file etc/docker/dev/docker-compose.yml up -d |
33 | | - |
34 | | -And verify that it is running properly:: |
35 | | - |
36 | | - docker ps |
37 | | - |
38 | | -This should show you a few running containers: the Rucio server, the PostgreSQL database, FTS and its associated MySQL database, the Graphite monitoring, and three XrootD storage servers. |
39 | | - |
40 | | -Finally, you can jump into the container with:: |
41 | | - |
42 | | - docker exec -it dev_rucio_1 bin/bash |
43 | | - |
44 | | -To verify that everything is in order, you can now run the unit tests. So again, inside the container, either run the full testing suite (which takes ~10 minutes):: |
45 | | - |
46 | | - tools/run_tests_docker.sh |
47 | | - |
48 | | -Or alternatively, just bootstrap the test environment once and then selectively run test case modules, test case groups, or even single test cases, for example:: |
49 | | - |
50 | | - tools/run_tests_docker.sh -i |
51 | | - nosetests -v lib/rucio/tests/test_replica.py |
52 | | - nosetests -v lib/rucio/tests/test_replica.py:TestReplicaCore |
53 | | - nosetests -v lib/rucio/tests/test_replica.py:TestReplicaCore.test_delete_replicas_from_datasets |
54 | | - |
55 | | -Development |
56 | | ------------ |
57 | | - |
58 | | -The idea for containerised development is that you use your host machine to edit the files, and test the changes within the container environment. On your host machine, you should be able to simply:: |
59 | | - |
60 | | - cd ~/dev/rucio |
61 | | - emacs <file> |
62 | | - |
63 | | -To see your changes in action the recommended way is to jump twice into the container in parallel. One terminal to follow the output of the Rucio server with a shortcut to tail the logfiles (`logshow`), and one terminal to actually run interactive commands: |
64 | | - |
65 | | -From your host, get a separate Terminal 1 (the Rucio "server log show"):: |
66 | | - |
67 | | - docker exec -it dev_rucio_1 /bin/bash |
68 | | - logshow |
69 | | - |
70 | | -Terminal 1 can now be left open, and then from your host go into a new Terminal 2 (the "interactive" terminal):: |
71 | | - |
72 | | - docker exec -it dev_rucio_1 /bin/bash |
73 | | - rucio whoami |
74 | | - |
75 | | -The command will output in Terminal 2, and at the same time the server debug output will be shown in Terminal 1. |
76 | | - |
77 | | -Development tricks |
78 | | ------------------- |
79 | | - |
80 | | -Server changes |
81 | | -~~~~~~~~~~~~~~ |
82 | | - |
83 | | -If you edit server-side files, e.g. in `lib/rucio/web`, and your changes are not showing up then it is usually helpful to flush the memcache and force the webserver to restart without having to restart the container. Inside the container execute:: |
84 | | - |
85 | | - echo 'flush_all' | nc localhost 11211 && httpd -k graceful |
86 | | - |
87 | | -Database access |
88 | | -~~~~~~~~~~~~~~~ |
89 | | - |
90 | | -The default database is PostgreSQL, and `docker-compose` is configured to open its port to the host machine. Using your favourite SQL navigator, e.g., `DBeaver <https://dbeaver.org>`_, you can connect to the database using the default access on `localhost:5432` to database name `rucio`, schema name `dev`, with username `rucio` and password `secret`. |
91 | | - |
92 | | -Docker is eating my disk space |
93 | | -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
94 | | - |
95 | | -You can reclaim this with:: |
96 | | - |
97 | | - docker system prune -f --volumes |
| 4 | +Please consult the relevant documentation at https://rucio.github.io/documentation/operator/setting_up_demo to learn more. |
0 commit comments