-
-
Notifications
You must be signed in to change notification settings - Fork 167
Docker newbie instructions
-- by Mike Gage
The docker files for webwork were constructed by Pan Luo (@xcompass) at UBC. Thanks Pan!!
This explanation is by and for newbies and may well be simplified, clarified, corrected in later editions. (Jan 6, 2018, July 21, 2018)
The purpose of using docker is to allow nearly everyone who has worked with webwork using the command line to download and test new patches or pull requests to WeBWorK. The initial setup takes a bit more than an hour. After that you can pull a new experimental version of WeBWorK in five minutes for review without interfering with any of the existing work on the computer. Faster testing will allow us to proceed more quickly with new WeBWorK features.
The ways of setting up Docker are evolving. What follows is a relatively simple (if slightly time consuming way) of installing WeBWorK inside docker. We are experimenting with installing webwork2, pg, the mysql database and OPL separately as docker components and then connecting them. This will make it easier to modify and save files in each of these components and to rebuild without having to rebuild everything. (Rebuilding the library database in particular takes some time.) Those with some experience using Docker and webwork may want to look at https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises) and review and contribute to experiments described in the develop branch and on the forum
I have run docker on the MacOS high sierra and am now running it on Mojave. You need at least El Capitan MacOS in order to run Docker.
Docker CE has been successfully installed on Linux machines.
Docker CE is available for Windows machines as well, but getting WeBWorK running under Docker CE for Windows is a bit more complicated than doing so on MacOS X or Linux.
- Docker CE for Windows depends on Hyper-V, so only works on Windows Professional, Education, or Enterprise, but not on Windows Home.
- There is an older Docker Toolbox for Windows which works differently, and which may work on on machines whose version of Windows does not support Docker CE for Windows.
- When Hyper-V is activated during the installation of Docker, it disables VirtualBox and other VM hypervisiors. (The cannot operate in parallel.)
- In order to run VirtualBox again you have to disable Hyper-V. To do so: uncheck Hyper-V in Control Panel -> Programs -> Programs and Features -> Turn Windows features on or off. Then reboot.
- Getting WeBWorK to work in Docker for Windows also requires:
- getting symbolic links to work properly in Windows (possible on recent versions of Windows when Developer Mode is enabled)
- and sharing the Windows drive on which you have your
webwork2folder.
- See:
Please report on your experiences. This wiki is versioned so don't worry about making mistakes when editing this document -- we can always restore an older version.
All commands are issued from the command line so you will need a terminal window. You will need some experience typing unix commands in order to follow these instructions.
Install Docker on your computer. (There are versions for Mac, Windows and Linux)
Docker download home Get the community edition. (CE)
The setup of docker on the Mac is standard and quick, see the Mac Instructions. For Windows see the Windows Instructions and https://servingmath.org/wp/2018/08/installing-webwork-on-docker-on-windows/
Test that it is installed with
$ docker --version
Docker version 18.06.0-ce, build 0ffa825
$ docker-compose --version
docker-compose version 1.22.0, build f46880f
$ docker-machine --version
docker-machine version 0.15.0, build b48dc28d
mkdir webwork_docker # (create a directory anywhere on your disk)
cd webwork_docker # all your work will be in this directory
docker run -d -p 80:80 --name webserver nginx # start a local webserver
after a few minutes an image of the nginx image will have been pulled from the cloud and a container running the webserver will have been created from the image.
In a web browser type http://localhost/ and return (see the Mac Instructions, Windows Instructions, pages above for expected response). Something along the lines of "welcome to nginx".
docker ps # see what containers are running, the first column is container ID
gage-imac-461:webwork_docker mgage$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
017118b37fef nginx "nginx -g 'daemon of�" 2 minutes ago Up 2 minutes 0.0.0.0:80->80/tcp webserver
docker logs CONTAINER_ID # print log/stdout of the container with CONTAINER_ID
gage-imac-461:webwork_docker mgage$ docker stop webserver
webserver
docker stop webserver # stop the server
docker ps # no container is running (they are all stopped and therefore "hidden").
docker ps -a # includes all of the containers including ones that are stopped.
gage-imac-461:webwork_docker mgage$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
017118b37fef nginx "nginx -g 'daemon of�" 5 minutes ago Exited (0) About a minute ago
webserver
docker start webserver # restart the server
docker rm -f webserver # remove the webserver (the -f stands for "force" )
# the image which you downloaded from the cloud and from which the container was created is still on your machine
docker images
gage-imac-461:webwork_docker mgage$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 7042885a156a 44 hours ago 109MB
docker rmi nginx # will remove the image completely
At this point you can read/skim the rest of the Mac Instructions, Windows Instructions, etc pages at your leisure. There are no changes that need to be made, but skimming to see what is available on this page can be helpful.
There is a second getting started page that is useful to read in order to get oriented. For example it explains the difference between a container and an image but first let's actually set up a webwork installation using docker.
- The two Git config settings included below
-c core.autocrlf=false -c core.symlinks=trueshould really only be needed on Windows (as these are the default values on Linux and MacOS X), however forcing them should not interfere with anything on Linux and MacOS X. - It is not necessary to install the
pgrepository below. You can skip thatgit cloneinstruction unless you plan to make changes to the corepgcode, in which case you will also need to editwebwork2/docker-compose.ymlto include your localpgtree at run-time. ( See https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises))
cd webwork_docker
ls # it should be empty -- running the demo above left no trace in this directory
git clone https://github.com/openwebwork/webwork2 -c core.autocrlf=false -c core.symlinks=true # grab the webwork2 repo
#optional
# git clone https://github.com/openwebwork/pg -c core.autocrlf=false -c core.symlinks=true # grab the pg repo
cd webwork2
# Change to the branch needed. (Docker is now part of the master branch so no change is needed. If you are doing
# advanced work you may want to switch to the develop branch.
git branch
# should show "* master" highlighted.
Depending on how fast and how accurately you type these git commands will take 5 minutes or less.
- The revised files will run
OPL-updateautomatically when the container is first run.
Before proceeding, shut everything down in Docker and clear images. docker ps should be clear.
We are still in the webwork2 directory:
ls
If everything is successful the ls listing will show the usual webwork files but also the files
Dockerfile, docker-compose.yml and docker-entrypoint.sh
Now while still in the webwork2 directory type
docker-compose up
You will now have to wait for a while. The terminal window will scroll through lots of lines to let you know that something is running.
Next time you will probably prefer to run
docker-compose up -d # detaches the terminal so you don't see all the output
# and so that you can quit the terminal without stopping the container.
You can see that it is installing a lot of things for the ubuntu version of the linux operating system.
Eventually it installs TeX and Apache, various CPAN modules (you'll see a lot of
Building and testing lines) and then MathJax. Then there are lines about creating the
mysql (dabase) application db_1 and finally the webwork application app_1. Then the admin
course is created and finally the lines will pause with "WebworkWebservice: webwork_directory is set to ..."
IMPORTANT: Don't close the terminal window since that will stop the container running the website.
Towards the end you should see
........
app_1 | Admin course is created.
app_1 | create modelCourse subdirectory
app_1 | defaultClasslist.lst is being created
app_1 | Library version is 2.5; using OPLtables!
app_1 | Mysql database reinitialized.
app_1 | Reading in textbook data from Textbooks in the library /opt/webwork/libraries/webwork-open-problem-library/OpenProblemLibrary.
app_1 | Reading in OPL taxonomy from Taxonomy2 in the library /opt/webwork/libraries/webwork-open-problem-library/OpenProblemLibrary.
app_1 | Saved taxonomy to /opt/webwork/webwork2/htdocs/DATA/tagging-taxonomy.json.
app_1 | Converting data from tagged pgfiles into mysql.
app_1 | Number of files processed:
app_1 | 100 200 300 400 500 600 700 800 900 1000
.....
as completes the installation of webwork (app_1), the database (db_1) and R (r_1) and it starts to build the OPL database. This takes the most time. There are about 35K files that have to be processed. Go get some coffee. After the files are processed there are 126 textbooks that need to be processed to line up problems relevant to those text books.
On my iMac the process took minutes from typing docker-compose up to the final result
app_1 | Updating Library Statistics.
app_1 | Importing statistics for 2 courses.
app_1 | 1 2
app_1 |
app_1 | mysql: [Warning] Using a password on the command line interface can be insecure.
app_1 |
app_1 | Done.
app_1 | Fixing ownership and permissions (just in case it is needed)
app_1 | AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.4. Set the 'ServerName' directive globally to suppress this message
app_1 | webwork.apache2-config: WeBWorK server is starting
app_1 | webwork.apache2-config: WeBWorK root directory set to /opt/webwork/webwork2 in webwork2/conf/webwork.apache2-config
app_1 | webwork.apache2-config: The following locations and urls are set in webwork2/conf/site.conf
app_1 | webwork.apache2-config: PG root directory set to /opt/webwork/pg
app_1 | webwork.apache2-config: WeBWorK server userID is www-data
app_1 | webwork.apache2-config: WeBWorK server groupID is root
app_1 | webwork.apache2-config: The webwork url on this site is http://localhost/webwork2
app_1 | webwork.apache2-config: The webwork smtp server address is localhost
app_1 | webwork.apache2-config: The webwork smtp server port is
app_1 | webwork.apache2-config: The webwork smtp server protocol is 'not ssl'
app_1 | WebworkSOAP::WSDL: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
app_1 | WebworkSOAP::WSDL: rpc_url set to http://localhost/webwork2_rpc
app_1 | WebworkWebservice: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
DO NOT CLOSE THIS WINDOW!!!! The window must be kept open while running the webserver. If you close the window it will shut down the webwork server.
Once the lines have stopped scrolling you can type
http://localhost:8080/webwork2 into your browser and you have a webwork site running.
The only course is "course administration" and it has login "admin" with password "admin".
You can login and add yourself as another admin in the admin course. (First add yourself as a student, then promote yourself to admin.)
You can check the library browser to make sure that it works and that the OPL database has been updated properly.
Create a new course 'test_course', add some guest "students" by importing from demoCourse.lst and add a few homework problems to the course using the LibraryBrowser.
In a new terminal window go to webwork_docker/webwork2. Don't close the first terminal window.
OLD:
cd webwork_docker/webwork2
ls .data
A new directory has been created to hold persistent webwork2 data
cd webwork_docker
ls ww-webwork-data
You'll see courses, which is where persistent data for courses run in the Docker container will be saved. You can add (upload) archived courses to the courses directory and then unarchive them using the admin course in the browser.
With older versions of the Docker config files, there will also be a db directory which is the location for the persistent storage for the database. More recent versions of the Docker config files put the databases into a named Docker volume. It's wise not to touch the db directory unless you are a MariaDB expert.
Read more about the persistent and transient storage at https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises).
- Recent versions of the Docker config files (those at https://github.com/taniwallach/ww_docker_config_sets/tree/master/WW-2.14-ubuntu_16.04) will run
OPL-updateautomatically the first time the container is started.- If you used those files, skip this section.
- In the older versions of the Docker config files (those currently in the main repository), you need to manually run
OPL-update.- This will take about 20 minutes. The download takes about 10 minutes and another 10 minutes of playing around.
- This won't be necessary in future versions -- it will get done automatically but for now.
In your second terminal window (in directory webwork_docker/webwork2) type
docker-compose down
If you go back to your first window you will see that the build process has exited and your browser can no longer access the webwork site.
In the first window type
docker-compose up -d
Notice that things are much quicker this time as the images are already built. And the webserver has started up right where you left off.
Close the first terminal window. And check that the website is still running. (This is what the -d accomplished.)
In the second terminal (webwork_docker/webwork2 directory) type
docker container ls # (or docker ps) and observer that there are two containers powering the website: webwork and mariadb(Mysql)
Observe the container id for webwork. Some commands will require at least the first 4 or 5 characters of that WeBWorK_id to identify the container. Sometimes the name "webwork" alone will work.
type
docker container exec -it WeBWorK_id bash
root@WeBWorK_id:/opt/webwork#
A terminal window has just been opened inside the "remote" machine at the directory /opt/webwork.
cd webwork2/bin
./OPL-update
and go get a cup of coffee while the database for the OPL tags are created. You will get some reassuring lines about how many problems have been processed but there are 35K of them so this process takes a while.
When it's finally done type
exit and you will close the terminal window "connection" to the container (the remote machine)
This takes about an hour from beginning the git downloads to this point. A good twenty minutes of that was waiting for the OPL-update script to populate the database with the problem tags.
In a second terminal (webwork_docker/webwork2 directory) type
docker container ls # (or docker ps) and observer that there are three containers powering the website: webwork and mariadb(Mysql) and r
Observe the container id for webwork. Some commands will require at least the first 4 or 5 characters of that WeBWorK_id to identify the container. Sometimes the name webwork alone will work.
type
docker container exec -it WeBWorK_id bash
root@WeBWorK_id:/opt/webwork#
A terminal window has just been opened inside the Docker container (as if it were a "remote" machine) with the current directory set to /opt/webwork.
Anything that you would do by logging in to your local WeBWorK server on the command line you can do by "logging" in to the container using the approach above.
Warning: Recall that changes made in the command line of the Docker container will only be persistent if they are making changes to files on the persistent storage. See https://github.com/openwebwork/webwork2/wiki/Docker---things-to-get-used-to-(and-avoid-surprises)
If you have a copy of an archived course that you've downloaded from your usual webwork site (a .gz file) we can place that in the container as follows.
- In webwork_docker/webwork2
docker-compose down
cp ../path/to/course/archive .data/courses
ls .data/courses # just to make sure the course is there
ls .data # should have courses and db
docker-compose up
- Check the web browser and the admin course. You should be able to "unarchive" your course using the "unarchive" tab.
- If it is an old course you may have to "update courses" to fix the database on the course you have imported.
- In the web browser open the library browser and make sure that it works.
- You can now add problems to your existing course or add
- The recent Docker control files set up the modelCourse so if you used them you can ignore this section.
- The older Docker control files do not... so in that case:
- Let's do one more thing so that it is easier to create new courses in your local website.
- One way we can fix that is this:
docker container exec -it webwork2_app_1 bash # open a terminal into the container
# into the terminal prompt (root@..... # type
ls # should see toplevel files of webwork
cd courses
ls # should see /admin course
ln -s ../webwork2/courses.dist/modelCourse/ modelCourse # link to the modelCourse distributed in webwork2
Double check that you are running the versions of webwork and pg that you want. At the bash prompt (which you opened inside the Docker container) type:
cd pg
git branch -a # if you are not seeing rel-2.14 and you want to use it
git branch -t rel-2.14 origin/rel-PG-2.14
git branch
git checkout rel-2.14
apachectl graceful # restarts the web server.
- Now try adding a new course from the browser using the Course Administration web page.
- You should see that you can build it using modelCourse as a template.
- When you are done using your Docker container type
docker-compose downand docker will stop using CPU's on your computer. - It will still take up some disk memory space.
- It can be restarted with
docker-compose up -d.
- see Docker for newbies part 2 for adding daemon_course
That's enough for now. Play with the WeBWorK site you have created and go back to skimming the getting started to try to figure out what you have just done. We've only used the first two sections of that document -- there is more that you can do to create a "production site" that can handle many students.
Warning With what we have so far you DO NOT want to use this a production WeBWorK site -- this is a local installation on your machine for you only. None of the databases are using passwords for example, so there is not much security if you start letting people do homework on a network connected to your computer.
To repeat, the purpose of using Docker (so far) is to allow nearly everyone who has worked with WeBWorK using the command line to download and test new patches or pull requests to WeBWorK. Faster testing will allow us to proceed more quickly with new WeBWorK features.
The UBC people are eventually planning to use docker on their production machines so that it is super simple to install WeBWorK.
All the persistent files (database and course files) are stored in webwork_docker/webwork2/.data directory (or a named volume for database storage in more recent configurations). In order to do a completely clean uninstall of this "webwork in a box", remove the .data directory or webwork_docker directory (and the named mysql storage volume).