Merge pull request #1 from mahos/merging

Add system to build from split content

Author: eywalker

.gitignore

@@ -10,6 +10,13 @@
 _build/
 _static/thumbs/
 site/
+build-all/
+build-local/
+loc_built_site/
+full_site/
+version-menu.html
+build_config.py
+
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

Makefile

@@ -23,6 +23,9 @@ site:
 	rm -rf site 
 	make clean
 	make html
+	make latex
 	cp -r $(BUILDDIR)/html site
 	@echo "Build finished. The HTML pages are in site/html."
 
+
+

README.md

@@ -8,18 +8,48 @@ largely based on the [Read The Doc theme](https://github.com/rtfd/sphinx_rtd_the
 # License
 The documentation can be distributed for free use under the [Creative Commons Attribution-ShareAlike 4.0 International Public License](https://creativecommons.org/licenses/by-sa/4.0/).  Any copy or derivation of the documentation must include attribution to "DataJoint contributors" and include the URL reference https://docs.datajoint.io
 
-# Building locally
-1. Fork and clone the repository to your local machine.
-2. Install requirements using `pip3 install -r requirements.txt`
-3. Build the website by running `make site`. This will build and generate the static website in the `site` directory. (Windows users should run `sphinx-build . _build` from the `contents` directory.)
-4. Move inside the `site` folder (or the `_build` folder on Windows) and run the following command to launch a locally web server:
+
+# In-Development: Building All Versions (Full Site)
+1. Clone the repository to your local machine.
+2. Currently this will build using `testDocMain.git`, `testDocMatlab.git` and `testDocPython.git` repo. This will eventually need to be switched out to the actual official DJ documentation repositories. 
+3. Build the website by running `python build-all.py`. This will build and generate the static website in the `full_site` directory.
+- Note for Windows users: Please manually remove the `build-all` folder before running `python build-all.py` for building second time and on... 
+4. Move inside the `full_site` folder and run the following command to launch a local web server:
+    ```bash
+    $ python3 -m http.server
+    ```
+    This should launch a HTTP server locally serving files from the `full_site` directory.
+5. Finally open up a web browser and navigate to `http://localhost:8000` - you should see the built documentation page. The port (i.e. the number after the colon `:`) may differ - refer to the output of the command from the step above for the actual port to use.
+6. If you made changes to the documentation source but you're not seeing the changes reflected, that is because this command builds from contents that are already pushed and tagged in the git repository. Please build locally to test, and then push with updated version numbers to see the changes in the full_site building.
+7. To stop the server, hit `Ctrl+C` in the terminal window that's running the server.
+
+# In-Development: Building Locally/Partially 
+1. Fork and clone the repository to your local machine. Note: datajoint-docs now only contains the common documentions. If you are writing for a specific language, you also need to clone the datajoint-matlab or datajoint-python repository and make sure they are placed on the same level as the datajoint-docs folder.
+2. Rename the cloned folder to `datajoint-docs`. If you cloned the Python/MATLAB repository for local development and building, make sure to rename the folders to `datajoint-python` and `datajoint-matlab` respectively.
+3. Build the website by running `python build-local.py`. This will build and generate the static website in the `loc_built_site` directory. 
+
+- Note for Windows users: Please manually remove the `build-local` folder before running `python build-local.py` for building second time and on...
+- Note 1: `python build-local.py` defaults to building the most updated local common documentation. If you have both `datajoint-matlab` or `datajoint-python` folder locally on the same level, then this will build using those local folders (check and edit the `build-config.py` to make sure the local language folders are correctly placed). If you don't have a local language-specific folder, then it will still build using the most current lang-specific documentation on its respective git repository.
+- Note 2: If you want to test-build a specific language version locally, then add `python/matlab_tag=(version)` after the `python build-local.py`. For example, `python build-local.py matlab_tag=v3.2.5` or `python build-local.py python_tag=v0.9.4` and this should automatically grab the matching common version for building. Make sure you are specifying the full version tag (and not the abbreviated v3.2 in this case).
+Note 3: If for some reason, you don't want to build using the local common version, you can also build using the most updated common version on the git repository by running `python build-local.py False` or `python build-local.py loc_comm=False`. This is mostly likely not going to work well if you already have a pre-existing local language-specific folder. 
+4. Move inside the `loc_built_site` folder and run the following command to launch a local web server:
     ```bash
     $ python3 -m http.server
     ```
-    This should launch a HTTP server locally serving files from the `site` directory.
-5. Finally open up a web browser and navigate to `http://localhost:8000` - you should see the built documentation page. The port (i.e. the number after the color `:`) may differ - refer to the output of the command from the step above for the actual port to use.
-6. If you made changes to the documentation source, rerun `make site` in a separate terminal window, and then refresh the page in the browser - you should see the changes reflected.
-7. To stop the server, hit `Ctrl+C` in the termianl window that's running the server.
+    This should launch a HTTP server locally serving files from the `loc_built_site` directory.
+5. Finally open up a web browser and navigate to `http://localhost:8000` - you should see the built documentation page. The port (i.e. the number after the colon `:`) may differ - refer to the output of the command from the step above for the actual port to use.
+6. To stop the server, hit `Ctrl+C` in the terminal window that's running the server.
+
+# Notes on Tagging
+1. Before you tag anything, please `git tag` to make sure you see the current tag status.
+2. In the `datajoint-docs` folder, `build-version.json` specifies which language versions to build in the build-all/full-build-mode. If you specify v3.2 in matlab, then the site will be built using the most recent tag (ex. v3.2.5 will be used to build, not v3.2.4). 
+3. In the language-specific folder, all documentation contents are inside the `/docs` directory. Inside, you will see a `_version_common.json` file, which should only contain one corresponding common version tag for this language folder to be build alongside. Note that this file specifies the full-tag version (ex. v0.0.3) for the common version.
+(Not sure about how we will coordinate this with the team yet: When you update content in the lang-specific documentation and you also have made changes in the common version to reflect the change, make sure you give a new tag to the common version, update this `_version_common.json` with the new tag, then also add a new tag (should we?) for the specific language version.)
+4. to add a tag: `git tag -a v3.2.5 -m "some message"`
+   to delete local tag: `git tag -d v3.2.5`
+   to delete already-pushed tag: `git push origin :refs/tags/v3.2.5`
+   to push with tag `git push origin v3.2.5`
+
 
 # Guidelines for Writing
 - For inserting a SQL code-block, be sure to use `.. code-block:: mysql` (and NOT `SQL`)

build-all.py

@@ -0,0 +1,176 @@
+import os
+from os import path
+import json
+import glob
+import shutil
+import subprocess
+import platform
+from util import get_newest_tag 
+from util import copy_contents
+
+# default values in case the build config file is missing
+git_urls = {
+    'common': "https://github.com/datajoint/datajoint-docs.git",
+    'matlab': "https://github.com/datajoint/datajoint-matlab.git",
+    'python': "https://github.com/datajoint/datajoint-python.git"
+}
+
+def create_build_folders(lang): 
+    """
+    Prepares the necessary parts for full-versioned documentation site building by 
+    cloning and checking out appropriate tags from the `lang` respective repositories. 
+    Prepared parts will be inside `build-all` directory 
+    """
+    raw_tags = subprocess.Popen(["git", "tag"], cwd= path.join("build-all", "datajoint-" + lang), stdout=subprocess.PIPE).communicate()[0].decode("utf-8").split()
+
+    with open("build_versions.json") as f:
+        print(f)
+        min_tags = json.load(f)
+
+    tags = [get_newest_tag(t, raw_tags) for t in min_tags[lang]]
+
+    for tag in tags:
+        subprocess.Popen(["git", "checkout", tag],
+                         cwd=path.join("build-all", "datajoint-" + lang), stdout=subprocess.PIPE).wait()
+        dsrc_lang = path.join("build-all", "datajoint-" + lang, "docs-parts")
+        dst_build_folder = path.join("build-all", lang + "-" + tag)
+        dst_main = path.join(dst_build_folder, "contents")
+        dst_temp = path.join(dst_main, "comm")
+
+        if path.exists(dst_build_folder):
+            shutil.rmtree(dst_build_folder)
+
+        # copy over the lang source doc contents into the build folder 
+        shutil.copytree(dsrc_lang, dst_main)
+
+        # grab which version of the common folder the lang doc needs to be merged with
+        with open(path.join(dsrc_lang, "version_common.json")) as f:
+            # expected in this format { "comm_version" : "v0.0"}
+            version_info = json.load(f)
+
+        raw_tags_comm = subprocess.Popen(["git", "tag"], cwd=path.join("build-all", "datajoint-docs"), stdout=subprocess.PIPE).communicate()[0].decode("utf-8").split()
+        comm_to_build = get_newest_tag(version_info['comm_version'], raw_tags_comm)
+
+        subprocess.Popen(["git", "checkout", comm_to_build],cwd=path.join("build-all", "datajoint-docs"), stdout=subprocess.PIPE).wait()
+        dsrc_comm = path.join("build-all", "datajoint-docs", "contents")
+        # copy over the cmmon source doc contents into the build folder 
+        shutil.copytree(dsrc_comm, dst_temp)
+
+        # unpacking the content of common into lang-specific build folder
+        copy_contents(dst_temp, dst_main)
+
+        # removing the temporary comm folder because that shouldn't get build
+        shutil.rmtree(dst_temp)
+
+        # copy the datajoint_theme folder, conf.py and makefile for individual lang-ver folder building
+        shutil.copytree("datajoint_theme", path.join(dst_build_folder, "datajoint_theme"))
+        shutil.copy2("Makefile", path.join(dst_build_folder, "Makefile"))
+        shutil.copy2(path.join("contents", "conf.py"), path.join(dst_build_folder, "contents", "conf.py"))
+        shutil.copy2("report.txt", path.join(dst_build_folder, "report.txt"))
+
+        # add current_version <p> tag into the datajoint_theme folder 
+        with open(path.join(dst_build_folder, 'datajoint_theme', 'this_version.html'), 'w+') as f:
+            f.write('<p class="thisVersion">' + lang + "-" + ".".join(tag.split(".")[:-1]) + '</p>')
+
+        # add current_version as release into the conf.py file (for pdf generation)
+        with open(path.join(dst_build_folder, "contents", "conf.py"), 'a+') as f:
+            f.write('release = "' + lang + "-" + ".".join(tag.split(".")[:-1]) + '"')
+
+
+# generate site folder with all contents using the above build folders
+def make_full_site():
+    """
+    Builds the full-versioned site using the `build-all` directory and puts the resulting html/pdf into `full_site` directory.
+    """
+
+    if path.exists('full_site'):
+        shutil.rmtree('full_site')
+    os.makedirs('full_site')
+    
+    # build individual lang-ver folder
+    to_make = [folder for folder in glob.glob(path.join('build-all', '**')) if not path.basename(folder).startswith('datajoint')]
+
+    with open("build_versions.json") as f:
+        min_tags = json.load(f)
+    # min_tags look like this {'python': ['v0.9'], 'matlab': ['v3.2']}
+
+    # create full version-menu listing using the built folders from above
+    with open(path.join('datajoint_theme', 'version-menu.html'), 'w+') as f:
+        for lang in min_tags:
+            for ver in min_tags[lang]:
+                # f.write('<li class="version-menu"><a href="/' + lang + "/" + ver + '">' + lang + "-" + ver + '</a></li>\n')
+                f.write('<li class="version-menu"><a href="/{lang}/{ver}">{lang}-{ver}</a></li>\n'.format(lang=lang, ver=ver))
+       
+    # copy over the full version-menu listing to datajoint_theme FIRST, 
+    # then build individual folders, and copy to full_site folder 
+
+    for folder in to_make:
+        shutil.copy2(path.join('datajoint_theme', 'version-menu.html'), path.join(folder, "datajoint_theme", "version-menu.html"))
+        if platform.system() == "Windows":
+            subprocess.Popen(["sphinx-build", ".", "..\_build\html"], cwd=path.join(folder, "contents")).wait() # builds html by default
+            subprocess.Popen(["sphinx-build", "-b", "latex", ".", "..\_build\latex"], cwd=path.join(folder, "contents")).wait()
+            if path.exists(path.join(folder, "site")):
+                shutil.rmtree(path.join(folder, "site"))
+            os.makedirs(path.join(folder, "site"))
+            copy_contents(path.join(folder, "_build", "html"), path.join(folder, "site"))
+        else:
+            subprocess.Popen(["make", "site"], cwd=folder).wait()
+
+        # making pdf out of the latex directory only if pdflatex runs
+        try:
+            subprocess.Popen(["pdflatex", "DataJointDocs.tex"], cwd=path.join(folder, '_build', 'latex')).wait()
+            subprocess.Popen(["pdflatex", "DataJointDocs.tex"], cwd=path.join(folder, '_build', 'latex')).wait()
+        except:
+            print("Latex environment not set up - no pdf will be generated")
+
+        lang_version = folder.split(os.sep)[1]  # 'matlab-v3.2.2'
+        lang, version = lang_version.split("-")   # e.g. 'matlab',  'v3.2.2'
+        abbr_ver = '.'.join(version.split('.')[:-1])  # e.g. 'v3.2'
+        abbr_lang_ver = lang + '-' + abbr_ver
+
+        shutil.copytree(path.join(folder, "site"), path.join('full_site', lang_version.split("-")[0], abbr_ver))
+
+        if path.exists(path.join(folder, '_build', 'latex', 'DataJointDocs.pdf')):
+            os.rename(path.join(folder, '_build', 'latex', 'DataJointDocs.pdf'), path.join(folder, '_build', 'latex', 'DataJointDocs_{}.pdf'.format(abbr_lang_ver)))
+            shutil.copy2(path.join(folder, '_build', 'latex', 'DataJointDocs_{}.pdf'.format(abbr_lang_ver)), path.join('full_site', lang_version.split("-")[0], abbr_ver))
+
+    for lang in min_tags:
+        ver_list=[]
+        for to_sort in glob.glob(path.join('full_site', lang, '**')):
+            ver_list.append(float((path.basename(to_sort).strip("v"))))
+        # ensure clean format in case float above produces something like 3.70000000000001
+        newest_ver = 'v{:.1f}'.format(max(ver_list)) 
+        src_path = path.join('full_site', lang, newest_ver)
+        copy_contents(src_path, path.join('full_site', lang))
+
+    copy_contents('dj_root_theme', 'full_site')
+    copy_contents(path.join('full_site', 'python', '_static'), path.join('full_site', '_static'))
+
+
+##########################################################
+####====== begin building full version doc here ======####
+if __name__ == "__main__":
+    # if build_config file exists, override the default git_url values with config values
+    try:
+        import build_config as config
+        git_urls = dict(git_urls, **config.config_urls)
+    except:
+        print("build_config.py file missing - will use default values for git repo")
+
+    # ensure build folder is clean before the build
+    if path.exists('build-all'):
+        shutil.rmtree('build-all')
+    os.makedirs('build-all')
+
+    subprocess.Popen(
+        ["git", "clone", git_urls['common'], "datajoint-docs"], cwd="build-all").wait()
+
+    subprocess.Popen(
+        ["git", "clone", git_urls['matlab'], "datajoint-matlab"], cwd="build-all").wait()
+
+    subprocess.Popen(
+        ["git", "clone", git_urls['python'], "datajoint-python"], cwd="build-all").wait()
+
+    create_build_folders("matlab")
+    create_build_folders("python")
+    make_full_site()