Data Release
The Data Release web page on the staging site allows the project team to upload lemmatised keywords in context (kwic) from Lemming and deploy the various converted TEI files to the public site (also called live site).
https://tvof3-stg.kdl.kcl.ac.uk/data_release
The page shows the status of the files on the staging site (left hand side) and the live web site (right hand side).
The TVOF editorial team regularly edits the following TEI documents using Oxygen on their personal computers:
- TEI files of the BNF, fr 20125
- TEI files of the BL, Royal 20 D I
- Select Bibliography
- Alignment files that describes paragraphs and match them among various manuscripts
Those documents are continuously synchronised to Dropbox. Most of the dynamic content of the websites derives directly from those files.
An automated Conversion program (also called task or job) runs every two hours on the staging site to download the files from Dropbox then aggregate and convert to HTML so they can be rendered in the Text Viewer, the Alignment visualisations and the Bibliography page.
The 'TEI files' section in the left column shows information about the Conversion task and all the converted files. For each file it gives you the date at which the originals were fetched from Dropbox and then converted.
The Conversion task takes less than 10 minutes to complete. It will be highlighted on screen while it runs. If you move your mouse over 'Conversion', the date and time it last execution appears in a tooltip. If you click the status message next to 'Conversion' it will open a new page showing the log of the internal processing steps carried out by the task. This is particularly useful in case of error.
We now look at the bottom section in the left column.
The Search page is based on three files that you have to manually upload to the staging site as a ZIP document in order to refresh its content.
The three files are listed in that section:
- kwic-out.xml: this is the XML file exported from lemmatiser software (Lemming);
- fr-tokenised.xml: a file that you have manually generated using the tokenisation scripts from the TEI files of your Fr edition;
- royal-tokenised.xml: a file that you have manually generated using the tokenisation scripts from the TEI files of your Royal edition;
To prepare the zip file: copy the three files in an empty folder on your computer. Select the three files and zip them together. Make sure you don't zip the folder itself, just the three files.
Once you have uploaded the zip file, the Indexing task/job will automatically begin. It also takes less than 10 minutes. At the end you should be able to check the results immediately on the search page.
You can generate them from the TEI files of the edition using the conversion and tokenisation scripts KDL has shared with you in Dropbox. You need python 2 on your computer to run them. Please let us know if you need help.
The scripts are also available on github. The one that creates everything for you is called doall.py.
It is important that the two tokenised files were generated at the same time as the kwic-in file that you last imported into Lemming. Otherwise the tokenised files may contain para, seg and word numbers that don't match those found on the kwic-out file exported from XML and the information on the search page will be either incomplete or point to the wrong place. You may want to always save the latest version of the three files in a folder on Dropbox.
Short answer: tick the checkbox in front of the files you want to publish then click 'Apply'. This will copy the files across and the results should be visible almost immediately, except on the Search page which may take up to ten minutes due to the re-indexing task.
Technically you can copy any number of file across at any time. However one golden rule is to always thoroughly test the content on the staging site first (by going to the Text Viewer, Alignment, Search, Bibliography pages) before you copy to the live site; then test thoroughly again on the live site to make sure everything went ok and corresponds to the staging site.
Another important point is that, to some extent, the files are dependent. The alignment files refer to seg numbers in Royal, which may point to particular bibliographic references and the search results point to paragraphs in the Text Viewer.
For that reason the recommended practice is to plan your next release in advance with the rest of the team and agree on a day where you test on the staging site then deploy everything together to the live site. With this approach it is easier to ensure that the release forms a coherent whole, especially prepared and 'finalised' for a given date. You might want to work in cycles where you publish a new version at the end of every two week or every month.
Note that the search files should always be copied together, I can't see a reason at the moment for copying the kwic and not the tokenised files or vice versa.
As explained below there is a very small risk of breaking specific parts of the live site. It is therefore preferable that you publish things on the live site in the morning on a weekday before Friday. If you want to be extra cautious you can let us know in advance, at least the first few times.
I believe that it should be very low since the data release doesn't affect the software and doesn't restart the sites. Especially if you have tested before deploying to the live site.
The worst that could happen is that some of your files are invalid (e.g. incorrect XML, or wrong file uploaded by mistake) and the corresponding part of the site will stop to work (e.g. Error message on the Text Viewer). Hence the importance of testing the staging site first.
If that happens on the staging site you can first look into the logs to see any possible errors, or look at the validation step of the conversions scripts on your machine. If you manage to find and correct the error, just wait until your corrected version is downloaded from Dropbox or manually upload a new version of the search files. If the files are valid then the error on the site should disappear.
If you can't find the error, please do let KDL know and we'll investigate as it might just be a bug in the software. Please let us know immediately if something wrong happens on the live site.