Prototype Thredds verify twitcher

[tlvu]

Prototype, not to be merged. PR just for discussion.

I was not able to get the "verify" to work when I keep the old Thredds context-root /twitcher/ows/proxy/thedds to avoid breaking all existing notebooks.

@mishaschwartz could you check which config I overlooked. I think verify was working initially when the context-root as simply /thredds.

However I was able to completely bypass Twitcher while keeping /twitcher/ows/proxy/thedds as context-root. Search for all changes containing the word bypass.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1629/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : thredds-verify-twitcher
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1170/

NOTEBOOK TEST RESULTS

    
[2023-06-12T18:21:45.827Z] ============================= test session starts ==============================
[2023-06-12T18:21:45.827Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-06-12T18:21:45.827Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-06-12T18:21:45.827Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-06-12T18:21:45.827Z] collected 245 items
[2023-06-12T18:21:45.827Z] 
[2023-06-12T18:21:51.242Z] notebooks-auth/test_thredds.ipynb ......FFF..                            [  4%]
[2023-06-12T18:21:59.403Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2023-06-12T18:22:07.541Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  9%]
[2023-06-12T18:22:14.295Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 13%]
[2023-06-12T18:22:16.778Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 17%]
[2023-06-12T18:36:33.019Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 22%]
[2023-06-12T18:36:33.019Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 22%]
[2023-06-12T18:36:40.963Z] ...............                                                          [ 28%]
[2023-06-12T18:36:49.609Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 30%]
[2023-06-12T18:36:56.234Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 33%]
[2023-06-12T18:37:14.861Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 35%]
[2023-06-12T18:37:16.242Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 35%]
[2023-06-12T18:37:20.920Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 38%]
[2023-06-12T18:37:25.291Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ..FF.       [ 40%]
[2023-06-12T18:40:11.652Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 46%]
[2023-06-12T18:41:22.687Z] .............                                                            [ 52%]
[2023-06-12T18:41:25.709Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 53%]
[2023-06-12T18:41:28.045Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 57%]
[2023-06-12T18:41:47.343Z] .................                                                        [ 64%]
[2023-06-12T18:41:53.062Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 66%]
[2023-06-12T18:41:54.456Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 66%]
[2023-06-12T18:42:03.314Z] .FF......                                                                [ 70%]
[2023-06-12T18:42:12.591Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 74%]
[2023-06-12T18:42:21.429Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 77%]
[2023-06-12T18:42:23.969Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 77%]
[2023-06-12T18:42:34.581Z] ......                                                                   [ 80%]
[2023-06-12T18:45:56.135Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 80%]
[2023-06-12T18:47:01.677Z] .............                                                            [ 85%]
[2023-06-12T18:47:16.574Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 86%]
[2023-06-12T18:51:21.449Z] ....s.                                                                   [ 88%]
[2023-06-12T18:51:28.026Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2023-06-12T18:51:41.354Z] ...                                                                      [ 90%]
[2023-06-12T18:51:53.592Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2023-06-12T18:52:14.556Z] ......                                                                   [ 93%]
[2023-06-12T18:52:16.989Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-06-12T18:55:11.818Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-06-12T18:55:11.818Z] 
[2023-06-12T18:55:11.818Z] =================================== FAILURES ===================================
    
  

[mishaschwartz]

@tlvu I would go with a different strategy actually. I'd keep the changes up to commit 7706c5d and then add the following line to the nginx configuration (in a separate optional component):

rewrite ^${TWITCHER_PROTECTED_PATH}/thredds/(.*) /thredds/$1 permanent;

This will ensure that the existing paths in the notebooks will still work with a lot fewer complex changes to the code.

[mishaschwartz]

@tlvu

rewrite ^${TWITCHER_PROTECTED_PATH}/thredds/(.*) /thredds/$1 permanent;

I've tested this out and can confirm that the redirect works as intended