xnat: automate query to establish a new provider for non-public project

[yarikoptic]

XNAT does not return 403 but 404:

$> wget -S 'https://central.xnat.org/data/projects/test-private?format=json'
--2021-04-20 08:39:25--  https://central.xnat.org/data/projects/test-private?format=json
Resolving central.xnat.org (central.xnat.org)... 128.252.217.126
Connecting to central.xnat.org (central.xnat.org)|128.252.217.126|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 404 Not Found
  Set-Cookie: JSESSIONID=central.xnat.org3~8D1E10CD81618689E10E473DC2E37F27; Path=/; HttpOnly
  Set-Cookie: SESSION_EXPIRATION_TIME="1618922365335,900000"; Version=1; Path=/
  Date: Tue, 20 Apr 2021 12:39:26 GMT
  Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
  Accept-Ranges: bytes
  Server: Noelios-Restlet-Engine/1.1.10
  X-Content-Type-Options: nosniff
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  Content-Security-Policy: frame-ancestors 'self'
  Content-Type: text/html;charset=ISO-8859-1
  Content-Length: 312
2021-04-20 08:39:26 ERROR 404: Not Found.

unfortunately this provider config I crafted

[provider:xnat-test-private]
#url_re = https://central.xnat.org/data/projects/test-private.*
url_re = https://central.xnat.org/.*
credential = xnat-test-private
authentication_type = html_form
html_form_url = https://central.xnat.org/app/template/Login.vm#!
html_form_tagid = login_form
html_form_fields = username={user}
                   password={password}
                   XNAT_CSRF=
html_form_success_re = Logged in as


[credential:xnat-test-private]
url = https://central.xnat.org
type = user_password

is not yet complete/working:

$> datalad -l debug download-url 'https://central.xnat.org/data/projects/test-private?format=json'
...
[DEBUG  ] Importing keyring 
[DEBUG  ] Posted to https://central.xnat.org/app/template/Login.vm#! fields ['username', 'password', 'XNAT_CSRF'], got response <Response [200]> with headers ['Server', 'Set-Cookie', 'X-Content-Type-Options', 'X-XSS-Protection', 'X-Frame-Options', 'Content-Security-Policy', 'Content-Type', 'Content-Language', 'Transfer-Encoding', 'Date'] 
download_url(error): /tmp/ (file) [Access to https://central.xnat.org/data/projects/test-private?format=json has failed: not found [http.py:check_response_status:104]]
[DEBUG  ] could not perform all requested actions: Command did not complete successfully. 1 failed:
[{'action': 'download_url',
  'exception_traceback': '[download_url.py:__call__:185,providers.py:download:482,base.py:download:521,base.py:access:169,base.py:_download:419,http.py:get_downloader_session:573,http.py:check_response_status:104]',
  'message': 'Access to '
             'https://central.xnat.org/data/projects/test-private?format=json '
             'has failed: not found [http.py:check_response_status:104]',
  'path': '/tmp/',
  'status': 'error',
  'type': 'file'}] [utils.py:generator_func:455] 

so more debugging/dance is needed. May be @chaselgrove remembers some detail?

attn @dnkennedy @satra

[yarikoptic]

1. returned response also does not 403, but rather presents a page with Logged in as: Guest where <span style="color:red;">Guest</span>
2. and looking at html points to

    XNAT.app = getObject(XNAT.app||{});
    XNAT.app.addFormCSRF = function(form){
        $$(form||'form').each(function(){
            var form$ = $(this);
            if (!form$.has('input[name="XNAT_CSRF"]').length){
                form$.append('<input type="hidden" name="XNAT_CSRF" value="' + csrfToken + '">')
            }
        });
    };

so that XNAT_CSRF should not be empty but populated with csrfToken which is populated in

    <script>

        var XNAT = XNAT || {};
        var serverRoot = '';
        var csrfToken = 'e86e98a9-moreofuuid;

which is minted upon each login attempt:

lena:/tmp
$> wget -q -O- 'https://central.xnat.org/app/template/Login.vm#!' | grep csrf
        var csrfToken = 'ec33fc85-e053-4b28-ae19-bc0544a287fa';
                form$.append('<input type="hidden" name="XNAT_CSRF" value="' + csrfToken + '">')

$> wget -q -O- 'https://central.xnat.org/app/template/Login.vm#!' | grep csrf
        var csrfToken = 'e3325b07-aa7f-4047-91f6-a60a202b8fd8';
                form$.append('<input type="hidden" name="XNAT_CSRF" value="' + csrfToken + '">')

and I do not think we have support for picking those up from html's <script> yet in datalad. So indeed more dance is needed.

[yarikoptic]

most likely this (HTML form) is a wrong way to login, and there should be API endpoint. Worth looking into pyxnat and/or datalad-xnat (once again -- may be that one already just works).

PS edit: i.e. for now I am stepping away from this challenge

[satra]

datalad-xnat did not work with the endpoint i tried. just returned the login page.

and datalad crawl-init gives a weird error

$ datalad crawl-init --save -t xnat url=https://intradb.humanconnectome.org/ project=CCF_BANDA_STG
[ERROR ] Failed to import pipeline from /Users/satra/software/miniconda3/envs/datalad/lib/python3.7/site-packages/datalad_crawler/pipelines/xnat.py: No module named 'nose' [utils.py::47] [pipeline.py:load_pipeline_from_module:403] (RuntimeError)