git-annex provided paths under windows are POSIX or not?

[mih]

Presently the tests fail, and this may be a reason why.

[adswa]

@bpoldrack and I tried on the remote Win box:

• annex gives windows paths to the special remote
• the export remote error is happening prior to the transfer store export. Annex fails finding the key file in the object tree before communicating to the special remote

I will try to copy the debug error messages

[adswa]

(base) C:\Users\mih\code\datalad-osf>python -m nose datalad_osf/tests 

 .testremote osfproject [2020-06-18 11:25:59.9951041] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2020-06-18 11:26:00.0208496] process done ExitSuccess    
[2020-06-18 11:26:00.0208496] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]                      
[2020-06-18 11:26:00.0364411] process done ExitSuccess     
[2020-06-18 11:26:00.0364411] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..215925bc28d0ef609dbefdd483e6be744a6c230c","--pretty=%H","-n1"]                                                               
[2020-06-18 11:26:00.0520763] process done ExitSuccess                                                                       
[2020-06-18 11:26:00.0520763] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]       
[2020-06-18 11:26:00.0520763] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)"]                                                                                     
[2020-06-18 11:26:00.067795] chat: C:\Users\mih\Miniconda3\Scripts\git-annex-remote-osf.exe []                               
[2020-06-18 11:26:00.8486627] git-annex-remote-osf[1] --> VERSION 1                                                          
[2020-06-18 11:26:00.8486627] git-annex-remote-osf[1] <-- EXTENSIONS INFO                                                    
[2020-06-18 11:26:00.8486627] git-annex-remote-osf[1] --> EXTENSIONS                                                         
[2020-06-18 11:26:00.864285] git-annex-remote-osf[1] <-- PREPARE                                                             
[2020-06-18 11:26:00.864285] git-annex-remote-osf[1] --> GETCONFIG project                                                   
[2020-06-18 11:26:00.864285] git-annex-remote-osf[1] <-- VALUE bgvkc                                                         
[2020-06-18 11:26:02.0057565] git-annex-remote-osf[1] --> PREPARE-SUCCESS                                                    
[2020-06-18 11:26:02.0057565] git-annex-remote-osf[1] <-- GETCOST                                                            
[2020-06-18 11:26:02.0057565] git-annex-remote-osf[1] --> UNSUPPORTED-REQUEST                                                
[2020-06-18 11:26:02.0057565] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","config","remote.osfproject.annex-cost","200.0"]                                                                                                          
[2020-06-18 11:26:02.0362887] process done ExitSuccess                                                                       
[2020-06-18 11:26:02.0362887] read: git ["config","--null","--list"]                                                         
[2020-06-18 11:26:02.0576673] process done ExitSuccess                                                                       
[2020-06-18 11:26:02.0576673] git-annex-remote-osf[1] <-- GETAVAILABILITY                                                    
[2020-06-18 11:26:02.0576673] git-annex-remote-osf[1] --> UNSUPPORTED-REQUEST                                                
[2020-06-18 11:26:02.0576673] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","config","remote.osfproject.annex-availability","GloballyAvailable"]                                                                                      
[2020-06-18 11:26:02.0831893] process done ExitSuccess                                                                       
[2020-06-18 11:26:02.0831893] read: git ["config","--null","--list"]                                                         
[2020-06-18 11:26:02.0987351] process done ExitSuccess                                                                       
[2020-06-18 11:26:02.0987351] chat: C:\Users\mih\Miniconda3\Scripts\git-annex-remote-osf.exe []                              
[2020-06-18 11:26:02.8996026] git-annex-remote-osf[1] --> VERSION 1                                                          
[2020-06-18 11:26:02.8996026] git-annex-remote-osf[1] <-- EXTENSIONS INFO                                                    
[2020-06-18 11:26:02.9034563] git-annex-remote-osf[1] --> EXTENSIONS                                                         
[2020-06-18 11:26:02.9034563] git-annex-remote-osf[1] <-- PREPARE                                                            
[2020-06-18 11:26:02.9034563] git-annex-remote-osf[1] --> GETCONFIG project                                                  
[2020-06-18 11:26:02.9034563] git-annex-remote-osf[1] <-- VALUE bgvkc                                                        
[2020-06-18 11:26:04.1303326] git-annex-remote-osf[1] --> PREPARE-SUCCESS                                                    
[2020-06-18 11:26:04.1303326] git-annex-remote-osf[1] <-- GETCOST                                                            
[2020-06-18 11:26:04.1303326] git-annex-remote-osf[1] --> UNSUPPORTED-REQUEST                                                
[2020-06-18 11:26:04.1303326] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","config","remote.osfproject.annex-cost","200.0"]                                                                                                          
[2020-06-18 11:26:04.1722288] process done ExitSuccess                                                                       
[2020-06-18 11:26:04.1722288] read: git ["config","--null","--list"]                                                         
[2020-06-18 11:26:04.1878596] process done ExitSuccess                                                                       
[2020-06-18 11:26:04.1878596] git-annex-remote-osf[1] <-- GETAVAILABILITY                                                    
[2020-06-18 11:26:04.1878596] git-annex-remote-osf[1] --> UNSUPPORTED-REQUEST                                                
[2020-06-18 11:26:04.1926713] call: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","config","remote.osfproject.annex-availability","GloballyAvailable"]                                                                                     
[2020-06-18 11:26:04.2083062] process done ExitSuccess                                                                       
[2020-06-18 11:26:04.2083062] read: git ["config","--null","--list"]                                                         
[2020-06-18 11:26:04.2238848] process done ExitSuccess                                                                       
(generating test keys...)                                                                                                    
git-annex: MoveFileEx "C:\\Users\\mih\\AppData\\Local\\Temp\\ran1776" Just ".git\\annex\\objects\\1a9\\009\\SHA256E-s1048576--301c64c65f3e4a9c90130881191433b4f106a8a803b56f65127e012df211bb1d.this-is-a-test-key\\SHA256E-s1048576--301c64c65f3e4a9c90130881191433b4f106a8a803b56f65127e012df211bb1d.this-is-a-test-key": does not exist (Das System kann den angegebenen Pfad nicht finden.)                                                                                                                      failed                                                                                                                      
[2020-06-18 11:26:04.2706779] process done ExitSuccess                                                                       
[2020-06-18 11:26:04.2706779] process done ExitSuccess                                                                       
[2020-06-18 11:26:04.2706779] process done ExitSuccess                                                                       
git-annex: testremote: 1 failed                                                                                              E                                                                                                                            
======================================================================                                                       
ERROR: datalad_osf.tests.test_remote.test_gitannex                                                                           
----------------------------------------------------------------------                                                       
Traceback (most recent call last):                                                                                             File 
"C:\Users\mih\Miniconda3\lib\site-packages\nose\case.py", line 198, in runTest                                            
self.test(*self.arg)                                                                                                       File "C:\Users
\mih\code\datalad-osf\datalad_osf\tests\utils.py", line 68, in new_func                                          
return f(*(args + (proj_id,)), **kwargs)                                                                                   File 
"C:\Users\mih\Miniconda3\lib\site-packages\datalad\tests\utils.py", line 714, in newfunc                                  
return t(*(arg + (filename,)), **kw)                                                                                       File 
"C:\Users\mih\code\datalad-osf\datalad_osf\tests\test_remote.py", line 49, in test_gitannex                               
['git', 'annex', 'testremote', 'osfproject', "--fast", "--debug"]                                                          File 
"C:\Users\mih\Miniconda3\lib\site-packages\datalad\cmd.py", line 477, in run                                              
**results,                                                                                                               
datalad.support.exceptions.CommandError: CommandError: '"git" "annex" "testremote" "osfproject" "--
fast" "--debug"' failed with exitcode 1 under C:\Users\mih\AppData\Local
\Temp\datalad_temp_x3_n7luq                                                    -------------------- >> begin captured 
logging << --------------------                                                       urllib3.connectionpool: DEBUG: 
Starting new HTTPS connection (1): api.osf.io:443                                             
urllib3.connectionpool: DEBUG: https://api.osf.io:443 "POST /v2//nodes/ HTTP/1.1" 201 3781                                   
asyncio: DEBUG: Using proactor: IocpProactor                                                                                 
asyncio: DEBUG: Using proactor: IocpProactor                                                                                 
asyncio: DEBUG: Using proactor: IocpProactor                                                                                 
asyncio: DEBUG: Using proactor: IocpProactor                                                                                 
urllib3.connectionpool: DEBUG: https://api.osf.io:443 "DELETE /v2//nodes/bgvkc/ HTTP/1.1" 204 0                              
--------------------- >> end captured logging << ---------------------                                                                                                                                                                                    
----------------------------------------------------------------------                                                       Ran 2 
tests in 12.726s                                                                                                                                                                                                                                    
FAILED (errors=1)        

[adswa]

This is the test_gitannex test ran with an additional --debug in the argument list.

[mih]

Thx!

Hmm, not clear to me. It could be a too long path, or it wants POSIX but get's Windows paths.

[mih]

It might simply be that git-annex uses SAH256 and that leads to very long paths, and they might just be too long for Windows. Examples taken from a session on windows:

In [2]: len(".git\\annex\\objects\\f15\\74f\\SHA256E-s1048576--966a8adac67da2766d779b64956dc6
   ...: 2b467723ac4b94be7e0a816bc627b4af43.this-is-a-test-key\\SHA256E-s1048576--966a8adac67d
   ...: a2766d779b64956dc62b467723ac4b94be7e0a816bc627b4af43.this-is-a-test-key")
Out[2]: 230