Tapis apps deploy upload failure

[JoshuaUrrutia]

Describe the bug
The entire assets directory is not being uploaded when I run tapis apps deploy --no-build, and there is no warning or error when a file fails to upload.

To Reproduce
Probably tricky to reproduce. But for this example the _lib/extend_runtime.sh doesn't always get uploaded to my app assets directory, and there is no error/warning when it fails. Strangely, it only seems to be that file and it fails pretty consistently.
Deploy:

urrutia$ tapis apps deploy --no-build
Pushing poldracklab/mriqc:0.15.1
Finished (182 msec)
Uploading app asset directory "assets" to agave://a2cps.home.urrutia/apps/mriqc-0.15.1
Finding file(s) to upload...
Found 4 file(s) in 0s
Creating remote directory "assets"...
Creating remote directory "assets/_lib"...
Uploading assets/runner-template.sh...
Uploading assets/tester.sh...
Uploading assets/_lib/CONTAINER_IMAGE...
Uploading assets/_lib/extend-runtime.sh...
Uploaded 4 files in 7s
Finished (17816 msec)
+--------+-------------------------------------------------------------+
| stage  | message                                                     |
+--------+-------------------------------------------------------------+
| push   | The push refers to repository [docker.io/poldracklab/mriqc] |
| upload | assets/runner-template.sh                                   |
| upload | assets/tester.sh                                            |
| upload | assets/_lib/CONTAINER_IMAGE                                 |
| upload | assets/_lib/extend-runtime.sh                               |
| create | Created Tapis app urrutia-mriqc-0.15.1 revision 5           |
+--------+-------------------------------------------------------------+

extend_runtime.sh missing when listing the assets directory:

urrutia$ tapis files list agave://a2cps.home.urrutia/apps/mriqc-0.15.1/_lib
+-----------------+---------------+--------+
| name            | lastModified  | length |
+-----------------+---------------+--------+
| CONTAINER_IMAGE | 5 minutes ago |      0 |
+-----------------+---------------+--------+

Expected behavior
I'd expect extend-runtime.sh to be uploaded or to generate an error when it fails. I'm able to upload it directly with:

tapis files upload  agave://a2cps.home.urrutia/apps/mriqc-0.15.1/_lib/ assets/_lib/extend-runtime.sh 

So I don't think there's anything wrong with the file itself.

Desktop (please complete the following information):

• OS: MAC OS catalina
• Version 10.15.6

[mwvaughn]

I have a question: Can you check to see if the file was eventually uploaded?

A persistent problem I have seen with the current deployment process is that there can a delay in completing the actual data movement on the backend when uploading the app asset files. If the app is tested before the data movement queue is cleared out, a failure will be encountered. This is very tricky to debug!

[JoshuaUrrutia]

Nope, doesn't seem to happen eventually. I've checked back hours later and the file wasn't there. One thing I did notice is when it fails, it has some extended attribute on it:

urrutia$ ls -lht assets/_lib/
total 16
-rw-r--r--@ 1 urrutia  staff   4.5K Aug  7 14:24 extend-runtime.sh
-rw-r--r--  1 urrutia  staff     0B Jul 15 17:27 CONTAINER_IMAGE

urrutia$ xattr -l assets/_lib/extend-runtime.sh 
com.apple.macl:
00000000  02 00 E0 A8 FC 9F 1D 50 4E DB AC D4 7A 4E E0 27  |.......PN...zN.'|
00000010  EC 9D 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00                          |........|
00000048

And if I delete, re-copy a "good" file (without the extended attribute), then the upload works fine.