UI problems

[ediril]

While running shaded jar version 0.22 (I downloaded it from here), I ran into the following problems with the UI:

1. On the Requests page, the DeployId links don't work because the links don't have the request-id in them. They are like this: http://localhost:7099/singularity/request/undefined/deploy/1. Notice it says undefined instead of the actual request id

2. After I navigate to the Deployment page for a specific Request id using the correct link from above, I can see the history of tasks that ran for that deployment. Each task in the history has a link to its logs. However Singularity can't seem to find/read the logs on the slave. Clicking on any of them says:

stdout does not exist in this directory.
It may have been moved to stdout
Back to Task Detail Page

I can actually view the logs if I look at them from the Mesos UI by clicking on the specific Sandbox link. In fact I can see both stdout and stderr. For some reason, Singularity framework is unable to get to the logs.

It would be great if these two UI issues could be fixed.
Thank you

[ssalinas]

First one will be addressed by #1978

The second one I'm not able to replicate. When you click the logs link:

1. Is it this one you are talking about?

2. What url does it put you on? I'm curious if it's a UI issue or possibly something with Singularity's access/config. Singularity tries to fetch the files from the mesos-slave api. So, it will hit the mesos slave on {mesos slave hostname}:5051. If it can't access that (due to security groups or something) then that could also be the cause

[ediril]

Regarding the second one, yes that's the link I'm talking about. It's unable to tail the slave logs for any task. Clicking that link takes me to http://localhost:7099/singularity/task/sleep-ondemand-C-1563394569546-1-mesos_slave-DEFAULT/tail/stdout where I see the message I posted above

When I look at Singularity logs, I can see that it tried to access the slave but it failed:

ERROR [2019-07-17 21:11:07,600] com.hubspot.singularity.mesos.SingularityMesosExecutorInfoSupport: While fetching directory and container id for task: sleep-ondemand-C-1563394569546-1-mesos_slave-DEFAULT
! java.io.IOException: Remotely Closed
! Causing: com.hubspot.horizon.HttpRuntimeException: java.io.IOException: Remotely Closed
! at com.hubspot.horizon.ning.NingHttpClient.execute(NingHttpClient.java:43)
! at com.hubspot.mesos.client.SingularityMesosClient.getFromMesos(SingularityMesosClient.java:68)
! ... 11 common frames omitted
! Causing: com.hubspot.mesos.client.MesosClient$MesosClientException: Exception fetching http://mesos-slave:5051/slave(1)/state after 00:14.445

The ip:port for the slave is correct, and I'm running the hubspot/singularityexecutorslave:0.21.0 docker image as my slave. The slave is able to execute Run-Once and On-Demand tasks. I just can't tail the logs for some reason from the web UI.

Any ideas?

[ssalinas]

Hmm, best guess is that the ip:port isn't accessible from singularity. Are you able to curl that same endpoint from the container that singularity is running in?

[ediril]

So you are saying when I navigate to http://mesos-slave:5051/slave(1)/state in the browser or via curl, I should get something back? This doesn't work for me, I get: mesos-slave didn’t send any data

What does it do for you? What data do you get back?

(For reference, my mesos-master is the mesosphere/mesos-master:1.7.1 docker image)

[ediril]

As another data point, if I switch to using mesosphere/mesos-slave:1.7.1 image, then http://mesos-slave:5051/slave(1)/state endpoint does return json data (and the exception goes away). However, the Logs link still doesn't work.

Does this functionality require using your hubspot/singularityexecutorslave:0.21.0 slave image?

[ssalinas]

Ah, it's missing #1949 , which we added as part of the upgrade to 1.8. We're overdue to release some new stuff anyways. I'll try and get a release put together today or tomorrow morning and get 0.23.0 out there

[ediril]

Awesome thank you! Could you also please make sure #1978 gets included in there as well?

[ssalinas]

https://github.com/HubSpot/Singularity/releases/tag/Singularity-0.23.0

just released in sonatype, jars should show up in a little while on maven central

[ediril]

I downloaded 0.23 (shaded) from here. #1978 should be in this release correct? The link still has undefined instead of request-id. Just wanted to let you know.

My environment at work is a bit locked down so I've been relying on these release jars to evaluate Singularity, but I'll try to build it locally.

[ssalinas]

It should be in there. May need to hard refresh. We have one open issue on the cache headers being too aggressive and not updating nicely between releases

[ediril]

Ah that was it, thank you very much!

[ediril]

Unfortunately, the Logs link still doesn't work for me, I get the same message. I'm not sure how to debug this on my end, I'm not getting any errors from SingularityService

I have the following images running on docker:

hubspot/singularityexecutorslave:0.23.0
mesosphere/mesos-master:1.7.1
mesoscloud/zookeeper:3.4.8-ubuntu-14.04

and I run SingularityService on the command line on the host machine (Windows 10) directly. Everything appears to be running properly and I can see the logs if I look at them in the mesos web UI.

[ssalinas]

Ok. Places to look for stack traces:

• console logs in your browser
• SingularityService logs

I'd expect the first in this case. Also, I've never actually tested this all out on a windows machine, only mac/linux. So there could possibly be some weirdness there