crond jobs don't seem to be triggering in k8s deployment

[eli-gc]

Describe the Bug

I have deployed zipkin-dependencies v3.2.0 on my k8s cluster but I can't get the crond job to work. I can see the crond process running when I do top and if I manually run the script in /etc/periodic/hourly, it works fine and I can see data in Elasticsearch and the Zipkin UI. So I know it's not a connection issue.
No container logs so it seems to me that it isn't running the job. I looked at #192, but no avail. Seems like the issues in that are already fixed so I think mine is different. I run it as the zipkin-dependencies user not root. I am using elasticsearch as the storage backend.

Steps to Reproduce

Deploy image to k8s using deployment.yaml. Use elasticsearch backend. Run as user 1000.

Expected Behaviour

The crond job should run and I should see dependencies data in Zipkin UI

[eli-gc]

I should mention that I am using a read-only root filesystem, but I have a writable volume mounted at /tmp

[eli-gc]

I noticed that there is no crontab entry for the zipkin-dependencies user at /var/spool/cron/crontabs
Only a root file. I am running the container as non root, could this be why?

[codefromthecrypt]

probably the best way to move this forward is to test it, try to reproduce with docker, as this repo doesn't have a helm chart.

e.g. https://github.com/openzipkin/zipkin-dependencies/tree/master/docker#cron

[codefromthecrypt]

fyi openzipkin/zipkin-helm#11 is the currently dormant effort to make this a helm chart

[eli-gc]

I tried adding a crontab for the zipkin-dependencies user in a custom image. While this got the crond to start triggering, I ran into permission issues with the zipkin-dependencies user. It seems like crond calls cron using root. I tested as the root user and it worked. Unfortunately, our security policy prevents us from running containers as root so I ended up having to implement as a k8s cronJob.

[codefromthecrypt]

So, I missed the major detail on #192 which is that the solution was to change to running as root. I think basically this issue should be renamed to "don't require root for scheduled runs of the dependency job".

Maybe someone can then try dcron or similar and raise a PR to change it. gliderlabs/docker-alpine#381

[codefromthecrypt]

sorry maybe supercronic as that's currently maintained. Basically something that does scheduling as non-root. After this is in, someone can also look at k8s native scheduling in openzipkin/zipkin-helm#11, but I think non-root should come first, to reduce guesswork.