Feature Idea: Time Lapse

[CpuID]

Hey @mattcallanan

An idea for showing people exactly whats going on under the hood would be a time lapse view, rather than just real-time.

This would likely require an architecture shift to make it work:

• persistent storage
• something that can pull the ECS API data regularly and store it with a timestamp
• the frontend moves to a model where it can read saved data in $somedatastore

Feelings/thoughts? I can maybe see a toggle to say "real-time" or "time lapse", to either pull direct from the API or look at the saved data.

I can picture a few ways to go about this, but mostly wanted to float the idea first and gauge interest.

cc @DanielRussell

[mattcallanan]

Hey!

Thanks for the feature suggestion. I love it. In fact, there’s been a similar item in TODO.md for a while (since the start of the project I think): “Sliding timebar to see historical data for comparison (like google street view)”.

Happy to hear your thoughts on how it could work. Also happy to review any PRs!

-Matt

[CpuID]

I'm picturing some form of API export snapshot (depending on what it includes, I suspect the output of a few API calls), either raw or pre-massaged, and then dumped into a datastore.

Technically you could use a dir of JSON files in S3 for this, or maybe DynamoDB, or something else? could be Elasticsearch as JSON documents, maybe even Redis as object blobs?

Lots of different datastore options, would depend on the preferred LoE to manage the storage I guess mostly, plus the desired retention periods.

Gut feeling is avoid Elasticsearch, too heavy for this purpose, I do like the idea of S3 as long as it can be made nice and efficient in terms of frontend UX (not too much waiting for S3 ListObjects API calls etc). You'd also want the system that polls ECS and populates the datastore to handle garbage collection/retention policies I suspect.

I could foresee some standalone process (Node? not my specialty unfortunately) to handle most of those tasks, and then have the rest implemented as frontend UI code that hits S3 directly potentially, or via some intermediary backend service for security/credentials reasons.