Worker Consumption

[B3DTech]

This is more a question that doesn't appear to be answered - does this worker run under the limits of the free tier, or does anyone have a cost idea on the worker?

Thanks.

[jorgelbg]

👋 @B3DTech, you can definitively run this worker within the limits of the Edge Workers free tier. That's how I am using it to gather stats from my personal blog. As long as the number of requests to the website that you want to monitor is below 100k requests/day and 1000 requests/minute you should be fine. It really depends on the traffic of your website(s).

You still need to host Loki/Dashflare for storage/visualization but you can do that pretty much everywhere. I'm using a small VM with 2 vCores and 4GB of RAM for hosting (among other things) Loki and Grafana, this instance receives the stats from 3 different websites.

[B3DTech]

Very good. I'm using Grafana Cloud for now, but may roll my own Loki instance on-prem. Cloud hosting for observably is insane.

For now I set up CloudFlare log push to S3 and am using Datadog Vector to pull the logs, ungzip them and push them up to Loki Cloud. I took your dashboards and converted them into Loki only queries using the log push json logs.

Do you know what metrics/logs are different between your worker code and is in the log push json files? We're moving all our websites into CloudFlare so I'm looking to reproduce all my dashboards that I use with Imperva now.

[jorgelbg]

Very good. I'm using Grafana Cloud for now, but may roll my own Loki instance on-prem. Cloud hosting for observably is insane.

Yep I agree that cloud pricing can get out of hand quickly 😅. I have a local branch that I haven't tested that should add support for using the Loki instance from Grafana Cloud (#25).

For now I set up CloudFlare log push to S3 and am using Datadog Vector to pull the logs, unzip them and push them up to Loki Cloud. I took your dashboards and converted them into Loki only queries using the log push json logs.

Ahh so you are relying on Logpush already, nice.

Do you know what metrics/logs are different between your worker code and is in the log push json files? We're moving all our websites into CloudFlare so I'm looking to reproduce all my dashboards that I use with Imperva now.

I think that Logpush offers some more details about Cloudflare infrastructure, like firewall events that we don't have a way of fetching. Regarding a pure HTTP request what the worker generates is similar in content to an access log but formatted using logfmt:

method=GET url=https://jorgelbg.me/tags/worldmap/ status=304 user_agent="Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.90 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" protocol=https domain=jorgelbg.me origin=https://jorgelbg.me path=/tags/worldmap/ browser=Chrome browser_version=91 os=Android os_version=6.0.1 device_type=mobile country=US duration=171 country_name="United States of America" level=warn session=c6c5611679e932df

From this link I see that there are a bunch of internal data like edge node and bot related fields that are accessible when using Logpush, currently we do not log any of that information but if some of those fields are set as headers in the request we may be able to ingest them from the worker too.

We have a debug mode in the worker that if enabled will store every request/response header (at the time of reaching the worker) as part of the event that the worker sends to loki. Last time that I tested it (it has been a while) there was a lot of interesting information there but I ended up going just for a subset.