feat: add webhook to settings for custom event types #2839
Conversation
|
Thanks for the PR @meysam81.
Most robust webhook implementations generally rely on some sort of persistence (Redis, Kafka, Sidekiq, RabbitMQ etc.). Bringing in a whole new external dependency isn't ideal at all, so here, again Postgres will have to be relied on for persistence. That brings in RDBMS performance considerations. Piled-up and persisted webhooks shouldn't cause the primary DB to choke and affect critical functionality in the app. |
|
thanks for reviewing and your notes I didn't get a clear message though. are you suggesting a re-architecture to make it more reliable and then merge it? or are you saying that we drop this altogether? cause if it's the former... I can suggest, from your notes, to make changes and perform the following:
every trigger is a sync db persistent to webhook_log with status set to pending then we'll have a single goroutine running with ticker (configurable) that will take and batch those webhooks and run them to completion so, in essence we're dealing with only one goroutine overall
we can drop it but I don't believe it adds a lot of overhead. it's just a few added CPU cycles IMHO
persisting webhooks to the db will survive restart based on the earlier recommendations what do you think? a few adjustments needed to deliver the reliability standard requested? |
Definitely not about dropping this. I am suggesting a discussion and analysis that'll lead to an ideal architecture.
Not really about CPU cycles, but just product spec complexity. Best to ensure consistency across various interfaces (Messengers uses BasicAuth. We can also consider supporting an additional
I'm suggesting this as a possible path ahead, but this has to be evaluated/load tested before its finalized. |
|
that's fair I'll perform the changes discussed so far do you have any recommendation on the benchmark tooling? any preference? or should I just search the web for what works the best or maybe write a custom client benchmark that runs (simulations) of different real world use-cases... for example importing a batch CSV as subscriber, many bounce reports by the upstream smtp server, etc. any suggestion is welcome to close the gap |
|
I looked at Svix webhook... those are the expert in webhook, its internal implementation and the quirks involved... is it completely unacceptable to add another moving part, accepting svix apikey as env var and using their webhook-as-a-service? |
|
Here's a Python snippet to generate dummy CSVs that you can import. Should easily trigger 10s of thousands of events. This is enough for load testing. import csv
import random
f = open("/tmp/subs.csv", "w+")
w = csv.writer(f)
w.writerow(["name", "email", "attributes"])
for n in range(0, 20000):
w.writerow([
"First%d Last%d" % (n, n),
"user%[email protected]" % (n,),
"{\"age\": %d, \"city\": \"Test\"}" % (random.randint(20,70), random.randint(10,99),)
])An external SaaS dependency for webhooks, definitely not. It should be fully self-contained. |
|
when an import of X number of subscriber happens from the admin panel, would you say you would want to receive one webhook for the entire batch? or one event per subscriber? I think the former should be the case, ain't it? |
|
we now have a pool of webhook worker that can be configured on the admin panel we also have 3 additional webhook event type for batch import the only thing left from our conversations so far is to remove the HMAC and add token everything else should be addressed I hope |
|
and that's it the HMAC is removed I think I've addressed your concerns in the new changes please review it at your own pace 🙏 |
2 participants
Hey @knadh
I studied the relevant issue (addressed below) and summed up the requirements into the following PR.
Summarize:
internal/core/directory are properly triggering the correct events. if not webhook is defined by admin or if the no event type is matched, we ignore and move on.X-Listmonk-Signature&X-Listmonk-Timestampheaders.User-Agent: listmonk/v5.1.0,X-Listmonk-Event: subscriber.createdDemo
I'm providing the relevant screenshots here, but to avoid cluttering the description, putting it in html detail.
Event types
Auth types
JSON Payload
The following json payloads are taken from the echo-server
Subscriber created payload
{ "name": "echo-server", "hostname": "aaf3a0b12c1f", "pid": 1, "level": 30, "host": { "hostname": "localhost", "ip": "::ffff:172.17.0.1", "ips": [] }, "http": { "method": "POST", "baseUrl": "", "originalUrl": "/listmonk", "protocol": "http" }, "request": { "params": {}, "query": {}, "cookies": {}, "body": { "event": "subscriber.created", "timestamp": "2025-12-28T13:54:35.544674992Z", "data": { "id": 11, "created_at": "2025-12-28T13:54:29.130854Z", "updated_at": "2025-12-28T13:54:29.130854Z", "uuid": "477f9364-8b0a-4e7b-b820-4b42bf2ff2d5", "email": "[email protected]", "name": "john", "attribs": {}, "status": "enabled", "lists": [ { "subscription_status": "unconfirmed", "subscription_created_at": "2025-12-28T13:54:29.130854+00:00", "subscription_updated_at": "2025-12-28T13:54:29.130854+00:00", "subscription_meta": {}, "id": 2, "uuid": "3e95d710-3873-4873-af68-f4de58331ca9", "name": "Opt-in list", "type": "public", "optin": "double", "status": "active", "tags": [ "test" ], "description": "", "created_at": "2025-12-27T04:18:58.360369+00:00", "updated_at": "2025-12-27T04:18:58.360369+00:00" } ] } }, "headers": { "host": "localhost:10000", "user-agent": "listmonk-webhook/1.0", "content-length": "748", "content-type": "application/json", "x-listmonk-event": "subscriber.created", "x-listmonk-signature": "sha256=acc61cddfa8ec62ddadd58f40426791b6874e27428f981554435bdf7ebaaaed5", "x-listmonk-timestamp": "1766930075", "accept-encoding": "gzip" } }, "msg": "Sun, 28 Dec 2025 13:54:35 GMT | [POST] - http://localhost:10000/listmonk", "time": "2025-12-28T13:54:35.554Z", "v": 0 }Campaign started payload
{ "name": "echo-server", "hostname": "aaf3a0b12c1f", "pid": 1, "level": 30, "host": { "hostname": "localhost", "ip": "::ffff:172.17.0.1", "ips": [] }, "http": { "method": "POST", "baseUrl": "", "originalUrl": "/listmonk", "protocol": "http" }, "request": { "params": {}, "query": {}, "cookies": {}, "body": { "event": "campaign.started", "timestamp": "2025-12-28T13:57:17.153960863Z", "data": { "id": 1, "created_at": "2025-12-27T04:18:59.092583Z", "updated_at": "2025-12-27T04:18:59.092583Z", "views": 0, "clicks": 0, "bounces": 0, "lists": [ { "id": 1, "name": "Default list" } ], "media": [], "started_at": null, "to_send": 0, "sent": 0, "uuid": "4ab67b3f-78b1-45a5-8254-eea019c9e79e", "type": "regular", "name": "Test campaign", "subject": "Welcome to listmonk", "from_email": "No Reply <[email protected]>", "body": "<h3>Hi {{ .Subscriber.FirstName }}!</h3>\n\t\t<p>This is a test e-mail campaign. Your second name is {{ .Subscriber.LastName }} and you are from {{ .Subscriber.Attribs.city }}.</p>\n\t\t<p>Here is a <a href=\"https://listmonk.app@TrackLink\">tracked link</a>.</p>\n\t\t<p>Use the link icon in the editor toolbar or when writing raw HTML or Markdown,\n\t\t\tsimply suffix @TrackLink to the end of a URL to turn it into a tracking link. Example:</p>\n\t\t<pre><a href="https:/‌/listmonk.app@TrackLink"></a></pre>\n\t\t<p>For help, refer to the <a href=\"https://listmonk.app/docs\">documentation</a>.</p>\n\t\t", "body_source": null, "altbody": null, "send_at": null, "status": "running", "content_type": "richtext", "tags": [ "test-campaign" ], "headers": [], "template_id": 1, "messenger": "email", "archive": false, "archive_slug": "welcome-to-listmonk", "archive_template_id": 2, "archive_meta": { "name": "Subscriber" } } }, "headers": { "host": "localhost:10000", "user-agent": "listmonk-webhook/1.0", "content-length": "1606", "content-type": "application/json", "x-listmonk-event": "campaign.started", "x-listmonk-signature": "sha256=020e28ab4718c9b99a01a1ef197e93f49c7ec1347162cfb9bc7858106fd310eb", "x-listmonk-timestamp": "1766930237", "accept-encoding": "gzip" } }, "msg": "Sun, 28 Dec 2025 13:57:17 GMT | [POST] - http://localhost:10000/listmonk", "time": "2025-12-28T13:57:17.157Z", "v": 0 }NOTE: if you still request additional changes, please let me know and I'll accommodate asap.
cheers 🥂
fixes #658
Future works