Skip to content

Commit 1bfbe78

Browse files
crowecawcawcrowecawcaw
authored
feat: add Lambda + EventBridge job event Slack notification sample (aws-deadline#253)
* feat: add Lambda + EventBridge job event Slack notification sample Add a CloudFormation sample that connects an AWS Lambda function to Deadline Cloud job events via Amazon EventBridge. An EventBridge rule matches "Job Run Status Change" events with a terminal (SUCCEEDED or FAILED) status and invokes a Lambda function that posts a notification to Slack using an incoming webhook. The webhook URL is held in a Lambda environment variable so it can be rotated without redeploying. Demonstrates the general mechanism for reacting to Deadline Cloud job events, using Slack notifications as the concrete scenario. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com> * fix: handle Slack POST errors in job events sample Lambda Wrap the webhook POST in try/except for urllib HTTPError and URLError so a mistyped, revoked, or rate-limited webhook logs a clear message to CloudWatch Logs and returns, instead of raising an opaque stack trace and triggering silent asynchronous Lambda retries. Addresses PR review feedback. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com> * fix: also catch OSError (timeouts) around Slack POST A read timeout on the webhook POST raises TimeoutError, which is an OSError but not a URLError, so it escaped both except clauses and would propagate -- triggering the async Lambda retries the handler tries to avoid. Add an OSError clause to cover socket timeouts and other low-level I/O errors. Addresses PR review feedback. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com> * docs: note other messaging apps with incoming webhooks Add a section listing messaging apps that support the same incoming-webhook style (Microsoft Teams, Discord, Google Chat, Mattermost) with links to each app's docs on creating a webhook, and note that the JSON body needs adjusting per app. Addresses PR review feedback. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com> --------- Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
1 parent b14bb69 commit 1bfbe78

3 files changed

Lines changed: 393 additions & 0 deletions

File tree

‎cloudformation/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,14 @@ The [budget_events_notification](notification_templates/budget_events_notificati
3131
to receive notifications via email and Slack when a budget threshold is reached in the aws.deadline service. It creates an SNS topic,
3232
an EventBridge rule, and a Chatbot configuration to send the notifications.
3333

34+
## Job event Slack notifications with Lambda
35+
36+
The [job_events_slack_lambda](notification_templates/job_events_slack_lambda/) CloudFormation template demonstrates
37+
how to connect an AWS Lambda function to Deadline Cloud job events through Amazon EventBridge. It creates an
38+
EventBridge rule that matches job completion and failure events and invokes a Lambda function that posts a
39+
notification to a Slack channel via an incoming webhook. Use it as a starting point for reacting to job events
40+
in your own automation.
41+
3442
## Scheduled standby workers
3543

3644
The [fleet_standby_scheduling](farm_templates/fleet_standby_scheduling/) sample CloudFormation template schedules

‎cloudformation/notification_templates/job_events_slack_lambda/README.md‎

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
# Job event Slack notifications with Lambda and EventBridge
2+
3+
This CloudFormation template demonstrates the general mechanism for connecting an AWS Lambda
4+
function to AWS Deadline Cloud job events through Amazon EventBridge. The specific scenario it
5+
implements is sending a [Slack](https://slack.com/) notification whenever a job completes
6+
(`SUCCEEDED`) or fails (`FAILED`).
7+
8+
Deadline Cloud publishes [events to EventBridge](https://docs.aws.amazon.com/deadline-cloud/latest/userguide/eventbridge.html)
9+
on the default event bus of the AWS account and Region that owns the farm. This template creates
10+
an EventBridge rule that matches those job events and invokes a Lambda function, which posts a
11+
formatted message to a Slack channel using an [Incoming Webhook](https://api.slack.com/messaging/webhooks).
12+
13+
You can adapt the Lambda function to do anything else you like with the event — send an email,
14+
open a ticket, update a dashboard, or trigger a downstream workflow.
15+
16+
## Using a different messaging app
17+
18+
This sample uses Slack, but many messaging apps expose the same style of incoming webhook: you
19+
create a webhook URL and `POST` a JSON message to it. To target one of these instead, set that
20+
app's webhook URL as the `SLACK_WEBHOOK_URL` environment variable and adjust the JSON body the
21+
Lambda function builds to match the app's expected payload (each app formats its message JSON
22+
differently). The docs below explain how to create a webhook for each:
23+
24+
- [Slack](https://api.slack.com/messaging/webhooks)
25+
- [Microsoft Teams](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook)
26+
- [Discord](https://discord.com/developers/docs/resources/webhook)
27+
- [Google Chat](https://developers.google.com/workspace/chat/quickstart/webhooks)
28+
- [Mattermost](https://developers.mattermost.com/integrate/webhooks/incoming/)
29+
30+
## How it works
31+
32+
```mermaid
33+
flowchart LR
34+
A["Deadline Cloud job finishes"] -->|"'Job Run Status Change' event<br/>(source: aws.deadline)"| B["EventBridge rule<br/>(matches SUCCEEDED or FAILED)"]
35+
B --> C["Lambda function<br/>(formats a message)"]
36+
C -->|HTTPS POST| D["Slack webhook"]
37+
```
38+
39+
The `Job Run Status Change` event carries a `detail` payload with the job's identifiers and its
40+
new status, for example:
41+
42+
```json
43+
{
44+
"version": "0",
45+
"detail-type": "Job Run Status Change",
46+
"source": "aws.deadline",
47+
"account": "111122223333",
48+
"region": "us-west-2",
49+
"resources": [],
50+
"detail": {
51+
"farmId": "farm-0123456789abcdef0123456789abcdef",
52+
"queueId": "queue-0123456789abcdef0123456789abcdef",
53+
"jobId": "job-0123456789abcdef0123456789abcdef",
54+
"previousTaskRunStatus": "RUNNING",
55+
"taskRunStatus": "SUCCEEDED",
56+
"taskRunStatusCounts": {
57+
"SUCCEEDED": 1,
58+
"FAILED": 0,
59+
"...": 0
60+
}
61+
}
62+
}
63+
```
64+
65+
The EventBridge rule filters on `detail.taskRunStatus` so the Lambda function is only invoked for
66+
finished jobs, not for every intermediate status change.
67+
68+
## Resources created
69+
70+
1. **IAM role** (`JobEventsSlackFunctionRole`): Execution role for the Lambda function. It grants
71+
only CloudWatch Logs write access (via the `AWSLambdaBasicExecutionRole` managed policy) — the
72+
function needs no other AWS permissions because it reaches Slack over HTTPS.
73+
2. **Lambda function** (`JobEventsSlackFunction`): A small Python function (standard library only,
74+
defined inline in the template) that formats the job event and posts it to the Slack webhook URL
75+
held in its `SLACK_WEBHOOK_URL` environment variable.
76+
3. **Lambda permission** (`JobEventsSlackFunctionPermission`): Allows EventBridge to invoke the
77+
function.
78+
4. **EventBridge rule** (`JobEventsRule`): Matches `Job Run Status Change` events with a
79+
`taskRunStatus` of `SUCCEEDED` or `FAILED` (optionally scoped to a single farm) and targets the
80+
Lambda function.
81+
82+
## Prerequisites
83+
84+
1. A Deadline Cloud farm in the same AWS account and Region where you deploy this stack. Job events
85+
are delivered to the event bus of the farm's account and Region, so the stack must be deployed
86+
there.
87+
2. A Slack Incoming Webhook URL. To create one:
88+
- Go to <https://api.slack.com/apps> and create (or open) a Slack app in your workspace.
89+
- Enable **Incoming Webhooks**, then **Add New Webhook to Workspace** and choose the channel to
90+
post to.
91+
- Copy the generated webhook URL. It has the form
92+
`https://hooks.slack.com/services/<workspace-id>/<channel-id>/<token>`.
93+
3. The AWS CLI installed and configured with credentials for the account (if deploying via CLI).
94+
95+
## Parameters
96+
97+
| Parameter | Default | Description |
98+
|---|---|---|
99+
| `EventSource` | `aws.deadline` | EventBridge source for Deadline Cloud events. Keep the default for production. |
100+
| `FarmId` | *(empty)* | Optional. Restrict notifications to a single farm ID. Leave blank for all farms. |
101+
| `SlackWebhookUrl` | *(empty)* | Optional. Slack webhook URL. Can be left blank and set on the Lambda function later. |
102+
103+
## Deployment
104+
105+
### AWS CLI
106+
107+
```bash
108+
aws cloudformation deploy \
109+
--stack-name deadline-job-events-slack \
110+
--template-file job_events_slack_lambda_template.yaml \
111+
--capabilities CAPABILITY_IAM \
112+
--region us-west-2 \
113+
--parameter-overrides \
114+
SlackWebhookUrl='https://hooks.slack.com/services/<workspace-id>/<channel-id>/<token>'
115+
```
116+
117+
Replace the webhook URL with your own. To scope notifications to a single farm, add
118+
`FarmId=farm-...` to `--parameter-overrides`.
119+
120+
### Setting the webhook URL after deployment
121+
122+
If you prefer not to pass the webhook URL as a stack parameter, deploy without it and then set the
123+
environment variable directly on the Lambda function:
124+
125+
```bash
126+
aws lambda update-function-configuration \
127+
--function-name <FunctionName-from-stack-outputs> \
128+
--environment "Variables={SLACK_WEBHOOK_URL=https://hooks.slack.com/services/<workspace-id>/<channel-id>/<token>}" \
129+
--region us-west-2
130+
```
131+
132+
Get `<FunctionName-from-stack-outputs>` from the stack's `FunctionName` output.
133+
134+
### AWS Console
135+
136+
1. Open the AWS CloudFormation console in the Region that owns your farm.
137+
2. Choose **Create stack** → **With new resources (standard)**.
138+
3. Upload `job_events_slack_lambda_template.yaml`.
139+
4. Provide the parameters (at minimum, your `SlackWebhookUrl`).
140+
5. Acknowledge that the stack creates IAM resources, and create the stack.
141+
142+
## Slack message format
143+
144+
The Lambda function posts a message using Slack's [`text`](https://api.slack.com/reference/messaging/payload)
145+
field, formatted with [mrkdwn](https://api.slack.com/reference/surfaces/formatting). The message is
146+
built from the following values taken from the event:
147+
148+
| Line | Source field | Example |
149+
|---|---|---|
150+
| `Deadline Cloud job <status>` | `detail.taskRunStatus` (lower-cased) | `Deadline Cloud job succeeded` |
151+
| `Job` | `detail.jobId` | `job-0123456789abcdef0123456789abcdef` |
152+
| `Queue` | `detail.queueId` | `queue-0123456789abcdef0123456789abcdef` |
153+
| `Farm` | `detail.farmId` | `farm-0123456789abcdef0123456789abcdef` |
154+
| `Region` | top-level `region` | `us-west-2` |
155+
156+
A delivered message looks like this:
157+
158+
```
159+
Deadline Cloud job succeeded
160+
Job: job-0123456789abcdef0123456789abcdef
161+
Queue: queue-0123456789abcdef0123456789abcdef
162+
Farm: farm-0123456789abcdef0123456789abcdef
163+
Region: us-west-2
164+
```
165+
166+
The exact JSON posted to the webhook is:
167+
168+
```json
169+
{
170+
"text": "*Deadline Cloud job succeeded*\nJob: `job-0123456789abcdef0123456789abcdef`\nQueue: `queue-0123456789abcdef0123456789abcdef`\nFarm: `farm-0123456789abcdef0123456789abcdef`\nRegion: `us-west-2`"
171+
}
172+
```
173+
174+
To change what the message says, edit the `text` value the Lambda function builds in the template.
175+
Any field from the event `detail` (see the JSON example above) is available to include.
176+
177+
## Testing
178+
179+
Submit a job to your farm and let it finish (or cancel a task so it fails). Within a few seconds of
180+
the job reaching `SUCCEEDED` or `FAILED`, a message appears in your Slack channel. You can also
181+
inspect the function's execution in its CloudWatch Logs log group (`/aws/lambda/<FunctionName>`).
182+
183+
## Further reading
184+
185+
- [Managing Deadline Cloud events using Amazon EventBridge](https://docs.aws.amazon.com/deadline-cloud/latest/userguide/eventbridge-integration.html)
186+
— how Deadline Cloud publishes events, the list of event detail types, and how to write rules to
187+
route them.
188+
- [Deadline Cloud events detail reference](https://docs.aws.amazon.com/deadline-cloud/latest/userguide/events-detail-reference.html)
189+
— the `detail` schema for each event, including `Job Run Status Change`.
190+
- [Amazon EventBridge event patterns](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html)
191+
— how the rule's `EventPattern` matching works.
192+
- [Amazon EventBridge rule targets](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-targets.html)
193+
— how a rule invokes a target such as a Lambda function.
194+
- [Using AWS Lambda with Amazon EventBridge](https://docs.aws.amazon.com/lambda/latest/dg/services-eventbridge.html)
195+
— the Lambda side of the integration.
196+
- [Sending messages using incoming webhooks](https://api.slack.com/messaging/webhooks) and
197+
[message payload formatting](https://api.slack.com/reference/surfaces/formatting) — the Slack APIs
198+
used by the Lambda function.

0 commit comments

Comments
 (0)