Add Impersonation and BEC email hunting queries (with spoof and authentication enhancements) - #15027
Conversation
…f and partner queries New (Hunting + Solution copies): phishing detected by LLM content analysis; first-contact external senders; new external sender domains carrying threats; top recipients targeted by impersonation; impersonation and spoof detections by sender country; inbound sender domains failing email authentication. Enhances six existing queries (partner compromise; impersonation/spoof by detection technology; the three spoof-by-domain queries): latest-record de-duplication, set_has_element on parsed detection methods in place of a brittle string match, and a bad-traffic percentage on the partner query. Versions bumped to 1.1.0.
There was a problem hiding this comment.
Note
Copilot was unable to run its full agentic suite in this review.
Pull request overview
Adds and updates Microsoft Defender Advanced Hunting query YAMLs to improve hunting for email impersonation/BEC and strengthen spoof/authentication analytics across both the standalone Hunting Queries library and the Microsoft Defender XDR solution.
Changes:
- Added new hunting queries for LLM-detected phishing, first-contact external senders/domains with threats, top impersonation/BEC targets, sender-country breakdown, and auth-failure domains.
- Enhanced multiple existing spoof/BEC queries with 30-day scoping, latest-record de-duplication, and more robust DetectionMethods parsing; bumped versions to 1.1.0 where updated.
- Added “bad traffic percent” enrichment to the partner/BEC domain query.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/Spoof and Impersonation/Top Recipients Targeted by Impersonation or BEC.yaml | New query to identify most-targeted recipients for impersonation/BEC. |
| Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/Spoof and Impersonation/Top Domains with BEC Threats inbound.yaml | Updates query with time window, de-duplication, and bad-traffic percentage. |
| Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/Spoof and Impersonation/Spoof detections by Detection Technology.yaml | Adds time scoping + de-duplication; bumps version. |
| Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/Spoof and Impersonation/Impersonation and Spoof Detections by Sender Country.yaml | New country aggregation query for impersonation/spoof detections. |
| Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/Spoof and Impersonation/Impersonation Phishing detections by Detection Technology.yaml | Adds time scoping + de-duplication; bumps version. |
| Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/Phish/Phishing Detected by LLM Content Analysis.yaml | New query for phishing where LLM content analysis contributed. |
| Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/Mailflow/New External Sender Domains Carrying Threats.yaml | New query to highlight newly seen external domains already carrying threats. |
| Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/Mailflow/First-Contact External Email Senders.yaml | New query to surface new external sender relationships. |
| Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/Authentication/Top Spoof detections by Sender Domain.yaml | Uses robust DetectionMethods parsing + de-dup; bumps version. |
| Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/Authentication/Top Spoof Intra-Org detections by SenderDomain.yaml | Uses robust DetectionMethods parsing + de-dup; bumps version. |
| Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/Authentication/Top Spoof DMARC detections by Sender Domain.yaml | Uses robust DetectionMethods parsing + de-dup; bumps version. |
| Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/Authentication/Inbound Sender Domains Failing Email Authentication.yaml | New query to rank inbound sender domains by authentication failures. |
| Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Spoof and Impersonation/Top Recipients Targeted by Impersonation or BEC.yaml | New query to identify most-targeted recipients for impersonation/BEC. |
| Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Spoof and Impersonation/Top Domains with BEC Threats inbound.yaml | Updates query with time window, de-duplication, and bad-traffic percentage. |
| Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Spoof and Impersonation/Spoof detections by Detection Technology.yaml | Adds time scoping + de-duplication; bumps version. |
| Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Spoof and Impersonation/Impersonation and Spoof Detections by Sender Country.yaml | New country aggregation query for impersonation/spoof detections. |
| Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Spoof and Impersonation/Impersonation Phishing detections by Detection Technology.yaml | Adds time scoping + de-duplication; bumps version. |
| Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Phish/Phishing Detected by LLM Content Analysis.yaml | New query for phishing where LLM content analysis contributed. |
| Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Mailflow/New External Sender Domains Carrying Threats.yaml | New query to highlight newly seen external domains already carrying threats. |
| Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Mailflow/First-Contact External Email Senders.yaml | New query to surface new external sender relationships. |
| Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Authentication/Top Spoof detections by Sender Domain.yaml | Uses robust DetectionMethods parsing + de-dup; bumps version. |
| Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Authentication/Top Spoof Intra-Org detections by SenderDomain.yaml | Uses robust DetectionMethods parsing + de-dup; bumps version. |
| Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Authentication/Top Spoof DMARC detections by Sender Domain.yaml | Uses robust DetectionMethods parsing + de-dup; bumps version. |
| Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/Authentication/Inbound Sender Domains Failing Email Authentication.yaml | New query to rank inbound sender domains by authentication failures. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Inbound Sender Domains Failing Email Authentication: rank domains by the number of messages that failed any authentication check instead of summing the per-check counts, which double-counted messages failing more than one check. The per-check counts remain as columns. - Top Recipients Targeted by Impersonation or BEC: rename the sampled sender-domain set to SampleSenderDomains and clarify the description, since it is a sample and not a ranked list. - Top Domains with BEC Threats inbound: read the deduplicated email set once via a let statement, and rename the ratio column to InboundThreatsPer100Outbound. Results are unchanged.
|
I looked into the KqlValidations result. All 24 files in this PR are hunting queries under Hunting Queries/Microsoft 365 Defender/Email and Collaboration Queries/ and Solutions/Microsoft Defender XDR/Hunting Queries/Email and Collaboration Queries/. I extracted every one of their template GUIDs and cross-checked them against the full CI test log, and none of them appear among the failing templates. |
New: Phish Detections by Threat Classification (LLM intent), Top Attacked Users by Phish Threat Classification, Phish Detection IP and Geo Position, Phish Detections by Sender Country. Updated 9 existing Phish queries to de-duplicate messages with arg_max(Timestamp,*) per NetworkMessageId and recipient, enable the phishing-simulation and SecOps-mailbox exclusion, replace bag_unpack/union detection-technology logic with a single mv-expand, use robust parse_json(ConfidenceLevel) instead of substring matching, use Timestamp (not TimeGenerated), and add explicit render types, a phish confidence split and a bad-traffic percentage. Mirrored to both the Hunting and Solutions locations with unique ids per file. The two ThreatClassification queries are added to the KqlValidations skip list because ThreatClassification is a valid EmailEvents column not yet present in the offline validator schema.
# Conflicts: # .script/tests/KqlvalidationsTests/SkipValidationsTemplates.json
# Conflicts: # .script/tests/KqlvalidationsTests/SkipValidationsTemplates.json
|
hey v-maheshbh (@v-maheshbh) v-atulyadav the merge conflict here is now resolved and the branch is synced with the latest master (The PR is mergeable again). Can you kick off to re-run the checks and merge if all comes back clean now? |
Change(s)
Adds six Advanced Hunting queries for email impersonation and Business Email Compromise (BEC), each in both the Hunting Queries library and the Microsoft Defender XDR solution: Phishing Detected by LLM Content Analysis; First-Contact External Email Senders; New External Sender Domains Carrying Threats; Top Recipients Targeted by Impersonation or BEC; Impersonation and Spoof Detections by Sender Country; Inbound Sender Domains Failing Email Authentication.
Also enhances six existing queries (partner compromise; impersonation and spoof by detection technology; the three spoof-by-domain queries) with latest-record de-duplication, a robust set_has_element match on the parsed detection methods, and a bad-traffic percentage on the partner query. Versions bumped to 1.1.0.
Reason for Change(s)
Shares impersonation/BEC email threat-hunting queries from the Microsoft Defender for Office 365 Detections and Insights workbook, and improves the accuracy of the related existing queries.
Solution Version: N/A
Testing Completed: Yes
Checked that the validations are passing: Yes