Skip to content

Conversation

@fukusuket
Copy link
Collaborator

@fukusuket fukusuket commented Dec 7, 2025

What Changed

Evidence

Integration-Test

I’d appreciate it if you could check it when you have time🙏

@fukusuket fukusuket changed the title feat: implement AWS CloudTrail search command with filtering options feat: implement AWS CloudTrail search command Dec 7, 2025
@fukusuket fukusuket requested a review from Copilot December 7, 2025 00:55
@fukusuket fukusuket self-assigned this Dec 7, 2025
@fukusuket fukusuket added the enhancement New feature or request label Dec 7, 2025
@fukusuket fukusuket added this to the v2.0.0 milestone Dec 7, 2025
@fukusuket fukusuket changed the title feat: implement AWS CloudTrail search command feat: AWS CloudTrail search command Dec 7, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements a new AWS CloudTrail search command that allows users to search and filter CloudTrail logs without requiring Sigma rules. The implementation refactors common CLI options into reusable structs and updates the GeoIP integration to use a newer maxminddb API.

Key changes:

  • Added aws-ct-search command with filtering by fields, keywords, regex, and time ranges
  • Refactored CLI options to create reusable OutputOption and SearchOptions structs
  • Updated GeoIP lookups to use the newer maxminddb lookup().decode() API pattern

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/cmd/aws/aws_search.rs New search command implementation with filter/keyword/regex support and comprehensive unit tests
src/cmd/aws.rs Registers the new aws_search module
src/option/cli.rs Refactors output-related options into OutputOption struct and adds SearchOptions with display_order for consistent help text
src/main.rs Integrates AwsCtSearch command with validation and dispatching logic
src/core/util.rs Extracts load_profile function with skip_sigma parameter for reuse across commands
src/core/timeline.rs Updates to use refactored OutputOption and new load_profile signature
src/core/timeline_writer.rs Supports optional rules by changing write_record to accept Option<&Rule>
src/core/scan.rs Updates to use refactored options.output_opt structure
src/option/geoip.rs Updates GeoIP API usage from lookup::<T>() to lookup().decode::<T>() pattern
Cargo.toml Adds regex = "1" dependency for regex search functionality
Cargo.lock Updates dependency versions including regex and various transitive dependencies
Comments suppressed due to low confidence (2)

src/option/geoip.rs:111

  • The country value is not being cached when lookup succeeds but decode fails. When country.decode() returns Err or Ok(None), the function returns "-" without caching this result. This means subsequent lookups for the same IP will repeatedly attempt to decode, which is inefficient. Consider caching the "-" result as well to avoid redundant lookups.
        match self.country.lookup(ip) {
            Ok(country) => {
                let mut ret = "-";
                if let Ok(Some(country)) = country.decode::<geoip2::Country>() {
                    let name_tree = country.country.names;
                    ret = name_tree.english.unwrap_or("")
                }
                ret.to_string()
            }
            _ => "-".to_string(),
        }

src/option/geoip.rs:131

  • The city value is not being cached when lookup succeeds but decode fails. When city.decode() returns Err or Ok(None), the function returns "-" without caching this result. This means subsequent lookups for the same IP will repeatedly attempt to decode, which is inefficient. Consider caching the "-" result as well to avoid redundant lookups.
        match self.city.lookup(ip) {
            Ok(city) => {
                let mut ret = "-";
                if let Ok(Some(city)) = city.decode::<geoip2::City>() {
                    let name_tree = city.city.names;
                    ret = name_tree.english.unwrap_or("")
                }
                ret.to_string()
            }
            _ => "-".to_string(),
        }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@fukusuket
Copy link
Collaborator Author

help

% ./suzaku aws-ct-search -h
Version: 2.0.0-dev Dev Build
Yamato Security (https://github.com/Yamato-Security/suzaku - @SecurityYamato)

Usage:
  suzaku aws-ct-search <INPUT> [OPTIONS]

General Options:
  -h, --help  Show the help menu

Display Settings:
  -K, --no-color  Disable color output
  -q, --quiet     Quiet mode: do not display the launch banner

Input:
  -d, --directory <DIR>  Directory of multiple gz/json files
  -f, --file <FILE>      File path to one gz/json file

Filtering:
  -F, --filter <FILTER...>     Filter by specific field(s)
  -c, --preserve-case          Case-sensitive keyword search
  -k, --keyword <KEYWORD...>   Search by keyword(s)
  -r, --regex <REGEX>          Search by regular expression
      --timeline-start <DATE>  Start time of the events to load (ex: "2022-02-22T23:59:59Z)
      --timeline-end <DATE>    End time of the events to load (ex: "2020-02-22T00:00:00Z")
      --time-offset <OFFSET>   Scan recent events based on an offset (ex: 1y, 3M, 30d, 24h, 30m)

Output:
  -C, --clobber                    Overwrite files when saving
  -G, --geo-ip <MAXMIND-DB-DIR>    Add GeoIP (ASN, city, country) info to IP addresses
  -o, --output <FILE>              Save the results to a file
  -t, --output-type <OUTPUT_TYPE>  Output type 1: CSV (default), 2: JSON, 3: JSONL, 4: CSV & JSON, 5: CSV & JSONL [default: 1]
      --raw-output                 Output the original JSON logs (only available in JSON formats or stdout)
      --threads <THREAD NUMBER>    Number of threads to use (default: same as CPU cores)

@fukusuket
Copy link
Collaborator Author

fukusuket commented Dec 7, 2025

-k, --keyword

fukusuke@fukusukenoMacBook-Air suzaku-2.0.0-mac-aarch64 % ./suzaku aws-ct-search -d ../data/aws/suzaku-sample-data-main -k CreateUser -q
Start time: 2025/12/07 10:20
Version: 2.0.0-dev (Dev Build)

Total log files: 3,980
Total file size: 251.1 MiB

Scanning now. Please wait.

2019-09-16 15:09:27 · CreateUser · iam.amazonaws.com · us-east-1 · 4.91.139.65 · aws-sdk-go/1.23.15 (go1.12.6; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.7 · backup · IAMUser · 811596193553 · arn:aws:iam::811596193553:user/backup · AIDA9BO36HFBHKGJAO9C1 · AKIA01U43UX3RBRDXF4Q · 571f5a95-8fa1-43b8-b48e-ba1d9b33cd21

2019-09-17 07:28:01 · CreateUser · iam.amazonaws.com · us-east-1 · 5.108.87.250 · aws-sdk-go/1.23.15 (go1.12.6; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.7 · backup · IAMUser · 811596193553 · arn:aws:iam::811596193553:user/backup · AIDA9BO36HFBHKGJAO9C1 · AKIA01U43UX3RBRDXF4Q · 4f5adf0b-97b3-469b-9b83-0c8e8bd5d432

2019-09-30 06:36:10 · CreateUser · iam.amazonaws.com · us-east-1 · 24.251.252.2 · Boto3/1.9.231 Python/2.7.15+ Linux/4.15.0-64-generic Botocore/1.12.231 · Level6 · IAMUser · 811596193553 · arn:aws:iam::811596193553:user/Level6 · AIDADO2GQD0K8TEF7KW1V · AKIA3Z2XBVUDFQ9TU4MD · 059193adb-fb5b-41c5-a878-d1635fe47a08

2019-09-30 06:36:15 · CreateUser · iam.amazonaws.com · us-east-1 · 24.251.252.2 · Boto3/1.9.231 Python/2.7.15+ Linux/4.15.0-64-generic Botocore/1.12.231 · backup · IAMUser · 811596193553 · arn:aws:iam::811596193553:user/backup · AIDA9BO36HFBHKGJAO9C1 · AKIA01U43UX3RBRDXF4Q · e393d54b-f11a-4151-a6c7-bc9cc23ca99e

2020-03-04 20:06:40 · CreateUser · iam.amazonaws.com · us-east-1 · 1.2.100.250 · aws-cli/2.0.2 Python/3.7.5 Windows/10 botocore/2.0.0dev6 · Level6 · IAMUser · 811596193553 · arn:aws:iam::811596193553:user/Level6 · AIDADO2GQD0K8TEF7KW1V · AKIA3Z2XBVUDFQ9TU4MD · e2ed42bb-6388-4d7c-bfe5-2ed2acbe67cf

@fukusuket
Copy link
Collaborator Author

-r, --regex

% ./suzaku aws-ct-search -d ../data/aws/suzaku-sample-data-main -r "Create.*" -q
Start time: 2025/12/07 10:22
Version: 2.0.0-dev (Dev Build)

Total log files: 3,980
Total file size: 251.1 MiB

Scanning now. Please wait.

2021-07-29 23:53:36 · CreateTrail · cloudtrail.amazonaws.com · us-west-1 · 96.253.26.224 · console.amazonaws.com · - · Root · 342082656213 · arn:aws:iam::342082656213:root · 342082656213 · ASIAU7JNXC7KWALOOGDA · 12848d00-3640-475e-9be1-e54782934b8a

2021-07-29 23:53:37 · CreateRole · iam.amazonaws.com · us-east-1 · 96.253.26.224 · Coral/Jakarta · - · Root · 342082656213 · arn:aws:iam::342082656213:root · 342082656213 · ASIAU7JNXC7KWALOOGDA · 045dbab5-d931-4810-8e6b-7042688a283a

2021-07-29 23:53:37 · CreatePolicy · iam.amazonaws.com · us-east-1 · 96.253.26.224 · Coral/Jakarta · - · Root · 342082656213 · arn:aws:iam::342082656213:root · 342082656213 · ASIAU7JNXC7KWALOOGDA · 5b0faa67-1a31-47ce-bc9c-d3c59164195a

2021-07-29 12:58:09 · CreateFlowLogs · ec2.amazonaws.com · us-west-1 · 96.253.26.224 · console.ec2.amazonaws.com · - · Root · 342082656213 · arn:aws:iam::342082656213:root · 342082656213 · ASIAU7JNXC7KYETYTWZT · e5211e1f-e673-449c-a608-a85fb6a5b10e

2021-07-29 12:58:10 · CreateFlowLogs · ec2.amazonaws.com · us-west-1 · 96.253.26.224 · console.ec2.amazonaws.com · - · Root · 342082656213 · arn:aws:iam::342082656213:root · 342082656213 · ASIAU7JNXC7KYETYTWZT · 97d2a78a-44ea-4c10-a9a3-1b56c56ec516

@fukusuket
Copy link
Collaborator Author

-F, --filter

./suzaku aws-ct-search -d ../data/aws/suzaku-sample-data-main -F ".sourceIPAddress:172.240.245.8" -q
Start time: 2025/12/07 10:25
Version: 2.0.0-dev (Dev Build)

Total log files: 3,980
Total file size: 251.1 MiB

Scanning now. Please wait.

2018-08-20 23:49:52 · ListBuckets · s3.amazonaws.com · us-east-1 · 172.240.245.8 · [aws-cli/1.9.10 Python/2.7.15 Darwin/17.7.0 botocore/1.3.10] · backup · IAMUser · 811596193553 · arn:aws:iam::811596193553:user/backup · AIDA9BO36HFBHKGJAO9C1 · AKIA01U43UX3RBRDXF4Q · a08cd126-0c6c-4d86-b2f0-2ddfbf2ce877

2018-08-21 00:11:08 · DescribeInstances · ec2.amazonaws.com · us-west-2 · 172.240.245.8 · aws-cli/1.9.10 Python/2.7.15 Darwin/17.7.0 botocore/1.3.10 · backup · IAMUser · 811596193553 · arn:aws:iam::811596193553:user/backup · AIDA9BO36HFBHKGJAO9C1 · AKIA01U43UX3RBRDXF4Q · ea162d49-4caa-456e-8042-a69c8d7ae8a0

2018-08-21 00:13:56 · DescribeSnapshots · ec2.amazonaws.com · us-west-2 · 172.240.245.8 · aws-cli/1.9.10 Python/2.7.15 Darwin/17.7.0 botocore/1.3.10 · backup · IAMUser · 811596193553 · arn:aws:iam::811596193553:user/backup · AIDA9BO36HFBHKGJAO9C1 · AKIA01U43UX3RBRDXF4Q · 0e70bf9a-6653-487e-a121-ecb3e161163c

2018-09-28 18:42:48 · ListBuckets · s3.amazonaws.com · us-east-1 · 172.240.245.8 · [aws-cli/1.15.79 Python/2.7.15 Darwin/17.7.0 botocore/1.10.78] · backup · IAMUser · 811596193553 · arn:aws:iam::811596193553:user/backup · AIDA9BO36HFBHKGJAO9C1 · AKIA01U43UX3RBRDXF4Q · 5267e47c-bd84-4e0a-8ae4-029432f7c8f2b

2018-09-28 18:46:57 · ListBuckets · s3.amazonaws.com · us-east-1 · 172.240.245.8 · [aws-cli/1.15.79 Python/2.7.15 Darwin/17.7.0 botocore/1.10.78] · backup · IAMUser · 811596193553 · arn:aws:iam::811596193553:user/backup · AIDA9BO36HFBHKGJAO9C1 · AKIA01U43UX3RBRDXF4Q · ca81b460-daee-4a0b-9645-1584b9d8e7b0

@fukusuket
Copy link
Collaborator Author

-t, --output-type (5: CSV & JSONL)

./suzaku aws-ct-search -d ../data/aws/suzaku-sample-data-main -F ".sourceIPAddress:172.240.245.8" -q -o search -t 5
Start time: 2025/12/07 10:26
Version: 2.0.0-dev (Dev Build)

Total log files: 3,980
Total file size: 251.1 MiB

Scanning now. Please wait.

[00:00:06] 3,980 / 3,980   [========================================] 100%

Scanning finished.
                                                                                                                                                          Total events scanned: 1,972,588
Matching events: 36

Results saved: search.csv (8.0 KiB) and search.jsonl (15.9 KiB)
Elapsed time: 00:00:06
cat search.csv | head
Timestamp,EventName,EventSource,AWS-Region,SrcIP,UserAgent,UserName,UserType,UserAccountID,UserARN,UserPrincipalID,UserAccessKeyID,EventID
2018-08-20 23:49:52,ListBuckets,s3.amazonaws.com,us-east-1,172.240.245.8,[aws-cli/1.9.10 Python/2.7.15 Darwin/17.7.0 botocore/1.3.10],backup,IAMUser,811596193553,arn:aws:iam::811596193553:user/backup,AIDA9BO36HFBHKGJAO9C1,AKIA01U43UX3RBRDXF4Q,a08cd126-0c6c-4d86-b2f0-2ddfbf2ce877
2018-08-21 00:11:08,DescribeInstances,ec2.amazonaws.com,us-west-2,172.240.245.8,aws-cli/1.9.10 Python/2.7.15 Darwin/17.7.0 botocore/1.3.10,backup,IAMUser,811596193553,arn:aws:iam::811596193553:user/backup,AIDA9BO36HFBHKGJAO9C1,AKIA01U43UX3RBRDXF4Q,ea162d49-4caa-456e-8042-a69c8d7ae8a0
2018-08-21 00:13:56,DescribeSnapshots,ec2.amazonaws.com,us-west-2,172.240.245.8,aws-cli/1.9.10 Python/2.7.15 Darwin/17.7.0 botocore/1.3.10,backup,IAMUser,811596193553,arn:aws:iam::811596193553:user/backup,AIDA9BO36HFBHKGJAO9C1,AKIA01U43UX3RBRDXF4Q,0e70bf9a-6653-487e-a121-ecb3e161163c
2018-09-28 18:42:48,ListBuckets,s3.amazonaws.com,us-east-1,172.240.245.8,[aws-cli/1.15.79 Python/2.7.15 Darwin/17.7.0 botocore/1.10.78],backup,IAMUser,811596193553,arn:aws:iam::811596193553:user/backup,AIDA9BO36HFBHKGJAO9C1,AKIA01U43UX3RBRDXF4Q,5267e47c-bd84-4e0a-8ae4-029432f7c8f2b
cat search.jsonl | head
{"AWS-Region":"us-east-1","EventID":"a08cd126-0c6c-4d86-b2f0-2ddfbf2ce877","EventName":"ListBuckets","EventSource":"s3.amazonaws.com","SrcIP":"172.240.245.8","Timestamp":"2018-08-20 23:49:52","UserARN":"arn:aws:iam::811596193553:user/backup","UserAccessKeyID":"AKIA01U43UX3RBRDXF4Q","UserAccountID":"811596193553","UserAgent":"[aws-cli/1.9.10 Python/2.7.15 Darwin/17.7.0 botocore/1.3.10]","UserName":"backup","UserPrincipalID":"AIDA9BO36HFBHKGJAO9C1","UserType":"IAMUser"}
{"AWS-Region":"us-west-2","EventID":"ea162d49-4caa-456e-8042-a69c8d7ae8a0","EventName":"DescribeInstances","EventSource":"ec2.amazonaws.com","SrcIP":"172.240.245.8","Timestamp":"2018-08-21 00:11:08","UserARN":"arn:aws:iam::811596193553:user/backup","UserAccessKeyID":"AKIA01U43UX3RBRDXF4Q","UserAccountID":"811596193553","UserAgent":"aws-cli/1.9.10 Python/2.7.15 Darwin/17.7.0 botocore/1.3.10","UserName":"backup","UserPrincipalID":"AIDA9BO36HFBHKGJAO9C1","UserType":"IAMUser"}
{"AWS-Region":"us-west-2","EventID":"0e70bf9a-6653-487e-a121-ecb3e161163c","EventName":"DescribeSnapshots","EventSource":"ec2.amazonaws.com","SrcIP":"172.240.245.8","Timestamp":"2018-08-21 00:13:56","UserARN":"arn:aws:iam::811596193553:user/backup","UserAccessKeyID":"AKIA01U43UX3RBRDXF4Q","UserAccountID":"811596193553","UserAgent":"aws-cli/1.9.10 Python/2.7.15 Darwin/17.7.0 botocore/1.3.10","UserName":"backup","UserPrincipalID":"AIDA9BO36HFBHKGJAO9C1","UserType":"IAMUser"}

@fukusuket fukusuket marked this pull request as ready for review December 7, 2025 01:35
@fukusuket
Copy link
Collaborator Author

-G, --geo-ip

fukusuke@fukusukenoMacBook-Air suzaku-2.0.0-mac-aarch64 % ./suzaku aws-ct-search -d ../data/aws/suzaku-sample-data-main -r Create -q -G ../data/maxmind -o search.csv -C
Start time: 2025/12/07 10:43
Version: 2.0.0-dev (Dev Build)

Total log files: 3,980
Total file size: 251.1 MiB

Scanning now. Please wait.

[00:00:10] 3,980 / 3,980   [========================================] 100%

Scanning finished.
                                                                                                                                                          Total events scanned: 1,972,588
Matching events: 19,473

Results saved: search.csv (5.9 MiB)
Elapsed time: 00:00:10

fukusuke@fukusukenoMacBook-Air suzaku-2.0.0-mac-aarch64 % head search.csv
Timestamp,EventName,EventSource,AWS-Region,SrcIP,SrcASN,SrcCity,SrcCountry,UserAgent,UserName,UserType,UserAccountID,UserARN,UserPrincipalID,UserAccessKeyID,EventID
2021-07-29 23:53:36,CreateTrail,cloudtrail.amazonaws.com,us-west-1,96.253.26.224,UUNET,Rumford,United States,console.amazonaws.com,-,Root,342082656213,arn:aws:iam::342082656213:root,342082656213,ASIAU7JNXC7KWALOOGDA,12848d00-3640-475e-9be1-e54782934b8a
2021-07-29 23:53:37,CreateRole,iam.amazonaws.com,us-east-1,96.253.26.224,UUNET,Rumford,United States,Coral/Jakarta,-,Root,342082656213,arn:aws:iam::342082656213:root,342082656213,ASIAU7JNXC7KWALOOGDA,045dbab5-d931-4810-8e6b-7042688a283a
2021-07-29 23:53:37,CreatePolicy,iam.amazonaws.com,us-east-1,96.253.26.224,UUNET,Rumford,United States,Coral/Jakarta,-,Root,342082656213,arn:aws:iam::342082656213:root,342082656213,ASIAU7JNXC7KWALOOGDA,5b0faa67-1a31-47ce-bc9c-d3c59164195a
2021-07-29 12:58:09,CreateFlowLogs,ec2.amazonaws.com,us-west-1,96.253.26.224,UUNET,Rumford,United States,console.ec2.amazonaws.com,-,Root,342082656213,arn:aws:iam::342082656213:root,342082656213,ASIAU7JNXC7KYETYTWZT,e5211e1f-e673-449c-a608-a85fb6a5b10e

Copy link
Contributor

@YamatoSecurity YamatoSecurity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fukusuket LGTM! Thanks so much!!

@YamatoSecurity YamatoSecurity merged commit f3ecc42 into main Dec 7, 2025
4 checks passed
@fukusuket fukusuket deleted the 55-search branch December 7, 2025 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Maxmind crate compile error aws-ct-search command

3 participants