Deploying hunting workbooks error

[siuolkl]

Hi everyone,

facing issues getting Sysmon threat hunting workbook to work.
have installed whitelisting CSV storage files to storage container.
from dashboard i am seeing this error " 'project' operator: Failed to resolve table or column expression named 'process_create_whitelist"

not sure if there anything i missed out. appreciate anyone's assistance.

[AU-Kurt]

Not sure if this will help, but I had the same issue and found the log analytics workspace had special characters in it which is not allowed for storage names. Also, found the storage name must be the log analytics workspace name with blobstore (example - log analytics workspace name is "logworksp", then the storage space should be "logworkspblobstore".

[Dutchb0y]

Did this work for you.

[AU-Kurt]

Partially. I was able to get the workspace setup, blobstorage, upload the files so the workbook did not show errors, but I do not see any data in the workbook. I can see data coming into Sentinel and have alert rules which work just fine. Have not found out why the Workbook isn't showing any information.

[damyanor]

Any method to workaround the issue with "static" blobstorage naming? I have a workspace with a "-" character (ie. "my-name") and already filled with logs. And that is why I cannot create a storage account like "my-nameblobstore".

[AU-Kurt]

I am not sure there is a way using the script, unless you make some major modifications. I would think it would be possible if you took the script and used it to set things up by hand, but also feel it would be a lot of work to get it setup properly.

When I found out the issue with the special characters, I ended up creating a new Sentinel workspace with the script and migrated my workbooks and rules to the new workspace. It was the best way forward for me and your mileage my vary.

[damyanor]

sysmon hunting workbook could be created using json, it somehow needs to know the location of csv files. I am not an expert, but from the workbook code analysis and the script I think the script saves custom queries or searches which use the external source. Then the workbook uses the queries. @AU-Kurt since you have a working environment, can you please check if you have queries, "savedSearches" or similar in log workspace which could read the data from these csvs? They should be named like "pipe_whitelist" "process_access_whitelist" etc.

{
            "type": "Microsoft.OperationalInsights/workspaces/savedSearches",
            "apiVersion": "2020-03-01-preview",
            "name": "[concat(parameters('workspace_name'), '/pipe_whitelist')]",
            "dependsOn": [
                "[resourceId('Microsoft.OperationalInsights/workspaces', parameters('workspace_name'))]"
            ],
            "properties": {
                "category": "sentinel-attack",
                "displayName": "pipe_whitelist",
                "query": "[concat('externaldata(host:string, process_path:string, pipe_name:string, reason:string) [h\"https://', parameters('storageAccounts_name'), '.blob.core.windows.net/', parameters('workspace_name') ,'-store/pipe_whitelist.csv?', listAccountSas(parameters('storageAccounts_name'), '2018-07-01', parameters('accountSasProperties')).accountSasToken,'\"] with (ignoreFirstRecord=true)\n')]",
                "functionAlias": "pipe_whitelist",
                "version": 2
            }
        }

[AU-Kurt]

I have the workbook and it references the files as you mention and showed in the script. The issue I ran into was finding the setting where it sets the 'storageAccounts_name' variable. I saw it working through the properties at one point, but it was not in an area where I could change it. If I get some time, I will do more looking to see if I can find the setting.

[AU-Kurt]

@damyanor I found the variable reference in the template used to create the workspace and queries, which can be downloaded and deployed, but have not found where to change the setting. I wonder if there is a way using PowerShell that you could find it and change it. I am no PowerShell expert, but it is a thought.

[damyanor]

I figured it out. First you need to get SAS (access token) of each csv file from the Storage Account -> Container, leaving the default in most sections, but setting access rights to read/write and expire date as you wish.

after that you need to write down a generated "blob SAS token" and a "bob SAS URL" (given at the bottom of the page).
Next, you need to switch to log analytics workspace -> logs and, as a query, enter the following code with modification described below:

externaldata(<csv_header_fields>) [
@"https://<workspacename>.blob.core.windows.net/<container_name>/<name_of_the_csv_file>.csv"
h@"?<token>"
] with (ignoreFirstRecord=true)

you'll need:

• an SAS URL copied earlier, only the path to the file, ie.
  https://someworkspace.blob.core.windows.net/whitelist/dns_whitelist.csv
• a generated token, ie:
  sp=rw&st=2022-02-15T10:52:59Z&se=2032-01-30T23:00:00Z&spr=https&sv=2020-08-04&sr=b&sig=somerandomsignature
• csv column names, got from https://github.com/BlueTeamLabs/sentinel-attack/blob/master/azuredeploy.json -> "query" section (differend for each csv):

"type": "Microsoft.OperationalInsights/workspaces/savedSearches",
(...)
"query": "[concat('externaldata(host:string, process_path:string, query_name:string, reason:string) [h\"https://',parameters('storageAccounts_name'),'.blob.core.windows.net/', parameters('workspace_name') ,'-store/dns_whitelist.csv?', listAccountSas(parameters('storageAccounts_name'), '2018-07-01', parameters('accountSasProperties')).accountSasToken,'\"] with (ignoreFirstRecord=true)\n')]",

we need only this part to copy-paste to the query (dns_whitelist as an example):
host:string, process_path:string, query_name:string, reason:string
so combining all together, the query for dns_whitelist should look like:

externaldata(host:string, process_path:string, query_name:string, reason:string) [
@"https://someworkspace.blob.core.windows.net/whitelist/dns_whitelist.csv"
h@"?sp=rw&st=2022-02-15T10:52:59Z&se=2032-01-30T23:00:00Z&spr=https&sv=2020-08-04&sr=b&sig=somerandomsignature"
] with (ignoreFirstRecord=true)

now we can run the query and see if it works. If succeded, we need to save it as a function, entering as a "legacy category" and "name" entries from azuredeploy.json, respectively: sentinel-attack and name as given in the json:

"category": "sentinel-attack",
 "displayName": "dns_whitelist",

Do it for all the files and then you will add the workbook succesfully.

[AU-Kurt]

I read about setting things up that way , but did not try it. Glad you persevered and was able to get it to work.
Do you see data in the Workbook? I do not get any errors in the Workbook and I have analytic rules in place that work by producing alerts, but I cannot get data in the Workbook.

[damyanor]

Yes, I can see the data, however, have to dig more in docs to tune in sysmon to log more key events. For now some of the fields remain empty. But in general it works.

[AU-Kurt]

Nice, I will have to do more digging. It became a back burner item since I could come up with the data in other ways. From: damyanor ***@***.***> Sent: Wednesday, February 16, 2022 8:47 AM To: BlueTeamLabs/sentinel-attack ***@***.***> Cc: Kurt Clark ***@***.***>; Mention ***@***.***> Subject: Re: [BlueTeamLabs/sentinel-attack] Deploying hunting workbooks error (Issue #50) Yes, I can see the data, however, have to dig more in docs to tune in sysmon to log more key events. For now some of the fields remain empty. But in general it works. - Reply to this email directly, view it on GitHub<https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2FBlueTeamLabs%2Fsentinel-attack%2Fissues%2F50%23issuecomment-1041507969&data=04%7C01%7Ckurt.clark%40air-usa.com%7C1c0b3993851d47e7c06408d9f152bf15%7Ce976109e157e442b8b620e15b979f14f%7C1%7C0%7C637806159974995963%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6NcLzoj5qIaiBNAdeBE95WgtyuKNMF5eqoum6U8HGEY%3D&reserved=0>, or unsubscribe<https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAXIWZCHEP7MZJH2TD2GZPN3U3OTDTANCNFSM5LI4DQGQ&data=04%7C01%7Ckurt.clark%40air-usa.com%7C1c0b3993851d47e7c06408d9f152bf15%7Ce976109e157e442b8b620e15b979f14f%7C1%7C0%7C637806159974995963%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZKY3lVSKDId55VlGi04X1jlHjRoVUoUSayozDnSFDms%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Ckurt.clark%40air-usa.com%7C1c0b3993851d47e7c06408d9f152bf15%7Ce976109e157e442b8b620e15b979f14f%7C1%7C0%7C637806159974995963%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=IzZrBs28Ae%2Fp1wtJ0ST0YijbhaKF6kdlXOwEmWjD4Aw%3D&reserved=0> or Android<https://usg02.safelinks.protection.office365.us/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Ckurt.clark%40air-usa.com%7C1c0b3993851d47e7c06408d9f152bf15%7Ce976109e157e442b8b620e15b979f14f%7C1%7C0%7C637806159974995963%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Hbaq2ca67LEKf2LA2kwMFACPDpISacEl4G38uJ%2F4%2FVY%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>