1.4.0 (#7)

* add facet, improve logging

* version

* Add facet and update examples

* add facet

* version

* update readme

Author: mike-mcgrail

README.md

@@ -29,7 +29,7 @@ Reference Splunk documentation for [installing add-ons](https://docs.splunk.com/
 
 ![Creating DataSet API keys](README_images/dataset_key.png)
 
-2. Click Add Key > Add Read Key (required for inputs and search command).
+2. Click Add Key > Add Read Key (required for search command and inputs).
 3. Click Add Key > Add Write Key (required for alert action).
 4. Optionally, click the pencil icon to rename the keys.
 
@@ -40,65 +40,31 @@ Reference Splunk documentation for [installing add-ons](https://docs.splunk.com/
 
 2. In configuration on DataSet Account tab:
 - Enter the DataSet URL (e.g.: `https://app.scalyr.com`).
-- Enter the DataSet read key.
-- Enter the DataSet write key.
+- Enter the DataSet read key from above.
+- Enter the DataSet write key from above.
 
 3. Optionally, configure logging level and proxy information on the associated tabs.
 4. Click Save.
-
-5. On the inputs page, click Create New Input and select the desired input
-
-6. For DataSet alerts, enter:
-
-![Setup alerts indexing](README_images/setup_alerts.png)
-- A name for the input.
-- Interval, in seconds. A good starting point is `300` seconds to collect every five mintues.
-- Splunk index name
-- Start time, in relative shorthand form, e.g.: `24h` for 24 hours before input execution time.
-
-7. For DataSet queries, enter:
-
-![Setup query indexing](README_images/setup_query.png)
-- A name for the input.
-- Interval, in seconds. A good starting point is `300` seconds to collect every five mintues.
-- Splunk index name
-- Start time, in relative shorthand form, e.g.: `24h` for 24 hours before input execution time.
-- *(optional)* End time, in relative shorthand form, e.g.: `5m` for 5 minutes before input execution time.
-- *(optional)* Query string used to return matching events.
-- *(optional)* Maximum number of events to return.
-
-8. For DataSet Power Queries, enter:
-- A name for the input.
-- Interval, in seconds. A good starting point is `300` seconds to collect every five mintues.
-- Splunk index name
-- Start time, in relative shorthand form, e.g.: `24h` for 24 hours before input execution time.
-- *(optional)* End time, in relative shorthand form, e.g.: `5m` for 5 minutes before input execution time.
-- Query string used to return matching events, including commands such as `| columns`, `| limit`, etc.
-
-## Usage
-
-### Inputs
-The DataSet Add-on for Splunk collects the following inputs utilizing time-based checkpointing to prevent reindexing the same data:
-
-| Source Type | Description | CIM Data Model |
-| ------ | ------ | ------ |
-| dataset:alerts | Predefined Power Query API call to index [alert state change records](https://app.scalyr.com/help/alerts#logging)  | [Alerts](https://docs.splunk.com/Documentation/CIM/latest/User/Alerts) |
-| dataset:query | User-defined standard [query](https://app.scalyr.com/help/api#query) API call to index events | - |
-| dataset:powerquery | User-defined [PowerQuery](https://app.scalyr.com/help/api#powerquery) API call to index events | - |
+5. To confirm connectivity, simply search `|dataset` and validate results.
 
 ## SPL Command
-The `| dataset` command allows queries against the DataSet API directly from Splunk's search bar. Optional parameters are supported:
+The `| dataset` command allows queries against the DataSet API directly from Splunk's search bar. 
 
-- **method** - Define `query`, `powerquery` or `timeseries` to call the appropriate REST endpoint. Default is query.
-- **query** - The DataSet [query](https://app.scalyr.com/help/query-language) or filter used to select events. Default is no filter (return all events limited by maxCount).
+Optional parameters are supported:
+
+- **method** - Define `query`, `powerquery`, `facet` or `timeseries` to call the appropriate REST endpoint. Default is query.
+- **query** - The DataSet [query](https://app.scalyr.com/help/query-language) or filter used to select events. Default is no filter (return all events limited by time and maxCount).
 - **starttime** - The Splunk time picker can be used (not "All Time"), but if starttime is defined it will take precedence to define the [start time](https://app.scalyr.com/help/time-reference) for DataSet events to return. Use epoch time or relative shorthand in the form of a number followed by d, h, m or s (for days, hours, minutes or seconds), e.g.: `24h`. Default is 24h.
 - **endtime** - The Splunk time picker can be used (not "All Time"), but if endtime is defined it will take precedence to define the [end time](https://app.scalyr.com/help/time-reference) for DataSet events to return. Use epoch time or relative shorthand in the form of a number followed by d, h, m or s (for days, hours, minutes or seconds), e.g.: `5m`. Default is current time at search.
 
-For query and powerquery, additional parameters include:
+For query and powerquery:
 - **maxcount** - Number of events to return.
 - **columns** - Specified fields to return from DataSet query (or powerquery, analogous to using `| columns` in a powerquery). Yields performance gains for high volume queries instead of returning and merging all fields.
 
-For timeseries, additional parameters include:
+For facet:
+- **field** - Define field to get most frequent values of. Default is logfile.
+
+For timeseries:
 - **function** - Define value to compute from matching events. Default is rate.
 - **buckets** - The number of numeric values to return by dividing time range into equal slices. Default is 1.
 - **createsummaries** - Specify whether to create summaries to automatically update on ingestion pipeline. Default is true; recommend setting to false for one-off or while testing new queries.
@@ -123,24 +89,71 @@ Power Query Example 2: `| dataset method=powerQuery search="$serverHost == 'clou
 | sort -DUR 
 | columns 'Request ID' = RID, 'Duration(ms)' = DUR, 'Charged delta (ms)' = deltaDUR, 'Used Memory (MB)' = UMEM, 'Charged delta Memory (MB)' = deltaMEM" starttime=5m`
 
-Since events are returned in JSON format, the Splunk [spath command](https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Spath) is useful. Additionally, the Splunk [collect command](https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/collect) can be used to add the events to a summary index:
-
+Facet Query Example:
 `
-| dataset query="serverHost = * AND Action = 'allow'" maxcount=50 starttime=10m endtime=1m
+| dataset method=facet search="serverHost = *" field=serverHost maxcount=25
 | spath
-| collect index=dataset
+| table value, count
 `
 
 Timeseries Query Example:
 `
 | dataset method=timeseries search="serverHost='scalyr-metalog'" function="p90(delayMedian)" starttime="24h" buckets=24 createsummaries=false onlyusesummaries=false
 `
 
+Since events are returned in JSON format, the Splunk [spath command](https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Spath) is useful. Additionally, the Splunk [collect command](https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/collect) can be used to add the events to a summary index:
+
+`
+| dataset query="serverHost = * AND Action = 'allow'" maxcount=50 starttime=10m endtime=1m
+| spath
+| collect index=dataset
+`
+
+## Inputs
+For use cases requiring data indexed in Splunk, optional inputs are provided utilizing time-based checkpointing to prevent reindexing the same data:
+
+| Source Type | Description | CIM Data Model |
+| ------ | ------ | ------ |
+| dataset:alerts | Predefined Power Query API call to index [alert state change records](https://app.scalyr.com/help/alerts#logging)  | [Alerts](https://docs.splunk.com/Documentation/CIM/latest/User/Alerts) |
+| dataset:query | User-defined standard [query](https://app.scalyr.com/help/api#query) API call to index events | - |
+| dataset:powerquery | User-defined [PowerQuery](https://app.scalyr.com/help/api#powerquery) API call to index events | - |
+
+1. On the inputs page, click Create New Input and select the desired input
+
+2. For DataSet alerts, enter:
+
+![Setup alerts indexing](README_images/setup_alerts.png)
+- A name for the input.
+- Interval, in seconds. A good starting point is `300` seconds to collect every five mintues.
+- Splunk index name
+- Start time, in relative shorthand form, e.g.: `24h` for 24 hours before input execution time.
+
+3. For DataSet queries, enter:
+
+![Setup query indexing](README_images/setup_query.png)
+- A name for the input.
+- Interval, in seconds. A good starting point is `300` seconds to collect every five mintues.
+- Splunk index name
+- Start time, in relative shorthand form, e.g.: `24h` for 24 hours before input execution time.
+- *(optional)* End time, in relative shorthand form, e.g.: `5m` for 5 minutes before input execution time.
+- *(optional)* Query string used to return matching events.
+- *(optional)* Maximum number of events to return.
+
+4. For DataSet Power Queries, enter:
+- A name for the input.
+- Interval, in seconds. A good starting point is `300` seconds to collect every five mintues.
+- Splunk index name
+- Start time, in relative shorthand form, e.g.: `24h` for 24 hours before input execution time.
+- *(optional)* End time, in relative shorthand form, e.g.: `5m` for 5 minutes before input execution time.
+- Query string used to return matching events, including commands such as `| columns`, `| limit`, etc.
+
 ## Alert Action
 An alert action allows sending an event to the DataSet [addEvents API](https://app.scalyr.com/help/api#addEvents). 
 
 ## Support
-For support, please open an issue on GitHub.
+To troubleshooting the custom command, check the Job Inspector search log, also available in the internal index: `index=_internal app="TA-dataset" sourcetype=splunk_search_messages`. Common issues include incorrect API key or firewalls blocking outbound traffic on port 443.
+
+For support, open a ticket with support, or open a GitHub issue.
 
 ##### Note
 This add-on was built with the [Splunk Add-on UCC framework](https://splunk.github.io/addonfactory-ucc-generator/).

globalConfig.json

@@ -2,7 +2,7 @@
     "meta": {
         "name": "TA-dataset",
         "displayName": "DataSet Add-on for Splunk",
-        "version": "1.3.0",
+        "version": "1.4.0",
         "restRoot": "TA_dataset",
         "schemaVersion": "0.0.3"
     },

output/TA-dataset-1.3.0.tar.gz

@@ -5,7 +5,7 @@
     "id": {
       "group": null,
       "name": "TA-dataset",
-      "version": "1.3.0"
+      "version": "1.4.0"
     },
     "author": [
       {