Skip to content

Commit be429d9

Browse files
committed
update readme and sample meltano.yml
1 parent fa62751 commit be429d9

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ Built with the [Meltano Singer SDK](https://sdk.meltano.com).
2626
| start_date | True | None | The earliest record date to sync |
2727
| log_group_name | True | None | The log group on which to perform the query. |
2828
| query | True | None | The query string to use. For more information, see [CloudWatch Logs Insights Query Syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html). |
29-
| batch_increment_mins | False | 1440 | The size of the time window to query by, default 1440 mins (i.e. 1 day). The tap will raise an exception if the result set is greater than the max limit of 10,000 records because it can't be sure that all data is extracted. If this happens you'll need to reduce this batch_increment_mins setting to retrieve a smaller record set per query. |
29+
| batch_increment_s | False | 3600 | The size of the time window to query by, default 3,600 seconds (i.e. 1 hour). If the result set for a batch is greater than the max limit of 10,000 records then the tap will query the same window again where >= the most recent record received. This means that the same data is potentially being scanned >1 times but < 2 times, depending on the amount the results set went over the 10k max. For example a batch window with 15k records would scan the 15k once, receiving 10k results, then scan ~5k again to get the rest. The net result is the same data was scanned ~1.5 times for that batch. To avoid this you should set the batch window to avoid exceeding the 10k limit. |
3030
| stream_maps | False | None | Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html). |
3131
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
3232
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |

meltano.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ plugins:
1616
- about
1717
- stream-maps
1818
config:
19-
start_date: '2010-01-01T00:00:00Z'
19+
start_date: '2023-01-01T00:00:00Z'
2020
settings:
2121
- name: aws_access_key_id
2222
kind: password

0 commit comments

Comments
 (0)