|
1 | 1 | # AWS Kinesis Data Streams for Dart |
2 | 2 |
|
3 | | -A Dart-only implementation for streaming data to Amazon Kinesis Data Streams with offline support. |
4 | | - |
5 | | -## Features |
6 | | - |
7 | | -- **Offline Support**: Records are persisted locally in SQLite and automatically sent when connectivity is restored |
8 | | -- **Automatic Retry**: Failed records are retried with configurable retry limits |
9 | | -- **Efficient Batching**: Records are batched according to Kinesis service limits (500 records or 5MB per batch) |
10 | | -- **Flexible Credentials**: Compatible with Amplify Auth credential providers via the `AWSCredentialsProvider` interface |
11 | | -- **Configurable Flushing**: Automatic interval-based flushing with manual flush support |
12 | | - |
13 | | -## Installation |
14 | | - |
15 | | -Add the package to your `pubspec.yaml`: |
16 | | - |
17 | | -```yaml |
18 | | -dependencies: |
19 | | - aws_kinesis_datastreams: ^0.1.0 |
20 | | -``` |
21 | | -
|
22 | | -## Usage |
23 | | -
|
24 | | -### Basic Setup |
25 | | -
|
26 | | -```dart |
27 | | -import 'dart:convert'; |
28 | | -import 'dart:typed_data'; |
29 | | - |
30 | | -import 'package:aws_kinesis_datastreams/aws_kinesis_datastreams.dart'; |
31 | | - |
32 | | -// Using static credentials (for testing only) |
33 | | -final kinesis = AmplifyKinesisClient( |
34 | | - region: 'us-east-1', |
35 | | - credentialsProvider: AWSCredentialsProvider( |
36 | | - AWSCredentials( |
37 | | - 'YOUR_ACCESS_KEY', |
38 | | - 'YOUR_SECRET_KEY', |
39 | | - 'YOUR_SESSION_TOKEN', // Optional |
40 | | - DateTime.now().add(Duration(hours: 1)), // Optional expiration |
41 | | - ), |
42 | | - ), |
43 | | -); |
44 | | -``` |
45 | | - |
46 | | -### Using with Amplify Auth |
47 | | - |
48 | | -For production applications, integrate with Amplify Auth for secure credential management: |
49 | | - |
50 | | -```dart |
51 | | -import 'package:amplify_auth_cognito/amplify_auth_cognito.dart'; |
52 | | -import 'package:aws_kinesis_datastreams/aws_kinesis_datastreams.dart'; |
53 | | -
|
54 | | -// Create an adapter for Amplify Auth credentials |
55 | | -class AmplifyCredentialsAdapter implements AWSCredentialsProvider { |
56 | | - final AmplifyAuthCognito _auth; |
57 | | - |
58 | | - AmplifyCredentialsAdapter(this._auth); |
59 | | - |
60 | | - @override |
61 | | - Future<AWSCredentials> retrieve() async { |
62 | | - final session = await _auth.fetchAuthSession(); |
63 | | - final credentials = session.credentialsResult.value; |
64 | | - return AWSCredentials( |
65 | | - credentials.accessKeyId, |
66 | | - credentials.secretAccessKey, |
67 | | - credentials.sessionToken, |
68 | | - credentials.expiration, |
69 | | - ); |
70 | | - } |
71 | | - |
72 | | - @override |
73 | | - String get runtimeTypeName => 'AmplifyCredentialsAdapter'; |
74 | | -} |
75 | | -
|
76 | | -// Initialize with Amplify Auth |
77 | | -final kinesis = AmplifyKinesisClient( |
78 | | - region: 'us-east-1', |
79 | | - credentialsProvider: AmplifyCredentialsAdapter(amplifyAuth), |
80 | | -); |
81 | | -``` |
82 | | - |
83 | | -### Custom Configuration |
84 | | - |
85 | | -```dart |
86 | | -final kinesis = AmplifyKinesisClient( |
87 | | - region: 'us-east-1', |
88 | | - credentialsProvider: myCredentialsProvider, |
89 | | - options: AmplifyKinesisClientOptions( |
90 | | - cacheMaxBytes: 10 * 1024 * 1024, // 10MB cache |
91 | | - maxRetries: 3, |
92 | | - flushStrategy: KinesisDataStreamsInterval( |
93 | | - interval: Duration(seconds: 60), // Flush every 60 seconds |
94 | | - ), |
95 | | - ), |
96 | | -); |
97 | | -``` |
98 | | - |
99 | | -### Recording Data |
100 | | - |
101 | | -```dart |
102 | | -// Record data to a stream |
103 | | -await kinesis.record( |
104 | | - data: Uint8List.fromList(utf8.encode('{"event": "user_action"}')), |
105 | | - partitionKey: 'user-123', |
106 | | - streamName: 'my-kinesis-stream', |
107 | | -); |
108 | | -``` |
109 | | - |
110 | | -### Manual Flush |
111 | | - |
112 | | -```dart |
113 | | -// Manually flush all cached records |
114 | | -await kinesis.flush(); |
115 | | -``` |
116 | | - |
117 | | -### Enable/Disable Recording |
118 | | - |
119 | | -```dart |
120 | | -// Disable recording (records will be silently ignored) |
121 | | -kinesis.disable(); |
122 | | -
|
123 | | -// Enable recording |
124 | | -kinesis.enable(); |
125 | | -``` |
126 | | - |
127 | | -### Clear Cache |
128 | | - |
129 | | -```dart |
130 | | -// Clear all cached records |
131 | | -await kinesis.clearCache(); |
132 | | -``` |
133 | | - |
134 | | -### Cleanup |
135 | | - |
136 | | -```dart |
137 | | -// Close the client when done |
138 | | -await kinesis.close(); |
139 | | -``` |
140 | | - |
141 | | -## Security Considerations |
142 | | - |
143 | | -### Unencrypted Storage |
144 | | - |
145 | | -Records are stored in an **unencrypted SQLite database** on the device. This means: |
146 | | - |
147 | | -- **Do not store sensitive data** (PII, credentials, etc.) without additional encryption |
148 | | -- Consider encrypting data before calling `record()` if it contains sensitive information |
149 | | -- The database file is stored in the application's data directory |
150 | | - |
151 | | -### Credential Handling |
152 | | - |
153 | | -The library handles credentials securely: |
154 | | - |
155 | | -- Credentials are **resolved on-demand** before each API request |
156 | | -- Credentials are **never persisted** to disk |
157 | | -- Credentials are **never logged** (even in debug mode) |
158 | | -- Error messages are **sanitized** to prevent credential exposure |
159 | | - |
160 | | -### Best Practices |
161 | | - |
162 | | -1. **Use temporary credentials**: Prefer IAM roles with temporary credentials over long-lived access keys |
163 | | -2. **Implement credential refresh**: Your `AWSCredentialsProvider` should handle credential expiration |
164 | | -3. **Encrypt sensitive data**: Encrypt any sensitive data before recording |
165 | | -4. **Use HTTPS**: All API calls use HTTPS (enforced by the AWS SDK) |
166 | | - |
167 | | -## API Reference |
168 | | - |
169 | | -### AmplifyKinesisClient |
170 | | - |
171 | | -The main client class for recording and streaming data. |
172 | | - |
173 | | -| Method | Description | |
174 | | -|--------|-------------| |
175 | | -| `record()` | Records data to be sent to a Kinesis stream | |
176 | | -| `flush()` | Manually flushes all cached records | |
177 | | -| `enable()` | Enables the client to accept records | |
178 | | -| `disable()` | Disables the client (records are ignored) | |
179 | | -| `clearCache()` | Removes all cached records | |
180 | | -| `close()` | Closes the client and releases resources | |
181 | | - |
182 | | -### AmplifyKinesisClientOptions |
183 | | - |
184 | | -Configuration options for the client. |
185 | | - |
186 | | -| Property | Default | Description | |
187 | | -|----------|---------|-------------| |
188 | | -| `cacheMaxBytes` | 5MB | Maximum size of the local cache | |
189 | | -| `maxRetries` | 5 | Maximum retry attempts for failed records | |
190 | | -| `flushStrategy` | 30s interval | Strategy for automatic flushing | |
191 | | - |
192 | | -### Exceptions |
193 | | - |
194 | | -| Exception | Description | |
195 | | -|-----------|-------------| |
196 | | -| `AmplifyKinesisException` | Base exception for all errors | |
197 | | -| `CacheFullException` | Thrown when the cache is full | |
198 | | -| `CredentialsException` | Thrown when credentials cannot be resolved | |
199 | | -| `NetworkException` | Thrown when a network error occurs | |
200 | | - |
201 | | -## License |
202 | | - |
203 | | -Apache License 2.0 |
| 3 | +Dart-only implementation for streaming data to Amazon Kinesis Data Streams. For internal use in the `aws_kinesis_datastreams` Flutter library. |
0 commit comments