Skip to content

Commit cd31ed2

Browse files
lordgamezlordgamez
committed
MINIFICPP-2644 Add proxy controller service support for AWS and Azure processors
1 parent 1976d80 commit cd31ed2

36 files changed

+824
-47
lines changed

‎CONTROLLERS.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ limitations under the License.
2727
- [NetworkPrioritizerService](#NetworkPrioritizerService)
2828
- [ODBCService](#ODBCService)
2929
- [PersistentMapStateStorage](#PersistentMapStateStorage)
30+
- [ProxyConfigurationService](#ProxyConfigurationService)
3031
- [RocksDbStateStorage](#RocksDbStateStorage)
3132
- [SmbConnectionControllerService](#SmbConnectionControllerService)
3233
- [SSLContextService](#SSLContextService)
@@ -244,6 +245,24 @@ In the list below, the names of required properties appear in bold. Any other pr
244245
| **File** | | | Path to a file to store state |
245246

246247

248+
## ProxyConfigurationService
249+
250+
### Description
251+
252+
Provides a set of configurations for different MiNiFi C++ components to use a proxy server.
253+
254+
### Properties
255+
256+
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
257+
258+
| Name | Default Value | Allowable Values | Description |
259+
|-----------------------|---------------|------------------|--------------------------------------------------------------------------------------------|
260+
| **Proxy Server Host** | | | Proxy server hostname or ip-address. |
261+
| Proxy Server Port | | | Proxy server port number. |
262+
| Proxy User Name | | | The name of the proxy client for user authentication. |
263+
| Proxy User Password | | | The password of the proxy client for user authentication.<br/>**Sensitive Property: true** |
264+
265+
247266
## RocksDbStateStorage
248267

249268
### Description

‎PROCESSORS.md‎

Lines changed: 16 additions & 4 deletions
Large diffs are not rendered by default.

‎docker/test/integration/features/azure_storage.feature‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,3 +109,84 @@ Feature: Sending data from MiNiFi-C++ to an Azure storage server
109109
Then the Minifi logs contain the following message: "key:azure.blobname value:test_1" in less than 60 seconds
110110
Then the Minifi logs contain the following message: "key:azure.blobname value:test_2" in less than 60 seconds
111111
And the Minifi logs do not contain the following message: "key:azure.blobname value:other_test" after 0 seconds
112+
113+
Scenario: A MiNiFi instance can upload data to Azure blob storage through a proxy
114+
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
115+
And a file with the content "#test_data$123$#" is present in "/tmp/input"
116+
And a PutAzureBlobStorage processor set up to communicate with an Azure blob storage
117+
And the "Proxy Configuration Service" property of the PutAzureBlobStorage processor is set to "ProxyConfigurationService"
118+
And a PutFile processor with the "Directory" property set to "/tmp/output"
119+
And the "success" relationship of the GetFile processor is connected to the PutAzureBlobStorage
120+
And the "success" relationship of the PutAzureBlobStorage processor is connected to the PutFile
121+
And the "failure" relationship of the PutAzureBlobStorage processor is connected to the PutAzureBlobStorage
122+
And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
123+
124+
And an Azure storage server is set up
125+
And the http proxy server is set up
126+
127+
When all instances start up
128+
129+
Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
130+
And the object on the Azure storage server is "#test_data$123$#"
131+
And no errors were generated on the http-proxy regarding "http://azure-storage-server-${feature_id}:10000/devstoreaccount1/test-container/test-blob"
132+
133+
Scenario: A MiNiFi instance can delete blob from Azure blob storage through a proxy
134+
Given a GenerateFlowFile processor with the "File Size" property set to "0B"
135+
And a DeleteAzureBlobStorage processor set up to communicate with an Azure blob storage
136+
And the "Blob" property of the DeleteAzureBlobStorage processor is set to "test"
137+
And the "Proxy Configuration Service" property of the DeleteAzureBlobStorage processor is set to "ProxyConfigurationService"
138+
And the "success" relationship of the GenerateFlowFile processor is connected to the DeleteAzureBlobStorage
139+
And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
140+
141+
And an Azure storage server is set up
142+
And the http proxy server is set up
143+
144+
When all instances start up
145+
And test blob "test" is created on Azure blob storage
146+
147+
Then the Azure blob storage becomes empty in 30 seconds
148+
And no errors were generated on the http-proxy regarding "http://azure-storage-server-${feature_id}:10000/devstoreaccount1/test-container/test"
149+
150+
Scenario: A MiNiFi instance can fetch a blob from Azure blob storage through a proxy
151+
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
152+
And the "Keep Source File" property of the GetFile processor is set to "true"
153+
And a file with the content "dummy" is present in "/tmp/input"
154+
And a FetchAzureBlobStorage processor set up to communicate with an Azure blob storage
155+
And the "Blob" property of the FetchAzureBlobStorage processor is set to "test"
156+
And the "Range Start" property of the FetchAzureBlobStorage processor is set to "6"
157+
And the "Range Length" property of the FetchAzureBlobStorage processor is set to "5"
158+
And the "Proxy Configuration Service" property of the FetchAzureBlobStorage processor is set to "ProxyConfigurationService"
159+
And a PutFile processor with the "Directory" property set to "/tmp/output"
160+
And the "success" relationship of the GetFile processor is connected to the FetchAzureBlobStorage
161+
And the "success" relationship of the FetchAzureBlobStorage processor is connected to the PutFile
162+
And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
163+
164+
And an Azure storage server is set up
165+
And the http proxy server is set up
166+
167+
When all instances start up
168+
And test blob "test" with the content "#test_data$123$#" is created on Azure blob storage
169+
170+
Then a flowfile with the content "data$" is placed in the monitored directory in less than 60 seconds
171+
And no errors were generated on the http-proxy regarding "http://azure-storage-server-${feature_id}:10000/devstoreaccount1/test-container/test"
172+
173+
Scenario: A MiNiFi instance can list a container on Azure blob storage through a proxy
174+
Given a ListAzureBlobStorage processor set up to communicate with an Azure blob storage
175+
And the "Prefix" property of the ListAzureBlobStorage processor is set to "test"
176+
And the "Proxy Configuration Service" property of the ListAzureBlobStorage processor is set to "ProxyConfigurationService"
177+
And a LogAttribute processor with the "FlowFiles To Log" property set to "0"
178+
And the "success" relationship of the ListAzureBlobStorage processor is connected to the LogAttribute
179+
And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
180+
181+
And an Azure storage server is set up
182+
And the http proxy server is set up
183+
184+
When all instances start up
185+
And test blob "test_1" with the content "data_1" is created on Azure blob storage
186+
And test blob "test_2" with the content "data_2" is created on Azure blob storage
187+
And test blob "other_test" with the content "data_3" is created on Azure blob storage
188+
189+
Then the Minifi logs contain the following message: "key:azure.blobname value:test_1" in less than 60 seconds
190+
And the Minifi logs contain the following message: "key:azure.blobname value:test_2" in less than 60 seconds
191+
And the Minifi logs do not contain the following message: "key:azure.blobname value:other_test" after 0 seconds
192+
And no errors were generated on the http-proxy regarding "http://azure-storage-server-${feature_id}:10000/devstoreaccount1/test-container"

‎docker/test/integration/features/s3.feature‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,25 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
8080
And the object content type on the s3 server is "application/octet-stream" and the object metadata matches use metadata
8181
And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket/test_object_key"
8282

83+
Scenario: A MiNiFi instance transfers encoded data through a http proxy to s3 using proxy configuration service
84+
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
85+
And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
86+
And a PutS3Object processor set up to communicate with an s3 server
87+
And the "Proxy Configuration Service" property of the PutS3Object processor is set to "ProxyConfigurationService"
88+
And a PutFile processor with the "Directory" property set to "/tmp/output"
89+
And the "success" relationship of the GetFile processor is connected to the PutS3Object
90+
And the "success" relationship of the PutS3Object processor is connected to the PutFile
91+
And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
92+
93+
And a s3 server is set up in correspondence with the PutS3Object
94+
And the http proxy server is set up
95+
When all instances start up
96+
97+
Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
98+
And the object on the s3 server is "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
99+
And the object content type on the s3 server is "application/octet-stream" and the object metadata matches use metadata
100+
And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket/test_object_key"
101+
83102
Scenario: A MiNiFi instance can remove s3 bucket objects
84103
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
85104
And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
@@ -141,6 +160,29 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
141160
And the object bucket on the s3 server is empty
142161
And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket/test_object_key"
143162

163+
Scenario: Deletion of a s3 object through a proxy-server succeeds using proxy configuration service
164+
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
165+
And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
166+
And a PutS3Object processor set up to communicate with an s3 server
167+
And a DeleteS3Object processor set up to communicate with the same s3 server
168+
And the "Proxy Configuration Service" property of the DeleteS3Object processor is set to "ProxyConfigurationService"
169+
And a PutFile processor with the "Directory" property set to "/tmp/output"
170+
And the processors are connected up as described here:
171+
| source name | relationship name | destination name |
172+
| GetFile | success | PutS3Object |
173+
| PutS3Object | success | DeleteS3Object |
174+
| DeleteS3Object | success | PutFile |
175+
And a ProxyConfigurationService controller service is set up with HTTP proxy configuration
176+
177+
And a s3 server is set up in correspondence with the PutS3Object
178+
And the http proxy server is set up
179+
180+
When all instances start up
181+
182+
Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
183+
And the object bucket on the s3 server is empty
184+
And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket/test_object_key"
185+
144186
Scenario: A MiNiFi instance can download s3 bucket objects directly
145187
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
146188
And a file with the content "test" is present in "/tmp/input"
@@ -189,6 +231,30 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
189231
Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
190232
And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket/test_object_key"
191233

234+
Scenario: A MiNiFi instance can download s3 bucket objects via a http-proxy using proxy configuration service
235+
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
236+
And a file with the content "test" is present in "/tmp/input"
237+
And a PutS3Object processor set up to communicate with an s3 server
238+
And the "success" relationship of the GetFile processor is connected to the PutS3Object
239+
240+
Given a GenerateFlowFile processor with the "File Size" property set to "1 kB" in a "secondary" flow
241+
And a FetchS3Object processor set up to communicate with the same s3 server
242+
And the "Proxy Configuration Service" property of the FetchS3Object processor is set to "ProxyConfigurationService"
243+
And a PutFile processor with the "Directory" property set to "/tmp/output"
244+
And the processors are connected up as described here:
245+
| source name | relationship name | destination name |
246+
| GenerateFlowFile | success | FetchS3Object |
247+
| FetchS3Object | success | PutFile |
248+
And a ProxyConfigurationService controller service is set up with HTTP proxy configuration in the "secondary" flow
249+
250+
And a s3 server is set up in correspondence with the PutS3Object
251+
And a http proxy server is set up accordingly
252+
253+
When all instances start up
254+
255+
Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
256+
And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket/test_object_key"
257+
192258
Scenario: A MiNiFi instance can list an S3 bucket directly
193259
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
194260
And the "Batch Size" property of the GetFile processor is set to "1"
@@ -233,6 +299,26 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
233299
Then 1 flowfile is placed in the monitored directory in 120 seconds
234300
And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket"
235301

302+
Scenario: A MiNiFi instance can list an S3 bucket objects via a http-proxy using proxy configuration service
303+
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
304+
And a file with the content "test" is present in "/tmp/input"
305+
And a PutS3Object processor set up to communicate with an s3 server
306+
And the "success" relationship of the GetFile processor is connected to the PutS3Object
307+
308+
Given a ListS3 processor in the "secondary" flow
309+
And the "Proxy Configuration Service" property of the ListS3 processor is set to "ProxyConfigurationService"
310+
And a PutFile processor with the "Directory" property set to "/tmp/output"
311+
And the "success" relationship of the ListS3 processor is connected to the PutFile
312+
And a ProxyConfigurationService controller service is set up with HTTP proxy configuration in the "secondary" flow
313+
314+
And a s3 server is set up in correspondence with the PutS3Object
315+
And a http proxy server is set up accordingly
316+
317+
When all instances start up
318+
319+
Then 1 flowfile is placed in the monitored directory in 120 seconds
320+
And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket"
321+
236322
Scenario: A MiNiFi instance transfers data in multiple parts to s3
237323
Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
238324
And a file of size 16MB is present in "/tmp/input"

‎docker/test/integration/features/steps/steps.py‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
from minifi.controllers.JsonTreeReader import JsonTreeReader
2828
from minifi.controllers.CouchbaseClusterService import CouchbaseClusterService
2929
from minifi.controllers.XMLReader import XMLReader
30+
from minifi.controllers.ProxyConfigurationService import ProxyConfigurationService
3031

3132
from behave import given, then, when
3233
from behave.model_describe import ModelDescriptor
@@ -405,6 +406,18 @@ def step_impl(context):
405406
context.test.acquire_container(context=context, name="http-proxy", engine="http-proxy")
406407

407408

409+
@given("a ProxyConfigurationService controller service is set up with HTTP proxy configuration in the \"{container_name}\" flow")
410+
def step_impl(context, container_name):
411+
proxy_service = ProxyConfigurationService("ProxyConfigurationService", host=f"http-proxy-{context.feature_id}", port=3128, username="admin", password="test101")
412+
container = context.test.acquire_container(context=context, name=container_name)
413+
container.add_controller(proxy_service)
414+
415+
416+
@given("a ProxyConfigurationService controller service is set up with HTTP proxy configuration")
417+
def step_impl(context):
418+
context.execute_steps("given a ProxyConfigurationService controller service is set up with HTTP proxy configuration in the \"{container_name}\" flow".format(container_name="minifi-cpp-flow"))
419+
420+
408421
# TLS
409422
@given("an ssl context service is set up for {processor_name}")
410423
@given("an ssl context service with a manual CA cert file is set up for {processor_name}")

‎docker/test/integration/minifi/controllers/ProxyConfigurationService.py‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
from ..core.ControllerService import ControllerService
16+
17+
18+
class ProxyConfigurationService(ControllerService):
19+
def __init__(self, name, host, port=None, username=None, password=None):
20+
super(ProxyConfigurationService, self).__init__(name=name)
21+
22+
self.service_class = 'ProxyConfigurationService'
23+
24+
self.properties['Proxy Server Host'] = host
25+
26+
if port is not None:
27+
self.properties['Proxy Server Port'] = port
28+
29+
if username is not None:
30+
self.properties['Proxy User Name'] = username
31+
32+
if password is not None:
33+
self.properties['Proxy User Password'] = password

0 commit comments

Comments
 (0)