Skip to content

Commit 047338b

Browse files
mmacatammacata
authored
Merge pull request #16 from mmacata/dev-setup-and-tests
Setup fixes
2 parents d4d2dd8 + 4c271f7 commit 047338b

File tree

14 files changed

+493
-107
lines changed

14 files changed

+493
-107
lines changed

‎.github/workflows/test.yml‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -37,14 +37,16 @@ jobs:
3737
steps:
3838
- uses: actions/checkout@v6
3939
- name: Start containers
40-
run: docker compose -f "docker/docker-compose.yml" up -d --build
40+
run: docker compose -f docker/docker-compose.yml up -d --build
4141
- name: List running docker
4242
run: docker ps
43-
- name: Docker logs actinia-cloudevent
44-
run: docker logs docker-actinia-cloudevent-1
45-
- name: Docker logs actinia-core
46-
run: docker logs docker-actinia-core-1
43+
- name: Docker logs
44+
run: |
45+
sleep 10 && \
46+
docker logs docker-actinia-cloudevent-1 && echo && \
47+
docker logs docker-actinia-core-1 && echo && \
48+
docker logs docker-event-receiver-server-1
4749
- name: Run integration test
4850
run: docker exec -t docker-actinia-cloudevent-1 make integrationtest
4951
- name: Stop containers
50-
run: docker compose -f "docker/docker-compose.yml" down
52+
run: docker compose -f docker/docker-compose.yml down

‎.vscode/tasks.json‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,18 @@
1111
"context": "${workspaceFolder}"
1212
}
1313
},
14+
{
15+
"label": "start-cloudevent-receiver-dummy-server",
16+
"type": "shell",
17+
"command": "docker compose -f docker/docker-compose-vscode.yml up -d --build --force-recreate event-receiver-server",
18+
"isBackground": true
19+
},
1420
{
1521
"type": "docker-run",
1622
"label": "docker-run: debug",
1723
"dependsOn": [
18-
"docker-build"
24+
"docker-build",
25+
"start-cloudevent-receiver-dummy-server"
1926
],
2027
"python": {
2128
"module": "flask",
@@ -29,6 +36,7 @@
2936
]
3037
},
3138
"dockerRun": {
39+
"containerName": "actinia-cloudevent",
3240
"remove": true,
3341
"network": "actinia-docker_actinia-dev",
3442
"ports": [

‎README.md‎

Lines changed: 131 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -1,108 +1,171 @@
11
# actinia-cloudevent-plugin
22

3-
This is a plugin for [actinia-core](https://github.com/mundialis/actinia_core) which adds cloudevent endpoints and runs as standalone app.
3+
This is a plugin for [actinia-core](https://github.com/mundialis/actinia_core) which translates cloudevents into a process definition
4+
for actinia-core and runs as standalone app.
45

56
## Installation and Setup
67

78
Use docker-compose for installation:
89
```bash
910
docker compose -f docker/docker-compose.yml build
10-
docker compose -f docker/docker-compose.yml run --rm --service-ports --entrypoint sh actinia-cloudevent
11-
# within docker
12-
gunicorn -b 0.0.0.0:5000 -w 8 --access-logfile=- -k gthread actinia_cloudevent_plugin.main:flask_app
11+
docker compose -f docker/docker-compose.yml up -d
1312
```
1413

15-
### DEV setup
16-
```bash
14+
### DEV setup with vscode
15+
1. Start actinia-core locally via local dev-setup as described in the
16+
[actinia-docker repository](https://github.com/actinia-org/actinia-docker#local-dev-setup-for-actinia-core-plugins-with-vscode)
17+
2. Have this repository open locally with vscode and press `F5`.
18+
After a few seconds, a browser window should be openend, pointing to
19+
an endpoint (showing 405 Method Not Allowed as this plugin has only
20+
HTTP POST endpoints so far).
21+
22+
Alternatively, configure actinia-core to run jobs via actinia-worker.
23+
In the actinia-core dev-setup, use the `per_job` queue in
24+
`actinia-docker/actinia-dev/actinia.cfg`:
25+
```ini
26+
[QUEUE]
27+
# queue_type = local
28+
29+
[QUEUE]
30+
queue_type = per_job
31+
kvdb_queue_server_url = valkey
32+
kvdb_queue_server_password = pass
33+
worker_queue_prefix = job_queue
34+
```
35+
And restart the vscode debugger. This way, a job is registered in the
36+
valkey DB, but not directly started.
37+
38+
39+
<!-- TODO: check update docker-compose DEV setup if needed -->
40+
<!-- ```bash
1741
# Uncomment the volume mount of the cloud-event-plugin within docker/docker-compose.yml,
1842
# then:
1943
docker compose -f docker/docker-compose.yml build
2044
docker compose -f docker/docker-compose.yml run --rm --service-ports --entrypoint sh actinia-cloudevent
21-
# within docker:
45+
# within docker
2246
# install the plugin
2347
pip3 install .
2448
# start flask app with actinia-cloudevent-plugin
49+
gunicorn -b 0.0.0.0:5000 -w 8 --access-logfile=- -k gthread actinia_cloudevent_plugin.main:flask_app
50+
# or directly via python
2551
python3 -m actinia_cloudevent_plugin.main
26-
```
27-
28-
### Installation hints
29-
* If you get an error like: `ERROR: for docker_kvdb_1 Cannot start service valkey: network xxx not found` you can try the following:
30-
```bash
31-
docker compose -f docker/docker-compose-dev.yml down
32-
# remove all custom networks not used by a container
33-
docker network prune
34-
docker compose -f docker/docker-compose-dev.yml up -d
35-
```
52+
``` -->
3653

3754
## Configuration
3855

39-
- the URL of the cloudevent receiver is defined within [config/mount/sample.ini](config/mount/sample.ini): `[EVENTRECEIVER]` (Default value defined within [src/actinia_cloudevent_plugin/resources/config.py](src/actinia_cloudevent_plugin/resources/config.py))
56+
- the URL of actinia-core and the cloudevent receiver is defined
57+
within [config/mount/sample.ini](config/mount/sample.ini): `
58+
[EVENTRECEIVER]` (Default value defined within
59+
[src/actinia_cloudevent_plugin/resources/config.py](src/actinia_cloudevent_plugin/resources/config.py))
4060

4161
## Requesting endpoint
42-
4362
**Note**: Assuming cloudevent-plugin is running as described in previous setup.
4463

4564
You can test the plugin and request the `/` endpoint, e.g. with:
4665
```bash
47-
# Start server for receiving of cloudevents (which are returned as response)
48-
# NOTE: as defined within config/mount/sample.ini: [EVENTRECEIVER]
49-
python3 tests/cloudevent_receiver_server.py
50-
51-
# In another terminal
5266
JSON=tests/examples/cloudevent_example.json
53-
curl -X POST -H 'Content-Type: application/json' --data @$JSON localhost:5000/api/v1/ | jq
67+
curl -X POST -H 'Content-Type: application/json' --data @$JSON localhost:3003/
68+
```
69+
Or test with `per_job` queue
70+
```bash
71+
JSON=tests/examples/cloudevent_example.json
72+
curl -X POST -H 'Content-Type: application/json' --data @$JSON localhost:3003/
73+
# Get the actinia job queue name from the response
74+
QUEUE_NAME=job_queue_resource_id-d4d9be86-5938-47ff-9c6d-7c79964862c0
75+
76+
docker run --rm --network actinia-docker_actinia-dev \
77+
-v $HOME/actinia/grassdb:/actinia_core/grassdb \
78+
-v $HOME/actinia/grassdb_user:/actinia_core/userdata \
79+
--entrypoint actinia-worker actiniacore -q $QUEUE_NAME
5480
```
5581

5682
Exemplary returned cloudevent: [tests/examples/cloudevent_example_return.json](tests/examples/cloudevent_example_return.json)
5783

58-
## Hints
59-
60-
* If you have no `.git` folder in the plugin folder, you need to set the
61-
`SETUPTOOLS_SCM_PRETEND_VERSION` before installing the plugin:
62-
```bash
63-
export SETUPTOOLS_SCM_PRETEND_VERSION=0.0
64-
```
65-
Otherwise you will get an error like this `LookupError: setuptools-scm was unable to detect version for '/src/actinia-cloudevent-plugin'.`.
66-
67-
* If you make changes in code and nothing changes you can try to uninstall the plugin:
68-
```bash
69-
pip3 uninstall actinia-cloudevent-plugin.wsgi -y
70-
rm -rf /usr/lib/python3.8/site-packages/actinia_cloudevent_plugin.wsgi-*.egg
71-
```
7284

7385
## Running tests
74-
You can run the tests in the actinia test docker:
86+
You can run the tests in the actinia test docker.
87+
These are the same steps which the github workflow is executing.
7588

7689
```bash
77-
# Uncomment the volume mount of the cloud-event-plugin within docker/docker-compose.yml,
78-
# then:
79-
docker compose -f docker/docker-compose.yml build
80-
docker compose -f docker/docker-compose.yml run --rm --service-ports --entrypoint sh actinia-cloudevent
81-
82-
# run all tests
83-
make test
84-
85-
# # run only unittests
86-
# make unittest
87-
88-
# run only integrationtests
89-
make integrationtest
90-
91-
# run only tests which are marked for development with the decorator '@pytest.mark.dev'
92-
make devtest
90+
docker compose -f docker/docker-compose.yml up -d --build
91+
sleep 10 && \
92+
docker logs docker-actinia-cloudevent-1 && echo && \
93+
docker logs docker-actinia-core-1 && echo && \
94+
docker logs docker-event-receiver-server-1
95+
docker exec -t docker-actinia-cloudevent-1 make integrationtest
96+
docker compose -f docker/docker-compose.yml down
9397
```
9498

95-
## Hint for the development of actinia plugins
99+
---
100+
101+
## Possible setup
102+
103+
```mermaid
104+
graph
105+
106+
1[Event Emitter, e.g. UI]
107+
2[Kafka Queue Broker]
108+
subgraph actinia
109+
direction TB
110+
3[actinia-cloudevent-plugin Transformer Service]
111+
%%3@{ shape: braces, label: "Transformer Service" }
112+
4[actinia worker JobSink]
113+
%%4@{ shape: braces, label: "JobSink" }
114+
5[actinia-core]
115+
%%5@{ shape: braces, label: " write job to valkey" }
116+
6[valkey]
117+
end
118+
119+
1 -- Cloudevent --> 2
120+
2 <-- Cloudevent --> 3
121+
3 -- 1: func,RDC --> 5
122+
5 -- 2: func,RDC --> 6
123+
3 -- 3: Cloudevent (queue Name) per HTTP --> 4
124+
4 -- 4: func,RDC --> 6
125+
4 -- 5: webhook --> 3
126+
3 -- Cloudevent (status) --> 2
96127
97-
### skip permission check
98-
The parameter [`skip_permission_check`](https://github.com/mundialis/actinia_core/blob/main/src/actinia_core/processing/actinia_processing/ephemeral_processing.py#L1420-L1422) (see [example in actinia-statistic plugin](https://github.com/mundialis/actinia_statistic_plugin/blob/master/src/actinia_statistic_plugin/vector_sampling.py#L207))
99-
should only be set to `True` if you are sure that you really don't want to check the permissions.
100-
101-
The skip of the permission check leads to a skipping of:
102-
* [the module check](https://github.com/mundialis/actinia_core/blob/main/src/actinia_core/processing/actinia_processing/ephemeral_processing.py#L579-L589)
103-
* [the limit of the number of processes](https://github.com/mundialis/actinia_core/blob/main/src/actinia_core/processing/actinia_processing/ephemeral_processing.py#L566-L570)
104-
* the limit of the processing time
128+
```
129+
---
130+
131+
<!-- <script>
132+
mermaid.initialize({ sequence: { showSequenceNumbers: true } });
133+
</script> -->
134+
135+
```mermaid
136+
sequenceDiagram
137+
autonumber
138+
create participant event receiver
139+
create actor event sender
140+
create participant A as actinia-cloudevent-plugin<br/>Transformer Service
141+
%% Note right of A: Transformer Service #9829;
142+
event sender-->>A: Cloudevent
143+
create participant D as actinia-core
144+
A->>D: func,RDC
145+
create participant V as valkey
146+
D->>V: func,RDC
147+
D->>A: return <<QUEUE NAME>>
148+
create participant W as actinia worker<br/>JobSink
149+
A-XW: Cloudevent with <<QUEUE NAME>> per HTTP
150+
A-->>event receiver: new Cloudevent with <<QUEUE NAME>>
151+
destroy event sender
152+
A-->>event sender: return "Received event <<ID>><br>and returned event <<ID 2>><br> with actinia-job <<QUEUE NAME>>."
153+
rect rgb(191, 223, 255)
154+
W->>V: func,RDC
155+
par Processing
156+
loop process
157+
W->>W:
158+
end
159+
and Webhook
160+
opt Optional
161+
W->>A: status webhook (running)
162+
A-->>event receiver: Cloudevent (status)
163+
end
164+
end
165+
destroy W
166+
W->>A: status webhook (finished / error)
167+
A-->>event receiver: Cloudevent (status)
168+
end
105169
106-
Not skipped are:
107-
* the limit of the cells
108-
* the mapset/project limitations of the user
170+
```
171+
---

‎config/mount/sample.ini‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ user = actinia-gdi
55
password = actinia-gdi
66

77
[EVENTRECEIVER]
8-
url = http://localhost:3000/
8+
url = http://event-receiver-server:3000/
99

1010
[LOGCONFIG]
1111
logfile = actinia-cloudevent-plugin.log

‎docker/Dockerfile_eventreceiver‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
FROM alpine:3.23
2+
3+
# python3 + pip3
4+
# hadolint ignore=DL3018
5+
RUN apk update; \
6+
apk add --no-cache python3 python3-dev make
7+
ENV PATH="/opt/venv/bin:$PATH"
8+
RUN /usr/bin/python -m venv --system-site-packages --without-pip /opt/venv
9+
# hadolint ignore=DL3013
10+
RUN python -m ensurepip && pip3 install --no-cache-dir --upgrade pip pep517 wheel
11+
12+
RUN pip install --no-cache-dir cloudevents flask
13+
14+
COPY tests/cloudevent_receiver_server.py /src/cloudevent_receiver_server.py
15+
16+
WORKDIR /src
17+
CMD ["python", "/src/cloudevent_receiver_server.py"]

‎docker/docker-compose-vscode.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
services:
2+
event-receiver-server:
3+
container_name: event-receiver-server
4+
build:
5+
context: ..
6+
dockerfile: docker/Dockerfile_eventreceiver
7+
ports:
8+
- "3000:3000"
9+
networks:
10+
actinia-dev:
11+
12+
networks:
13+
actinia-dev:
14+
name: actinia-docker_actinia-dev

‎docker/docker-compose.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,11 @@ services:
1616
- SYS_PTRACE
1717
ports:
1818
- "3003:3003"
19-
# network_mode: "host"
19+
20+
event-receiver-server:
21+
build:
22+
context: ..
23+
dockerfile: docker/Dockerfile_eventreceiver
2024

2125
actinia-core:
2226
image: mundialis/actinia:2.12.2

‎ruff.toml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
# PLR2004 magic-value-comparison (unnamed numerical constants ("magic") values)
1313
# PLR6301 Method could be a function, class method, or static method
1414
# PLW0603 (checks for use of) global-statement
15+
# S104 Possible binding to all interfaces
1516
# S107 hardcoded-password-default
1617
# S113 Probable use of `requests` call without timeout
1718
# S606 start-process-with-no-shell
@@ -24,3 +25,5 @@ lint.ignore = ["ANN001", "ANN201", "ANN202", "E501", "FA102", "N802", "N816", "P
2425
"src/actinia_cloudevent_plugin/resources/logging.py" = ["A005",]
2526
"src/actinia_cloudevent_plugin/resources/config.py" = ["SIM102","S105"]
2627
"tests/integrationtests/test_cloudevent.py" = ["PLR2004",]
28+
"tests/cloudevent_receiver_server.py" = ["S104"]
29+
"tests/integrationtests/test_hook.py" = ["PLR2004",]

0 commit comments

Comments
 (0)