Skip to content

Commit cd07892

Browse files
mmacatammacata
committed
update README
1 parent 656b0c5 commit cd07892

File tree

2 files changed

+133
-70
lines changed

2 files changed

+133
-70
lines changed

‎.github/workflows/test.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ jobs:
3737
steps:
3838
- uses: actions/checkout@v6
3939
- name: Start containers
40-
run: docker compose -f "docker/docker-compose.yml" up -d --build
40+
run: docker compose -f docker/docker-compose.yml up -d --build
4141
- name: List running docker
4242
run: docker ps
4343
- name: Docker logs
@@ -49,4 +49,4 @@ jobs:
4949
- name: Run integration test
5050
run: docker exec -t docker-actinia-cloudevent-1 make integrationtest
5151
- name: Stop containers
52-
run: docker compose -f "docker/docker-compose.yml" down
52+
run: docker compose -f docker/docker-compose.yml down

‎README.md‎

Lines changed: 131 additions & 68 deletions
Original file line numberDiff line numberDiff line change
@@ -1,108 +1,171 @@
11
# actinia-cloudevent-plugin
22

3-
This is a plugin for [actinia-core](https://github.com/mundialis/actinia_core) which adds cloudevent endpoints and runs as standalone app.
3+
This is a plugin for [actinia-core](https://github.com/mundialis/actinia_core) which translates cloudevents into a process definition
4+
for actinia-core and runs as standalone app.
45

56
## Installation and Setup
67

78
Use docker-compose for installation:
89
```bash
910
docker compose -f docker/docker-compose.yml build
10-
docker compose -f docker/docker-compose.yml run --rm --service-ports --entrypoint sh actinia-cloudevent
11-
# within docker
12-
gunicorn -b 0.0.0.0:5000 -w 8 --access-logfile=- -k gthread actinia_cloudevent_plugin.main:flask_app
11+
docker compose -f docker/docker-compose.yml up -d
1312
```
1413

15-
### DEV setup
16-
```bash
14+
### DEV setup with vscode
15+
1. Start actinia-core locally via local dev-setup as described in the
16+
[actinia-docker repository](https://github.com/actinia-org/actinia-docker#local-dev-setup-for-actinia-core-plugins-with-vscode)
17+
2. Have this repository open locally with vscode and press `F5`.
18+
After a few seconds, a browser window should be openend, pointing to
19+
an endpoint (showing 405 Method Not Allowed as this plugin has only
20+
HTTP POST endpoints so far).
21+
22+
Alternatively, configure actinia-core to run jobs via actinia-worker.
23+
In the actinia-core dev-setup, use the `per_job` queue in
24+
`actinia-docker/actinia-dev/actinia.cfg`:
25+
```
26+
[QUEUE]
27+
# queue_type = local
28+
29+
[QUEUE]
30+
queue_type = per_job
31+
kvdb_queue_server_url = valkey
32+
kvdb_queue_server_password = pass
33+
worker_queue_prefix = job_queue
34+
```
35+
And restart the vscode debugger. This way, a job is registered in the
36+
valkey DB, but not directly started.
37+
38+
39+
<!-- TODO: check update docker-compose DEV setup if needed -->
40+
<!-- ```bash
1741
# Uncomment the volume mount of the cloud-event-plugin within docker/docker-compose.yml,
1842
# then:
1943
docker compose -f docker/docker-compose.yml build
2044
docker compose -f docker/docker-compose.yml run --rm --service-ports --entrypoint sh actinia-cloudevent
21-
# within docker:
45+
# within docker
2246
# install the plugin
2347
pip3 install .
2448
# start flask app with actinia-cloudevent-plugin
49+
gunicorn -b 0.0.0.0:5000 -w 8 --access-logfile=- -k gthread actinia_cloudevent_plugin.main:flask_app
50+
# or directly via python
2551
python3 -m actinia_cloudevent_plugin.main
26-
```
27-
28-
### Installation hints
29-
* If you get an error like: `ERROR: for docker_kvdb_1 Cannot start service valkey: network xxx not found` you can try the following:
30-
```bash
31-
docker compose -f docker/docker-compose-dev.yml down
32-
# remove all custom networks not used by a container
33-
docker network prune
34-
docker compose -f docker/docker-compose-dev.yml up -d
35-
```
52+
``` -->
3653

3754
## Configuration
3855

39-
- the URL of the cloudevent receiver is defined within [config/mount/sample.ini](config/mount/sample.ini): `[EVENTRECEIVER]` (Default value defined within [src/actinia_cloudevent_plugin/resources/config.py](src/actinia_cloudevent_plugin/resources/config.py))
56+
- the URL of actinia-core and the cloudevent receiver is defined
57+
within [config/mount/sample.ini](config/mount/sample.ini): `
58+
[EVENTRECEIVER]` (Default value defined within
59+
[src/actinia_cloudevent_plugin/resources/config.py](src/actinia_cloudevent_plugin/resources/config.py))
4060

4161
## Requesting endpoint
42-
4362
**Note**: Assuming cloudevent-plugin is running as described in previous setup.
4463

4564
You can test the plugin and request the `/` endpoint, e.g. with:
4665
```bash
47-
# Start server for receiving of cloudevents (which are returned as response)
48-
# NOTE: as defined within config/mount/sample.ini: [EVENTRECEIVER]
49-
python3 tests/cloudevent_receiver_server.py
50-
51-
# In another terminal
5266
JSON=tests/examples/cloudevent_example.json
53-
curl -X POST -H 'Content-Type: application/json' --data @$JSON localhost:5000/api/v1/ | jq
67+
curl -X POST -H 'Content-Type: application/json' --data @$JSON localhost:3003/
68+
```
69+
Or test with `per_job` queue
70+
```bash
71+
JSON=tests/examples/cloudevent_example.json
72+
curl -X POST -H 'Content-Type: application/json' --data @$JSON localhost:3003/
73+
# Get the actinia job queue name from the response
74+
QUEUE_NAME=job_queue_resource_id-d4d9be86-5938-47ff-9c6d-7c79964862c0
75+
76+
docker run --rm --network actinia-docker_actinia-dev \
77+
-v $HOME/actinia/grassdb:/actinia_core/grassdb \
78+
-v $HOME/actinia/grassdb_user:/actinia_core/userdata \
79+
--entrypoint actinia-worker actiniacore -q $QUEUE_NAME
5480
```
5581

5682
Exemplary returned cloudevent: [tests/examples/cloudevent_example_return.json](tests/examples/cloudevent_example_return.json)
5783

58-
## Hints
59-
60-
* If you have no `.git` folder in the plugin folder, you need to set the
61-
`SETUPTOOLS_SCM_PRETEND_VERSION` before installing the plugin:
62-
```bash
63-
export SETUPTOOLS_SCM_PRETEND_VERSION=0.0
64-
```
65-
Otherwise you will get an error like this `LookupError: setuptools-scm was unable to detect version for '/src/actinia-cloudevent-plugin'.`.
66-
67-
* If you make changes in code and nothing changes you can try to uninstall the plugin:
68-
```bash
69-
pip3 uninstall actinia-cloudevent-plugin.wsgi -y
70-
rm -rf /usr/lib/python3.8/site-packages/actinia_cloudevent_plugin.wsgi-*.egg
71-
```
7284

7385
## Running tests
74-
You can run the tests in the actinia test docker:
86+
You can run the tests in the actinia test docker.
87+
These are the same steps which the github workflow is executing.
7588

7689
```bash
77-
# Uncomment the volume mount of the cloud-event-plugin within docker/docker-compose.yml,
78-
# then:
79-
docker compose -f docker/docker-compose.yml build
80-
docker compose -f docker/docker-compose.yml run --rm --service-ports --entrypoint sh actinia-cloudevent
81-
82-
# run all tests
83-
make test
84-
85-
# # run only unittests
86-
# make unittest
87-
88-
# run only integrationtests
89-
make integrationtest
90-
91-
# run only tests which are marked for development with the decorator '@pytest.mark.dev'
92-
make devtest
90+
docker compose -f docker/docker-compose.yml up -d --build
91+
sleep 10 && \
92+
docker logs docker-actinia-cloudevent-1 && echo && \
93+
docker logs docker-actinia-core-1 && echo && \
94+
docker logs docker-event-receiver-server-1
95+
docker exec -t docker-actinia-cloudevent-1 make integrationtest
96+
docker compose -f docker/docker-compose.yml down
9397
```
9498

95-
## Hint for the development of actinia plugins
96-
97-
### skip permission check
98-
The parameter [`skip_permission_check`](https://github.com/mundialis/actinia_core/blob/main/src/actinia_core/processing/actinia_processing/ephemeral_processing.py#L1420-L1422) (see [example in actinia-statistic plugin](https://github.com/mundialis/actinia_statistic_plugin/blob/master/src/actinia_statistic_plugin/vector_sampling.py#L207))
99-
should only be set to `True` if you are sure that you really don't want to check the permissions.
99+
---
100+
101+
## Possible setup
102+
103+
```mermaid
104+
graph
105+
106+
1[Event Emitter, e.g. UI]
107+
2[Kafka Queue Broker]
108+
subgraph actinia
109+
direction TB
110+
3[actinia-cloudevent-plugin Transformer Service]
111+
%%3@{ shape: braces, label: "Transformer Service" }
112+
4[actinia worker JobSink]
113+
%%4@{ shape: braces, label: "JobSink" }
114+
5[actinia-core]
115+
%%5@{ shape: braces, label: " write job to valkey" }
116+
6[valkey]
117+
end
118+
119+
1 -- Cloudevent --> 2
120+
2 <-- Cloudevent --> 3
121+
3 -- 1: func,RDC --> 5
122+
5 -- 2: func,RDC --> 6
123+
3 -- 3: Cloudevent (queue Name) per HTTP --> 4
124+
4 -- 4: func,RDC --> 6
125+
4 -- 5: webhook --> 3
126+
3 -- Cloudevent (status) --> 2
100127
101-
The skip of the permission check leads to a skipping of:
102-
* [the module check](https://github.com/mundialis/actinia_core/blob/main/src/actinia_core/processing/actinia_processing/ephemeral_processing.py#L579-L589)
103-
* [the limit of the number of processes](https://github.com/mundialis/actinia_core/blob/main/src/actinia_core/processing/actinia_processing/ephemeral_processing.py#L566-L570)
104-
* the limit of the processing time
128+
```
129+
---
130+
131+
<!-- <script>
132+
mermaid.initialize({ sequence: { showSequenceNumbers: true } });
133+
</script> -->
134+
135+
```mermaid
136+
sequenceDiagram
137+
autonumber
138+
create participant event receiver
139+
create actor event sender
140+
create participant A as actinia-cloudevent-plugin<br/>Transformer Service
141+
%% Note right of A: Transformer Service #9829;
142+
event sender-->>A: Cloudevent
143+
create participant D as actinia-core
144+
A->>D: func,RDC
145+
create participant V as valkey
146+
D->>V: func,RDC
147+
D->>A: return <<QUEUE NAME>>
148+
create participant W as actinia worker<br/>JobSink
149+
A-XW: Cloudevent with <<QUEUE NAME>> per HTTP
150+
A-->>event receiver: new Cloudevent with <<QUEUE NAME>>
151+
destroy event sender
152+
A-->>event sender: return "Received event <<ID>><br>and returned event <<ID 2>><br> with actinia-job <<QUEUE NAME>>."
153+
rect rgb(191, 223, 255)
154+
W->>V: func,RDC
155+
par Processing
156+
loop process
157+
W->>W:
158+
end
159+
and Webhook
160+
opt Optional
161+
W->>A: status webhook (running)
162+
A-->>event receiver: Cloudevent (status)
163+
end
164+
end
165+
destroy W
166+
W->>A: status webhook (finished / error)
167+
A-->>event receiver: Cloudevent (status)
168+
end
105169
106-
Not skipped are:
107-
* the limit of the cells
108-
* the mapset/project limitations of the user
170+
```
171+
---

0 commit comments

Comments
 (0)