Skip to content

Commit b4a0470

Browse files
WontersWonters
authored
Merge pull request #1 from Wonters/develop
Develop
2 parents 17f20d4 + f88feaf commit b4a0470

69 files changed

Lines changed: 2191 additions & 332 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
IP=shift.python.software.fr

‎.github/workflows/deploy-action.yml‎

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,32 @@
1-
name: Deploy alerting alerting
1+
name: Deploy alerting
22
run-name: ${{ github.actor }} is testing out GitHub Actions 🚀
3-
on: [push]
3+
4+
on:
5+
push:
6+
branches:
7+
- main
8+
- develop
9+
410
jobs:
511
test:
6-
runs-on: python
12+
runs-on: ubuntu-latest
13+
if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop'
714
steps:
8-
- run: pytest --version
9-
- run: pip install -r requirements.txt
10-
- run: pytest tests.py
11-
branch: develop, main
15+
- uses: actions/checkout@v3
16+
- name: Install dependencies
17+
run: pip install -r requirements.txt
18+
- name: Show Pytest Version
19+
run: pytest --version
20+
- name: Run tests
21+
run: pytest src/tests
22+
1223
deploy:
13-
runs-on: docker
24+
runs-on: ubuntu-latest
25+
if: github.ref == 'refs/heads/main'
1426
steps:
27+
- uses: actions/checkout@v3
1528
- run: echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event."
1629
- run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by GitHub!"
1730
- run: echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."
1831
- run: echo "🖥️ The workflow is now ready to test your code on the runner."
19-
- run: ssh root@38.0.101.76 -i ~/.ssh/id_rsa "cd /home/ubuntu/sentimental_analyses && git pull origin main && ./install.sh"
20-
branch: main
32+
- run: ssh -i ${{ secrets.SSH_PRIVATE_KEY }} debian@shift.python.software.fr "cd /home/debian && git pull origin main && ./install.sh"

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
11
__pycache__/
2+
mlruns/
3+
checkpoints/

‎Dockerfile‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,14 @@
1-
FROM python-slim
2-
RUN apt-get update && apt-get install -y procps
3-
LABEL authors="wonters"
1+
FROM python:3.10-slim
2+
RUN apt-get update && apt-get install -y procps git
3+
LABEL authors="shift"
44
WORKDIR /app
5-
COPY . .
5+
COPY requirements.txt .
66
RUN pip install -r requirements.txt
7-
ENTRYPOINT ["gunicorn", "--bind", "0.0.0.0:5000", "app:server"]
7+
COPY src src/
8+
COPY templates templates/
9+
COPY scripts/entrypoint.sh .
10+
COPY supervisord.conf .
11+
EXPOSE 5000
12+
EXPOSE 5001
13+
14+
ENTRYPOINT ["./entrypoint.sh"]

‎README.md‎

Lines changed: 89 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,89 @@
1-
# sentimental_analyses
2-
Tweet sentimental analyses
1+
# Sentimental analyses with MLFLOW and models Wrappers
2+
3+
Tweet sentimental analyses with different models.
4+
5+
Four wrapper of models:
6+
- Logistic Regression
7+
- Random Forest
8+
- LightGBM
9+
- Bert
10+
- Roberta
11+
- LSTM
12+
13+
MLFlow is used to list all experiments and easily commpare results for several differents configurations and select the bests
14+
15+
Optuna is used to optimise parameters. It run a set of experiments with a variation of parameters and select the best configuration
16+
maximising the accuracy.
17+
18+
19+
20+
The app is dockerised and can be installed launching the command
21+
```bash
22+
docker compose up
23+
```
24+
or to run in background
25+
```bash
26+
docker compose up -d
27+
```
28+
29+
## Access and architecture
30+
The application contains alerting system and monitoring on grafana on port 3000
31+
APP PORT
32+
MLFLOW 5001
33+
API 5000
34+
GRAFANA 3000
35+
MONGO
36+
PROMETHEUS
37+
LOKI
38+
39+
A loki message and prometheus services are define
40+
Loki message show the new tweets on grafana. New tweets are saved on a Mongo db.
41+
Prometheus send metrics as the number of prediction running.
42+
An alert is send by mail when number of predictions in concurrency are up to 5.
43+
An alert is send when the result of the prediction is too bad, probability < 0.5.
44+
45+
## Installation in dev
46+
# Install uv (Rust package to fastly install package)
47+
```bash
48+
curl -Ls https://astral.sh/uv/install.sh | bash
49+
export PATH="$HOME/.cargo/bin:$PATH"
50+
```
51+
52+
## OVH Train with AI train
53+
54+
Create an object storage on OVH managed with ovhai cli
55+
The secret key is obtain clicking on the user object storage line 'access secret key'
56+
```bash
57+
ovhai datastore add s3 datastore-model https://s3.gra.io.cloud.ovh.net/ gra <acces_key> <secret_key> --store-credentials-locally
58+
# Upload to training file
59+
ovhai bucket object upload datastore-model@GRA ../data/training.1600000.processed.noemoticon.csv --object-name training.1600000.processed.noemoticon.csv
60+
```
61+
1 Datastore is associate to one bucket, it is a gateway
62+
Credentials are stored in ~/.config/ovhai/context.json
63+
64+
```bash
65+
uv pip install boto3 awscli ovhai
66+
```
67+
68+
## Run on multi GPU
69+
DEBUG
70+
```bash
71+
export TORCH_DISTRIBUTED_DEBUG=DETAIL
72+
```
73+
```bash
74+
python -m torch.distributed.run --nproc_per_node=2 train.py
75+
```
76+
77+
## Tests
78+
```bash
79+
pytest src/tests
80+
```
81+
82+
## Launch a test to verify the prection from the API
83+
Go on 127.0.0.1:5000, tap your tweet and click on predict button
84+
85+
Par requête http
86+
```bash
87+
export $(cat .env | xargs)
88+
python post.py
89+
```

‎checkpoints/Logistic_regression_tokenizer.pkl‎

-13.7 MB
Binary file not shown.

‎checkpoints/logistic_regression.pkl‎

-4.5 MB
Binary file not shown.

‎data/tweets_test_train.csv‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
"0","1467810369","Mon Apr 06 22:19:45 PDT 2009","NO_QUERY","_TheSpecialOne_","@switchfoot http://twitpic.com/2y1zl - Awww, that's a bummer. You shoulda got David Carr of Third Day to do it. ;D"
2+
"4","1467822272","Mon Apr 06 22:22:45 PDT 2009","NO_QUERY","ersle","I LOVE @Health4UandPets u guys r the best!! "
3+
"0","1467810672","Mon Apr 06 22:19:49 PDT 2009","NO_QUERY","scotthamilton","is upset that he can't update his Facebook by texting it... and might cry as a result School today also. Blah!"
4+
"4","1467822273","Mon Apr 06 22:22:45 PDT 2009","NO_QUERY","becca210","im meeting up with one of my besties tonight! Cant wait!! - GIRL TALK!!"
5+
"4","1467822287","Mon Apr 06 22:22:46 PDT 2009","NO_QUERY","katarinka","Being sick can be really cheap when it hurts too much to eat real food Plus, your friends make you soup"
6+
"0","1467810917","Mon Apr 06 22:19:53 PDT 2009","NO_QUERY","mattycus","@Kenichan I dived many times for the ball. Managed to save 50% The rest go out of bounds"
7+
"0","1467811184","Mon Apr 06 22:19:57 PDT 2009","NO_QUERY","ElleCTF","my whole body feels itchy and like its on fire "
8+
"0","1467811193","Mon Apr 06 22:19:57 PDT 2009","NO_QUERY","Karoli","@nationwideclass no, it's not behaving at all. i'm mad. why am i here? because I can't see you all over there. "
9+
"4","1467822283","Mon Apr 06 22:22:46 PDT 2009","NO_QUERY","Wingman29","@DaRealSunisaKim Thanks for the Twitter add, Sunisa! I got to meet you once at a HIN show here in the DC area and you were a sweetheart. "
10+
"0","1467811372","Mon Apr 06 22:20:00 PDT 2009","NO_QUERY","joy_wolf","@Kwesidei not the whole crew "
11+
"0","1467811592","Mon Apr 06 22:20:03 PDT 2009","NO_QUERY","mybirch","Need a hug "
12+
"4","1467822391","Mon Apr 06 22:22:47 PDT 2009","NO_QUERY","ajarofalmonds","@ProductOfFear You can tell him that I just burst out laughing really loud because of that Thanks for making me come out of my sulk!"
13+
"0","1467811594","Mon Apr 06 22:20:03 PDT 2009","NO_QUERY","coZZ","@LOLTrish hey long time no see! Yes.. Rains a bit ,only a bit LOL , I'm fine thanks , how's you ?"
14+
"0","1467811795","Mon Apr 06 22:20:05 PDT 2009","NO_QUERY","2Hood4Hollywood","@Tatiana_K nope they didn't have it "
15+
"0","1467812025","Mon Apr 06 22:20:09 PDT 2009","NO_QUERY","mimismo","@twittera que me muera ? "
16+
"4","1467822293","Mon Apr 06 22:22:46 PDT 2009","NO_QUERY","_EmilyYoung","@LovesBrooklyn2 he has that effect on everyone "
17+
"0","1467812416","Mon Apr 06 22:20:16 PDT 2009","NO_QUERY","erinx3leannexo","spring break in plain city... it's snowing "
18+
"0","1467812579","Mon Apr 06 22:20:17 PDT 2009","NO_QUERY","pardonlauren","I just re-pierced my ears "
19+
"0","1467812723","Mon Apr 06 22:20:19 PDT 2009","NO_QUERY","TLeC","@caregiving I couldn't bear to watch it. And I thought the UA loss was embarrassing . . . . ."
20+
"4","1467822447","Mon Apr 06 22:22:51 PDT 2009","NO_QUERY","vmdavinci","@r_keith_hill Thans for your response. Ihad already find this answer "
21+
"4","1467822465","Mon Apr 06 22:22:48 PDT 2009","NO_QUERY","jessicavaliyi","@KeepinUpWKris I am so jealous, hope you had a great time in vegas! how did you like the ACM's?! LOVE YOUR SHOW!! "
22+
"0","1467812771","Mon Apr 06 22:20:19 PDT 2009","NO_QUERY","robrobbierobert","@octolinz16 It it counts, idk why I did either. you never talk to me anymore "
23+
"0","1467812784","Mon Apr 06 22:20:20 PDT 2009","NO_QUERY","bayofwolves","@smarrison i would've been the first, but i didn't have a gun. not really though, zac snyder's just a doucheclown."
24+
"4","1467822489","Mon Apr 06 22:22:49 PDT 2009","NO_QUERY","emmasaur28","@tommcfly ah, congrats mr fletcher for finally joining twitter "
25+
"0","1467812799","Mon Apr 06 22:20:20 PDT 2009","NO_QUERY","HairByJess","@iamjazzyfizzle I wish I got to watch it with you!! I miss you and @iamlilnicki how was the premiere?!"
26+
"0","1467812964","Mon Apr 06 22:20:22 PDT 2009","NO_QUERY","lovesongwriter","Hollis' death scene will hurt me severely to watch on film wry is directors cut not out now?"
27+
"0","1467813137","Mon Apr 06 22:20:25 PDT 2009","NO_QUERY","armotley","about to file taxes "
28+
"4","1467822496","Mon Apr 06 22:22:49 PDT 2009","NO_QUERY","SherylBreuker","@e4VoIP I RESPONDED Stupid cat is helping me type. Forgive errors "

‎docker-compose.yml‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
services:
2+
backend:
3+
build: .
4+
container_name: sentimental_analyses_backend
5+
ports:
6+
- "5000:5000"
7+
- "5001:5001"
8+
volumes:
9+
- ./mlruns:/app/mlruns/
10+
- ./src:/app/src/
11+
# stdin_open: true
12+
# tty: true
13+
entrypoint: "./entrypoint.sh"
14+
env_file:
15+
- .env
16+
environment:
17+
- MLFLOW_TRACKING_URI=http://0.0.0.0:5001
18+
- HOST_DEPLOY = 0.0.0.0
19+
20+
prometheus:
21+
image: prom/prometheus:latest
22+
container_name: sentimental_analyses_prometheus
23+
ports:
24+
- "9090:9090"
25+
volumes:
26+
- ./prometheus.yml:/etc/prometheus/prometheus.yml
27+
command:
28+
- --config.file=/etc/prometheus/prometheus.yml
29+
restart: unless-stopped
30+
31+
grafana:
32+
image: grafana/grafana:latest
33+
container_name: sentimental_analyses_grafana
34+
ports:
35+
- "3000:3000"
36+
depends_on:
37+
- loki
38+
volumes:
39+
- grafana-storage:/var/lib/grafana
40+
- ./grafana/dashboards:/etc/grafana/dashboards
41+
- ./grafana/dashboards.yaml:/etc/grafana/provisioning/dashboards/tweet_dashboards.yaml
42+
- ./grafana/datasources:/etc/grafana/provisioning/datasources
43+
- ./grafana/alerting:/etc/grafana/provisioning/alerting
44+
environment:
45+
- GF_SECURITY_ADMIN_PASSWORD=admin
46+
- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/etc/grafana/dashboards/tweet_dashboard.json
47+
- GF_SMTP_ENABLED=true
48+
- GF_SMTP_HOST=smtp.gmail.com:587
49+
- GF_SMTP_USER=shift.python.software@gmail.com
50+
- GF_SMTP_PASSWORD=meph potg kjia pfah
51+
- GF_SMTP_FROM_ADDRESS=shift.python.software@gmail.com
52+
- GF_SMTP_FROM_NAME=Grafana
53+
restart: unless-stopped
54+
55+
loki:
56+
image: grafana/loki:2.9.4
57+
container_name: sentimental_analyses_loki
58+
ports:
59+
- "3100:3100"
60+
command: -config.file=/etc/loki/local-config.yaml
61+
restart: unless-stopped
62+
63+
db:
64+
image: mongo:4.4
65+
container_name: sentimental_analyses_db
66+
ports:
67+
- "27017:27017"
68+
restart: unless-stopped
69+
70+
promtail:
71+
image: grafana/promtail:2.9.4
72+
container_name: sentimental_analyses_promtail
73+
volumes:
74+
- ./log:/var/log
75+
- ./promtail-config.yaml:/etc/promtail/config.yaml
76+
command: -config.file=/etc/promtail/config.yaml
77+
depends_on:
78+
- loki
79+
restart: unless-stopped
80+
# nginx:
81+
# image: nginx
82+
# container_name: sentimental_analyses_nginx
83+
# ports:
84+
# - "81:80"
85+
# - '444:443'
86+
# volumes:
87+
# - ./nginx.conf:/etc/nginx/conf.d/default.conf
88+
# depends_on:
89+
# - backend
90+
91+
volumes:
92+
grafana-storage:

‎grafana/alerting/contact-point.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
apiVersion: 1
2+
3+
contactPoints:
4+
- orgId: 1
5+
name: admin-email
6+
receivers:
7+
- uid: admin-email
8+
type: email
9+
settings:
10+
addresses: "shift.python.software@gmail.com"

0 commit comments

Comments
 (0)