Move Wav2Lip model download commands from Dockerfiles to entrypoint.sh (opea-project#811)

* Update audioqna gateway to print text, in gateway.py

Signed-off-by: Chun Tao <[email protected]>

* updates needed for demo

Signed-off-by: Chun Tao <[email protected]>

* original pr content

Signed-off-by: Chun Tao <[email protected]>

* Revert "updates needed for demo"

This reverts commit f0c7a02.

Signed-off-by: Chun Tao <[email protected]>

* remove improper images

Signed-off-by: Chun Tao <[email protected]>

* Addressed some comments on previous pr

Signed-off-by: Chun Tao <[email protected]>

* Add Dockerfile for cpu support

Signed-off-by: Chun Tao <[email protected]>

* CODEOWNER: Update comp CODEOWNER (opea-project#757)

Signed-off-by: Yeoh, Hoong Tee <[email protected]>
Signed-off-by: Chun Tao <[email protected]>

* Add stable diffusion microservice (opea-project#729)

* add stable diffusion microservice.

Signed-off-by: Ye, Xinyu <[email protected]>

* added test.

Signed-off-by: Ye, Xinyu <[email protected]>

* changed output to images bytes data

Signed-off-by: Ye, Xinyu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* unified inference and wrapper into one microservice.

Signed-off-by: Ye, Xinyu <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix test.

Signed-off-by: Ye, Xinyu <[email protected]>

---------

Signed-off-by: Ye, Xinyu <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Chun Tao <[email protected]>

* Compatible with different platforms. (opea-project#766)

* Compatible with different platforms.

Signed-off-by: ZePan110 <[email protected]>

* Fix issue.

Signed-off-by: ZePan110 <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix issue

Signed-off-by: ZePan110 <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: ZePan110 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Chun Tao <[email protected]>

* Optimize path and link validity check. (opea-project#745)

Signed-off-by: ZePan110 <[email protected]>
Signed-off-by: Chun Tao <[email protected]>

* Add timeout for ut test (opea-project#773)

Signed-off-by: chensuyue <[email protected]>
Signed-off-by: Chun Tao <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Chun Tao <[email protected]>

* test hyperlink

Signed-off-by: Chun Tao <[email protected]>

* test hyperlink

Signed-off-by: Chun Tao <[email protected]>

* test hyperlink issue

Signed-off-by: Chun Tao <[email protected]>

* test hyperlink issue

Signed-off-by: Chun Tao <[email protected]>

* put back hyperlinks in readme

Signed-off-by: Chun Tao <[email protected]>

* remove possible error hyperlink

Signed-off-by: Chun Tao <[email protected]>

* put hyperlink back

Signed-off-by: Chun Tao <[email protected]>

* major update to use FastAPI for wav2lip, and structure component format

Signed-off-by: Chun Tao <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add dockerfiles in animation-compose-cd.yaml

Signed-off-by: Chun Tao <[email protected]>

* Fix end of file issue in animation-compose-cd.yaml

Signed-off-by: Chun Tao <[email protected]>

* Fix Docker deployment on Xeon

Signed-off-by: Chun Tao <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add versioning for all pip packages

Signed-off-by: Chun Tao <[email protected]>

* e2e test script for animation

Signed-off-by: Chun Tao <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update e2e test script

Signed-off-by: Chun Tao <[email protected]>

* update e2e test script

Signed-off-by: Chun Tao <[email protected]>

* update readme

Signed-off-by: Chun Tao <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update

Signed-off-by: Chun Tao <[email protected]>

* update

Signed-off-by: Chun Tao <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update gateway

Signed-off-by: Chun Tao <[email protected]>

* udpate gateway

Signed-off-by: Chun Tao <[email protected]>

* Fix AVATAR_CHATBOT

Signed-off-by: Chun Tao <[email protected]>

* update gateway

Signed-off-by: Chun Tao <[email protected]>

* update

Signed-off-by: Chun Tao <[email protected]>

* test

Signed-off-by: Chun Tao <[email protected]>

* update

Signed-off-by: Chun Tao <[email protected]>

* update

Signed-off-by: Chun Tao <[email protected]>

* update gateway

Signed-off-by: Chun Tao <[email protected]>

* fix max_tokens in AvatarChatbot gateway

Signed-off-by: Chun Tao <[email protected]>

* test

Signed-off-by: Chun Tao <[email protected]>

* update

Signed-off-by: Chun Tao <[email protected]>

* update

Signed-off-by: Chun Tao <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Model download script moved from Dockerfiles to Docker entrypoint

Signed-off-by: Chun Tao <[email protected]>

* update paths

Signed-off-by: Chun Tao <[email protected]>

* Correct paths in readme

Signed-off-by: Chun Tao <[email protected]>

* revert changes to audioqna gateway

Signed-off-by: Chun Tao <[email protected]>

* longer wait time after docker run

Signed-off-by: Chun Tao <[email protected]>

* add mount volume in test scripts

Signed-off-by: Chun Tao <[email protected]>

* add volume mount in test scripts

Signed-off-by: Chun Tao <[email protected]>

* udpate test script

Signed-off-by: Chun Tao <[email protected]>

---------

Signed-off-by: Chun Tao <[email protected]>
Signed-off-by: Yeoh, Hoong Tee <[email protected]>
Signed-off-by: Ye, Xinyu <[email protected]>
Signed-off-by: ZePan110 <[email protected]>
Signed-off-by: chensuyue <[email protected]>
Co-authored-by: Hoong Tee, Yeoh <[email protected]>
Co-authored-by: XinyuYe-Intel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: ZePan110 <[email protected]>
Co-authored-by: chen, suyue <[email protected]>

Author: 6 people

comps/animation/wav2lip/README.md

@@ -42,11 +42,11 @@ export WAV2LIP_PORT=7860
 export ANIMATION_PORT=9066
 export INFERENCE_MODE='wav2lip+gfpgan'
 export CHECKPOINT_PATH='/usr/local/lib/python3.11/site-packages/Wav2Lip/checkpoints/wav2lip_gan.pth'
-export FACE="comps/animation/wav2lip/assets/img/avatar1.jpg"
+export FACE="assets/img/avatar1.jpg"
 # export AUDIO='assets/audio/eg3_ref.wav' # audio file path is optional, will use base64str in the post request as input if is 'None'
 export AUDIO='None'
 export FACESIZE=96
-export OUTFILE="comps/animation/wav2lip/assets/outputs/result.mp4"
+export OUTFILE="assets/outputs/result.mp4"
 export GFPGAN_MODEL_VERSION=1.4 # latest version, can roll back to v1.3 if needed
 export UPSCALE_FACTOR=1
 export FPS=10
@@ -61,11 +61,11 @@ export WAV2LIP_PORT=7860
 export ANIMATION_PORT=9066
 export INFERENCE_MODE='wav2lip+gfpgan'
 export CHECKPOINT_PATH='/usr/local/lib/python3.10/dist-packages/Wav2Lip/checkpoints/wav2lip_gan.pth'
-export FACE="comps/animation/wav2lip/assets/img/avatar1.jpg"
+export FACE="assets/img/avatar1.jpg"
 # export AUDIO='assets/audio/eg3_ref.wav' # audio file path is optional, will use base64str in the post request as input if is 'None'
 export AUDIO='None'
 export FACESIZE=96
-export OUTFILE="comps/animation/wav2lip/assets/outputs/result.mp4"
+export OUTFILE="assets/outputs/result.mp4"
 export GFPGAN_MODEL_VERSION=1.4 # latest version, can roll back to v1.3 if needed
 export UPSCALE_FACTOR=1
 export FPS=10
@@ -78,13 +78,13 @@ export FPS=10
 - Xeon CPU
 
 ```bash
-docker run --privileged -d --name "wav2lip-service" -p 7860:7860 --ipc=host -w /home/user/comps/animation/wav2lip -e PYTHON=/usr/bin/python3.11 -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT opea/wav2lip:latest
+docker run --privileged -d --name "wav2lip-service" -p 7860:7860 --ipc=host -w /home/user/comps/animation/wav2lip -e PYTHON=/usr/bin/python3.11 -v $(pwd)/comps/animation/wav2lip/assets:/home/user/comps/animation/wav2lip/assets -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT opea/wav2lip:latest
 ```
 
 - Gaudi2 HPU
 
 ```bash
-docker run --privileged -d --name "wav2lip-gaudi-service" -p 7860:7860 --runtime=habana --cap-add=sys_nice --net=host --ipc=host -w /home/user/comps/animation/wav2lip -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e PYTHON=/usr/bin/python3.10 -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT opea/wav2lip-gaudi:latest
+docker run --privileged -d --name "wav2lip-gaudi-service" -p 7860:7860 --runtime=habana --cap-add=sys_nice --net=host --ipc=host -w /home/user/comps/animation/wav2lip -v $(pwd)/comps/animation/wav2lip/assets:/home/user/comps/animation/wav2lip/assets -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e PYTHON=/usr/bin/python3.10 -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT opea/wav2lip-gaudi:latest
 ```
 
 ## 2.2 Run Animation Microservice
@@ -100,8 +100,8 @@ Once microservice starts, user can use below script to validate the running micr
 ## 3.1 Validate Wav2Lip service
 
 ```bash
-cd GenAIComps/comps/animation/wav2lip
-python3 dependency/check_wav2lip_server.py
+cd GenAIComps
+python3 comps/animation/wav2lip/dependency/check_wav2lip_server.py
 ```
 
 ## 3.2 Validate Animation service
@@ -115,14 +115,14 @@ curl http://${ip_address}:9066/v1/animation -X POST -H "Content-Type: applicatio
 or
 
 ```bash
-cd GenAIComps/comps/animation/wav2lip
-python3 check_animation_server.py
+cd GenAIComps
+python3 comps/animation/wav2lip/dependency/check_animation_server.py
 ```
 
 The expected output is a message similar to the following:
 
 ```bash
-{'wav2lip_result': '.../GenAIComps/comps/animation/wav2lip/assets/outputs/result.mp4'}
+{'wav2lip_result': '....../GenAIComps/comps/animation/wav2lip/assets/outputs/result.mp4'}
 ```
 
 Please find "comps/animation/wav2lip/assets/outputs/result.mp4" as a reference generated video.

comps/animation/wav2lip/assets/outputs/result.mp4

@@ -11,7 +11,7 @@
 outfile = os.environ.get("OUTFILE")
 
 # Read the JSON file
-with open("assets/audio/sample_question.json", "r") as file:
+with open("comps/animation/wav2lip/assets/audio/sample_question.json", "r") as file:
     data = json.load(file)
 
 response = requests.post(url=endpoint, json=data, headers={"Content-Type": "application/json"}, proxies={"http": None})

comps/animation/wav2lip/check_animation_server.py

@@ -51,8 +51,6 @@ ENV PYTHONPATH="$PYTHONPATH:/usr/local/lib/python3.11/site-packages/gfpgan"
 
 # Download pre-trained models
 WORKDIR /usr/local/lib/python3.11/site-packages
-RUN chmod +x /home/user/comps/animation/wav2lip/dependency/download_ckpts.sh
-RUN /home/user/comps/animation/wav2lip/dependency/download_ckpts.sh
 
 # Install pip dependencies
 RUN pip install -r /home/user/comps/animation/wav2lip/requirements.txt

comps/animation/wav2lip/dependency/Dockerfile

@@ -45,8 +45,6 @@ ENV PYTHONPATH="$PYTHONPATH:/usr/local/lib/python3.10/dist-packages/gfpgan"
 
 # Download pre-trained models
 WORKDIR /usr/local/lib/python3.10/dist-packages
-RUN chmod +x /home/user/comps/animation/wav2lip/dependency/download_ckpts.sh
-RUN /home/user/comps/animation/wav2lip/dependency/download_ckpts.sh
 
 # Install pip dependencies
 RUN pip install -r /home/user/comps/animation/wav2lip/requirements.txt

comps/animation/wav2lip/dependency/Dockerfile.intel_hpu

@@ -10,7 +10,7 @@
 outfile = os.environ.get("OUTFILE")
 
 # Read the JSON file
-with open("assets/audio/sample_question.json", "r") as file:
+with open("comps/animation/wav2lip/assets/audio/sample_question.json", "r") as file:
     data = json.load(file)
 
 inputs = {"audio": data["byte_str"]}

comps/animation/wav2lip/dependency/check_wav2lip_server.py

@@ -1,14 +1,29 @@
 #!/bin/sh
-cd /home/user/comps/animation/wav2lip/ || exit
 
 # Copyright (C) 2024 Intel Corporation
 # SPDX-License-Identifier: Apache-2.0
 
+# Check the value of $DEVICE and cd to the download path accordingly
+if [ "$DEVICE" = "hpu" ]; then
+    cd /usr/local/lib/python3.10/dist-packages
+else
+    cd /usr/local/lib/python3.11/site-packages
+fi
+
+# Download model weights
+wget https://www.adrianbulat.com/downloads/python-fan/s3fd-619a316812.pth -O Wav2Lip/face_detection/detection/sfd/s3fd.pth
+mkdir -p Wav2Lip/checkpoints
+wget "https://iiitaphyd-my.sharepoint.com/:f:/g/personal/radrabha_m_research_iiit_ac_in/Eb3LEzbfuKlJiR600lQWRxgBIY27JZg80f7V9jtMfbNDaQ?download=1" -O Wav2Lip/checkpoints/wav2lip.pth
+wget "https://iiitaphyd-my.sharepoint.com/:f:/g/personal/radrabha_m_research_iiit_ac_in/EdjI7bZlgApMqsVoEUUXpLsBxqXbn5z8VTmoxp55YNDcIA?download=1" -O Wav2Lip/checkpoints/wav2lip_gan.pth
+wget https://github.com/TencentARC/GFPGAN/releases/download/v1.3.0/GFPGANv1.4.pth -P gfpgan/experiments/pretrained_models
+echo "Face Detector, Wav2Lip, GFPGAN weights downloaded."
+
 # Environment variables
 export PT_HPU_LAZY_MODE=0
 export PT_HPU_ENABLE_REFINE_DYNAMIC_SHAPES=1
 
 # Wav2Lip, GFPGAN
+cd /home/user/comps/animation/wav2lip/ || exit
 python3 dependency/wav2lip_server.py \
 --device $DEVICE \
 --port $((WAV2LIP_PORT)) \

comps/animation/wav2lip/dependency/entrypoint.sh

@@ -17,7 +17,7 @@ function build_docker_images() {
     else
         echo "opea/wav2lip built successful"
     fi
-    docker build -t opea/animation:comps -f comps/animation/wav2lip/Dockerfile .
+    docker build --no-cache -t opea/animation:comps -f comps/animation/wav2lip/Dockerfile .
     if [ $? -ne 0 ]; then
         echo "opea/animation built fail"
         exit 1
@@ -43,12 +43,13 @@ function start_service() {
     export UPSCALE_FACTOR=1
     export FPS=10
 
-    docker run -d --name="test-comps-animation-wav2lip" -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT -p 7860:7860 --ipc=host opea/wav2lip:comps
-    docker run -d --name="test-comps-animation" -e WAV2LIP_ENDPOINT=http://$ip_address:7860 -e http_proxy=$http_proxy -e https_proxy=$https_proxy -p 9066:9066 --ipc=host opea/animation:comps
-    sleep 1m
+    docker run -d --name="test-comps-animation-wav2lip" -v $WORKPATH/comps/animation/wav2lip/assets:/home/user/comps/animation/wav2lip/assets -e http_proxy=$http_proxy -e https_proxy=$https_proxy -e DEVICE=$DEVICE -e INFERENCE_MODE=$INFERENCE_MODE -e CHECKPOINT_PATH=$CHECKPOINT_PATH -e FACE=$FACE -e AUDIO=$AUDIO -e FACESIZE=$FACESIZE -e OUTFILE=$OUTFILE -e GFPGAN_MODEL_VERSION=$GFPGAN_MODEL_VERSION -e UPSCALE_FACTOR=$UPSCALE_FACTOR -e FPS=$FPS -e WAV2LIP_PORT=$WAV2LIP_PORT -p 7860:7860 --ipc=host opea/wav2lip:comps
+    docker run -d --name="test-comps-animation" -v $WORKPATH/comps/animation/wav2lip/assets:/home/user/comps/animation/wav2lip/assets -e WAV2LIP_ENDPOINT=http://$ip_address:7860 -e http_proxy=$http_proxy -e https_proxy=$https_proxy -p 9066:9066 --ipc=host opea/animation:comps
+    sleep 3m
 }
 
 function validate_microservice() {
+    cd $WORKPATH
     result=$(http_proxy="" curl http://localhost:9066/v1/animation -X POST -H "Content-Type: application/json" -d @comps/animation/wav2lip/assets/audio/sample_question.json)
     if [[ $result == *"result.mp4"* ]]; then
         echo "Result correct."