Skip to content

README_en.md

Latest commit

 

History

History
100 lines (79 loc) · 4.69 KB

README_en.md

File metadata and controls

100 lines (79 loc) · 4.69 KB
Raw

Digital Human Dialogue Demo

Digital human dialogue demo based on open-source projects, offering two digital human generation methods: MLLM (GLM-4-Voice)-THG and ASR-LLM-TTS-THG, with the initial package delay as low as 3 seconds.

Online demo：https://www.modelscope.cn/studios/AI-ModelScope/video_chat

中文简体 | English

TODO

  • Add voice cloning feature to the TTS module
  • Add edge-tts to the TTS module
  • Add local inference for the Qwen to the LLM module
  • Support GLM-4-Voice, offering two generation methods: ASR-LLM-TTS-THG and MLLM-THG
  • Optimize the pipeline

Technology Stack

Local Deployment

1. Environment Setup

  • ubuntu 22.04
  • python 3.10
  • torch 2.1.2
$ git lfs install
$ git clone https://www.modelscope.cn/studios/AI-ModelScope/video_chat.git
$ conda create -n metahuman python=3.10
$ conda activate metahuman
$ cd video_chat
$ pip install -r requirement.txt
$ pip install --upgrade gradio # install gradio 5

2, Download Weights

2.1 Clone from ModelScope (Recommended)

The Creative Space repository is already set up to track weight files with git lfs.

If you clone via git clone https://www.modelscope.cn/studios/AI-ModelScope/video_chat.git, no additional configuration is required.

2.2 Manual Download

2.2.1 MuseTalk weights

Pls refer to this link

The weights should be organized as follows:

./weights/
├── dwpose
│   └── dw-ll_ucoco_384.pth
├── face-parse-bisent
│   ├── 79999_iter.pth
│   └── resnet18-5c106cde.pth
├── musetalk
│   ├── musetalk.json
│   └── pytorch_model.bin
├── sd-vae-ft-mse
│   ├── config.json
│   └── diffusion_pytorch_model.bin
└── whisper
    └── tiny.pt

2.2.2 GPT-SoVITS weights

Pls refer to this link

3. Other Configurations

The LLM and TTS modules offer various inference options for you to choose from.

3.1 Using API-KEY (Default)

For the LLM and TTS modules, if your local machine has limited performance, you can use the Qwen API and CosyVoice API provided by Alibaba Cloud's AI Model Service Platform. Please configure the API-KEY in app.py (line 14). Refer to this link to complete the acquisition and configuration of the API-KEY.

os.environ["DASHSCOPE_API_KEY"] = "INPUT YOUR API-KEY HERE"

3.2 Without Using API-KEY

If you do not wish to use an API-KEY, please refer to the instructions below to modify the relevant code.

3.2.1 LLM Module

In src/llm.py, the Qwen and Qwen_API classes handle local inference and API calls respectively. If you are not using an API-KEY, there are two ways to perform local inference:

  1. Use Qwen for local inference.
  2. Qwen_API calls the API by default for inference. If you do not use an API-KEY, you can also use vLLM to deploy the model inference service locally. Refer to this link for deployment instructions. After deployment, initialize the instance with Qwen_API(api_key="EMPTY", base_url="http://localhost:8000/v1") to call the local inference service.
3.2.2 TTS Module

In src/tts.py, GPT_SoVits_TTS and CosyVoice_API handle local inference and API calls respectively. If you are not using an API-KEY, you can directly remove the CosyVoice_API related content and use Edge_TTS to call the free TTS service of Edge browser for inference.

4. Starting the Service

$ python app.py

5. Using a Custom Digital Human Avatar (Optional)

Add the recorded digital human avatar video to the /data/video/ directory. Modify the avatar_list in the Muse_Talk class in /src/thg.py to include (avatar_name, bbox_shift). For details on bbox_shift, refer to this link. Add the digital human avatar name to the avatar_name field in Gradio within app.py, then restart the service and wait for initialization to complete.