Deploying MiniCPM-o 2.6 with FastAPI

Who this is for

This guide is for developers who want to wrap a multimodal model, one that handles images, video and text, in a local API that their own programs can call. By the end you should be able to start a FastAPI server for MiniCPM-o 2.6 and send single-image, multi-turn, multi-image and video requests from Python.

MiniCPM-o 2.6 and this server

MiniCPM-o 2.6 is a multimodal model from OpenBMB. It can be downloaded from ModelScope, and its model page is at modelscope.cn/models/OpenBMB/MiniCPM-o-2_6^[2]. This setup wraps the model in a FastAPI application, so clients only send JSON requests to one address^[1].

Environment

The test environment was Ubuntu 22.04, Python 3.12.3, CUDA 12.1 and PyTorch 2.3.0^[1]:

----------------
ubuntu 22.04
Python 3.12.3
cuda 12.1
pytorch 2.3.0
----------------

First switch pip to the Tsinghua mirror and install modelscope^[1]:

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

pip install modelscope==1.20.0

There are two ways to install the dependencies. One is to download the official requirements file from OpenBMB's GitHub repository and install from it in one step^[1]:

pip install -r requirements_o2.6.txt

The other is a manual install^[1]:

pip install Pillow==10.1.0 torch==2.3.1 torchaudio==2.3.1 torchvision==0.18.1 transformers==4.44.2 sentencepiece==0.2.0 vector-quantize-pytorch==1.18.5 vocos==0.1.0 accelerate==1.2.1 timm==0.9.10 soundfile==0.12.1 librosa==0.9.0 decord moviepy fastapi uvicorn python-multipart

After installing, check for missing packages or errors^[1]. Use one of the two install methods, not a mix of them, so the versions stay consistent.

Download the model

Download the model with snapshot_download from modelscope. The first argument is the model name, and cache_dir is the download location^[1]:

from modelscope import snapshot_download
# cache_dir记得修改为自己的目录路径
model_dir = snapshot_download('OpenBMB/MiniCPM-o-2_6', cache_dir='/root/autodl-tmp', revision='master')

Change cache_dir to a directory of your own^[1].

Start the FastAPI server

The server exposes one POST route at /, which reads a JSON body^[1]. The body carries prompt, an optional video_path, and a history list of messages^[1]. Base64 images in history are decoded to RGB; if decoding fails, the server prints an error and skips that image^[1].

For video, frames are read with decord at about one frame per second. When there are more than MAX_NUM_FRAMES frames, the server samples them uniformly^[1]:

MAX_NUM_FRAMES = 64  # 视频处理的最大帧数，如果显存不足可以设置更小的值

The chat call sets use_image_id to False and max_slice_nums to 2. A comment notes that the second value can be 1 when memory is short and the video resolution is above 448×448^[1]:

            "max_slice_nums": 2  # 如果显存不足且视频分辨率>448*448时使用1

After each request the server empties the CUDA cache^[1]. Responses include response, status (200 on success) and time; errors return an error message with status 500^[1]. Each request's time, prompt and response are also printed to the log^[1].

The entry point loads the model and tokenizer, then starts uvicorn on port 6006^[1]. workers=1 means a single worker process, so every request is handled by the same loaded model^[1]:

if __name__ == '__main__':
    # 设置随机种子
    torch.manual_seed(1000)
    
    # 加载模型和分词器
    model = AutoModel.from_pretrained(
        'autodl-tmp/OpenBMB/MiniCPM-o-2_6',  # 替换为自己的具体路径
        trust_remote_code=True,
        attn_implementation='sdpa', 
        torch_dtype=torch.bfloat16
    )
    model = model.eval().cuda()  # 将模型设置为评估模式并移到GPU
    
    tokenizer = AutoTokenizer.from_pretrained(
        'autodl-tmp/OpenBMB/MiniCPM-o-2_6',  # 替换为自己的具体路径
        trust_remote_code=True
    )
    
    # 启动FastAPI服务器
    uvicorn.run(app, host='0.0.0.0', port=6006, workers=1)

Replace the two autodl-tmp/OpenBMB/MiniCPM-o-2_6 paths with the location of your own download, then start the service^[1]:

# cd到自己存放代码的文件的路径，如 cd /root/autodl-tmp
python api_server.py

Once it starts, the service listens on port 6006 by default^[1].

Single image, single turn

The client's send_request posts JSON to http://localhost:6006. It prints the answer when the status is 200 and the error otherwise^[1]:

def send_request(data):
    """发送请求并处理响应"""
    response = requests.post("http://localhost:6006", json=data)
    try:
        result = response.json()
        if result.get('status') == 200:
            print(f"MiniCPM-o: {result.get('response')}")
        else:
            print(f"错误: {result.get('error')}")
    except Exception as e:
        print(f"解析响应时出错: {str(e)}")
        print(f"原始响应: {response.text}")

To read a local image, encode it as a base64 data URI^[1]:

with open("autodl-tmp/input/1-01.jpg", 'rb') as f: # 注意修改为自己输入图片的存放路径
    image_base64 = f"data:image/jpeg;base64,{base64.b64encode(f.read()).decode('utf-8')}"

The data URI and the question go together in one user message's content list inside history^[1].

Single image, multiple turns

After the first answer, append the assistant's reply to the message list, add the next question, and send the whole list again^[1]:

    msgs.append({'role': 'assistant', 'content': [first_response]})

Comparing several images

Several images can go into the same user message with the question. The example reads two images, places both in the content list, and puts the question last^[1]:

with open("autodl-tmp/input/1-02.jpg", 'rb') as f: # 在这里修改
    image2_base64 = f"data:image/jpeg;base64,{base64.b64encode(f.read()).decode('utf-8')}"

Video input

For video, set video_path to a local .mp4 file. The example checks that the file exists before it sends the request^[1]:

video_path = os.path.join(os.path.expanduser("~"), "autodl-tmp", "input", "1-04.mp4") # 在这里修改

Next steps

More model details are on the ModelScope model page^[2], and dependency versions should follow the official requirements file^[1].

Adapted from the tutorial "MiniCPM-o 2.6 FastApi部署调用" in Datawhale's open-source project self-llm^[1], released under the Apache-2.0 License.