Mistral Medium 3.5 128B与vLLM集成构建高性能AI服务的完整方案【免费下载链接】Mistral-Medium-3.5-128B项目地址: https://ai.gitcode.com/hf_mirrors/mistralai/Mistral-Medium-3.5-128BMistral Medium 3.5 128B是一款具有1280亿参数的强大AI模型支持256k上下文窗口能够处理指令遵循、推理和编码等多种任务。将其与vLLM集成可以构建高性能的AI服务满足生产环境的需求。本文将详细介绍如何实现Mistral Medium 3.5 128B与vLLM的集成包括安装、部署和优化等方面的内容。为什么选择Mistral Medium 3.5 128B与vLLM集成Mistral Medium 3.5 128B具有以下核心优势强大的性能在各项基准测试中表现优异如在τ³-Telecom上达到91.4%的得分在SWE-Bench Verified上达到77.6%的得分。多模态能力支持文本和图像输入输出文本内容。可配置的推理模式可以根据需求在快速即时回复模式和推理模式之间切换在需要时通过测试时计算提升性能。vLLM是一个高性能的LLM服务库具有以下特点高吞吐量采用PagedAttention技术有效提高内存利用率和吞吐量。低延迟优化的推理引擎减少响应时间。易于部署提供简单易用的API和部署工具。将Mistral Medium 3.5 128B与vLLM集成可以充分发挥两者的优势构建高性能、低延迟的AI服务。准备工作安装vLLM和相关依赖在开始集成之前需要安装vLLM和相关依赖。以下是详细的安装步骤安装vLLM确保安装vLLM的 nightly 版本uv pip install -U vllm \ --torch-backendauto \ --extra-index-url https://wheels.vllm.ai/nightly验证安装安装完成后验证相关库的版本python -c import mistral_common; print(mistral_common.__version__) python -c import transformers; print(transformers.__version__)确保mistral_common 1.11.1和transformers 5.4.0。获取模型通过以下命令克隆Mistral Medium 3.5 128B模型仓库git clone https://gitcode.com/hf_mirrors/mistralai/Mistral-Medium-3.5-128B部署vLLM服务启动Mistral Medium 3.5 128B使用vLLM部署Mistral Medium 3.5 128B模型推荐使用服务器/客户端设置。以下是启动服务器的命令vllm serve mistralai/Mistral-Medium-3.5-128B --tensor-parallel-size 8 \ --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral --max_num_batched_tokens 16384 --max_num_seqs 128 \ --gpu_memory_utilization 0.8其中--tensor-parallel-size 8指定了使用8个GPU进行张量并行根据实际GPU数量进行调整。--gpu_memory_utilization 0.8设置GPU内存利用率为80%可以根据实际情况调整。与vLLM交互使用API调用模型部署完成后可以通过API与vLLM服务进行交互。以下是使用Python调用vLLM服务的示例代码指令遵循示例from datetime import datetime, timedelta from huggingface_hub import hf_hub_download from openai import OpenAI # Modify OpenAIs API key and API base to use vLLMs API server. openai_api_key EMPTY openai_api_base http://localhost:8000/v1 REASONING_EFFORT none # Toggle reasoning with high. match REASONING_EFFORT: case none: TEMP 0.1 TOP_P None case high: TEMP 0.7 TOP_P 0.95 case _: raise ValueError(Only REASONING_EFFORT in [none, high] are supported.) client OpenAI( api_keyopenai_api_key, base_urlopenai_api_base, ) models client.models.list() model models.data[0].id def load_system_prompt(repo_id: str, filename: str) - str: file_path hf_hub_download(repo_idrepo_id, filenamefilename) with open(file_path, r) as file: system_prompt file.read() today datetime.today().strftime(%Y-%m-%d) yesterday (datetime.today() - timedelta(days1)).strftime(%Y-%m-%d) model_name repo_id.split(/)[-1] return system_prompt.format(namemodel_name, todaytoday, yesterdayyesterday) SYSTEM_PROMPT load_system_prompt(model, SYSTEM_PROMPT.txt) messages [ {role: system, content: SYSTEM_PROMPT}, { role: user, content: Write me a sentence where every word starts with the next letter in the alphabet - start with a and end with z., }, ] response client.chat.completions.create( modelmodel, messagesmessages, reasoning_effortREASONING_EFFORT, temperatureTEMP, top_pTOP_P, ) print() print(fRequest with {REASONING_EFFORT}, {TEMP} and {TOP_P}.) print() print(REASONING) print(~~~~~~~~~) print(response.choices[0].message.reasoning) print() print(CONTENT) print(~~~~~~~) print(response.choices[0].message.content)工具调用示例Mistral Medium 3.5 128B支持工具调用可以通过API实现与外部工具的集成。以下是一个使用计算器工具的示例import json from datetime import datetime, timedelta from openai import OpenAI from huggingface_hub import hf_hub_download # Modify OpenAIs API key and API base to use vLLMs API server. openai_api_key EMPTY openai_api_base http://localhost:8000/v1 REASONING_EFFORT high # Toggle reasoning with high. match REASONING_EFFORT: case none: TEMP 0.1 TOP_P None case high: TEMP 0.7 TOP_P 0.95 case _: raise ValueError(Only REASONING_EFFORT in [none, high] are supported.) client OpenAI( api_keyopenai_api_key, base_urlopenai_api_base, ) models client.models.list() model models.data[0].id def load_system_prompt(repo_id: str, filename: str) - str: file_path hf_hub_download(repo_idrepo_id, filenamefilename) with open(file_path, r) as file: system_prompt file.read() today datetime.today().strftime(%Y-%m-%d) yesterday (datetime.today() - timedelta(days1)).strftime(%Y-%m-%d) model_name repo_id.split(/)[-1] return system_prompt.format(namemodel_name, todaytoday, yesterdayyesterday) SYSTEM_PROMPT load_system_prompt(model, SYSTEM_PROMPT.txt) def my_calculator(expression: str) - str: return str(eval(expression)) tools [ { type: function, function: { name: my_calculator, description: A calculator that can evaluate a mathematical expression., parameters: { type: object, properties: { expression: { type: string, description: The mathematical expression to evaluate., }, }, required: [expression], }, }, }, ] messages [ {role: system, content: SYSTEM_PROMPT}, { role: user, content: What is 2 2 * 2?, }, ] response client.chat.completions.create( modelmodel, messagesmessages, toolstools, tool_choiceauto, reasoning_effortREASONING_EFFORT, temperatureTEMP, top_pTOP_P, ) tool_calls response.choices[0].message.tool_calls results [] for tool_call in tool_calls: function_name tool_call.function.name function_args tool_call.function.arguments if function_name my_calculator: result my_calculator(**json.loads(function_args)) results.append(result) messages.append({role: assistant, tool_calls: tool_calls}) for tool_call, result in zip(tool_calls, results): messages.append( { role: tool, tool_call_id: tool_call.id, name: tool_call.function.name, content: result, } ) response client.chat.completions.create( modelmodel, messagesmessages, reasoning_effortREASONING_EFFORT, temperatureTEMP, top_pTOP_P, ) print() print(fRequest with {REASONING_EFFORT}, {TEMP} and {TOP_P}.) print() print(REASONING) print(~~~~~~~~~) print(response.choices[0].message.reasoning) print() print(CONTENT) print(~~~~~~~) print(response.choices[0].message.content)性能优化提升Mistral Medium 3.5 128B与vLLM集成的效率为了提升集成后的性能可以从以下几个方面进行优化调整推理参数根据任务需求调整推理参数如reasoning_effort、temperature和top_p等。对于复杂任务使用reasoning_efforthigh可以提升推理性能对于简单任务使用reasoning_effortnone可以提高响应速度。优化GPU资源利用通过调整--tensor-parallel-size和--gpu_memory_utilization等参数优化GPU资源的利用。根据实际GPU数量和内存大小合理设置并行度和内存利用率避免资源浪费。使用EAGLE模型为了加速本地推理可以使用Mistral发布的EAGLE模型。EAGLE模型针对vLLM和SGLang进行了优化可以显著提升推理速度。更多信息请参考EAGLE模型。常见问题解决集成过程中可能遇到的问题及解决方案长上下文性能下降如果遇到长上下文性能下降的问题可能是由于Transformers配置不正确导致的。请确保使用最新的配置修复已在commit中引入。依赖版本问题确保mistral_common 1.11.1和transformers 5.4.0。如果版本过低可能会导致功能异常或性能问题。可以通过以下命令升级依赖uv pip install -U mistral_common transformers模型加载失败如果模型加载失败可能是由于模型文件不完整或损坏。请检查模型文件是否完整或重新克隆模型仓库。总结构建高性能AI服务的最佳实践通过本文的介绍我们了解了如何将Mistral Medium 3.5 128B与vLLM集成构建高性能的AI服务。以下是最佳实践总结选择合适的工具使用vLLM作为推理引擎充分发挥其高吞吐量和低延迟的优势。优化推理参数根据任务需求调整推理参数平衡性能和速度。合理利用GPU资源根据实际GPU配置优化并行度和内存利用率。使用优化模型对于本地部署优先使用EAGLE模型提升推理速度。及时更新配置和依赖确保使用最新的模型配置和依赖版本避免已知问题。通过遵循这些最佳实践可以构建稳定、高效的AI服务满足各种应用场景的需求。参考资料Mistral Medium 3.5 128B模型仓库vLLM官方文档EAGLE模型Mistral Medium 3.5 128B博客【免费下载链接】Mistral-Medium-3.5-128B项目地址: https://ai.gitcode.com/hf_mirrors/mistralai/Mistral-Medium-3.5-128B创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考