Multi-modal MCP: images, audio and generated images in tools

Who this is for and what you can do after reading

This is for developers who want MCP tools to process images and audio, or to return generated images. After reading, you can design the inputs and outputs of multi-modal tools, follow how the image analysis, audio transcription and image generation samples are written, and recognize which parts are placeholders that should not go live as they are.

What a multi-modal MCP implementation is made of

A multi-modal implementation typically has four parts^[1]:

• Modal-specific parsers that convert different media types into formats the model can process^[1].
• Modal-specific tools built for one kind of data, such as image analysis or audio processing^[1].
• Unified context management that maintains context across modalities^[1].
• Response generation that can produce replies containing more than one modality^[1].

Context management is the part most often overlooked in image-and-text conversations. A user sends a picture and then asks a text question. The second answer still has to refer to the picture from the first turn. If each tool handles its input in isolation, that reference breaks.

Image analysis: how the tool takes parameters and returns results

The C# sample defines a tool named imageAnalysis. It requires imageUrl and accepts an optional analysisType with the values general, objects, text or faces, defaulting to general^[1]. The IImageAnalysisService in the sample is hypothetical, so you must implement it against a real image recognition service^[1].

Execution has three steps: download the image, dispatch to the method matching analysisType, and serialize the result to JSON^[1]:

        public async Task<ToolResponse> ExecuteAsync(IDictionary<string, object> parameters)
        {
            // Extract parameters
            string imageUrl = parameters["imageUrl"].ToString();
            string analysisType = parameters.ContainsKey("analysisType") 
                ? parameters["analysisType"].ToString() 
                : "general";
              // Download or access the image
            byte[] imageData = await DownloadImageAsync(imageUrl);
            
            // Analyze based on the requested analysis type
            var analysisResult = analysisType switch
            {
                "objects" => await _imageService.DetectObjectsAsync(imageData),                "text" => await _imageService.RecognizeTextAsync(imageData),
                "faces" => await _imageService.DetectFacesAsync(imageData),
                _ => await _imageService.AnalyzeGeneralAsync(imageData) // Default general analysis
            };

One design choice deserves attention. The analysis comes back as a JSON string inside a content item of type Text, not as an image or structured content type^[1]. Clients that only need text results can use it directly. Clients that expect structured data have to parse that JSON themselves.

The sample sets the server's maximum request size to 10 MB and lists the supported content types: image/jpeg, image/png, text/plain and application/json^[1]. A request body above the limit should be rejected at the entry point rather than passed to a tool.

Audio transcription: a choice between a URL and Base64 data

This tool follows a common pattern: two kinds of input, one output path. Whether the caller supplies an address or the data itself, the transcription logic receives a byte array. Keeping the difference between input sources inside the tool is the benefit of this pattern.

The Java sample implements a tool named audioTranscription. It accepts either audioUrl or Base64-encoded audioData, and the language parameter defaults to en-US^[1]. Execution checks audioUrl first, then audioData. If neither is present, it throws "Either audioUrl or audioData must be provided"^[1]:

    public ToolResponse execute(ToolRequest request) {
        try {
            byte[] audioData;
            String language = request.getParameters().has("language") ? 
                request.getParameters().get("language").asText() : "en-US";
                
            // Get audio either from URL or direct data
            if (request.getParameters().has("audioUrl")) {
                String audioUrl = request.getParameters().get("audioUrl").asText();
                audioData = downloadAudio(audioUrl);
            } else if (request.getParameters().has("audioData")) {
                String base64Audio = request.getParameters().get("audioData").asText();
                audioData = Base64.getDecoder().decode(base64Audio);
            } else {
                throw new ToolExecutionException("Either audioUrl or audioData must be provided");
            }

Two things in this code need fixing first. The downloadAudio method is a placeholder that returns an empty byte array, so the URL branch does nothing useful yet^[1]. The getSchema method calls Arrays.asList, but the sample's imports do not include java.util.Arrays, so add that import before compiling^[1].

The sample sets the server's maximum request size to 20 MB, with a comment saying the room is for audio and video content^[1]. Audio files are usually much larger than text, so set the limit against the sizes you actually expect.

Image generation: returning a Base64 PNG

Returning the image inside JSON means the caller can use the result immediately, without fetching it from another address. The cost is a large response that must be transmitted and decoded every time. The sample takes this route, so a bigger generated image means a bigger response.

The Python imageGeneration tool takes prompt as required, style with the values realistic, artistic, cartoon and sketch (default realistic), and width and height (both defaulting to 512)^[1]. It encodes the generated image as a Base64 PNG and returns it in a result object together with format, width, height, generationPrompt and style^[1].

The real generation logic belongs in _generate_image, and the sample's version is a placeholder that paints a solid-colour image. The docstring says an actual image generation API should be called there^[1]:

    async def _generate_image(self, prompt: str, style: str, width: int, height: int) -> Image.Image:
        """
        This would call an actual image generation API
        Simplified placeholder implementation
        """
        # Return a placeholder image or call actual image generation API
        # For this example, we'll create a simple colored image
        image = Image.new('RGB', (width, height), color=(73, 109, 137))
        return image

The MultiModalResponseHandler assembles multi-modal replies. When generate_images is true, it calls imageGeneration once for each prompt and collects the Base64 data into an images list^[1]:

        if generate_images and image_prompts:
            for prompt in image_prompts:
                image_result = await self.client.execute_tool(
                    "imageGeneration",
                    {
                        "prompt": prompt,
                        "style": "realistic",
                        "width": 512,
                        "height": 512
                    }
                )
                
                response["images"].append({
                    "imageData": image_result.result["imageBase64"],
                    "format": image_result.result["format"],
                    "prompt": prompt
                })

The main function also registers AudioAnalysisTool and VideoFrameExtractionTool, but the sample does not define either class^[1]. Copying the whole file as it stands will fail, so you need to write those implementations yourself.

URL or Base64 for input

The same kind of image or audio can be sent in two ways in these samples, and the choice depends on the setting.

With a URL, the server fetches the file itself. The request stays small, but the server must be able to reach the address, and an expired or login-protected link makes the download step fail. This is why the C# sample calls DownloadImageAsync and does not add extra handling for failures.

With Base64, the file content travels inside the request. Base64 encodes every three bytes as four characters, so the payload grows by about a third. Large files reach the request-size limit quickly. The Java sample offers both options in one tool so the caller can choose by situation^[1].

A checklist for new multi-modal tools

Before writing a new multi-modal tool, check the following against the samples:

• Put the modality and format in the parameter definition. The C# sample lists the allowed content types in the server options, so clients know what they can send^[1].
• Set the request-size limit against the real file sizes, and reject oversize requests at the entry point^[1].
• Decide the output content type early. Plain text, structured JSON and Base64 images each require different amounts of decoding and display work on the client.
• Make error messages specific. The Java sample gives an explicit message when both inputs are missing, which is far more useful than a generic failure^[1].
• Make placeholders visible. downloadAudio and _generate_image in the samples are placeholders that must be replaced before launch, or the tool appears to work while doing nothing^[1].

Common problems

• The client cannot display images: all three samples return text or Base64 strings, not image content types. The client has to decode the data and show it.
• Generated images are all one colour: this is the placeholder implementation, not a failed generation. Real output appears only after you connect a real generation service^[1].
• Audio results are empty: downloadAudio currently returns an empty byte array^[1]. The fault lies in the placeholder code, not in the transcription service.
• Compilation fails on Arrays: add the java.util.Arrays import^[1].
• Requests are rejected: check whether the body exceeds the server's configured maximum^[1].

Next steps

OAuth 2 is the next topic to read. Once a multi-modal tool calls an external image or speech service, authentication becomes unavoidable^[1].

Adapted from lesson material in Microsoft's open-source course *MCP for Beginners*, covering multi-modal integration in its advanced topics^[1], released under the MIT License.