Multimodal Models

A picture is worth a thousand words (or at least a thousand tokens). With NobodyWho, you can easily provide image and audio information directly to a multimodal LLM.

info

This is about models that natively ingest images and audio - no transcription step involved. That matters for audio in particular: the model hears the raw sound, not just words that were said, so it can react to tone of voice, music, or other non-speech noises. If you only need to convert speech to text, see Speech-to-Text instead. If you need to generate spoken audio from text, see Text-to-Speech.

Choosing a model

Not all models have built-in image and audio capabilities. Generally, you will need two parts for making this work:

1. Multimodal LLM, so the LLM can consume image-tokens or/and audio-tokens
2. Projection model, which converts images to image-tokens or/and audio to audio-tokens

To find such a model, refer to the HuggingFace Image-Text-to-Text section and Audio-Text-to-Text. Some models like Gemma 4 even manage both! Usually, the projection model then includes mmproj in its name.

If you are unsure which ones to pick, or just want a reasonable default, you can try Gemma 4 with its BF16 projection model, which can do both image and audio.

With the downloaded GGUFs, you can simply add the projection model as:

from nobodywho import Model, Chat

model = Model("./vision-model.gguf", projection_model_path="./projection_model.gguf")
chat = Chat(
    model, system_prompt="You are a helpful assistant, that can hear and see stuff!"
)

info

The language model and projection model have to fit together, as they are trained together! Unfortunately you can't just take projection model and a LLM that you like and expect them to work together.

Composing a prompt object

With the model configured, all that is left is to compose the prompt and send it to the model. That is done through the Prompt object.

from nobodywho import Audio, Image, Prompt, Text

prompt = Prompt([
    Text("Tell me what you see in the image and what you hear in the audio."),
    Image("./dog.png"),
    Audio("./sound.mp3")
])

chat.ask(prompt).completed() # It's a dog and a penguin!

That should be it! Beware though, that consuming images and audio can quickly drain the context, and larger context sizes may be needed for smooth usage.

Media in a message list

Prompt is for ask(). When you pass a whole conversation to complete(), the same interleaving is expressed as a list of content parts — the shape the OpenAI and Anthropic libraries use:

chat.complete([
    {"role": "user", "content": [
        {"type": "text",  "text": "Tell me what you see in the image."},
        {"type": "image", "path": "./dog.png"},
        {"type": "text",  "text": "Answer in one word."},
    ]},
]).completed()

Parts are {"type": "text", "text": ...}, {"type": "image", "path": ...} and {"type": "audio", "path": ...}. The order is the order the model sees them in. A plain string stays valid wherever a content list is accepted, so text-only messages need no change.

text, image and audio are reserved part types. A content list using none of them is passed to the chat template untouched, which is what models finetuned on structured turns expect.

Media in a saved conversation

get_chat_history() records the file path of every image and audio clip in the content part it belongs to, so a conversation containing media can be saved and replayed later:

import json

# save it, maybe to a save file
history = json.dumps(chat.get_chat_history())

# ... later, in a new process ...
chat.complete(json.loads(history) + [
    {"role": "user", "content": "What colour was the dog?"}
])

complete() re-reads each file, so the model sees the images and audio again rather than a conversation with holes in it. The files therefore have to still be where they were — if one cannot be read, complete() fails instead of quietly answering without it.