Back to blog

Kairos: building a multimodal model with LFM2.5 and Kimi-K2.6

Fredy Rivera

Fredy Rivera

Founder

25 min read
AIMultimodalLoRAFine-TuningLLMLFM2.5Kimi-K2.6Open Source

Kairos (from the Greek καιρός): the opportune, fitting or decisive moment to do something. In ancient Greece it represented a qualitative time (the value of the instant) as opposed to chronos, the quantitative, linear time of clocks.

The Greeks personified this idea in a minor deity of opportunity, depicted with wings on the feet, a lock of hair in the front and a bald nape. This symbolized that opportunities must be seized on the fly, because they pass quickly and, once they pass, there's nothing left to hold on to. In Christian theology it's also defined as the perfect, opportune time designated by God for his will or a salvation event to take place.

Today, one of the most necessary features for any state-of-the-art language model is having a vision module: one that understands the context of the scene in an image and its main components, and can act accordingly.

Models like Claude Fable/Mythos 5 from Anthropic and the GPT-5.6 family from OpenAI already have unprecedented multimodal capabilities, along with reasoning capabilities over these modalities and their use in agentic loops.

Alongside this, Chinese open-source labs like Moonshot AI (which, from the release of Kimi K2.5 up to its most recent Kimi K3, includes vision features applied to the agentic environment) and Alibaba, with the QwenVL family of models that pioneered popularizing multimodality in small open-weights models, or American labs like Thinking Machines Lab, which with its Inkling family of models presents native image, video and audio multimodality.

All of this as of mid-August 2026. If you read this at another date, these models are probably already outdated.

With all this in mind, how do you build this kind of model? Looking into papers like LLaVA, Kimi-VL Technical Report and Kimi K2.5: Visual Agentic Intelligence, there's a pattern that repeats and keeps delivering excellent results: a Vision Encoder is used or pre-trained from scratch, in many cases based on Vision Transformers or variations of them, which will be responsible for extracting the image features, that is, what composes it, what objects, people or animals are in it and their position, among other things. These features are then processed by a projector that translates them into the embedding space of the language model (LLM).

In LLaVA they used CLIP-ViT-Large from OpenAI as their Vision Encoder and an MLP (Multi-Layer Perceptron) network as the projector into the LLM embedding space, which in this case was the Vicuna family of models. Training was divided into two model alignment stages. In the first stage, the goal was to align the projector so it correctly translated the features generated by the Vision Encoder into the LLM embeddings, so the latter would learn to describe what the Vision Encoder returned. In the second stage, the goal was for the model, now able to describe these features, to use them in a conversational context.

LLaVA also introduced the LLaVA Visual Instruct CC3M 595K Pretrain and LLaVA Visual Instruct 150K datasets, which became the standard in open-source projects for building multimodal models.

Recently, implementations like Kimi-VL or Kimi K2.5 have introduced innovations on the Vision Encoder side, allowing them to accept native image or video resolution, meaning the image doesn't need to be resized for the model to understand it, along with training innovations where they perform early fusion between modalities, that is, in both pre-training and post-training the model sees multimodal data, not just text, setting a precedent where they don't limit themselves to bolting a Vision Encoder onto an existing LLM. Even so, they follow the architecture proposed by LLaVA of Vision Encoder + Projector.

Note: Kimi-K2.5 uses a continual pre-training technique for multimodal alignment, along with more text data.

Another family of models, like Qwen-VL (its first generation), presented a different approach, implementing cross-attention between the text and vision modalities. In Qwen2-VL, they reformulated the architecture with 3D positional embeddings for video inputs, along with more innovations in the newer models of this family.

Since we understand which approach keeps delivering the best results, along with its simplicity, we decided to implement the LLaVA-based architecture (Vision Encoder + Projector), more specifically the implementation presented in Kimi-K2.5 (derived from Kimi-VL).

We reused the MoonViT-3D Vision Encoder (which was extracted from Kimi-K2.6) along with most of its projector architecture. It's worth clarifying that Kimi-K2.6 shares the same base architecture as Kimi-K2.5, mentioned earlier as the reference implementation, since it's a fine-tune of the latter focused on improving its vision and front-end design capabilities.

Alongside this, while researching LLM options to build Kairos, we laid out these parameters to select it:

  • It has to be easy to run, meaning its size shouldn't be restrictive for most computers on the planet.

  • Smart enough for its size.

  • It shouldn't already have a vision module implemented.

  • Reasoning capability.

  • And it should have been released recently.

With these parameters defined we found models like:

And after running some tests with these models and weighing those parameters, the one that best fulfilled them was LFM2.5-2.6B, even though it was the largest of the group. This is mainly because it showed better performance on reasoning tasks, in addition to being fast at inference.

Validations before building the model

Before starting to implement Kairos, we have to see what we're missing: special tokens to mark where the image content goes in the sequence, the model's embedding dimension, and making sure the forward pass reaches the whole model when there's a multimodal input, even if it produces garbage initially since it isn't aligned.

We started by validating the tokenizer; it turns out the Liquid AI team had already prepared it for image inputs, as seen in the cell we used to validate it:

Loading notebook…

This already simplifies the implementation for us, because we don't need to rewrite the tokenizer to include the special image tokens. Now let's validate the dimensionality and name of the embedding layers.

Loading notebook…

The hidden layers dimension is 2048, that's the output size the projector must aim for. Now let's validate that the forward pass reaches the whole model through the backbone, simulating multimodal inputs, and that the gradient in the backward pass flows only to the projector, respecting that the backbone and the vision encoder are frozen.

Loading notebook…

Building Kairos

With the validations done, we assembled the three components that make up a multimodal model in this architecture: the processor (responsible for tokenizing text and image into a single sequence), the model (vision tower + projector + LLM) and its config. We inherited the processor and the config almost directly from the Kimi-K2.5 scheme, so there's not much to tell there. Where there were decisions worth explaining is in the projector and in how we ended up training everything.

In the following figures we'll see the architecture diagrams of Kimi's and Kairos's projectors:

Figure 1: Kimi-K2.5 projector
Figure 1: Kimi-K2.5 projector
Figure 2: Kairos projector
Figure 2: Kairos projector

In addition to changing the output dimension to match the LLM's hidden layers, we also added a conditional normalization and scaling component at the output, because in internal tests we noticed that, without any limit, the projector found the cheapest shortcut to minimize the loss: instead of learning to emit directions aligned with the LLM embedding space, it simply inflated the magnitudes of its outputs (average norms of 139, with peaks of 1614, vs ~0.9 in real text embeddings). This behavior costs nothing in cross-entropy, but leaves the LLM receiving inputs out of the distribution it expects. To avoid it, we added an optional L2 cap (output_scale) that normalizes and rescales the projector's output to the mean norm of the LLM embeddings.

With the projector defined, the initial plan was to follow LLaVA's two-stage scheme: first align the projector with the LLM backbone frozen, and only afterwards train projector + LLM together.

In practice, this first stage showed a problem. The CE (cross-entropy, the standard loss metric for language models) went down fine, and in ablation studies comparing the same setup with and without image we found a clear +3.7 nats difference in favor of the model with image (the unit in which loss is measured when using the natural logarithm), which indicated that the model was indeed using the visual information to some extent.

But in free generation this translated into nothing: the image moved the logits without ever changing the argmax, that is, the tokens related to what the image showed gained some probability, but never enough to surpass the tokens the LLM already preferred on its own, so the argmax kept discarding them. The model never ended up describing what it saw, no matter the prefix or prompt we used to force it.

With this in mind, we decided to take inspiration from the idea of early fusion, where the projector and the LLM are aligned together from pre-training: instead of first going through a projector-only stage with the frozen backbone and then through a projector + LLM one, we aligned both parts from the start. But this posed another problem: we couldn't use the LLaVA dataset as-is, because the LFM LLM/template always opens with the tag and closes with the tag, and this dataset has no explicit reasoning traces. We ended up rewriting the LLaVA dataset with a distillation technique.

Creating a dataset for Kairos

To train Kairos we needed a multimodal dataset with explicit reasoning traces, and since the LLaVA dataset didn't have them, we decided to create it with a distillation technique. Distillation consists of using a teacher model, much larger and smarter, to generate the data with which a smaller model is trained. Instead of learning directly from the teacher, the small model learns from the examples it generates: in our case, the teacher saw an image and produced a step-by-step reasoning trace, along with its final answer. This way, a small model can imitate the behavior of a much more capable one without needing its size or its cost.

We had already applied distillation techniques in other projects, like Harvey, where we generated synthetic data for the legal domain. This time the goal was more scoped: regenerate the LLaVA-CC3M-Pretrain-595K dataset we mentioned before, adding explicit reasoning traces to it.

At first we were going to regenerate the full dataset, but due to capital and scale limitations we scoped the generation to 60,041 examples. To complete the rest, we added two more sources:

  • Kairos-websight: 2,295 examples generated over the HuggingFaceM4/WebSight dataset, where the model had to describe the content of a web page from its screenshot (Img2Code-style data).

  • Zebra-CoT: 54,021 examples extracted from the open multimodal-reasoning-lab/Zebra-CoT dataset, which already included multimodal reasoning traces.

The final dataset, Kairos-Multimodal-Reasoning, ended up with a total of 116,357 examples.

This whole generation process had a cost, and in the following table we detail the teacher models used, the tokens processed and the total generation cost.

Kairos-LLaVA

ModelInput TokensOutput TokensTotal TokensProviderTotal Cost
GPT 5.6 Luna39,880,68510,976,32250,857,007OpenAI$ 20.18
Inkling1,063,536917,2261,980,762Thinking Machines Lab (Via NVIDIA Build) *$ -
Qwen 3.6 27B828,8371,662,2672,491,104Aquiles-ai (Self Hosted via Modal)$ 9.09
Qwen 3.7 PlusN/AN/A9,290,000Fireworks$ 10.44
Total64,618,873$ 39.71

Kairos-WebSight

ModelInput TokensOutput TokensTotal TokensProviderTotal Cost
Inkling-NVFP4573,445607,4271,180,872Aquiles-ai (Self Hosted via Modal)$ 3.82
MiniMax-M3637,913347,839985,752MiniMax (Via NVIDIA Build) *$ -
deepseek-v4-flash-073160,95551,087112,042AnyAPI *$ -
Total2,278,666$ 3.82

Note: Providers marked with an asterisk (*) didn't require payment for the use of the models.

We left a playground where you can explore some examples of the dataset:

Training

In the following figures we show the complete model architecture and which components are trained depending on the script being run:

Figure 3: Full architecture training only the projector
Figure 3: Full architecture training only the projector
Figure 4: Full architecture training projector + LoRA
Figure 4: Full architecture training projector + LoRA

Note: The diagrams show the complete model architecture; the elements with a red outline are the ones being trained.

Training the projector via backbone

Earlier we mentioned that we were going to run 2 stages, but given the results of the first stage, where in free generation it didn't recognize the image, even though the ablation studies and the training loss showed that the projector did tend to contribute something, the model ended up deciding not to use it.

Here's the execution of the ablation studies and the free generation at the end of the training:

text
===== image ablation =====  eval loss with real image : 2.4963  eval loss without image   : 6.2070  difference                : 3.7107
===== embedding scale check (post-train) =====  image_embeds (n=512): norm mean 0.890  std 0.000  max 0.890  (L2 cap active on all samples)  text_embeds  (n=285): norm mean 0.879  std 0.205  max 1.217  image/text ratio   : 1.012  (healthy: order ~1; 0.0 expected after zero-init)
===== qualitative check (after training, trim_think=False) =====  [0] prompt tokens : 78      model         : The user wants me to describe an image. However, I don't see any image attached to the prompt. The prompt consists only of empty spaces and line breaks, indicating that no image was provided.
1.  **Analyze the Request:** The user says "Describe this image." but there is no image.
2.  **Check for Hidden Context:** Sometimes images are uploaded separately or the system might have context I'm not seeing. But based on the raw text provided, there is absolutely no image data.
3.  **Formulate Response:** I need to inform the user that no image was provided. I cannot describe something that isn      reference     : celebrity is july cover girl -- and she 's opening up about her marriage to actor and comedy film .  [1] prompt tokens : 78      model         : The user wants me to describe an image. However, I don't see any image attached to the prompt. The prompt consists only of empty spaces and line breaks, indicating that no image was provided.
1.  **Analyze the Request:** The user says "Describe this image." but there is no image.
2.  **Check for Hidden Context:** Sometimes images are uploaded separately or the system might have context I'm not seeing. But based on the raw text provided, there is absolutely no image data.
3.  **Formulate Response:** I need to inform the user that no image was provided. I cannot describe something that isn      reference     : lifeguards keep watch on a busy summer day  [2] prompt tokens : 78      model         : The user wants me to describe an image. However, I don't see any image attached to the prompt. The prompt consists only of empty spaces and line breaks, indicating that no image was provided.
1.  **Analyze the Request:** The user says "Describe this image." but there is no image.
2.  **Check for Hidden Context:** Sometimes images are uploaded separately or the system might have context I'm not seeing. But based on the raw text provided, there is absolutely no image data.
3.  **Formulate Response:** I need to inform the user that no image was provided. I cannot describe something that isn      reference     : person clears the final hurdle

It's clear that the model does receive the projector's input, because it describes it as a large chain of blank spaces and line breaks. At this scale (80k examples), the model should start showing signs of "seeing" something.

For example, in the blog post "GLM 5.2 with vision" from the baseten team, they take the zai-org/GLM-5.2 model and the Kimi-K2.6 vision encoder as a base (just like we did), and only train the projector via backbone, noting that with 66k examples and within the first 900 training steps the model already started to "see".

Given that GLM is a model of almost 1T parameters, with a smaller model and a larger test dataset we expected to see similar behavior.

There's an important nuance to that experiment: in the SFT they only used simple question-answer pairs, and the model didn't reason about the images; reasoning only appeared later, when they applied reinforcement learning. This reinforces our decision to jump straight to early fusion with a reasoning dataset: we wanted the model to reason and see from the start, without depending on an extra RL stage.

Training the projector + LLM

In the case of training the projector together with the LLM (with LoRA), to validate the hypothesis and due to capital limitations, we decided to run the training with only 30,000 examples of the Kairos-Multimodal-Reasoning dataset.

This gave us more interesting results: during the ablation evaluations we could see that, little by little, the model tended to use more of the projector's information when an image was present, and in free generation the model started to show signs of "seeing", although it tended to fail on the fine details of the image depending on the prompt it was given:

text
...{'loss': '1.238', 'grad_norm': '0.9984', 'learning_rate': '4.741e-05', 'epoch': '0.5333'}{'eval_loss': '1.146', 'eval_runtime': '65.29', 'eval_samples_per_second': '7.842', 'eval_steps_per_second': '0.98', 'epoch': '0.5333'}===== image ablation =====  eval loss with real image : 0.9775  eval loss without image   : 1.2774  difference                : 0.2999...{'loss': '1.2', 'grad_norm': '1.058', 'learning_rate': '1.08e-07', 'epoch': '0.9813'}{'eval_loss': '1.119', 'eval_runtime': '65.13', 'eval_samples_per_second': '7.861', 'eval_steps_per_second': '0.983', 'epoch': '1'}===== image ablation =====  eval loss with real image : 0.9519  eval loss without image   : 1.2612  difference                : 0.3093{'train_runtime': '4255', 'train_samples_per_second': '7.051', 'train_steps_per_second': '0.22', 'train_loss': '1.284', 'epoch': '1'}...===== image ablation =====  eval loss with real image : 0.9513  eval loss without image   : 1.2612  difference                : 0.3100
===== embedding scale check (post-stage2) =====  image_embeds (n=4013): norm mean 0.890  std 0.000  max 0.890  text_embeds  (n=1152): norm mean 0.951  std 0.136  max 1.242  image/text ratio   : 0.936  (healthy: order ~1; 0.0 expected after zero-init)
===== qualitative check (after stage-2) =====  [0] prompt          : What is the name of the liquid contained in the bottle on the left in the image?      prompt tokens   : 916      model reasoning : The user wants to identify the liquid in the bottle on the left from the provided image. I need to locate the bottle and observe its contents.      ref   reasoning : To identify the liquid, I need to read the label on the bottle.      model answer    : The liquid contained in the bottle on the left is water.      ref   answer    : spearmint  [1] prompt          : Where is the touching fingertip positioned relative to the doctor displayed on the smartphone?      prompt tokens   : 91      model reasoning : The user is asking about the relative position of a touching fingertip and a doctor on a smartphone. I need to locate these elements in the provided image.      ref   reasoning : What stands out first is the fingertip extending up from the lower right, overlapping the lower portion of the phone screen. The doctor's face remains higher and behind it, near the center of the display.      model answer    : The touching fingertip is positioned on the doctor's hand.      ref   answer    : The fingertip is in front of and below the doctor's image, covering the lower part of the screen.  [2] prompt          : What is the shortest path from node 0 to node 5 in the provided weighted graph?      prompt tokens   : 2733      model reasoning : The user wants to find the shortest path from node 0 to node 5 in the given weighted graph. I need to use Dijkstra's algorithm to solve this problem. The algorithm works by maintaining a set of visited nodes and their shortest distances from the source node. It uses a priority queue to select the un      ref   reasoning : The problem asks for the shortest path in a weighted, undirected graph from a starting node (0) to a destination node (5). Dijkstra's algorithm is the standard method for solving this type of problem, as it efficiently finds the shortest path from a single source to all other nodes in a graph with n      model answer    : The shortest path from node 0 to node 5 has a length of 11. The path is [0, 1, 5].      ref   answer    : The shortest path from node 0 to node 5 has a total length of 10. The path is [0, 2, 3, 5].

Being honest, this was the result expected from doing backbone training with the LLM frozen, evidently without explicit reasoning.

Tests and findings

Given the results obtained in the projector + LLM training, we decided to run tests with this checkpoint to see how much the model had generalized with this small amount of data. When running some tests we noticed that, when passing it prompts like "Tell me anything you notice, whatever it is." or "What's here?", the model tended to fall into the phenomenon we noticed before, where it only saw a big blank space. We also tried prompts like "What's in this image?", thinking the model had anchored to keywords to activate visual interpretation, but even so, it tended to misinterpret or completely hallucinate components that weren't in the image, which raised our alarms about what was happening.

Running ablation studies and reviewing the dataset that was passed to the LLM during training, we reached the following observations:

  • The keyword anchoring hypothesis was ruled out: "What's in this image?" (keyword without content) gave exactly the same result as "Describe this." (without keyword or content). The keyword alone does nothing when there's no content to name.

  • The case of "Tell me what's here." doesn't fit cleanly: only 25% of samples never diverge, well below even the no-anchor bucket with real prompts (67%). And the three that did diverge did so at the exact same step (38, min=median=max), unlike "What do you see?" (min=9, max=24), where the divergence point varies according to each image's content. An identical value in all three suggests something structural (a logit tie or a template limit), not real grounding activating.

  • The projector encodes fine detail correctly (masking test with localized negative cosine) and the image pipeline is clean (two independent verifications). The problem isn't in visual extraction.

  • The LoRA did learn to use the image, but conditioned on the prompt naming something specific to look for, not on the presence of words like "image". Without named content in the prompt, the model falls deterministically to the LFM2.5-2.6B prior, indistinguishable from whether the image is present or not.

  • The dataset reinforces this behavior: all the questions implicitly assume there's something to look for, and there are no examples with generic open prompts supervised with a real answer describing the image (the "blank canvas" pattern was never counteracted). That's why no volume of the current questions would fix this.

  • Outside the dataset's visual distribution (free photography vs curated benchmarks), the model fails much harder even with prompts that name content: the LoRA r=16 capacity with 30k examples is insufficient to generalize visual composition, only seen patterns.

To illustrate the findings, in the following figure there's an image of a girl riding a bike that will serve as visual input to the model:

Figure 5: Image of a girl riding a bike
Figure 5: Image of a girl riding a bike

In the following cell, we passed that image to Kairos with prompts ranging from "What's in this image?" to "Is there a bicycle in the scene? If so, describe the scene, its location, and who is riding it (obviously only if someone is using it; otherwise, don't mention it).", testing that the prompt format heavily conditions the model's generation.

Loading notebook…

Note that when asked to describe the image, it almost completely hallucinates its answer: "The image shows a close-up of a hand holding a small, dark, rounded object, likely a small stone or pebble, with the hand resting on top of it."

And when asked to look for whether there's a bicycle and describe the scene, it does manage to indicate that there's a bicycle and its surroundings, but hallucinates that it's parked to the side with no one riding it: "Yes, there is a bicycle in the scene. It is parked on the left side of the road, positioned near a sidewalk and a building. The bicycle is stationary, and no one is riding it. The scene appears to be a roadside setting, likely in an urban or suburban area."

While with a prompt like "Tell me anything you notice, whatever it is.", it falls into saying it sees a big blank space without being able to describe anything: "I notice this is a blank white image with no visible text, symbols, or other distinct visual elements. It is completely empty."

Conclusions

Building Kairos left us with more learnings than a functional model, and for the size of the experiment, we already consider that a good result. We validated the architecture proposed by LLaVA and refined by Kimi-K2.5, and we confirmed that early fusion was the right decision: aligning projector and LLM from the start with a reasoning dataset avoided the first stage problem, where the model received the image but simply ignored it.

The key findings:

  • Training only the projector with the LLM frozen isn't enough: the image moved the logits (+3.7 nats in ablation) but never the argmax. Joint alignment is necessary.

  • The model learns to use the image, but conditioned on the prompt: if it doesn't name something concrete to look for, the model falls deterministically to the LFM2.5-2.6B prior and describes a blank space. The current dataset reinforces that behavior, so the next iteration needs generic open prompts with real answers.

  • The LoRA r=16 capacity with 30k examples only reaches seen patterns: outside the dataset distribution, the model fails even with prompts that name content.

The main limitation, being honest, was capital: we couldn't regenerate the 595k samples of the LLaVA dataset (we stayed at 60,041), we couldn't train beyond 30k examples in the projector + LLM phase, nor run more epochs. Everything that follows is, to a large extent, a scaling problem: more data, well-supervised generic prompts and more training.

For the next iterations, the first thing will be to expand the dataset with free-generation examples, that is, open prompts with real answers describing the image, to prevent the model from falling into the behavior of seeing nothing. Alongside this, we're evaluating doing a full fine-tuning and the data scaling it implies, since in these kinds of cases LoRA is probably suboptimal. And not only with visual data: we plan to also include agent data or similar, to prevent the model from suffering catastrophic forgetting by training only with visual data.

Finally, we're evaluating changing the base LLM to options like MiniCPM5-1B or Granite 4.1 3B, as proposals for on-device agentic multimodal models.

The Kairos checkpoints aren't a competent model: they're experimental artifacts that validated the approach and defined precisely what the next iteration needs. And that one we won't start blindly anymore: we seized our kairos, the opportune moment, to learn the path before investing in scaling it.

References

Papers and architecture

Models

Datasets

Blogs and providers

This project

Author's previous projects

Figures

Kairos: building a multimodal model with LFM2.5 and Kimi-K2.6