Cache-to-Cache
Cache-to-Cache (C2C) is a framework that enables direct semantic communication between Large Language Models by projecting and fusing their KV-Caches, bypassing text generation. It solves the problem of inefficient and lossy text-based communication between models, achieving higher accuracy and faster inference. It is for developers and researchers working with LLMs who want to combine the strengths of multiple models.
✨ Key features
- Direct KV-Cache communication between LLMs
- 8.5–10.5% higher accuracy than individual models
- 3.0–5.0% better performance than text-based communication
- 2.0× speedup in latency
- Supports arbitrary model pairs with different architectures
- Pre-trained fusers available for popular Qwen and Llama models
🎯 Use cases
- Combine multiple LLMs to answer complex questions more accurately
- Enable real-time collaborative reasoning between models
- Transfer knowledge from a larger teacher model to a smaller receiver
- Build multi-model chat systems with shared semantic understanding
📦 Installation
🧰 Requirements: Python 3.10, PyTorch, and Hugging Face Transformers. Requires GPU for inference and training. No API keys needed.
conda create -n rosetta python=3.10
conda activate rosetta
pip install -e .
For training and evaluation, install additional dependencies:
pip install -e ".[training,evaluation]"
🚀 Usage
import torch
from huggingface_hub import snapshot_download
from script.playground.inference_example import load_rosetta_model, run_inference_example
checkpoint_dir = snapshot_download(
repo_id="nics-efc/C2C_Fuser",
allow_patterns=["qwen3_0.6b+qwen2.5_0.5b_Fuser/*"],
)
model_config = {
"rosetta_config": {
"base_model": "Qwen/Qwen3-0.6B",
"teacher_model": "Qwen/Qwen2.5-0.5B-Instruct",
"checkpoints_dir": f"{checkpoint_dir}/qwen3_0.6b+qwen2.5_0.5b_Fuser/final",
}
}
rosetta_model, tokenizer = load_rosetta_model(model_config, eval_config={}, device=torch.device("cuda"))
device = rosetta_model.device
prompt = [{"role": "user", "content": "Say hello in one short sentence."}]
input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tokenizer(input_text, return_tensors="pt").to(device)
instruction_index = torch.tensor([1, 0], dtype=torch.long).repeat(inputs['input_ids'].shape[1] - 1, 1).unsqueeze(0).to(device)
label_index = torch.tensor([-1, 0], dtype=torch.long).repeat(1, 1).unsqueeze(0).to(device)
kv_cache_index = [instruction_index, label_index]
with torch.no_grad(
⚠️ Good to know
Multi-sharer support is in preliminary stages and still being actively worked on.
❓ FAQ
What is Cache-to-Cache (C2C)?
C2C is a framework that lets LLMs communicate directly through their KV-Caches, bypassing text generation, by projecting and fusing KV-Caches between models.
How do I use a pre-trained C2C fuser?
Download the fuser checkpoint from Hugging Face and load it with the provided inference script, as shown in the usage example.
Can I apply C2C to my own LLMs?
Yes, you can wrap your models with the RosettaModel and add C2CProjector layers, as shown in the 'Apply C2C to any LLMs' section.
What are the requirements for training a C2C fuser?
You need to prepare a training configuration file and run the training script. Only the projector parameters are updated; the source and target models remain frozen.
📊 Repository
🤖 Overview, features, install steps and FAQ were generated from the project's README on Sep 4, 2026. Always check the original source before running commands.