webAI-Official

/

webAI-ColVec1.1-8b

like 5

Follow

webAI 110

Hello!

This mirrors the integration proposed for the 4B sibling (PR here), so the two checkpoints stay usable through exactly the same code. The MultiVectorEncoder class ships in the next Sentence Transformers release, planned for around the 18th, so for now the install below pulls from source. I would love to feature this model in that release's blog post and documentation, especially once it loads without the revision pin (that is, once this PR is merged).

Heads up, this PR was AI-generated and human-reviewed. Here's a summary of the changes as reported by my agent:

Pull Request overview

• Integrate webAI-Official/webAI-ColVec1.1-8b with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via MultiVectorEncoder.
• Implement ColQwen35BidirectionProcessor.replace_image_token, which ProcessorMixin requires and which currently raises NotImplementedError.

Details

The integration is config-only on the model side: sentence_bert_config.json sets transformer_task="retrieval", so Sentence Transformers resolves the model through your auto_map and loads ColQwen35Bidirection itself, keeping the bidirectional attention patching, the projection, the L2 normalization and the masking in your code. The pipeline is therefore just Transformer(retrieval) -> MultiVectorMask. The prompt formats are reproduced in a named chat template selected through processing_kwargs, so encode_query and encode_document produce byte-identical token ids to your process_queries / process_images helpers, and the existing AutoProcessor / AutoModel / score_retrieval path is unchanged.

One code change: ColQwen35BidirectionProcessor.replace_image_token. ProcessorMixin.__call__ (and apply_chat_template through it) delegates image-placeholder expansion to that method, which currently raises NotImplementedError, so any image input fails while text-only input works. The fix implements it with the same grid arithmetic as _process_single_image, so it is a strict addition: process_images, process_queries and score_retrieval are unchanged, and processor(text=..., images=...) now works for everyone, not only through Sentence Transformers.

Verified against your AutoProcessor + AutoModel + score_retrieval path, bit-exact throughout. The bf16 checkpoint fits a 24 GB card but float32 does not, so parity was established in bf16 and extended down to the preprocessed tensors (identical input_ids, attention_mask, image_grid_thw, mm_token_type_ids, and pixel_values): embeddings and MaxSim scores match with max absolute difference exactly 0.0, per item and batched, including a page that saturates the 1792-token budget, a grayscale page, a file-path input, and queries from 18 to 138 tokens. Loading from the config files reports all 8 full-attention layers with is_causal=False, confirming the bidirectional patching survives the Sentence Transformers load path.

pip install "sentence-transformers[image] @ git+https://github.com/huggingface/sentence-transformers.git"

from io import BytesIO

import requests
from PIL import Image
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("webAI-Official/webAI-ColVec1.1-8b", revision="refs/pr/1", trust_remote_code=True)

queries = [
    "When was the United States Declaration of Independence proclaimed?",
    "Who printed the edition of Romeo and Juliet?",
]
document_urls = [
    "https://upload.wikimedia.org/wikipedia/commons/8/89/US-original-Declaration-1776.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Romeoandjuliet1597.jpg/500px-Romeoandjuliet1597.jpg",
]
documents = [
    Image.open(BytesIO(requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=30).content))
    for url in document_urls
]

query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (27, 640) (523, 640)

scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[22.1212,  4.7561],
#         [ 6.7469, 22.3831]])

• Tom Aarsen