Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring

Peichun Hua; Hao Li; Shanghao Shi; Zhiyuan Yu; Ning Zhang

Published · ACL 2026 · Main conference, oral presentation

Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring

Peichun Hua, Hao Li, Shanghao Shi, Zhiyuan Yu, Ning Zhang

Read paper (PDF) Publication record Code Slides Video

Overview

Representational Contrastive Scoring (RCS) detects jailbreak attempts against large vision-language models using their internal representations. It addresses over-rejection caused by treating unfamiliar benign inputs as malicious anomalies.

Method

RCS identifies safety-relevant hidden layers, learns a lightweight projection, and scores inputs by their relative proximity to benign and malicious representations. The paper instantiates the approach as Mahalanobis Contrastive Detection (MCD) and K-nearest Contrastive Detection (KCD).

Representation-space diagrams contrasting Mahalanobis Contrastive Detection and K-nearest Contrastive Detection relative to benign and malicious reference groups. — Contrastive detection compares proximity to benign and malicious representations using cluster distances or nearest neighbors. Figure 3 in the paper

Evaluation

Evaluation tests generalization to unseen attack types across mixed sources and modalities. The study compares MCD and KCD with baseline defenses and examines how contrastive representation geometry separates malicious inputs from benign distribution shifts.

Detection cost relative to the host model forward pass
Operation	Time	Relative overhead
Host LVLM forward pass	0.6383 s	Reference
KCD: projection + nearest-neighbor search	0.0260 s	≈ 4.0%
MCD: projection + Mahalanobis distance	0.0355 s	≈ 5.5%

Total rows from Table 4 using LLaVA-v1.6-Vicuna-7B. Overhead is relative to a standard forward pass, excluding autoregressive generation; these are detector costs rather than full-answer latency. Table 4 in the paper

multimodal jailbreak detection
vision-language models
representation learning
Mahalanobis distance
model safety

Citation

@inproceedings{hua2026rethinking,
    title = "Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring",
    author = {Hua, Peichun and Li, Hao and Shi, Shanghao and Yu, Zhiyuan and Zhang, Ning},
    editor = {Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David},
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.992/",
    pages = "21748--21785",
    ISBN = "979-8-89176-390-6",
}