Enroll Once, Identify Everywhere: Few-Shot Cross-Scenario Visual Speaker Recognition via Event-driven SlowFast Networks

Summary

This paper recognizes who is speaking from lip motion recorded by an event camera, using only three enrollment samples per new user. An event camera reports each pixel's brightness change as it happens instead of capturing full frames, so it keeps fast lip movements sharp and works in dim light. The model is trained once on known speakers and then reused unchanged for new ones. On the DVSpeaker dataset it reaches 99.18% Rank-1 accuracy (the nearest enrolled sample belongs to the right person) in matched frontal, bright conditions, and 87.25% when the probe is recorded in low light.

Key numbers

• 99.18%Rank-1 accuracy with frontal, bright enrollment and probes, on 20 speakers unseen in training and three enrollment samples each
• 87.25%Rank-1 accuracy when the probe is recorded in low light and the enrollment was bright
• +28.75 pointsLead over the strongest of four event-based baselines when the probe is rotated 45° from the enrollment view (74.95% vs 46.20%)

Method

Visual speaker recognition (VSR) identifies a person from the motion of the lips and the area around them. It is silent, works without contact, and does not depend on ambient sound. Frame-based cameras blur fast lip movements and lose detail in low light. An event camera ^[1] instead reports per-pixel brightness changes asynchronously, each as an event with a position, a timestamp, and a polarity (brighter or darker), at microsecond latency and over a wide dynamic range.

The paper fixes a deployment-oriented few-shot enrollment protocol. Training and test speakers are disjoint. Each unseen test speaker contributes K = 3 enrollment samples to a gallery, and the model is never fine-tuned on them. The enrollment and probe recordings may differ in viewpoint or illumination, which defines the cross-scenario setting. Prior closed-set protocols such as NeuroLip ^[2] keep the same identities in training and test and vary only the scenario.

Events are accumulated into a count volume, a four-dimensional array of event counts per polarity over 32 time bins, with a logarithm to stabilize brightness. A SlowFast network ^[3] runs two 3D convolutional pathways over this volume. The Slow pathway sees a temporally subsampled volume with wider channels, and the Fast pathway sees the full rate with narrower channels. Lateral connections add Fast features into the Slow stream at four points. The network is trained with cross-entropy on the training speakers, and its 384-dimensional embedding is then used as a frozen feature extractor. A probe is matched to the nearest gallery embedding by Euclidean distance.

Evaluation

The evaluation uses DVSpeaker ^[2], which holds 20,000 recordings of 50 subjects saying digits 0 to 9 in four scenarios: three viewpoints (0°, 45°, 90°) in bright light and a frontal view in dim light. The default split trains on 30 speakers and tests on 20. With matched enrollment and probes, Rank-1 accuracy is 99.18% (frontal, bright), 97.32% (45°), 98.04% (90°), and 96.34% (low light). With only 10 training speakers, matched accuracy is already above 74% in every scenario.

Accuracy falls when the scenario shifts. With frontal, bright enrollment, the probe reaches 87.25% under low light, 74.95% at 45°, and 50.20% at 90°. The strongest of four reproduced event-based baselines (EventFace, EV-Gait-IMG, a frame-rate convolutional network, and SpikGRU++) trails the SlowFast model by 28.75 points at 45°, 21.15 points at 90°, and 8.55 points under low light.

Ablations show that neither pathway is enough alone. The Slow-only and Fast-only variants match the full model on matched probes (about 98%) and lag it by 15 to 17 points at 45° and 19 to 22 points at 90°. Lateral fusion adds 8 to 10 points on the viewpoint shifts and does not help under low light, where the model without fusion scores 89.85% against 87.25%.

Limitations

DVSpeaker is, to the authors' knowledge, the only event-camera lip-motion corpus with controlled multi-scenario coverage, so cross-dataset generalization is untested. It contains 50 healthy adult speakers saying digits, and a deployment-grade biometric study would need a larger and more diverse pool and non-digit speech.

The 90° viewpoint stays hard: matched accuracy is 98.04%, but frontal-bright enrollment gives 50.20% at 90°. The authors attribute this to the side profile occluding the inner mouth, which a different backbone design would have to address. Because the protocol trains on all scenarios, it assumes that scenario-labeled data can be collected before deployment.

Citation

@inproceedings{yao2026enroll,
  title={Enroll Once, Identify Everywhere: Few-Shot Cross-Scenario Visual Speaker Recognition via Event-driven SlowFast Networks},
  author={Junguang Yao and Peichun Hua and Yue Zheng},
  booktitle={The 23rd Pacific Rim International Conference on Artificial Intelligence (PRICAI)},
  year={2026},
  organization={Springer}
}

References

1. Guillermo Gallego et al. Event-based Vision: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. arxiv.org/abs/1904.08405
2. Junguang Yao, Wenye Liu, Stjepan Picek, and Yue Zheng. NeuroLip: An event-driven spatiotemporal learning framework for cross-scene lip-motion-based visual speaker recognition. arXiv, 2026. arxiv.org/abs/2604.15718
3. Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast Networks for Video Recognition. ICCV, 2019. arxiv.org/abs/1812.03982