Abstract
This UCF invention is SiA, a simple and scalable architecture for open vocabulary action detection in videos. Unlike conventional closed set action detectors that are limited to a predefined list of trained actions, SiA enables detection of previously unseen human actions through textual descriptions provided at inference time. The model achieves strong generalization by combining video text contrastive pretraining with a weakly supervised training strategy that leverages large scale action datasets. SiA is designed to balance flexibility, performance, and computational efficiency, making it suitable for real world applications where action sets evolve dynamically and retraining costs must be minimized.
Technical Details: SiA employs a transformer based dual encoder architecture, consisting of a video transformer encoder and a text transformer encoder. Following large scale video text contrastive pretraining, the standard classification token is replaced with multiple trainable [DET] tokens, enabling DETR style spatio-temporal action localization. These [DET] tokens are processed through dedicated MLP heads for human classification, bounding box regression, and action embedding projection.
Action recognition is performed by computing cosine similarity between normalized video embeddings and text embeddings corresponding to candidate action prompts. A predefined similarity threshold determines action occurrence. To preserve language semantics while adapting the model for detection, the text encoder incorporates Low Rank Adaptation (LoRA) modules.
A key feature is the Assignment based Weak Supervision (AWS) scheme, which enables training on the AVA Kinetics dataset using more than 700 action labels rather than the standard 80. AWS operates in two stages: an initial naive allocation of video level action labels, followed by a refinement stage where the model selectively assigns actions only to detected humans actually performing them. This substantially improves label quality, scalability, and generalization without requiring costly manual annotation.
Benefit
Eliminates the need for task specific retraining when new actions must be detected.Scales to an order of magnitude more action categories than existing open vocabulary detectors.Computationally efficient and edge deployable, unlike large compute heavy vision language models.Enables fast adaptation to evolving operational requirements in safety critical and dynamic environments.Market Application
Security and surveillance systems, including suspicious activity and threat detection.Healthcare and assisted-living environments, such as fall detection and elderly monitoring.Smart home and automation platforms requiring flexible human activity understanding.Sports analytics and performance monitoring, supporting dynamic and custom action queries.
Brochure
