Software Is Designed to Protect Privacy in Surveillance Videos
#NoSeeMe
Software Is Designed to Protect Privacy in Surveillance Videos
#NoSeeMe
Research Terms
The University of Central Florida invention is a real-time online system and method that can detect multiple activities occurring in long, untrimmed security videos. The invention uses a deep learning approach to process videos in an online fashion at a clip level—drastically reducing the computation time in detecting activities. The ability of the method to process one video clip at a time in an online fashion makes it robust against varying length activities. The methodology was tested on the VIRAT and MEVA (Multiview Extended Video with Activities) datasets with more than 250 hours of videos and demonstrated effective performance in terms of processing speed as well as activity detection. The invention can process high-resolution security videos at 100 frames per second.
The University of Central Florida invention provides a low-cost system that can significantly reduce the annotation cost for videos. Video activity detection requires annotations at every frame, which drastically increases the labeling cost. As a solution, the UCF invention offers a way to greatly reduce annotation costs while using only a few annotations. An example application is preparing large-scale video datasets for various video analysis tasks such as video tracking, detection and segmentation.
Technical Details: The invention’s Active Sparse Labeling (ASL) algorithm identifies the usefulness of each frame of a video. ASL then suggests the frames and videos that can improve dense video understanding tasks such as activity detection in videos. Along with the algorithm, the invention uses a unique method called Spatio-Temporal Weighted loss (STeW loss) to train video models using datasets with sparsely annotated frames. The invention works in two stages, first, it trains a deep-learning video model using very sparse frames and then uses the trained deep-learning model to select more frames for annotation based on their utility value. When tested on public benchmark datasets, UCF-101 and J-HMDB, with more than 400,000 frames, the invention effectively reduced annotation cost by 90 percent and learned action detection in videos using only 10 percent of annotated video frames.
Partnering Opportunity: The research team is seeking partners for licensing, research collaboration, or both.
Stage of Development: Prototype available.
Are All Frames Equal? Active Sparse Labeling for Video Action Detection, 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
This UCF invention is SiA, a simple and scalable architecture for open vocabulary action detection in videos. Unlike conventional closed set action detectors that are limited to a predefined list of trained actions, SiA enables detection of previously unseen human actions through textual descriptions provided at inference time. The model achieves strong generalization by combining video text contrastive pretraining with a weakly supervised training strategy that leverages large scale action datasets. SiA is designed to balance flexibility, performance, and computational efficiency, making it suitable for real world applications where action sets evolve dynamically and retraining costs must be minimized.
Technical Details: SiA employs a transformer based dual encoder architecture, consisting of a video transformer encoder and a text transformer encoder. Following large scale video text contrastive pretraining, the standard classification token is replaced with multiple trainable [DET] tokens, enabling DETR style spatio-temporal action localization. These [DET] tokens are processed through dedicated MLP heads for human classification, bounding box regression, and action embedding projection.
Action recognition is performed by computing cosine similarity between normalized video embeddings and text embeddings corresponding to candidate action prompts. A predefined similarity threshold determines action occurrence. To preserve language semantics while adapting the model for detection, the text encoder incorporates Low Rank Adaptation (LoRA) modules.
A key feature is the Assignment based Weak Supervision (AWS) scheme, which enables training on the AVA Kinetics dataset using more than 700 action labels rather than the standard 80. AWS operates in two stages: an initial naive allocation of video level action labels, followed by a refinement stage where the model selectively assigns actions only to detected humans actually performing them. This substantially improves label quality, scalability, and generalization without requiring costly manual annotation.
This UCF invention is a novel deep learning architecture that enables compact neural networks to perform high-level visual understanding with significantly reduced computational requirements. At a high level, this technology makes it possible to deliver advanced AI perception without the heavy compute typically required, opening the door to real-time deployment on resource-constrained devices. The system introduces an Asynchronous Perception Machine that processes visual information incrementally and adapts to new inputs on the fly, achieving competitive performance with far fewer resources than traditional CNN or transformer-based models. It is particularly suited for real-time, resource-constrained environments where speed, efficiency, and adaptability are critical.
Technical Details: The technology combines a simple convolutional front-end with a shared multi-layer perceptron (MLP). Instead of processing entire images at once, it evaluates smaller regions and builds a full understanding by aggregating these responses. A key feature is its ability to learn from a single input at inference time by refining a compact representation, eliminating the need for repeated full-model computation. This approach significantly lowers compute requirements while maintaining strong performance on challenging datasets.
This UCF invention is a deep learning framework that enables reliable identification of individuals across significant appearance changes, such as different outfits or environmental conditions. Simply put, the system learns to recognize the person themselves rather than what they are wearing or how they look at a given moment. The system isolates consistent identity characteristics while minimizing the influence of transient visual factors that typically degrade performance in conventional methods. By integrating AI-generated textual descriptions with visual data during training, the model learns to emphasize biometric traits and ignore non-essential details. This results in stronger and more consistent identification performance in real-world scenarios where appearance variability is common, particularly in surveillance and long-term tracking applications.
Technical Details: The framework incorporates a vision-language model to generate descriptive text capturing both biometric attributes (e.g., body structure, age, gender) and non-biometric attributes (e.g., clothing, hair, pose). These descriptions are encoded and used to guide the decomposition of image features into separate subspaces through contrastive learning. A key component, the NBDetach module, introduces gradient reversal to actively remove non-biometric information from the learned representation while reinforcing identity-relevant features. The training process combines image–text alignment losses with identity classification objectives. During deployment, the additional text-based components are not required, allowing the system to operate efficiently using only a visual encoder on standard RGB inputs.
The University of Central Florida invention introduces a novel approach to person identification based on daily activities, addressing the limitations of traditional methods such as face recognition and gait analysis. Face recognition techniques, while advanced, often fail in real-world scenarios where facial features are not visible due to factors like long distances, environmental disturbances, occlusions (e.g., mask-wearing), and uncooperative subjects. Gait recognition, which analyzes walking patterns, also has limitations as individuals are not always walking in real-world situations.
This invention focuses on identifying individuals based on a wide range of daily activities, such as sitting, taking off a jacket, drinking water, and more. These activities provide unique behavioral cues that can be instrumental in identifying individuals even when facial information is unavailable. The approach leverages video analysis to process both biometric features (e.g., body shape, gait) and non-biometric features (e.g., clothing, background) from video data.
Technical Details: The UCF invention utilizes advanced video analysis techniques to process both biometric and non-biometric features from video data. Initially, an RGB video input containing a mix of biometric features (such as body shape and gait) and non-biometric features (such as clothing and background) is received. This video is passed through a sophisticated video encoder that extracts spatial-temporal features, which are divided into two streams. The "activity head" analyzes the specific activities depicted in the video, while the "actor head" distinguishes between the biometric and non-biometric features of the individual.
To enhance identification accuracy, the invention employs a bias-less distillation process, where a silhouette version of the video is processed through a teacher network to distill unbiased biometric features back into the main model, filtering out appearance-related biases. Additionally, a bias-learning technique distorts the original video to obscure biometric features while keeping non-biometric features intact, allowing the system to learn and compensate for appearance biases during training. The method concludes with a joint training process that refines both activity and actor features, enabling the system to use these enhanced features for accurate identification by comparing them against a pre-existing gallery of identities.
