Rozprawy doktorskie na temat „VIDEO ANOMALY DETECTION”

Anomaly detection is an area of video analysis that has great importance in automated surveillance. Although it has been extensively studied, there has been little work on using deep convolutional neural networks to learn spatio-temporal feature representations. In this thesis we present novel approaches for learning motion features and modelling normal spatio-temporal dynamics for anomaly detection. The contributions are divided into two main chapters. The first introduces a method that uses a convolutional autoencoder to learn motion features from foreground optical flow patches. The autoencoder is coupled with a spatial sparsity constraint, known as Winner-Take-All, to learn shift-invariant and generic flow-features. This method solves the problem of using hand-crafted feature representations in state of the art methods. Moreover, to capture variations in scale of the patterns of motion as an object moves in depth through the scene,we also divide the image plane into regions and learn a separate normality model in each region. We compare the methods with state of the art approaches on two datasets and demonstrate improved performance. The second main chapter presents a end-to-end method that learns normal spatio-temporal dynamics from video volumes using a sequence-to-sequence encoder-decoder for prediction and reconstruction. This work is based on the intuition that the encoder-decoder learns to estimate normal sequences in a training set with low error, thus it estimates an abnormal sequence with high error. Error between the network's output and the target is used to classify a video volume as normal or abnormal. In addition to the use of reconstruction error, we also use prediction error for anomaly detection. We evaluate the second method on three datasets. The prediction models show comparable performance with state of the art methods. In comparison with the first proposed method, performance is improved in one dataset. Moreover, running time is significantly faster.

In automated video surveillance anomaly detection is a challenging task. We address this task as a novelty detection problem where pattern description is limited and labelling information is available only for a small sample of normal instances. Classification under these conditions is prone to over-fitting. The contribution of this work is to propose a novel video abnormality detection method that does not need object detection and tracking. The method is based on subspace learning to discover a subspace where abnormality detection is easier to perform, without the need of detailed annotation and description of these patterns. The problem is formulated as one-class classification utilising a low dimensional subspace, where a novelty classifier is used to learn normal actions automatically and then to detect abnormal actions from low-level features extracted from a region of interest. The subspace is discovered (using both labelled and unlabelled data) by a locality preserving graph-based algorithm that utilises the Graph Laplacian of a specially designed parameter-less nearest neighbour graph. The methodology compares favourably with alternative subspace learning algorithms (both linear and non-linear) and direct one-class classification schemes commonly used for off-line abnormality detection in synthetic and real data. Based on these findings, the framework is extended to on-line abnormality detection in video sequences, utilising multiple independent detectors deployed over the image frame to learn the local normal patterns and infer abnormality for the complete scene. The method is compared with an alternative linear method to establish advantages and limitations in on-line abnormality detection scenarios. Analysis shows that the alternative approach is better suited for cases where the subspace learning is restricted on the labelled samples, while in the presence of additional unlabelled data the proposed approach using graph-based subspace learning is more appropriate.

This thesis work focusses upon developing the capability to automatically evaluate and detect anomalies in human behaviour from surveillance video. We work with static monocular cameras in crowded urban surveillance scenarios, particularly air- ports and commercial shopping areas. Typically a person is 100 to 200 pixels high in a scene ranging from 10 - 20 meters width and depth, populated by 5 to 40 peo- ple at any given time. Our procedure evaluates human behaviour unobtrusively to determine outlying behavioural events, agging abnormal events to the operator. In order to achieve automatic human behaviour anomaly detection we address the challenge of interpreting behaviour within the context of the social and physical environment. We develop and evaluate a process for measuring social connectivity between individuals in a scene using motion and visual attention features. To do this we use mutual information and Euclidean distance to build a social similarity matrix which encodes the social connection strength between any two individuals. We de- velop a second contextual basis which acts by segmenting a surveillance environment into behaviourally homogeneous subregions which represent high tra c slow regions and queuing areas. We model the heterogeneous scene in homogeneous subgroups using both contextual elements. We bring the social contextual information, the scene context, the motion, and visual attention features together to demonstrate a novel human behaviour anomaly detection process which nds outlier behaviour from a short sequence of video. The method, Nearest Neighbour Ranked Outlier Clusters (NN-RCO), is based upon modelling behaviour as a time independent se- quence of behaviour events, can be trained in advance or set upon a single sequence. We nd that in a crowded scene the application of Mutual Information-based social context permits the ability to prevent self-justifying groups and propagate anomalies in a social network, granting a greater anomaly detection capability. Scene context uniformly improves the detection of anomalies in all the datasets we test upon. We additionally demonstrate that our work is applicable to other data domains. We demonstrate upon the Automatic Identi cation Signal data in the maritime domain. Our work is capable of identifying abnormal shipping behaviour using joint motion dependency as analogous for social connectivity, and similarly segmenting the shipping environment into homogeneous regions.

In light of increased demands for security, we propose a unique approach to automated video surveillance using anomaly detection. The success of this approach is dependent on the ability of the system to ascertain the novelty of a given image acquired by a video camera. We adopt a compression-based similarity measure to determine similarity between images in a video sequence. Images that are sufficiently similar to the previously-seen images are discarded; conversely, images that are sufficiently dissimilar are stored for comparison with future incoming images.
The use of a compression-based technique inherently reduces the heavy computational and storage demands that other video surveillance applications typically have placed on the system. In order to further reduce the computational and storage load, the anomaly detection algorithm is applied to edges and people, which are image features that have been extracted from the images acquired by the camera.

Human operators of modern surveillance systems are confronted with an increasing amount of trajectory data from moving objects, such as people, vehicles, vessels, and aircraft. A large majority of these trajectories reflect routine traffic and are uninteresting. Nevertheless, some objects are engaged in dangerous, illegal or otherwise interesting activities, which may manifest themselves as unusual and abnormal trajectories. These anomalous trajectories can be difficult to detect by human operators due to cognitive limitations. In this thesis, we study algorithms for the automated detection of anomalous trajectories in surveillance applications. The main results and contributions of the thesis are two-fold. Firstly, we propose and discuss a novel approach for anomaly detection, called conformal anomaly detection, which is based on conformal prediction (Vovk et al.). In particular, we propose two general algorithms for anomaly detection: the conformal anomaly detector (CAD) and the computationally more efficient inductive conformal anomaly detector (ICAD). A key property of conformal anomaly detection, in contrast to previous methods, is that it provides a well-founded approach for the tuning of the anomaly threshold that can be directly related to the expected or desired alarm rate. Secondly, we propose and analyse two parameter-light algorithms for unsupervised online learning and sequential detection of anomalous trajectories based on CAD and ICAD: the sequential Hausdorff nearest neighbours conformal anomaly detector (SHNN-CAD) and the sequential sub-trajectory local outlier inductive conformal anomaly detector (SSTLO-ICAD), which is more sensitive to local anomalous sub-trajectories. We implement the proposed algorithms and investigate their classification performance on a number of real and synthetic datasets from the video and maritime surveillance domains. The results show that SHNN-CAD achieves competitive classification performance with minimum parameter tuning on video trajectories. Moreover, we demonstrate that SSTLO-ICAD is able to accurately discriminate realistic anomalous vessel trajectories from normal background traffic.

Behaviour analysis and anomaly detection are key components of intelligent vision systems. Anomaly detection can be considered from two perspectives: abnormal events can be defined as those that violate typical activities or as a sudden change in behaviour. Topic modeling and change point detection methodologies, respectively, are employed to achieve these objectives. The thesis starts with development of novel learning algorithms for a dynamic topic model. Topics extracted by the learning algorithms represent typical activities happening within an observed scene. These typical activities are used for likelihood computation. The likelihood serves as a normality measure in anomaly detection decision making. A novel anomaly localisation procedure is proposed. In the considered dynamic topic model a number of topics, i.e., typical activities, should be specified in advance. A novel dynamic nonparametric hierarchical Dirichlet process topic model is then developed where the number of topics is determined from data. Conventional posterior inference algorithms require processing of the whole data through several passes. It is computationally intractable for massive or sequential data. Therefore, batch and online inference algorithms for the proposed model are developed. A novel normality measure is derived for decision making in anomaly detection. The latter part of the thesis considers behaviour analysis and anomaly detection within the change point detection methodology. A novel general framework for change point detection is introduced. Gaussian process time series data is considered and a change is defined as an alteration in hyperparameters of the Gaussian process prior. The problem is formulated in the context of statistical hypothesis testing and several tests suitable both for offline and online data processing and multiple change point detection are proposed. Theoretical properties of the proposed tests are derived based on the distribution of the test statistics.

World wide focus has over the years been shifting towards security issues, not in least due to recent world wide terrorist activities. Several researchers have proposed state of the art surveillance systems to help with some of the security issues with varying success. Recent studies have suggested that the ability of these surveillance systems to learn common environmental behaviour patterns as wells as to detect and predict unusual, or anomalous, activities based on those learnt patterns are possible improvements to those systems. In addition, some of these surveillance systems are still run by human operators, who are prone to mistakes and may need some help from the surveillance systems themselves in detection of anomalous activities. This dissertation attempts to address these suggestions by combining the fields of Image Understanding and Artificial Intelligence, specifically Bayesian Networks, to develop a prototype video surveillance system that can learn common environmental behaviour patterns, thus being able to detect and predict anomalous activity in the environment based on those learnt patterns. In addition, this dissertation aims to show how the prototype system can adapt to these anomalous behaviours and integrate them into its common patterns over a prolonged occurrence period. The prototype video surveillance system showed good performance and ability to detect, predict and integrate anomalous activity in the evaluation tests that were performed using a volunteer in an experimental indoor environment. In addition, the prototype system performed quite well on the PETS 2002 dataset 1, which it was not designed for. The evaluation procedure used some of the evaluation metrics commonly used on the PETS datasets. Hence, the prototype system provides a good approach to anomaly detection and prediction using Bayesian Networks trained on common environmental activities.

This work addresses the following research question: Can we detect videogame glitches using Convolutional Neural Networks? Focusing on the most common types of glitches, texture glitches (Stretched, Lower Resolution, Missing, and Placeholder). We first systematically generate a dataset with both images with texture glitches and normal samples.  To detect the faulty images we try both Classification and Semantic Segmentation approaches, with a clear focus on the former. The best setting in classification uses a ShuffleNetV2 architecture and obtains precisions of 80.0%, 64.3%, 99.2%, and 97.0% in the respective glitch classes Stretched, Lower Resolution, Missing, and Placeholder. All of this with a low false positive rate of 6.7%. To complement this study, we also discuss how the models extrapolate to different graphical environments, which are the main sources of confusion for the model, how to estimate the confidence of the network, and ways to interpret the internal behavior of the models.
Detta projekt svarar på följande forskningsfråga: Kan man använda Convolutional Neural Networks för att upptäcka felaktiga bilder i videospel? Vi fokuserar på de vanligast förekommande grafiska defekter i videospel, felaktiga textures (sträckt, lågupplöst, saknas och platshållare). Med hjälp av en systematisk process genererar vi data med både normala och felaktiga bilder. För att hitta defekter använder vi CNN via både Classification och Semantic Segmentation, med fokus på den första metoden. Den bäst presterande Classification-modellen baseras på ShuffleNetV2 och når 80.0%, 64.3%, 99.2% och 97.0% precision på respektive sträckt-, lågupplöst-, saknas- och platshållare-buggar. Detta medan endast 6.7% av negativa datapunkter felaktigt klassifieras som positiva. Denna undersökning ser även till hur modellen generaliserar till olika grafiska miljöer, vilka de primära orsakerna till förvirring hos modellen är, hur man kan bedöma säkerheten i nätverkets prediktion och hur man bättre kan förstå modellens interna struktur.

Aircraft-mounted cameras have potential to greatly increase the effectiveness of wilderness search and rescue efforts by collecting photographs or video of the search area. The more data that is collected, the more difficult it becomes to process it by visual inspection alone. This work presents a method for automatically detecting unusual objects in aerial video to assist people in locating signs of missing persons in wilderness areas. The detector presented here makes use of anomaly detection methods originally designed for hyperspectral imagery. Multiple anomaly detection methods are considered, implemented, and evaluated. These anomalies are then aggregated into spatiotemporal objects by using the video's inherent spatial and temporal redundancy. The results are therefore summarized into a list of unusual objects to enhance the search technician's video review interface. In the user study reported here, unusual objects found by the detector were overlaid on the video during review. This increased participants' ability to find relevant objects in a simulated search without significantly affecting the rate of false detection. Other effects and possible ways to improve the user interface are also discussed.

As video capturing devices become more ubiquitous from surveillance cameras to smart phones, the demand of automated video analysis is increasing as never before. One obstacle in this process is to efficiently locate where a human operator’s attention should be, and another is to determine the specific types of activities or actions without ambiguity. It is the special interest of this dissertation to locate spatial and temporal regions of interest in videos and to develop a better action representation for video-based activity analysis. This dissertation follows the scheme of “locating then recognizing” activities of interest in videos, i.e., locations of potentially interesting activities are estimated before performing in-depth analysis. Theoretical properties of regions of interest in videos are first exploited, based on which a unifying framework is proposed to locate both spatial and temporal regions of interest with the same settings of parameters. The approach estimates the distribution of motion based on 3D structure tensors, and locates regions of interest according to persistent occurrences of low probability. Two contributions are further made to better represent the actions. The first is to construct a unifying model of spatio-temporal relationships between reusable mid-level actions which bridge low-level pixels and high-level activities. Dense trajectories are clustered to construct mid-level actionlets, and the temporal relationships between actionlets are modeled as Action Graphs based on Allen interval predicates. The second is an effort for a novel and efficient representation of action graphs based on a sparse coding framework. Action graphs are first represented using Laplacian matrices and then decomposed as a linear combination of primitive dictionary items following sparse coding scheme. The optimization is eventually formulated and solved as a determinant maximization problem, and 1-nearest neighbor is used for action classification. The experiments have shown better results than existing approaches for regions-of-interest detection and action recognition.

There have been lots of new drone applications in recent years. Drones are also often used in the field of national security forces. The aim of this work is to design and implement a tool intended for crowd behavior analysis in drone videodata. This tool ensures identification of suspicious behavior of persons and facilitates its localization. The main benefits include the design of a suitable video stabilization algorithm to stabilize small jitters, as well as trace back of the lost scene. Furthermore, two anomaly detectors were proposed, differing in the method of feature vector extraction and background modeling. Compared to the state of the art approaches, they achieved comparable results, but at the same time they brought the possibility of online data processing.

Cette thèse est dédiée es débruitage vidéo. La première partie se concentre sur les méthodes de débruitage de vidéo à patches. Nous étudions en détail VBM3D, une méthode populaire de débruitage vidéo, pour comprendre les méchanismes qui ont fait son succès. Nous présentons aussi une implémentation temps-réel sur care graphique de cette méthode. Nous étudions ensuite l'impacte de la recherche de patches pour le débruitage vidéo et en particulier commen une recherche globale peut améliorer la qualité du débruitage. Enfin, nous proposons une nouvelle méthode causale et récursive appelée NL-Kalman qui produit ne rès bonne consistance temporelle.Dans la deuxième partie, nous étudions les méthodes d'apprentissage pour le débruitage. Nous présentons l'une des toutes premières architecture de réseau qui est compétitive avec l'état de l'art. Nous montrons aussi que les méthodes basées sur l'apprentissage profond offrent de nouvelles opportunités. En particulier, il devient possible de débruiter sans connaître le modèle du bruit. Grâce à la méthode proposée, même les vidéos traitées par une chaîne de traitement inconnue peuvent être débruitées. Nous étudions aussi le cas de données mosaïquées. En particulier, nous montrons que les réseaux de neurones sont largement supérieurs aux méthodes précédentes. Nous proposons aussi une nouvelle méthode d'apprentissage pour démosaïckage sans avoir besoin de vérité terrain.Dans une troisième partie nous présentons différentes application aux techniques utilisées pour le débruitage. Le premier problème étudié est la détection d'anomalie. Nous montrons que ce problème peut être ramené à détecter des anomalies dans du bruit. Nous regardons aussi la détection de falsification et en particulier la détection de copié-collé. Tout comme le débruitage à patches, ce problème peut être résolu à l'aide d'une recherche de patches similaires. Pour cela, nous étudions en détail PatchMatch et l'utilisons pour détecter des falsifications. Nous présentons aussi une méthode basée sur une association de patches parcimonieuse
This thesis studies the problem of video denoising. In the first part we focus on patch-based video denoising methods. We study in details VBM3D, a popular video denoising method, to understand the mechanisms that made its success. We also present a real-time implementation on GPU of this method. We then study the impact of patch search in video denoising and in particular how searching for similar patches in the entire video, a global patch search, improves the denoising quality. Finally, we propose a novel causal and recursive method called NL-Kalman that produces very good temporal consistency.In the second part, we look at the newer trend of deep learning for image and video denoising. We present one of the first neural network architecture, using temporal self-similarity, competitive with state-of-the-art patch-based video denoising methods. We also show that deep learning offers new opportunities. In particular, it allows for denoising without knowing the noise model. We propose a framework that allows denoising of videos that have been through an unknown processing pipeline. We then look at the case of mosaicked data. In particular, we show that deep learning is undeniably superior to previous approaches for demosaicking. We also propose a novel training process for demosaicking without ground-truth based on multiple raw acquisition. This allows training for real case applications. In the third part we present different applications taking advantage of mechanisms similar those studied for denoising. The first problem studied is anomaly detection. We show that this problem can be reduced to detecting anomalies in noise. We also look at forgery detection and in particular copy-paste forgeries. Just like for patch-based denoising, solving this problem requires searching for similar patches. For that, we do an in-depth study of PatchMatch and see how it can be used for detecting forgeries. We also present an efficient method based on sparse patch matching

Ce travail porte sur la surveillance des infrastructures critiques dans le cadre d'un projet nommé VIRTUALIS mené par Thales. La thèse se concentre sur la détection de comportements inhabituels dans un bâtiment. Deux approches ont été proposées à ce problème de surveillance en fonction de la qualité des données disponibles. La première approche vise à surveiller la région identifiée d'un bâtiment, généralement la zone d'importance particulière telle que salle informatique, salle prototype... Les capteurs recueillent des données qui permettent de construire un modèle statistique simple basé sur des mesures clés qui captent le comportement habituel des personnes dans un lieu. Sur la base de la distribution estimée de ces indicateurs clés, les objectifs sont de détecter des situations inhabituelles. Il a appliqué cette approche aux 2 cas d'utilisation et a montré que lorsque le but de la surveillance est clairement lié à la zone critique, la méthode fonctionne bien. La deuxième approche nécessite plus de données car elle est basée sur la trajectoire des personnes dans le bâtiment. L'idée principale est de détecter des trajectoires inhabituelles. La première étape consiste à transformer les données brutes provenant des capteurs dans un graphique d'activités plutôt simple, où les activités sont déduites des données recueillies. Ensuite, une SVM à une ou deux classes est formée en utilisant des noyaux ad hoc pour détecter une séquence inhabituelle d'activités. En mode détection, les trajectoires sont construites en temps réel sur une fenêtre temporelle donnée
This work is about critical infrastructure monitoring within a project named VIRTUALIS leaded by Thales. VIRTUALIS project is about convergence between cyber and physical security systems. The thesis concentrates on the detection of unusual behaviors in a building. Two approaches were proposed to this monitoring problem depending on the quality of the available data. The first approach intends to monitor identified region of a building, generally the area of particular importance such as computer room, prototype room… Sensors gather data that enable to build simple statistical model based on key metrics which capture the usual behavior of people in a given area. Based on the estimated distribution of these key metrics the aims is to detect unusual situations. He applied this approach on the 2 use cases and show that when the aim of monitoring is clearly related to critical area, the method performs well. The second approach necessitates more data since it is based on trajectory of people in the building. The leading idea is to detect unusual trajectories. The first step is to transform the raw data coming from sensors in a rather simple graph of activities, where activities are inferred from gathered data. Next a one or two class SVM is trained using ad-hoc kernels to detect unusual sequence of activities. In detection mode, trajectories are constructed in real time over a given time window

When photos and videos are increasingly used as evidence material, it is of importance to know if these materials can be used as evidence material or if the risk of them being forged is impending. This thesis investigates methods for detecting anomalous regions in images and videos using photo-response non-uniformity -- a fixed-pattern sensor noise that can be estimated from photos or videos. For photos, experiments were performed on a method that assumes other photos from the same camera are available. For videos, experiments were performed on a method further developed from the still image method, with other videos from the same camera being available. The last experiments were performed on videos when only the video that was about to be investigated was available. The experiments on the still image method were performed on images with three different kinds of forged regions: a forged region from somewhere else in the same photo, a forged region from a photo taken by another camera, and a forged region from the same sensor position in a photo taken by the same camera. The method should not be able to detect the third kind of forged region. Experiments performed on videos had a forged region in several adjacent frames in the video. The forged region was from another video, and it moved and changed shape between the frames. The methods mainly consist of a classification process and some post-processing. In the classification process, features were extracted from images/videos and used in a random forest classifier. The results are presented in precision, recall, F1 score and false positive rate. The quality of the still images was generally better than the videos, which also resulted in better results. For the cameras used in the experiments, it seemed easier to estimate a good PRNU pattern from photos and videos from older cameras. Probably due to sensor differences and extra processing in newer camera models. How the images and videos are compressed also affects the possibility to estimate a good PRNU pattern, because important information may then be lost.

L'utilisation des caméras de surveillance s'est considérablement accru ces dernières années. Cette prolifération pose un problème sociétal de premier ordre, celui de l’exploitation des flux générés. Actuellement, ces données sont en majorité analysées par des opérateurs humains. Cependant, de nombreuses études remettent en cause la pertinence de cette approche. Il est chronophage et laborieux pour un opérateur de visionner des vidéos de surveillance durant de longues périodes. Compte tenu des progrès réalisés récemment dans le domaine de la vision par ordinateur, notamment par l'intermédiaire de l'apprentissage profond, une solution à ce problème réside dans le développement de systèmes intelligents capables d'épauler l'opérateur humain dans l'exploitation de ces données. Ces systèmes intelligents auront pour objectifs de modéliser les comportements normaux d'une scène surveillée et de détecter tout événement déviant, pouvant conduire à une faille de sécurité. Dans le cadre de cette thèse intitulée « Apprentissage profond appliqué à la détection d'événements anormaux dans les flux vidéos », on se propose de développer des algorithmes se basant sur l’apprentissage profond pour la détection et la localisation des événements vidéo anormaux pouvant refléter des situations à risque. Il s’agit, en fait, d’extraire des descripteurs spatiotemporels robustes et de définir des algorithmes de classification adaptés pour détecter des comportements suspects avec le minimum possible de fausses alarmes, tout en assurant un taux élevé de détection
The use of surveillance cameras has increased considerably in recent years. This proliferation poses a major societal problem, which is the exploitation of the generated video streams. Currently, most of these data are being analyzed by human operators. However, several studies question the relevance of this approach. It is time-consuming and laborious for an operator to monitor surveillance videos for long time periods. Given recent advances in computer vision, particularly through deep learning, one solution to this problem consists in the development of intelligent systems that can support the human operator in the exploitation of this data. These intelligent systems will aim to model the normal behaviours of a monitored scene and detect any deviant event that could lead to a security breach. Within the context of this thesis entitled "Deep learning applied to the detection of abnormal events in video streams", we propose to develop algorithms based on deep learning for the detection and localization of abnormal video events that may reflect dangerous situations. The purpose is to extract robust spatial and temporal descriptors and define classification algorithms adapted to detect suspicious behaviour with the minimum possible number of false alarms, while ensuring a high detection rate

In this dissertation, we address the problem of understanding human activities in videos by developing a two-pronged approach: coarse level modeling of scene activities and fine level modeling of individual activities. At the coarse level, where the resolution of the video is low, we rely on person tracks. At the fine level, richer features are available to identify different parts of the human body, therefore we rely on the body joint tracks. There are three main goals of this dissertation: (1) identify unusual activities at the coarse level, (2) recognize different activities at the fine level, and (3) predict the behavior for synthesizing and tracking activities at the fine level. The first goal is addressed by modeling activities at the coarse level through two novel and complementing approaches. The first approach learns the behavior of individuals by capturing the patterns of motion and size of objects in a compact model. Probability density function (pdf) at each pixel is modeled as a multivariate Gaussian Mixture Model (GMM), which is learnt using unsupervised expectation maximization (EM). In contrast, the second approach learns the interaction of object pairs concurrently present in the scene. This can be useful in detecting more complex activities than those modeled by the first approach. We use a 14-dimensional Kernel Density Estimation (KDE) that captures motion and size of concurrently tracked objects. The proposed models have been successfully used to automatically detect activities like unusual person drop-off and pickup, jaywalking, etc. The second and third goals of modeling human activities at the fine level are addressed by employing concepts from theory of chaos and non-linear dynamical systems. We show that the proposed model is useful for recognition and prediction of the underlying dynamics of human activities. We treat the trajectories of human body joints as the observed time series generated from an underlying dynamical system. The observed data is used to reconstruct a phase (or state) space of appropriate dimension by employing the delay-embedding technique. This transformation is performed without assuming an exact model of the underlying dynamics and provides a characteristic representation that will prove to be vital for recognition and prediction tasks. For recognition, properties of phase space are captured in terms of dynamical and metric invariants, which include the Lyapunov exponent, correlation integral, and correlation dimension. A composite feature vector containing these invariants represents the action and will be used for classification. For prediction, kernel regression is used in the phase space to compute predictions with a specified initial condition. This approach has the advantage of modeling dynamics without making any assumptions about the exact form (polynomial, radial basis, etc.) of the mapping function. We demonstrate the utility of these predictions for human activity synthesis and tracking.
Ph.D.
School of Electrical Engineering and Computer Science
Engineering and Computer Science
Computer Science PhD

All cities are getting smart with the intervention of latest technologies, their infrastructure is getting upgraded with each day. Critical information is provided to us by these infrastructures. There is growing prevalence of AI in today’s world, with the help of which a real-time system can be developed that can assist in detecting crimes as they occur. The surveillance platform’s information may include both aberrant and conventional footage. We propose developing an aberrant event identification system based on weakly annotated training videos, and so when such behavior is discovered, suitable action may be taken. For extraction of features, we deployed I3D-Resnet-50, a deep residual model. The Kinetics video action dataset was used to train this network. There are 13 unique abnormalities in our dataset. Crime, Attack, Firing, Burglaries, Thieving, Prison, Fight, Thefts, Breaking and entering, Bomb, Criminal damage, Torture, and Traffic Accident are all unusual incidents. The proposed approach for visual anomaly detection achieves considerable improvements in terms of correctness and recall.

M.TECH
Public security has become a major issue in public places such as subway stations, banks, malls, airports, etc. Recently we have seen that terrorist activities are growing all over the world. To monitor these kinds of activities, there is an increasing demand of automatic video surveillance systems. In a surveillance system, we need to study the behaviour of the environment whether there is any abnormality in the video or not, in real time. Due to this for real time application in surveillance systems, video behaviour profiling has been a topic of great interest in real time. In this work we have implemented a method for detecting the abnormality in the video. We have tested this method for classroom video surveillance. In case of video profiling we tend to find the behaviour of the video. There may be three types of behaviour in a classroom- normal, empty, abnormal. Normal means class is going on smoothly, empty means there is no one in the class and abnormal means there is some abnormal activity in the classroom. We have found out the behaviour of the class by finding out energy of the video. Hidden markov model has been used as classifier. This method gives results in real time.

碩士
國立臺灣大學
電信工程學研究所
107
Video anomaly detection which intents to identify rarely-happened or unexpected events is a worthy and developmental problem in video understanding tasks. Most of the previous works deal with the problem in an unsupervised way by learning normal representations of training data and identified the outliers as anomalies. Common deep learning-based methods are reconstruction-based. They train an autoencoder by minimizing the reconstruction errors of regular videos. Nevertheless, abnormal events don''t always lead to larger reconstruction errors. To address this issue, We propose using multi-frame prediction framework to enlarge the unexpected change and overcome the generalization property which stems from the use of an autoencoder. We use ConvLSTM model as the multi-frame predictor and show the effectivenes of utilizing latter frames for computing the frame anomaly scores. Experimental results show that our model leads to better performance on motion and appearance deformation irregularities. In addition, we collect a new car crash dataset which contains various car accidents as abnormal events from YouTube for evaluation. Compared to existing anomaly detection datasets, it is a more challenging and practical dataset due to the diversity of events and its different environmental conditions. Our model achieves comparable results in popular existing anomaly detection datasets and outperforms the state-of-the-art on the new proposed dataset.

博士
國立臺灣科技大學
電子工程系
105
This dissertation presents a unified framework for video anomaly detection and localization via hierarchical feature representations, kernel-based statistical models, and tree-based search algorithms. While most research on this topic has focused more on detecting local anomalies, which refer to video events with unusual appearances or motions, we are more interested in global anomalies that involve multiple video events interacting in an unusual manner, even if any individual video event can be normal. To simultaneously detect local and global anomalies, we first introduce a hierarchical feature structure for video event representation. Then, a statistical model is built to understand the normal events in a training set which does not contain any anomalies, based on which a tree-based inference algorithm is developed to detect and locate abnormal events in unseen-before test videos. Along the same structure, we gradually enrich our feature structures, statistical models, and inference algorithms to increasingly improve our previous methods. In this dissertation, we investigate two different hierarchical feature representations: 1) the bag-of-words histogram (BOW) and 2) the {\it ensemble} of nearby spatio-temporal interest points (STIP); two different kernel-based statistical models: 1) one-class support vector machine (SVM) and 2) Gaussian process regression (GPR); and two different inference algorithms: 1) single-instance path search and 2) multiple-instance path search (MiPS). Simulations on five popular benchmarks show that the proposed methods significantly outperform the main state-of-the-art methods, yet with lower computation time. We also demonstrate that such a framework can be successfully applied to improve many convolution neural network (CNN) based object recognition methods. This is achieved by developing an iterative localization refinement (ILR) algorithm as a post-processing scheme to refine these object detection results in an iterative manner in order to match as much ground-truth as possible. Simulations show that the proposed method can improve the main state-of-the-art works on the large-scale PASCAL VOC 2007, 2012, and Youtube-Object datasets.

碩士
輔仁大學
電機工程學系
100
The number of cameras is greatly increased due to security, road monitoring, and home-care demanded. Images remained clear and correct field of view (FOV) are very important for video surveillance, and yet a large-scale system installed with a huge amount of cameras is hard to maintain. This paper presents a camera anomaly detection method based on holistic feature analysis over time in salient regions for automatically online determination. The salient regions are constructed from a Markov Random Field framework, which is modeled by pixel-based accumulated movement. There are a handful of holistic features extracted from salient regions, and an online Kalman filter is introduced for recursive smoothing uncertain features. A finite state machine, then, is further designed for real-time event detection. The proposed method yields a robust solution for reducing noise produced from real-world complexities. Experiments are conducted on a set of recorded videos simulating various challenging situations. The test results show that the camera anomaly detection method is superior to other methods in terms of precision rate, false alarm rate, and time complexity.

Motion is an important cue in videos that captures the dynamics of moving objects. It helps in effective analysis of various event related tasks such as human action recognition, anomaly detection, tracking, crowd behavior analysis, traffic monitoring, etc. Generally, accurate motion information is computed using various optical flow estimation techniques. On the other hand, coarse motion information is readily available in the form of motion vectors in compressed videos. Utilizing these encoded motion vectors reduces the computational burden involved in flow estimation and enables rapid analysis of video streams. In this work, the focus is on analyzing motion patterns, retrieved from either motion vectors or optical flow, in order to do various event analysis tasks such as video classification, anomaly detection and crowd flow segmentation. In the first section, we utilize the motion vectors from H.264 compressed videos, a compression standard widely used due to its high compression ratio, to address the following problems. i) Video classification: This work proposes an approach to classify videos based on human action by capturing spatio-temporal motion pattern of the actions using Histogram of Oriented Motion Vector (HOMV) ii) Crowd flow segmentation: In this work, we have addressed the problem of flow segmentation of the dominant motion patterns of the crowds. The proposed approach combines multi-scale super-pixel segmentation of the motion vectors to obtain the final flow segmentation. iii) Anomaly detection: This problem is addressed by local modeling of usual behavior by capturing features such as magnitude and orientation of each moving object. In all the above approaches, the focus was to reduce computations while retaining comparable accuracy to pixel domain processing. In second section, we propose two approaches for anomaly detection using optical flow. The first approach uses spatio-temporal low level motion features and detects anomalies based on the reconstruction error of the sparse representation of the candidate feature over a dictionary of usual behavior features. The main contribution is in enhancing each local dictionary by applying an appropriate transformation on dictionaries of the neighboring regions. The other algorithm aims to improve the accuracy of anomaly localization through short local trajectories of super pixels belonging to moving objects. These trajectories capture both spatial as well as temporal information effectively. In contrast to compressed domain analysis, these pixel level approaches focus on improving the accuracy of detection with reasonable detection speed.

Motion is an important cue in videos that captures the dynamics of moving objects. It helps in effective analysis of various event related tasks such as human action recognition, anomaly detection, tracking, crowd behavior analysis, traffic monitoring, etc. Generally, accurate motion information is computed using various optical flow estimation techniques. On the other hand, coarse motion information is readily available in the form of motion vectors in compressed videos. Utilizing these encoded motion vectors reduces the computational burden involved in flow estimation and enables rapid analysis of video streams. In this work, the focus is on analyzing motion patterns, retrieved from either motion vectors or optical flow, in order to do various event analysis tasks such as video classification, anomaly detection and crowd flow segmentation. In the first section, we utilize the motion vectors from H.264 compressed videos, a compression standard widely used due to its high compression ratio, to address the following problems. i) Video classification: This work proposes an approach to classify videos based on human action by capturing spatio-temporal motion pattern of the actions using Histogram of Oriented Motion Vector (HOMV) ii) Crowd flow segmentation: In this work, we have addressed the problem of flow segmentation of the dominant motion patterns of the crowds. The proposed approach combines multi-scale super-pixel segmentation of the motion vectors to obtain the final flow segmentation. iii) Anomaly detection: This problem is addressed by local modeling of usual behavior by capturing features such as magnitude and orientation of each moving object. In all the above approaches, the focus was to reduce computations while retaining comparable accuracy to pixel domain processing. In second section, we propose two approaches for anomaly detection using optical flow. The first approach uses spatio-temporal low level motion features and detects anomalies based on the reconstruction error of the sparse representation of the candidate feature over a dictionary of usual behavior features. The main contribution is in enhancing each local dictionary by applying an appropriate transformation on dictionaries of the neighboring regions. The other algorithm aims to improve the accuracy of anomaly localization through short local trajectories of super pixels belonging to moving objects. These trajectories capture both spatial as well as temporal information effectively. In contrast to compressed domain analysis, these pixel level approaches focus on improving the accuracy of detection with reasonable detection speed.

Anomaly detection and classification is a crucial task for ensuring public safety in real world surveillance videos. Due to the increasing number of crimes, act of terrorism, and accidents, timely detection and classification of such anomaly cases can minimize the loss. However, the task of anomaly detection and classification is challenging due to the unavailability of large annotated video data, untrimmed nature of the surveillance videos, and difficulties in obtaining discriminative feature representations for anomaly categories. This thesis aims at mitigating the mentioned challenges by adopting weakly supervised learning methods, efficient temporal dependency modeling schemes in untrimmed videos and extracting discriminative feature representations from space time convolutional neural network (3DCNN). For this, three contributions have been made in this thesis and each of the contribution mitigate one or more challenges. The first contribution focus on extracting discriminative spatiotemporal feature representation from a proposed multilevel 3DCNN for anomaly video sequences and subsequently utilizes long short term memory (LSTM) module for obtaining effective temporal dependency encoding in anomaly detection task. From result analysis, it is found that the feature representation obtained from the multilevel 3DCNN is not only superior over existing competent methods but also it is robust to challenges like illumination changes, partial occlusion. In the second contribution, the knowledge of image and video classification is used through inflated 3DCNN for obtaining enhanced feature representation. In addition, a temporal interintra clip pooling strategy is proposed for efficient temporal encoding by LSTM module in anomaly detection task. The effectiveness of this contribution has been demonstrated over existing methods and the previous contribution through exhaustive experimentation. Since the first two contributions only focus on anomaly detection task in untrimmed surveillance videos, the task of anomaly classification still remains challenging. Thus, the third contribution of this thesis proposes a framework that jointly handles the anomaly detection and classification task. In addition, the proposed framework is driven by twolevels of temporal attention mechanism which highlights the temporal saliency in the feature map as well as develops a dependency between the detection and classification task. From exhaustive experimental analysis and visualization, it is observed that the proposed method not only boosts the anomaly detection performance but also substantially improves the classification accuracy.