Rozprawy doktorskie na temat „Landmark Estimation”

This thesis describes a method for the recognition of objects in an unconstrained environment with a widely ranging illumination, imaged from unknown view points and complicated background. The general problem is simplified by placing specially designed patterns on the object that allows us to solve the pose determination problem easily. There are several key components involved in the proposed recognition approach. They include pattern detection, pose estimation, model acquisition and matching, searching and indexing the model database. Other crucial issues pertaining to the individual components of the recognition system such as the choice of the pattern, the reliability and accuracy of the pattern detector, pose estimator and matching and the speed of the overall system are addressed. After establishing the methodological framework, experiments are carried out on a wide range of both synthetic and real data to illustrate the validity and usefulness of the proposed methods. The principal contribution of this research is a methodology for Tagged Object Recognition (TOR) in unconstrained conditions. A robust pattern (calibration chart) detector is developed for off-the-shelf use. To empirically assess the effectiveness of the pattern detector and the pose estimator under various scenarios, simulated data generated using a graphics rendering process is used. This simulated data provides ground truth which is difficult to obtain in projected images. Using the ground truth, the detection error, which is usually ignored, can be analysed. For model matching, the Chamfer matching algorithm is modified to get a more reliable matching score. The technique facilitates reliable Tagged Object Recognition (TOR). Finally, the results of extensive quantitative and qualitative tests are presented that show the plausibility of practical use of Tagged Object Recognition (TOR). The features characterising the enabling technology developed are the ability to a) recognise an object which is tagged with the calibration chart, b) establish camera position with respect to a landmark and c) test any camera calibration and 3D pose estimation routines, thus facilitating future research and applications in mobile robots navigations, 3D reconstruction and stereo vision.

Driver's distraction is one of the main areas, which researchers are focusing on, in design of Advanced Drivers Assistance Systems (ADASs). Head pose and eye-gaze direction are two reliable indicators of a driver's gaze and the current focus of attention. Compared with other methods that make use of head pose only, methods that combine eye information can achieve higher accuracy. The naturalistic driving environment always presents unique challenges (e.g., unstable illumination, jolts, etc.) to video-based gaze estimation and tracking systems. Some methods can achieve relatively high proficiency in the stationary laboratory environment, but they may not be suitable for the unstable driving environment. In addition, performing in real time or near-real time is another consideration for gaze estimation in an ADAS. Therefore, these special challenges need to be overcome to design ADASs. In this thesis, we proposed a new driver's gaze zone estimation framework designed for the naturalistic driving environment. The framework combines head and eye information to estimate the gaze zone of the driver in both daytime and nighttime. The framework is composed of five main components: Facial Landmark Detection, Head Pose Estimation, Iris Center Detection, Upper Eyelid Information Extraction, and Gaze Zone Estimation. First, Constrained Local Neural Field (CLNF) is applied to obtain the facial landmarks in the image plane and the 3D model of the face in the object frame. In addition, extracting region of interest (ROI) is utilized as an optimization strategy for CLNF facial landmark detection. Second, head pose estimation can be regarded as a Perspective-n-Point (PnP) problem. Levenberg-Marquardt optimization method is used to solve the PnP problem based on the 2D landmark locations in the image plane and their corresponding 3D locations in the object frame. Third, a regression model-based method is employed to obtain the iris center from eye landmarks detected in the previous part. For upper eyelid information extraction, a quadratic function is utilized to model the upper eyelid, and the second-order coefficient is extracted. Finally, the head pose and the eye information are combined to form a feature vector, and Random Decision Forest classifier is utilized to estimate the current gaze zone of the driver from the feature vector extracted. The experiment is carried out in the realistic driving environment in both daytime and nighttime with three volunteers by Kinect sensor V2 for Windows that is put at the back of windshield. Weighted and unweighted accuracy are utilized as evaluation metrics in gaze zone estimation. Weighted accuracy evaluates gaze zones with different significance while unweighted accuracy treats each gaze zone equally. Experiment results show that the gaze zone estimation framework proposed in this work has better performance compared to the reference in the daytime. The weighted and unweighted accuracy of gaze zone estimation reach 96.6% and 95.0% for daytime, respectively. For nighttime, the weighted and unweighted accuracy can reach 96% and 91.4%.

Automatic face recognition has been an active research field during the last few decades. Existing face recognition systems have demonstrated acceptable recognition performance under controlled conditions. However, practical and robust face recognition which is tolerant to various interferential variations remains a difficult and unsolved problem in the research community. In the first part of this thesis, we propose to use the concept of sparse point representation to address four important challenges in face recognition: wider-range tolerance to pose variation, face misalignment, facial landmark localisation and head pose estimation. The sparse point representation can be classified into two different categories. In the first category, equal numbers of feature points are predefined on different individuals. Each feature point refers to a specific physical location on a face while all the feature points have explicit correspondence across different individuals. In the second category, a set of feature points are detected at different locations with discriminative information content on a face image. Both the number and the positions of the feature points are varied from person to person such that diverse facial characteristics of different individuals can be represented. Based on the first category of sparse point representation, we propose a new Constrained Profile Model (CPM) to form an efficient facial landmark localisation framework. We also propose a novel Elastic Energy Model (EEM) to automatically conduct head pose estimation. Based on the second category of sparse point representation, we propose a new Textural Hausdorff Distance (THD), which has demonstrated a considerably wider range of tolerance against both in-depth head rotation and face misalignment. In the second part of this thesis, we focus on recently proposed micropattern based approaches which have proven to outperform classical face recognition methods and provided a new way of investigation into face analysis. We first apply a new Multidirectional Binary Pattern (MBP) representation upon sparse points to establish point correspondences for face recognition. We further propose an enhanced Sobel-LBP operator for face representation, which has demonstrated better performance than the original Local Binary Pattern (LBP). We finally present a novel high-order Local Derivative Pattern (LDP) for face recognition, which can capture more detailed and discriminative information than the first-order local pattern used in LBP. It should be noted that the concept of LDP for face recognition was pioneered by Dr. Baochang Zhang, but we have significantly extended and elaborated this concept. We have extended the concept of LDP from its original usage on Gabor phase features only to much more generalised definition on gray-level images. We have rewritten and enlarged the original draft of his manuscript. Some of the experiments were also implemented and reported by us. In the third part of this thesis, we pay attention to the representation of 'Average Face', which was newly published on Science and claimed to be capable of dramatically improving performance of face recognition systems. To reveal its working mechanism, we conduct a comparative study to observe its effectiveness on holistic and local face recognition approaches. Our experimental results reveal that the process of face averaging does not necessarily improve all the face recognition systems. Its usefulness is dependent on the specific methods employed in practice.
Thesis (PhD Doctorate)
Doctor of Philosophy (PhD)
Griffith School of Engineering
Science, Environment, Engineering and Technology
Full Text

This thesis develops an algorithm enabling accurate tracking of human faces, precise estimation of head poses, efficient resolution of occlusions and improved depth perception under different lighting conditions. The system also utilises two stereo cameras that have the ability to track movements across six degrees of freedom, thereby accounting for pose variations. The system can address circumstances in which facial features are no longer discernible, as the results demonstrate increased accuracy in real-time estimation of head poses and facial landmark features. It can also precisely map facial features in different head poses, making it extremely robust for applying 3D facial tracking solutions.

Tato práce se zaměřuje na problém počítání vozidel v statickém obraze bez znalosti geometrických vlastností scény. V rámci řešení bylo implementováno a natrénováno 5 architektur konvolučních neuronových sítí. Také byl pořízen rozsáhlý dataset s 19 310 snímky pořízených z 12pohledů a zachycujících 7 různých scén. Použité konvoluční sítě mapují vstupní vzorek na mapu hustoty vozidel, ze které lze získat jejich počet a lokalizaci v kontextu vstupního snímku. Hlavním přínosem této práce je porovnání a aplikace dosavadních nejlepších řešení pro počítání objektů v obraze. Většina z těchto architektur byla navržena pro počítání lidí v obraze, proto musely být uzpůsobeny pro potřeby počítání vozidel v statickém obraze. Natrénované modely jsou vyhodnoceny GAME metrikou na TRANCOS datasetu a na velkém spojeném datasetu. Dosažené výsledky všech modelů jsou následně popsány a porovnány.

Dissertação de Mestrado Integrado em Engenharia Electrotécnica e de Computadores apresentada à Faculdade de Ciências e Tecnologia
A Artroplastia total do joelho (ATJ) é um procedimento cirúrgico que consiste na substituição da região articular do joelho por uma prótese do joelho. Foram investigados e desenvolvidos sistemas de navegação baseados em computadores para melhorar o resultados destes procedimento cirúrgicos. Estes sistemas ajudam o cirurgião em planear a posição mais adequada para as próteses e auxiliam-no durante o procedimento cirúrgico a seguir da forma mais eficiente o plano cirúrgico definido.Esta tese é desenvolvida na recente introdução de sistemas de navegação por vídeo para ATJ e é focada em navegação image-free. Este tipo de navegação, requer a aquisição de determinadas pontos de referência anatómicas durante a cirurgia. No entanto, muitas vezes o processo de identificação destes pontos de referência é realizado manualmente, o que é demorado, tem falta de precisão e alta variabilidade, levando a erros significativos no posicionamento das próteses. Este trabalho apresenta uma pipeline para deteção e localização automática dos pontos de referência a partir de imagens RGB capturadas durante a cirurgia. O objetivo é guiar e assistir cirurgiões em localizar os pontos de referência fornecendo sugestões em tempo real sobre a localização dos mesmos. A solução proposta é baseada em algoritmos do estado da arte de deep learning, que combinamos com uma estratégia rápida e eficiente de gerar dados de treino que desenvolvemos especificamente para o nosso problema. Os resultados experimentais obtidos utilizando dados de cirurgias reais revelam um desempenho promissor, apresentando capacidade de generalização para diferentes dados intra-operatórios e estimações confiáveis que atendem aos requisitos clínicos e funciona em tempo real.
Total knee arthroplasty (TKA) is a surgical procedure that consists in replacing the entire knee joint by artificial knee implants. Computer-based navigation systems have been investigated and developed to improve the outcome of TKA procedures. These systems support the surgeon in planning the most adequate position for the implants, and assist during the procedure in effectively following the defined surgical plan. This thesis is built on the recent introduction of video-based navigation systems in the context of TKA and is focused on image-free navigation. This type of navigation requires the acquisition of particular anatomical landmarks intra-operatively. The accurate localization of these anatomical landmarks is essential for the success of the surgery. However, the landmark identification process is often conducted manually, which is time-consuming, lacks accuracy and has high variability, leading to significant errors in implant positioning. This work presents an end-to-end pipeline for automatic detection and localization of anatomical landmarks from RGB images acquired during the surgery. The aim is to guide and assist surgeons in locating the anatomical landmarks by providing real time suggestions about the landmark location. The proposed solution is based in state-of-the-art deep learning strategies, which we combine with a fast and effective labelling framework which we developed specifically to provide the required annotations. The experimental results using real surgery data show encouraging performance, presenting generelization capabilities for unseen data and reliable predictions that meet the clinical requirements, running in real-time.

碩士
國立臺灣科技大學
機械工程系
104
We propose the Generalized Biologically Inspired Features (GBIFs) and a moving segmentation scheme followed by soft boundary regression for age estimation. The GBIF is more advantageous than the Bio-Inspired Feature (BIF) for capturing age-related facial traits. The moving segmentation is proposed to better determine the age groups, leading to an improvement on the age estimation accuracy. Different from most approaches that segment the age groups in an ad-hoc way, the moving segmentation allows one to define age groups using the local minima in the misclassification rate across ages. The extraction of the GBIF depends on the partition of component regions defined by facial landmarks. In addition to the partition of component regions, we also study the appropriate age grouping and hierarchical classification, and determine the best configuration for age estimation. The proposed approach with the most appropriate settings outperforms most of the state of the art on two benchmarks, FG-NET and MORPH.

The exponential rise in the availability of data over the past decade has fuelled research in deep learning. While supervised deep learning models achieve near-human performance using annotated data, it comes with an additional cost of annotation. Additionally, there could be ambiguity in annotations due to human error. While an image classification task assigns one label to the whole image, as we increase the granularity of the task to landmark estimation, the annotator needs to pinpoint the landmark accurately. The self-supervised learning (SSL) paradigm overcomes these concerns by using pretext task based objectives to learn from large-scale unannotated data. In this work, we show how to extract relevant signals from pretrained self-supervised networks for a) a discriminative task of landmark estimation under limited annotations, and b) increasing perceptual quality of the images generated by generative adversarial network. In this first part, we demonstrate the emergent correspondence tracking properties in the non-contrastive SSL framework. Using this as supervision, we propose LEAD which is an approach to discover landmarks from an unannotated collection of category-specific images. Existing works in self-supervised landmark detection are based on learning dense (pixel-level) feature representations from an image, which are further used to learn landmarks in a semi-supervised manner. While there have been advances in self-supervised learning of image features for instance-level tasks like classification, these methods do not ensure dense equivariant representations. The property of equivariance is of interest for dense prediction tasks like landmark estimation. In this work, we introduce an approach to enhance the learning of dense equivariant representations in a self-supervised fashion. We follow a two-stage training approach: first, we train a network using the BYOL objective which operates at an instance level. The correspondences obtained through this network are further used to train a dense and compact representation of the image using a lightweight network. We show that having such a prior in the feature extractor helps in landmark detection, even under a drastically limited number of annotations while also improving generalization across scale variations. Next, we utilize the rich feature space from the SSL framework as a “naturalness” prior to alleviate unnatural image generation from Generative Adversarial Networks (GAN), which is a popular class of generative models. Progress in GANs has enabled the generation of high-resolution photorealistic images of astonishing quality. StyleGANs allow for compelling attribute modification on such images via mathematical operations on the latent style vectors in the W/W+ space that effectively modulates the rich hierarchical representations of the generator. Such operations have recently been generalized beyond mere attribute swapping in the original StyleGAN paper to include interpolations. In spite of many significant improvements in StyleGANs, they are still seen to generate unnatural images. The quality of the generated images is a function of, (a) richness of the hierarchical representations learned by the generator, and, (b) linearity and smoothness of the style spaces. In this work, we propose Hierarchical Semantic Regularizer (HSR) which aligns the hierarchical representations learnt by the generator to corresponding powerful features learned by pretrained networks on large amounts of data. HSR not only improves generator representations but also the linearity and smoothness of the latent style spaces, leading to the generation of more natural-looking style-edited images. To demonstrate improved linearity, we propose a novel metric - Attribute Linearity Score (ALS). A significant reduction in the generation of unnatural images is corroborated by improvement in the Perceptual Path Length (PPL) metric by 15% across different standard datasets while simultaneously improving the linearity of attribute-change in the attribute editing tasks.

碩士
國立臺灣科技大學
機械工程系
103
Tree Structured model (TSM) is proven effective for face detection, landmark localization and pose estimation. It is a rare approach that can solve all three issues using one single unified model. However, it can be too slow to handle real-time applications because of the heavy computation involved. Besides, it cannot detect faces less than 80x80 in size. A bilayer structure, coined Bilayer Tree Structure Model(BTSM), is proposed in this study to solve these two issues. The BTSM has a downscaled model with fewer parts and trained on down-scaled samples, and therefore, can detect faces as small as 50x50. When the down-scaled model finds faces of sufficient sizes, it would activate a full-scaled model to locate more landmarks without performing convolution through the image pyramid. Compared on various databases, the BTSM can be 30x faster than the original TSM, while keeping almost all advantages of TSM the same.

碩士
育達商業技術學院
資訊管理所
94
Abstract Position and velocity errors of Inertial Navigation System (INS) increase unboundedly in time. In order to improve the performance parameters, the alternative of an INS assisted by Global Positioning System (GPS) is frequently adopted. However, GPS still has many unsolved problems existing, for instance the degradation of positioning precision when the view of sky is obstructed. Pseudolite (pseudo-satellite, PL), a ground-based GPS satellite-like signal transmitter, can provide extra measurements in order to improve the positioning precision from GPS alone in the obstructed areas. The pseudolite observations reduced the effect of dilution of precision (DOP) that leads navigation strategy of mobile vehicles. In this research, the pseudolites can be used as landmarks. The problem dealt with here is that of estimating the kinematic state components of a vehicle in autonomous navigation using range, elevation and azimuth angles measured by Line of Sight (LOS) coordinate system. The estimates of the position and velocity of the vehicle are provided by the hybrid coordinate filtering approach (Modified Spherical Coordinates and Cartesian Coordinates Systems). The data fusion algorithm called covariance matching method is employed to reduce the estimation errors. Results of this research show that the Averaged Root Mean Square Error (ARMSE) of position and velocity with the filters was found to be larger (about 87% and 76%) than with the data fusion. Simulation results of proposed algorithm show that the two landmarks may influence the estimating accuracies of mobile vehicle navigation.

Flying animals such as bats, birds, and moths are actively studied by researchers wanting to better understand these animals’ behavior and flight characteristics. Towards this goal, multi-view videos of flying animals have been recorded both in lab- oratory conditions and natural habitats. The analysis of these videos has shifted over time from manual inspection by scientists to more automated and quantitative approaches based on computer vision algorithms. This thesis describes a study on the largely unexplored problem of 3D pose estimation of flying animals in multi-view video data. This problem has received little attention in the computer vision community where few flying animal datasets exist. Additionally, published solutions from researchers in the natural sciences have not taken full advantage of advancements in computer vision research. This thesis addresses this gap by proposing three different approaches for 3D pose estimation of flying animals in multi-view video datasets, which evolve from successful pose estimation paradigms used in computer vision. The first approach models the appearance of a flying animal with a synthetic 3D graphics model and then uses a Markov Random Field to model 3D pose estimation over time as a single optimization problem. The second approach builds on the success of Pictorial Structures models and further improves them for the case where only a sparse set of landmarks are annotated in training data. The proposed approach first discovers parts from regions of the training images that are not annotated. The discovered parts are then used to generate more accurate appearance likelihood terms which in turn produce more accurate landmark localizations. The third approach takes advantage of the success of deep learning models and adapts existing deep architectures to perform landmark localization. Both the second and third approaches perform 3D pose estimation by first obtaining accurate localization of key landmarks in individual views, and then using calibrated cameras and camera geometry to reconstruct the 3D position of key landmarks. This thesis shows that the proposed algorithms generate first-of-a-kind and leading results on real world datasets of bats and moths, respectively. Furthermore, a variety of resources are made freely available to the public to further strengthen the connection between research communities.

碩士
國立成功大學
醫學工程研究所
83
The hip joint can be closely approximated as a ball-and-socket joint. The accuracy of method of estimation of hip joint centers (HJC) from external landmarks can greatly influence the kinematic and kinetic analysis of gait and other human motions. The purposes of this study were (1) to estimate the three-dimensional HJC location by using function method as in Cappozzo (where the HC is assumed to be the center of the sphere described by markers located on the thigh during a flexion-extension followed by an ab-adduction of the hip) and (2) to determine the effects of three different methods of the HJC parameter on gait analysis application. The video-based Motion Analysis system was used for hip motion measurements. The results show that the center of the sphere method was reliable and the estimated HJC was within 1.01 cm radius sphere. In fact, for level walking of the normal subject, these results demonstrate that the joint angles are insensitive, however, the joint moments (10%) and joint powers (20%) are sensitive to the location of the hip center. The future studies suggest to combine the detailed anatomy from 3-D CT images and the joint forces from the gait analysis together, to study the pathomechanics of developmental dysplasia of hip (DDH).

碩士
育達商業技術學院
資訊管理所
94
[Abstract] The mobile location is to continuously confirm the position, direction, and velocity of a mobile vehicle. When the vehicle move, the vehicle’s position, velocity and direction have to be known and predicted. The mobile vehicle has to confirm its position and the trajectory for not being lost. Global Positioning System (GPS) is often utilized to eliminate the accumulated error from the Inertial Navigation System (INS). However, the accuracy of GPS positioning is highly related to the number and distribution of the available GPS satellites being tracked. This research uses the landmark of wireless communication by using the pseudo-satellite (pseudolite, PL), which is a ground-based GPS satellite-like signal transmitter. The problem dealt with here is that of estimating the kinematic state components of a vehicle in autonomous navigation using range, elevation and azimuth angles measured by Line of Sight (LOS) coordinate system. The estimates of the absolute position and velocity of the vehicle in Local Inertial Cartesian Coordinate System (LICCS) are provided by Kalman filter and the data fusion algorithm called covariance matching method. Performance results for the proposed algorithm are compared with those of Kalman filters, using difference simulations of typical vehicle maneuvering scenarios. Results of this research show that the Averaged Root Mean Square Error (ARMSE) of position and velocity with the filters was found to be larger (about 26% and 8%) than with the data fusion. Keyword：Landmark, Kalman Filter, Data Fusion.

Cette thèse porte sur le recalage de signaux à partir de l'alignement de leurs landmarks, pour la comparaison d'ensembles de courbes ou d'images. Après une revue des techniques de recalage qui existent dans la littérature, une approche nonparamétrique est proposée pour estimer les landmarks d'une fonction 1D bruitée à partir des lignes de maxima d'ondelettes et de zero-crossings de sa transformée continue en ondelettes. Un nouvel outil, l'intensité structurelle, est introduit pour représenter les positions des points caractéristiques d'une courbe sous forme d'une densité de probabilité. Cette méthode conduit à une nouvelle technique de mise en correspondance automatique des landmarks de deux fonctions. L'approche envisagée dans le cas 1D est étendue au cas 2D en utilisant des décompositions en wedgelets/platelets pour détecter les contours d'une image. De nombreuses simulations et des problèmes réels d'analyse de variance fonctionnelle servent d'illustration des méthodes proposées.