Dissertations / Theses on the topic 'Visual Odometry'

Extrair informação de profundidade a partir de imagens bidimensionais é um importante problema na área de visão computacional. Diversas aplicações se beneficiam desta classe de algoritmos tais como: robótica, a indústria de entretenimento, aplicações médicas para diagnóstico e confecção de próteses e até mesmo exploração interplanetária. Esta aplicação pode ser dividida em duas etapas interdependentes: a estimação da posição e orientação da câmera no momento em que a imagem foi gerada, e a estimativa da estrutura tridimensional da cena. Este trabalho foca em técnicas de visão computacional usadas para estimar a trajetória de um veículo equipado com uma câmera, problema conhecido como odometria visual. Para obter medidas objetivas de eficiência e precisão, e poder comparar os resultados obtidos com o estado da arte, uma base de dados de alta precisão, bastante utilizada pela comunidade científica foi utilizada. No curso deste trabalho novas técnicas para rastreamento de detalhes, estimativa de posição de câmera, cálculo de posição 3D de pontos e recuperação de escala são propostos. Os resultados alcançados superam os mais bem ranqueados trabalhos na base de dados escolhida até o momento da publicação desta tese.
Recovering three-dimensional information from bi-dimensional images is an important problem in computer vision that finds several applications in our society. Robotics, entertainment industry, medical diagnose and prosthesis, and even interplanetary exploration benefit from vision based 3D estimation. The problem can be divided in two interdependent operations: estimating the camera position and orientation when each image was produced, and estimating the 3D scene structure. This work focuses on computer vision techniques, used to estimate the trajectory of a vehicle equipped camera, a problem known as visual odometry. In order to provide an objective measure of estimation efficiency and to compare the achieved results to the state-of-the-art works in visual odometry a high precision popular dataset was selected and used. In the course of this work new techniques for image feature tracking, camera pose estimation, point 3D position calculation and scale recovery are proposed. The achieved results outperform the best ranked results in the popular chosen dataset.

This Master thesis tackles the problem of measuring objects’ directions from a motionlessobservation point. A new method based on a single rotating camera requiring the knowledge ofonly two (or more) landmarks’ direction is proposed. In a first phase, multi-view geometry isused to estimate camera rotations and key elements’ direction from a set of overlapping images.Then in a second phase, the direction of any object can be estimated by resectioning the cameraassociated to a picture showing this object. A detailed description of the algorithmic chain isgiven, along with test results on both synthetic data and real images taken with an infraredcamera.
Detta masterarbete behandlar problemet med att mäta objekts riktningar från en fastobservationspunkt. En ny metod föreslås, baserad på en enda roterande kamera som kräverendast två (eller flera) landmärkens riktningar. I en första fas används multiperspektivgeometri,för att uppskatta kamerarotationer och nyckelelements riktningar utifrån en uppsättningöverlappande bilder. I en andra fas kan sedan riktningen hos vilket objekt som helst uppskattasgenom att kameran, associerad till en bild visande detta objekt, omsektioneras. En detaljeradbeskrivning av den algoritmiska kedjan ges, tillsammans med testresultat av både syntetisk dataoch verkliga bilder tagen med en infraröd kamera.

In this master thesis a visual odometry system is implemented and explained. Visual odometry is a technique, which could be used on autonomous vehicles to determine its current position and is preferably used indoors when GPS is notworking. The only input to the system are the images from a stereo camera and the output is the current location given in relative position. In the C++ implementation, image features are found and matched between the stereo images and the previous stereo pair, which gives a range of 150-250 verified feature matchings. The image coordinates are triangulated into a 3D-point cloud. The distance between two subsequent point clouds is minimized with respect to rigid transformations, which gives the motion described with six parameters, three for the translation and three for the rotation. Noise in the image coordinates gives reconstruction errors which makes the motion estimation very sensitive. The results from six experiments show that the weakness of the system is the ability to distinguish rotations from translations. However, if the system has additional knowledge of how it is moving, the minimization can be done with only three parameters and the system can estimate its position with less than 5 % error.

Thesis: S.M., Massachusetts Institute of Technology, Department of Aeronautics and Astronautics, 2019
Cataloged from PDF version of thesis.
Includes bibliographical references (pages 57-62).
Recent advances in technology indicate that autonomous vehicles and self-driving cats in particular may become commonplace in the near future. This thesis contributes to that scenario by studying the problem of depth perception based on sequences of camera images. We start by presenting a sensor fusion framework that achieves state-of-the-art performance when completing depth from sparse LiDAR measurements and a camera. Then, we study how the system performs under a variety of modifications of the sparse input until we ultimately replace LiDAR measurements with triangulations from a typical sparse visual odometry pipeline. We are then able to achieve a small improvement over the single image baseline and chart guidelines to assist in designing a system with even more substantial gains.
by Guilherme Venturelli Cavalheiro.
S.M.
S.M. Massachusetts Institute of Technology, Department of Aeronautics and Astronautics

Monocular cameras are prominently used for estimating motion of Unmanned Aerial Vehicles. With growing interest in autonomous vehicle technology, the use of monocular cameras in ground vehicles is on the rise. This is especially favorable for localization in situations where Global Navigation Satellite System (GNSS) is unreliable, such as open-pit mining environments. However, most monocular camera based approaches suffer due to obscure scale information. Ground vehicles impose a greater difficulty due to high speeds and fast movements. This thesis aims to estimate the scale of monocular vision data by using an inertial sensor in addition to the camera. It is shown that the simultaneous estimation of pose and scale in autonomous ground vehicles is possible by the fusion of visual and inertial sensors in an Extended Kalman Filter (EKF) framework. However, the convergence of scale is sensitive to several factors including the initialization error. An accurate estimation of scale allows the accurate estimation of pose. This facilitates the localization of ground vehicles in the absence of GNSS, providing a reliable fall-back option.
Monokulära kameror används ofta vid rörelseestimering av obemannade flygande farkoster. Med det ökade intresset för autonoma fordon har även användningen av monokulära kameror i fordon ökat. Detta är fram för allt fördelaktigt i situationer där satellitnavigering (Global Navigation Satellite System (GNSS)) äropålitlig, exempelvis i dagbrott. De flesta system som använder sig av monokulära kameror har problem med att estimera skalan. Denna estimering blir ännu svårare på grund av ett fordons större hastigheter och snabbare rörelser. Syftet med detta exjobb är att försöka estimera skalan baserat på bild data från en monokulär kamera, genom att komplettera med data från tröghetssensorer. Det visas att simultan estimering av position och skala för ett fordon är möjligt genom fusion av bild- och tröghetsdata från sensorer med hjälp av ett utökat Kalmanfilter (EKF). Estimeringens konvergens beror på flera faktorer, inklusive initialiseringsfel. En noggrann estimering av skalan möjliggör också en noggrann estimering av positionen. Detta möjliggör lokalisering av fordon vid avsaknad av GNSS och erbjuder därmed en ökad redundans.

This thesis addresses the problem of incremental localization from visual information, a scenario commonly known as visual odometry. Current visual odometry algorithms are heavily dependent on camera calibration, using a pre-established geometric model to provide the transformation between input (optical flow estimates) and output (vehicle motion estimates) information. A novel approach to visual odometry is proposed in this thesis where the need for camera calibration, or even for a geometric model, is circumvented by the use of machine learning principles and techniques. A non-parametric Bayesian regression technique, the Gaussian Process (GP), is used to elect the most probable transformation function hypothesis from input to output, based on training data collected prior and during navigation. Other than eliminating the need for a geometric model and traditional camera calibration, this approach also allows for scale recovery even in a monocular configuration, and provides a natural treatment of uncertainties due to the probabilistic nature of GPs. Several extensions to the traditional GP framework are introduced and discussed in depth, and they constitute the core of the contributions of this thesis to the machine learning and robotics community. The proposed framework is tested in a wide variety of scenarios, ranging from urban and off-road ground vehicles to unconstrained 3D unmanned aircrafts. The results show a significant improvement over traditional visual odometry algorithms, and also surpass results obtained using other sensors, such as laser scanners and IMUs. The incorporation of these results to a SLAM scenario, using a Exact Sparse Information Filter (ESIF), is shown to decrease global uncertainty by exploiting revisited areas of the environment. Finally, a technique for the automatic segmentation of dynamic objects is presented, as a way to increase the robustness of image information and further improve visual odometry results.

Thesis: S.M., Massachusetts Institute of Technology, Department of Aeronautics and Astronautics, 2019
Cataloged from PDF version of thesis.
Includes bibliographical references (pages 51-52).
Autonomous navigation for robotic platforms, particularly techniques that leverage an onboard camera, are of currently of significant interest to the robotics community. Designing methods to localize small, resource-constrained robots is a particular challenge due to limited availability of computing power and physical space for sensors. A computer vision, machine learning-based localization method was proposed by researchers investigating the automation of medical procedures. However, we believed the method to also be promising for low size, weight, and power (SWAP) budget robots. Unlike for traditional odometry methods, in this case, a machine learning model can be trained offline, and can then generate odometry measurements quickly and efficiently. This thesis describes the implementation of the learning-based, visual odometry method in the context of autonomous drones. We refer to the method as RetiNav due to its similarities with the way the human eye processes light signals from its surroundings. We make several modifications to the method relative to the initial design based on a detailed parameter study, and we test the method on a variety of challenging flight datasets. We show that over the course of a trajectory, RetiNav achieves as low as 1.4% error in predicting the distance traveled. We conclude that such a method is a viable component of a localization system, and propose the next steps for work in this area.
by Tori Wuthrich.
S.M.
S.M. Massachusetts Institute of Technology, Department of Aeronautics and Astronautics

A new visual registration algorithm (Adaptive Iterative Closest Keypoint, AICK) is tested and evaluated as a positioning tool on a Micro Aerial Vehicle (MAV). Captured frames from a Kinect like RGB-D camera are analyzed and an estimated position of the MAV is extracted. The hope is to find a positioning solution for GPS-denied environments. This thesis is focused on an indoor office environment. The MAV is flown manually, capturing in-flight RGB-D images which are registered with the AICK algorithm. The result is analyzed to come to a conclusion if AICK is viable or not for autonomous flight based on on-board positioning estimates. The result shows potential for a working autonomous MAV in GPS-denied environments, however there are some surroundings that have proven difficult. The lack of visual features on e.g., a white wall causes problems and uncertainties in the positioning, which is even more troublesome when the distance to the surroundings exceed the RGB-D cameras depth range. With further work on these weaknesses we believe that a robust autonomous MAV using AICK for positioning is plausible.
En ny visuell registreringsalgoritm (Adaptive Iterative Closest Keypoint, AICK) testas och utvärderas som ett positioneringsverktyg på en Micro Aerial Vehicle (MAV). Tagna bilder från en Kinect liknande RGB-D kamera analyseras och en approximerad position av MAVen beräknas. Förhoppningen är att hitta en positioneringslösning för miljöer utan GPS förbindelse, där detta arbete fokuserar på kontorsmiljöer inomhus. MAVen flygs manuellt samtidigt som RGB-D bilder tas, dessa registreras sedan med hjälp av AICK. Resultatet analyseras för att kunna dra en slutsats om AICK är en rimlig metod eller inte för att åstadkomma autonom flygning med hjälp av den uppskattade positionen. Resultatet visar potentialen för en fungerande autonom MAV i miljöer utan GPS förbindelse, men det finns testade miljöer där AICK i dagsläget fungerar undermåligt. Bristen på visuella särdrag på t.ex. en vit vägg inför problem och osäkerheter i positioneringen, ännu mer besvärande är det när avståndet till omgivningen överskrider RGB-D kamerornas räckvidd. Med fortsatt arbete med dessa svagheter är en robust autonom MAV som använder AICK för positioneringen rimlig.

Precise pose information is a fundamental prerequisite for numerous applications in robotics, AI and mobile computing. Monocular cameras are the ideal sensor for this purpose - they are cheap, lightweight and ubiquitous. As such, monocular visual localization is widely regarded as a cornerstone requirement of machine perception. However, a large gap still exists between the performance that these applications require and that which is achievable through existing monocular perception algorithms. In this thesis we directly tackle the issue of robust egocentric visual localization and mapping through a data-centric approach. As a first major contribution we propose novel learnt models for visual odometry which form the basis of the ego-motion estimates used in later chapters. The proposed approaches are less fragile and much more robust than existing approaches. We present experimental evidence that these approaches can not only approach the accuracy of standard methods but in many cases also show major improvements in computational and memory efficiency. To cope with the drift inherent to the odometry methods, we then introduce a novel learnt spatio-temporal model for performing global relocalization updates. The proposed approach allows one to efficiently infer the global location of an image stream at the fraction of the time of traditional feature-based approaches with minimal loss in localization accuracy. Finally, we present a novel SLAM system integrating our learnt priors for creating 3D maps from monocular image sequences. The approach is designed to harness multiple input sources, including prior depth and ego-motion estimates and incorporates both loop-closure and relocalization updates. The approach, based on the well-established standard visual-inertial structure-from-motion process, allows us to perform accurate posterior inference of camera poses and scene structure to significantly boost the reconstruction robustness and fidelity. Through our qualitative and quantitative experimentation on a wide range of datasets, we conclude that the proposed methods can bring accurate visual localization to a wide class of consumer devices and robotic platforms.

This thesis work belongs to the field of self-supervised monocular depth estimation and constitutes a conversion of the work done in [1]. The purpose is to consider the computationally expensive model in [1] as the baseline model of this work and try to create a lightweight model out of it. The current work proposes a network suited to be deployed on embedded devices such as NVIDIA Jetson TX2 where the needs for short runtime, small memory footprint, and power consumption matters the most. In other words, if those requirements are missing, no matter if precision is extraordinarily high, the model cannot be functional on embedded processors. Thus, mobile platforms with small size such as drones, delivery robots, etc. cannot exploit the benefits of deep learning. The proposed network has _29.7 less parameters than the baseline model [1] and uses only 10.6 MB for a forward pass in contrast to 227MB used by the network in [1]. Consequently, the proposed model can be functional on embedded devices’ GPU. Lastly, it is able to infer depth with promising speed even on standard CPUs and at the same time provides comparable or higher accuracy than other works.
Detta examensarbete tillhör området för självkontrollerad monokulär djupbedömning och utgör en omvandling av det arbete som gjorts under [1]. Syftet är att överväga den beräkningsmässiga dyra modellen i [1] som basmodellen för detta arbete och försöka skapa en lätt modell ur den. Det nuvarande arbetet förutsätter ett nätverk som är lämpligt att distribueras på inbäddade enheter som NVIDIA Jetson TX2 där behoven för kort driftstid, liten minnesfotavtryck och kraftförbrukning är viktigast. Med andra ord, om dessa krav saknas, oavsett om precisionen är extra hög, kan modellen inte fungera på inbäddade processorer. Således kan mobilplattformar med små storlekar som drönare, leveransrobotar, etc. inte utnyttja fördelarna med djupbildning. Det föreslagna nätverket har _29,7 mindre parametrar än baselinemodellen [1] och använder endast 10,6MB för ett framåtpass i motsats till 227MB som används av nätverket i [1]. Följaktligen kan den föreslagna modellen fungera på inbäddade enheters GPU. Slutligen kan den dra slutsatsen med lovande hastighet på standard CPUs och samtidigt ger jämförbar eller högre noggrannhet än andra arbete.

This thesis investigates techniques and designs an autonomous visual stereo based navigation sensor to improve stereo visual odometry for purpose of navigation in unknown environments. In particular, autonomous navigation in a space mission context which imposes challenging constraints on algorithm development and hardware requirements. For instance, Global Positioning System (GPS) is not available in this context. Thus, a solution for navigation cannot rely on similar external sources of information. Support to handle this problem is required with the conception of an intelligent perception-sensing device that provides precise outputs related to absolute and relative 6 degrees of freedom (DOF) positioning. This is achieved using only images from stereo calibrated cameras possibly coupled with an inertial measurement unit (IMU) while fulfilling real time processing requirements. Moreover, no prior knowledge about the environment is assumed. Robotic navigation has been the motivating research to investigate different and complementary areas such as stereovision, visual motion estimation, optimisation and data fusion. Several contributions have been made in these areas. Firstly, an efficient feature detection, stereo matching and feature tracking strategy based on Kanade-Lucas-Tomasi (KLT) feature tracker is proposed to form the base of the visual motion estimation. Secondly, in order to cope with extreme illumination changes, High dynamic range (HDR) imaging solution is investigated and a comparative assessment of feature tracking performance is conducted. Thirdly, a two views local bundle adjustment scheme based on trust region minimisation is proposed for precise visual motion estimation. Fourthly, a novel KLT feature tracker using IMU information is integrated into the visual odometry pipeline. Finally, a smart standalone stereo visual/IMU navigation sensor has been designed integrating an innovative combination of hardware as well as the novel software solutions proposed above. As a result of a balanced combination of hardware and software implementation, we achieved 5fps frame rate processing up to 750 initials features at a resolution of 1280x960. This is the highest reached resolution in real time for visual odometry applications to our knowledge. In addition visual odometry accuracy of our algorithm achieves the state of the art with less than 1% relative error in the estimated trajectories.

Nowadays visual and inertial information is readily available from small mobile platforms, such as quadcopters. However, due to the limitation of onboard resource and capability, it is still a challenge to developing localisation and mapping estimation algorithms for small size mobile platforms. Visual-based techniques for tracking or motion estimation related tasks have been developed abundantly, especially using interest points as features. However, such sparse feature-based methods are quickly getting divergence, due to noise, partial occlusion or light condition variation in views. Only in recent years, direct visual based approaches, which densely, semi-densely or statistically use pixel information reveal significant improvement in algorithm robustness and stability. On the other hand, inertial sensors measure the changes in angular velocity and linear acceleration, which can be further integrated to predict relative velocity, position and orientation for mobile platforms. In practical usage, the accumulated error from inertial sensors is often compensated by cameras, while the loss of agile egomotion from visual sensors can be compensated by inertial-based motion estimation. Based on the complementary nature of visual and inertial information, in this research, we focus on how to use the direct visual based approaches to providing location information through a monocular camera, while fusing with the inertial information to enhance the robustness and accuracy. The proposed algorithms can be applied to practical datasets which are collected from mobile platforms. Particularly, direct-based and mutual information based methods are explored in details. Two visual-inertial odometry algorithms are proposed in the framework of multi-state constraint Kalman filter. They are also tested with the real data from a flying robot in complex indoor and outdoor environments. The results show that the direct-based methods have the merits of robustness in image processing and accuracy in the case of moving along straight lines with a slight rotation. Furthermore, the visual and inertial fusion strategies are investigated to build their intrinsic links, then the improvement done by iterative steps in filtering propagation is proposed. As an addition, for experimental implementation, a self-made flying robot for data collection is also developed.

This thesis explored the utility of long-range stereo visual odometry for application on Unmanned Aerial Vehicles. Novel parameterisations and initialisation routines were developed for the long-range case of stereo visual odometry and new optimisation techniques were implemented to improve the robustness of visual odometry in this difficult scenario. In doing so, the applications of stereo visual odometry were expanded and shown to perform adequately in situations that were previously unworkable.

Plutôt que de générer des images de manière constante et synchrone, les capteurs neuromorphiques de vision -également connus sous le nom de caméras événementielles, permettent à chaque pixel de fournir des informations de manière indépendante et asynchrone chaque fois qu'un changement de luminosité est détecté. Par conséquent, les capteurs de vision neuromorphiques n'ont pas les problèmes des caméras conventionnelles telles que les artefacts d'image et le Flou cinétique. De plus, ils peuvent fournir une compression sans perte de donné avec une résolution temporelle et une plage dynamique plus élevée. Par conséquent, les caméras événmentielles remplacent commodément les caméras conventionelles dans les applications robotiques nécessitant une grande maniabilité et des conditions environnementales variables. Dans cette thèse, nous abordons le problème de l'odométrie visio-inertielle à l'aide de caméras événementielles et d'une centrale inertielle. En exploitant la cohérence des caméras événementielles avec les conditions de constance de la luminosité, nous discutons de la possibilité de construire un système d'odométrie visuelle basé sur l'estimation du flot optique. Nous développons notre approche basée sur l'hypothèse que ces caméras fournissent des informations des contours des objets de la scène et appliquons un algorithme de détection de ligne pour la réduction des données. Le suivi de ligne nous permet de gagner plus de temps pour les calculs et fournit une meilleure représentation de l'environnement que les points d'intérêt. Dans cette thèse, nous ne montrons pas seulement une approche pour l'odométrie visio-inertielle basée sur les événements, mais également des algorithmes qui peuvent être utilisés comme algorithmes des caméras événementielles autonomes ou intégrés dans d'autres approches si nécessaire
Rather than generating images constantly and synchronously, neuromorphic vision sensors -also known as event-based cameras- permit each pixel to provide information independently and asynchronously whenever brightness change is detected. Consequently, neuromorphic vision sensors do not encounter the problems of conventional frame-based cameras like image artifacts and motion blur. Furthermore, they can provide lossless data compression, higher temporal resolution and higher dynamic range. Hence, event-based cameras conveniently replace frame-based cameras in robotic applications requiring high maneuverability and varying environmental conditions. In this thesis, we address the problem of visual-inertial odometry using event-based cameras and an inertial measurement unit. Exploiting the consistency of event-based cameras with the brightness constancy conditions, we discuss the availability of building a visual odometry system based on optical flow estimation. We develop our approach based on the assumption that event-based cameras provide edge-like information about the objects in the scene and apply a line detection algorithm for data reduction. Line tracking allows us to gain more time for computations and provides a better representation of the environment than feature points. In this thesis, we do not only show an approach for event-based visual-inertial odometry but also event-based algorithms that can be used as stand-alone algorithms or integrated into other approaches if needed

Thesis: S.M., Massachusetts Institute of Technology, Department of Aeronautics and Astronautics, 2017.
Cataloged from PDF version of thesis.
Includes bibliographical references (pages 147-151).
The navigation problem for mobile robots operating in unknown environments can be posed as a subset of Simultaneous Localization and Mapping (SLAM). For computationally-constrained systems, maintaining and promoting system sparsity is key to achieving the high-rate solutions required for agile trajectory tracking. This thesis focuses on the computation involved in the elimination step of optimization, showing it to be a function of the corresponding graph structure. This observation directly motivates the search for measurement selection techniques to promote sparse structure and reduce computation. While many sophisticated selection techniques exist in the literature, relatively little attention has been paid to the simple yet ubiquitous heuristic of decimation. This thesis shows that decimation produces graphs with an inherently sparse, partitioned super-structure. Furthermore, it is shown analytically for single-landmark graphs that the even spacing of observations characteristic of decimation is near optimal in a weighted number of spanning trees sense. Recent results in the SLAM community suggest that maximizing this connectivity metric corresponds to good information-theoretic performance. Simulation results confirm that decimation-style strategies perform as well or better than sophisticated policies which require significant computation to execute. Given that decimation consumes negligible computation to evaluate, its performance demonstrated here makes decimation a formidable measurement selection strategy for high-rate, realtime SLAM solutions. Finally, the SAMWISE visual-inertial estimator is described, and thorough experimental results demonstrate its robustness in a variety of scenarios, particularly to the challenges prescribed by the DARPA Fast Lightweight Autonomy program.
This thesis was supported by the Defense Advanced Research Projects Agency (DARPA) under the Fast Lightweight Autonomy program.
by Kristoffer M. Frey.
S.M.

In this degree project global optimization methods, for a previously developedsoftwareprototype of a stereo odometry system, were studied. The existing softwareestimatesthe motion between stereo frames and builds up a map of selected stereo frameswhich accumulates increasing error over time. The aim of the project was to studymethods to mitigate the error accumulated over time in the step-wise motionestimation.One approach based on relative pose estimates and another approach based onreprojection optimization were implemented and evaluated for the existing platform.The results indicate that optimization based on relative keyframe estimates ispromising for real-time usage. The second strategy based on reprojection of stereotriangulatedpoints proved useful as a refinement step but the relatively small errorreduction comes at an increased computational cost. Therefore, this approachrequiresfurther improvements to become applicable in situations where corrections areneededin real-time, and it is hard to justify the increased computations for the relatively smallerror reduction.The results also show that the global optimization primarily improves the absolutetrajectory error.

The purpose of this project is to implement, analyze and compare visual odometry approaches to help the localization task in autonomous vehicles. The stereo visual odometry algorithm Libviso2 is compared with a proposed omnidirectional multi-camera approach. The proposed method consists of performing monocular visual odometry on all cameras individually and selecting the best estimate through a voting scheme involving all cameras. The omnidirectionality of the vision system allows the part of the surroundings richest in features to be used in the relative pose estimation. Experiments are carried out using cameras Bumblebee XB3 and Ladybug 2, fixed on the roof of a vehicle. The voting process of the proposed omnidirectional multi-camera method leads to some improvements relatively to the individual monocular estimates. However, stereo visual odometry provides considerably more accurate results.
O objetivo deste mestrado é implementar, analisar e comparar abordagens de odometria visual, de forma a contribuir para a localização de um veículo autônomo. O algoritmo de odometria visual estéreo Libviso2 é comparado com um método proposto, que usa um sistema multi-câmera omnidirecional. De acordo com este método, odometria visual monocular é calculada para cada câmera individualmente e, seguidamente, a melhor estimativa é selecionada através de um processo de votação que involve todas as câmeras. O fato de o sistema de visão ser omnidirecional faz com que a parte dos arredores mais rica em características possa sempre ser usada para estimar a pose relativa do veículo. Nas experiências são utilizadas as câmeras Bumblebee XB3 e Ladybug 2, fixadas no teto de um veículo. O processo de votação do método multi-câmera omnidirecional proposto apresenta melhorias relativamente às estimativas monoculares individuais. No entanto, a odometria visual estéreo fornece resultados mais precisos.

Simultaneous localization and mapping is an important problem in robotics that can be solved using visual odometry -- the process of estimating ego-motion from subsequent camera images. In turn, visual odometry systems rely on point matching between different frames. This work presents a novel method for matching key-points by applying neural networks to point detection and description. Traditionally, point detectors are used in order to select good key-points (like corners) and then these key-points are matched using features extracted with descriptors. However, in this work a descriptor is trained to match points densely and then a detector is trained to predict, which points are more likely to be matched with the descriptor. This information is further used for selection of good key-points. The results of this project show that this approach can lead to more accurate results compared to model-based methods.
Samtidig lokalisering och kartläggning är ett viktigt problem inom robotik som kan lösas med hjälp av visuell odometri -- processen att uppskatta självrörelse från efterföljande kamerabilder. Visuella odometrisystem förlitar sig i sin tur på punktmatchningar mellan olika bildrutor. Detta arbete presenterar en ny metod för matchning av nyckelpunkter genom att applicera neurala nätverk för detektion av punkter och deskriptorer. Traditionellt sett används punktdetektorer för att välja ut bra nyckelpunkter (som hörn) och sedan används dessa nyckelpunkter för att matcha särdrag. I detta arbete tränas istället en deskriptor att matcha punkterna. Sedan tränas en detektor till att förutspå vilka punker som är mest troliga att matchas korrekt med deskriptorn. Denna information används sedan för att välja ut bra nyckelpunkter. Resultatet av projektet visar att det kan leda till mer precisa resultat jämfört med andra modellbaserade metoder.

Au sein des systèmes avancés d’aide à la conduite (Advanced Driver Assistance Systems - ADAS) pour les systèmes de transport intelligents (Intelligent Transport Systems - ITS), les systèmes de positionnement, ou de localisation, du véhicule jouent un rôle primordial. Le système GPS (Global Positioning System) largement employé ne peut donner seul un résultat précis à cause de facteurs extérieurs comme un environnement contraint ou l’affaiblissement des signaux. Ces erreurs peuvent être en partie corrigées en fusionnant les données GPS avec des informations supplémentaires provenant d'autres capteurs. La multiplication des systèmes d’aide à la conduite disponibles dans les véhicules nécessite de plus en plus de capteurs installés et augmente le volume de données utilisables. Dans ce cadre, nous nous sommes intéressés à la fusion des données provenant de capteurs bas cout pour améliorer le positionnement du véhicule. Parmi ces sources d’information, en parallèle au GPS, nous avons considérés les caméras disponibles sur les véhicules dans le but de faire de l’odométrie visuelle (Visual Odometry - VO), couplée à une carte de l’environnement. Nous avons étudié les caractéristiques de cette trajectoire reconstituée dans le but d’améliorer la qualité du positionnement latéral et longitudinal du véhicule sur la route, et de détecter les changements de voies possibles. Après avoir été fusionnée avec les données GPS, cette trajectoire générée est couplée avec la carte de l’environnement provenant d’Open-StreetMap (OSM). L'erreur de positionnement latérale est réduite en utilisant les informations de distribution de voie fournies par OSM, tandis que le positionnement longitudinal est optimisé avec une correspondance de courbes entre la trajectoire provenant de l’odométrie visuelle et les routes segmentées décrites dans OSM. Pour vérifier la robustesse du système, la méthode a été validée avec des jeux de données KITTI en considérant des données GPS bruitées par des modèles de bruits usuels. Plusieurs méthodes d’odométrie visuelle ont été utilisées pour comparer l’influence de la méthode sur le niveau d'amélioration du résultat après fusion des données. En utilisant la technique d’appariement des courbes que nous proposons, la précision du positionnement connait une amélioration significative, en particulier pour l’erreur longitudinale. Les performances de localisation sont comparables à celles des techniques SLAM (Simultaneous Localization And Mapping), corrigeant l’erreur d’orientation initiale provenant de l’odométrie visuelle. Nous avons ensuite employé la trajectoire provenant de l’odométrie visuelle dans le cadre de la détection de changement de voie. Cette indication est utile dans pour les systèmes de navigation des véhicules. La détection de changement de voie a été réalisée par une somme cumulative et une technique d’ajustement de courbe et obtient de très bon taux de réussite. Des perspectives de recherche sur la stratégie de détection sont proposées pour déterminer la voie initiale du véhicule. En conclusion, les résultats obtenus lors de ces travaux montrent l’intérêt de l’utilisation de la trajectoire provenant de l’odométrie visuelle comme source d’information pour la fusion de données à faible coût pour la localisation des véhicules. Cette source d’information provenant de la caméra est complémentaire aux données d’images traitées qui pourront par ailleurs être utilisées pour les différentes taches visée par les systèmes d’aides à la conduite
With the growing research on Advanced Driver Assistance Systems (ADAS) for Intelligent Transport Systems (ITS), accurate vehicle localization plays an important role in intelligent vehicles. The Global Positioning System (GPS) has been widely used but its accuracy deteriorates and susceptible to positioning error due to factors such as the restricting environments that results in signal weakening. This problem can be addressed by integrating the GPS data with additional information from other sensors. Meanwhile, nowadays, we can find vehicles equipped with sensors for ADAS applications. In this research, fusion of GPS with visual odometry (VO) and digital map is proposed as a solution to localization improvement with low-cost data fusion. From the published works on VO, it is interesting to know how the generated trajectory can further improve vehicle localization. By integrating the VO output with GPS and OpenStreetMap (OSM) data, estimates of vehicle position on the map can be obtained. The lateral positioning error is reduced by utilizing lane distribution information provided by OSM while the longitudinal positioning is optimized with curve matching between VO trajectory trail and segmented roads. To observe the system robustness, the method was validated with KITTI datasets tested with different common GPS noise. Several published VO methods were also used to compare improvement level after data fusion. Validation results show that the positioning accuracy achieved significant improvement especially for the longitudinal error with curve matching technique. The localization performance is on par with Simultaneous Localization and Mapping (SLAM) SLAM techniques despite the drift in VO trajectory input. The research on employability of VO trajectory is extended for a deterministic task in lane-change detection. This is to assist the routing service for lane-level direction in navigation. The lane-change detection was conducted by CUSUM and curve fitting technique that resulted in 100% successful detection for stereo VO. Further study for the detection strategy is however required to obtain the current true lane of the vehicle for lane-level accurate localization. With the results obtained from the proposed low-cost data fusion for localization, we see a bright prospect of utilizing VO trajectory with information from OSM to improve the performance. In addition to obtain VO trajectory, the camera mounted on the vehicle can also be used for other image processing applications to complement the system. This research will continue to develop with future works concluded in the last chapter of this thesis

O problema da localização consiste em estimar a posição de um robô com relação a algum referencial externo e é parte essencial de sistemas de navegação de robôs e veículos autônomos. A localização baseada em odometria visual destaca-se em relação a odometria de encoders na obtenção da rotação e direção do movimento do robô. Esse tipo de abordagem é também uma escolha atrativa para sistemas de controle de veículos autônomos em ambientes urbanos, onde a informação visual é necessária para a extração de informações semânticas de placas, semáforos e outras sinalizações. Neste contexto este trabalho propõe o desenvolvimento de um sistema de odometria visual utilizando informação visual de uma câmera monocular baseado em reconstrução 3D para estimar o posicionamento do veículo. O problema da escala absoluta, inerente ao uso de câmeras monoculares, é resolvido utilizando um conhecimento prévio da relação métrica entre os pontos da imagem e pontos do mundo em um mesmo plano.
The localization problem consists of estimating the position of the robot with regards to some external reference and it is an essential part of robots and autonomous vehicles navigation systems. Localization based on visual odometry, compared to encoder based odometry, stands out at the estimation of rotation and direction of the movement. This kind of approach is an interesting choice for vehicle control systems in urban environment, where the visual information is mandatory for the extraction of semantic information contained in the street signs and marks. In this context this project propose the development of a visual odometry system based on structure from motion using visual information acquired from a monocular camera to estimate the vehicle pose. The absolute scale problem, inherent with the use of monocular cameras, is achieved using som previous known information regarding the metric relation between image points and points lying on a same world plane.

Planetary exploration rovers should be capable of operating autonomously also for long paths with minimal human input. Control operations must be minimized in order to reduce traverse time, optimize the resources allocated for telecommunications and maximize the scientific output of the mission. Knowing the goal position and considering the vehicle dynamics, control algorithms have to provide the appropriate inputs to actuators. Path planning algorithms use three-dimensional models of the surrounding terrain in order to safely avoid obstacles. Moreover, rovers, for the sample and return missions planned for the next years, have to demonstrate the capability to return to a previously visited place for sampling scientific data or to return a sample to an ascent vehicle. Motion measurement is a fundamental task in rover control, and planetary environment presents some specific issues. Wheel odometry has wide uncertainty due to slippage of wheels on a sandy surface, inertial measurement has drift problems and GPS-like positioning systems is not available on extraterrestrial planets. Vision systems have demonstrated to be reliable and accurate motion tracking measurement methods. One of these methods is stereo Visual Odometry. Stereo-processing allows estimation of the three-dimensional location of landmarks observed by a pair of cameras by means of triangulation. Point cloud matching between two subsequent frames allows stereo-camera motion computation. Thanks to Visual SLAM (Simultaneous Localization and Mapping) techniques a rover is able to reconstruct a consistent map of the environment and to localize itself with reference to this map. SLAM technique presents two main advantages: the map of the environment construction and a more accurate motion tracking, thanks to the solutions of a large minimization problem which involves multiple camera poses and measurements of map landmarks. After rover touchdown, one of the key tasks requested to the operations center is the accurate measurement of the rover position on the inertial and fixed coordinate systems, such as the J2000 frame and the Mars Body-Fixed (MBF) frame. For engineering and science operations, high precision global localization and detailed Digital Elevation Models (DEM) of the landing site are crucial. The first part of this dissertation treats the problem of localizing a rover with respect to a satellite geo-referenced and ortho-rectified images, and the localization with respect to a digital elevation model (DEM) realized starting from satellite images A sensitivity analysis of the Visual Position Estimator for Rover (VIPER) algorithm outputs is presented. By comparing the local skyline, extracted form a panoramic image, and a skyline rendered from a Digital Elevation Model (DEM), the algorithm retrieve the camera position and orientation relative to the DEM map. This algorithm has been proposed as part of the localization procedure realized by the Rover Operation Control Center (ROCC), located in ALTEC, to localize ExoMars 2020 rover after landing and as initialization and verification of rover guidance and navigation outputs. Images from Mars Exploration Rover mission and HiRISE DEM have been used to test the algorithm performances. During rover traverse, Visual Odometry methods could be used as an asset to refine the path estimation. The second part of this dissertation treats an experimental analysis of how landmark distributions in a scene, as observed by a stereo-camera, affect Visual Odometry measurement performances. Translational and rotational tests have been performed in many different positions in an indoor environment. The Visual Odometry algorithm, which has been implemented, firstly guesses motion by a linear 3D-to-3D method embedded within a RANdom SAmple Consensus (RANSAC) process to remove outliers. Then, motion estimation is computed from the inliers by minimizing the Euclidean distance between the triangulated landmarks. The last part of this dissertation has been developed in collaboration with NASA Jet Propulsion Laboratory and presents an innovative visual localization method for hopping and tumbling platforms. These new mobility systems for the exploration of comets, asteroids, and other small Solar System bodies, require new approaches for localization. The choice of a monocular onboard camera for perception is constrained by the rover’s limited weight and size. Visual localization near the surface of small bodies is difficult due to large scale changes, frequent occlusions, high-contrast, rapidly changing shadows and relatively featureless terrains. A synergistic localization and mapping approach between the mother spacecraft and the deployed hopping/tumbling daughter-craft rover has been studied and developed. We have evaluated various open-source visual SLAM algorithms. Between them, ORB-SLAM2 has been chosen and adapted for this application. The possibility to save the map made by orbiter observations and re-load it for rover localization has been introduced. Moreover, now it is possible to fuse the map with other orbiter sensor pose measurement. Collaborative localization method accuracy has been estimated. A series of realistic images of an asteroid mockup have been captured and a Vicon system has been used in order to give the trajectory ground truth. In addition, we had evaluated this method robustness to illumination changes.
I rover marziani e, più in generale, i robot per l’esplorazione di asteroidi e piccoli corpi celesti, richiedono un alto livello di autonomia. Il controllo da parte di un operatore deve essere ridotto al minimo, al fine di ridurre i tempi di percorrenza, ottimizzare le risorse allocate per le tele-comunicazioni e massimizzare l’output scientifico della missione. Conoscendo la posizione obiettivo e considerando la dinamica del veicolo, gli algoritmi di controllo forniscono gli input adeguati agli attuatori. Algoritmi di pianificazione della traiettoria, sfruttando modelli tridimensionali del terreno circostante, evitano gli ostacoli con ampi margini di sicurezza. Inoltre i rover per le missioni di sample and return, previste per i prossimi anni, devono dimostrare la capacità di tornare in un luogo già visitato per il campionamento di dati scientifici o per riportare i campioni raccolti ad un veicolo di risalita. In tutte queste task la stima del moto risulta essere fondamentale. La stima del moto su altri pianeti ha la sua peculiarità. L’odometria tramite encoder, infatti, presenta elevate incertezze a causa dello slittamento delle ruote su superfici sabbiose o scivolose; i sistemi di navigazione inerziale, nel caso della dinamica lenta dei rover, presentano derive non tollerabili per una stima accurata dell’assetto; infine non sono disponibili sistemi di posizionamento globale analoghi al GPS. Sistemi della stima del moto basati su telecamere hanno dimostrato, già con le missioni MER della NASA, di essere affidabili e accurati. Uno di questi sistemi è l’odometria visuale stereo. In questo algoritmo il moto è stimato calcolando la roto-traslazione di due nuvole di punti misurate a due istanti successivi. La nuvola di punti è generata tramite triangolazione di punti salienti presenti nelle due immagini. Grazie a tecniche di Simultaneous Localization and Mapping (SLAM) si dà la capacità ad un rover di costruire una mappa dell’ambiente circostante e di localizzarsi rispetto ad essa. Le tecniche di SLAM presentano due vantaggi: la costruzione della mappa e una stima della traiettoria più accurata, grazie alla soluzione di problemi di minimizzazione che coinvolgono la stima di più posizioni e landmark allo stesso tempo. Subito dopo l’atterraggio, una delle task principali che devono essere svolte dal centro operativo per il controllo di rover è il calcolo accurato della posizione del lander/rover rispetto al sisma di riferimento inerziale e il sistema di riferimento solidale al pianeta, come il sistema J2000 e il Mars Body-Fixed (MBF) frame. Sia per le operazioni scientifiche che ingegneristiche risulta fondamentale la localizzazione accurata rispetto a immagini satellitari e a modelli tridimensionali della zona di atterraggio. Nella prima parte della tesi viene trattato il problema della localizzazione di un rover rispetto ad un’immagine satellitare geo referenziata e orto rettificata e la localizzazione rispetto ad un modello di elevazione digitale (DEM), realizzato da immagini satellitari. È stata svolta l’analisi di una versione modificata dell’algoritmo Visual Position Estimator for Rover (VIPER). L’algoritmo trova la posizione e l’assetto di un rover rispetto ad un DEM, comparando la linea d’orizzonte locale con le linee d’orizzonte calcolate in posizioni a priori del DEM. Queste analisi sono state svolte in collaborazione con ALTEC S.p.A., con lo scopo di definire le operazioni che il Rover Operation Control Center (ROCC) dovrà svolgere per la localizzazione del rover ExoMars 2020. Una volta effettuate le operazioni di localizzazione, questi metodi possono essere nuovamente utilizzati come verifica e correzione della stima della traiettoria. Nella seconda parte della dissertazione è presentato un metodo di odometria visuale stereo per rover ed un’analisi di come la distribuzione dei landmark triangolati influisca sulla stima del moto. A questo scopo sono stati svolti dei test in laboratorio, variando la distanza della scena. L’algoritmo di odometria visiva implementato è un metodo 3D-to-3D con rimozione dei falsi positivi tramite procedura di RANdom SAmple Consensus. La stima del moto è effettuata minimizzando la distanza euclidea tra le due nuvole di punti. L’ultima parte di questa dissertazione è stata sviluppata in collaborazione con il Jet Propulsion Laboratory (NASA) e presenta un sistema di localizzazione per rover hopping/tumbling per l’esplorazione di comete e asteroidi. Tali sistemi innovativi richiedono nuovi approcci per la localizzazione. Viste le risorse limitate di spazio, peso e energia disponibile e le limitate capacità computazionali, si è scelto di basare il sistema di localizzazione su una monocamera. La localizzazione visuale in prossimità di una cometa, inoltre, presenta alcune peculiarità che la rendono più difficoltosa. Questo a causa dei grandi cambiamenti di scala che si presentano durante il movimento della piattaforma, le frequenti occlusioni del campo di vista, la presenza di ombre nette che cambiano con il periodo di rotazione dell’asteroide e la caratteristica visiva del terreno, che risulta essere omogeno nel campo del visibile. È stato proposto un sistema di visual SLAM collaborativo tra il rover tumbling/hopping e il satellite “madre”, che ha portato il rover nell’orbita di rilascio. È stato effettuato lo stato dell’arte dei più recenti algoritmi di visual SLAM open-source e, dopo un’accurata analisi, si è optato per l’utilizzo di ORB-SLAM2, che è stato modificato per far fronte al tipo di applicazione richiesta. È stata introdotta la possibilità di salvare la mappa realizzata dall’orbiter, che viene utilizzata dal rover per la sua localizzazione. È possibile, inoltre, fondere la mappa realizzata da orbiter con altre misure d’assetto provenienti da altri sensori a bordo dell’orbiter. L’accuratezza di tale metodo è stata valutata utilizzando una sequenza di immagini raccolta in ambiente rappresentativo e utilizzando un sistema di riferimento esterno. Sono state effettuate simulazioni della fase di mappatura dell’asteroide e localizzazione della piattaforma hopping/tumbling e, infine, è stato valutato come migliorare le performances di questo metodo, in seguito al cambiamento delle condizioni di illuminazione.

This is a thesis on outdoor monocular visual SLAM in natural environments. The techniques proposed herein aim at estimating camera pose and 3D geometrical structure of the surrounding environment. This problem statement was motivated by the GPS-denied scenario for a sea-surface vehicle developed at Plymouth University named Springer. The algorithms proposed in this thesis are mainly adapted for the Springer’s environmental conditions, so that the vehicle can navigate on a vision based localization system when GPS is not available; such environments include estuarine areas, forests and the occasional semi-urban territories. The research objectives are constrained versions of the ever-abiding problems in the fields of multiple view geometry and mobile robotics. The research is proposing new techniques or improving existing ones for problems such as scene reconstruction, relative camera pose recovery and filtering, always in the context of the aforementioned landscapes (i.e., rivers, forests, etc.). Although visual tracking is paramount for the generation of data point correspondences, this thesis focuses primarily on the geometric aspect of the problem as well as with the probabilistic framework in which the optimization of pose and structure estimates takes place. Besides algorithms, the deliverables of this research should include the respective implementations and test data for these algorithms in the form of a software library and a dataset containing footage of estuarine regions taken from a boat, along with synchronized sensor logs. This thesis is not the final analysis on vision based navigation. It merely proposes various solutions for the localization problem of a vehicle navigating in natural environments either on land or on the surface of the water. Although these solutions can be used to provide position and orientation estimates when GPS is not available, they have limitations and there is still a vast new world of ideas to be explored.

PONTIFÍCIA UNIVERSIDADE CATÓLICA DO RIO DE JANEIRO
Alvo de intenso estudo da visão computacional, a reconstrução densa 3D teve um importante marco com os primeiros sistemas em tempo real a alcançarem precisão milimétrica com uso de câmeras RGBD e GPUs. Entretanto estes métodos não são aplicáveis a dispositivos de menor poder computacional. Tendo a limitação de recursos computacionais como requisito, o objetivo deste trabalho é apresentar um método de odometria visual utilizando câmeras comuns e sem a necessidade de GPU, baseado em técnicas de Structure from Motion (SFM) com features esparsos, utilizando as informações de uma reconstrução densa. A Odometria visual é o processo de estimar a orientação e posição de um agente (um robô, por exemplo), a partir das imagens. Esta dissertação fornece uma comparação entre a precisão da odometria calculada pelo método proposto e pela reconstrução densa utilizando o Kinect Fusion. O resultado desta pesquisa é diretamente aplicável na área de realidade aumentada, tanto pelas informações da odometria que podem ser usadas para definir a posição de uma câmera, como pela reconstrução densa, que pode tratar aspectos como oclusão dos objetos virtuais com reais.
Aim of intense research in the field computational vision, dense 3D reconstruction achieves an important landmark with first methods running in real time with millimetric precision, using RGBD cameras and GPUs. However these methods are not suitable for low computational resources. Having low computational resources as requirement, the goal of this work is to show a method of visual odometry using regular cameras, without using a GPU. The proposed method is based on technics of sparse Structure From Motion (SFM), using data provided by dense 3D reconstruction. Visual odometry is the process of estimating the position and orientation of an agent (a robot, for instance), based on images. This dissertation compares the proposed method with the odometry calculated by Kinect Fusion. Results of this research are applicable in augmented reality. Odometry provided by this work can be used to model a camera and the data from dense 3D reconstruction, can be used to handle occlusion between virtual and real objects.

Thesis: S.M., Massachusetts Institute of Technology, Department of Aeronautics and Astronautics, 2014.
Cataloged from PDF version of thesis.
Includes bibliographical references (pages 93-97).
The goal of visual inertial odometry (VIO) is to estimate a moving vehicle's trajectory using inertial measurements and observations, obtained by a camera, of naturally occurring point features. One existing VIO estimation algorithm for use with a monocular system, is the multi-state constraint Kalman filter (MSCKF), proposed by Mourikis and Li [34, 29]. The way the MSCKF uses feature measurements drastically improves its performance, in terms of consistency, observability, computational complexity and accuracy, compared to other VIO algorithms [29]. For this reason, the MSCKF is chosen as the basis for the estimation algorithm presented in this thesis. A VIO estimation algorithm for a system consisting of an IMU, a monocular camera and a depth sensor is presented in this thesis. The addition of the depth sensor to the monocular camera system produces three-dimensional feature locations rather than two-dimensional locations. Therefore, the MSCKF algorithm is extended to use the extra information. This is accomplished using a model proposed by Dryanovski et al. that estimates the 3D location and uncertainty of each feature observation by approximating it as a multivariate Gaussian distribution [11]. The extended MSCKF algorithm is presented and its performance is compared to the original MSCKF algorithm using real-world data obtained by flying a custom-built quadrotor in an indoor office environment.
by Marissa N. Galfond.
S.M.

Cette thèse de doctorat aborde les défis de la fusion de capteurs et de la localisation et de la cartographie simultanées (SLAM) pour les systèmes autonomes, en se concentrant spécifiquement sur les véhicules terrestres autonomes (AGV) et les micro-véhicules aériens (MAV) naviguant dans des environnements dynamiques et à grande échelle. La thèse présente une gamme de solutions innovantes pour améliorer la performance et la fiabilité des systèmes SLAM à travers cinq chapitres méthodologiques.Le chapitre d'introduction établit la motivation de la recherche, en soulignant les défis et les limitations de l'odométrie visuelle utilisant des caméras hétérogènes. Il décrit également la structure de la thèse et fournit une analyse approfondie de la littérature pertinente. Le deuxième chapitre présente IBISCape, une référence simulée pour valider les systèmes SLAM haute fidélité basés sur le simulateur CARLA. Le troisième chapitre présente une nouvelle méthode basée sur l'optimisation pour calibrer une configuration visuelle-inertielle RGB-D-IMU, validée par des expériences approfondies sur des séquences réelles et simulées. Le quatrième chapitre propose une approche d'estimation d'état optimale linéaire pour les MAV afin d'obtenir une localisation de haute précision avec un retard minimal du système.Le cinquième chapitre présente le système DH-PTAM pour un suivi et une cartographie parallèles robustes dans des environnements dynamiques utilisant des images stéréo et des flux d'événements. Le sixième chapitre explore de nouvelles frontières dans le domaine du SLAM dense à l'aide de caméras Event, présentant une nouvelle approche de bout en bout pour les événements hybrides et le système SLAM dense à nuages de points. Le septième et dernier chapitre résume les contributions et les principaux résultats de la thèse, en mettant l'accent sur les progrès réalisés dans la fusion de capteurs hétérogènes multimodaux pour les systèmes autonomes naviguant dans des environnements dynamiques et à grande échelle. Les travaux futurs comprennent l'étude du potentiel d'intégration de capteurs de navigation inertielle et l'exploration de composants supplémentaires d'apprentissage en profondeur pour améliorer la robustesse et la précision de la fermeture de boucle
This Ph.D. thesis addresses the challenges of sensor fusion and Simultaneous Localization And Mapping (SLAM) for autonomous systems, specifically focusing on Autonomous Ground Vehicles (AGVs) and Micro Aerial Vehicles (MAVs) navigating large-scale and dynamic environments. The thesis presents a range of innovative solutions to enhance the performance and reliability of SLAM systems through five methodological chapters.The introductory chapter establishes the research motivation, highlighting the challenges and limitations of visual odometry using heterogeneous cameras. It also outlines the thesis structure and extensively reviews relevant literature. The second chapter introduces IBISCape, a simulated benchmark for validating high-fidelity SLAM systems based on the CARLA simulator. The third chapter presents a novel optimization-based method for calibrating an RGB-D-IMU visual-inertial setup, validated through extensive experiments on real-world and simulated sequences. The fourth chapter proposes a linear optimal state estimation approach for MAVs to achieve high-accuracy localization with minimal system delay.The fifth chapter introduces the DH-PTAM system for robust parallel tracking and mapping in dynamic environments using stereo images and event streams. The sixth chapter explores new frontiers in the field of dense SLAM using Event cameras, presenting a novel end-to-end approach for hybrid events and point clouds dense SLAM system. The seventh and final chapter summarizes the thesis's contributions and main findings, emphasizing the advancements made in multi-modal heterogeneous sensor fusion for autonomous systems navigating large-scale and dynamic environments. Future work includes investigating the potential of integrating inertial navigation sensors and exploring additional deep-learning components for improving loop-closure robustness and accuracy

Made available in DSpace on 2014-12-17T14:55:51Z (GMT). No. of bitstreams: 1 BrunoMFS_DISSERT.pdf: 2462891 bytes, checksum: b8ea846d0fcc23b0777a6002e9ba92ac (MD5) Previous issue date: 2011-02-15
Coordena??o de Aperfei?oamento de Pessoal de N?vel Superior
Visual Odometry is the process that estimates camera position and orientation based solely on images and in features (projections of visual landmarks present in the scene) extraced from them. With the increasing advance of Computer Vision algorithms and computer processing power, the subarea known as Structure from Motion (SFM) started to supply mathematical tools composing localization systems for robotics and Augmented Reality applications, in contrast with its initial purpose of being used in inherently offline solutions aiming 3D reconstruction and image based modelling. In that way, this work proposes a pipeline to obtain relative position featuring a previously calibrated camera as positional sensor and based entirely on models and algorithms from SFM. Techniques usually applied in camera localization systems such as Kalman filters and particle filters are not used, making unnecessary additional information like probabilistic models for camera state transition. Experiments assessing both 3D reconstruction quality and camera position estimated by the system were performed, in which image sequences captured in reallistic scenarios were processed and compared to localization data gathered from a mobile robotic platform
Odometria Visual ? o processo pelo qual consegue-se obter a posi??o e orienta??o de uma c?mera, baseado somente em imagens e consequentemente, em caracter?sticas (proje??es de marcos visuais da cena) nelas contidas. Com o avan?o nos algoritmos e no poder de processamento dos computadores, a sub?rea de Vis?o Computacional denominada de Structure from Motion (SFM) passou a fornecer ferramentas que comp?em sistemas de localiza??o visando aplica??es como rob?tica e Realidade Aumentada, em contraste com o seu prop?sito inicial de ser usada em aplica??es predominantemente offline como reconstru??o 3D e modelagem baseada em imagens. Sendo assim, este trabalho prop?e um pipeline de obten??o de posi??o relativa que tem como caracter?sticas fazer uso de uma ?nica c?mera calibrada como sensor posicional e ser baseado interamente nos modelos e algoritmos de SFM. T?cnicas usualmente presentes em sistemas de localiza??o de c?mera como filtros de Kalman e filtros de part?culas n?o s?o empregadas, dispensando que informa??es adicionais como um modelo probabil?stico de transi??o de estados para a c?mera sejam necess?rias. Experimentos foram realizados com o prop?sito de avaliar tanto a reconstru??o 3D quanto a posi??o de c?mera retornada pelo sistema, atrav?s de sequ?ncias de imagens capturadas em ambientes reais de opera??o e compara??es com um ground truth fornecido pelos dados do od?metro de uma plataforma rob?tica

GPS/IMU integrated systems are commonly used for vehicle navigation. The algorithm for this coupled system is normally based on Kalman filter. However, oscillated GPS measurements in the urban environment can lead to localization divergence easily. Moreover, heading estimation may be sensitive to magnetic interference if it relies on IMU with integrated magnetometer. This report tries to solve the localization problem on GPS oscillation and outage, based on adaptive extended Kalman filter(AEKF). In terms of the heading estimation, stereo visual odometry(VO) is fused to overcome the effect by magnetic disturbance. Vision-aided AEKF based algorithm is tested in the cases of both good GPS condition and GPS oscillation with magnetic interference. Under the situations considered, the algorithm is verified to outperform conventional extended Kalman filter(CEKF) and unscented Kalman filter(UKF) in position estimation by 53.74% and 40.09% respectively, and decrease the drifting of heading estimation.
GPS/IMU integrerade system används ofta för navigering av fordon. Algoritmen för detta kopplade system är normalt baserat på ett Kalmanfilter. Ett problem med systemet är att oscillerade GPS mätningar i stadsmiljöer enkelt kan leda till en lokaliseringsdivergens. Dessutom kan riktningsuppskattningen vara känslig för magnetiska störningar om den är beroende av en IMU med integrerad magnetometer. Rapporten försöker lösa lokaliseringsproblemet som skapas av GPS-oscillationer och avbrott med hjälp av ett adaptivt förlängt Kalmanfilter (AEKF). När det gäller riktningsuppskattningen används stereovisuell odometri (VO) för att försvaga effekten av magnetiska störningar genom sensorfusion. En Visionsstödd AEKF-baserad algoritm testas i fall med både goda GPS omständigheter och med oscillationer i GPS mätningar med magnetiska störningar. Under de fallen som är aktuella är algoritmen verifierad för att överträffa det konventionella utökade Kalmanfilteret (CEKF) och ”Unscented Kalman filter” (UKF) när det kommer till positionsuppskattning med 53,74% respektive 40,09% samt minska fel i riktningsuppskattningen.

Mestrado em Engenharia Mecânica
O principal objetivo deste trabalho é o desenvolvimento de uma solução de hodometria para veículos com direção Ackermann. A solução tinha que ser portátil, exível e fácil de montar. Após o estudo do estado da arte e uma pesquisa de soluções, a solução escolhida foi baseada em hodometria visual. Os passos seguintes do trabalho foram estudar a viabilidade de utilizar câmaras lineares para hodometria visual. O sensor de imagem foi usado para calcular a velocidade longitudinal; e a orientação da movimento foi calculado usando dois giroscópios. Para testar o método, várias experiências foram feitas; as experiências ocorreram indoor, sob condições controladas. Foi testada a capacidade de medir a velocidade em movimentos de linha reta, movimentos diagonais, movimentos circulares e movimentos com variação da distância ao solo. Os dados foram processados usando algoritmos de correlação e os foram resultados documentados. Com base nos resultados, é seguro concluir que hodometria com câmaras lineares auxiliado por sensores inerciais tem um potencial de aplicabilidade no mundo real.
The main objective of this work is to develop a solution of odometry for vehicles with Ackermann steering. The solution had to be portable, exible and easy to mount. After the study of the state of the art and a survey of solutions, the solution chosen was based on visual odometry. The following steps of the work were to study the feasibility to use line scan image sensors for visual odometry. The image sensor was used to compute the longitudinal velocity; and the orientation of motion was computed using two gyroscopes. To test the method, several experiments were made; the experiments took place indoor, under controlled conditions. It was tested the ability to measure velocity on straight line movements, diagonal movements, circular movements and movements with a changing distance from the ground. The data was processed with correlation algorithms and the results were documented. Based on the results it is safe to conclude that odometry with line scan sensors aided by inertial sensors has a potential for a real world applicability.

This thesis examines two methods for monocular visual odometry, FAST + KLT and ORBSLAM2, in the case of underwater environments.This is done by implementing and testing the methods on different underwater datasets. The results for the FAST + KLT provide no evidence that this method is effective in underwater settings. However, results for the ORBSLAM2 indicate that good performance is possible whenproperly tuned and provided with good camera calibration. Still, thereremain challenges related to, for example, sand bottom environments and scale estimation in monocular setups. The conclusion is therefore that the ORBSLAM2 is the most promising method of the two tested for underwater monocular visual odometry.
Denna uppsats undersöker två metoder för monokulär visuell odometri, FAST + KLT och ORBSLAM2, i det särskilda fallet av miljöer under vatten. Detta görs genom att implementera och testa metoderna på olika undervattensdataset. Resultaten för FAST + KLT ger inget stöd för att metoden skulle vara effektiv i undervattensmiljöer. Resultaten för ORBSLAM2, däremot, indikerar att denna metod kan prestera bra om den justeras på rätt sätt och får bra kamerakalibrering. Samtidigt återstår dock utmaningar relaterade till exempelvis miljöer med sandbottnar och uppskattning av skala i monokulära setups. Slutsatsen är därför att ORBSLAM2 är den mest lovande metoden av de två testade för monokulär visuell odometri under vatten.

Visual Odometry (VO) is a localisation technology that has many potential benefits for the mining industry. However, the performance of VO degrades in challenging lighting conditions such as underground mines. This study evaluated the effect of key illumination strategies (the spatial position, spatial distribution and directional properties of light) on the performance of VO using real-world evaluation data. The results showed that the illumination strategies affected the performance of VO. The most significant impact (an improvement in accuracy and reliability of 71% and 70% respectively) was observed by increasing the distance between the light sources and the camera.

A classic Computer Vision task is the estimation of a 3D map from a collection of images. This thesis explores the online simultaneous estimation of camera poses and map points, often called Visual Simultaneous Localisation and Mapping [VSLAM]. In the near future the use of visual information by autonomous cars is likely, since driving is a vision dominated process. For example, VSLAM could be used to estimate the position of the car in relation to objects of interest, such as the road, other cars and pedestrians. Aimed at the creation of a real-time, robust, loop closing, single camera SLAM system, the properties of several state-of-the-art VSLAM systems and related techniques are studied. The system goals cover several important, if difficult, problems, which makes a solution widely applicable. This thesis makes two contributions: A rigorous qualitative analysis of VSLAM methods and a system designed accordingly. A novel tracking by matching scheme is proposed, which, unlike the trackers used by many similar systems, is able to deal better with forward camera motion. The system estimates general motion with loop closure in real time. The system is compared to a state-of-the-art monocular VSLAM algorithm and found to be similar in speed and performance.

To remove the dependency on external structure for drone positioning in GPS-denied environments, it is desirable to estimate the ego-motion of drones on-board. Visual positioning systems have been studied for quite some time and the literature on the area is diligent. The aim of this project is to investigate the currently available methods and implement a visual odometry system for drones which is capable of giving continuous estimates with a lightweight solution. In that manner, the state of the art systems are investigated and a visual odometry system is implemented based on the design decisions. The resulting system is shown to give acceptable estimates.
För att avlägsna behovet av extern infrastruktur så som GPS, som dessutominte är tillgänglig i många miljöer, är det önskvärt att uppskatta en drönares rörelse med sensor ombord. Visuella positioneringssystem har studerats under lång tid och litteraturen på området är ymnig. Syftet med detta projekt är att undersöka de för närvarande tillgängliga metodernaoch designa ett visuellt baserat positioneringssystem för drönare. Det resulterande systemet utvärderas och visas ge acceptabla positionsuppskattningar.

The three-dimensional mapping environments has been intensively studied in the last decade. Among the benefits of this research topic is possible to highlight the addition of autonomy for car or even drones. The three-dimensional representation also allows viewing of a given scenario iteratively and with greater detail. However, until the time of this work was not found one framework to present in detail the implementation of algorithms to perform 3D mapping outdoor approaching a real-time processing. In view of this, in this work we developed a framework with the main stages of three-dimensional reconstruction. Therefore, stereoscopy was chosen as a technique for acquiring the depth information of the scene. In addition, this study evaluated four algorithms depth map generation, where it was possible to achieve the rate of 9 frames per second.
O mapeamento tridimensional de ambientes tem sido intensivamente estudado na última década. Entre os benefícios deste tema de pesquisa é possível destacar adição de autonomia á automóveis ou mesmo drones. A representação tridimensional também permite a visualização de um dado cenário de modo iterativo e com maior riqueza de detalhes. No entanto, até o momento da elaboração deste trabalho não foi encontrado um framework que apresente em detalhes a implementação de algoritmos para realização do mapeamento 3D de ambientes externos que se aproximasse de um processamento em tempo real. Diante disto, neste trabalho foi desenvolvido um framework com as principais etapas de reconstrução tridimensional. Para tanto, a estereoscopia foi escolhida como técnica para a aquisição da informação de profundidade do cenário. Além disto, neste trabalho foram avaliados 4 algoritmos de geração do mapa de profundidade, onde foi possível atingir a taxa de 9 quadros por segundo.

This thesis deals with algorithms of visual odometry and its application on the experimental vehicle Car4. The first part contains different researches in this area on which the solution process is based. Next chapters introduce theoretical design and ideas of monocular and stereo visual odometry algorithms. The third part deals with the implementation in the software MATLAB with the use of Image processing toolbox. After tests done and based on real data, the chosen algorithm is applied to the vehicle Car4 used in practical conditions of interior and exterior. The last part summarizes the results of the work and address the problems which are asociated with the application of visual obmetry algorithms.

In this dissertation, we have focused on conflicts that occur due to disagreeing motions in multi-modal localization algorithms. In spite of the recent achievements in robust localization by means of multi-sensor fusion, these algorithms are not applicable to all environments. This is primarily attributed to the following fundamental assumptions: (i) the environment is predominantly stationary, (ii) only ego-motion of the sensor platform exists, and (iii) multiple sensors are always in agreement with each other regarding the observed motion. Recently, studies have shown how to relax the static environment assumption using outlier rejection techniques and dynamic object segmentation. Additionally, to handle non ego-motion, approaches that extend the localization algorithm to multi-body tracking have been studied. However, there has been no attention given to the conditions where multiple sensors contradict each other with regard to the motions observed. Vision based localization has become an attractive approach for both indoor and outdoor applications due to the large information bandwidth provided by images and reduced cost of the cameras used. In order to improve the robustness and overcome the limitations of vision, an Inertial Measurement Unit (IMU) may be used. Even though visual-inertial localization has better accuracy and improved robustness due to the complementary nature of camera and IMU sensor, they are affected by disagreements in motion observations. We term such dynamic situations as environments with motion conflictbecause these are caused when multiple different but self- consistent motions are observed by different sensors. Tightly coupled visual inertial fusion approaches that disregard such challenging situations exhibit drift that can lead to catastrophic errors. We have provided a probabilistic model for motion conflict. Additionally, a novel algorithm to detect and resolve motion conflicts is also presented. Our method to detect motion conflicts is based on per-frame positional estimate discrepancy and per- landmark reprojection errors. Motion conflicts were resolved by eliminating inconsistent IMU and landmark measurements. Finally, a Motion Conflict aware Visual Inertial Odometry (MC- VIO) algorithm that combined both detection and resolution of motion conflict was implemented. Both quantitative and qualitative evaluation of MC-VIO on visually and inertially challenging datasets were obtained. Experimental results indicated that MC-VIO algorithm reduced the absolute trajectory error by 70% and the relative pose error by 34% in scenes with motion conflict, in comparison to the reference VIO algorithm. Motion conflict detection and resolution enables the application of visual inertial localization algorithms to real dynamic environments. This paves the way for articulate object tracking in robotics. It may also find numerous applications in active long term augmented reality.

Tato diplomová práce se zabývá tvorbou fúze pozičních dat z existující realtimové im- plementace vizuálního SLAMu a kolové odometrie. Výsledkem spojení dat je potlačení nežádoucích chyb u každé ze zmíněných metod měření, díky čemuž je možné vytvořit přesnější 3D model zkoumaného prostředí. Práce nejprve uvádí teorií potřebnou pro zvládnutí problematiky 3D SLAMu. Dále popisuje vlastnosti použitého open source SLAM projektu a jeho jednotlivé softwarové úpravy. Následně popisuje principy spo- jení pozičních informací získaných vizuálními a odometrickými snímači, dále uvádí popis diferenciálního podvozku, který byl použit pro tvorbu kolové odometrie. Na závěr práce shrnuje výsledky dosažené datovou fúzí a srovnává je s původní přesností vizuálního SLAMu.

In pursuit of creating a fully automated navigation system that is capable of operating in dynamic environments, a large amount of research is being devoted to systems that use visual odometry assisted methods to estimate the position of a platform with regards to the environment surrounding it. This includes systems that do and do not know the environment a priori, as both rely on the same methods for localisation. For the combined problem of localisation and mapping, Simultaneous Localisation and Mapping (SLAM) is the de facto choice, and in recent years with the advent of color and depth (RGB-D) sensors, RGB-D SLAM has become a hot topic for research. Most research being performed is on improving the overall system accuracy or more specifically the performance with regards to the overall trajectory error. While this approach quantifies the performance of the system as a whole, the individual frame-to-frame performance is often not mentioned or explored properly. While this will directly tie in to the overall performance, the level of scene cohesion experienced between two successive observations can vary greatly over a single dataset of observations. The focus of this dissertation will be the relevant levels of translational and rotational velocities experienced by the sensor between two successive observations and the effect on the final accuracy of the SLAM implementation. The frame rate will specifically be used to alter and evaluate the different spatial velocities experienced over multiple datasets of RGB-D data. Two systems were developed to illustrate and evaluate the potential of various approaches to RGB-D SLAM. The first system is a real-world implementation where SLAM is used to localise and map the environment surrounding a quadcopter platform. A Microsoft Kinect is directly mounted to the quadcopter and is used to provide a RGB-D datastream to a remote processing terminal. This terminal runs a SLAM implementation that can alternate between different visual odometry methods. The remote terminal acts as the position controller for the quadcopter, replacing the need for a direct human operator. A semi-automated system is implemented, that allows a human operator to designate waypoints within the environment that the quadcopter moves to. The second system uses a series of publicly available RGB-D datasets with their accompanying ground-truth readings to simulate a real RGB-D datasteam. This is used to evaluate the performance of the various RGB-D SLAM approaches to visual odometry. For each of the datasets, the accompanying translational and angular velocity on a frame-to-frame basis can be calculated. This can, in turn, be used to evaluate the frame-to-frame accuracy of the SLAM implementation, where the spatial velocity can be manually altered by occluding frames within the sequence. Thus, an accurate relationship can be calculated between the frame rate, the spatial velocity and the performance of the SLAM implementation. Three image processing techniques were used to implement the visual odometry for RGB-D SLAM. SIFT, SURF and ORB were compared across eight of the TUM database datasets. SIFT had the best performance, with a 30% increase over SURF and doubling the performance of ORB. By implementing SIFT using CUDA, the feature detection and description process only takes 18ms, negating the disadvantage that SIFT has compared to SURF and ORB. The RGB-D SLAM implementation was compared to four prominent research papers, and showed comparable results. The effect of rotation and translation was evaluated, based on the effect of each rotation and translation axis. It was found that the z-axis (scale) and the roll-axis (scene orientation) have a lower effect on the average RPE error in a frame-to-frame basis. It was found that rotation has a much greater impact on the performance, when evaluating rotation and translation separately. On average, a rotation of 1deg resulted in a 4mm translation error and a 20% rotation error , where a translation of 10mm resulted in a rotation error of 0.2deg and a translation error of 45%. The combined effect of rotation and translation had a multiplicative effect on the error metric. The quadcopter platform designed to work with the SLAM implementation did not function ideally, but it was sufficient for the purpose. The quadcopter is able to self stabilise within the environment, given a spacious area. For smaller, enclosed areas the backdraft generated by the quadcopter motors lead to some instability in the system. A frame-to-frame error of 40.34mm and 1.93deg was estimated for the quadcopter system.
Dissertation (MEng)--University of Pretoria, 2017.
Electrical, Electronic and Computer Engineering
MEng
Unrestricted

Submitted by Luciana Ferreira (lucgeral@gmail.com) on 2018-11-21T11:05:44Z No. of bitstreams: 2 Dissertação - Vinícius Araújo Santos - 2018.pdf: 14601054 bytes, checksum: e02a8bcd3cdc93bf2bf202c3933b3f27 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5)
Approved for entry into archive by Luciana Ferreira (lucgeral@gmail.com) on 2018-11-21T11:06:26Z (GMT) No. of bitstreams: 2 Dissertação - Vinícius Araújo Santos - 2018.pdf: 14601054 bytes, checksum: e02a8bcd3cdc93bf2bf202c3933b3f27 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5)
Made available in DSpace on 2018-11-21T11:06:26Z (GMT). No. of bitstreams: 2 Dissertação - Vinícius Araújo Santos - 2018.pdf: 14601054 bytes, checksum: e02a8bcd3cdc93bf2bf202c3933b3f27 (MD5) license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) Previous issue date: 2018-10-11
Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES
Visual Odometry is an important process in image based navigation of robots. The standard methods of this field rely on the good feature matching between frames where feature detection on images stands as a well adressed problem within Computer Vision. Such techniques are subject to illumination problems, noise and poor feature localization accuracy. Thus, 3D information on a scene may mitigate the uncertainty of the features on images. Deep Learning techniques show great results when dealing with common difficulties of VO such as low illumination conditions and bad feature selection. While Visual Odometry and Deep Learning have been connected previously, no techniques applying Siamese Convolutional Networks on depth infomation given by disparity maps have been acknowledged as far as this work’s researches went. This work aims to fill this gap by applying Deep Learning to estimate egomotion through disparity maps on an Siamese architeture. The SiameseVO-Depth architeture is compared to state of the art techniques on OV by using the KITTI Vision Benchmark Suite. The results reveal that the chosen methodology succeeded on the estimation of Visual Odometry although it doesn’t outperform the state-of-the-art techniques. This work presents fewer steps in relation to standard VO techniques for it consists of an end-to-end solution and demonstrates a new approach of Deep Learning applied to Visual Odometry.
Odometria Visual é um importante processo na navegação de robôs baseada em imagens. Os métodos clássicos deste tema dependem de boas correspondências de características feitas entre imagens sendo que a detecção de características em imagens é um tema amplamente discutido no campo de Visão Computacional. Estas técnicas estão sujeitas a problemas de iluminação, presença de ruído e baixa de acurácia de localização. Nesse contexto, a informação tridimensional de uma cena pode ser uma forma de mitigar as incertezas sobre as características em imagens. Técnicas de Deep Learning têm demonstrado bons resultados lidando com problemas comuns em técnicas de OV como insuficiente iluminação e erros na seleção de características. Ainda que já existam trabalhos que relacionam Odometria Visual e Deep Learning, não foram encontradas técnicas que utilizem Redes Convolucionais Siamesas com sucesso utilizando informações de profundidade de mapas de disparidade durante esta pesquisa. Este trabalho visa preencher esta lacuna aplicando Deep Learning na estimativa do movimento por de mapas de disparidade em uma arquitetura Siamesa. A arquitetura SiameseVO-Depth proposta neste trabalho é comparada à técnicas do estado da arte em OV utilizando a base de dados KITTI Vision Benchmark Suite. Os resultados demonstram que através da metodologia proposta é possível a estimativa dos valores de uma Odometria Visual ainda que o desempenho não supere técnicas consideradas estado da arte. O trabalho proposto possui menos etapas em comparação com técnicas clássicas de OV por apresentar-se como uma solução fim-a-fim e apresenta nova abordagem no campo de Deep Learning aplicado à Odometria Visual.

The focus of the thesis is to develop navigation and perception system for small UAVs using low cost inertial and vision sensors, for situations where external positioning information such as GPS losses or totally unavailable in the isolated environments such as valleys, tunnels, mines or indoor areas. Two important areas related to the robot navigation and perception are considered: (1) vision and inertial based navigation system using low cost cameras and inertial sensors in GPS denied situations and (2) Interpretation of the environment simultaneously. A robot’s poses are estimated by visual odometry (VO), which estimates the ego-motion by detecting, tracking and matching the features from a sequence of images and/or point cloud. Due to uncertainties and ambiguities in features and 3D point, outliers add noise in estimated egomotion. Visual odometry estimates the relative position of robot, therefore, in order to compute the current position with respect to the starting point, needs to concatenate the egomotion estimated by all previous frames, therefore drift accumulates. Inertial sensor measurements are fused with visual odometry (VO) to improve accuracy and restrict drifts. A 3D spatial Map is generated using these poses estimated and optimized by visual odometry (VO) and back-end library, respectively. Perception is important for a robot to interact with the complex environment and operate in it safely. The aim is to extract semantic information from 3D map, to develop object or semantic map. Semantic mapping is a process of labeling or tagging the entities in the environment, for example objects, other coarse classes and/or regions in the map, which should be meaningful for the humans and other robots. The spatial map is semantically divided, and partial map is segmented. The features are extracted from each segment, and labels are assigned to the objects and classes in the segment by the classifier.

Recent data availability of three dimensional imaging brought new possibilities for the guided intervention. The management of complex data requires the use of new tools and new technologies. We analyzed the use of algorithms for the automatic, ai-driven segmentation to extrapolate the best systems for the use of CBCTs. The tool improves the timing and precision of the segmentation and allows to give normal values for the cephalometry. The segmented data can be analyzed through three dimensional alignment, the new techniques will be explored for data synthesis. Finally three dimensional resulting datasets can be exported and used to virtually plan an intervention. After the plan is complete they can be used to bring the clinician a new powerful system for the AR guided intervention. Clinical and research data results and outcomes will be given, bringing both preliminary as well as multidisciplinary researchers in the field of: orthodontics, endodontics and oral surgery with clinical direct performances and results.

Visual Simultaneous Localization And Mapping (SLAM) allows for three dimensionalreconstruction from a camera’s output and simultaneous positioning of the camera withinthe reconstruction. With use cases ranging from autonomous vehicles to augmentedreality, the SLAM field has garnered interest both commercially and academically. A SLAM system performs odometry as it estimates the camera’s movement throughthe scene. The incremental estimation of odometry is not error free and exhibits driftover time with map inconsistencies as a result. Detecting the return to a previously seenplace, a loop, means that this new information regarding our position can be incorporatedto correct the trajectory retroactively. Loop detection can also facilitate relocalization ifthe system loses tracking due to e.g. heavy motion blur. This thesis proposes an odometric system making use of bundle adjustment within akeyframe based stereo SLAM application. This system is capable of detecting loops byutilizing the algorithm FAB-MAP. Two aspects of this system is evaluated, the odometryand the capability to relocate. Both of these are evaluated using the EuRoC MAV dataset,with an absolute trajectory RMS error ranging from 0.80 m to 1.70 m for the machinehall sequences. The capability to relocate is evaluated using a novel methodology that intuitively canbe interpreted. Results are given for different levels of strictness to encompass differentuse cases. The method makes use of reprojection of points seen in keyframes to definewhether a relocalization is possible or not. The system shows a capability to relocate inup to 85% of all cases when a keyframe exists that can project 90% of its points intothe current view. Errors in estimated poses were found to be correlated with the relativedistance, with errors less than 10 cm in 23% to 73% of all cases. The evaluation of the whole system is augmented with an evaluation of local imagedescriptors and pose estimation algorithms. The descriptor SIFT was found to performbest overall, but demanding to compute. BRISK was deemed the best alternative for afast yet accurate descriptor. Conclusions that can be drawn from this thesis is that FAB-MAP works well fordetecting loops as long as the addition of keyframes is handled appropriately.

Modern cars can be bought with camera systems that watch the road ahead. They can be used for many purposes, one use is to alert the driver when other cars are in the path of collision. If the warning system is to be reliable, the input data must be correct. One input can be the depth image from a stereo camera system; one reason for the depth image to be wrong is if the vergence angle between the cameras are erroneously calibrated. Even if the calibration is accurate from production there's a risk that the vergence changes due to temperature variations when the car is started. This thesis proposes one solution for short-time live calibration of a stereo camera system; where the speedometer data available on the CAN-bus is used as reference. The motion of the car is estimated using visual odometry, which will be affected by any errors in the calibration. The vergence angle is then altered virtually until the estimated speed is equal to the reference speed. The method is analyzed for noise and tested on real data. It is shown that detection of calibration errors down to 0.01 degrees is possible under certain circumstances using the proposed method.

Un grand nombre d'applications de robotique embarquées pourrait bénéficier d'une détection explicite des objets mobiles. A ce jour, la majorité des approches présentées repose sur la classification, ou sur une analyse structurelle de la scène (la V-Disparité est un bon exemple de ces approches). Depuis quelques années, nous sommes témoins d'un intérêt croissant pour les méthodes faisant collaborer activement l'analyse structurelle et l'analyse du mouvement. Ces deux processus sont en effet étroitement liés. Dans ce contexte, nous proposons, à travers de travail de thèse, deux approches différentes. Si la première fait appel à l'intégralité de l'information stéréo/mouvement, la seconde se penche sur le cas des capteurs monoculaires, et permet de retrouver une information partielle.La première approche présentée consiste en un système innovation d'odométrie visuelle. Nous avons en effet démontré que le problème d'odométrie visuelle peut être posé de façon linéaire, alors que l'immense majorité des auteurs sont contraint de faire appel à des méthodes d'optimisation non-linéaires. Nous avons également montré que notre approche permet d'atteindre, voire de dépasser le niveau de performances présenté par des système matériels haut de gamme (type centrale inertielle). A partir de ce système d'odométrie visuelle, nous définissons une procédure permettant de détecter les objets mobiles. Cette procédure repose sur une compensation de l'influence de l'égo-mouvement, puis une mesure du mouvement résiduel. Nous avons ensuite mené une réflexion de fond sur les limitations et les sources d'amélioration de ce système. Il nous est apparu que les principaux paramètres du système de vision (base, focale) ont un impact de premier plan sur les performances du détecteur. A notre connaissance, cet impact n'a jamais été décrit dans la littérature. Il nous semble cependant que nos conclusions peuvent constituer un ensemble de recommandations utiles à tout concepteur de système de vision intelligent.La seconde partie de ce travail porte sur les systèmes de vision monoculaire, et plus précisément sur le concept de C-Vélocité. Alors que la V-Disparité a défini une transformée de la carte de disparité permettant de mettre en avant certains plans de l'image, la C-Vélocité défini une transformée du champ de flot optique, et qui utilise la position du FoE, qui permet une détection facile de certains plans spécifiques de l'image. Dans ce travail, nous présentons une modification de la C-Vélocité. Au lieu d'utiliser un a priori sur l'égo-mouvement (la position du FoE) afin d'inférer la structure de la scène, nous utilisons un a priori sur la structure de la scène afin de localiser le FoE, donc d'estimer l'égo-mouvement translationnel. Les premiers résultats de ce travail sont encourageants et nous permettent d'ouvrir plusieurs pistes de recherches futures
Many embedded robotic applications could benefit from an explicit detection of mobile objects. To this day, most approaches rely on classification, or on some structural scene analysis (for instance, V-Disparity). During the last few years, we've witnessed a growing interest for collaboration methods, that use actively btw structural analysis and motion analysis. These two processes are, indeed, closely related. In this context, we propose, through this study, two novel approaches that address this issue. While the first one use information from stereo and motion, the second one focuses on monocular systems, and allows us to retrieve a partial information.The first presented approach consists in a novel visual odometry system. We have shown that, even though the wide majority of authors tackle the visual odometry problem as non-linear, it can be shown to be purely linear. We have also shown that our approach achieves performances, as good as, or even better than the ones achieved by high-end IMUs. Given this visual odometry system, we then define a procedure allowing us to detect mobile objects. This procedure relies on a compensation of the ego-motion and a measure of the residual motion. We then lead a reflexion on the causes of limitation and the possible sources of improvement of this system. It appeared that the main parameters of the vision system (baseline, focal length) have a major impact on the performances of our detector. To the best of our knowledge, this impact had never been discussed, prior to our study. However, we think that our conclusion could be used as a set of recommendations, useful for every designer of intelligent vision system.the second part of this work focuses on monocular systems, and more specifically on the concept of C-Velocity. When V-Disparity defined a disparity map transform, allowing an easy detection of specific planes, C-Velocity defines a transform of the optical flow field, using the position of the FoE, allowing an easy detection of specific planes. Through this work, we present a modification of the C-Velocity concept. Instead of using a priori knowledge of the ego-motion (the position of the FoE) in order to determine the scene structure, we use a prior knowledge of the scene structure in order to localize the FoE, thus the translational ego-motion. the first results of this work are promising, and allow us to define several future works