Dissertations / Theses on the topic 'Point cloud instance segmentation'

The camera is the cheapest and computationally real-time option for detecting or segmenting the environment for an autonomous vehicle, but it does not provide the depth information and is undoubtedly not reliable during the night, bad weather, and tunnel flash outs. The risk of an accident gets higher for autonomous cars when driven by a camera in such situations. The industry has been relying on LiDAR for the past decade to solve this problem and focus on depth information of the environment, but LiDAR also has its shortcoming. The industry methods commonly use projections methods to create a projection image and run detection and localization network for inference, but LiDAR sees obscurants in bad weather and is sensitive enough to detect snow, making it difficult for robustness in projection based methods. We propose a novel pointwise and Instance segmentation deep learning architecture for the point clouds focused on self-driving application. The model is only dependent on LiDAR data making it light invariant and overcoming the shortcoming of the camera in the perception stack. The pipeline takes advantage of both global and local/edge features of points in points clouds to generate high-level feature. We also propose Pointer-Capsnet which is an extension of CapsNet for small 3D point clouds.

This thesis provides a comparison between instance segmentation methods using point clouds and depth images. Specifically, their performance on cluttered scenes of irregular objects in an industrial environment is investigated. Recent work by Wang et al. [1] has suggested potential benefits of a point cloud representation when performing deep learning on data from 3D cameras. However, little work has been done to enable quantifiable comparisons between methods based on different representations, particularly on industrial data. Generating synthetic data provides accurate grayscale, depth map, and point cloud representations for a large number of scenes and can thus be used to compare methods regardless of datatype. The datasets in this work are created using a tool provided by SICK. They simulate postal packages on a conveyor belt scanned by a LiDAR, closely resembling a common industry application. Two datasets are generated. One dataset has low complexity, containing only boxes.The other has higher complexity, containing a combination of boxes and multiple types of irregularly shaped parcels. State-of-the-art instance segmentation methods are selected based on their performance on existing benchmarks. We chose PointGroup by Jiang et al. [2], which uses point clouds, and Mask R-CNN by He et al. [3], which uses images. The results support that there may be benefits of using a point cloud representation over depth images. PointGroup performs better in terms of the chosen metric on both datasets. On low complexity scenes, the inference times are similar between the two methods tested. However, on higher complexity scenes, MaskR-CNN is significantly faster.

Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2016.
This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.
Cataloged from student-submitted PDF version of thesis.
Includes bibliographical references (pages 47-48).
In this thesis, we develop a system for estimating a belief state for a scene over multiple observations of the scene. Given as input a sequence of observed RGB-D point clouds of a scene, a list of known objects in the scene and their pose distributions as a prior, and a black-box object detector, our system outputs a belief state of what is believed to be in the scene. This belief state consists of the states of known objects, walls, the floor, and "stuff" in the scene based on the observed point clouds. The system first segments the observed point clouds and then incrementally updates the belief state with each segmented point cloud.
by Charlotte Zhu.
M. Eng.

Achieving robot autonomy is an extremely challenging task and it starts with developing algorithms that help the robot understand how humans perceive the environment around them. Once the robot understands how to make sense of its environment, it is easy to make efficient decisions about safe movement. It is hard for robots to perform tasks that come naturally to humans like understanding signboards, classifying traffic lights, planning path around dynamic obstacles, etc. In this work, we take up one such challenge of motion segmentation using Light Detection and Ranging (LiDAR) point clouds. Motion segmentation is the task of classifying a point as either moving or static. As the ego-vehicle moves along the road, it needs to detect moving cars with very high certainty as they are the areas of interest which provide cues to the ego-vehicle to plan it's motion. Motion segmentation algorithms segregate moving cars from static cars to give more importance to dynamic obstacles. In contrast to the usual LiDAR scan representations like range images and regular grid, this work uses a modern representation of LiDAR scans using permutohedral lattices. This representation gives ease of representing unstructured LiDAR points in an efficient lattice structure. We propose a machine learning approach to perform motion segmentation. The network architecture takes in two sequential point clouds and performs convolutions on them to estimate if 3D points from the first point cloud are moving or static. Using two temporal point clouds help the network in learning what features constitute motion. We have trained and tested our learning algorithm on the FlyingThings3D dataset and a modified KITTI dataset with simulated motion.

Semantic segmentation is a key approach to comprehensive image data analysis. It can be applied to analyze 2D images, videos, and even point clouds that contain 3D data points. On the first two problems, CNNs have achieved remarkable progress, but on point cloud segmentation, the results are less satisfactory due to challenges such as limited memory resource and difficulties in 3D point annotation. One of the research studies carried out by the Computer Vision Lab at Linköping University was aiming to ease the semantic segmentation of 3D point cloud. The idea is that by first projecting 3D data points to 2D space and then focusing only on the analysis of 2D images, we can reduce the overall workload for the segmentation process as well as exploit the existing well-developed 2D semantic segmentation techniques. In order to improve the performance of CNNs for 2D semantic segmentation, the study has used input data derived from different modalities. However, how different modalities can be optimally fused is still an open question. Based on the above-mentioned study, this thesis aims to improve the multistream framework architecture. More concretely, we investigate how different singlestream architectures impact the multistream framework with a given fusion method, and how different fusion methods contribute to the overall performance of a given multistream framework. As a result, our proposed fusion architecture outperformed all the investigated traditional fusion methods. Along with the best singlestream candidate and few additional training techniques, our final proposed multistream framework obtained a relative gain of 7.3\% mIoU compared to the baseline on the semantic3D point cloud test set, increasing the ranking from 12th to 5th position on the benchmark leaderboard.

Light Detection And Ranging (LiDAR), as well as many other applications and sensors, involve segmenting sparse sets of points (point clouds) for which point density is the only discriminating feature. The segmentation of these point clouds is challenging for several reasons, including the fact that the points are not associated with a regular grid. Moreover, the presence of noise, particularly impulsive noise with varying density, can make it difficult to obtain a good segmentation using traditional techniques, including the algorithms that had been developed to process LiDAR data. This dissertation introduces novel algorithms and frameworks based on statistical techniques and image analysis in order to segment and extract surfaces from sparse noisy point clouds. We introduce an adaptive method for mapping point clouds onto an image grid followed by a contour detection approach that is based on an enhanced version of region-based Active Contours Without Edges (ACWE). We also proposed a noise reduction method using Bayesian approach and incorporated it, along with other noise reduction approaches, into a joint framework that produces robust results. We combined the aforementioned techniques with a statistical surface refinement method to introduce a novel framework to detect ground and canopy surfaces in micropulse photon-counting LiDAR data. The algorithm is fully automatic and uses no prior elevation or geographic information to extract surfaces. Moreover, we propose a novel segmentation framework for noisy point clouds in the plane based on a Markov random field (MRF) optimization that we call Point Cloud Densitybased Segmentation (PCDS). We also developed a large synthetic dataset of in plane point clouds that includes either a set of randomly placed, sized and oriented primitive objects (circle, rectangle and triangle) or an arbitrary shape that forms a simple approximation for the LiDAR point clouds. The experiment performed on a large number of real LiDAR and synthetic point clouds showed that our proposed frameworks and algorithms outperforms the state-of-the-art algorithms in terms of segmentation accuracy and surface RMSE.
Ph. D.

The presented diploma thesis deals with the division of points into homogeneous groups. The work provides a broad overview of the current state in this topic and a brief explanation of the main segmentation methods principles. From the analysis of the articles are selected and programmed five algorithms. The work defines the principles of selected algorithms and explains their mathematical models. For each algorithm is also given a code design description. The diploma thesis also contains a cross comparison of segmentation capabilities of individual algorithms on created as well as on measured data. The results of the curves extraction are compared with each other graphically and numerically. At the end of the work is a comparison graph of time dependence on the number of points and the table that includes a mutual comparison of algorithms in specific areas.

Orchards demand large areas of land, thus they are often situated far from major population centres. As a result it is often difficult to obtain the necessary personnel, limiting both growth and productivity. However, if autonomous robots could be integrated into the operation of the orchard, the manpower demand could be reduced. A key problem for any autonomous robot is localisation; how does the robot know where it is? In agriculture robots, the most common approach is to use GPS positioning. However, in an orchard environment, the dense and tall vegetation restricts the usage to large robots that reach above the surroundings. In order to enable the use of smaller robots, it is instead necessary to use a GPS independent system. However, due to the similarity of the environment and the lack of strong recognisable features, it appears unlikely that typical non-GPS solutions will prove successful. Therefore we present a GPS independent localisation system, specifically aimed for orchards, that utilises the inherent structure of the surroundings. Furthermore, we examine and individually evaluate three related sub-problems. The proposed system utilises a 3D point cloud created from a 2D LIDAR and the robot’s movement. First, we show how the data can be segmented into individual trees using a Hidden Semi-Markov Model. Second, we introduce a set of descriptors for describing the geometric characteristics of the individual trees. Third, we present a robust localisation method based on Hidden Markov Models. Finally, we propose a method for detecting segmentation errors when associating new tree measurements with previously measured trees. Evaluation shows that the proposed segmentation method is accurate and yields very few segmentation errors. Furthermore, the introduced descriptors are determined to be consistent and informative enough to allow localisation. Third, we show that the presented localisation method is robust both to noise and segmentation errors. Finally it is shown that a significant majority of all segmentation errors can be detected without falsely labeling correct segmentations as incorrect.
Eftersom fruktodlingar kräver stora markområden är de ofta belägna långt från större befolkningscentra. Detta gör det svårt att finna tillräckligt med arbetskraft och begränsar expansionsmöjligheterna. Genom att integrera autonoma robotar i drivandet av odlingarna skulle arbetet kunna effektiviseras och behovet av arbetskraft minska. Ett nyckelproblem för alla autonoma robotar är lokalisering; hur vet roboten var den är? I jordbruksrobotar är standardlösningen att använda GPS-positionering. Detta är dock problematiskt i fruktodlingar, då den höga och täta vegetationen begränsar användandet till större robotar som når ovanför omgivningen. För att möjliggöra användandet av mindre robotar är det istället nödvändigt att använda ett GPS-oberoende lokaliseringssystem. Detta problematiseras dock av den likartade omgivningen och bristen på distinkta riktpunkter, varför det framstår som osannolikt att existerande standardlösningar kommer fungera i denna omgivning. Därför presenterar vi ett GPS-oberoende lokaliseringssystem, speciellt riktat mot fruktodlingar, som utnyttjar den naturliga strukturen hos omgivningen.Därutöver undersöker vi och utvärderar tre relaterade delproblem. Det föreslagna systemet använder ett 3D-punktmoln skapat av en 2D-LIDAR och robotens rörelse. Först visas hur en dold semi-markovmodell kan användas för att segmentera datasetet i enskilda träd. Därefter introducerar vi ett antal deskriptorer för att beskriva trädens geometriska form. Vi visar därefter hur detta kan kombineras med en dold markovmodell för att skapa ett robust lokaliseringssystem.Slutligen föreslår vi en metod för att detektera segmenteringsfel när nya mätningar av träd associeras med tidigare uppmätta träd. De föreslagna metoderna utvärderas individuellt och visar på goda resultat. Den föreslagna segmenteringsmetoden visas vara noggrann och ge upphov till få segmenteringsfel. Därutöver visas att de introducerade deskriptorerna är tillräckligt konsistenta och informativa för att möjliggöra lokalisering. Ytterligare visas att den presenterade lokaliseringsmetoden är robust både mot brus och segmenteringsfel. Slutligen visas att en signifikant majoritet av alla segmenteringsfel kan detekteras utan att felaktigt beteckna korrekta segmenteringar som inkorrekta.

Light Detection and Ranging (LiDAR) sensors have many different application areas, from revealing archaeological structures to aiding navigation of vehicles. However, it is challenging to interpret and fully use the vast amount of unstructured data that LiDARs collect. Automatic classification of LiDAR data would ease the utilization, whether it is for examining structures or aiding vehicles. In recent years, there have been many advances in deep learning for semantic segmentation of automotive LiDAR data, but there is less research on aerial LiDAR data. This thesis investigates the current state-of-the-art deep learning architectures, and how well they perform on LiDAR data acquired by an Unmanned Aerial Vehicle (UAV). It also investigates different training techniques for class imbalanced and limited datasets, which are common challenges for semantic segmentation networks. Lastly, this thesis investigates if pre-training can improve the performance of the models. The LiDAR scans were first projected to range images and then a fully convolutional semantic segmentation network was used. Three different training techniques were evaluated: weighted sampling, data augmentation, and grouping of classes. No improvement was observed by the weighted sampling, neither did grouping of classes have a substantial effect on the performance. Pre-training on the large public dataset SemanticKITTI resulted in a small performance improvement, but the data augmentation seemed to have the largest positive impact. The mIoU of the best model, which was trained with data augmentation, was 63.7% and it performed very well on the classes Ground, Vegetation, and Vehicle. The other classes in the UAV dataset, Person and Structure, had very little data and were challenging for most models to classify correctly. In general, the models trained on UAV data performed similarly as the state-of-the-art models trained on automotive data.

Virtual reconstruction of historic sites, planning of restorations and attachments of new building parts, as well as forest inventory are few examples of fields that benefit from the application of 3D surveying data. Originally using 2D photo based documentation and manual distance measurements, the 3D information obtained from multi camera and laser scanning systems realizes a noticeable improvement regarding the surveying times and the amount of generated 3D information. The 3D data allows a detailed post processing and better visualization of all relevant spatial information. Yet, for the extraction of the required information from the raw scan data and for the generation of useable visual output, time-consuming, complex user-based data processing is still required, using the commercially available 3D software tools. In this context, the automatic object recognition from 3D point cloud and depth data has been discussed in many different works. The developed tools and methods however, usually only focus on a certain kind of object or the detection of learned invariant surface shapes. Although the resulting methods are applicable for certain practices of data segmentation, they are not necessarily suitable for arbitrary tasks due to the varying requirements of the different fields of research. This thesis presents a more widespread solution for automatic scene reconstruction from 3D point clouds, targeting street scenarios, specifically for the task of traffic accident scene analysis and documentation. The data, obtained by sampling the scene using a mobile scanning system is evaluated, segmented, and finally used to generate detailed 3D information of the scanned environment. To realize this aim, this work adapts and validates various existing approaches on laser scan segmentation regarding the application on accident relevant scene information, including road surfaces and markings, vehicles, walls, trees and other salient objects. The approaches are therefore evaluated regarding their suitability and limitations for the given tasks, as well as for possibilities concerning the combined application together with other procedures. The obtained knowledge is used for the development of new algorithms and procedures to allow a satisfying segmentation and reconstruction of the scene, corresponding to the available sampling densities and precisions. Besides the segmentation of the point cloud data, this thesis presents different visualization and reconstruction methods to achieve a wider range of possible applications of the developed system for data export and utilization in different third party software tools.

Many laser devices acquire directly 3D objects and reconstruct their surface. Nevertheless, the final reconstructed surface is usually smoothed out as a result of the scanner internal de-noising process and the offsets between different scans. This thesis, working on results from high precision scans, adopts the somewhat extreme conservative position, not to loose or alter any raw sample throughout the whole processing pipeline, and to attempt to visualize them. Indeed, it is the only way to discover all surface imperfections (holes, offsets). Furthermore, since high precision data can capture the slightest surface variation, any smoothing and any sub-sampling can incur in the loss of textural detail.The thesis attempts to prove that one can triangulate the raw point cloud with almost no sample loss. It solves the exact visualization problem on large data sets of up to 35 million points made of 300 different scan sweeps and more. Two major problems are addressed. The first one is the orientation of the complete raw point set, an the building of a high precision mesh. The second one is the correction of the tiny scan misalignments which can cause strong high frequency aliasing and hamper completely a direct visualization.The second development of the thesis is a general low-high frequency decomposition algorithm for any point cloud. Thus classic image analysis tools, the level set tree and the MSER representations, are extended to meshes, yielding an intrinsic mesh segmentation method.The underlying mathematical development focuses on an analysis of a half dozen discrete differential operators acting on raw point clouds which have been proposed in the literature. By considering the asymptotic behavior of these operators on a smooth surface, a classification by their underlying curvature operators is obtained.This analysis leads to the development of a discrete operator consistent with the mean curvature motion (the intrinsic heat equation) defining a remarkably simple and robust numerical scale space. By this scale space all of the above mentioned problems (point set orientation, raw point set triangulation, scan merging, segmentation), usually addressed by separated techniques, are solved in a unified framework.

Le graphe est un concept puissant pour la représentation des relations entre des paires d'entités. Les données ayant une structure de graphes sous-jacente peuvent être trouvées dans de nombreuses disciplines, décrivant des composés chimiques, des surfaces des modèles tridimensionnels, des interactions sociales ou des bases de connaissance, pour n'en nommer que quelques-unes. L'apprentissage profond (DL) a accompli des avancées significatives dans une variété de tâches d'apprentissage automatique au cours des dernières années, particulièrement lorsque les données sont structurées sur une grille, comme dans la compréhension du texte, de la parole ou des images. Cependant, étonnamment peu de choses ont été faites pour explorer l'applicabilité de DL directement sur des données structurées sous forme des graphes. L'objectif de cette thèse est d'étudier des architectures de DL sur des graphes et de rechercher comment transférer, adapter ou généraliser à ce domaine des concepts qui fonctionnent bien sur des données séquentielles et des images. Nous nous concentrons sur deux primitives importantes : le plongement de graphes ou leurs nœuds dans une représentation de l'espace vectorielle continue (codage) et, inversement, la génération des graphes à partir de ces vecteurs (décodage). Nous faisons les contributions suivantes. Tout d'abord, nous introduisons Edge-Conditioned Convolutions (ECC), une opération de type convolution sur les graphes réalisés dans le domaine spatial où les filtres sont générés dynamiquement en fonction des attributs des arêtes. La méthode est utilisée pour coder des graphes avec une structure arbitraire et variable. Deuxièmement, nous proposons SuperPoint Graph, une représentation intermédiaire de nuages de points avec de riches attributs des arêtes codant la relation contextuelle entre des parties des objets. Sur la base de cette représentation, l'ECC est utilisé pour segmenter les nuages de points à grande échelle sans sacrifier les détails les plus fins. Troisièmement, nous présentons GraphVAE, un générateur de graphes permettant de décoder des graphes avec un nombre de nœuds variable mais limité en haut, en utilisant la correspondance approximative des graphes pour aligner les prédictions d'un auto-encodeur avec ses entrées. La méthode est appliquée à génération de molécules
Graph is a powerful concept for representation of relations between pairs of entities. Data with underlying graph structure can be found across many disciplines, describing chemical compounds, surfaces of three-dimensional models, social interactions, or knowledge bases, to name only a few. There is a natural desire for understanding such data better. Deep learning (DL) has achieved significant breakthroughs in a variety of machine learning tasks in recent years, especially where data is structured on a grid, such as in text, speech, or image understanding. However, surprisingly little has been done to explore the applicability of DL on graph-structured data directly.The goal of this thesis is to investigate architectures for DL on graphs and study how to transfer, adapt or generalize concepts working well on sequential and image data to this domain. We concentrate on two important primitives: embedding graphs or their nodes into a continuous vector space representation (encoding) and, conversely, generating graphs from such vectors back (decoding). To that end, we make the following contributions.First, we introduce Edge-Conditioned Convolutions (ECC), a convolution-like operation on graphs performed in the spatial domain where filters are dynamically generated based on edge attributes. The method is used to encode graphs with arbitrary and varying structure.Second, we propose SuperPoint Graph, an intermediate point cloud representation with rich edge attributes encoding the contextual relationship between object parts. Based on this representation, ECC is employed to segment large-scale point clouds without major sacrifice in fine details.Third, we present GraphVAE, a graph generator allowing to decode graphs with variable but upper-bounded number of nodes making use of approximate graph matching for aligning the predictions of an autoencoder with its inputs. The method is applied to the task of molecule generation

Traditional forest management relies on a small field sample and interpretation of aerial photography that not only are costly to execute but also yield inaccurate estimates of the entire forest in question. Airborne light detection and ranging (LiDAR) is a remote sensing technology that records point clouds representing the 3D structure of a forest canopy and the terrain underneath. We present a method for segmenting individual trees from the LiDAR point clouds without making prior assumptions about tree crown shapes and sizes. We then present a method that vertically stratifies the point cloud to an overstory and multiple understory tree canopy layers. Using the stratification method, we modeled the occlusion of higher canopy layers with respect to point density. We also present a distributed computing approach that enables processing the massive data of an arbitrarily large forest. Lastly, we investigated using deep learning for coniferous/deciduous classification of point cloud segments representing individual tree crowns. We applied the developed methods to the University of Kentucky Robinson Forest, a natural, majorly deciduous, closed-canopy forest. 90% of overstory and 47% of understory trees were detected with false positive rates of 14% and 2% respectively. Vertical stratification improved the detection rate of understory trees to 67% at the cost of increasing their false positive rate to 12%. According to our occlusion model, a point density of about 170 pt/m² is needed to segment understory trees located in the third layer as accurately as overstory trees. Using our distributed processing method, we segmented about two million trees within a 7400-ha forest in 2.5 hours using 192 processing cores, showing a speedup of ~170. Our deep learning experiments showed high classification accuracies (~82% coniferous and ~90% deciduous) without the need to manually assemble the features. In conclusion, the methods developed are steps forward to remote, accurate quantification of large natural forests at the individual tree level.

Disertační práce se zabývá zpracováním mračenen bodů z laserových skenerů pomocí vektorizace a následnému vyhledávání korespondencí mezi takto získanými aproximacemi pro potřeby současné sebelokalizace a mapování v mobilní robotice. První nová metoda je určena pro segmentaci a filtraci surových dat a realizuje obě operace najednou v jednom algoritmu. Pro vektorizaci je představen optimalizovaný algoritmus založený na úplné metodě nejmenších čtverců, který je v současnosti patrně nejrychlejší ve své třídě a blíží se tak eliminačním metodám, které ovšem produkují výrazně horší aproxi- mace. Inovativní analytické metody jsou představeny i pro účely vyjádření podobnosti mezi dvěma vektorizovanými skeny, pro jejich optimální sesazení a pro vyhledávání korespondencí mezi nimi. Všechny představené algoritmy jsou intezivně testovány a jejich vlastnosti ověřeny množstvím experimentů.

La modélisation géométrique d'objets fabriqués par l'homme à partir de données 3D est l'un des plus grands défis de la vision par ordinateur et de l'infographie. L'objectif à long terme est de générer des modèles de type CAO de la manière la plus automatique possible. Pour atteindre cet objectif, des problèmes difficiles doivent être résolus, notamment (i) le passage à l'échelle du processus de modélisation sur des données d'entrée massives, (ii) la robustesse de la méthodologie contre des mesures d'entrées erronés, et (iii) la qualité géométrique des modèles de sortie. Les méthodes existantes fonctionnent efficacement pour reconstruire la surface des objets de forme libre. Cependant, dans le cas d'objets fabriqués par l'homme, il est difficile d'obtenir des résultats dont la qualité approche celle des représentations hautement structurées, comme les modèles CAO. Dans cette thèse, nous présentons une série de contributions dans ce domaine. Tout d'abord, nous proposons une méthode de classification basée sur l'apprentissage en profondeur pour distinguer des objets dans des environnements complexes à partir de nuages de points 3D. Deuxièmement, nous proposons un algorithme pour détecter des primitives planaires dans des données 3D à différents niveaux d'abstraction. Enfin, nous proposons un mécanisme pour assembler des primitives planaires en maillages polygonaux compacts. Ces contributions sont complémentaires et peuvent être utilisées de manière séquentielle pour reconstruire des modèles de ville à différents niveaux de détail à partir de données 3D aéroportées. Nous illustrons la robustesse, le passage à l'échelle et l'efficacité de nos méthodes sur des données laser et multi-vues stéréo sur des scènes composées d'objets fabriqués par l'homme
Geometric modeling of man-made objects from 3D data is one of the biggest challenges in Computer Vision and Computer Graphics. The long term goal is to generate a CAD-style model in an as-automatic-as-possible way. To achieve this goal, difficult issues have to be addressed including (i) the scalability of the modeling process with respect to massive input data, (ii) the robustness of the methodology to various defect-laden input measurements, and (iii) the geometric quality of output models. Existing methods work well to recover the surface of free-form objects. However, in case of manmade objects, it is difficult to produce results that approach the quality of high-structured representations as CAD models.In this thesis, we present a series of contributions to the field. First, we propose a classification method based on deep learning to distinguish objects from raw 3D point cloud. Second, we propose an algorithm to detect planar primitives in 3D data at different level of abstraction. Finally, we propose a mechanism to assemble planar primitives into compact polygonal meshes. These contributions are complementary and can be used sequentially to reconstruct city models at various level-of-details from airborne 3D data. We illustrate the robustness, scalability and efficiency of our methods on both laser and multi-view stereo data composed of man-made objects

Cette thèse s'inscrit dans le contexte du développement d'outils numériques innovants au service de ce qui est communément désigné par Usine du Futur. Nos travaux de recherche ont été menés dans le cadre du laboratoire de recherche commun "Inspection 4.0" entre IMT Mines Albi/ICA et la Sté DIOTA spécialisée dans le développement d'outils numériques pour l'Industrie 4.0. Dans cette thèse, nous nous sommes intéressés au développement de systèmes exploitant des images 2D ou des nuages de points 3D pour l'inspection automatique d'assemblages mécaniques aéronautiques complexes (typiquement un moteur d'avion). Nous disposons du modèle CAO de l'assemblage (aussi désigné par maquette numérique) et il s'agit de vérifier que l'assemblage a été correctement assemblé, i.e que tous les éléments constituant l'assemblage sont présents, dans la bonne position et à la bonne place. La maquette numérique sert de référence. Nous avons développé deux scénarios d'inspection qui exploitent les moyens d'inspection développés par DIOTA : (1) un scénario basé sur une tablette équipée d'une caméra, portée par un opérateur pour un contrôle interactif temps-réel, (2) un scénario basé sur un robot équipé de capteurs (deux caméras et un scanner 3D) pour un contrôle totalement automatique. Dans les deux scénarios, une caméra dite de localisation fournit en temps-réel la pose entre le modèle CAO et les capteurs mis en œuvre (ce qui permet de relier directement la maquette numérique 3D avec les images 2D ou les nuages de points 3D analysés). Nous avons d'abord développé des méthodes d'inspection 2D, basées uniquement sur l'analyse d'images 2D puis, pour certains types d'inspection qui ne pouvaient pas être réalisés à partir d'images 2D (typiquement nécessitant la mesure de distances 3D), nous avons développé des méthodes d'inspection 3D basées sur l'analyse de nuages de points 3D. Pour l'inspection 3D de câbles électriques présents au sein de l'assemblage, nous avons proposé une méthode originale de segmentation 3D des câbles. Nous avons aussi traité la problématique de choix automatique de point de vue qui permet de positionner le capteur d'inspection dans une position d'observation optimale. Les méthodes développées ont été validées sur de nombreux cas industriels. Certains des algorithmes d’inspection développés durant cette thèse ont été intégrés dans le logiciel DIOTA Inspect© et sont utilisés quotidiennement chez les clients de DIOTA pour réaliser des inspections sur site industriel
This thesis makes part of a research aimed towards innovative digital tools for the service of what is commonly referred to as Factory of the Future. Our work was conducted in the scope of the joint research laboratory "Inspection 4.0" founded by IMT Mines Albi/ICA and the company DIOTA specialized in the development of numerical tools for Industry 4.0. In the thesis, we were interested in the development of systems exploiting 2D images or (and) 3D point clouds for the automatic inspection of complex aeronautical mechanical assemblies (typically an aircraft engine). The CAD (Computer Aided Design) model of the assembly is at our disposal and our task is to verify that the assembly has been correctly assembled, i.e. that all the elements constituting the assembly are present in the right position and at the right place. The CAD model serves as a reference. We have developed two inspection scenarios that exploit the inspection systems designed and implemented by DIOTA: (1) a scenario based on a tablet equipped with a camera, carried by a human operator for real-time interactive control, (2) a scenario based on a robot equipped with sensors (two cameras and a 3D scanner) for fully automatic control. In both scenarios, a so-called localisation camera provides in real-time the pose between the CAD model and the sensors (which allows to directly link the 3D digital model with the 2D images or the 3D point clouds analysed). We first developed 2D inspection methods, based solely on the analysis of 2D images. Then, for certain types of inspection that could not be performed by using 2D images only (typically requiring the measurement of 3D distances), we developed 3D inspection methods based on the analysis of 3D point clouds. For the 3D inspection of electrical cables, we proposed an original method for segmenting a cable within a point cloud. We have also tackled the problem of automatic selection of best view point, which allows the inspection sensor to be placed in an optimal observation position. The developed methods have been validated on many industrial cases. Some of the inspection algorithms developed during this thesis have been integrated into the DIOTA Inspect© software and are used daily by DIOTA's customers to perform inspections on industrial sites

This thesis examines an automated method to detect surface deformations on automotive panels for the purpose of quality control along a manufacturing assembly line. Automation in the automotive manufacturing industry is becoming more prominent, but quality control is still largely performed by human workers. Quality control is important in the context of automotive body panels as deformations can occur along the assembly line such as inadequate handling of parts or tools around a vehicle during assembly, rack storage, and shipping from subcontractors. These defects are currently identified and marked, before panels are either rectified or discarded. This work attempts to develop an automated system to detect deformations to alleviate the dependence on human workers in quality control and improve performance by increasing speed and accuracy. Some techniques make use of an ideal CAD model behaving as a master work, and panels scanned on the assembly line are compared to this model to determine the location of deformations. This thesis presents a solution for detecting deformations of various scales without a master work. It also focuses on automated analysis requiring minimal intuitive operator-set parameters and provides the ability to classify the deformations as dings, which are deformations that protrude from the surface, or dents, which are depressions into the surface. A complete automated deformation detection system is proposed, comprised of a feature extraction module, segmentation module, and classification module, which outputs the locations of deformations when provided with the 3D mesh of an automotive panel. Two feature extraction techniques are proposed. The first is a general feature extraction technique for 3D meshes using octrees for multi-resolution analysis and evaluates the amount of surface variation to locate deformations. The second is specifically designed for the purpose of deformation detection, and analyzes multi-resolution cross-sections of a 3D mesh to locate deformations based on their estimated size. The performance of the proposed automated deformation detection system, and all of its sub-modules, is tested on a set of meshes which represent differing characteristics of deformations in surface panels, including deformations of different scales. Noisy, low resolution meshes are captured from a 3D acquisition, while artificial meshes are generated to simulate ideal acquisition conditions. The proposed system shows accurate results in both ideal situations as well as non-ideal situations under the condition of noise and complex surface curvature by extracting only the deformations of interest and accurately classifying them as dings or dents.

The point cloud is an unorganized set of points with 3D coordinates (x, y, z) which represents a real object. These point clouds are acquired by the technology called 3D scanning. This scanning technique can be done by various methods, such as LIDAR (Light Detection And Ranging) or by utilizing recently developed 3D scanners. Point clouds can be therefore used in various applications, such as mechanical or reverse engineering, rapid prototyping, biology, nuclear physics or virtual reality. Therefore in this doctoral Ph.D. thesis, I focus on feature detection and visualization in a point cloud. These features represent parts of the object that can be described by the well--known mathematical model (lines, planes, helices etc.). The points on the sharp edges are especialy problematic for commonly used methods. Therefore, I focus on detection of these problematic points. This doctoral Ph.D. thesis presents a new algorithm for precise detection of these problematic points. Visualization of these points is done by a modified curve fitting algoritm with a new weight function that leads to better results. Each of the proposed methods were tested on real data sets and compared with contemporary published methods.

Pour agir de manière autonome et intelligente dans un environnement, un robot mobile doit disposer de cartes. Une carte contient les informations spatiales sur l’environnement. La géométrie 3D ainsi connue par le robot est utilisée non seulement pour éviter la collision avec des obstacles, mais aussi pour se localiser et pour planifier des déplacements. Les robots de prochaine génération ont besoin de davantage de capacités que de simples cartographies et d’une localisation pour coexister avec nous. La quintessence du robot humanoïde de service devra disposer de la capacité de voir comme les humains, de reconnaître, classer, interpréter la scène et exécuter les tâches de manière quasi-anthropomorphique. Par conséquent, augmenter les caractéristiques des cartes du robot à l’aide d’attributs sémiologiques à la façon des humains, afin de préciser les types de pièces, d’objets et leur aménagement spatial, est considéré comme un plus pour la robotique d’industrie et de services à venir. Une carte sémantique enrichit une carte générale avec les informations sur les entités, les fonctionnalités ou les événements qui sont situés dans l’espace. Quelques approches ont été proposées pour résoudre le problème de la cartographie sémantique en exploitant des scanners lasers ou des capteurs de temps de vol RGB-D, mais ce sujet est encore dans sa phase naissante. Dans cette thèse, une tentative de reconstruction sémantisée d’environnement d’intérieur en utilisant une caméra temps de vol qui ne délivre que des informations de profondeur est proposée. Les caméras temps de vol ont modifié le domaine de l’imagerie tridimensionnelle discrète. Elles ont dépassé les scanners traditionnels en termes de rapidité d’acquisition des données, de simplicité fonctionnement et de prix. Ces capteurs de profondeur sont destinés à occuper plus d’importance dans les futures applications robotiques. Après un bref aperçu des approches les plus récentes pour résoudre le sujet de la cartographie sémantique, en particulier en environnement intérieur. Ensuite, la calibration de la caméra a été étudiée ainsi que la nature de ses bruits. La suppression du bruit dans les données issues du capteur est menée. L’acquisition d’une collection d’images de points 3D en environnement intérieur a été réalisée. La séquence d’images ainsi acquise a alimenté un algorithme de SLAM pour reconstruire l’environnement visité. La performance du système SLAM est évaluée à partir des poses estimées en utilisant une nouvelle métrique qui est basée sur la prise en compte du contexte. L’extraction des surfaces planes est réalisée sur la carte reconstruite à partir des nuages de points en utilisant la transformation de Hough. Une interprétation sémantique de l’environnement reconstruit est réalisée. L’annotation de la scène avec informations sémantiques se déroule sur deux niveaux : l’un effectue la détection de grandes surfaces planes et procède ensuite en les classant en tant que porte, mur ou plafond; l’autre niveau de sémantisation opère au niveau des objets et traite de la reconnaissance des objets dans une scène donnée. A partir de l’élaboration d’une signature de forme invariante à la pose et en passant par une phase d’apprentissage exploitant cette signature, une interprétation de la scène contenant des objets connus et inconnus, en présence ou non d’occultations, est obtenue. Les jeux de données ont été mis à la disposition du public de la recherche universitaire
Intelligent autonomous actions in an ordinary environment by a mobile robot require maps. A map holds the spatial information about the environment and gives the 3D geometry of the surrounding of the robot to not only avoid collision with complex obstacles, but also selflocalization and for task planning. However, in the future, service and personal robots will prevail and need arises for the robot to interact with the environment in addition to localize and navigate. This interaction demands the next generation robots to understand, interpret its environment and perform tasks in human-centric form. A simple map of the environment is far from being sufficient for the robots to co-exist and assist humans in the future. Human beings effortlessly make map and interact with environment, and it is trivial task for them. However, for robots these frivolous tasks are complex conundrums. Layering the semantic information on regular geometric maps is the leap that helps an ordinary mobile robot to be a more intelligent autonomous system. A semantic map augments a general map with the information about entities, i.e., objects, functionalities, or events, that are located in the space. The inclusion of semantics in the map enhances the robot’s spatial knowledge representation and improves its performance in managing complex tasks and human interaction. Many approaches have been proposed to address the semantic SLAM problem with laser scanners and RGB-D time-of-flight sensors, but it is still in its nascent phase. In this thesis, an endeavour to solve semantic SLAM using one of the time-of-flight sensors which gives only depth information is proposed. Time-of-flight cameras have dramatically changed the field of range imaging, and surpassed the traditional scanners in terms of rapid acquisition of data, simplicity and price. And it is believed that these depth sensors will be ubiquitous in future robotic applications. In this thesis, an endeavour to solve semantic SLAM using one of the time-of-flight sensors which gives only depth information is proposed. Starting with a brief motivation in the first chapter for semantic stance in normal maps, the state-of-the-art methods are discussed in the second chapter. Before using the camera for data acquisition, the noise characteristics of it has been studied meticulously, and properly calibrated. The novel noise filtering algorithm developed in the process, helps to get clean data for better scan matching and SLAM. The quality of the SLAM process is evaluated using a context-based similarity score metric, which has been specifically designed for the type of acquisition parameters and the data which have been used. Abstracting semantic layer on the reconstructed point cloud from SLAM has been done in two stages. In large-scale higher-level semantic interpretation, the prominent surfaces in the indoor environment are extracted and recognized, they include surfaces like walls, door, ceiling, clutter. However, in indoor single scene object-level semantic interpretation, a single 2.5D scene from the camera is parsed and the objects, surfaces are recognized. The object recognition is achieved using a novel shape signature based on probability distribution of 3D keypoints that are most stable and repeatable. The classification of prominent surfaces and single scene semantic interpretation is done using supervised machine learning and deep learning systems. To this end, the object dataset and SLAM data are also made publicly available for academic research

Dans cette thèse, nous abordons les défis de la rareté des annotations et la fusion de données hétérogènes tels que les nuages de points 3D et images 2D. D’abord, nous adoptons une stratégie de conduite de bout en bout où un réseau de neurones est entraîné pour directement traduire l'entrée capteur (image caméra) en contrôles-commandes, ce qui rend cette approche indépendante des annotations dans le domaine visuel. Nous utilisons l’apprentissage par renforcement profond où l'algorithme apprend de la récompense, obtenue par interaction avec un simulateur réaliste. Nous proposons de nouvelles stratégies d'entraînement et fonctions de récompense pour une meilleure conduite et une convergence plus rapide. Cependant, le temps d’apprentissage reste élevé. C'est pourquoi nous nous concentrons sur la perception dans le reste de cette thèse pour étudier la fusion de nuage de points et d'images. Nous proposons deux méthodes différentes pour la fusion 2D-3D. Premièrement, nous projetons des nuages de points LiDAR 3D dans l’espace image 2D, résultant en des cartes de profondeur éparses. Nous proposons une nouvelle architecture encodeur-décodeur qui fusionne les informations de l’image et la profondeur pour la tâche de complétion de carte de profondeur, améliorant ainsi la résolution du nuage de points projeté dans l'espace image. Deuxièmement, nous fusionnons directement dans l'espace 3D pour éviter la perte d'informations dû à la projection. Pour cela, nous calculons les caractéristiques d’image issues de plusieurs vues avec un CNN 2D, puis nous les projetons dans un nuage de points 3D global pour les fusionner avec l’information 3D. Par la suite, ce nuage de point enrichi sert d'entrée à un réseau "point-based" dont la tâche est l'inférence de la sémantique 3D par point. Sur la base de ce travail, nous introduisons la nouvelle tâche d'adaptation de domaine non supervisée inter-modalités où on a accès à des données multi-capteurs dans une base de données source annotée et une base cible non annotée. Nous proposons une méthode d’apprentissage inter-modalités 2D-3D via une imitation mutuelle entre les réseaux d'images et de nuages de points pour résoudre l’écart de domaine source-cible. Nous montrons en outre que notre méthode est complémentaire à la technique unimodale existante dite de pseudo-labeling
In this thesis, we address the challenges of label scarcity and fusion of heterogeneous 3D point clouds and 2D images. We adopt the strategy of end-to-end race driving where a neural network is trained to directly map sensor input (camera image) to control output, which makes this strategy independent from annotations in the visual domain. We employ deep reinforcement learning where the algorithm learns from reward by interaction with a realistic simulator. We propose new training strategies and reward functions for better driving and faster convergence. However, training time is still very long which is why we focus on perception to study point cloud and image fusion in the remainder of this thesis. We propose two different methods for 2D-3D fusion. First, we project 3D LiDAR point clouds into 2D image space, resulting in sparse depth maps. We propose a novel encoder-decoder architecture to fuse dense RGB and sparse depth for the task of depth completion that enhances point cloud resolution to image level. Second, we fuse directly in 3D space to prevent information loss through projection. Therefore, we compute image features with a 2D CNN of multiple views and then lift them all to a global 3D point cloud for fusion, followed by a point-based network to predict 3D semantic labels. Building on this work, we introduce the more difficult novel task of cross-modal unsupervised domain adaptation, where one is provided with multi-modal data in a labeled source and an unlabeled target dataset. We propose to perform 2D-3D cross-modal learning via mutual mimicking between image and point cloud networks to address the source-target domain shift. We further showcase that our method is complementary to the existing uni-modal technique of pseudo-labeling

Les capacités des technologies de télédétection ont augmenté exponentiellement au cours des dernières années : de nouveaux scanners fournissent maintenant une représentation géométrique de leur environnement sous la forme de nuage de points avec une précision jusqu'ici inégalée. Le traitement de nuages de points est donc devenu une discipline à part entière avec ses problématiques propres et de nombreux défis à relever. Le coeur de cette thèse porte sur la modélisation géométrique et introduit une méthode robuste d'extraction de formes tubulaires à partir de nuages de points. Nous avons choisi de tester nos méthodes dans le contexte applicatif difficile de la foresterie pour mettre en valeur la robustesse de nos algorithmes et leur application à des données volumineuses. Nos méthodes intègrent les normales aux points comme information supplémentaire pour atteindre les objectifs de performance nécessaire au traitement de nuages de points volumineux.Cependant, ces normales ne sont généralement pas fournies par les capteurs, il est donc nécessaire de les pré-calculer.Pour préserver la rapidité d'exécution, notre premier développement a donc consisté à présenter une méthode rapide d'estimation de normales. Pour ce faire nous avons approximé localement la géométrie du nuage de points en utilisant des "patchs" lisses dont la taille s'adapte à la complexité locale des nuages de points. Nos travaux se sont ensuite concentrés sur l’extraction robuste de formes tubulaires dans des nuages de points denses, occlus, bruités et de densité inhomogène. Dans cette optique, nous avons développé une variante de la transformée de Hough dont la complexité est réduite grâce aux normales calculées. Nous avons ensuite couplé ces travaux à une proposition de contours actifs indépendants de leur paramétrisation. Cette combinaison assure la cohérence interne des formes reconstruites et s’affranchit ainsi des problèmes liés à l'occlusion, au bruit et aux variations de densité. Notre méthode a été validée en environnement complexe forestier pour reconstruire des troncs d'arbre afin d'en relever les qualités par comparaison à des méthodes existantes. La reconstruction de troncs d'arbre ouvre d'autres questions à mi-chemin entre foresterie et géométrie. La segmentation des arbres d'une placette forestière est l'une d’entre elles. C'est pourquoi nous proposons également une méthode de segmentation conçue pour contourner les défauts des nuages de points forestiers et isoler les différents objets d'un jeu de données. Durant nos travaux nous avons utilisé des approches de modélisation pour répondre à des questions géométriques, et nous les avons appliqué à des problématiques forestières.Il en résulte un pipeline de traitements cohérent qui, bien qu'illustré sur des données forestières, est applicable dans des contextes variés.
Abstract : The potential of remote sensing technologies has recently increased exponentially: new sensors now provide a geometric representation of their environment in the form of point clouds with unrivalled accuracy. Point cloud processing hence became a full discipline, including specific problems and many challenges to face. The core of this thesis concerns geometric modelling and introduces a fast and robust method for the extraction of tubular shapes from point clouds. We hence chose to test our method in the difficult applicative context of forestry in order to highlight the robustness of our algorithms and their application to large data sets. Our methods integrate normal vectors as a supplementary geometric information in order to achieve the performance goal necessary for large point cloud processing. However, remote sensing techniques do not commonly provide normal vectors, thus they have to be computed. Our first development hence consisted in the development of a fast normal estimation method on point cloud in order to reduce the computing time on large point clouds. To do so, we locally approximated the point cloud geometry using smooth ''patches`` of points which size adapts to the local complexity of the point cloud geometry. We then focused our work on the robust extraction of tubular shapes from dense, occluded, noisy point clouds suffering from non-homogeneous sampling density. For this objective, we developed a variant of the Hough transform which complexity is reduced thanks to the computed normal vectors. We then combined this research with a new definition of parametrisation-invariant active contours. This combination ensures the internal coherence of the reconstructed shapes and alleviates issues related to occlusion, noise and variation of sampling density. We validated our method in complex forest environments with the reconstruction of tree stems to emphasize its advantages and compare it to existing methods. Tree stem reconstruction also opens new perspectives halfway in between forestry and geometry. One of them is the segmentation of trees from a forest plot. Therefore we also propose a segmentation approach designed to overcome the defects of forest point clouds and capable of isolating objects inside a point cloud. During our work we used modelling approaches to answer geometric questions and we applied our methods to forestry problems. Therefore, our studies result in a processing pipeline adapted to forest point cloud analyses, but the general geometric algorithms we propose can also be applied in various contexts.

This thesis is aimed at detection of lines of horizontal road markings from a point cloud, which was obtained using mobile laser mapping. The system works interactively in cooperation with user, which marks the beginning of the traffic line. The program gradually detects the remaining parts of the traffic line and creates its vector representation. Initially, a point cloud is projected into a horizontal plane, crating a 2D image that is segmented by a U-Net convolutional neural network. Segmentation marks one traffic line. Segmentation is converted to a polyline, which can be used in a geo-information system. During testing, the U-Net achieved a segmentation accuracy of 98.8\%, a specificity of 99.5\% and a sensitivity of 72.9\%. The estimated polyline reached an average deviation of 1.8cm.

Scene parsing, aiming to provide a comprehensive understanding of the scene, is a fundamental task in the field of computer vision and remains a challenging problem for the unconstrained environment and open scenes. The results of scene parsing can generate semantic labels, location distribution, as well as for instance shape information for each element, which has shown great potential in the applications like automatic driving, video surveillance, just to name a few. Also, the efficiency of the methods determines whether it can be used on a large scale. With the easy availability of various sensors, more and more solutions resort to different data modalities according to the requirements of the applications. Imagery and point cloud are two representative data sources. How to design efficient frameworks in separate domains remains an open problem and more importantly, lays a solid foundation for multimodal fusion. In this thesis, we study the task of scene parsing under different data modalities, i.e., imagery and point cloud data, by deep neural networks. The first part of this thesis addresses the task of efficient semantic segmentation in 2D image data. The aim is to improve the accuracy of small models while maintaining their fast inference speed without introducing extra computation overhead. To achieve this, we propose a knowledge-distillation-based method tailored for semantic segmentation to improve the performance of the small Fully Convolution Network (FCN) model by injecting compact feature representation and long-tail dependencies from the large complex FCN model (incorporated in Chapter 3). The second part of this thesis addresses the task of semantic and instance segmentation on point cloud data. Compared to rasterized image data, point cloud data often suffer from two problems: (1) how to efficiently extract and aggregate context information. (2) how to solve the forgetting issue Lin et al., 2017c caused by extreme data imbalance. For the first problem, we study the influence of instance-aware knowledge by proposing an Instance-Aware Module by learning discriminative instance embedding features via metric learning (incorporated in Chapter 4). We also address the second problem by proposing a memory-augmented network to learn and memorize the representative prototypes that cover diverse samples universally (incorporated in Chapter 5).
Thesis (Ph.D.) -- University of Adelaide, School of Computer Science, 2020

碩士
國立臺灣大學
土木工程學研究所
103
The point cloud segmentation has been a significant progress to point cloud classification and ground object reconstruction. In addition, the result of segmentation has directly influence over the following analysis and utilization. Considering that LiDAR (light detection and ranging) scanners are attributes of blind systems, the object-based concept is used to analyze point clouds from large amounts of discrete data to point cloud objects, which are composed of parent-child relationships. The methods of point cloud segmentation are diverse in accordance with purposes and demands. For instance, a model-driven approach, RANSAC (random sample consensus), which is robust and efficient, is used to building extraction and reconstruction. Moreover, a data-driven approach, clustering, which clusters highly correlated points into objects, is applied to irregular object identification and classification by calculating Euclidean distance between points. The study is essentially built on the object-based point cloud analysis (OBPCA) and proposes a suitable segmentation method to point clouds. Since the features, also known as attributes, are considered in the object-based point cloud analysis, they are not only beneficial to object analysis, but also provide heterogeneities to the progress of segmentation. The heterogeneity is exploited to simplify the procedure, to improve the efficiency of point cloud segmentation, and to adapt different point cloud distributions of scenes. Therefore, in this research, current methods of segmentation are consolidated and interpreted, and a multi-scale segmentation algorithm is developed for increasing operational efficiency without reducing overall accuracy of classification.

碩士
國立雲林科技大學
資訊工程系
106
In this thesis, a moving detection and tracking method is proposed for multiple targets by using point cloud segmentation. LIDAR systems are widely used in autonomous systems. In an ego-motion system, it is an interesting research topic to identify moving objects from scene point clouds obtained by the mobile LIDAR. The proposed method can detect moving objects within a moving scene and the information of moving objects, e.g., relative velocity, can be used for collision avoidance for a driverless vehicle. The proposed approach consists of five steps: (1) point cloud capturing, (2) ground point removal, (3) segmentation, (4) foreground and background detection, (5) moving object tracking. Firstly, the 3D point cloud scene is retrieved by LiDAR mounted on a ego-motion system. Then, in order to reduce the computation complexity, ground points are removed by the ground detection algorithm. In third step, the rest points are grouped and segmented by the voxel grouping method to eliminate the noise point and to form objects. The velocities of objects are computed with respect to the ego-motion system for identifying the foreground (moving object) and the background (static objects). Finally, Kalman filter is used to track moving objects and to expect the position of these objects. The expecting position of moving objects can be used for collision avoidance.

碩士
國立臺北科技大學
電機工程系
107
In this thesis, the simultaneous localization and mapping (SLAM) will be employed to establish a three-dimensional indoor environment map of a real scene, and then the users engaged in a virtual experience can use the head-mounted display to view the established real scene model. For the environment scene construction, the original point cloud obtained by the depth camera will have empty points. Also, since the sensing range is limited during the SLAM construction process, and the established three-dimensional environment map may be defective and invisible during the virtual experience. First, the semantic segmentation is utilized to classify the point cloud groups according to the image of color camera, and then the point clouds of different objects are repaired according to the category of semantic segmentation. Since the classification results of point clouds is not high in some objects, it is improved by clustering the position and classified label of point clouds. Then, the partition planes such as the wall surface, the ground and the ceiling are reconstructed first. The plane reconstruction of the partition mainly solves the unevenness in the planes due to the depth error of the depth camera. For the reconstruction of furniture objects, the original point cloud of the furniture and the point cloud of complete object models will be matched to select the appropriate model point cloud to replace the original point cloud of the furniture such that the problem of the original point cloud missing or invisible during SLAM can be improved. After that, the density of the point cloud is increased by upsampling to produce a better reconstruction effect and to avoid the occurrence of virtual reality display delay due to excessive point clouds. Finally, the triangular mesh reconstruction is used to convert the map form of point cloud into a form of surface, thereby improving the detail of the map. Through the Unity engine, the reconstructed environment map is displayed on the virtual helmet, allowing the user to enjoy the virtually experience of the real scene in any locations.

碩士
國立雲林科技大學
資訊工程系
106
Since autonomous robots are often used for navigating unknown or dangerous environments, multiple sensing devices are required to enable autonomous robots to plan the collision-free path to the destination and to record the trajectories. In this thesis, an autonomous car mounted with LiDAR, GPS, gyroscopes, and camera is proposed for navigation, collision avoidance and path planning. The proposed system is a four-wheel-drive electric scooter carrier which is designed as four independent drive axles of the motor and a joints between front and back of the chassis. 　　In this paper, point clouds captured by LiDAR is used as obstacle detection. Firstly, point clouds are reduced to 2D dimensions by Voxel algorithm and then the reduced points are clustered by flood-fill grouping algorithm into several objects which are considered as obstacles. The bug algorithm adopted as path planning algorithm plans a local path to destination to avoid those obstacles. The GPS and gyroscopes are used for locating the robot position and identifying its orientation. 　　The experimental results show that the implemented autonomous car can reach the target position safely. Since the proposed pre-processing method can reduce the amount of point cloud and can improve the efficiency of obstacle clustering significantly, the proposed system is practicable in the field.

Graph convolutional networks (GCNs) showed promising results in learning from point cloud data. Applications of GCNs include point cloud classi cation, point cloud segmentation, point cloud upsampling, and more. Recently, the introduction of Deep Graph Convolutional Networks (DeepGCNs) allowed GCNs to go deeper, and thus resulted in better graph learning while avoiding the vanishing gradient problem in GCNs. By adapting impactful methods from convolutional neural networks (CNNs) such as residual connections, dense connections, and dilated convolutions, DeepGCNs allowed GCNs to learn better from non-Euclidean data. In addition, deep learning methods proved very e ective in the task of point cloud upsampling. Unlike traditional optimization-based methods, deep learning-based methods to point cloud upsampling does not rely on priors nor hand-crafted features to learn how to upsample point clouds. In this thesis, I discuss the impact and show the performance results of DeepGCNs in the task of point cloud part segmentation on PartNet dataset. I also illustrate the signi cance of using GCNs as upsampling modules in the task of point cloud upsampling by introducing two novel upsampling modules: Multi-branch GCN and Clone GCN. I show quantitatively and qualitatively the performance results of our novel and versatile upsampling modules when evaluated on a new proposed standardized dataset: PU600, which is the largest and most diverse point cloud upsampling dataset currently in the literature.

As buildings have almost come to a saturation point in most developed countries, the management and maintenance of existing buildings have become the major problem of the field. Building Information Modeling (BIM) is the underlying technology to solve this problem. It is a 3D semantic representation of building construction and facilities that contributes to not only the design phase but also the construction and maintenance phases, such as life-cycle management and building energy performance measurement. This study aims at the processes of creating as-built BIM models, which are constructed after the design phase. Point cloud, a set of points in 3D space, is an intermediate product of as-built BIM models that is often acquired by 3D laser scanning and photogrammetry. A raw point cloud typically requires further procedures, e.g. registration, segmentation, classification, etc. In terms of segmentation and classification, machine learning methodologies are trending due to the enhanced speed of computation. However, supervised machine learning methodologies require labelling the training point clouds in advance, which is time-consuming and often leads to inevitable errors. And due to the complexity and uncertainty of real-world environments, the attributes of one point vary from the attributes of others. These situations make it difficult to analyze how one single attribute contributes to the result of segmentation and classification. This study developed a method of producing point clouds from a fast-generating 3D virtual indoor environment using procedural modeling. This research focused on two attributes of simulated point clouds, point density and the level of random errors. According to Silverman (1986), point density is associated with the point features around each output raster cell. The number of points within a neighborhood divided the area of the neighborhood is the point density. However, in this study, there was a little different. The point density was defined as the number of points on a surface divided by the surface area. And the unit is points per square meters (pts/m²). This research compared the performances of a machine learning segmentation and classification algorithm on ten different point cloud datasets. The mean loss and accuracy of segmentation and classification were analyzed and evaluated to show how the point density and level of random errors affect the performance of the segmentation and classification models. Moreover, the real-world point cloud data were used as additional data to evaluate the applicability of produced models.

Large scale semantic segmentation is considered as one of the fundamental tasks in 3D scene understanding. Point clouds provide a basic and rich geometric rep- resentation of scenes and tangible objects. Convolutional Neural Networks (CNNs) have demonstrated an impressive success in processing regular discrete data such as 2D images and 1D audio. However, CNNs do not directly generalize to point cloud processing due to their irregular and un-ordered nature. One way to extend CNNs to point cloud understanding is to derive an intermediate euclidean representation of a point cloud by projecting onto image domain, voxelizing, or treating points as vertices of an un-directed graph. Graph-CNNs (GCNs) have demonstrated to be a very promising solution for deep learning on irregular data such as social networks, bi- ological systems, and recently point clouds. Early works in literature for graph based point networks relied on constructing dynamic graphs in the node feature space to define a convolution kernel. Later works constructed hierarchical static graphs in 3D space for an encoder-decoder framework inspired from image segmentation. This thesis takes a closer look at both dynamic and static graph neighborhoods of graph- based point networks for the task of semantic segmentation in order to: 1) discuss a potential cause for why going deep in dynamic GCNs does not necessarily lead to an improved performance, and 2) propose a new approach in treating points in a static graph neighborhood for an improved information aggregation. The proposed method leads to an efficient graph based 3D semantic segmentation network that is on par with current state-of-the-art methods on both indoor and outdoor scene semantic segmentation benchmarks such as S3DIS and Semantic3D.

碩士
國立臺灣大學
資訊工程學研究所
107
The deep learning technology has brought great success in image classification, object detection and semantic segmentation tasks. Recent years, the advent of inexpensive depth sensors hugely motivate 3D research area and real scene reconstruction datasets such as ScanNet [5] and Matterport3D [1] have been proposed. However, the problem of 3D scene semantic segmentation still remains new and challenging due to many variance of 3D data type (e.g. image, voxel, point cloud). Other difficulties such as suffering from high computation cost and the scarcity of data dispel the research progress of 3D segmentation. In this paper, we study 3D indoor scene segmentation problem with three different types of 3D data, which we categorize into image-based, voxel-based and point-based. We experiment on different input signals (e.g. color, depth, normal) and verify their effectiveness and performance in different data type networks. We further study fusion methods and improve the performance by using off-the-shelf deep models and by leveraging data modalities in the paper.

Dissertação de Mestrado Integrado em Engenharia Electrotécnica e de Computadores apresentada à Faculdade de Ciências e Tecnologia
Nuvens de pontos são conjuntos de pontos que representam um objeto ou cena 3D, em que os pontos são representados pelas respetivas coordenadas 3D e atributos opcionais tais como cor, reflectância, entre outros. Estas são usadas em várias áreas de aplicação como, por exemplo, entretenimento, representação de terrenos, imagens médicas e, mais recentemente, sistemas de condução autónoma de veículos. No entanto, devido ao grande volume de dados necessários para representar as nuvens de pontos, essas aplicações iriam precisar de um grande poder de processamento e, em alguns casos, poderia não ser possível realizar as tarefas em tempo real. Portanto, a compressão é usada para combater o problema de armazenamento e transmissão em tempo real. Sabe-se que a compressão com perdas introduz distorções geométricas que, geralmente, dependem do grau de compressão. Dado que em algumas aplicações é necessário segmentar os objectos que compõem a nuvem de pontos reconstruída/descomprimida, é importante perceber e caracterizar o efeito do tipo e grau de compressão na performance das tarefas de segmentação e classificação.Nesta dissertação, são descritos dois tipos de experiências: uma com nuvens de pontos de uso geral e a outra usando um caso particular das nuvens de pontos, mais precisamente, o LiDAR. Esta divisão foi feita, pois é provável que os resultados destas duas experiências sejam diferentes devido às aplicações distintas destas classes de nuvens de pontos, assim como respetivos requisitos de precisão. Estas experiências foram criadas para avaliar empiricamente o efeito de diferentes métodos de compressão de nuvens de pontos usando diferentes graus de compressão na performance de vários algoritmos de segmentação e classificação. Para isso, várias medidas de performance são usadas para avaliar o comportamento de cada caso.
Point clouds are sets of points which represent a 3D object/scene represented by their coordinates and optional attributes such as color, reflectance or other. Point clouds are being used in several application areas such as entertainment, terrain representation, medical imaging and, more recently, autonomous vehicle guidance systems. Due to the large size of point clouds, these applications would require a huge power processing and, in some cases, tasks may not be able to be performed in real time. Thus, compression is used to tackle the challenges of storage and real-time transmission. It is known that lossy compression introduces geometric distortions to the point clouds which are usually dependent on the compression rate. Therefore, in some cases, it is necessary to segment the component objects of the reconstructed/decompressed point cloud, it is important to understand and characterize the effect of the type and degree of compression on the performance of the segmentation and classification tasks. In this dissertation, two sets of experiments are described: one with general use point clouds and the other using a particular type of point clouds, more precisely LiDAR. This division was made because it is likely that the results are different for these two types due to the amount of precision and uses of each point cloud type. These experiments are designed to empirically evaluate the effect of different point cloud compression methods, employed at different compression rates, on the performance of several point cloud segmentation and classification algorithms. For that, several performance measures are used to evaluate the behavior of each case.

The goal of this work was to design fast and simple methods for processing point-cloud-data of urban areas for virtual reality applications. For the visualization of methods, we developed a simple renderer written in C++ and HLSL. The renderer is based on DirectX 11. For point-cloud processing, we designed a method based on height-histograms for filtering ground points out of point cloud. We also proposed a parallel method for point cloud segmentation based on the region growing algorithm. The individual segments are then tested by simple rules to check if it is or it is not corresponding to a predefined object.

3D point cloud processing has been a critical task due to the increasing demand of a variety of applications such as urban planning and management, as-built mapping of industrial sites, infrastructure monitoring, and road safety inspection. Point clouds are mainly acquired from two sources, laser scanning and optical imaging systems. However, the original point clouds usually do not provide explicit semantic information, and the collected data needs to undergo a sequence of processing steps to derive and extract the required information. Moreover, according to application requirements, the outcomes from the point cloud processing could be different. This dissertation presents two tiers of data processing. The first tier proposes an adaptive data processing framework to deal with multi-source and multi-platform point clouds. The second tier introduces two point clouds processing strategies targeting applications mainly from urban environments and transportation corridors.

Virtual reconstruction of historic sites, planning of restorations and attachments of new building parts, as well as forest inventory are few examples of fields that benefit from the application of 3D surveying data. Originally using 2D photo based documentation and manual distance measurements, the 3D information obtained from multi camera and laser scanning systems realizes a noticeable improvement regarding the surveying times and the amount of generated 3D information. The 3D data allows a detailed post processing and better visualization of all relevant spatial information. Yet, for the extraction of the required information from the raw scan data and for the generation of useable visual output, time-consuming, complex user-based data processing is still required, using the commercially available 3D software tools. In this context, the automatic object recognition from 3D point cloud and depth data has been discussed in many different works. The developed tools and methods however, usually only focus on a certain kind of object or the detection of learned invariant surface shapes. Although the resulting methods are applicable for certain practices of data segmentation, they are not necessarily suitable for arbitrary tasks due to the varying requirements of the different fields of research. This thesis presents a more widespread solution for automatic scene reconstruction from 3D point clouds, targeting street scenarios, specifically for the task of traffic accident scene analysis and documentation. The data, obtained by sampling the scene using a mobile scanning system is evaluated, segmented, and finally used to generate detailed 3D information of the scanned environment. To realize this aim, this work adapts and validates various existing approaches on laser scan segmentation regarding the application on accident relevant scene information, including road surfaces and markings, vehicles, walls, trees and other salient objects. The approaches are therefore evaluated regarding their suitability and limitations for the given tasks, as well as for possibilities concerning the combined application together with other procedures. The obtained knowledge is used for the development of new algorithms and procedures to allow a satisfying segmentation and reconstruction of the scene, corresponding to the available sampling densities and precisions. Besides the segmentation of the point cloud data, this thesis presents different visualization and reconstruction methods to achieve a wider range of possible applications of the developed system for data export and utilization in different third party software tools.

Dissertação de Mestrado Integrado em Engenharia Electrotécnica e de Computadores apresentada à Faculdade de Ciências e Tecnologia
O constante desenvolvimento de sistemas robóticos autónomos tem aumentando o interesse em utilizar os robôs como alternativa ao ser humano no desempenho de tarefas repetitivas, árduas, e perigosas.Face a uma alta densidade florestal existente em Portugal, mas também noutros países da Europa e de outros continentes, a necessidade de diminuir a matéria inflamável existente na floresta tornou-se um dos grandes objetivos da prevenção de grandes incêndios florestais.Os desenvolvimentos na área da robótica permitem aos robôs mapear os ambientes florestais de modo a obter informação útil que permita percecionar qual a matéria inflamável existente. A necessidade de perceber qual a vegetação que o robô deve ou não cortar, torna-se uma tarefa muito importante para o desempenho do robô. Esta dissertação está focada na perceção do ambiente que rodeia o robô, ou seja, perceber quais os objetos que rodeiam o robô, quais são obstáculos, qual a vegetação a cortar e não cortar.São propostas soluções usando LiDAR ou usando uma câmara RGB. Em relação ao LiDAR as soluções implementadas têm como base a altura dos objetos, a reflexão do “laser” do LiDAR conforme a superfície do objeto e também o tamanho dos objetos. Enquanto usando a câmara RGB a solução passa pelo uso de índices de vegetação e segmentação.As soluções foram validadas usando data ‘sets’ e fotografias de ambiente real. No final foi possível classificar os objetos como obstáculos, neste caso carros, paredes e troncos, mas também vegetação cortada através de um trator equipado com uma capinadeira e vegetação não cortada.
The constant development of autonomous robotic systems has open up the interest in using robots as an alternative to humans in the performance of repetitive, arduous, and dangerous tasks.Given the high forest density in Portugal, but also in other countries in Europe and other continents, the need to reduce the inflammable matter in the forest has become one of the major goals in the prevention of large forest fires.Developments in robotics allow robots to map forest environments in order to obtain useful information to understand the existing inflammable matter.The need to understand which vegetation the robot should or should not cut becomes a very important task for the robot performance.This dissertation is focused on the perception of the environment that surrounds the robot, that is, to understand which objects surround the robot, which are obstacles, which vegetation to cut and not to cut.Solutions are proposed using LiDAR or using an RGB camera. Regarding LiDAR the solutions implemented are based on the height of the objects, the reflection of the LiDAR laser according to the surface of the object and also the size of the objects. While using the RGB camera the solution goes through the use of vegetation indexes and segmentation. The solutions were validated using data sets and real environment photographs. In the end it was possible to classify the objects as obstacles, in this case cars, walls and trunks, but also vegetation cut by a tractor equipped with a clearing machine and uncut vegetation.

Recent advances in Light Detection and Ranging (LIDAR) technology and integration have resulted in vehicle-borne platforms for urban LIDAR scanning, such as Terrapoint Inc.'s TITAN system. Such technology has lead to an explosion in ground LIDAR data. The large size of such mobile urban LIDAR data sets, and the ease at which they may now be collected, has shifted the bottleneck of creating abstract urban models for Geographical Information Systems (GIS) from data collection to data processing. While turning such data into useful models has traditionally relied on human analysis, this is no longer practical. This thesis outlines a methodology for automatically recovering the necessary information to create abstract urban models from mobile urban LIDAR data using computer vision methods. As an integral part of the methodology, a novel scale-based interest operator is introduced (Di erence of Normals) that is e cient enough to process large datasets, while accurately isolating objects of interest in the scene according to real-world parameters. Finally a novel localized object recognition algorithm is introduced (Local Potential Well Space Embedding), derived from a proven global method for object recognition (Potential Well Space Embedding). The object recognition phase of our methodology is discussed with these two algorithms as a focus.
Thesis (Master, Computing) -- Queen's University, 2010-03-01 12:26:34.698