Academic literature on the topic 'Semantic coupling of task and motion planning'

Solving problems combining task and motion planning requires searching across a symbolic search space and a geometric search space. Because of the semantic gap between symbolic and geometric representations, symbolic sequences of actions are not guaranteed to be geometrically feasible. This compels us to search in the combined search space, in which frequent backtracks between symbolic and geometric levels make the search inefficient. We address this problem by guiding symbolic search with rich information extracted from the geometric level through culprit detection mechanisms.

Task and motion planning (TAMP) is a key research field for robotic manipulation tasks. The goal of TAMP is to generate motion-feasible task plan automatically. Existing methods for checking motion feasibility of task plan skeletons have some limitations of semantic-free pose candidate sampling, weak search heuristics, and early value commitment. In order to overcome these limitations, we propose a novel constraint satisfaction framework for checking motion feasibility of task plan skeletons. Our framework provides (1) a semantic pose candidate sampling method, (2) novel variable and constraint ordering heuristics based on intra- and inter-action dependencies in a task plan skeleton, and (3) an efficient search strategy using constraint propagation. Based upon these techniques, our framework can improve the efficiency of motion feasibility checking for TAMP. From experiments using the humanoid robot PR2, we show that the motion feasibility checking in our framework is 1.4x to 6.0x faster than previous ones.

Large language models have found utility in the domain of robot task planning and task decomposition. Nevertheless, the direct application of these models for instructing robots in task execution is not without its challenges. Limitations arise in handling more intricate tasks, encountering difficulties in effective interaction with the environment, and facing constraints in the practical executability of machine control instructions directly generated by such models. In response to these challenges, this research advocates for the implementation of a multi-layer large language model to augment a robot’s proficiency in handling complex tasks. The proposed model facilitates a meticulous layer-by-layer decomposition of tasks through the integration of multiple large language models, with the overarching goal of enhancing the accuracy of task planning. Within the task decomposition process, a visual language model is introduced as a sensor for environment perception. The outcomes of this perception process are subsequently assimilated into the large language model, thereby amalgamating the task objectives with environmental information. This integration, in turn, results in the generation of robot motion planning tailored to the specific characteristics of the current environment. Furthermore, to enhance the executability of task planning outputs from the large language model, a semantic alignment method is introduced. This method aligns task planning descriptions with the functional requirements of robot motion, thereby refining the overall compatibility and coherence of the generated instructions. To validate the efficacy of the proposed approach, an experimental platform is established utilizing an intelligent unmanned vehicle. This platform serves as a means to empirically verify the proficiency of the multi-layer large language model in addressing the intricate challenges associated with both robot task planning and execution.

The Continuum Robot has a multiredundant dof structure, which is extremely advantageous in the unstructured environment, and can complete such tasks as aircraft fuel tank inspection. However, due to its complex kinematics and coupling of joint motion, its motion path planning is also a challenging task. In this paper, a path planning method for Continuum Robot based on an equal curvature model in an aircraft fuel tank environment is proposed. Considering the complexity of calculation and the structural characteristics of Continuum Robot, a feasible obstacle avoidance discrete path is obtained by using the improved RRT algorithm. Then, joint fitting is performed on the existing discrete path according to the kinematic model of Continuum Robot, joint obstacle avoidance was conducted in the process of fitting, and finally, a motion path suitable for the Continuum Robot was selected. A reasonable experiment is designed based on MATLAB, and simulation and analysis results demonstrate excellent performance of this method and feasibility of path planning.

Observation planning for Unmanned Aerial Vehicles (UAVs) is a challenging task as it requires planning trajectories over a large continuous space and with motion models that can not be directly encoded into current planners. Furthermore, realistic problems often require complex objective functions that complicate problem decomposition. In this paper, we propose a local search approach to plan the trajectories of a fleet of UAVs on an observation mission. The strength of the approach lies in its loose coupling with domain specific requirements such as the UAV model or the objective function that are both used as black boxes. Furthermore, the Variable Neighborhood Search (VNS) procedure considered facilitates the adaptation of the algorithm to specific requirements through the addition of new neighborhoods. We demonstrate the feasibility and convenience of the method on a large joint observation task in which a fleet of fixed-wing UAVs maps wildfires over areas of a hundred square kilometers. The approach allows generating plans over tens of minutes for a handful of UAVs in matter of seconds, even when considering very short primitive maneuvers.

For the free-floating space manipulator with free-swinging joint failure, motions among its active joints, passive joints, free-floating base, and end-effector are coupled. It is significant to make clear all motion coupling relationships, which are defined as “kinematic coupling relationships” and “dynamic coupling relationships,” inside the system. With the help of conservation of system momentum, the kinematic model is established, and velocity mapping relation between active joints and passive joints, velocity mapping relation between active joints and base, velocity mapping relation between active joints and end-effector. We establish the dynamic model based on the Lagrange equation, and the system inertia matrix is partitioned according to the distribution of active joints, passive joints, and the base. Then, kinematic and dynamic coupling relationships are explicitly derived, and coupling indexes are defined to depict coupling degree. Motions of a space manipulator with free-swinging joint failure simultaneously satisfy the first-order nonholonomic constraint (kinematic coupling relationships) and the second-order nonholonomic constraint (dynamic coupling relationships), and the manipulator can perform tasks through motion planning and control. Finally, simulation experiments are carried out to verify the existence and correctness of the first-order and second-order nonholonomic constraints and display task execution effects of the space manipulator. This research analyzes the kinematic and dynamic characteristics of the free-floating space manipulator with free-swinging joint failure for the first time. It is the theoretical basis of free-swinging joint failure treatment for a space manipulator.

Planning a sequence of two motor elements is much more than concatenating two independent movements. However, very little is known about the cognitive strategies that are used to perform fluent sequences for intentional object manipulation. In this series of studies, the participants’ task was to reach for and pick to place a wooden cylinder to set it on a place pad of three different diameters, which served to modify terminal accuracy constraints. Participants were required to perform the sequences (1) at their preferred speed or (2) as fast as possible. Action kinematics were recorded with the Qualisys motion-capture system in order to implement a real-time protocol to get participants to engage in a true interactive relation. Results revealed that with low internal constraints (at preferred speed), low coupling between the two elements of the motor sequence was observed, suggesting a step-by-step planning strategy. Under high constraints (at fastest speed), an important terminal accuracy effect back propagated to modify early kinematic parameters of the first element, suggesting strong coupling of the parameters in an encapsulated planning strategy. In Studies 2 and 3, we further manipulated instructions and timing constraints to confirm the importance of time and predictability of external information for coupled planning. These findings overall sustain the hypothesis that coupled planning can take place in a pick and place task when anticipatory strategies are possible. This mode of action planning may be the key reason why motor intention can be read through the observation of micro variations in body kinematics.

The task of 3D reconstruction of urban targets holds pivotal importance for various applications, including autonomous driving, digital twin technology, and urban planning and development. The intricate nature of urban landscapes presents substantial challenges in attaining 3D reconstructions with high precision. In this paper, we propose a semantically aware multi-view 3D reconstruction method for urban applications which incorporates semantic information into the technical 3D reconstruction. Our research primarily focuses on two major components: sparse reconstruction and dense reconstruction. For the sparse reconstruction process, we present a semantic consistency-based error filtering approach for feature matching. To address the challenge of errors introduced by the presence of numerous dynamic objects in an urban scene, which affects the Structure-from-Motion (SfM) process, we propose a computation strategy based on dynamic–static separation to effectively eliminate mismatches. For the dense reconstruction process, we present a semantic-based Semi-Global Matching (sSGM) method. This method leverages semantic consistency to assess depth continuity, thereby enhancing the cost function during depth estimation. The improved sSGM method not only significantly enhances the accuracy of reconstructing the edges of the targets but also yields a dense point cloud containing semantic information. Through validation using architectural datasets, the proposed method was found to increase the reconstruction accuracy by 32.79% compared to the original SGM, and by 63.06% compared to the PatchMatch method. Therefore, the proposed reconstruction method holds significant potential in urban applications.

Human activities in maintenance of industrial plants pose elevated risks as well as significant costs due to the required shutdowns of the facility. An aerial robotic system with two arms for long reach manipulation in cluttered environments is presented to alleviate these constraints. The system consists of a multirotor with a long bar extension that incorporates a lightweight dual arm in the tip. This configuration allows aerial manipulation tasks even in hard-to-reach places. The objective of this work is the development of planning strategies to move the aerial robotic system with two arms for long reach manipulation in a safe and efficient way for both navigation and manipulation tasks. The motion planning problem is addressed considering jointly the aerial platform and the dual arm in order to achieve wider operating conditions. Since there exists a strong dynamical coupling between the multirotor and the dual arm, safety in obstacle avoidance will be assured by introducing dynamics awareness in the operation of the planner. On the other hand, the limited maneuverability of the system emphasizes the importance of energy and time efficiency in the generated trajectories. Accordingly, an adapted version of the optimal Rapidly-exploring Random Tree algorithm has been employed to guarantee their optimality. The resulting motion planning strategy has been evaluated through simulation in two realistic industrial scenarios, a riveting application and a chimney repairing task. To this end, the dynamics of the aerial robotic system with two arms for long reach manipulation has been properly modeled, and a distributed control scheme has been derived to complete the test bed. The satisfactory results of the simulations are presented as a first validation of the proposed approach.

Pour rester compétitifs, les industriels doivent réduire de plus en plus les coûts et les temps de développement de leurs nouveaux produits. Ceux-ci sont aujourd'hui de plus en plus intégrés, plus petits, plus légers et moins gourmands en énergie. Ils sont plus difficiles à concevoir et doivent être assemblés, maintenus et désassemblés sous de très fortes contraintes géométriques. Traditionnellement, en phase de conception, on établit le modèle CAO du produit, puis on fabrique les différentes parties physiques de celui-ci pour s’apercevoir trop souvent ensuite, que tout ou partie des tâches associées au cycle de vie du produit sont difficiles ou impossibles à réaliser. Une détection tardive de ces problèmes nécessite alors de remettre en cause la conception du produit. Les travaux de cette thèse s'intéressent à valider, dès la phase de conception et par simulation numérique avant la fabrication des prototypes physiques, l'ensemble des tâches associées au PLM, ce qui permettrait de réduire les temps et coûts de développement et de viser des procédés de fabrication plus respectueux de l’environnement en réduisant le nombre de prototypes physiques fabriqués. Une étape clef dans la validation par simulation des tâches du PLM consiste à trouver une trajectoire réalisable et sans collision afin de prouver leur faisabilité. La communauté robotique a, depuis les années 80, mis en oeuvre des méthodes de planification automatiques de trajectoires pour résoudre cette problématique. Toutefois, ces méthodes ont des limites, principalement liées à la complexité des modèles de l'environnement, traditionnellement purement géométriques. Dans des environnements très complexes, les planificateurs de trajectoires peuvent proposer des trajectoires peu pertinentes, dans des temps pouvant être très longs, voire échouer. Pour répondre à ces limites, des travaux ont considéré des approches collaboratives homme - planificateur mais qui ne permettent que rarement une interaction continue. Par ailleurs, les techniques de RV permettent la simulation avec un opérateur humain dans la boucle, en immersion dans l’environnement virtuel et en interaction avec celui-ci. Une approche originale liant planification automatique de trajectoires et RV a ainsi été développée au LGP permettant de profiter de la puissance de calcul des ordinateurs et des capacités cognitives d'un opérateur humain. Toutefois, dans cette approche, l'assistance proposée à l'opérateur n'est pas orientée vers le métier et la tâche à réaliser. Pour pouvoir raisonner au niveau de la tâche à réaliser il faut considérer conjointement planification de tâches et planification de trajectoires et s’intéresser à la capacité de modéliser des informations relatives à cette tâche et de raisonner sur celles-ci ; les ontologies sont un outil prometteur. L'objectif de cette thèse concerne l'élaboration d'une méthodologie pour le couplage sémantique des planificateurs de trajectoires et de tâches pour l’assistance à la manipulation en RV ou la robotique. Dans ce cadre, nous proposons deux contributions principales : La première contribution de ce travail propose deux ontologies originales. La première, ENVOn-2, concerne la modélisation de l'environnement dans lequel se déroule une tâche. La seconde, TAMPO, est une ontologie développée pour le planification conjointe de tâches et de trajectoires. La seconde contribution porte sur l'élaboration d'une méthodologie pour le couplage sémantique des planificateurs de tâches et de trajectoires. Cette méthodologie, par l'utilisation conjointe des deux ontologies, permet d'améliorer la planification de trajectoires d'une action primitive tout en proposant un plan (ou des plans) de tâche (s) pertinent(s) pour la manipulation effectuée. Ces développements ont ensuite été validés à l'aide de scénarios variés et de complexités croissantes. Les résultats obtenus montrent la pertinence de l'approche
To remain competitive, manufacturers need to reduce the costs and development times of their new products. Current products are increasingly integrated, smaller, lighter and more energyefficient. They are more difficult to design and have to be assembled, maintained and disassembled under very high geometric constraints. Traditionally, during the design phase, the CAD model of the product is established, then the physical parts of the product are manufactured, to discover all too often that some or all of the tasks associated with the product's life cycle are difficult or impossible to carry out. If these problems are detected too late, the product design has to be reconsidered. The aim of this thesis is to validate, at the design stage, all the tasks associated to the PLM using digital simulation before the physical prototypes are manufactured. This would make it possible to reduce development times and costs and to aim for more environmentally-friendly manufacturing processes by reducing the number of physical prototypes manufactured. A key step in the simulation-based validation of PLM tasks is to find a feasible collision-free trajectory in order to prove their feasibility. Since the 1980s, the robotics community has been using automatic path planning methods to solve this problem. However, these methods have limitations, mainly linked to the complexity of the environment models, which are traditionally purely geometric. In very complex environments, path planners can propose trajectories that are not very relevant, in times that can be very long, or even fail. To overcome these limitations, some works has considered collaborative human-planner approaches, but these rarely enable continuous interaction. On the other hand, VR techniques allow simulation with a human operator in the loop, immersed in the virtual environment and interacting with it. An original approach linking automatic path planning and VR has been developed at LGP, taking advantage of the computing power of computers and the cognitive abilities of a human operator. However, in this approach, the assistance offered to the operator is not oriented towards the task to be carried out. In order to be able to reason at the level of the task to be carried out, task planning and path planning must be considered together, and attention must be paid to the ability to model information relating to the task and to reason about these information; ontologies are a promising tool. The aim of this thesis is to develop a common framework for the semantic coupling of path and task planners for manipulation assistance in VR or robotics. Within this framework, we propose two main contributions: The first contribution of this work proposes two original ontologies. The first, ENVOn-2, concerns the modelling of the environment in which a manipulation task takes place. The second, TAMPO, is an ontology developed for jointly use path and task planning. The second contribution concerns the development of a methodology for the semantic coupling of task and trajectory planners. This methodology, through the joint use of the two ontologies, makes it possible to improve the path planning of a primitive action while proposing a task plan (or plans) that is (are) relevant to the manipulation being carried out. These developments were then validated using a variety of scenarios of increasing complexity. The results obtained demonstrate the relevance of the approach

The development of autonomous vehicles requires extensive testing of software modules. Developing a reliable software platform which allows testing on a real vehicle is yet a challenging task. Therefore, open-source software platforms are becoming more important for researchers in the field of autonomous driving. For example, Baidu provides the open-source autonomous driving platform Apollo which aims to accelerate testing and deployment of autonomous vehicles. However, the complex software structure hinders an easy integration of developed software modules, especially the motion planning module. Moreover, Baidu's Apollo provides only one possibility to test one's own algorithms in simulation, namely to upload the algorithm in Baidu's cloud platform, which is unacceptable for most autonomous driving companies. In contrast, the open-source CommonRoad benchmark suites contain diverse testing scenarios, e.g., highway, urban, dense traffic, and interaction with bicyclists and pedestrians. In addition, CommonRoad provides a motion planning framework in Python which enables rapid planner prototyping, along with additional tools, e.g., efficient collision checker, map format converter, and interface with the traffic simulator SUMO. In this work, we introduce a Python API between the planning module of the Baidu Apollo platform and the CommonRoad software framework. The developed interface aims to bridge the gap between rapid prototyping for safe planning algorithms and real-time test drives. The API transfers perception and map information to the planner and then returns the planned trajectory. The users can either replace the Apollo planner with their own planner or integrate their planner as a fail-safe planner if the planned trajectory by Apollo is unsafe. With our interface, developers can first test their planners in diverse scenarios from the CommonRoad benchmark, and directly on a real vehicle afterwards using the Apollo platform. The latter can be performed without adapting their algorithms to Apollo software structures. Moreover, developers can record their test drives in CommonRoad format for offline analyses. We demonstrate our interface in several scenarios with increasing complexity.

MOVEit, a widely used Robot Operating System framework, plans composite tasks, where the high-level sequence of actions is fixed and known in advance. However, these tasks need to be tailored and adapted to the environmental context. This framework uses custom trajectory planners, known as controllers, to solve goals that are fully defined within the configuration space. Libraries, such as the Open Motion Planning Library, provide a collection of motion planners that can solve task-space goals. An exact spatial and joint replication of the robotic manipulator’s mechanics, typically Universal Robot Description Format and Semantic Robot Description Format files, is required. Common arms such as the Panda-Manipulator and OpenMANIPULATOR-X provide these files in their respective public repositories, but custom arms require significant modification or even a complete rewrite of these files.