Littérature scientifique sur le sujet « Heterogeneous neural networks »

Aquest treball presenta una classe de funcions que serveixen de models neuronals generalitzats per ser usats en xarxes neuronals artificials. Es defineixen com una mesura de similitud que actúa com una definició flexible de neurona vista com un reconeixedor de patrons. <br/>La similitud proporciona una marc conceptual i serveix de cobertura unificadora de molts models neuronals de la literatura i d'exploració de noves instàncies de models de neurona.<br/> <br/>La visió basada en similitud porta amb naturalitat a integrar informació heterogènia, com ara quantitats contínues i discretes (nominals i ordinals), i difuses ó imprecises. Els valors perduts es tracten de manera explícita. <br/>Una neurona d'aquesta classe s'anomena neurona heterogènia i qualsevol arquitectura neuronal que en faci ús serà una Xarxa Neuronal Heterogènia.<br/>En aquest treball ens concentrem en xarxes neuronals endavant, com focus inicial d'estudi. Els algorismes d'aprenentatge són basats en algorisms evolutius, especialment extesos per treballar amb informació heterogènia. <br/><br/>En aquesta tesi es descriu com una certa classe de neurones heterogènies porten a xarxes neuronals que mostren un rendiment molt satisfactori, comparable o superior al de xarxes neuronals tradicionals (com el perceptró multicapa ó la xarxa de base radial), molt especialment en presència d'informació heterogènia, usual en les bases de dades actuals.<br>This work presents a class of functions serving as generalized neuron models to be used in artificial neural networks. They are cast into the common framework of computing a similarity function, a flexible definition of a neuron as a pattern recognizer. The similarity endows the model with a clear conceptual view and serves as a unification cover for many of the existing neural models, including those classically used for the MultiLayer Perceptron (MLP) and most of those used in Radial Basis Function Networks (RBF). These families of models are conceptually unified and their relation is clarified. <br/>The possibilities of deriving new instances are explored and several neuron models --representative of their families-- are proposed.<br/> <br/>The similarity view naturally leads to further extensions of the models to handle heterogeneous information, that is to say, information coming from sources radically different in character, including continuous and discrete (ordinal) numerical quantities, nominal (categorical) quantities, and fuzzy quantities. Missing data are also explicitly considered. A neuron of this class is called an heterogeneous neuron and any neural structure making use of them is an Heterogeneous Neural Network (HNN), regardless of the specific architecture or learning algorithm. Among them, in this work we concentrate on feed-forward networks, as the initial focus of study. The learning procedures may include a great variety of techniques, basically divided in derivative-based methods (such as the conjugate gradient)and evolutionary ones (such as variants of genetic algorithms).<br/><br/>In this Thesis we also explore a number of directions towards the construction of better neuron models --within an integrant envelope-- more adapted to the problems they are meant to solve.<br/>It is described how a certain generic class of heterogeneous models leads to a satisfactory performance, comparable, and often better, to that of classical neural models, especially in the presence of heterogeneous information, imprecise or incomplete data, in a wide range of domains, most of them corresponding to real-world problems.

Aquest treball presenta una classe de funcions que serveixen de models neuronals generalitzats per ser usats en xarxes neuronals artificials. Es defineixen com una mesura de similitud que actúa com una definició flexible de neurona vista com un reconeixedor de patrons. La similitud proporciona una marc conceptual i serveix de cobertura unificadora de molts models neuronals de la literatura i d'exploració de noves instàncies de models de neurona. La visió basada en similitud porta amb naturalitat a integrar informació heterogènia, com ara quantitats contínues i discretes (nominals i ordinals), i difuses ó imprecises. Els valors perduts es tracten de manera explícita. Una neurona d'aquesta classe s'anomena neurona heterogènia i qualsevol arquitectura neuronal que en faci ús serà una Xarxa Neuronal Heterogènia.En aquest treball ens concentrem en xarxes neuronals endavant, com focus inicial d'estudi. Els algorismes d'aprenentatge són basats en algorisms evolutius, especialment extesos per treballar amb informació heterogènia. En aquesta tesi es descriu com una certa classe de neurones heterogènies porten a xarxes neuronals que mostren un rendiment molt satisfactori, comparable o superior al de xarxes neuronals tradicionals (com el perceptró multicapa ó la xarxa de base radial), molt especialment en presència d'informació heterogènia, usual en les bases de dades actuals.<br>This work presents a class of functions serving as generalized neuron models to be used in artificial neural networks. They are cast into the common framework of computing a similarity function, a flexible definition of a neuron as a pattern recognizer. The similarity endows the model with a clear conceptual view and serves as a unification cover for many of the existing neural models, including those classically used for the MultiLayer Perceptron (MLP) and most of those used in Radial Basis Function Networks (RBF). These families of models are conceptually unified and their relation is clarified. The possibilities of deriving new instances are explored and several neuron models --representative of their families-- are proposed. The similarity view naturally leads to further extensions of the models to handle heterogeneous information, that is to say, information coming from sources radically different in character, including continuous and discrete (ordinal) numerical quantities, nominal (categorical) quantities, and fuzzy quantities. Missing data are also explicitly considered. A neuron of this class is called an heterogeneous neuron and any neural structure making use of them is an Heterogeneous Neural Network (HNN), regardless of the specific architecture or learning algorithm. Among them, in this work we concentrate on feed-forward networks, as the initial focus of study. The learning procedures may include a great variety of techniques, basically divided in derivative-based methods (such as the conjugate gradient)and evolutionary ones (such as variants of genetic algorithms).In this Thesis we also explore a number of directions towards the construction of better neuron models --within an integrant envelope-- more adapted to the problems they are meant to solve.It is described how a certain generic class of heterogeneous models leads to a satisfactory performance, comparable, and often better, to that of classical neural models, especially in the presence of heterogeneous information, imprecise or incomplete data, in a wide range of domains, most of them corresponding to real-world problems.

Cette thèse porte sur l'obtention rigoureuse de limites de champ moyen pour la dynamique continue de grands réseaux de neurones hétérogènes. Nous considérons des neurones à taux de décharge, et sujets à un bruit Brownien additif. Le réseau est entièrement connecté, avec des poids de connections dont la variance décroît comme l'inverse du nombre de neurones conservant un effet non trivial dans la limite thermodynamique. Un second type d'hétérogénéité, interprété comme une position spatiale, est considéré au niveau de chaque cellule. Pour la pertinence biologique, nos modèles incluent ou bien des délais, ainsi que des moyennes et variances de connections, dépendants de la distance entre les cellules, ou bien des synapses dépendantes de l'état des deux neurones post- et présynaptique. Ce dernier cas s'applique au modèle de Kuramoto pour les oscillateurs couplés. Quand les poids synaptiques sont Gaussiens et indépendants, nous prouvons un principe de grandes déviations pour la mesure empirique de l'état des neurones. La bonne fonction de taux associée atteint son minimum en une unique mesure de probabilité, impliquant convergence et propagation du chaos sous la loi "averaged". Dans certains cas, des résultats "quenched" sont obtenus. La limite est solution d'une équation implicite, non Markovienne, dans laquelle le terme d'interactions est remplacé par un processus Gaussien qui dépend de la loi de la solution du réseau entier. Une universalité de cette limite est prouvée, dans le cas de poids synaptiques non-Gaussiens avec queues sous-Gaussiennes. Enfin, quelques résultats numérique sur les réseau aléatoires sont présentés, et des perspectives discutées<br>This thesis addresses the rigorous derivation of mean-field results for the continuous time dynamics of heterogeneous large neural networks. In our models, we consider firing-rate neurons subject to additive noise. The network is fully connected, with highly random connectivity weights. Their variance scales as the inverse of the network size, and thus conserves a non-trivial role in the thermodynamic limit. Moreover, another heterogeneity is considered at the level of each neuron. It is interpreted as a spatial location. For biological relevance, a model considered includes delays, mean and variance of connections depending on the distance between cells. A second model considers interactions depending on the states of both neurons at play. This last case notably applies to Kuramoto's model of coupled oscillators. When the weights are independent Gaussian random variables, we show that the empirical measure of the neurons' states satisfies a large deviations principle, with a good rate function achieving its minimum at a unique probability measure, implying averaged convergence of the empirical measure and propagation of chaos. In certain cases, we also obtained quenched results. The limit is characterized through a complex non Markovian implicit equation in which the network interaction term is replaced by a non-local Gaussian process whose statistics depend on the solution over the whole neural field. We further demonstrate the universality of this limit, in the sense that neuronal networks with non-Gaussian interconnections but sub-Gaussian tails converge towards it. Moreover, we present a few numerical applications, and discuss possible perspectives

Federated learning allows the training of a model from the distributed data of many clients under the orchestration of a central server. With the increasing concern on privacy, federated learning draws great attention from both academia and industry. However, the heterogeneous challenges introduced by natural characters of federated learning settings significantly degrade the performance of federated learning methods. Specifically, these heterogeneous challenges include the heterogeneous data challenges and the heterogeneous scenario challenges. Data heterogeneous challenges mean the significant differences between the datasets of numerous users. In federated learning, the data is stored separately on many distanced clients, causing these challenges. In addition, the heterogeneous scenario challenges refer to the differences between the devices participating in federated learning. Furthermore, the suitable models vary among the different scenarios. However, many existing federated learning methods use a single global model for all the devices' scenarios, which is not optimal for these two challenges. We first propose a novel federated learning framework called local union in federated learning (LU-FL) to address these challenges. LU-FL incorporates the hierarchical knowledge distillation mechanism that effectively transfers knowledge among different models. So, LU-FL can enable any number of models to be used on each client. Allocating the specially designed models to different clients can mitigate the adverse effects caused by these challenges. At the same time, it can further improve the accuracy of the output models. Extensive experimental results over several popular datasets demonstrate the effectiveness of our proposed method. It can effectively reduce the harmful effects of heterogeneous challenges, improving the accuracy of the final output models and the adaptability of the clients to various scenarios. So, it lets federated learning methods be applied in more diverse scenarios. Keywords: federated learning, neural networks, knowledge distillation, computer vision

This thesis proposes a novel feature selection and classification method employing evolving spiking neural networks (eSNN) and evolutionary algorithms (EA). The method is named the Quantum-inspired Spiking Neural Network (QiSNN) framework. QiSNN represents an integrated wrapper approach. An evolutionary process evolves appropriate feature subsets for a given classification task and simultaneously optimises the neural and learning-related parameters of the network. Unlike other methods, the connection weights of this network are determined by a fast one-pass learning algorithm which dramatically reduces the training time. In its core, QiSNN employs the Thorpe neural model that allows the efficient simulation of even large networks. In QiSNN, the presence or absence of features is represented by a string of concatenated bits, while the parameters of the neural network are continuous. For the exploration of these two entirely different search spaces, a novel Estimation of Distribution Algorithm (EDA) is developed. The method maintains a population of probabilistic models specialised for the optimisation of either binary, continuous or heterogeneous search spaces while utilising a small and intuitive set of parameters. The EDA extends the Quantum-inspired Evolutionary Algorithm (QEA) proposed by Han and Kim (2002) and was named the Heterogeneous Hierarchical Model EDA (hHM-EDA). The algorithm is compared to numerous contemporary optimisation methods and studied in terms of convergence speed, solution quality and robustness in noisy search spaces. The thesis investigates the functioning and the characteristics of QiSNN using both synthetic feature selection benchmarks and a real-world case study on ecological modelling. By evolving suitable feature subsets, QiSNN significantly enhances the classification accuracy of eSNN. Compared to numerous other feature selection techniques, like the wrapper-based Multilayer Perceptron (MLP) and the Naive Bayesian Classifier (NBC), QiSNN demonstrates a competitive classification and feature selection performance while requiring comparatively low computational costs.

The process of data integration is an important part of the database field when it comes to database migrations and the merging of data. The research in the area has grown with the addition of machine learning approaches in the last 20 years. Due to the complexity of the research field, no go-to solutions have appeared. Instead, a wide variety of ways of enhancing database migrations have emerged. This thesis examines how well a learning-based solution performs for the semantic integration problem in database migrations. Two algorithms are implemented. One that is based on information retrieval theory, with the goal of yielding a matching result that can be used as a benchmark for measuring the performance of the machine learning algorithm. The machine learning approach is based on grouping data with agglomerative hierarchical clustering and then training a neural network to recognize patterns in the data. This allows making predictions about potential data correspondences across two databases. The results show that agglomerative hierarchical clustering performs well in the task of grouping the data into classes. The classes can in turn be used for training a neural network. The matching algorithm gives a high recall of matching tables, but improvements are needed to both receive a high recall and precision. The conclusion is that the proposed learning-based approach, using agglomerative hierarchical clustering and a neural network, works as a solid base to semi-automate the data integration problem seen in this thesis. But the solution needs to be enhanced with scenario specific algorithms and rules, to reach desired performance.<br>Dataintegrering är en viktig del inom området databaser när det kommer till databasmigreringar och sammanslagning av data. Forskning inom området har ökat i takt med att maskininlärning blivit ett attraktivt tillvägagångssätt under de senaste 20 åren. På grund av komplexiteten av forskningsområdet, har inga optimala lösningar hittats. Istället har flera olika tekniker framställts, som tillsammans kan förbättra databasmigreringar. Denna avhandling undersöker hur bra en lösning baserad på maskininlärning presterar för dataintegreringsproblemet vid databasmigreringar. Två algoritmer har implementerats. En är baserad på informationssökningsteori, som främst används för att ha en prestandamässig utgångspunkt för algoritmen som är baserad på maskininlärning. Den algoritmen består av ett första steg, där data grupperas med hjälp av hierarkisk klustring. Sedan tränas ett artificiellt neuronnät att hitta mönster i dessa grupperingar, för att kunna göra förutsägelser huruvida olika datainstanser har ett samband mellan två databaser. Resultatet visar att agglomerativ hierarkisk klustring presterar väl i uppgiften att klassificera den data som använts. Resultatet av matchningsalgoritmen visar på att en stor mängd av de matchande tabellerna kan hittas. Men förbättringar behöver göras för att både ge hög en hög återkallelse av matchningar och hög precision för de matchningar som hittas. Slutsatsen är att ett inlärningsbaserat tillvägagångssätt, i detta fall att använda agglomerativ hierarkisk klustring och sedan träna ett artificiellt neuronnät, fungerar bra som en basis för att till viss del automatisera ett dataintegreringsproblem likt det som presenterats i denna avhandling. För att få bättre resultat, krävs att lösningen förbättras med mer situationsspecifika algoritmer och regler.

The objective of this study was firstly to investigate and understand sorption properties of earlywood, latewood, annual rings and gross wood. Secondly, to develop a heterogeneous sorption model for earlywood, latewood and annual rings by taking into consideration unified complex interactions of anatomy, chemical composition and thermodynamic parameters. Thirdly, to upscale the annual ring level model to gross wood by applying artificial neural networks (ANNs) modeling tools using dimensionally reduced inputs through dimensional analysis and genetic algorithms. Four novel physical models, namely, dynamical two-level systems (TLS) model of annual rings, sorption kinetics, sorption isotherms and TLS model of physical properties and chemical composition were derived and successfully validated using experimental data of Douglas-fir. The annual ring’s TLS model was capable to generate novel physical quantities, namely, golden ring volume (GRV) and golden ring cube (GRC) to which the sorption properties are very sensitive, according to the validation tests. A new heterogeneity test criterion (HTC) was also derived. Validations of the TLS sorption models revealed new evidence showing a transient nature of sorption hysteresis in which boundary sorption isotherms asymptotically converged to a single isotherm at large time limit. A novel method for the computation of internal surface area of wood was also validated using the TLS model of sorption isotherms. The fibre saturation point prediction of the model was also found to agree well with earlier reports. The TLS model of physical properties and chemical composition was able to reveal the self-organization in Douglas-fir that gives rise to allometric scaling. The TLS modeling revealed existence of self-organizing criticality (SOC) in Douglas-fir and demonstrated mechanisms by which it is generated. Ten categories of unified ANNs Douglas-fir sorption models that predict equilibrium moisture content, diffusion and surface emission coefficients were successfully developed and validated. The network models predict sorption properties of Douglas-fir using thermodynamic variables and parameters generated by the four TLS models from chemical composition and physical properties of annual rings. The findings of this study contribute to the creation of a decision support system that would allow predicting wood properties and processing characteristics based on chemical and structural attributes.

L’objectiu d’aquesta tesi és l’extracció d’Informació de documents total o parcialment manuscrits amb una certa estructura. Bàsicament treballem amb dos escenaris d’aplicació diferent. El primer escenari són els documents moderns altament estructurats, com formularis. En aquests documents, la informació semàntica està ja definida en camps, amb una posició concreta al document i l’extracció de la informació és equivalent a una transcripció. El segon escenari son els documents semi-estructurats totalment manuscrits on, a més de transcriure, cal associar un valor semàntic, d’entre un conjunt conegut de valors possibles, a les paraules que es transcriuen. En ambdós casos la qualitat de la transcripció té un gran pes en la precisió del sistema, per això proposem models basats en xarxes neuronals per a transcriure text manuscrit. Per a poder afrontar el repte dels documents semi-estructurats hem generat un benchmark, compost de dataset, una sèrie de tasques definides i una mètrica que es va presentar a la comunitat científica com a una competició internacional. També proposem diferents models basats en Xarxes Neuronals Convolucionals i recurrents, capaços de transcriure i assignar diferent etiquetes semàntiques a cada paraula manuscrita, és a dir, capaços d'extreure informació.<br>El objetivo de esta tesis es la extracción de Información de documentos total o parcialmente manuscritos, con una cierta estructura. Básicamente trabajamos con dos escenarios de aplicación diferentes. El primer escenario son los documentos modernos altamente estructurados, como los formularios. En estos documentos, la información semántica está pre-definida en campos con una posición concreta en el documento i la extracción de información es equivalente a una transcripción. El segundo escenario son los documentos semi-estructurados totalmente manuscritos, donde, además de transcribir, es necesario asociar un valor semántico, de entre un conjunto conocido de valores posibles, a las palabras manuscritas. En ambos casos, la calidad de la transcripción tiene un gran peso en la precisión del sistema. Por ese motivo proponemos modelos basados en redes neuronales para transcribir el texto manuscrito. Para poder afrontar el reto de los documentos semi-estructurados, hemos generado un benchmark, compuesto de dataset, una serie de tareas y una métrica que fue presentado a la comunidad científica a modo de competición internacional. También proponemos diferentes modelos basados en Redes Neuronales Convolucionales y Recurrentes, capaces de transcribir y asignar diferentes etiquetas semánticas a cada palabra manuscrita, es decir, capaces de extraer información.<br>The goal of this thesis is information Extraction from totally or partially handwritten documents. Basically we are dealing with two different application scenarios. The first scenario are modern highly structured documents like forms. In this kind of documents, the semantic information is encoded in different fields with a pre-defined location in the document, therefore, information extraction becomes equivalent to transcription. The second application scenario are loosely structured totally handwritten documents, besides transcribing them, we need to assign a semantic label, from a set of known values to the handwritten words. In both scenarios, transcription is an important part of the information extraction. For that reason in this thesis we present two methods based on Neural Networks, to transcribe handwritten text.In order to tackle the challenge of loosely structured documents, we have produced a benchmark, consisting of a dataset, a defined set of tasks and a metric, that was presented to the community as an international competition. Also, we propose different models based on Convolutional and Recurrent neural networks that are able to transcribe and assign different semantic labels to each handwritten words, that is, able to perform Information Extraction.