Dissertations / Theses on the topic 'Neural Sequence Models'

La thèse propose une approche de l'apprentissage temporel par des mécanismes d'auto-organisation à grain fin. Le manuscrit situe dans un premier temps le travail dans la perspective de contribuer à promouvoir une informatique cellulaire. Il s'agit d'une informatique où les calculs se répartissent en un grand nombre de calculs élémentaires, exécutés en parallèle, échangeant de l'information entre eux. Le caractère cellulaire tient à ce qu'en plus d’être à grain fin, une telle architecture assure que les connexions entre calculateurs respectent une topologie spatiale, en accord avec les contraintes des évolutions technologiques futures des matériels. Dans le manuscrit, la plupart des architectures informatiques distribuées sont examinées suivant cette perspective, pour conclure que peu d'entre elles relèvent strictement du paradigme cellulaire.Nous nous sommes intéressé à la capacité d'apprentissage de ces architectures, du fait de l'importance de cette notion dans le domaine connexe des réseaux de neurones par exemple, sans oublier toutefois que les systèmes cellulaires sont par construction des systèmes complexes dynamiques. Cette composante dynamique incontournable a motivé notre focalisation sur l'apprentissage temporel, dont nous avons passé en revue les déclinaisons dans les domaines des réseaux de neurones supervisés et des cartes auto-organisatrices.Nous avons finalement proposé une architecture qui contribue à la promotion du calcul cellulaire en ce sens qu'elle exhibe des propriétés d'auto-organisation pour l'extraction de la représentation des états du système dynamique qui lui fournit ses entrées, même si ces dernières sont ambiguës et ne reflètent que partiellement cet état. Du fait de la présence d'un cluster pour nos simulations, nous avons pu mettre en œuvre une architecture complexe, et voir émerger des phénomènes nouveaux. Sur la base de ces résultats, nous développons une critique qui ouvre des perspectives sur la suite à donner à nos travaux
The thesis proposes a sequence learning approach that uses the mechanism of fine grain self-organization. The manuscript initially starts by situating this effort in the perspective of contributing to the promotion of cellular computing paradigm in computer science. Computation within this paradigm is divided into a large number of elementary calculations carried out in parallel by computing cells, with information exchange between them.In addition to their fine grain nature, the cellular nature of such architectures lies in the spatial topology of the connections between cells that complies with to the constraints of the technological evolution of hardware in the future. In the manuscript, most of the distributed architecture known in computer science are examined following this perspective, to find that very few of them fall within the cellular paradigm.We are interested in the learning capacity of these architectures, because of the importance of this notion in the related domain of neural networks for example, without forgetting, however, that cellular systems are complex dynamical systems by construction.This inevitable dynamical component has motivated our focus on the learning of temporal sequences, for which we reviewed the different models in the domains of neural networks and self-organization maps.At the end, we proposed an architecture that contributes to the promotion of cellular computing in the sense that it exhibits self-organization properties employed in the extraction of a representation of a dynamical system states that provides the architecture with its entries, even if the latter are ambiguous such that they partially reflect the system state. We profited from an existing supercomputer to simulate complex architecture, that indeed exhibited a new emergent behavior. Based on these results we pursued a critical study that sets the perspective for future work

Data-driven sequence models have long played a role in the analysis and generation of musical information. Such models are of interest in computational musicology, computer-aided music composition, and tools for music education among other applications. This dissertation beginswith an experiment tomodel sequences of musical pitch in melodies with a class of purely data-driven predictive models collectively known as Connectionist models. It was demonstrated that a set of six such models could performon par with, or better than state-of-the-art n-gram models previously evaluated in an identical setting. A new model known as the Recurrent Temporal Discriminative Restricted Boltzmann Machine (RTDRBM), was introduced in the process and found to outperform the rest of the models. A generalisation of this modelling task was also explored, and involved extending the set of musical features used as input by the models while still predicting pitch as before. The improvement in predictive performance which resulted from adding these new input features is encouraging for future work in this direction. Based on the above success of the RTDRBM, its application was extended to a non-musical sequence labelling task, namely Optical Character Recognition. This extension involved a modification to the model’s original prediction algorithm as a result of relaxing an assumption specific to the melody modelling task. The generalised model was evaluated on a benchmark dataset and compared against a set of 8 baseline models where it faired better than all of them. Furthermore, a theoretical extension to an existingmodel which was also employed in the above pitch prediction task - the Discriminative Restricted Boltzmann Machine (DRBM) - was proposed. This led to three new variants of the DRBM (which originally contained Logistic Sigmoid hidden layer activations), withHyperbolic Tangent, Binomial and Rectified Linear hidden layer activations respectively. The first two of these have been evaluated here on the benchmark MNIST dataset and shown to perform on par with the original DRBM.

Negation is a complex phenomenon present in all human languages, allowing for the uniquely human capacities of denial, contradiction, misrepresentation, lying, and irony. It is in the first place a phenomenon of semantical opposition. Sentences containing negation are generally (a) less informative than affirmative ones, (b) morphosyntactically more marked—all languages have negative markers while only a few have affirmative markers, and (c) psychologically more complex and harder to process. Negation often conveys positive meaning. This meaning ranges from implicatures to entailments. In this dissertation, I develop a system to reveal the underlying positive interpretation of negation. I first identify which words are intended to be negated (i.e, the focus of negation) and second, I rewrite those tokens to generate an actual positive interpretation. I identify the focus of negation by scoring probable foci along a continuous scale. One of the obstacles to exploring foci scoring is that no public datasets exist for this task. Thus, to study this problem I create new corpora. The corpora contain verbal, nominal and adjectival negations and their potential positive interpretations along with their scores ranging from 1 to 5. Then, I use supervised learning models for scoring the focus of negation. In order to rewrite the focus of negation with its positive interpretation, I work with negations from Simple Wikipedia, automatically generate potential positive interpretations, and then collect manual annotations that effectively rewrite the negation in positive terms. This procedure yields positive interpretations for approximately 77% of negations, and the final corpus includes over 5,700 negations and over 5,900 positive interpretations. I then use sequence-to-sequence neural models and provide baseline results.

Denna avhandling i datalogi föreslår modeller för hur vissa beräkningsmässiga uppgifter kan utföras av hjärnbarken. Utgångspunkten är dels kända fakta om hur en area i hjärnbarken är uppbyggd och fungerar, dels etablerade modellklasser inom beräkningsneurobiologi, såsom attraktorminnen och system för gles kodning. Ett neuralt nätverk som producerar en effektiv gles kod i binär mening för sensoriska, särskilt visuella, intryck presenteras. Jag visar att detta nätverk, när det har tränats med naturliga bilder, reproducerar vissa egenskaper (receptiva fält) hos nervceller i lager IV i den primära synbarken och att de koder som det producerar är lämpliga för lagring i associativa minnesmodeller. Vidare visar jag hur ett enkelt autoassociativt minne kan modifieras till att fungera som ett generellt sekvenslärande system genom att utrustas med synapsdynamik. Jag undersöker hur ett abstrakt attraktorminnessystem kan implementeras i en detaljerad modell baserad på data om hjärnbarken. Denna modell kan sedan analyseras med verktyg som simulerar experiment som kan utföras på en riktig hjärnbark. Hypotesen att hjärnbarken till avsevärd del fungerar som ett attraktorminne undersöks och visar sig leda till prediktioner för dess kopplingsstruktur. Jag diskuterar också metodologiska aspekter på beräkningsneurobiologin idag.
In this thesis I take a modular approach to cortical function. I investigate how the cerebral cortex may realise a number of basic computational tasks, within the framework of its generic architecture. I present novel mechanisms for certain assumed computational capabilities of the cerebral cortex, building on the established notions of attractor memory and sparse coding. A sparse binary coding network for generating efficient representations of sensory input is presented. It is demonstrated that this network model well reproduces the simple cell receptive field shapes seen in the primary visual cortex and that its representations are efficient with respect to storage in associative memory. I show how an autoassociative memory, augmented with dynamical synapses, can function as a general sequence learning network. I demonstrate how an abstract attractor memory system may be realised on the microcircuit level -- and how it may be analysed using tools similar to those used experimentally. I outline some predictions from the hypothesis that the macroscopic connectivity of the cortex is optimised for attractor memory function. I also discuss methodological aspects of modelling in computational neuroscience.
QC 20100916

Supplying the right amount of taxis in the right place at the right time is very important for taxi companies. In this paper, the machine learning model Taxi Demand Net (TDNet) is presented which predicts short-term taxi demand in different zones of a city. It is based on WaveNet which is a causal dilated convolutional neural net for time-series generation. TDNet uses historical demand from the last years and transforms features such as time of day, day of week and day of month into 26-hour taxi demand forecasts for all zones in a city. It has been applied to one city in northern Europe and one in South America. In northern europe, an error of one taxi or less per hour per zone was achieved in 64% of the cases, in South America the number was 40%. In both cities, it beat the SARIMA and stacked ensemble benchmarks. This performance has been achieved by tuning the hyperparameters with a Bayesian optimization algorithm. Additionally, weather and holiday features were added as input features in the northern European city and they did not improve the accuracy of TDNet.

The Neutral Indel Model is able to predict accurately the distribution of indel events in alignments of neutrally evolving genomic sequence. Here, I apply this model to a diverse range of metazoan species pairs, to a number of ends. First, I apply the Neutral Indel Model to alignments of genome sequences for species within the mammalian clade in order to estimate the quantities of functional DNA shared between species pairs. I demonstrate that as the evolutionary divergence between species pairs increases, estimates of functional sequence drop off dramatically. This pattern is not replicated in extensive simulations of genome sequence alignments, suggesting that functional (and mostly non-coding) sequence is turning over at a rapid rate. I also estimate that between 200 and 300 Mb (6.5-10%) of the human genome is under evolutionary constraint, a considerably higher quantity of sequence than has been estimated by previous whole genome analyses. Second, extending my analyses to consider more diverse metazoan species, I provide estimates for functional bases within organisms’ genomes that appear to mirror our conceptions of organismal complexity. Thirdly, I develop the Neutral Indel Model as a method for assessing genome sequence quality, by quantifying indel errors within alignments of closely related (ds < 0.1) species pairs. Applying this method to six primate genome sequence assemblies, I demonstrate that the frequency of indel error events per base varies up to six-fold. Further to this, I show that second generation sequencing technologies can be used to create high quality genome sequence assemblies and to ameliorate errors in pre-existing assemblies. Finally, I analyse patterns of indel mutations in primate transposable elements and show that indels are not randomly distributed within these sequences due to regularly spaced homo-nucleotide motifs.

The goal of this thesis is to describe and build a system for neural machine translation. System is built with recurrent neural networks - encoder-decoder architecture in particular. The result is a nmt library used to conduct experiments with different model parameters. Results of the experiments are compared with system built with the statistical tool Moses.

[EN] This work is focused on problems (like automatic speech recognition (ASR) and handwritten text recognition (HTR)) that: 1) can be represented (at least approximately) in terms of one-dimensional sequences, and 2) solving these problems entails breaking the observed sequence down into segments which are associated to units taken from a finite repertoire. The required segmentation and classification tasks are so intrinsically interrelated ("Sayre's Paradox") that they have to be performed jointly. We have been inspired by what some works call the "successful trilogy", which refers to the synergistic improvements obtained when considering: - a good formalization framework and powerful algorithms; - a clever design and implementation taking the best profit of hardware; - an adequate preprocessing and a careful tuning of all heuristics. We describe and study "two stage generative models" (TSGMs) comprising two stacked probabilistic generative stages without reordering. This model not only includes Hidden Markov Models (HMMs, but also "segmental models" (SMs). "Two stage decoders" may be deduced by simply running a TSGM in reversed way, introducing non determinism when required: 1) A directed acyclic graph (DAG) is generated and 2) it is used together with a language model (LM). One-pass decoders constitute a particular case. A formalization of parsing and decoding in terms of semiring values and language equations proposes the use of recurrent transition networks (RTNs) as a normal form for Context Free Grammars (CFGs), using them in a parsing-as-composition paradigm, so that parsing CFGs result in a slight extension of regular ones. Novel transducer composition algorithms have been proposed that can work with RTNs and can deal with null transitions without resorting to filter-composition even in the presence of null transitions and non-idempotent semirings. A review of LMs is described and some contributions mainly focused on LM interfaces, LM representation and on the evaluation of Neural Network LMs (NNLMs) are provided. A review of SMs includes the combination of generative and discriminative segmental models and general scheme of frame emission and another one of SMs. Some fast cache-friendly specialized Viterbi lexicon decoders taking profit of particular HMM topologies are proposed. They are able to manage sets of active states without requiring dictionary look-ups (e.g. hashing). A dataflow architecture allowing the design of flexible and diverse recognition systems from a little repertoire of components has been proposed, including a novel DAG serialization protocol. DAG generators can take over-segmentation constraints into account, make use SMs other than HMMs, take profit of the specialized decoders proposed in this work and use a transducer model to control its behavior making it possible, for instance, to use context dependent units. Relating DAG decoders, they take profit of a general LM interface that can be extended to deal with RTNs. Some improvements for one pass decoders are proposed by combining the specialized lexicon decoders and the "bunch" extension of the LM interface, including an adequate parallelization. The experimental part is mainly focused on HTR tasks on different input modalities (offline, bimodal). We have proposed some novel preprocessing techniques for offline HTR which replace classical geometrical heuristics and make use of automatic learning techniques (neural networks). Experiments conducted on the IAM database using this new preprocessing and HMM hybridized with Multilayer Perceptrons (MLPs) have obtained some of the best results reported for this reference database. Among other HTR experiments described in this work, we have used over-segmentation information, tried lexicon free approaches, performed bimodal experiments and experimented with the combination of hybrid HMMs with holistic classifiers.
[ES] Este trabajo se centra en problemas (como reconocimiento automático del habla (ASR) o de escritura manuscrita (HTR)) que cumplen: 1) pueden representarse (quizás aproximadamente) en términos de secuencias unidimensionales, 2) su resolución implica descomponer la secuencia en segmentos que se pueden clasificar en un conjunto finito de unidades. Las tareas de segmentación y de clasificación necesarias están tan intrínsecamente interrelacionadas ("paradoja de Sayre") que deben realizarse conjuntamente. Nos hemos inspirado en lo que algunos autores denominan "La trilogía exitosa", refereido a la sinergia obtenida cuando se tiene: - un buen formalismo, que dé lugar a buenos algoritmos; - un diseño e implementación ingeniosos y eficientes, que saquen provecho de las características del hardware; - no descuidar el "saber hacer" de la tarea, un buen preproceso y el ajuste adecuado de los diversos parámetros. Describimos y estudiamos "modelos generativos en dos etapas" sin reordenamientos (TSGMs), que incluyen no sólo los modelos ocultos de Markov (HMM), sino también modelos segmentales (SMs). Se puede obtener un decodificador de "dos pasos" considerando a la inversa un TSGM introduciendo no determinismo: 1) se genera un grafo acíclico dirigido (DAG) y 2) se utiliza conjuntamente con un modelo de lenguaje (LM). El decodificador de "un paso" es un caso particular. Se formaliza el proceso de decodificación con ecuaciones de lenguajes y semianillos, se propone el uso de redes de transición recurrente (RTNs) como forma normal de gramáticas de contexto libre (CFGs) y se utiliza el paradigma de análisis por composición de manera que el análisis de CFGs resulta una extensión del análisis de FSA. Se proponen algoritmos de composición de transductores que permite el uso de RTNs y que no necesita recurrir a composición de filtros incluso en presencia de transiciones nulas y semianillos no idempotentes. Se propone una extensa revisión de LMs y algunas contribuciones relacionadas con su interfaz, con su representación y con la evaluación de LMs basados en redes neuronales (NNLMs). Se ha realizado una revisión de SMs que incluye SMs basados en combinación de modelos generativos y discriminativos, así como un esquema general de tipos de emisión de tramas y de SMs. Se proponen versiones especializadas del algoritmo de Viterbi para modelos de léxico y que manipulan estados activos sin recurrir a estructuras de tipo diccionario, sacando provecho de la caché. Se ha propuesto una arquitectura "dataflow" para obtener reconocedores a partir de un pequeño conjunto de piezas básicas con un protocolo de serialización de DAGs. Describimos generadores de DAGs que pueden tener en cuenta restricciones sobre la segmentación, utilizar modelos segmentales no limitados a HMMs, hacer uso de los decodificadores especializados propuestos en este trabajo y utilizar un transductor de control que permite el uso de unidades dependientes del contexto. Los decodificadores de DAGs hacen uso de un interfaz bastante general de LMs que ha sido extendido para permitir el uso de RTNs. Se proponen también mejoras para reconocedores "un paso" basados en algoritmos especializados para léxicos y en la interfaz de LMs en modo "bunch", así como su paralelización. La parte experimental está centrada en HTR en diversas modalidades de adquisición (offline, bimodal). Hemos propuesto técnicas novedosas para el preproceso de escritura que evita el uso de heurísticos geométricos. En su lugar, utiliza redes neuronales. Se ha probado con HMMs hibridados con redes neuronales consiguiendo, para la base de datos IAM, algunos de los mejores resultados publicados. También podemos mencionar el uso de información de sobre-segmentación, aproximaciones sin restricción de un léxico, experimentos con datos bimodales o la combinación de HMMs híbridos con reconocedores de tipo holístico.
[CAT] Aquest treball es centra en problemes (com el reconeiximent automàtic de la parla (ASR) o de l'escriptura manuscrita (HTR)) on: 1) les dades es poden representar (almenys aproximadament) mitjançant seqüències unidimensionals, 2) cal descompondre la seqüència en segments que poden pertanyer a un nombre finit de tipus. Sovint, ambdues tasques es relacionen de manera tan estreta que resulta impossible separar-les ("paradoxa de Sayre") i s'han de realitzar de manera conjunta. Ens hem inspirat pel que alguns autors anomenen "trilogia exitosa", referit a la sinèrgia obtinguda quan prenim en compte: - un bon formalisme, que done lloc a bons algorismes; - un diseny i una implementació eficients, amb ingeni, que facen bon us de les particularitats del maquinari; - no perdre de vista el "saber fer", emprar un preprocés adequat i fer bon us dels diversos paràmetres. Descrivim i estudiem "models generatiu amb dues etapes" sense reordenaments (TSGMs), que inclouen no sols inclouen els models ocults de Markov (HMM), sinò també models segmentals (SM). Es pot obtindre un decodificador "en dues etapes" considerant a l'inrevés un TSGM introduint no determinisme: 1) es genera un graf acíclic dirigit (DAG) que 2) és emprat conjuntament amb un model de llenguatge (LM). El decodificador "d'un pas" en és un cas particular. Descrivim i formalitzem del procés de decodificació basada en equacions de llenguatges i en semianells. Proposem emprar xarxes de transició recurrent (RTNs) com forma normal de gramàtiques incontextuals (CFGs) i s'empra el paradigma d'anàlisi sintàctic mitjançant composició de manera que l'anàlisi de CFGs resulta una lleugera extensió de l'anàlisi de FSA. Es proposen algorismes de composició de transductors que poden emprar RTNs i que no necessiten recorrer a la composició amb filtres fins i tot amb transicions nul.les i semianells no idempotents. Es proposa una extensa revisió de LMs i algunes contribucions relacionades amb la seva interfície, amb la seva representació i amb l'avaluació de LMs basats en xarxes neuronals (NNLMs). S'ha realitzat una revisió de SMs que inclou SMs basats en la combinació de models generatius i discriminatius, així com un esquema general de tipus d'emissió de trames i altre de SMs. Es proposen versions especialitzades de l'algorisme de Viterbi per a models de lèxic que permeten emprar estats actius sense haver de recórrer a estructures de dades de tipus diccionari, i que trauen profit de la caché. S'ha proposat una arquitectura de flux de dades o "dataflow" per obtindre diversos reconeixedors a partir d'un xicotet conjunt de peces amb un protocol de serialització de DAGs. Descrivim generadors de DAGs capaços de tindre en compte restriccions sobre la segmentació, emprar models segmentals no limitats a HMMs, fer us dels decodificadors especialitzats proposats en aquest treball i emprar un transductor de control que permet emprar unitats dependents del contexte. Els decodificadors de DAGs fan us d'una interfície de LMs prou general que ha segut extesa per permetre l'ús de RTNs. Es proposen millores per a reconeixedors de tipus "un pas" basats en els algorismes especialitzats per a lèxics i en la interfície de LMs en mode "bunch", així com la seua paral.lelització. La part experimental està centrada en el reconeiximent d'escriptura en diverses modalitats d'adquisició (offline, bimodal). Proposem un preprocés d'escriptura manuscrita evitant l'us d'heurístics geomètrics, en el seu lloc emprem xarxes neuronals. S'han emprat HMMs hibridats amb xarxes neuronals aconseguint, per a la base de dades IAM, alguns dels millors resultats publicats. També podem mencionar l'ús d'informació de sobre-segmentació, aproximacions sense restricció a un lèxic, experiments amb dades bimodals o la combinació de HMMs híbrids amb classificadors holístics.
España Boquera, S. (2016). Contributions to the joint segmentation and classification of sequences (My two cents on decoding and handwriting recognition) [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/62215
TESIS
Premiado

Il seguente progetto è volto ad implementare sul microcontroller "Raspberry Pi v.4 Model B" un software utilizzante a mo' di scatola nera, anche, in parte, una rete neurale che sulla base di una classificazione precedentemente realizzata da soggetti terzi e sulla base di opportuni modelli preaddestrati sfrutti un meccanismo di apprendimento supervisionato per stimare ragionevolmente, secondo opportuni criteri, il sesso, la fascia d'età, lo stato emotivo (caricaturale, ossia forzato) e la distanza approssimativa di uno o più soggetti ripresi frontalmente in volto da una telecamera in tempo reale.

This master thesis deals with the implementation of various types of recurrent neural networks via programming language lua using torch library. It focuses on finding optimal strategy for training recurrent neural networks and also tries to minimize the duration of the training. Furthermore various types of regularization techniques are investigated and implemented into the recurrent neural network architecture. Implemented recurrent neural networks are compared on the speech recognition task using AMI dataset, where they model the acustic information. Their performance is also compared to standard feedforward neural network. Best results are achieved using BLSTM architecture. The recurrent neural network are also trained via CTC objective function on the TIMIT dataset. Best result is again achieved using BLSTM architecture.

碩士
國立臺北科技大學
工業工程與管理研究所
96
This research attempts to use probabilistic neural networks(PNN) and binary sequence algorithm(BSA) to build financial prediction models, regard listed company as the research object, take three annual financial materials of company. The main purpose to build this financial prediction models, lie in finding the potential financial crisis inside enterprises ahead of time, offer investors and electronic industry one to consult alert news by this. This research is divided into two stages and built the model, the first stage is to use two kinds of data type and four kinds of period to build financial classification model, elect the best model to produce the classifying value, The second stage is to rise from these classifying value prediction pattern through BSA, predicting via these prediction patterns. Looked by the real example result, with appropriate prediction pattern, can offer better prediction result.

The advent of efficient genome sequencing tools and high-throughput experimental biotechnology has lead to an enormous progress in life sciences. Among the most important innovations is the microarray technology. It allows to quantify the expression of thousands of genes simultaneously by measuring the hybridization from a tissue of interest to probes on a small glass or plastic slide. Before launching into microarray research it is important to recall that the characteristics of this data include a fair amount of noise and an atypical dimensionality (which makes difficult the use of classic statistics tools – experimental samples in the order of dozens and measured parameters in thousands or tens of thousands). Therefore, the main goal of this thesis is the development of adequate computational methods and algorithms, capable of extracting valuable biological knowledge from this type of data. Applications of microarray technology as a tool for gene expression analysis range from the assignment of functional categories for genes of unknown biological function (based on the analysis of genes with already established biological role), to precise and early diagnosis of different tumor malignancies. However, the main goal of computational analysis of gene expression data is the extraction of regulatory knowledge at genetic level that may be used to provide a broader understanding on the functioning of complex cellular systems. In this direction, revealing the structures of regulatory networks based of gene expression data becomes a pivotal task. The thesis contributes with a framework for the discovery of biological functional category of genes based on the synergy of ICA and a dynamic SOM-based clustering algorithm, that accurately finds groups of co-regulated genes, while identifying interesting regulatory signals within the data with the help of ICA decomposition. We also pursue the task of molecular characterization of different tumor types using gene expression profiling, by providing a novel method for tissue samples classification, based on an ensemble of classifiers sequentially trained on reweighted versions of the data. The algorithm, known as boosting, is adapted to peculiarities of gene expression data and employed in conjunction with SVMs. Additionally, the novel concept of finding predictive genes whose signatures are significant for phenotype discrimination is treated. Finally, the thesis presents a method developed for reverse-engineering gene regulatory networks based on recurrent neuro-fuzzy networks, which exploits the advantages of fuzzy-based models, in terms of results interpretability, and those of neural systems, in terms of computational power and time series prediction capabilities.
H έλευση ικανών υπολογιστικών εργαλείων για την μελέτη της γενομικής ακολουθίας και της ερευνητικής βιοτεχνολογίας υψηλής ανάλυσης, οδήγησε σε μια τεράστια πρόοδο στις επιστήμες ζωής. Μεταξύ των πιο σημαντικών καινοτομιών είναι η τεχνολογία μικροσυστοιχιών. H τεχνολογία αυτή επιτρέπει την ποσοτικοποίηση της έκφρασης χιλιάδων γονιδίων ταυτόχρονα, μετρώντας τον υβριδισμό από έναν ιστό ενδιαφέροντος έως σε δείγματα σε μικρό γυαλί η σε πλαστικά τσιπ. Πριν ξεκινήσουμε την έρευνα πάνω στις μικροσυστοιχίες είναι σημαντικό να θυμόμαστε ότι τα χαρακτηριστικά των δεδομένων αυτής περιλαμβάνουν αρκετό ποσό θορύβου και ένα μη τυπικό αριθμό διαστάσεων (το οποίο καθιστά δύσκολη την χρήση κλασσικών στατιστικών μεθόδων – μέγεθος δείγματος σε δωδεκάδες και μέγεθος χαρακτηριστικών σε χιλιάδες η δεκάδες η εκατοντάδες). Επομένως, ο κύριος στόχος αυτής της διδακτορικής εργασίας είναι η ανάπτυξη ικανών υπολογιστικών μεθόδων και αλγόριθμων έτσι ώστε να εξάγουν πολύτιμη βιολογική γνώση από τον συγκεκριμένο τύπο δεδομένων. Εφαρμογές της τεχνολογίας μικροσυστοιχιών σαν ένα εργαλείο για την ανάλυση έκφρασης γονιδίων ξεκινούν από την εύρεση και απόδοση λειτουργικών κατηγοριών για γονίδια άγνωστης βιολογικής λειτουργικότητας (βασισμένη στην ανάλυση των γονιδίων ήδη εδραιωμένου βιολογικού ρόλου) έως την ακριβή και πρώιμη διάγνωση διαφορετικών κακοήθων όγκων. Όμως ο κύριος στόχος της υπολογιστικής ανάλυσης της έκφρασης γονιδίων είναι η εξαγωγή ρυθμιζόμενης γνώσης στο γενετικό επίπεδο το οποίο μπορεί να χρησιμοποιηθεί ώστε να παρέχει μία ευρύτερη κατανόηση της λειτουργίας πολύπλοκων κυτταρικών συστημάτων. Σε αυτή την κατεύθυνση, το να αναδεικνύεις τις δομές ρυθμιστικών δικτύων βασισμένων στην έκφραση γονιδίων γίνεται καίριο έργο. Η διδακτορική διατριβή συνεισφέρει στο πλαίσιο για την ανακάλυψη βιολογικά λειτουργικών κατηγοριών γονιδίων βασισμένη στην συνεργία της ΙCA και της δυναμικού βασισμένου στη SOM ομαδοποίηση αλγορίθμου η οποία με ακρίβεια βρίσκει ομάδες γονιδίων που συν-ρυθμίζονται ενώ παράλληλα αναγνωρίζει ενδιαφέροντα ρυθμιστικά σήματα μέσα στα δεδομένα με τη βοήθεια της ΙCA αποδόμησης. Eπίσης, προσανατολιζόμαστε στην εύρεση του μοριακού χαρακτηρισμού διαφορετικών τύπων όγκων χρησιμοποιώντας το προφίλ της γονιδιακής έκφρασης, βασισμένο σε ένα σύνολο κατηγοριοποιητών οι οποίοι εκπαιδεύτηκαν σειριακά σε επανασταθμισμένες παραλλαγές των δεδομένων. Ο αλγόριθμος, γνωστός και σαν boosting, έχει προσαρμοστεί στις ιδιαιτερότητες των δεδομένων έκφρασης γονιδίου και εφαρμόζεται σε συνδυασμό με τα SVMs. Επιπλέον, εξετάζεται η πρωτοποριακή τεχνική της εύρεσης προβλέψιμων τιμών των οποίων οι υπογραφές είναι σημαντικές για τον χαρακτηρισμό φαινότυπου. Τελικά, η παρούσα διδακτορική διατριβή παρουσιάζει μια μέθοδο που αναπτύχθηκε για αντίστροφα μηχανικά ελεγχόμενα από γονίδια νευρωνικά δίκτυα βασισμένα σε αναδρομικά νευρωνικά δίκτυα τύπου fuzzy, τα οποία αξιοποιούν τα πλεονεκτήματα των μοντέλων τύπου fuzzy σε βάση επεξηγηματικότητας αποτελεσμάτων, και αυτών των νευρωνικών δικτύων σε βάση υπολογιστικής δύναμης και ικανότητας πρόβλεψης χρονοσειρών.

A family of Natural Language Processing (NLP) tasks such as part-of- speech (PoS) tagging, Named Entity Recognition (NER), and Multiword Expression (MWE) identification all involve assigning labels to sequences of words in text (sequence labeling). Most modern machine learning approaches to sequence labeling utilize word embeddings, learned representations of text, in which words with similar meanings have similar representations. Quite recently, contextualized word embeddings have garnered much attention because, unlike pretrained context- insensitive embeddings such as word2vec, they are able to capture word meaning in context. In this thesis, I evaluate the performance of different embedding setups (context-sensitive, context-insensitive word, as well as task-specific word, character, lemma, and PoS) on the three abovementioned sequence labeling tasks using a deep learning model (BiLSTM) and Portuguese datasets. v

碩士
國立成功大學
資訊工程學系
106
With the accelerated development of artificial intelligence, some people want to use it to predict market trends. Simultaneously, digital currency, headed by Bitcoin and Ethereum, caught people’s attention because of its soaring price in last year. The reputation of digital currency get higher and higher in social media and traditional media. People certainly hope to use AI to predict the digital currency market. In this research, we use Twitter posts as training data and vectored method to represent the tweet information (day vector) per day. After cleaning Twitter raw data, we converted the tweets in the giving day as day vector and feed the day vector to sequence to Sequence model use to predict the change of Bitcoin price. The entire system uses attention model in day vector model and the sequence to sequence model, respectively. The experiments show that the prediction accuracy rise slightly by increasing day vector dimension and the attention model of the SequenceDecoder model can significantly improve the accuracy. Finally, we analyzed the 7-day predicted results individually and found that the accuracy decrease when predicting latter day. This meet our understanding that it is harder to predict the latter day than the near day.

Many complex actions are precomposed, by sequencing simpler motor actions. For such a complex action to be executed accurately, those simpler actions must be planned in the desired order, held in working memory, and then enacted one-by-one until the sequence is complete. Examples of this phenomenon include writing, typing, and speaking. Under most circumstances, the ability to learn and reproduce novel motor sequences is hindered when additional information is presented. However, in cases where the motor sequence is musical in nature (e.g. a choreographed dance or a piano melody), one must learn two sequences at the same time, one of motor actions and one of the time intervals between actions. Despite this added complexity, humans learn and perform rhythm-based motor sequences regularly. It has been shown that people can learn motoric and rhythmic sequences separately and then combine them with little trouble (Ullén & Bengtsson 2003). Also, functional MRI data suggest that there are distinct sets of neural regions responsible for the two different sequence types (Bengtsson et al. 2004). Although research on musical rhythm is extensive, few computational models exist to extend and inform our understanding of its neural bases. To that end, this dissertation introduces the TAMSIN (Timing And Motor System Integration Network) model, a systems-level neural network model designed to replicate rhythm-based motor sequence performance. TAMSIN utilizes separate Competitive Queuing (CQ) modules for motoric and temporal sequences, as well as modules designed to coordinate these sequence types into a cogent output performance consistent with a perceived beat and tempo. Chapters 1-4 explore prior literature on CQ architectures, rhythmic perception/production, and computational modeling, thereby illustrating the need for a model to tie those research areas together. Chapter 5 details the structure of the TAMSIN model and its mathematical specification. Chapter 6 presents and discusses the results of the model simulated under various circumstances. Chapter 7 compares the simulation results to behavioral and imaging results from the experimental literature. The final chapter discusses future modifications that could be made to TAMSIN to simulate aspects of rhythm learning, rhythm perception, and disordered productions, such as those seen in Parkinson’s disease.

Cette thèse étudie des modèles de séquences de haute dimension basés sur des réseaux de neurones récurrents (RNN) et leur application à la musique et à la parole. Bien qu'en principe les RNN puissent représenter les dépendances à long terme et la dynamique temporelle complexe propres aux séquences d'intérêt comme la vidéo, l'audio et la langue naturelle, ceux-ci n'ont pas été utilisés à leur plein potentiel depuis leur introduction par Rumelhart et al. (1986a) en raison de la difficulté de les entraîner efficacement par descente de gradient. Récemment, l'application fructueuse de l'optimisation Hessian-free et d'autres techniques d'entraînement avancées ont entraîné la recrudescence de leur utilisation dans plusieurs systèmes de l'état de l'art. Le travail de cette thèse prend part à ce développement. L'idée centrale consiste à exploiter la flexibilité des RNN pour apprendre une description probabiliste de séquences de symboles, c'est-à-dire une information de haut niveau associée aux signaux observés, qui en retour pourra servir d'à priori pour améliorer la précision de la recherche d'information. Par exemple, en modélisant l'évolution de groupes de notes dans la musique polyphonique, d'accords dans une progression harmonique, de phonèmes dans un énoncé oral ou encore de sources individuelles dans un mélange audio, nous pouvons améliorer significativement les méthodes de transcription polyphonique, de reconnaissance d'accords, de reconnaissance de la parole et de séparation de sources audio respectivement. L'application pratique de nos modèles à ces tâches est détaillée dans les quatre derniers articles présentés dans cette thèse. Dans le premier article, nous remplaçons la couche de sortie d'un RNN par des machines de Boltzmann restreintes conditionnelles pour décrire des distributions de sortie multimodales beaucoup plus riches. Dans le deuxième article, nous évaluons et proposons des méthodes avancées pour entraîner les RNN. Dans les quatre derniers articles, nous examinons différentes façons de combiner nos modèles symboliques à des réseaux profonds et à la factorisation matricielle non-négative, notamment par des produits d'experts, des architectures entrée/sortie et des cadres génératifs généralisant les modèles de Markov cachés. Nous proposons et analysons également des méthodes d'inférence efficaces pour ces modèles, telles la recherche vorace chronologique, la recherche en faisceau à haute dimension, la recherche en faisceau élagué et la descente de gradient. Finalement, nous abordons les questions de l'étiquette biaisée, du maître imposant, du lissage temporel, de la régularisation et du pré-entraînement.
This thesis studies models of high-dimensional sequences based on recurrent neural networks (RNNs) and their application to music and speech. While in principle RNNs can represent the long-term dependencies and complex temporal dynamics present in real-world sequences such as video, audio and natural language, they have not been used to their full potential since their introduction by Rumelhart et al. (1986a) due to the difficulty to train them efficiently by gradient-based optimization. In recent years, the successful application of Hessian-free optimization and other advanced training techniques motivated an increase of their use in many state-of-the-art systems. The work of this thesis is part of this development. The main idea is to exploit the power of RNNs to learn a probabilistic description of sequences of symbols, i.e. high-level information associated with observed signals, that in turn can be used as a prior to improve the accuracy of information retrieval. For example, by modeling the evolution of note patterns in polyphonic music, chords in a harmonic progression, phones in a spoken utterance, or individual sources in an audio mixture, we can improve significantly the accuracy of polyphonic transcription, chord recognition, speech recognition and audio source separation respectively. The practical application of our models to these tasks is detailed in the last four articles presented in this thesis. In the first article, we replace the output layer of an RNN with conditional restricted Boltzmann machines to describe much richer multimodal output distributions. In the second article, we review and develop advanced techniques to train RNNs. In the last four articles, we explore various ways to combine our symbolic models with deep networks and non-negative matrix factorization algorithms, namely using products of experts, input/output architectures, and generative frameworks that generalize hidden Markov models. We also propose and analyze efficient inference procedures for those models, such as greedy chronological search, high-dimensional beam search, dynamic programming-like pruned beam search and gradient descent. Finally, we explore issues such as label bias, teacher forcing, temporal smoothing, regularization and pre-training.

Yes
A confocal microscope provides a sequence of images, at incremental depths, of the various corneal layers and structures. From these, medical practioners can extract clinical information on the state of health of the patient's cornea. In this work we are addressing problems associated with capturing and processing these images including blurring, non-uniform illumination and noise, as well as the displacement of images laterally and in the anterior posterior direction caused by subject movement. The latter may cause some of the captured images to be out of sequence in terms of depth. In this paper we introduce automated algorithms for classification, reordering, registration and segmentation to solve these problems. The successful implementation of these algorithms could open the door for another interesting development, which is the 3D modelling of these sequences.

碩士
國立中興大學
資訊管理學系所
107
There are many types of single-gene disorders, which could affect a wide range of human bodies, include heart disease, metabolic abnormality, brain or neurological disorders, skin lesion, etc., and even lead to death. Nowadays, machine learning and deep learning techniques have been able to assist physicians in clinical diagnosis with objective and accurate advantages. In order to prevent diseases onset or from getting worse, these techniques could perform analysis of human genes and let patients to receive early treatment or adjust their habits of eating and living. NCBI GenBank database is applied to gather DNA sequences in this study. These sequences are transformed into global data and local data as inputs by multiple algorithms and tools. Convolutional Neural Networks, Naïve Bayes, Support Vector Machine, C4.5 algorithm and Random Forest are implemented to construct classification models of sequences of single-gene disorders. Performance of various models would be compared by validation indexes of confusion matrix. The experimental results show that when the global data is used as the input data, a higher classification effect could be obtained. Among all algorithms, Random Forest and Convolutional Neural Networks have the best performance with accuracy over 97%. Performances of other algorithms are sorted from best to worst in the following order: Naïve Bayes > C4.5 algorithm > Support Vector Machine. In the analysis of local data, the 10-second segmented audio signal images have the best classification effect in the Convolutional Neural Networks model with sensitivity 84.81%, F1 score 84.08%, MCC 82.64% and accuracy 84.28%. Multiple classification models of single-gene disorders are proposed in this study. The combination of algorithms and input data with best performance could be selected as a tool and direction for genetic disorders diagnosis and screening. This study expects that these classification models could assist physicians in clinical diagnosis of single-gene disorders and as a research basis of bioinformatics.