Tesi sul tema "Data processing and Chemometrics"

One trend in the ‘omics’ sciences is the generation of increasing amounts of data, describing complex biological samples. To cope with this and facilitate progress towards reliable diagnostic tools, it is crucial to develop methods for extracting representative and predictive information. In global metabolite analysis (metabolomics and metabonomics) NMR, GC/MS and LC/MS are the main platforms for data generation. Multivariate projection methods (e.g. PCA, PLS and O-PLS) have been recognized as efficient tools for data analysis within subjects such as biology and chemistry due to their ability to provide interpretable models based on many, correlated variables. In global metabolite analysis, these methods have been successfully applied in areas such as toxicology, disease diagnosis and plant functional genomics. This thesis describes the development of processing methods for the unbiased extraction of representative and predictive information from metabolic GC/MS and LC/MS data characterizing biofluids, e.g. plant extracts, urine and blood plasma. In order to allow the multivariate projections to detect and highlight differences between samples, one requirement of the processing methods is that they must extract a common set of descriptors from all samples and still retain the metabolically relevant information in the data. In Papers I and II this was done by applying a hierarchical multivariate compression approach to both GC/MS and LC/MS data. In the study described in Paper III a hierarchical multivariate curve resolution strategy (H-MCR) was developed for simultaneously resolving multiple GC/MS samples into pure profiles. In Paper IV the H-MCR method was applied to a drug toxicity study in rats, where the method’s potential for biomarker detection and identification was exemplified. Finally, the H-MCR method was extended, as described in Paper V, allowing independent samples to be processed and predicted using a model based on an existing set of representative samples. The fact that these processing methods proved to be valid for predicting the properties of new independent samples indicates that it is now possible for global metabolite analysis to be extended beyond isolated studies. In addition, the results facilitate high through-put analysis, because predicting the nature of samples is rapid compared to the actual processing. In summary this research highlights the possibilities for using global metabolite analysis in diagnosis.

The present Ph.D. thesis, primarily conceived to support and reinforce the relation between academic and industrial worlds, was developed in collaboration with Shell Global Solutions (Amsterdam, The Netherlands) in the endeavour of applying and possibly extending well-established latent variable-based approaches (i.e. Principal Component Analysis - PCA - Partial Least Squares regression - PLS - or Partial Least Squares Discriminant Analysis - PLSDA) for complex problem solving not only in the fields of manufacturing troubleshooting and optimisation, but also in the wider environment of multivariate data analysis. To this end, novel efficient algorithmic solutions are proposed throughout all chapters to address very disparate tasks, from calibration transfer in spectroscopy to real-time modelling of streaming flows of data. The manuscript is divided into the following six parts, focused on various topics of interest: Part I - Preface, where an overview of this research work, its main aims and justification is given together with a brief introduction on PCA, PLS and PLSDA; Part II - On kernel-based extensions of PCA, PLS and PLSDA, where the potential of kernel techniques, possibly coupled to specific variants of the recently rediscovered pseudo-sample projection, formulated by the English statistician John C. Gower, is explored and their performance compared to that of more classical methodologies in four different applications scenarios: segmentation of Red-Green-Blue (RGB) images, discrimination of on-/off-specification batch runs, monitoring of batch processes and analysis of mixture designs of experiments; Part III - On the selection of the number of factors in PCA by permutation testing, where an extensive guideline on how to accomplish the selection of PCA components by permutation testing is provided through the comprehensive illustration of an original algorithmic procedure implemented for such a purpose; Part IV - On modelling common and distinctive sources of variability in multi-set data analysis, where several practical aspects of two-block common and distinctive component analysis (carried out by methods like Simultaneous Component Analysis - SCA - DIStinctive and COmmon Simultaneous Component Analysis - DISCO-SCA - Adapted Generalised Singular Value Decomposition - Adapted GSVD - ECO-POWER, Canonical Correlation Analysis - CCA - and 2-block Orthogonal Projections to Latent Structures - O2PLS) are discussed, a new computational strategy for determining the number of common factors underlying two data matrices sharing the same row- or column-dimension is described, and two innovative approaches for calibration transfer between near-infrared spectrometers are presented; Part V - On the on-the-fly processing and modelling of continuous high-dimensional data streams, where a novel software system for rational handling of multi-channel measurements recorded in real time, the On-The-Fly Processing (OTFP) tool, is designed; Part VI - Epilogue, where final conclusions are drawn, future perspectives are delineated, and annexes are included.
La presente tesis doctoral, concebida principalmente para apoyar y reforzar la relación entre la academia y la industria, se desarrolló en colaboración con Shell Global Solutions (Amsterdam, Países Bajos) en el esfuerzo de aplicar y posiblemente extender los enfoques ya consolidados basados en variables latentes (es decir, Análisis de Componentes Principales - PCA - Regresión en Mínimos Cuadrados Parciales - PLS - o PLS discriminante - PLSDA) para la resolución de problemas complejos no sólo en los campos de mejora y optimización de procesos, sino también en el entorno más amplio del análisis de datos multivariados. Con este fin, en todos los capítulos proponemos nuevas soluciones algorítmicas eficientes para abordar tareas dispares, desde la transferencia de calibración en espectroscopia hasta el modelado en tiempo real de flujos de datos. El manuscrito se divide en las seis partes siguientes, centradas en diversos temas de interés: Parte I - Prefacio, donde presentamos un resumen de este trabajo de investigación, damos sus principales objetivos y justificaciones junto con una breve introducción sobre PCA, PLS y PLSDA; Parte II - Sobre las extensiones basadas en kernels de PCA, PLS y PLSDA, donde presentamos el potencial de las técnicas de kernel, eventualmente acopladas a variantes específicas de la recién redescubierta proyección de pseudo-muestras, formulada por el estadista inglés John C. Gower, y comparamos su rendimiento respecto a metodologías más clásicas en cuatro aplicaciones a escenarios diferentes: segmentación de imágenes Rojo-Verde-Azul (RGB), discriminación y monitorización de procesos por lotes y análisis de diseños de experimentos de mezclas; Parte III - Sobre la selección del número de factores en el PCA por pruebas de permutación, donde aportamos una guía extensa sobre cómo conseguir la selección de componentes de PCA mediante pruebas de permutación y una ilustración completa de un procedimiento algorítmico original implementado para tal fin; Parte IV - Sobre la modelización de fuentes de variabilidad común y distintiva en el análisis de datos multi-conjunto, donde discutimos varios aspectos prácticos del análisis de componentes comunes y distintivos de dos bloques de datos (realizado por métodos como el Análisis Simultáneo de Componentes - SCA - Análisis Simultáneo de Componentes Distintivos y Comunes - DISCO-SCA - Descomposición Adaptada Generalizada de Valores Singulares - Adapted GSVD - ECO-POWER, Análisis de Correlaciones Canónicas - CCA - y Proyecciones Ortogonales de 2 conjuntos a Estructuras Latentes - O2PLS). Presentamos a su vez una nueva estrategia computacional para determinar el número de factores comunes subyacentes a dos matrices de datos que comparten la misma dimensión de fila o columna y dos planteamientos novedosos para la transferencia de calibración entre espectrómetros de infrarrojo cercano; Parte V - Sobre el procesamiento y la modelización en tiempo real de flujos de datos de alta dimensión, donde diseñamos la herramienta de Procesamiento en Tiempo Real (OTFP), un nuevo sistema de manejo racional de mediciones multi-canal registradas en tiempo real; Parte VI - Epílogo, donde presentamos las conclusiones finales, delimitamos las perspectivas futuras, e incluimos los anexos.
La present tesi doctoral, concebuda principalment per a recolzar i reforçar la relació entre l'acadèmia i la indústria, es va desenvolupar en col·laboració amb Shell Global Solutions (Amsterdam, Països Baixos) amb l'esforç d'aplicar i possiblement estendre els enfocaments ja consolidats basats en variables latents (és a dir, Anàlisi de Components Principals - PCA - Regressió en Mínims Quadrats Parcials - PLS - o PLS discriminant - PLSDA) per a la resolució de problemes complexos no solament en els camps de la millora i optimització de processos, sinó també en l'entorn més ampli de l'anàlisi de dades multivariades. A aquest efecte, en tots els capítols proposem noves solucions algorítmiques eficients per a abordar tasques dispars, des de la transferència de calibratge en espectroscopia fins al modelatge en temps real de fluxos de dades. El manuscrit es divideix en les sis parts següents, centrades en diversos temes d'interès: Part I - Prefaci, on presentem un resum d'aquest treball de recerca, es donen els seus principals objectius i justificacions juntament amb una breu introducció sobre PCA, PLS i PLSDA; Part II - Sobre les extensions basades en kernels de PCA, PLS i PLSDA, on presentem el potencial de les tècniques de kernel, eventualment acoblades a variants específiques de la recentment redescoberta projecció de pseudo-mostres, formulada per l'estadista anglés John C. Gower, i comparem el seu rendiment respecte a metodologies més clàssiques en quatre aplicacions a escenaris diferents: segmentació d'imatges Roig-Verd-Blau (RGB), discriminació i monitorització de processos per lots i anàlisi de dissenys d'experiments de mescles; Part III - Sobre la selecció del nombre de factors en el PCA per proves de permutació, on aportem una guia extensa sobre com aconseguir la selecció de components de PCA a través de proves de permutació i una il·lustració completa d'un procediment algorítmic original implementat per a la finalitat esmentada; Part IV - Sobre la modelització de fonts de variabilitat comuna i distintiva en l'anàlisi de dades multi-conjunt, on discutim diversos aspectes pràctics de l'anàlisis de components comuns i distintius de dos blocs de dades (realitzat per mètodes com l'Anàlisi Simultània de Components - SCA - Anàlisi Simultània de Components Distintius i Comuns - DISCO-SCA - Descomposició Adaptada Generalitzada en Valors Singulars - Adapted GSVD - ECO-POWER, Anàlisi de Correlacions Canòniques - CCA - i Projeccions Ortogonals de 2 blocs a Estructures Latents - O2PLS). Presentem al mateix temps una nova estratègia computacional per a determinar el nombre de factors comuns subjacents a dues matrius de dades que comparteixen la mateixa dimensió de fila o columna, i dos plantejaments nous per a la transferència de calibratge entre espectròmetres d'infraroig proper; Part V - Sobre el processament i la modelització en temps real de fluxos de dades d'alta dimensió, on dissenyem l'eina de Processament en Temps Real (OTFP), un nou sistema de tractament racional de mesures multi-canal registrades en temps real; Part VI - Epíleg, on presentem les conclusions finals, delimitem les perspectives futures, i incloem annexos.
Vitale, R. (2017). Novel chemometric proposals for advanced multivariate data analysis, processing and interpretation [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/90442
TESIS

Due to recent progress, higher-order chemical instrumentation provides large amounts of data which need automated processing in order to extract relevant information. In most cases, the raw signals or spectra are too complex for manual analysis. The ability to detect, identify and quantitate chemical substances in gas phase in field operations is required in a huge number of applications. Among them, I would like to highlight the need for chemical sensing on diverse humanitarian, safety and security applications. In these cases, it becomes extremely important to continuously monitor the environments where chemicals are spread in order to be ready to act when abnormal events are discovered. In most critical scenarios the sample can not just be taken to the laboratory and analyzed, since an immediate answer is needed. In some other scenarios, the exploration of the area must be performed because the localization of the gas source or material of interest is unknown. This exploration can be performed using multiple mobile sensors in order to localize the chemical source or material. Different sensing technologies have been successfully used to detect and identify different chemical substances (gases or volatile compounds). These compounds could be either toxic or hazardous, or they can be signatures of the materials to be detected, for instance, explosives or drugs. Among these technologies, mobility based analyzers provide fast responses with high sensitivity. However, IMS instruments are not exempt of problems. Typically, they provide moderate selectivity, appearing overlapped peaks in the spectra. Moreover, the presence of humidity makes peaks wider, thus worsening the resolving power and the resolution. Furthermore, the response of IMS is non-linear as substance concentration increases and more than one peak can appear in the spectra due to the same compound. In the present thesis, these problems are addressed and applications using an Ion Mobility Spectrometer (IMS) and a Differential Mobility Analyzer (DMA) are shown. It is demonstrated that multivariate data analysis tools are more effective when dealing with these technologies. For the first time, multivariate data analysis tools have been applied to a novel DMA. It is shown that DMA could be established as a good instrumentation for the detection of explosives and the detection and quantitation of VOCs. Furthermore, Multivariate curve resolution Alternating Least Squares (MCR-ALS) is shown to be suitable to analyze IMS spectra qualitatively when interfering chemicals appear in the spectra and even when their behaviour is non-linear. Partial Least Squares (PLS) methods are demonstrated to work properly for the quantitative analysis of these signals; from this analysis the chemical concentrations of the target substances are obtained. It is also demonstrated in this thesis that the quantitative measurements from these sensors can be integrated in a gas source localization algorithm in order to improve the localization of the source in those scenarios where it is required. It is shown that the new proposal works significantly better in cases where the source strength is weak. This is illustrated presenting results from simulations generated under realistic conditions. Moreover, real-world data were obtained using a mobile robot mounting a photo ionization detector (PID). Experiments were carried out under forced ventilation and turbulences in indoors and outdoors environments. The results obtained validate the simulation results and confirm that the new localization algorithm can effectively operate in real environments.
Debido a los progresos recientes, la instrumentación química genera mayores volúmenes de datos los cuales requieren de un procesado automático con la finalidad de extraer la información relevante, ya que un análisis manual no suele ser viable debido a la elevada complejidad de los datos. La habilidad de detectar, identificar y cuantificar sustancias químicas en fase gas en operaciones de campo es requerida en un gran número de aplicaciones. Entre ellas, aplicaciones humanitarias y de seguridad. En estos casos, la monitorización continua de los entornos es extremadamente importante, ya que se debe estar alerta de eventos anormales. En los escenarios más críticos, debe realizarse una exploración del área porque la posición de la fuente de gas de interés es desconocida. Esta exploración puede realizarse usando múltiples robots. Diferentes tecnologías de sensores se han aplicado con éxito a la detección e identificación de diferentes sustancias químicas (gases o compuestos volátiles). Estos compuestos pueden ser tóxicos, peligrosos, o precursores de explosivos o drogas. De entre estas tecnologías, los analizadores basados en movilidad iónica (IMS) proporcionan rápidas respuestas con gran sensibilidad. Sin embargo, estos instrumentos no están exentos de problemas. Típicamente, proporcionan una moderada selectividad, apareciendo picos solapados en los espectros. Además, la presencia de humedad provoca que los picos se ensanchen, así empeorando la resolución. Además, la respuesta de IMS es no lineal al incrementar la concentración y es posible que más de un pico debido al mismo compuesto aparezca en el espectro. En la presente tesis se trata con estos problemas y se demuestra que las herramientas de análisis de datos multivariantes son más efectivas que las herramientas típicas univariantes al tratar con tecnologías de movilidad iónica (IMS y DMA), especialmente para el análisis cualitativo y cuantitativo de sus espectros. Además, se demuestra que las medidas cuantitativas pueden integrarse de manera efectiva en un algoritmo de localización de fuentes químicas. Los resultados obtenidos (simulaciones realistas y datos reales) muestran que el algoritmo desarrollado durante la tesis puede funcionar especialmente bien en situaciones en las que la potencia de emisión de la fuente a detectar sea débil.

L'évolution de l'environnement et le climat sont, actuellement, au centre de toutes les attentions. Les impacts de l'activité des sociétés actuelles et passées sur l'environnement sont notamment questionnés pour mieux anticiper les implications de nos activités sur le futur. Mieux décrire les environnements passés et leurs évolutions sont possibles grâce à l'étude de nombreux enregistreurs naturels (sédiments, spéléothèmes, cernes, coraux). Grâce à eux, il est possible de caractériser des évolutions bio-physico-chimiques à différentes résolutions temporelles et pour différentes périodes. La haute résolution entendue ici comme la résolution su sante pour l'étude de l'environnement en lien avec l'évolution des sociétés constitue le principal verrou de l'étude de ces archives naturelles notamment en raison de la capacité analytique des appareils qui ne peuvent que rarement voir des structures fines inframillimétriques. Ce travail est bâti autour de l'hypothèse que l'utilisation de caméras hyperspectrales (VNIR, SWIR, LIF) couplée à des méthodes statistiques pertinentes doivent permettre d'accéder aux informations spectrales et donc bio-physico-chimiques contenues dans ces archives naturelles à une résolution spatiale de quelques dizaines de micromètres et, donc, de proposer des méthodes pour atteindre la haute résolution temporelle (saisonnière). De plus, a n d'avoir des estimations ables, plusieurs capteurs d'imageries et de spectroscopies linéaires (XRF, TRES) sont utilisés avec leurs propres caractéristiques (résolutions, gammes spectrales, interactions atomiques/moléculaires). Ces méthodes analytiques sont utilisées pour la caractérisation de la surface des carottes sédimentaires. Ces analyses spectrales micrométriques sont mises en correspondance avec des analyses géochimiques millimétriques usuelles. Optimiser la complémentarité de toutes ces données, implique de développer des méthodes permettant de dépasser la difficulté inhérente au couplage de données considérées par essence dissimilaire (résolutions, décalages spatiaux, non-recouvrement spectral). Ainsi, quatre méthodes ont été développées. La première consiste à associer les méthodes hyperspectrales et usuelles pour la création de modèles prédictifs quantitatifs. La seconde permet le recalage spatial des différentes images hyperspectrales à la plus basse des résolutions. La troisième s'intéresse à la fusion de ces dernières à la plus haute des résolutions. Enfin, la dernière s'intéresse aux dépôts présents dans les sédiments (lamines, crues, tephras) pour ajouter une dimension temporelle à nos études. Grâce à l'ensemble de ces informations et méthodes, des modèles prédictifs multivariés ont été estimés pour l'étude de la matière organique, des paramètres texturaux et de la distribution granulométrique. Les dépôts laminés et instantanés au sein des échantillons ont été caractérisés. Ceci a permis d'estimer des chroniques de crues, ainsi que des variations biophysico-chimiques à l'échelle de la saison. L'imagerie hyperspectrale couplée à des méthodes d'analyse des données sont donc des outils performants pour l'étude des archives naturelles à des résolutions temporelles fines. L'approfondissement des approches proposées dans ces travaux permettra d'étudier de multiples archives pour caractériser des évolutions à l'échelle d'un ou de plusieurs bassin(s) versant(s)
The evolution of the environment and climate are, currently, the focus of all attention. The impacts of the activities of present and past societies on the environment are in particular questioned in order to better anticipate the implications of our current activities on the future. Better describing past environments and their evolutions are possible thanks to the study of many natural recorders (sediments, speleothems, tree rings, corals). Thanks to them, it is possible to characterize biological-physical-chemical evolutions at di erent temporal resolutions and for di erent periods. The high resolution understood here as the su cient resolution for the study of the environment in connection with the evolution of societies constitutes the main lock of the study of these natural archives in particular because of the analytical capacity devices that can only rarely see ne inframillimetre structures. This work is built on the assumption that the use of hyperspectral sensors (VNIR, SWIR, LIF) coupled with relevant statistical methods should allow access to the spectral and therefore biological-physical-chemical contained in these natural archives at a spatial resolution of a few tens of micrometers and, therefore, to propose methods to reach the high temporal resolution (season). Besides, to obtain reliable estimates, several imaging sensors and linear spectroscopy (XRF, TRES) are used with their own characteristics (resolutions, spectral ranges, atomic/molecular interactions). These analytical methods are used for surface characterization of sediment cores. These micrometric spectral analyses are mapped to usual millimeter geochemical analyses. Optimizing the complementarity of all these data involves developing methods to overcome the di culty inherent in coupling data considered essentially dissimilar (resolutions, spatial shifts, spectral non-recovery). Thus, four methods were developed. The rst consists in combining hyperspectral and usual methods for the creation of quantitative predictive models. The second allows the spatial registration of di erent hyperspectral images at the lowest resolution. The third focuses on their merging with the highest of the resolutions. Finally, the last one focuses on deposits in sediments (laminae, oods, tephras) to add a temporal dimension to our studies. Through all this information and methods, multivariate predictive models were estimated for the study of organic matter, textural parameters and particle size distribution. The laminated and instantaneous deposits within the samples were characterized. These made it possible to estimate oods chronicles, as well as biological-physical-chemical variations at the season scale. Hyperspectral imaging coupled with data analysis methods are therefore powerful tools for the study of natural archives at ne temporal resolutions. The further development of the approaches proposed in this work will make it possible to study multiple archives to characterize evolutions at the scale of one or more watershed(s)

In the last two decades chemometrics has become an essential tool for the experimental biologist and chemist. The level of contribution varies strongly depending on the type of research performed. Therefore, chemometrics may be used to interpret and explain results, to compare experimental data with real-word ‘unseen’ data, to accurately detect certain chemical vapour, to identify cancerous related metabolites, to identify and rank potentially relevant/important variables or simply just for a pictorial interpretation and understanding of the results. Whilst many chemometrics methods are well-established in the area of chemistry and metabolomics many scientists are still using them with what is often referred to as a ‘black-box’ approach, that is without prior knowledge of the methods and well-recognised statistical properties. This lack of knowledge is thanks to the wide availability of powerful computers and – perhaps more notably – up-to-date, easy to use and reliable software. The main aim of this study is to reduce this gap by providing extensive demonstration of several approaches applied at different stages of the data analysis pipeline highlighting the importance of appropriate method selection. The comparisons are based both on chemical and biochemical (metabolomics) data and construct a firm basis for the researchers in terms of understanding of chemometric methods and the influence of parameter selection. Consequently, in this thesis the exploration and comparison of different approaches employed for various statistical steps are investigated. These include pre-treatment steps such as dealing with missing data and scaling. First, different substitution of missing values and their influence on unsupervised and supervised learning have been compared, where it has been shown that metabolites that display skewness in distribution can have a significant impact on the replacement approach. The scaling approaches were compared in terms of effect on classification accuracy for variety of metabolomics data sets. It was shown that the most standard option which is autoscaling is not always the best. In the next step a comparison of various variable selection methods which are commonly used for the analysis of chemical data has been carried out. The results revealed that random forests, with its variable selection techniques, and support vector machines, combined with recursive feature elimination as a variable selection method, displayed the best results in comparison to other approaches. Moreover, in this study a double cross-validation procedure was applied to minimize the consequence of over-fitting. Finally, seven different algorithms and two model validation procedures based on either 10-fold cross-validation or bootstrapping were investigated in order to allow direct comparison between different classification approaches.

The research has included collaboration with number of different companies and consortiums involving spectroscopic measurements with the application of chemometric techniques. For the 'European Framework 5', Standards Measurements and Testing (SMT) chemometrics network consortium a certified reference dataset based on visible metals complex spectra was developed. An inter-laboratory study was carried out which demonstrated the between subject significant difference for chemometric data analysis. An industrial collaboration with BNFL, Springfield's, this work consisted of producing a PLS regression model which could be used to predict levels of uranyl and nitrate in uranyl nitrate liquors samples, which were analysed by Raman spectroscopy which was insensitive to temperature. A substantial amount of work has been in the development of GMS with multivariate calibration for process analysis. The GMS is designed for the analysis of flowing mixtures, slurries and moisture content. The method is currently hindered by the existing calibration method; here PCA, PLS and weighted ridge regression (WRR) have been applied to the broadband, complex spectra to successfully allow measurement of a range of samples including; aqueous, organic, fermentation and non-homogeneous samples.

Thesis (MSc)--Stellenbosch University, 2014.
ENGLISH ABSTRACT: Background New methods to facilitate exploratory analysis in scientific data are in high demand. There is an abundance of available data used only for confirmatory analysis from which new hypotheses can be drawn. To this end, two new exploratory techniques are developed: one for chemometrics and another for visualisation of fundamental scientific experiments. The former transforms large-scale multiple raw HPLC/UV-vis data into a conserved set of putative features - something not often attempted outside of Mass-Spectrometry. The latter method ('StatNet'), applies network techniques to the results of designed experiments to gain new perspective on variable relations. Results The resultant data format from un-targeted chemometric processing was amenable to both chemical and statistical analysis. It proved to have integrity when machine-learning techniques were applied to infer attributes of the experimental set-up. The visualisation techniques were equally successful in generating hypotheses, and were easily extendible to three different types of experimental results. Conclusion The overall aim was to create useful tools for hypothesis generation in a variety of data. This has been largely reached through a combination of novel and existing techniques. It is hoped that the methods here presented are further applied and developed.
AFRIKAANSE OPSOMMING: Agtergrond Nuwe metodes om ondersoekende ontleding in wetenskaplike data te fasiliteer is in groot aanvraag. Daar is 'n oorvloed van beskikbaar data wat slegs gebruik word vir bevestigende ontleding waaruit nuwe hipoteses opgestel kan word. Vir hierdie doel, word twee nuwe ondersoekende tegnieke ontwikkel: een vir chemometrie en 'n ander vir die visualisering van fundamentele wetenskaplike eksperimente. Die eersgenoemde transformeer grootskaalse veelvoudige rou HPLC / UV-vis data in 'n bewaarde stel putatiewe funksies - iets wat nie gereeld buite Massaspektrometrie aangepak word nie. Die laasgenoemde metode ('StatNet') pas netwerktegnieke tot die resultate van ontwerpte eksperimente toe om sodoende ân nuwe perspektief op veranderlike verhoudings te verkry. Resultate Die gevolglike data formaat van die ongeteikende chemometriese verwerking was in 'n formaat wat vatbaar is vir beide chemiese en statistiese analise. Daar is bewys dat dit integriteit gehad het wanneer masjienleertegnieke toegepas is om eienskappe van die eksperimentele opstelling af te lei. Die visualiseringtegnieke was ewe suksesvol in die generering van hipoteses, en ook maklik uitbreibaar na drie verskillende tipes eksperimentele resultate. Samevatting Die hoofdoel was om nuttige middele vir hipotese generasie in 'n verskeidenheid van data te skep. Dit is grootliks bereik deur 'n kombinasie van oorspronklike en bestaande tegnieke. Hopelik sal die metodes wat hier aangebied is verder toegepas en ontwikkel word.

La metabolomica, che consiste nella identificazione di tutti i metaboliti presenti all’interno dei campioni biologici analizzati, è un approccio ampiamente applicato in diversi campi di ricerca quali: identificazione di biomarcatori, sviluppo di nuovi farmaci, scienze alimentari e ambientali. La metabolomica è strettamente legata alla capacità di tecniche analitiche fra queste una delle più applicate è la gas cromatografia accoppiata alla spettrometria di massa. Moderne piattaforme analitiche possono generare centinaia di migliaia di spettri, rilevando una quantità impressionante di molecole distinte. Nonostante i progressi tecnici raggiunti sul lato sperimentale, la conversione dei segnali misurati dagli strumenti in informazioni utili non è un passaggio scontato in studi metabolomici. Per ogni composto identificato, l’obbiettivo è ottenere la concentrazione relativa tra tutti i campioni analizzati e lo spettro di massa associato al composto, necessario per l’identificazione della molecola stessa. I software disponibili per l’analisi dei dati sperimentali sono stati ripetutamente indicati come una fonte importante di incertezza, limitando fortemente sia la quantità che la qualità delle informazioni estratte. Gli strumenti più applicati richiedono l’impostazione di diversi parametri da parte dell’operatore, influenzando il risultato dell’analisi. In questa tesi è descritto un nuovo approccio, chiamato AutoDise, per l’analisi dei dati GC-MS. L’elaborazione dei segnali sperimentali si basa su PARAFAC2. PARAFAC2 è un modello che scompone dati multidimensionali, discriminando tra i diversi segnali nei campioni. Grazie alle sue proprietà, PARAFAC2 non ha bisogno che i dati siano pretrattati e non richiede di impostare parametri, mentre software utilizzati in questo ambito richiedono di definire diversi parametri e un laborioso pretrattamento dei dati, richiedendo l’intervento di un utente esperto, inoltre la riproducibilità dei risultati è limitata, dipendendo i parametri scelti dall’utente. Tuttavia, il fitting di modelli PARAFAC2 coinvolge diverse fasi ed è necessario un esperto analista per l’analisi e l’interpretazione dei modelli. AutoDise è un sistema esperto in grado di gestire tutti i passaggi riguardanti la modellazione e di generare una tabella dei picchi in cui ogni composto è identificato in modo univoco, con risultati completamente riproducibili. Questo è possibile grazie alla combinazione di diversi strumenti diagnostici e grazie all’ applicazione di modelli d’intelligenza artificiale. Le prestazioni dell’approccio sono state testate su un complesso dataset di oli d’oliva ottenuto tramite analisi GC-MS. I dati sono stati analizzati sia manualmente, da utenti esperti, sia automaticamente con il metodo AutoDise proposto e le tabelle dei picchi risultanti sono state confrontate. I risultati mostrano che AutoDise supera l’analisi manuale sia in termini di numero di composti identificati che di qualità dell’identificazione e della quantificazione. Inoltre, è stata sviluppata una GUI per rendere l’algoritmo più accessibile a persone non esperte nel linguaggio di programmazione. La tesi include un tutorial che mostra le caratteristiche principali e come utilizzare l’interfaccia grafica. Un’altra parte importante della tesi è stata dedicata al test e allo sviluppo di nuove reti neurali artificiali da implementare nel software AutoDise per rilevare quali componenti PARAFAC2 stanno fornendo informazioni chimicamente utili. A tal fine, più di 170.000 profili sono stati etichettati manualmente, al fine di addestrare, validare e testare una rete neurale convoluzionale e una rete bilineare con memoria a breve termine e un modello k-nearest neighbour. I risultati suggeriscono che le reti di deep learning possono essere efficacemente applicate per la classificazione automatica dei profili cromatografici.
Metabolomics, which consists of identifying all the metabolites present in the biological samples analysed, is an approach widely applied in various research fields such as biomarker identification, new drug development, food and environmental sciences. Metabolomics is closely linked to the ability of analytical techniques, one of the most widely applied being gas chromatography coupled to mass spectrometry. Modern analytical platforms can generate hundreds of thousands of spectra, detecting an impressive number of distinct molecules. Despite the technical progress achieved on the experimental side, the conversion of signals measured by instruments into useful information is not an obvious step in metabolomic studies. For each identified compound, the goal is to obtain the relative concentration among all analysed samples and the mass spectrum associated with the compound needed to identify the molecule itself. The software available for analysing experimental data has repeatedly been cited as a major source of uncertainty, severely limiting both the quantity and quality of the information extracted. The most applied tools are based on univariate data analysis, considering each sample separately from the others and requiring the operator to set several parameters, affecting the result of the analysis. In this thesis, a new approach, called AutoDise, for the analysis of GC-MS data is described. The processing of the experimental signals is based on PARAFAC2. PARAFAC2 is a model that decomposes multidimensional data, discriminating between different signals in the samples. Due to its properties, PARAFAC2 does not need the data to be pre-processed and does not require parameters to be set, whereas software used in this field requires several parameters to be defined and laborious pre-processing of the data, requiring the intervention of an expert user, and the reproducibility of the results is limited, depending on the parameters chosen by the user. However, fitting PARAFAC2 models involves several steps and an experienced analyst is needed to analyse and interpret the models. AutoDise is an expert system capable of handling all modelling steps and generating a peak table in which each compound is uniquely identified, with fully reproducible results. This is possible thanks to the combination of different diagnostic tools and the application of artificial intelligence models. The performance of the approach was tested on a complex dataset of olive oils obtained by GC-MS analysis. The data were analysed both manually, by experienced users, and automatically with the proposed AutoDise method and the resulting peak tables were compared. The results show that AutoDise outperforms manual analysis both in terms of the number of compounds identified and the quality of identification and quantification. In addition, a GUI was developed to make the algorithm more accessible to people not skilled in the programming language. The thesis includes a tutorial showing the main features and how to use the GUI. Another important part of the thesis was devoted to testing and developing new artificial neural networks to be implemented in the AutoDise software to detect which PARAFAC2 components are providing chemically useful information. To this end, more than 170,000 profiles were manually labelled in order to train, validate and test a convolutional neural network and a bilinear network with short-term memory and a k-nearest neighbour model. The results suggest that deep learning networks can be effectively applied for the automatic classification of chromatographic profiles.

Thesis (Ph. D.)--University of Kentucky, 2006.
Title from document title page (viewed on August 22, 2006). Document formatted into pages; contains: xvi, 229 p. : ill. (some col.). Includes abstract and vita. Includes bibliographical references (p. 213-227).

Data processing tools are valuable to the analytical chemist as they can speed up the analysis, and sometimes solve problems that are not feasible to solve in a traditional manner. However, the complexity of many data processing tools can make their use daunting for the inexperienced user. This thesis includes two applications and two tools for data processing. The first application focuses on minimizing the manual input, reducing the time required for a simple task. The second application required more manual input, in the form of parameter selection, but process far more data.  The data processing tools both include features that simplify the manual work required. The first by including visual diagnostics tools that helps in setting the parameters. The second via internal validation that makes the tool’s process more robust and reliable, and thereby less sensitive to small changes in the parameters. No matter how good or precise a data processing tool is, if it is so cumbersome that it is not used by the analytical chemists that need it, it is useless. Therefore, the main focus of this thesis is to make data processing easier.

At the time of the doctoral defense, the following paper was unpublished and had a status as follows: Paper 4: Submitted.

Big data is a concept that is expanding rapidly. As more and more data is generatedand garnered, there is an increasing need for efficient solutions that can be utilized to process all this data in attempts to gain value from it. The purpose of this thesis is to find an efficient way to quickly process a large number of relatively small files. More specifically, the purpose is to test two frameworks that can be used for processing big data. The frameworks that are tested against each other are Apache NiFi and Apache Storm. A method is devised in order to, firstly, construct a data flow and secondly, construct a method for testing the performance and scalability of the frameworks running this data flow. The results reveal that Apache Storm is faster than Apache NiFi, at the sort of task that was tested. As the number of nodes included in the tests went up, the performance did not always do the same. This indicates that adding more nodes to a big data processing pipeline, does not always result in a better performing setup and that, sometimes, other measures must be made to heighten the performance.
Big data är ett koncept som växer snabbt. När mer och mer data genereras och samlas in finns det ett ökande behov av effektiva lösningar som kan användas föratt behandla all denna data, i försök att utvinna värde från den. Syftet med detta examensarbete är att hitta ett effektivt sätt att snabbt behandla ett stort antal filer, av relativt liten storlek. Mer specifikt så är det för att testa två ramverk som kan användas vid big data-behandling. De två ramverken som testas mot varandra är Apache NiFi och Apache Storm. En metod beskrivs för att, för det första, konstruera ett dataflöde och, för det andra, konstruera en metod för att testa prestandan och skalbarheten av de ramverk som kör dataflödet. Resultaten avslöjar att Apache Storm är snabbare än NiFi, på den typen av test som gjordes. När antalet noder som var med i testerna ökades, så ökade inte alltid prestandan. Detta visar att en ökning av antalet noder, i en big data-behandlingskedja, inte alltid leder till bättre prestanda och att det ibland krävs andra åtgärder för att öka prestandan.

Apache Spark è un framework open source per la computazione distribuita su larga scala, caratterizzato da un engine in-memory che permette prestazioni superiori a soluzioni concorrenti nell’elaborazione di dati a riposo (batch) o in movimento (streaming). In questo lavoro presenteremo alcune tecniche progettate e implementate per migliorare l’elasticità e l’adattabilità del framework rispetto a modifiche dinamiche nell’ambiente di esecuzione o nel workload. Lo scopo primario di tali tecniche è di permettere ad applicazioni concorrenti di condividere le risorse fisiche disponibili nell’infrastruttura cluster sottostante in modo efficiente. Il contesto nel quale le applicazioni distribuite vengono eseguite difficilmente può essere considerato statico: le componenti hardware possono fallire, i processi possono interrompersi, gli utenti possono allocare risorse aggiuntive in modo imprevedibile nel tentativo di accelerare la computazione o di allegerire il carico di lavoro. Infine, non soltanto le risorse fisiche ma anche i dati in input possono variare di dimensione e complessità durante l’esecuzione, così che sia dati sia risorse non possano essere considerati statici. Una configurazione immutabile del cluster non riuscirà a ottenere la migliore efficienza possibile per tutti i differenti carichi di lavoro. Ne consegue che un framework per il calcolo distribuito che sia "consapevole" delle modifiche ambientali e delle modifiche al workload e che sia in grado di adattarsi a esse puo risultare piu performante di un framework che permetta unicamente configurazioni statiche. Gli esperimenti da noi compiuti con applicazioni Big Data altamente parallelizzabili mostrano come il costo della soluzione proposta sia minimo e come la nostra version di Spark più dinamica e adattiva possa portare a benefici in termini di flessibilità, scalabilità ed efficienza.

ITC/USA 2009 Conference Proceedings / The Forty-Fifth Annual International Telemetering Conference and Technical Exhibition / October 26-29, 2009 / Riviera Hotel & Convention Center, Las Vegas, Nevada
Once the test is complete, the job of the Data Analyst has begun. Files from the various acquisition systems are collected. It is the job of the analyst to put together these files in a readable format so the success or failure of the test can be attained. This paper will discuss the process of breaking down these files, comparing data from different systems, and methods of presenting the data.

Democratisation of data means that more people than ever are involved in the data analysis process. This is beneficial - it brings domain-specific knowledge from broad fields - but data scientists do not have adequate tools to write algorithms and execute them at scale. Processing models of current data-parallel processing systems, designed for scalability and fault tolerance, are stateless. Stateless processing facilitates capturing parallelisation opportunities and hides fault tolerance. However, data scientists want to write stateful programs - with explicit state that they can update, such as matrices in machine learning algorithms - and are used to imperative-style languages. These programs struggle to execute with high-performance in stateless data-parallel systems. Representing state explicitly makes data-parallel processing at scale challenging. To achieve scalability, state must be distributed and coordinated across machines. In the event of failures, state must be recovered to provide correct results. We introduce stateful data-parallel processing that addresses the previous challenges by: (i) representing state as a first-class citizen so that a system can manipulate it; (ii) introducing two distributed mutable state abstractions for scalability; and (iii) an integrated approach to scale out and fault tolerance that recovers large state - spanning the memory of multiple machines. To support imperative-style programs a static analysis tool analyses Java programs that manipulate state and translates them to a representation that can execute on SEEP, an implementation of a stateful data-parallel processing model. SEEP is evaluated with stateful Big Data applications and shows comparable or better performance than state-of-the-art stateless systems.

Multivariate analysis has rapidly developed in the past few years. This rise is due to advances in intelligent instruments and laboratory automation as well the possibility of using powerful computers and user-friendly software. In the field of analytical chemistry, the capability of newer, mostly multicomponent or multielement analytical methods produces so many data, that only the use of mathematical and statistical techniques can provide a suitable interpretation. The aim of the present work is to develop multivariate methods for processing experimental data obtained through non-destructive techniques, in which it is possible to investigate samples without altering them. For qualitative investigation, such “direct” analytical procedures like infrared spectroscopy are available; however, the univariate approach is not exaustive in case of very complex matrices. The quantitative approach is still an open issue, due to the strong matrix effect hindering the creation of univariate calibration methods in interpolation mode. Multivariate analysis may be the solution. This thesis is organized as follows: In Section 1 the general problem of high-dimensional data is introduced, reviewing the basic principles of Principal Components and their implementation for descriptive and predictive purposes. In the last part of this section the core of the present work is discussed: advanced algorithms aimed to perform standard addition method in multivariate analysis Section 2 is dedicated to theory of the employed analytical technique: the basis of infrared spectroscopy, focusing particular attention to reflectance technique Section 3 describes the typologies of the analysed samples (marine sediments) and the reason of interest of one of their specific components (biogenic silica) In Section 4 experimental data, their computational treatment and a final discussion of results compared with other reference methods are presented.

Large data processing systems require a high degree of coordination, and exhibit network bottlenecks due to massive communication data. This motivates my PhD study to propose system control mechanisms that improve monitoring and coordination, and efficient communication methods by bridging applications and networks. The first result is Chi, a new control plane for stateful streaming systems. Chi has a control loop that embeds control messages in data channels to seamlessly monitor and coordinate a streaming pipeline. This design helps monitor system and application-specific metrics in a scalable manner, and perform complex modification with on-the-fly data. The behaviours of control messages are customisable, thus enabling various control algorithms. Chi has been deployed into production systems, and exhibits high performance and scalability in test-bed experiments. With effective coordination, data-intensive systems need to remove network bottlenecks. This is important in data centres as their networks are usually over-subscribed. Hence, my study explores an idea that bridges applications and networks for accelerating communication. This idea can be realised (i) in the network core through a middlebox platform called NetAgg that can efficiently execute application-specific aggregation functions along busy network paths, and (ii) at network edges through a server network stack that provides powerful communication primitives and traffic management services. Test-bed experiments show that these methods can improve the communication of important analytics systems. A tight integration of applications and networks, however, requires an intuitive network programming model. My study thus proposes a network programming framework named Flick. Flick has a high-level programming language for application-specific network services. The services are compiled to dataflows and executed by a high-performance runtime. To be production-friendly, this runtime can run in commodity network elements and guarantee fair resource sharing among services. Flick has been used for developing popular network services, and its performance is shown in real-world benchmarks.

International Telemetering Conference Proceedings / October 23-26, 2000 / Town & Country Hotel and Conference Center, San Diego, California
With the introduction of digital data recorders new ways of data processing have been developed. The three most important improvements are discussed in this paper: A) By processing PCM Data from a digital recorder by using the SCSI-Interface our ground station has developed software to detect the synchronization pattern of the PCM data and then perform software frame decommutation. Many advantages will be found with this method. B) New digital recorders already use the CCSDS Standard as the internal recording format. Once this technique is implemented in our ground station’s software and becomes part of our software engineering team’s general know-how, the switch to CCSDS telemetry in the future will require no quantum leap in effort. C) Digital recorders offer a very new application: Writing data to a digital tape in the recorder’s own format, allows the replay of data using the recorder’s interfaces; i.e. writing vibration data from the host system to tape, using the analog format of the digital recorder, allows the analysis of the data either in analog form, using the analog interface of the recorder, or in digital form.

International Telemetering Conference Proceedings / October 28-31, 1996 / Town and Country Hotel and Convention Center, San Diego, California
The Telemetry Data Center (TDC) at White Sands Missile Range (WSMR) is now beginning to modernize its existing telemetry data processing system. Modern networking and interactive graphical displays are now being introduced. This infusion of modern technology will allow the TDC to provide our customers with enhanced data processing and display capability. The intent of this project is to outline this undertaking.

Thesis (MTech (Information Technology))--Cape Peninsula University of Technology, 2016.
Health information policies are constantly becoming a key feature in directing information usage in healthcare. After the passing of the Health Information Technology for Economic and Clinical Health (HITECH) Act in 2009 and the Affordable Care Act (ACA) passed in 2010, in the United States, there has been an increase in health systems innovations. Coupling this health systems hype is the current buzz concept in Information Technology, „Big data‟. The prospects of big data are full of potential, even more so in the healthcare field where the accuracy of data is life critical. How big health data can be used to achieve improved health is now the goal of the current health informatics practitioner. Even more exciting is the amount of health data being generated by patients via personal handheld devices and other forms of technology that exclude the healthcare practitioner. This patient-generated data is also known as Personal Health Records, PHR. To achieve meaningful use of PHRs and healthcare data in general through big data, a couple of hurdles have to be overcome. First and foremost is the issue of privacy and confidentiality of the patients whose data is in concern. Secondly is the perceived trustworthiness of PHRs by healthcare practitioners. Other issues to take into context are data rights and ownership, data suppression, IP protection, data anonymisation and reidentification, information flow and regulations as well as consent biases. This study sought to understand the role of data policies in the process of data utilisation in the healthcare sector with added interest on PHRs utilisation as part of big health data.

Studies have proven that the desired signal for Magnetotellurics (MT) in the electromagnetic (EM) field can be regarded as 'quasi stationary' (i.e. sufficiently stationary to apply a windowed Fourier transform). However, measured time series often contain environmental noise. Hence, they may not fulfill the stationarity requirement for the application of the Fourier Transform (FT) and therefore may lead to false or unreliable results under methods that rely on the FT. In light of paucity of algorithms of MT data processing in the presence of non stationary noise, it is the goal of this thesis to elaborate a robust, non stationary algorithm, which can compete with sophisticated, state-of-the-art algorithms in terms of accuracy and precision. In addition, I proof mathematically the algorithm's viability and validate its superiority to other codes processing non stationary, synthetic and real MT data. Non stationary EM data may affect the computation of Fourier spectra in unforeseeable manners and consequently, the traditional estimation of the MT transfer functions (TF). The TF estimation scheme developed in this work is based on an emerging nonlinear, non stationary time series analysis tool, called Empirical Mode Decomposition (EMD). EMD decomposes time series into Intrinsic Mode Functions (IMF) in the time-frequency domain, which can be represented by the instantaneous parameters amplitude, phase and frequency. In the first part of my thesis, I show that time slices of well defined IMFs equal time slices of Fourier Series, where the instantaneous parameters of the IMF define amplitude and phase of the Fourier Series parameters. Based on these findings I formulate the theorem that non stationary convolution of an IMF with a general time domain response function translates into a multiplication of the IMF with the respective spectral domain response function, which is explicitly permitted to vary over time. Further, I employ real world MT data to illustrate that a de-trended signal's IMFs can be convolved independently and then be used for further time-frequency analysis as done for MT processing. In the second part of my thesis, I apply the newly formulated theorem to the MT method. The MT method analyses the correlation between the electric and magnetic field due to the conductivity structure of the subsurface. For sufficiently low frequencies (i.e. when the EM field interacts diffusively), the conductive body of the Earth acts as an inductive system response, which convolves with magnetic field variations and results in electric field variations. The frequency representation of this system response is commonly referred to as MT TF and its estimation from measured electric and magnetic time series is summarized as MT processing. The main contribution in this thesis is the design of the MT TF estimation algorithm based on EMD. In contrast to previous works that employ EMD for MT data processing, I (i) point out the advantages of a multivariate decomposition, (ii) highlight the possibility to use instantaneous parameters, and (iii) define the homogenization of frequency discrepancies between data channels. In addition, my algorithm estimates the transfer functions using robust statistical methods such as (i) robust principal component analysis and (ii) iteratively re-weighted least squares regression with a Huber weight function. Finally, TF uncertainties are estimated by iterating the complete robust regression, including the robust weight computation, by means of a bootstrap routine. The proposed methodology is applied to synthetic and real data with and without non stationary character and the results are compared with other processing techniques. I conclude that non stationary noise can heavily affect Fourier based MT data processing but the presented non stationary approach is nonetheless able to extract the impedances correctly even when the other methods fail.

Thesis (M.S. in Electrical Engineering and M.S. in Applied Physics) Naval Postgraduate School, September 1994.
Thesis advisor(s): Richard Christopher Olsen, Ralph Hippenstiel. "September 1994." Bibliography: p. 71. Also available online.

Thesis (B. Sc.(Hons.))--University of Adelaide, Dept. of Geology and Geophysics, 1994.
Cover title: Advantages of multi-channel radiometric processing. Two maps have overlays. National map series reference Forbes, N.S.W. 1:250,000 S heet SI/55-7. Includes bibliographical references (leaf 38).

The scale-out approach of modern data-parallel frameworks such as Apache Flink or Apache Spark has enabled them to deal with large amounts of data. These applications are often deployed in large-scale data centres with many resources. However, as deployments and data continue to grow, more network communication is incurred during a data processing query. At the same time, data centre networks (DCNs) are becoming increasingly more complex in terms of the physical network topology, the variety of applications that are sharing the network, and the different requirements of these applications on the network. The high complexity of DCNs combined with the increased traffic demands of applications has made the network a bottleneck for query performance. In this thesis, we explore ways of making data-parallel frameworks network-aware, i.e. we combine specific knowledge about the application and the physical network to reduce query completion times. We identify three main types of traffic that occur during query processing and add network-awareness to each of them to optimise network usage. 1) Traffic reduction for aggregatable traffic exploits the physical network topology and the associativity and commutativity of aggregation queries to reduce traffic as early as possible. In-network aggregation trees utilise existing networking hardware and the tree topology of DCNs to partially aggregate and thereby reduce data as it flows through the network. 2) Traffic balancing for non-aggregatable traffic monitors the network throughput of an application and uses knowledge about the query to optimise the overall network utilisation. By dynamically changing the destinations of parts of the transferred data, network hotspots, which can occur when many applications share the network, can be avoided. 3) Traffic elimination for storage traffic gives control over data placement to the application instead of the distributed storage system. This allows the application to optimise where data is stored across the cluster based on application properties and thereby eliminate unnecessary network traffic.

Thesis (M.S.)--Massachusetts Institute of Technology, Dept. of Mechanical Engineering, 1995.
Includes bibliographical references (p. 176-177).
New developments on the nanoscale are taking place rapidly in many fields. Instrumentation used to measure and understand the geometry and property of the small scale structure is therefore essential. One of the most promising devices to head the measurement science into the nanoscale is the scanning probe microscope. A prototype of a nanoscale profilometer based on the scanning probe microscope has been built in the Laboratory for Manufacturing and Productivity at MIT. A sample is placed on a precision flip stage and different sides of the sample are scanned under the SPM to acquire its separate surface topography. To reconstruct the original three dimensional profile, many techniques like digital filtering, edge identification, and image matching are investigated and implemented in the computer programs to post process the data, and with greater emphasis placed on the nanoscale application. The important programming issues are addressed, too. Finally, this system's error sources are discussed and analyzed.
by Cheng-Jung Chiu.
M.S.

International Telemetering Conference Proceedings / October 28-31, 1996 / Town and Country Hotel and Convention Center, San Diego, California
This paper will discuss the functions performed by the Spatial Infrared Imaging Telescope (SPIRIT) III Data Processing Center (DPC) at Utah State University (USU). The SPIRIT III sensor is the primary instrument on the Midcourse Space Experiment (MSX) satellite; and as builder of this sensor system, USU is responsible for developing and operating the associated DPC. The SPIRIT III sensor consists of a six-color long-wave infrared (LWIR) radiometer system, an LWIR spectrographic interferometer, contamination sensors, and housekeeping monitoring systems. The MSX spacecraft recorders can capture up to 8+ gigabytes of data a day from this sensor. The DPC is subsequently required to provide a 24-hour turnaround to verify and qualify these data by implementing a complex set of sensor and data verification and quality checks. This paper addresses the computing architecture, distributed processing software, and automated data verification processes implemented to meet these requirements.

Sensor it is the most powerful part of any system. Aviation industry is the plase where milions of sensors is be used for difetrent purpuses. Othe wery important task of avionics equipment is data transfer between sensors to processing equipment. Why it is so important to transmit data online into MatLab? Nowadays rapidly are developing unmanned aerial vehicles. If we can transmit data from UAV sensors into MatLab, then we can process it and get the desired information about UAV. Of course we have to use the most chipiest way to data transfer. Today everyone in the world has mobile phone. Many of them has different sensors, such as: pressure sensor, temperature sensor, gravity sensor, gyroscope, rotation vector sensor, proximity sensor, light sensor, orientation sensor, magnetic field sensor, accelerometer, GPS receiver and so on. It will be cool if we can use real time data from cell phone sensors for some navigation tasks. In our work we use mobile phone Samsung Galaxy SIII with all sensors which are listed above except temperature sensor. There are existing many programs for reading and displaying data from sensors, such as: “Sensor Kinetics”, “Sensors”, “Data Recording”, “Android Sensors Viewer”. We used “Data Recording”. For the purpose of transmitting data from cell phone there are following methods: - GPRS (Mobile internet); - Bluetooth; - USB cable; - Wi-Fi. After comparing this methods we analyzed that GPRS is uncomfortable for us because we should pay for it, Bluetooth has small coverage, USB cable has not such portability as others methods. So we decided that Wi-Fi is optimal method on transmitting data for our goal

Mestrado em Engenharia de Computadores e Telemática
Field-Programmable Gate Arrays (FPGAs) were invented by Xilinx in 1985, i.e. less than 30 years ago. The influence of FPGAs on many directions in engineering is growing continuously and rapidly. There are many reasons for such progress and the most important are the inherent reconfigurability of FPGAs and relatively cheap development cost. Recent field-configurable micro-chips combine the capabilities of software and hardware by incorporating multi-core processors and reconfigurable logic enabling the development of highly optimized computational systems for a vast variety of practical applications, including high-performance computing, data, signal and image processing, embedded systems, and many others. In this context, the main goals of the thesis are to study the new micro-chips, namely the Zynq-7000 family and to apply them to two selected case studies: data sort and Hamming weight calculation for long vectors.
Field-Programmable Gate Arrays (FPGAs) foram inventadas pela Xilinx em 1985, ou seja, há menos de 30 anos. A influência das FPGAs está a crescer continua e rapidamente em muitos ramos de engenharia. Há varias razões para esta evolução, as mais importantes são a sua capacidade de reconfiguração inerente e os baixos custos de desenvolvimento. Os micro-chips mais recentes baseados em FPGAs combinam capacidades de software e hardware através da incorporação de processadores multi-core e lógica reconfigurável permitindo o desenvolvimento de sistemas computacionais altamente otimizados para uma grande variedade de aplicações práticas, incluindo computação de alto desempenho, processamento de dados, de sinal e imagem, sistemas embutidos, e muitos outros. Neste contexto, este trabalho tem como o objetivo principal estudar estes novos micro-chips, nomeadamente a família Zynq-7000, para encontrar as melhores formas de potenciar as vantagens deste sistema usando casos de estudo como ordenação de dados e cálculo do peso de Hamming para vetores longos.

Pattern recognition is an analytical process that is now playing an increasingly important role in the interpretation and evaluation of data across many scientific disciplines. This thesis examines the application of pattern recognition techniques to both pharmaceutical process data and metabolomic data, and attempts to identify trends, as well as provide solutions to problems in these fields. Pattern recognition can be divided into univariate and multivariate techniques. Although this thesis examines both types of methods, the focus is primarily on multivariate techniques, because the pharmaceutical process data and the metabolomic data are both highly multivariate. Previous studies have provided data from these areas using a variety of analytical techniques such as acoustic emission (AE), nuclear magnetic resonance (NMR) and denaturing gradient gel electrophoresis (DGGE), and in this work multivariate pattern recognition is successful at extracting valuable information from these data. The first approach is the monitoring of a high-shear wet granulation process using AE spectroscopy. Granulation is a highly complex multi-step process, and there are many factors and variables, such as the duration of different phases, quantities of added materials, and speed of the granulator equipment. These factors can all have a considerable effect on the quality of the final granules that are produced, although it can be very difficult to pinpoint exactly to what degree they each influence the process. The combination of AE with pattern recognition techniques is studied here as an approach to identify the trends that are due to each variable. This study uses multiway analysis such as parallel factor analysis (PARAFAC) and multiway partial least squares (MPLS) to examine the data, as well as slope analysis techniques to analyse the trends between the acoustic profiles of batches that are characterized by different process variables. The results suggest that AE combined with pattern recognition techniques can be particularly useful to the pharmaceutical industry for the monitoring and control of granulation processes. The second approach looks at pattern recognition applied to the metabolomic data produced by two sources. An NMR dataset was obtained from human samples of saliva, half of which had been treated with a mouthwash. Multiple factors contributed to variance in the dataset. The total variance was split into parts characterized by these underlying experimental factors. Multilevel techniques were applied, based on a hierarchical relationship between the factors. Also, as the experimental factors in the study varied due to the programmed experimental design, ANOVA based techniques were utilized to interpret this variance. The results suggest that in metabolomic studies on humans there is a high variation between the subjects. This variation was dominant and once extracted, trends that were due to other experimental factors could be analysed. The second metabolomic dataset was a DGGE dataset. Pattern recognition techniques, in particular • principal coordinates analysis (PCO), were used to analyse the trends in microbial profiles of human sweat. It was shown that PCO components can be effectively used as an input for various classification techniques in order to observe individual or group separation.

During this project three distinct methods to improve the data processing of Controlled Source Audio Magnetotellurics (CSAMT) data are implemented and their advantages and disadvantages are discussed. The methods in question are: Detrending the time series in the time domain, instead of detrending in the frequencydomain. Implementation of a coherency test to pinpoint data segments of low quality andremove these data from the calculations. Implementing a method to detect and remove transients from the time series toreduce background noise in the frequency spectra. Both the detrending in time domain and the transient removal shows potential in improvingdata quality even if the improvements are small(both in the (1-10% range). Due totechnical limitations no coherency test was implemented. Overall the processes discussedin the report did improve the data quality and may serve as groundwork for further improvementsto come.
Projektet behandlar tre stycken metoder för att förbättra signalkvaliten hos Controlled Source Audio Magnetotellurics (CSAMT) data, dessa implementeras och deras för- och nackdelar diskuteras. Metoderna som hanteras är: Avlägsnandet av trender från tidsserier i tidsdomänen istället för i frekvensdomänen. Implementationen av ett koherenstest för att identifiera ”dåliga” datasegment ochavlägsna dessa från vidare beräkningar. Implementationen av en metod för att både hitta och avlägsna transienter (dataspikar) från tidsserien för att minska bakgrundsbruset i frekvensspektrat. Både avlägsnandet av trender samt transienter visar positiv inverkan på datakvaliteten,även om skillnaderna är relativt små (båda på ungefär 1-10%). På grund av begränsningarfrån mätdatan kunde inget meningsfullt koherenstest utformas. Överlag har processernasom diskuteras i rapporten förbättrat datakvaliten och kan ses som ett grundarbete förfortsatta förbättringar inom området.

In this thesis, improving the performance of the automatic recognition of the Doppler radar targets is studied. The radar used in this study is a ground-surveillance doppler radar. Target types are car, truck, bus, tank, helicopter, moving man and running man. The input of this thesis is the output of the real doppler radar signals which are normalized and preprocessed (TRP vectors: Target Recognition Pattern vectors) in the doctorate thesis by Erdogan (2002). TRP vectors are normalized and homogenized doppler radar target signals with respect to target speed, target aspect angle and target range. Some target classes have repetitions in time in their TRPs. By the use of these repetitions, improvement of the target type classification performance is studied. K-Nearest Neighbor (KNN) and Support Vector Machine (SVM) algorithms are used for doppler radar target classification and the results are evaluated. Before classification PCA (Principal Component Analysis), LDA (Linear Discriminant Analysis), NMF (Nonnegative Matrix Factorization) and ICA (Independent Component Analysis) are implemented and applied to normalized doppler radar signals for feature extraction and dimension reduction in an efficient way. These techniques transform the input vectors, which are the normalized doppler radar signals, to another space. The effects of the implementation of these feature extraction algoritms and the use of the repetitions in doppler radar target signals on the doppler radar target classification performance are studied.

This thesis introduces new algorithms for analysis and classification of multivariate data. Statistical approaches are devised for the objectives of data clustering, data classification and object recognition. An initial investigation begins with the application of fundamental pattern recognition principles. Where such fundamental principles meet their limitations, statistical and neural algorithms are integrated to augment the overall approach for an enhanced solution. This thesis provides a new dimension to the problem of classification of data as a result of the following developments: (1) application of algorithms for object classification and recognition; (2) integration of a neural network algorithm which determines the decision functions associated with the task of classification; (3) determination and use of the eigensystem using newly developed methods with the objectives of achieving optimized data clustering and data classification, and dynamic monitoring of time-varying data; and (4) use of the principal component transform to exploit the eigensystem in order to perform the important tasks of orientation-independent object recognition, and di mensionality reduction of the data such as to optimize the processing time without compromising accuracy in the analysis of this data.

The paper examines the application of the concept of economic efficiency to organizational issues of collective information processing in decision making. Information processing is modeled in the framework of the dynamic parallel-processing model of associative computation with an endogenous set-up cost of the processors. The model is extended to include the specific features of collective information processing in the team of decision makers which could cause an error in data analysis. In such a model, the conditions for efficient organization of information processing are defined and the architecture of the efficient structures is considered. We show that specific features of collective decision making procedures require a broader framework for judging organizational efficiency than has traditionally been adopted. In particular, and contrary to the results presented in economic literature, we show that in human data processing (unlike in computer systems), there is no unique architecture for efficient information processing structures, but a number of various efficient forms can be observed. The results indicate that technological progress resulting in faster data processing (ceteris paribus) will lead to more regular information processing structures. However, if the relative cost of the delay in data analysis increases significantly, less regular structures could be efficient. (authors' abstract)
Series: Discussion Papers of the Institute for Economic Geography and GIScience

This thesis describes the application of a wide variety of data processing methods, in particular the Maximum Entropy Method (MEM), to data from Nuclear Magnetic Resonance (NMR) experiments. Chapter 1 provides a brief introduction to NMR and to data processing, which is developed in chapter 2. NMR is described in terms of the classical model due to Bloch, and the principles of conventional (Fourier transform) data processing developed. This is followed by a description of less conventional techniques. The MEM is derived on several grounds, and related to both Bayesian reasoning and Shannon information theory. Chapter 3 describes several methods of evaluating the quality of NMR spectra obtained by a variety of data processing techniques; the simple criterion of spectral appearance is shown to be completely unsatisfactory. A Monte Carlo method is described which allows several different techniques to be compared, and the relative advantages of Fourier transformation and the MEM are assessed. Chapter 4 describes in vivo NMR, particularly the application of the MEM to data from Phase Modulated Rotating Frame Imaging (PMRFI) experiments. In this case the conventional data processing is highly unsatisfactory, and MEM processing results in much clearer spectra. Chapter 5 describes the application of a range of techniques to the estimation and removal of splittings from NMR spectra. The various techniques are discussed using simple examples, and then applied to data from the amino acid iso-leucine. The thesis ends with five appendices which contain historical and philosophical notes, detailed calculations pertaining to PMRFI spectra, and a listing of the MEM computer program.

Thesis (S.M. in Geosystems)--Massachusetts Institute of Technology, Dept. of Earth, Atmospheric, and Planetary Sciences, 1998.
Includes bibliographical references (leaves 62-64).
by Youshun Sun.
S.M.in Geosystems

For steel industry, knowing and understanding characteristics of a steel strip surface at every steps of the production process is a key element to control final product quality. Today as the quality requirements increase this task gets more and more important. The surface of new steel grades with complex chemical compositions has behaviors especially hard to master. For those grades in particular, surface control is critical and difficult. One of the promising technics to assess the problem of surface quality control is spectra analysis. Over the last few years, ArcelorMittal, world’s leading integrated steel and mining company, has led several projects to investigate the possibility of using devices to measure light spectrum of their product at different stage of the production. The large amount of data generated by these devices makes it absolutely necessary to develop efficient data treatment pipelines to get meaningful information out of the recorded spectra. In this thesis, we developed mathematical models and statistical tools to treat signal measured with spectrometers in the framework of different research projects.
För stålindustrin, att veta och förstå ytegenskaperna på ett stålband vid varje steg i produktionsprocessen är en nyckelfaktor för att styra slutproduktens kvalitet. Den senaste tidens ökande kvalitetskraven har gjort denna uppgift allt mer viktigare. Ytan på nya stål kvaliteter med komplexa kemiska sammansättningar har egenskaper som är särskilt svårt att hantera. För dess kvaliteter är ytkontroll kritisk och svår. En av de tekniker som används för att kontrollera ytans kvalitet är spektrum analys. Arcelor Mittal, världens ledande integrerade stål- och gruvföretag, har under de senaste åren lett flera projekt för att undersöka möjligheten att använda mätinstrument för att mäta spektrum ljuset från sin produkt i olika stadier av produktionen. En av de tekniker som används för att kontrollera ytans kvalitet är spektrum analys. I denna avhandling har vi utvecklat matematiska modeller och statistiska verktyg för att kunna hanskas med signaler som är uppmätt med spektrometrar inom ramen av olika forskningsprojekt hos Arcelor Mittal.