Academic literature on the topic 'Hierarchical Spatio-Temporal Feature Maps'

AbstractThe hierarchical nature of language requires human brain to internally parse connected-speech and incrementally construct abstract linguistic structures. Recent research revealed multiple neural processing timescales underlying grammar-based configuration of linguistic hierarchies. However, little is known about where in the whole cerebral cortex such temporally scaled neural processes occur. This study used novel magnetoencephalography source imaging techniques combined with a unique language stimulation paradigm to segregate cortical maps synchronized to 3 levels of linguistic units (i.e., words, phrases, and sentences). Notably, distinct ensembles of cortical loci were identified to feature structures at different levels. The superior temporal gyrus was found to be involved in processing all 3 linguistic levels while distinct ensembles of other brain regions were recruited to encode each linguistic level. Neural activities in the right motor cortex only followed the rhythm of monosyllabic words which have clear acoustic boundaries, whereas the left anterior temporal lobe and the left inferior frontal gyrus were selectively recruited in processing phrases or sentences. Our results ground a multi-timescale hierarchical neural processing of speech in neuroanatomical reality with specific sets of cortices responsible for different levels of linguistic units.

Ship type classification is an essential task in maritime navigation domains, contributing to shipping monitoring, analysis, and forecasting. Presently, with the development of ship positioning and monitoring systems, many ship trajectory acquisitions make it possible to classify ships according to their movement pattern. Existing methods of ship classification based on trajectory include classical sequence analysis and deep learning methods. However, the real ship trajectories are unevenly distributed in geographical space, which leads to many problems in inferring the ship movement mode on the original ship trajectory. This paper proposes a hierarchical spatial-temporal embedding method based on enhanced trajectory features for ship type classification. We first preprocess the trajectory and combine the port information to transform the original ship trajectory into the moored records of ships, removing the unevenly distributed points in the trajectory data and enhancing key points’ semantic information. Then, we propose a Hierarchical Spatial-Temporal Embedding Method (Hi-STEM) for ship classification. Hi-STEM maps moored records in the original geographical space into the feature space and can efficiently find the classification plane in the feature space. Experiments are conducted on real-world datasets and compared with several existing methods. The result shows that our approach has high accuracy in ship classification on ship moored records. We make the source code and datasets publicly available.

Os mapas auto-organizáveis (SOFM) consistem em um tipo de rede neural artificial que permite a conversão de dados de alta dimensão, complexos e não lineares, em simples relações geométricas com baixa dimensionalidade. Este método também pode ser utilizado para a classificação de imagens de sensoriamento remoto, pois permite a compressão de dados de alta dimensão preservando as relações topológicas dos dados primários. Este trabalho objetiva desenvolver uma metodologia eficaz para a utilização de mapas auto-organizáveis na detecção de mudanças. No presente estudo o SOFM é utilizado para a classificação não supervisionada de dados de sensoriamento remoto, considerando os seguintes atributos: espaciais (x, y), espectrais e temporais. O método é empregado na região oeste da Bahia, que teve recentemente um aumento significativo em monoculturas. Testes foram realizados com os parâmetros do SOFM com o objetivo de refinar o mapa de detecção demudanças. O SOFM possibilita uma melhor seleção de células e dos correspondentes vetores de peso, que mostram o processo de ordenação e agrupamento hierárquicodos dados. Esta informação é essencial para identificar mudanças ao longo do tempo. Um programa em linguagem C ++ do método proposto foi desenvolvido. ABSTRACT. Self-organizing feature maps (SOFM) consist of a type of artificial neural network that allows the conversion from high-dimensional data into simple geometric relationships with low-dimensionality. This method can also be used for classification of remote sensing images because it allows the compression of high dimensional data while preserving the most important topological and metric relationships of the primary data. This paper aims to develop an effective methodology forusing self-organizing maps in change detection. In this study, SOFM is used for unsupervised classification of remote sensing data, considering the following attributes: spatial (x and y), spectral and temporal. The method is tested and simulated in the western region of Bahia that has observed a significant increase in mechanized agriculture. Tests were performed with the SOFM parameters for the purpose of fine tuning a change detection map. The SOFM provides the best selection of cell and corresponding adjustment of weight vectors, which show the process of ordering and hierarchical clustering of the data. This information is essential to identify changes over time. All algorithms were implemented in C++ language.Keywords: unsupervised classification; land cover; multitemporal analysis; remote sensing

Paddy rice cropping patterns (PRCPs) play important roles in both agroecosystem modeling and food security. Although paddy rice maps have been generated over several regions using satellite observations, few studies have focused on mapping diverse smallholder PRCPs, which include crop rotation and are dominant cropping structures in South China. Here, an approach called the feature selection and hierarchical classification (FSHC) method was proposed to effectively identify paddy rice and its rotation types. Considering the cloudy and rainy weather in South China, a harmonized Landsat and Sentinel-2 (HLS) surface reflectance product was employed to increase high-quality observations. The FSHC method consists of three processes: cropping intensity mapping, feature selection, and decision tree (DT) model development. The FSHC performance was carefully evaluated using crop field samples obtained in 2018 and 2019. Results suggested that the derived cropping intensity map based on the Savitzky–Golay (S-G) filtered normalized difference vegetation index (NDVI) time series was reliable, with an overall accuracy greater than 93%. Additionally, the optimal spectral (i.e., normalized difference water index (NDWI) and land surface water index (LSWI)) and temporal (start-of-season (SOS) date) features for distinguishing different PRCPs were successfully identified, and these features are highly related to the critical growth stage of paddy rice. The developed DT model with three hierarchical levels based on optimal features performed satisfactorily, and the identification accuracy of each PRCP can be achieved approximately 85%. Furthermore, the FSHC method exhibited similar performances when mapping PRCPs in adjacent years. These results demonstrate that the proposed FSHC approach with HLS data can accurately extract diverse PRCPs over fragmented croplands; thus, this approach represents a promising opportunity for generating refined crop type maps.

To solve the problems of susceptibility to image noise, subjectivity of training sample selection, and inefficiency of state-of-the-art change detection methods with heterogeneous images, this study proposes a post-classification change detection method for heterogeneous images with improved training of hierarchical extreme learning machine (HELM). After smoothing the images to suppress noise, a sample selection method is defined to train the HELM for each image, in which the feature extraction is respectively implemented for heterogeneous images and the parameters need not be fine-tuned. Then, the multi-temporal feature maps extracted from the trained HELM are segmented to obtain classification maps and then compared to generate a change map with changed types. The proposed method is validated experimentally by using one set of synthetic aperture radar (SAR) images obtained from Sentinel-1, one set of optical images acquired from Google Earth, and two sets of heterogeneous SAR and optical images. The results show that compared to state-of-the-art change detection methods, the proposed method can improve the accuracy of change detection by more than 8% in terms of the kappa coefficient and greatly reduce run time regardless of the type of images used. Such enhancement reflects the robustness and superiority of the proposed method.

Abstract The luxury car market has demanding product development standards aimed at providing state-of-the-art features in the automotive domain. Handling performance is amongst the most important properties that must be assessed when developing a new car model. In this work, we analyse the problem of predicting subjective evaluations of automobiles handling performances from objective records of driving sessions. A record is a multi-dimensional time series describing the temporal evolution of the mechanical state of an automobile. A categorical variable quantifies the evaluations of handling properties. We describe an original deep learning system, featuring a denoising autoencoder and hierarchical attention mechanisms, that we designed to solve this task. Attention mechanisms intrinsically compute probability distributions over their inputs’ components. Combining this feature with the saliency maps technique, our system can compute heatmaps that provide a visual aid to identify the physical events conditioning its predictions.

<p><strong>Abstract.</strong> Hyperspectral images (HSIs) contain hundreds of spectral bands, providing high-resolution spectral information pertaining to the Earth’s surface. Additionally, abundant spatial contextual information can also be obtained simultaneously from a HSI. To characterize the properties of ground objects, classification is the most widely-used technology in the field of remote sensing, where each pixel in a HSI is assigned to a pre-defined class. Over the past decade, deep learning has attracted increasing attention in the machine-learning and computer-vision domains, due to its favourable performances for various types of tasks, and it has been successfully introduced to the remote-sensing community. Instead of utilizing the shallow features within in a given image, which is the approach that is generally adopted in other conventional classification methods, deep-learning algorithms can extract hierarchical features from raw HSI data. Within the deep-learning framework, recurrent neural networks (RNNs), which are able to encode sequential features, have exhibited promising capabilities and have achieved encouraging performances, especially for the natural-language processing and speech-recognition communities. As multi-temporal remote-sensing images can be readily obtained from increasing numbers of satellite and unmanned aircraft systems, and since analysis of such multi-temporal data comprises a critical issue within numerous research subfields, including land-cover and land-change analyses, and land-resource management, RNNs have been applied in recent studies in order to extract temporal sequential features from multi-temporal remote-sensing images for the purpose of image classification. Apart from using multi-temporal image datasets, RNNs can also be utilized on a single image, where the spectral feature/band of each individual pixel can be taken as a sequential feature for the input layer of RNNs. However, the application of such sequential feature extraction that relies on a single image still needs to be further investigated since applying RNNs to spectral bands will directly introduce more parameters that need to be optimized, consequently increasing the total training time.</p><p>In this study, we propose a novel RNN-based HSI classification framework. In this framework, unlabelled pixels obtained from a single image are considered when constructing sequential features. Two spatial similarity measurements, referred to as pixel-matching and block-matching, respectively, are employed to extract pixels that are “similar” to the target pixel. Then, the sequential feature of the target pixel is constructed by exploiting several of the most “similar” pixels and ordering them based on their similarities to the target pixel. The aforementioned two schemes are advantageous, as unlabelled pixels within the given HSI are taken into consideration for similarity measurement and sequential feature construction for the RNN model. Moreover, the block-matching scheme also takes advantage of spatial contextual information, which has been widely utilized in spatial-spectral-based HSI classification methods. To evaluate the proposed methods, two benchmark HSIs are used, including a HSI collected over Pavia University, Italy by the airborne Reflective Optics System Imaging Spectrometer (ROSIS) sensor, and an image acquired over the Salinas Valley, California, USA via the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor. Spatio-temporally coincident ground-reference data accompanies each of these respective HSIs. In addition, the proposed methods are compared with three state-of-the-art algorithms, including support vector machine (SVM), the 1-dimensional convolutional neural network (1DCNN), and the 1-dimensional RNN (1DRNN).</p><p>Experimental results indicate that our proposed methods achieve markedly better classification performance compared with the baseline algorithms on both datasets. For example, for the Pavia University image, the block-matching based RNN achieves the highest overall classification accuracy, with 94.32% accuracy, which is 9.87% higher than the next most accurate algorithm of the aforementioned three baseline methods, which in this case is the 1DCNN, with 84.45% overall accuracy. More specifically, the block-matching method performs better than the pixel-matching method in terms of both quantitative and qualitative assessments. Based on visual assessment/interpretation of the classification maps, it is apparent that “salt-and-pepper” noise is markedly alleviated; with block-matching, smoother classified images are generated compared with pixel-matching-based methods and the three baseline algorithms. Such results demonstrate the effectiveness of utilizing spatial contextual information in the similarity measurement.</p>

The astounding development of optical sensing imaging technology, coupled with the impressive improvements in machine learning algorithms, has increased our ability to understand and extract information from scenic events. In most cases, Convolution neural networks (CNNs) are largely adopted to infer knowledge due to their surprising success in automation, surveillance, and many other application domains. However, the convolution operations’ overwhelming computation demand has somewhat limited their use in remote sensing edge devices. In these platforms, real-time processing remains a challenging task due to the tight constraints on resources and power. Here, the transfer and processing of non-relevant image pixels act as a bottleneck on the entire system. It is possible to overcome this bottleneck by exploiting the high bandwidth available at the sensor interface by designing a CNN inference architecture near the sensor. This paper presents an attention-based pixel processing architecture to facilitate the CNN inference near the image sensor. We propose an efficient computation method to reduce the dynamic power by decreasing the overall computation of the convolution operations. The proposed method reduces redundancies by using a hierarchical optimization approach. The approach minimizes power consumption for convolution operations by exploiting the Spatio-temporal redundancies found in the incoming feature maps and performs computations only on selected regions based on their relevance score. The proposed design addresses problems related to the mapping of computations onto an array of processing elements (PEs) and introduces a suitable network structure for communication. The PEs are highly optimized to provide low latency and power for CNN applications. While designing the model, we exploit the concepts of biological vision systems to reduce computation and energy. We prototype the model in a Virtex UltraScale+ FPGA and implement it in Application Specific Integrated Circuit (ASIC) using the TSMC 90nm technology library. The results suggest that the proposed architecture significantly reduces dynamic power consumption and achieves high-speed up surpassing existing embedded processors’ computational capabilities.

Abstract. The restrictions of the analysis of natural processes which are observed at any point in space or time to a purely spatial or purely temporal domain may cause loss of information and larger prediction errors. Moreover, the arbitrary combinations of purely spatial and purely temporal models may not yield valid models for the space-time domain. For such processes the variation can be characterized by sophisticated spatio-temporal modeling. In the present study the composite spatio-temporal Bayesian maximum entropy (BME) method and transformed hierarchical Bayesian space-time interpolation are used in order to predict precipitation in Pakistan during the monsoon period. Monthly average precipitation data whose time domain is the monsoon period for the years 1974–2000 and whose spatial domain are various regions in Pakistan are considered. The prediction of space-time precipitation is applicable in many sectors of industry and economy in Pakistan especially; the agricultural sector. Mean field maps and prediction error maps for both methods are estimated and compared. In this paper it is shown that the transformed hierarchical Bayesian model is providing more accuracy and lower prediction error compared to the spatio-temporal Bayesian maximum entropy method; additionally, the transformed hierarchical Bayesian model also provides predictive distributions.

This dissertation is primarily an applied statistical modelling investigation, motivated by a case study comprising real data and real questions. Theoretical questions on modelling and computation of normalization constants arose from pursuit of these data analytic questions. The essence of the thesis can be described as follows. Consider binary data observed on a two-dimensional lattice. A common problem with such data is the ambiguity of zeroes recorded. These may represent zero response given some threshold (presence) or that the threshold has not been triggered (absence). Suppose that the researcher wishes to estimate the effects of covariates on the binary responses, whilst taking into account underlying spatial variation, which is itself of some interest. This situation arises in many contexts and the dingo, cypress and toad case studies described in the motivation chapter are examples of this. Two main approaches to modelling and inference are investigated in this thesis. The first is frequentist and based on generalized linear models, with spatial variation modelled by using a block structure or by smoothing the residuals spatially. The EM algorithm can be used to obtain point estimates, coupled with bootstrapping or asymptotic MLE estimates for standard errors. The second approach is Bayesian and based on a three- or four-tier hierarchical model, comprising a logistic regression with covariates for the data layer, a binary Markov Random field (MRF) for the underlying spatial process, and suitable priors for parameters in these main models. The three-parameter autologistic model is a particular MRF of interest. Markov chain Monte Carlo (MCMC) methods comprising hybrid Metropolis/Gibbs samplers is suitable for computation in this situation. Model performance can be gauged by MCMC diagnostics. Model choice can be assessed by incorporating another tier in the modelling hierarchy. This requires evaluation of a normalization constant, a notoriously difficult problem. Difficulty with estimating the normalization constant for the MRF can be overcome by using a path integral approach, although this is a highly computationally intensive method. Different methods of estimating ratios of normalization constants (N Cs) are investigated, including importance sampling Monte Carlo (ISMC), dependent Monte Carlo based on MCMC simulations (MCMC), and reverse logistic regression (RLR). I develop an idea present though not fully developed in the literature, and propose the Integrated mean canonical statistic (IMCS) method for estimating log NC ratios for binary MRFs. The IMCS method falls within the framework of the newly identified path sampling methods of Gelman & Meng (1998) and outperforms ISMC, MCMC and RLR. It also does not rely on simplifying assumptions, such as ignoring spatio-temporal dependence in the process. A thorough investigation is made of the application of IMCS to the three-parameter Autologistic model. This work introduces background computations required for the full implementation of the four-tier model in Chapter 7. Two different extensions of the three-tier model to a four-tier version are investigated. The first extension incorporates temporal dependence in the underlying spatio-temporal process. The second extensions allows the successes and failures in the data layer to depend on time. The MCMC computational method is extended to incorporate the extra layer. A major contribution of the thesis is the development of a fully Bayesian approach to inference for these hierarchical models for the first time. Note: The author of this thesis has agreed to make it open access but invites people downloading the thesis to send her an email via the 'Contact Author' function.

With the volume of data generated in today's internet-of-things, learning algorithms to extract and understand the underlying relations between the various attributes of data have gained momentum. This thesis is focused on learning algorithms to extract meaningful relations from the data using both unsupervised and supervised learning algorithms. Vector quantization techniques are popularly used for applications in contextual data clustering, data visualization and high-dimensional data exploration. Existing vector quantization techniques, such as, the K-means and its variants and those derived from the self-organizing maps consider the input data vector as a whole without prioritizing over individual coordinates. Motivated by applications requiring priorities over data coordinates, we develop a theory for clustering data with different priorities over the coordinates called the data-dependent priority-based soft vector quantization. Based on the input data distribution, the priorities over the data coordinates are learnt by estimating the marginal distributions over each coordinate. The number of neurons approximating each coordinate based on the priority are determined through a reinforcement learning algorithm. Analysis on the convergence of the proposed algorithm and the probability of misclassification are presented along with simulation results on various data sets. Self-organizing maps (SOM) are popularly used for applications in learning features, vector quantization, and recalling spatial input patterns. The adaptation rule in SOMs is based on the Euclidean distance between the input vector and the neuronal weight vector along with a neighborhood function that brings in topological arrangement of the neurons in the output space. It is capable of learning the spatial correlations among the data but fails to capture temporal correlations present in a sequence of inputs. We formulate a potential function based on a spatio-temporal metric and create hierarchical vector quantization feature maps by embedding memory structures similar to long short-term memories across the feature maps to learn the spatio-temporal correlations in the data across clusters. Error correction codes such as low density parity check codes are popularly used to enhance the performance of digital communication systems. The current decoding framework relies on exchanging beliefs over a Tanner graph, which the encoder and decoder are aware of. However, this information may not be available readily, for example, in covert communication. The main idea is to build a neural network to learn the encoder mappings in the absence of knowledge of the Tanner graph. We propose a scheme to learn the mappings using the back-propagation algorithm. We investigate into the choice of different cost functions and the number of hidden neurons for learning the encoding function. The proposed scheme is capable of learning the parity check equations over a binary field towards identifying the validity of a codeword. Simulation results over synthetic data show that our algorithm is indeed capable of learning the encoder mappings. We also propose an approach to identify noisy codes using uncertainty estimation and to decode them using autoencoders. In the next work, we consider the convolutional neural networks which are widely used in natural language processing, video analysis, and image recognition. However, the popularly used max-pooling layer discards most of the data, which is a drawback in applications, such as, prediction of video frames. We propose an adaptive prediction and classification network based on a data-dependent pooling architecture. We formulate a combined cost function for minimizing the prediction and classification errors. We also detect the presence of an unseen class during testing for digit prediction in videos.

Forecasting of air pollutant concentration, which is influenced by air pollution accumulation, traffic flow and industrial emissions, has attracted extensive attention for decades. In this paper, we propose a spatio-temporal attention convolutional long short term memory neural networks (Attention-CNN-LSTM) for air pollutant concentration forecasting. Firstly, we analyze the Granger causalities between different stations and establish a hyperparametric Gaussian vector weight function to determine spatial autocorrelation variables, which is used as part of the input feature. Secondly, convolutional neural networks (CNN) is employed to extract the temporal dependence and spatial correlation of the input, while feature maps and channels are weighted by attention mechanism, so as to improve the effectiveness of the features. Finally, a depth long short term memory (LSTM) based time series predictor is established for learning the long-term and short-term dependence of pollutant concentration. In order to reduce the effect of diverse complex factors on LSTM, inherent features are extracted from historical air pollutant concentration data meteorological data and timestamp information are incorporated into the proposed model. Extensive experiments were performed using the Attention-CNNLSTM, autoregressive integrated moving average (ARIMA), support vector regression (SVR), traditional LSTM and CNN, respectively. The results demonstrated that the feasibility and practicability of Attention-CNN-LSTM on estimating CO and NO concentration.

Abstract Geologic CO2 sequestration and CO2 enhanced oil recovery (EOR) have received significant attention from the scientific community as a response to climate change from greenhouse gases. Safe and efficient management of a CO2 injection site requires spatio-temporal tracking of the CO2 plume in the reservoir during geologic sequestration. The goal of this paper is to develop robust modeling and monitoring technologies for imaging and visualization of the CO2 plume using routine pressure/temperature measurements. The streamline-based technology has proven to be effective and efficient for reconciling geologic models to various types of reservoir dynamic response. In this paper, we first extend the streamline-based data integration approach to incorporate distributed temperature sensor (DTS) data using the concept of thermal tracer travel time. Then, a hierarchical workflow composed of evolutionary and streamline methods is employed to jointly history match the DTS and pressure data. Finally, CO2 saturation and streamline maps are used to visualize the CO2 plume movement during the sequestration process. The power and utility of our approach are demonstrated using both synthetic and field applications. We first validate the streamline-based DTS data inversion using a synthetic example. Next, the hierarchical workflow is applied to a carbon sequestration project in a carbonate reef reservoir within the Northern Niagaran Pinnacle Reef Trend in Michigan, USA. The monitoring data set consists of distributed temperature sensing (DTS) data acquired at the injection well and a monitoring well, flowing bottom-hole pressure data at the injection well, and time-lapse pressure measurements at several locations along the monitoring well. The history matching results indicate that the CO2 movement is mostly restricted to the intended zones of injection which is consistent with an independent warmback analysis of the temperature data. The novelty of this work is the streamline-based history matching method for the DTS data and its field application to the Department of Engergy regional carbon sequestration project in Michigan.