Journal articles on the topic 'Variational Infernce'
To see the other types of publications on this topic, follow the link: Variational Infernce.
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the top 50 journal articles for your research on the topic 'Variational Infernce.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Browse journal articles on a wide variety of disciplines and organise your bibliography correctly.
1
Yun-Shan Sun, Yun-Shan Sun, Hong-Yan Xu Yun-Shan Sun, and Yan-Qin Li Hong-Yan Xu. "Missing Data Interpolation with Variational Bayesian Inference for Socio-economic Statistics Applications." 電腦學刊 33, no. 2 (April 2022): 169–76. http://dx.doi.org/10.53106/199115992022043302015.
Abstract:
<p>The information integrity is needed to solving socio-economic statistical problems. However, the information integrity is destroyed by missing data which is caused by various subjective and objective reasons. So the missing data interpolation is used to supplement missing data. In this paper, missing data interpolation with variational Bayesian inference is proposed. This method is combined with Gaussian model to approximate the posterior distribution to obtain complete data. The experiments include two datasets (artificial dataset and actual dataset) based on three missing ratios separately. The missing data interpolation performance of variational Bayesian method is compared with that which is obtained by mean interpolation and K-nearest neighbor interpolation methods separately in MSE (Mean Square Error) and MAPE (Mean Absolute Percentage Error). The experimental results show that the proposed variational Bayesian method is better in MSE and MAPE.</p> <p> </p>
2
Yun-Shan Sun, Yun-Shan Sun, Hong-Yan Xu Yun-Shan Sun, and Yan-Qin Li Hong-Yan Xu. "Missing Data Interpolation with Variational Bayesian Inference for Socio-economic Statistics Applications." 電腦學刊 33, no. 2 (April 2022): 169–76. http://dx.doi.org/10.53106/199115992022043302015.
Abstract:
<p>The information integrity is needed to solving socio-economic statistical problems. However, the information integrity is destroyed by missing data which is caused by various subjective and objective reasons. So the missing data interpolation is used to supplement missing data. In this paper, missing data interpolation with variational Bayesian inference is proposed. This method is combined with Gaussian model to approximate the posterior distribution to obtain complete data. The experiments include two datasets (artificial dataset and actual dataset) based on three missing ratios separately. The missing data interpolation performance of variational Bayesian method is compared with that which is obtained by mean interpolation and K-nearest neighbor interpolation methods separately in MSE (Mean Square Error) and MAPE (Mean Absolute Percentage Error). The experimental results show that the proposed variational Bayesian method is better in MSE and MAPE.</p> <p> </p>
3
Jaakkola, T. S., and M. I. Jordan. "Variational Probabilistic Inference and the QMR-DT Network." Journal of Artificial Intelligence Research 10 (May 1, 1999): 291–322. http://dx.doi.org/10.1613/jair.583.
Abstract:
We describe a variational approximation method for efficient inference in large-scale probabilistic models. Variational methods are deterministic procedures that provide approximations to marginal and conditional probabilities of interest. They provide alternatives to approximate inference methods based on stochastic sampling or search. We describe a variational approach to the problem of diagnostic inference in the `Quick Medical Reference' (QMR) network. The QMR network is a large-scale probabilistic graphical model built on statistical and expert knowledge. Exact probabilistic inference is infeasible in this model for all but a small set of cases. We evaluate our variational inference algorithm on a large set of diagnostic test cases, comparing the algorithm to a state-of-the-art stochastic sampling method.
4
Unlu, Ali, and Laurence Aitchison. "Gradient Regularization as Approximate Variational Inference." Entropy 23, no. 12 (December 3, 2021): 1629. http://dx.doi.org/10.3390/e23121629.
Abstract:
We developed Variational Laplace for Bayesian neural networks (BNNs), which exploits a local approximation of the curvature of the likelihood to estimate the ELBO without the need for stochastic sampling of the neural-network weights. The Variational Laplace objective is simple to evaluate, as it is the log-likelihood plus weight-decay, plus a squared-gradient regularizer. Variational Laplace gave better test performance and expected calibration errors than maximum a posteriori inference and standard sampling-based variational inference, despite using the same variational approximate posterior. Finally, we emphasize the care needed in benchmarking standard VI, as there is a risk of stopping before the variance parameters have converged. We show that early-stopping can be avoided by increasing the learning rate for the variance parameters.
5
Merlo, A., A. Pavone, D. Böckenhoff, E. Pasch, G. Fuchert, K. J. Brunner, K. Rahbarnia, et al. "Accelerated Bayesian inference of plasma profiles with self-consistent MHD equilibria at W7-X via neural networks." Journal of Instrumentation 18, no. 11 (November 1, 2023): P11012. http://dx.doi.org/10.1088/1748-0221/18/11/p11012.
Abstract:
Abstract High-β operations require a fast and robust inference of plasma parameters with a self-consistent magnetohydrodynamic (MHD) equilibrium. Precalculated MHD equilibria are usually employed at Wendelstein 7-X (W7-X) due to the high computational cost. To address this, we couple a physics-regularized artificial neural network (NN) model that approximates the ideal-MHD equilibrium with the Bayesian modeling framework Minerva. We show the fast and robust inference of plasma profiles (electron temperature and density) with a self-consistent MHD equilibrium approximated by the NN model. We investigate the robustness of the inference across diverse synthetic W7-X plasma scenarios. The inferred plasma parameters and their uncertainties are compatible with the parameters inferred using the variational moments equilibrium code (VMEC), and the inference time is reduced by more than two orders of magnitude. This work suggests that MHD self-consistent inferences of plasma parameters can be performed between shots.
6
Becker, McCoy R., Alexander K. Lew, Xiaoyan Wang, Matin Ghavami, Mathieu Huot, Martin C. Rinard, and Vikash K. Mansinghka. "Probabilistic Programming with Programmable Variational Inference." Proceedings of the ACM on Programming Languages 8, PLDI (June 20, 2024): 2123–47. http://dx.doi.org/10.1145/3656463.
Abstract:
Compared to the wide array of advanced Monte Carlo methods supported by modern probabilistic programming languages (PPLs), PPL support for variational inference (VI) is less developed: users are typically limited to a predefined selection of variational objectives and gradient estimators, which are implemented monolithically (and without formal correctness arguments) in PPL backends. In this paper, we propose a more modular approach to supporting variational inference in PPLs, based on compositional program transformation. In our approach, variational objectives are expressed as programs, that may employ first-class constructs for computing densities of and expected values under user-defined models and variational families. We then transform these programs systematically into unbiased gradient estimators for optimizing the objectives they define. Our design makes it possible to prove unbiasedness by reasoning modularly about many interacting concerns in PPL implementations of variational inference, including automatic differentiation, density accumulation, tracing, and the application of unbiased gradient estimation strategies. Additionally, relative to existing support for VI in PPLs, our design increases expressiveness along three axes: (1) it supports an open-ended set of user-defined variational objectives, rather than a fixed menu of options; (2) it supports a combinatorial space of gradient estimation strategies, many not automated by today’s PPLs; and (3) it supports a broader class of models and variational families, because it supports constructs for approximate marginalization and normalization (previously introduced for Monte Carlo inference). We implement our approach in an extension to the Gen probabilistic programming system (genjax.vi, implemented in JAX), and evaluate our automation on several deep generative modeling tasks, showing minimal performance overhead vs. hand-coded implementations and performance competitive with well-established open-source PPLs.
7
Fourment, Mathieu, and Aaron E. Darling. "Evaluating probabilistic programming and fast variational Bayesian inference in phylogenetics." PeerJ 7 (December 18, 2019): e8272. http://dx.doi.org/10.7717/peerj.8272.
Abstract:
Recent advances in statistical machine learning techniques have led to the creation of probabilistic programming frameworks. These frameworks enable probabilistic models to be rapidly prototyped and fit to data using scalable approximation methods such as variational inference. In this work, we explore the use of the Stan language for probabilistic programming in application to phylogenetic models. We show that many commonly used phylogenetic models including the general time reversible substitution model, rate heterogeneity among sites, and a range of coalescent models can be implemented using a probabilistic programming language. The posterior probability distributions obtained via the black box variational inference engine in Stan were compared to those obtained with reference implementations of Markov chain Monte Carlo (MCMC) for phylogenetic inference. We find that black box variational inference in Stan is less accurate than MCMC methods for phylogenetic models, but requires far less compute time. Finally, we evaluate a custom implementation of mean-field variational inference on the Jukes–Cantor substitution model and show that a specialized implementation of variational inference can be two orders of magnitude faster and more accurate than a general purpose probabilistic implementation.
8
Frank, Philipp, Reimar Leike, and Torsten A. Enßlin. "Geometric Variational Inference." Entropy 23, no. 7 (July 2, 2021): 853. http://dx.doi.org/10.3390/e23070853.
Abstract:
Efficiently accessing the information contained in non-linear and high dimensional probability distributions remains a core challenge in modern statistics. Traditionally, estimators that go beyond point estimates are either categorized as Variational Inference (VI) or Markov-Chain Monte-Carlo (MCMC) techniques. While MCMC methods that utilize the geometric properties of continuous probability distributions to increase their efficiency have been proposed, VI methods rarely use the geometry. This work aims to fill this gap and proposes geometric Variational Inference (geoVI), a method based on Riemannian geometry and the Fisher information metric. It is used to construct a coordinate transformation that relates the Riemannian manifold associated with the metric to Euclidean space. The distribution, expressed in the coordinate system induced by the transformation, takes a particularly simple form that allows for an accurate variational approximation by a normal distribution. Furthermore, the algorithmic structure allows for an efficient implementation of geoVI which is demonstrated on multiple examples, ranging from low-dimensional illustrative ones to non-linear, hierarchical Bayesian inverse problems in thousands of dimensions.
9
Kiselev, Igor. "Variational BEJG Solvers for Marginal-MAP Inference with Accurate Approximation of B-Conditional Entropy." Proceedings of the AAAI Conference on Artificial Intelligence 33 (July 17, 2019): 9957–58. http://dx.doi.org/10.1609/aaai.v33i01.33019957.
Abstract:
Previously proposed variational techniques for approximate MMAP inference in complex graphical models of high-order factors relax a dual variational objective function to obtain its tractable approximation, and further perform MMAP inference in the resulting simplified graphical model, where the sub-graph with decision variables is assumed to be a disconnected forest. In contrast, we developed novel variational MMAP inference algorithms and proximal convergent solvers, where we can improve the approximation accuracy while better preserving the original MMAP query by designing such a dual variational objective function that an upper bound approximation is applied only to the entropy of decision variables. We evaluate the proposed algorithms on both simulated synthetic datasets and diagnostic Bayesian networks taken from the UAI inference challenge, and our solvers outperform other variational algorithms in a majority of reported cases. Additionally, we demonstrate the important real-life application of the proposed variational approaches to solve complex tasks of policy optimization by MMAP inference, and performance of the implemented approximation algorithms is compared. Here, we demonstrate that the original task of optimizing POMDP controllers can be approached by its reformulation as the equivalent problem of marginal-MAP inference in a novel single-DBN generative model, which guarantees that the control policies computed by probabilistic inference over this model are optimal in the traditional sense. Our motivation for approaching the planning problem through probabilistic inference in graphical models is explained by the fact that by transforming a Markovian planning problem into the task of probabilistic inference (a marginal MAP problem) and applying belief propagation techniques in generative models, we can achieve a computational complexity reduction from PSPACE-complete or NEXP-complete to NPPP-complete in comparison to solving the POMDP and Dec-POMDP models respectively search vs. dynamic programming).
10
Chi, Jinjin, Zhichao Zhang, Zhiyao Yang, Jihong Ouyang, and Hongbin Pei. "Generalized Variational Inference via Optimal Transport." Proceedings of the AAAI Conference on Artificial Intelligence 38, no. 10 (March 24, 2024): 11534–42. http://dx.doi.org/10.1609/aaai.v38i10.29035.
Abstract:
Variational Inference (VI) has gained popularity as a flexible approximate inference scheme for computing posterior distributions in Bayesian models. Original VI methods use Kullback-Leibler (KL) divergence to construct variational objectives. However, KL divergence has zero-forcing behavior and is completely agnostic to the metric of the underlying data distribution, resulting in bad approximations. To alleviate this issue, we propose a new variational objective by using Optimal Transport (OT) distance, which is a metric-aware divergence, to measure the difference between approximate posteriors and priors. The superior performance of OT distance enables us to learn more accurate approximations. We further enhance the objective by gradually including the OT term using a hyperparameter λ for over-parameterized models. We develop a Variational inference method with OT (VOT) which presents a gradient-based black-box framework for solving Bayesian models, even when the density function of approximate distribution is not available. We provide the consistency analysis of approximate posteriors and demonstrate the practical effectiveness on Bayesian neural networks and variational autoencoders.
11
Havasi, Marton, Jasper Snoek, Dustin Tran, Jonathan Gordon, and José Miguel Hernández-Lobato. "Sampling the Variational Posterior with Local Refinement." Entropy 23, no. 11 (November 8, 2021): 1475. http://dx.doi.org/10.3390/e23111475.
Abstract:
Variational inference is an optimization-based method for approximating the posterior distribution of the parameters in Bayesian probabilistic models. A key challenge of variational inference is to approximate the posterior with a distribution that is computationally tractable yet sufficiently expressive. We propose a novel method for generating samples from a highly flexible variational approximation. The method starts with a coarse initial approximation and generates samples by refining it in selected, local regions. This allows the samples to capture dependencies and multi-modality in the posterior, even when these are absent from the initial approximation. We demonstrate theoretically that our method always improves the quality of the approximation (as measured by the evidence lower bound). In experiments, our method consistently outperforms recent variational inference methods in terms of log-likelihood and ELBO across three example tasks: the Eight-Schools example (an inference task in a hierarchical model), training a ResNet-20 (Bayesian inference in a large neural network), and the Mushroom task (posterior sampling in a contextual bandit problem).
12
Krishnan, Ranganath, Mahesh Subedar, and Omesh Tickoo. "Specifying Weight Priors in Bayesian Deep Neural Networks with Empirical Bayes." Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 04 (April 3, 2020): 4477–84. http://dx.doi.org/10.1609/aaai.v34i04.5875.
Abstract:
Stochastic variational inference for Bayesian deep neural network (DNN) requires specifying priors and approximate posterior distributions over neural network weights. Specifying meaningful weight priors is a challenging problem, particularly for scaling variational inference to deeper architectures involving high dimensional weight space. We propose MOdel Priors with Empirical Bayes using DNN (MOPED) method to choose informed weight priors in Bayesian neural networks. We formulate a two-stage hierarchical modeling, first find the maximum likelihood estimates of weights with DNN, and then set the weight priors using empirical Bayes approach to infer the posterior with variational inference. We empirically evaluate the proposed approach on real-world tasks including image classification, video activity recognition and audio classification with varying complex neural network architectures. We also evaluate our proposed approach on diabetic retinopathy diagnosis task and benchmark with the state-of-the-art Bayesian deep learning techniques. We demonstrate MOPED method enables scalable variational inference and provides reliable uncertainty quantification.
13
Grimmer, Justin. "An Introduction to Bayesian Inference via Variational Approximations." Political Analysis 19, no. 1 (2011): 32–47. http://dx.doi.org/10.1093/pan/mpq027.
Abstract:
Markov chain Monte Carlo (MCMC) methods have facilitated an explosion of interest in Bayesian methods. MCMC is an incredibly useful and important tool but can face difficulties when used to estimate complex posteriors or models applied to large data sets. In this paper, we show how a recently developed tool in computer science for fitting Bayesian models, variational approximations, can be used to facilitate the application of Bayesian models to political science data. Variational approximations are often much faster than MCMC for fully Bayesian inference and in some instances facilitate the estimation of models that would be otherwise impossible to estimate. As a deterministic posterior approximation method, variational approximations are guaranteed to converge and convergence is easily assessed. But variational approximations do have some limitations, which we detail below. Therefore, variational approximations are best suited to problems when fully Bayesian inference would otherwise be impossible. Through a series of examples, we demonstrate how variational approximations are useful for a variety of political science research. This includes models to describe legislative voting blocs and statistical models for political texts. The code that implements the models in this paper is available in the supplementary material.
14
Perez, Iker, and Giuliano Casale. "Variational inference for Markovian queueing networks." Advances in Applied Probability 53, no. 3 (September 2021): 687–715. http://dx.doi.org/10.1017/apr.2020.72.
Abstract:
AbstractQueueing networks are stochastic systems formed by interconnected resources routing and serving jobs. They induce jump processes with distinctive properties, and find widespread use in inferential tasks. Here, service rates for jobs and potential bottlenecks in the routing mechanism must be estimated from a reduced set of observations. However, this calls for the derivation of complex conditional density representations, over both the stochastic network trajectories and the rates, which is considered an intractable problem. Numerical simulation procedures designed for this purpose do not scale, because of high computational costs; furthermore, variational approaches relying on approximating measures and full independence assumptions are unsuitable. In this paper, we offer a probabilistic interpretation of variational methods applied to inference tasks with queueing networks, and show that approximating measure choices routinely used with jump processes yield ill-defined optimization problems. Yet we demonstrate that it is still possible to enable a variational inferential task, by considering a novel space expansion treatment over an analogous counting process for job transitions. We present and compare exemplary use cases with practical queueing networks, showing that our framework offers an efficient and improved alternative where existing variational or numerically intensive solutions fail.
15
Ma, Jirong, Qinghua Ma, Shujun Yang, Jianqiang zheng, and Shuaiwei Wang. "Survey of state estimation based on variational bayesian inference." Journal of Physics: Conference Series 2352, no. 1 (October 1, 2022): 012002. http://dx.doi.org/10.1088/1742-6596/2352/1/012002.
Abstract:
State estimation problem in multiple target tracking usually faces high-dimensional uncertainty, including target model uncertainty, data association uncertainty, deep coupling and so on. Variational Bayesian inference provides a way to get the approximation for high-dimensional intractable problem. In this paper, we give the survey of state estimation based on variational Bayesian inference.
16
Benallal, Abdellah, Nawal Cheggaga, Adrian Ilinca, Selma Tchoketch-Kebir, Camelia Ait Hammouda, and Noureddine Barka. "Bayesian Inference-Based Energy Management Strategy for Techno-Economic Optimization of a Hybrid Microgrid." Energies 17, no. 1 (December 24, 2023): 114. http://dx.doi.org/10.3390/en17010114.
Abstract:
This paper introduces a novel techno-economic feasibility analysis of energy management utilizing the Homer software v3.14.5 environment for an independent hybrid microgrid. This study focuses on a school with twelve classes, classifying the electrical components of the total load into three priority profiles: green, orange, and red. The developed approach involves implementing demand management for the hybrid microgrid through Bayesian inference, emphasizing goal-directed decision making within embodied or active inference. The Bayesian inference employs three parameters as inputs: the total production of the hybrid system, the load demand, and the state of charge of batteries to determine the supply for charge consumption. By framing decision making and action selection as variational Bayesian inference, the approach transforms the problem from selecting an optimal action to making optimal inferences about control. The results have led to the creation of a Bayesian inference approach for the new demand management strategy, applicable to load profiles resembling those of commercial and service institutions. Furthermore, Bayesian inference management has successfully reduced the total unmet load on secondary and tertiary priority charges to 1.9%, thereby decreasing the net present cost, initial cost, and energy cost by 37.93%, 41.43%, and 36.71%, respectively. This significant cost reduction has enabled a substantial decrease in investments for the same total energy consumption.
17
Friston, Karl, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, and Giovanni Pezzulo. "Active Inference: A Process Theory." Neural Computation 29, no. 1 (January 2017): 1–49. http://dx.doi.org/10.1162/neco_a_00912.
Abstract:
This article describes a process theory based on active inference and belief propagation. Starting from the premise that all neuronal processing (and action selection) can be explained by maximizing Bayesian model evidence—or minimizing variational free energy—we ask whether neuronal responses can be described as a gradient descent on variational free energy. Using a standard (Markov decision process) generative model, we derive the neuronal dynamics implicit in this description and reproduce a remarkable range of well-characterized neuronal phenomena. These include repetition suppression, mismatch negativity, violation responses, place-cell activity, phase precession, theta sequences, theta-gamma coupling, evidence accumulation, race-to-bound dynamics, and transfer of dopamine responses. Furthermore, the (approximately Bayes’ optimal) behavior prescribed by these dynamics has a degree of face validity, providing a formal explanation for reward seeking, context learning, and epistemic foraging. Technically, the fact that a gradient descent appears to be a valid description of neuronal activity means that variational free energy is a Lyapunov function for neuronal dynamics, which therefore conform to Hamilton’s principle of least action.
18
Park, Cheoneum, and Changki Lee. "Sentimental Analysis of Korean Movie Review using Variational Inference and RNN based on BERT." KIISE Transactions on Computing Practices 25, no. 11 (November 30, 2019): 552–58. http://dx.doi.org/10.5626/ktcp.2019.25.11.552.
19
Zheng, Kai, Xianjun Yang, Yilei Wang, Yingjie Wu, and Xianghan Zheng. "Collaborative filtering recommendation algorithm based on variational inference." International Journal of Crowd Science 4, no. 1 (January 31, 2020): 31–44. http://dx.doi.org/10.1108/ijcs-10-2019-0030.
Abstract:
Purpose The purpose of this paper is to alleviate the problem of poor robustness and over-fitting caused by large-scale data in collaborative filtering recommendation algorithms. Design/methodology/approach Interpreting user behavior from the probabilistic perspective of hidden variables is helpful to improve robustness and over-fitting problems. Constructing a recommendation network by variational inference can effectively solve the complex distribution calculation in the probabilistic recommendation model. Based on the aforementioned analysis, this paper uses variational auto-encoder to construct a generating network, which can restore user-rating data to solve the problem of poor robustness and over-fitting caused by large-scale data. Meanwhile, for the existing KL-vanishing problem in the variational inference deep learning model, this paper optimizes the model by the KL annealing and Free Bits methods. Findings The effect of the basic model is considerably improved after using the KL annealing or Free Bits method to solve KL vanishing. The proposed models evidently perform worse than competitors on small data sets, such as MovieLens 1 M. By contrast, they have better effects on large data sets such as MovieLens 10 M and MovieLens 20 M. Originality/value This paper presents the usage of the variational inference model for collaborative filtering recommendation and introduces the KL annealing and Free Bits methods to improve the basic model effect. Because the variational inference training denotes the probability distribution of the hidden vector, the problem of poor robustness and overfitting is alleviated. When the amount of data is relatively large in the actual application scenario, the probability distribution of the fitted actual data can better represent the user and the item. Therefore, using variational inference for collaborative filtering recommendation is of practical value.
20
Champion, Théophile, Marek Grześ, and Howard Bowman. "Realizing Active Inference in Variational Message Passing: The Outcome-Blind Certainty Seeker." Neural Computation 33, no. 10 (September 16, 2021): 2762–826. http://dx.doi.org/10.1162/neco_a_01422.
Abstract:
Abstract Active inference is a state-of-the-art framework in neuroscience that offers a unified theory of brain function. It is also proposed as a framework for planning in AI. Unfortunately, the complex mathematics required to create new models can impede application of active inference in neuroscience and AI research. This letter addresses this problem by providing a complete mathematical treatment of the active inference framework in discrete time and state spaces and the derivation of the update equations for any new model. We leverage the theoretical connection between active inference and variational message passing as described by John Winn and Christopher M. Bishop in 2005. Since variational message passing is a well-defined methodology for deriving Bayesian belief update equations, this letter opens the door to advanced generative models for active inference. We show that using a fully factorized variational distribution simplifies the expected free energy, which furnishes priors over policies so that agents seek unambiguous states. Finally, we consider future extensions that support deep tree searches for sequential policy optimization based on structure learning and belief propagation.
21
Ahn, Sungsoo, Michael Chertkov, and Jinwoo Shin. "Gauging variational inference." Journal of Statistical Mechanics: Theory and Experiment 2019, no. 12 (December 20, 2019): 124015. http://dx.doi.org/10.1088/1742-5468/ab3217.
22
Lian, Huiqiang, Bing Liu, and Pengyuan Li. "A fuel sales forecast method based on variational Bayesian structural time series." Journal of High Speed Networks 27, no. 1 (March 29, 2021): 45–66. http://dx.doi.org/10.3233/jhs-210651.
Abstract:
Fuel prices, which are of broad concern to the general public, are always seen as a challenging research topic. This paper proposes a variational Bayesian structural time-series model (STM) to effectively process complex fuel sales data online and provide real-time forecasting of fuel sales. While a traditional STM normally uses a probability model and the Markov chain Monte Carlo (MCMC) method to process change points, using the MCMC method to train the online model can be difficult given a relatively heavy computing load and time consumption. We thus consider the variational Bayesian STM, which uses variational Bayesian inference to make a reliable judgment of the trend change points without relying on artificial prior information, for our prediction method. With the inferences being driven by the data, our model passes the quantitative uncertainties to the forecast stage of the time series, which improves the robustness and reliability of the model. After conducting several experiments by using a self-collected dataset, we show that compared with a traditional STM, the proposed model has significantly shorter computing times for approximate forecast precision. Moreover, our model improves the forecast efficiency for fuel sales and the synergy of the distributed forecast platform based on an architecture of network.
23
Zalman (Oshri), Dana, and Shai Fine. "Variational Inference via Rényi Bound Optimization and Multiple-Source Adaptation." Entropy 25, no. 10 (October 20, 2023): 1468. http://dx.doi.org/10.3390/e25101468.
Abstract:
Variational inference provides a way to approximate probability densities through optimization. It does so by optimizing an upper or a lower bound of the likelihood of the observed data (the evidence). The classic variational inference approach suggests maximizing the Evidence Lower Bound (ELBO). Recent studies proposed to optimize the variational Rényi bound (VR) and the χ upper bound. However, these estimates, which are based on the Monte Carlo (MC) approximation, either underestimate the bound or exhibit a high variance. In this work, we introduce a new upper bound, termed the Variational Rényi Log Upper bound (VRLU), which is based on the existing VR bound. In contrast to the existing VR bound, the MC approximation of the VRLU bound maintains the upper bound property. Furthermore, we devise a (sandwiched) upper–lower bound variational inference method, termed the Variational Rényi Sandwich (VRS), to jointly optimize the upper and lower bounds. We present a set of experiments, designed to evaluate the new VRLU bound and to compare the VRS method with the classic Variational Autoencoder (VAE) and the VR methods. Next, we apply the VRS approximation to the Multiple-Source Adaptation problem (MSA). MSA is a real-world scenario where data are collected from multiple sources that differ from one another by their probability distribution over the input space. The main aim is to combine fairly accurate predictive models from these sources and create an accurate model for new, mixed target domains. However, many domain adaptation methods assume prior knowledge of the data distribution in the source domains. In this work, we apply the suggested VRS density estimate to the Multiple-Source Adaptation problem (MSA) and show, both theoretically and empirically, that it provides tighter error bounds and improved performance, compared to leading MSA methods.
24
Hubin, Aliaksandr, and Geir Storvik. "Sparse Bayesian Neural Networks: Bridging Model and Parameter Uncertainty through Scalable Variational Inference." Mathematics 12, no. 6 (March 7, 2024): 788. http://dx.doi.org/10.3390/math12060788.
Abstract:
Bayesian neural networks (BNNs) have recently regained a significant amount of attention in the deep learning community due to the development of scalable approximate Bayesian inference techniques. There are several advantages of using a Bayesian approach: parameter and prediction uncertainties become easily available, facilitating more rigorous statistical analysis. Furthermore, prior knowledge can be incorporated. However, the construction of scalable techniques that combine both structural and parameter uncertainty remains a challenge. In this paper, we apply the concept of model uncertainty as a framework for structural learning in BNNs and, hence, make inferences in the joint space of structures/models and parameters. Moreover, we suggest an adaptation of a scalable variational inference approach with reparametrization of marginal inclusion probabilities to incorporate the model space constraints. Experimental results on a range of benchmark datasets show that we obtain comparable accuracy results with the competing models, but based on methods that are much more sparse than ordinary BNNs.
25
Zhao, Shengjia, Jiaming Song, and Stefano Ermon. "InfoVAE: Balancing Learning and Inference in Variational Autoencoders." Proceedings of the AAAI Conference on Artificial Intelligence 33 (July 17, 2019): 5885–92. http://dx.doi.org/10.1609/aaai.v33i01.33015885.
Abstract:
A key advance in learning generative models is the use of amortized inference distributions that are jointly trained with the models. We find that existing training objectives for variational autoencoders can lead to inaccurate amortized inference distributions and, in some cases, improving the objective provably degrades the inference quality. In addition, it has been observed that variational autoencoders tend to ignore the latent variables when combined with a decoding distribution that is too flexible. We again identify the cause in existing training criteria and propose a new class of objectives (Info-VAE) that mitigate these problems. We show that our model can significantly improve the quality of the variational posterior and can make effective use of the latent features regardless of the flexibility of the decoding distribution. Through extensive qualitative and quantitative analyses, we demonstrate that our models outperform competing approaches on multiple performance metrics
26
Frey, Brendan J., and Geoffrey E. Hinton. "Variational Learning in Nonlinear Gaussian Belief Networks." Neural Computation 11, no. 1 (January 1, 1999): 193–213. http://dx.doi.org/10.1162/089976699300016872.
Abstract:
We view perceptual tasks such as vision and speech recognition as inference problems where the goal is to estimate the posterior distribution over latent variables (e.g., depth in stereo vision) given the sensory input. The recent flurry of research in independent component analysis exemplifies the importance of inferring the continuous-valued latent variables of input data. The latent variables found by this method are linearly related to the input, but perception requires nonlinear inferences such as classification and depth estimation. In this article, we present a unifying framework for stochastic neural networks with nonlinear latent variables. Nonlinear units are obtained by passing the outputs of linear gaussian units through various nonlinearities. We present a general variational method that maximizes a lower bound on the likelihood of a training set and give results on two visual feature extraction problems. We also show how the variational method can be used for pattern classification and compare the performance of these nonlinear networks with other methods on the problem of handwritten digit recognition.
27
Yamaguchi, Kazuhiro, and Kensuke Okada. "Variational Bayes Inference for the DINA Model." Journal of Educational and Behavioral Statistics 45, no. 5 (March 31, 2020): 569–97. http://dx.doi.org/10.3102/1076998620911934.
Abstract:
In this article, we propose a variational Bayes (VB) inference method for the deterministic input noisy AND gate model of cognitive diagnostic assessment. The proposed method, which applies the iterative algorithm for optimization, is derived based on the optimal variational posteriors of the model parameters. The proposed VB inference enables much faster computation than the existing Markov chain Monte Carlo (MCMC) method, while still offering the benefits of a full Bayesian framework. A simulation study revealed that the proposed VB estimation adequately recovered the parameter values. Moreover, an example using real data revealed that the proposed VB inference method provided similar estimates to MCMC estimation with much faster computation.
28
Dong, Ping, Jianhua Cheng, and Liqiang Liu. "A Novel Anti-Jamming Technique for INS/GNSS Integration Based on Black Box Variational Inference." Applied Sciences 11, no. 8 (April 19, 2021): 3664. http://dx.doi.org/10.3390/app11083664.
Abstract:
In this paper, a novel anti-jamming technique based on black box variational inference for INS/GNSS integration with time-varying measurement noise covariance matrices is presented. We proved that the time-varying measurement noise is more similar to the Gaussian distribution with time-varying mean value than to the Inv-Gamma or Inv-Wishart distribution found by Kullback–Leibler divergence. Therefore, we assumed the prior distribution of measurement noise covariance matrices as Gaussian, and calculated the Gaussian parameters by the black box variational inference method. Finally, we obtained the measurement noise covariance matrices by using the Gaussian parameters. The experimental results illustrate that the proposed algorithm performs better in resisting time-varying measurement noise than the existing Variational Bayesian adaptive filter.
29
Yamaguchi, Nobuhiko. "Constructing Generative Topographic Mapping by Variational Bayes with ARD Hierarchical Prior." Journal of Advanced Computational Intelligence and Intelligent Informatics 17, no. 4 (July 20, 2013): 473–79. http://dx.doi.org/10.20965/jaciii.2013.p0473.
Abstract:
Generative Topographic Mapping (GTM) is a nonlinear latent variable model introduced as a data visualization technique by Bishop et al. In this paper, we focus on variational Bayesian inference in GTM. Variational Bayesian GTM, first proposed by Olier et al., uses a single regularization term and regularization parameter to avoid overfitting and therefore cannot be used to control the degree of regularization locally. To overcome this problem, we propose variational Bayesian inference with Automatic Relevance Determination (ARD) hierarchical prior for use with GTM. The proposed model uses multiple regularization parameters and therefore can be used to control the degree of regularization in local areas of data space individually. Several experiments show that GTM that we propose provides better visualization than conventional GTM approaches.
30
Galy-Fajou, Théo, Valerio Perrone, and Manfred Opper. "Flexible and Efficient Inference with Particles for the Variational Gaussian Approximation." Entropy 23, no. 8 (July 30, 2021): 990. http://dx.doi.org/10.3390/e23080990.
Abstract:
Variational inference is a powerful framework, used to approximate intractable posteriors through variational distributions. The de facto standard is to rely on Gaussian variational families, which come with numerous advantages: they are easy to sample from, simple to parametrize, and many expectations are known in closed-form or readily computed by quadrature. In this paper, we view the Gaussian variational approximation problem through the lens of gradient flows. We introduce a flexible and efficient algorithm based on a linear flow leading to a particle-based approximation. We prove that, with a sufficient number of particles, our algorithm converges linearly to the exact solution for Gaussian targets, and a low-rank approximation otherwise. In addition to the theoretical analysis, we show, on a set of synthetic and real-world high-dimensional problems, that our algorithm outperforms existing methods with Gaussian targets while performing on a par with non-Gaussian targets.
31
Gao, Di, Xiaoru Xie, and Dongxu Wei. "A Design Methodology for Fault-Tolerant Neuromorphic Computing Using Bayesian Neural Network." Micromachines 14, no. 10 (September 27, 2023): 1840. http://dx.doi.org/10.3390/mi14101840.
Abstract:
Memristor crossbar arrays are a promising platform for neuromorphic computing. In practical scenarios, the synapse weights represented by the memristors for the underlying system are subject to process variations, in which the programmed weight when read out for inference is no longer deterministic but a stochastic distribution. It is therefore highly desired to learn the weight distribution accounting for process variations, to ensure the same inference performance in memristor crossbar arrays as the design value. In this paper, we introduce a design methodology for fault-tolerant neuromorphic computing using a Bayesian neural network, which combines the variational Bayesian inference technique with a fault-aware variational posterior distribution. The proposed framework based on Bayesian inference incorporates the impacts of memristor deviations into algorithmic training, where the weight distributions of neural networks are optimized to accommodate uncertainties and minimize inference degradation. The experimental results confirm the capability of the proposed methodology to tolerate both process variations and noise, while achieving more robust computing in memristor crossbar arrays.
32
Vlastelica, Marin, Patrick Ernst, and Gyuri Szarvas. "Taming Continuous Posteriors for Latent Variational Dialogue Policies." Proceedings of the AAAI Conference on Artificial Intelligence 37, no. 11 (June 26, 2023): 13673–81. http://dx.doi.org/10.1609/aaai.v37i11.26602.
Abstract:
Utilizing amortized variational inference for latent-action reinforcement learning (RL) has been shown to be an effective approach in Task-oriented Dialogue (ToD) systems for optimizing dialogue success.Until now, categorical posteriors have been argued to be one of the main drivers of performance. In this work we revisit Gaussian variational posteriors for latent-action RL and show that they can yield even better performance than categoricals. We achieve this by introducing an improved variational inference objective for learning continuous representations without auxiliary learning objectives, which streamlines the training procedure. Moreover, we propose ways to regularize the latent dialogue policy, which helps to retain good response coherence. Using continuous latent representations our model achieves state of the art dialogue success rate on the MultiWOZ benchmark, and also compares well to categorical latent methods in response coherence.
33
Su, Hang, and Wei Wang. "An Out-of-Distribution Generalization Framework Based on Variational Backdoor Adjustment." Mathematics 12, no. 1 (December 26, 2023): 85. http://dx.doi.org/10.3390/math12010085.
Abstract:
In practical applications, learning models that can perform well even when the data distribution is different from the training set are essential and meaningful. Such problems are often referred to as out-of-distribution (OOD) generalization problems. In this paper, we propose a method for OOD generalization based on causal inference. Unlike the prevalent OOD generalization methods, our approach does not require the environment labels associated with the data in the training set. We analyze the causes of distributional shifts in data from a causal modeling perspective and then propose a backdoor adjustment method based on variational inference. Finally, we constructed a unique network structure to simulate the variational inference process. The proposed variational backdoor adjustment (VBA) framework can be combined with any mainstream backbone network. In addition to theoretical derivation, we conduct experiments on different datasets to demonstrate that our method performs well in prediction accuracy and generalization gaps. Furthermore, by comparing the VBA framework with other mainstream OOD methods, we show that VBA performs better than mainstream methods.
34
Zhai, Ke, Jordan Boyd-Graber, and Shay B. Cohen. "Online Adaptor Grammars with Hybrid Inference." Transactions of the Association for Computational Linguistics 2 (December 2014): 465–76. http://dx.doi.org/10.1162/tacl_a_00196.
Abstract:
Adaptor grammars are a flexible, powerful formalism for defining nonparametric, unsupervised models of grammar productions. This flexibility comes at the cost of expensive inference. We address the difficulty of inference through an online algorithm which uses a hybrid of Markov chain Monte Carlo and variational inference. We show that this inference strategy improves scalability without sacrificing performance on unsupervised word segmentation and topic modeling tasks.
35
Zhang, Chendong, and Ting Chen. "Bayesian slip inversion with automatic differentiation variational inference." Geophysical Journal International 229, no. 1 (October 29, 2021): 546–65. http://dx.doi.org/10.1093/gji/ggab438.
Abstract:
SUMMARY The Bayesian slip inversion offers a powerful tool for modelling the earthquake source mechanism. It can provide a fully probabilistic result and thus permits us to quantitatively assess the inversion uncertainty. The Bayesian problem is usually solved with Monte Carlo methods, but they are computationally expensive and are inapplicable for high-dimensional and large-scale problems. Variational inference is an alternative solver to the Bayesian problem. It turns Bayesian inference into an optimization task and thus enjoys better computational performances. In this study, we introduce a general variational inference algorithm, automatic differentiation variational inference (ADVI), to the Bayesian slip inversion and compare it with the classic Metropolis–Hastings (MH) sampling method. The synthetic test shows that the two methods generate nearly identical mean slip distributions and standard deviation maps. In the real case study, the two methods produce highly consistent mean slip distributions, but the ADVI-derived standard deviation map differs from that produced by the MH method, possibly because of the limitation of the Gaussian approximation in the ADVI method. In both cases, ADVI can give comparable results to the MH method but with a significantly lower computational cost. Our results show that ADVI is a promising and competitive method for the Bayesian slip inversion.
36
Dang, Tung, and Hirohisa Kishino. "Stochastic Variational Inference for Bayesian Phylogenetics: A Case of CAT Model." Molecular Biology and Evolution 36, no. 4 (February 1, 2019): 825–33. http://dx.doi.org/10.1093/molbev/msz020.
Abstract:
Abstract The pattern of molecular evolution varies among gene sites and genes in a genome. By taking into account the complex heterogeneity of evolutionary processes among sites in a genome, Bayesian infinite mixture models of genomic evolution enable robust phylogenetic inference. With large modern data sets, however, the computational burden of Markov chain Monte Carlo sampling techniques becomes prohibitive. Here, we have developed a variational Bayesian procedure to speed up the widely used PhyloBayes MPI program, which deals with the heterogeneity of amino acid profiles. Rather than sampling from the posterior distribution, the procedure approximates the (unknown) posterior distribution using a manageable distribution called the variational distribution. The parameters in the variational distribution are estimated by minimizing Kullback–Leibler divergence. To examine performance, we analyzed three empirical data sets consisting of mitochondrial, plastid-encoded, and nuclear proteins. Our variational method accurately approximated the Bayesian inference of phylogenetic tree, mixture proportions, and the amino acid propensity of each component of the mixture while using orders of magnitude less computational time.
37
Zhang, Cheng, Judith Butepage, Hedvig Kjellstrom, and Stephan Mandt. "Advances in Variational Inference." IEEE Transactions on Pattern Analysis and Machine Intelligence 41, no. 8 (August 1, 2019): 2008–26. http://dx.doi.org/10.1109/tpami.2018.2889774.
38
Saeedi, Ardavan, Yuria Utsumi, Li Sun, Kayhan Batmanghelich, and Li-wei Lehman. "Knowledge Distillation via Constrained Variational Inference." Proceedings of the AAAI Conference on Artificial Intelligence 36, no. 7 (June 28, 2022): 8132–40. http://dx.doi.org/10.1609/aaai.v36i7.20786.
Abstract:
Knowledge distillation has been used to capture the knowledge of a teacher model and distill it into a student model with some desirable characteristics such as being smaller, more efficient, or more generalizable. In this paper, we propose a framework for distilling the knowledge of a powerful discriminative model such as a neural network into commonly used graphical models known to be more interpretable (e.g., topic models, autoregressive Hidden Markov Models). Posterior of latent variables in these graphical models (e.g., topic proportions in topic models) is often used as feature representation for predictive tasks. However, these posterior-derived features are known to have poor predictive performance compared to the features learned via purely discriminative approaches. Our framework constrains variational inference for posterior variables in graphical models with a similarity preserving constraint. This constraint distills the knowledge of the discriminative model into the graphical model by ensuring that input pairs with (dis)similar representation in the teacher model also have (dis)similar representation in the student model. By adding this constraint to the variational inference scheme, we guide the graphical model to be a reasonable density model for the data while having predictive features which are as close as possible to those of a discriminative model. To make our framework applicable to a wide range of graphical models, we build upon the Automatic Differentiation Variational Inference (ADVI), a black-box inference framework for graphical models. We demonstrate the effectiveness of our framework on two real-world tasks of disease subtyping and disease trajectory modeling.
39
Li, Ang, Luis Pericchi, and Kun Wang. "Objective Bayesian Inference in Probit Models with Intrinsic Priors Using Variational Approximations." Entropy 22, no. 5 (April 30, 2020): 513. http://dx.doi.org/10.3390/e22050513.
Abstract:
There is not much literature on objective Bayesian analysis for binary classification problems, especially for intrinsic prior related methods. On the other hand, variational inference methods have been employed to solve classification problems using probit regression and logistic regression with normal priors. In this article, we propose to apply the variational approximation on probit regression models with intrinsic prior. We review the mean-field variational method and the procedure of developing intrinsic prior for the probit regression model. We then present our work on implementing the variational Bayesian probit regression model using intrinsic prior. Publicly available data from the world’s largest peer-to-peer lending platform, LendingClub, will be used to illustrate how model output uncertainties are addressed through the framework we proposed. With LendingClub data, the target variable is the final status of a loan, either charged-off or fully paid. Investors may very well be interested in how predictive features like FICO, amount financed, income, etc. may affect the final loan status.
40
Damgaard, Malte Rørmose, Rasmus Pedersen, and Thomas Bak. "Study of Variational Inference for Flexible Distributed Probabilistic Robotics." Robotics 11, no. 2 (March 24, 2022): 38. http://dx.doi.org/10.3390/robotics11020038.
Abstract:
By combining stochastic variational inference with message passing algorithms, we show how to solve the highly complex problem of navigation and avoidance in distributed multi-robot systems in a computationally tractable manner, allowing online implementation. Subsequently, the proposed variational method lends itself to more flexible solutions than prior methodologies. Furthermore, the derived method is verified both through simulations with multiple mobile robots and a real world experiment with two mobile robots. In both cases, the robots share the operating space and need to cross each other’s paths multiple times without colliding.
41
Kiefer, Alex B. "Psychophysical identity and free energy." Journal of The Royal Society Interface 17, no. 169 (August 2020): 20200370. http://dx.doi.org/10.1098/rsif.2020.0370.
Abstract:
An approach to implementing variational Bayesian inference in biological systems is considered, under which the thermodynamic free energy of a system directly encodes its variational free energy. In the case of the brain, this assumption places constraints on the neuronal encoding of generative and recognition densities, in particular requiring a stochastic population code. The resulting relationship between thermodynamic and variational free energies is prefigured in mind–brain identity theses in philosophy and in the Gestalt hypothesis of psychophysical isomorphism.
42
Rezek, I., D. S. Leslie, S. Reece, S. J. Roberts, A. Rogers, R. K. Dash, and N. R. Jennings. "On Similarities between Inference in Game Theory and Machine Learning." Journal of Artificial Intelligence Research 33 (October 23, 2008): 259–83. http://dx.doi.org/10.1613/jair.2523.
Abstract:
In this paper, we elucidate the equivalence between inference in game theory and machine learning. Our aim in so doing is to establish an equivalent vocabulary between the two domains so as to facilitate developments at the intersection of both fields, and as proof of the usefulness of this approach, we use recent developments in each field to make useful improvements to the other. More specifically, we consider the analogies between smooth best responses in fictitious play and Bayesian inference methods. Initially, we use these insights to develop and demonstrate an improved algorithm for learning in games based on probabilistic moderation. That is, by integrating over the distribution of opponent strategies (a Bayesian approach within machine learning) rather than taking a simple empirical average (the approach used in standard fictitious play) we derive a novel moderated fictitious play algorithm and show that it is more likely than standard fictitious play to converge to a payoff-dominant but risk-dominated Nash equilibrium in a simple coordination game. Furthermore we consider the converse case, and show how insights from game theory can be used to derive two improved mean field variational learning algorithms. We first show that the standard update rule of mean field variational learning is analogous to a Cournot adjustment within game theory. By analogy with fictitious play, we then suggest an improved update rule, and show that this results in fictitious variational play, an improved mean field variational learning algorithm that exhibits better convergence in highly or strongly connected graphical models. Second, we use a recent advance in fictitious play, namely dynamic fictitious play, to derive a derivative action variational learning algorithm, that exhibits superior convergence properties on a canonical machine learning problem (clustering a mixture distribution).
43
Vedadi, Elahe, Joshua V. Dillon, Philip Andrew Mansfield, Karan Singhal, Arash Afkanpour, and Warren Richard Morningstar. "Federated Variational Inference: Towards Improved Personalization and Generalization." Proceedings of the AAAI Symposium Series 3, no. 1 (May 20, 2024): 323–27. http://dx.doi.org/10.1609/aaaiss.v3i1.31228.
Abstract:
Conventional federated learning algorithms train a single global model by leveraging all participating clients’ data. However, due to heterogeneity in client generative distributions and predictive models, these approaches may not appropriately approximate the predictive process, converge to an optimal state, or generalize to new clients. We study personalization and generalization in stateless cross-device federated learning setups assuming heterogeneity in client data distributions and predictive models. We first propose a hierarchical generative model and formalize it using Bayesian Inference. We then approximate this process using Variational Inference to train our model efficiently. We call this algorithm Federated Variational Inference (FedVI). We use PAC-Bayes analysis to provide generalization bounds for FedVI. We evaluate our model on FEMNIST and CIFAR-100 image classification and show that FedVI beats the state-of-the-art on both tasks.
44
Park, Mijung, James Foulds, Kamalika Chaudhuri, and Max Welling. "Variational Bayes In Private Settings (VIPS)." Journal of Artificial Intelligence Research 68 (May 5, 2020): 109–57. http://dx.doi.org/10.1613/jair.1.11763.
Abstract:
Many applications of Bayesian data analysis involve sensitive information such as personal documents or medical records, motivating methods which ensure that privacy is protected. We introduce a general privacy-preserving framework for Variational Bayes (VB), a widely used optimization-based Bayesian inference method. Our framework respects differential privacy, the gold-standard privacy criterion, and encompasses a large class of probabilistic models, called the Conjugate Exponential (CE) family. We observe that we can straightforwardly privatise VB’s approximate posterior distributions for models in the CE family, by perturbing the expected sufficient statistics of the complete-data likelihood. For a broadly-used class of non-CE models, those with binomial likelihoods, we show how to bring such models into the CE family, such that inferences in the modified model resemble the private variational Bayes algorithm as closely as possible, using the Pólya-Gamma data augmentation scheme. The iterative nature of variational Bayes presents a further challenge since iterations increase the amount of noise needed. We overcome this by combining: (1) an improved composition method for differential privacy, called the moments accountant, which provides a tight bound on the privacy cost of multiple VB iterations and thus significantly decreases the amount of additive noise; and (2) the privacy amplification effect of subsampling mini-batches from large-scale data in stochastic learning. We empirically demonstrate the effectiveness of our method in CE and non-CE models including latent Dirichlet allocation, Bayesian logistic regression, and sigmoid belief networks, evaluated on real-world datasets.
45
Gallego, Víctor, and David Ríos Insua. "Variationally Inferred Sampling through a Refined Bound." Entropy 23, no. 1 (January 19, 2021): 123. http://dx.doi.org/10.3390/e23010123.
Abstract:
In this work, a framework to boost the efficiency of Bayesian inference in probabilistic models is introduced by embedding a Markov chain sampler within a variational posterior approximation. We call this framework “refined variational approximation”. Its strengths are its ease of implementation and the automatic tuning of sampler parameters, leading to a faster mixing time through automatic differentiation. Several strategies to approximate evidence lower bound (ELBO) computation are also introduced. Its efficient performance is showcased experimentally using state-space models for time-series data, a variational encoder for density estimation and a conditional variational autoencoder as a deep Bayes classifier.
46
Zhang, Rui, Christian Walder, and Marian-Andrei Rizoiu. "Variational Inference for Sparse Gaussian Process Modulated Hawkes Process." Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 04 (April 3, 2020): 6803–10. http://dx.doi.org/10.1609/aaai.v34i04.6160.
Abstract:
The Hawkes process (HP) has been widely applied to modeling self-exciting events including neuron spikes, earthquakes and tweets. To avoid designing parametric triggering kernel and to be able to quantify the prediction confidence, the non-parametric Bayesian HP has been proposed. However, the inference of such models suffers from unscalability or slow convergence. In this paper, we aim to solve both problems. Specifically, first, we propose a new non-parametric Bayesian HP in which the triggering kernel is modeled as a squared sparse Gaussian process. Then, we propose a novel variational inference schema for model optimization. We employ the branching structure of the HP so that maximization of evidence lower bound (ELBO) is tractable by the expectation-maximization algorithm. We propose a tighter ELBO which improves the fitting performance. Further, we accelerate the novel variational inference schema to linear time complexity by leveraging the stationarity of the triggering kernel. Different from prior acceleration methods, ours enjoys higher efficiency. Finally, we exploit synthetic data and two large social media datasets to evaluate our method. We show that our approach outperforms state-of-the-art non-parametric frequentist and Bayesian methods. We validate the efficiency of our accelerated variational inference schema and practical utility of our tighter ELBO for model selection. We observe that the tighter ELBO exceeds the common one in model selection.
47
Masegosa, Andrés R., Darío Ramos-López, Antonio Salmerón, Helge Langseth, and Thomas D. Nielsen. "Variational Inference over Nonstationary Data Streams for Exponential Family Models." Mathematics 8, no. 11 (November 3, 2020): 1942. http://dx.doi.org/10.3390/math8111942.
Abstract:
In many modern data analysis problems, the available data is not static but, instead, comes in a streaming fashion. Performing Bayesian inference on a data stream is challenging for several reasons. First, it requires continuous model updating and the ability to handle a posterior distribution conditioned on an unbounded data set. Secondly, the underlying data distribution may drift from one time step to another, and the classic i.i.d. (independent and identically distributed), or data exchangeability assumption does not hold anymore. In this paper, we present an approximate Bayesian inference approach using variational methods that addresses these issues for conjugate exponential family models with latent variables. Our proposal makes use of a novel scheme based on hierarchical priors to explicitly model temporal changes of the model parameters. We show how this approach induces an exponential forgetting mechanism with adaptive forgetting rates. The method is able to capture the smoothness of the concept drift, ranging from no drift to abrupt drift. The proposed variational inference scheme maintains the computational efficiency of variational methods over conjugate models, which is critical in streaming settings. The approach is validated on four different domains (energy, finance, geolocation, and text) using four real-world data sets.
48
Hilprecht, Benjamin, Martin Härterich, and Daniel Bernau. "Monte Carlo and Reconstruction Membership Inference Attacks against Generative Models." Proceedings on Privacy Enhancing Technologies 2019, no. 4 (October 1, 2019): 232–49. http://dx.doi.org/10.2478/popets-2019-0067.
Abstract:
Abstract We present two information leakage attacks that outperform previous work on membership inference against generative models. The first attack allows membership inference without assumptions on the type of the generative model. Contrary to previous evaluation metrics for generative models, like Kernel Density Estimation, it only considers samples of the model which are close to training data records. The second attack specifically targets Variational Autoencoders, achieving high membership inference accuracy. Furthermore, previous work mostly considers membership inference adversaries who perform single record membership inference. We argue for considering regulatory actors who perform set membership inference to identify the use of specific datasets for training. The attacks are evaluated on two generative model architectures, Generative Adversarial Networks (GANs) and Variational Autoen-coders (VAEs), trained on standard image datasets. Our results show that the two attacks yield success rates superior to previous work on most data sets while at the same time having only very mild assumptions. We envision the two attacks in combination with the membership inference attack type formalization as especially useful. For example, to enforce data privacy standards and automatically assessing model quality in machine learning as a service setups. In practice, our work motivates the use of GANs since they prove less vulnerable against information leakage attacks while producing detailed samples.
49
Friston, Karl, Philipp Schwartenbeck, Thomas FitzGerald, Michael Moutoussis, Timothy Behrens, and Raymond J. Dolan. "The anatomy of choice: dopamine and decision-making." Philosophical Transactions of the Royal Society B: Biological Sciences 369, no. 1655 (November 5, 2014): 20130481. http://dx.doi.org/10.1098/rstb.2013.0481.
Abstract:
This paper considers goal-directed decision-making in terms of embodied or active inference. We associate bounded rationality with approximate Bayesian inference that optimizes a free energy bound on model evidence. Several constructs such as expected utility, exploration or novelty bonuses, softmax choice rules and optimism bias emerge as natural consequences of free energy minimization. Previous accounts of active inference have focused on predictive coding . In this paper, we consider variational Bayes as a scheme that the brain might use for approximate Bayesian inference. This scheme provides formal constraints on the computational anatomy of inference and action, which appear to be remarkably consistent with neuroanatomy. Active inference contextualizes optimal decision theory within embodied inference, where goals become prior beliefs. For example, expected utility theory emerges as a special case of free energy minimization, where the sensitivity or inverse temperature (associated with softmax functions and quantal response equilibria) has a unique and Bayes-optimal solution. Crucially, this sensitivity corresponds to the precision of beliefs about behaviour. The changes in precision during variational updates are remarkably reminiscent of empirical dopaminergic responses—and they may provide a new perspective on the role of dopamine in assimilating reward prediction errors to optimize decision-making.
50
BOCCIGNONE, GIUSEPPE, PAOLO NAPOLETANO, and MARIO FERRARO. "EMBEDDING DIFFUSION IN VARIATIONAL BAYES: A TECHNIQUE FOR SEGMENTING IMAGES." International Journal of Pattern Recognition and Artificial Intelligence 22, no. 05 (August 2008): 811–27. http://dx.doi.org/10.1142/s0218001408006533.
Abstract:
In this paper, we discuss how image segmentation can be handled by using Bayesian learning and inference. In particular variational techniques relying on free energy minimization will be introduced. It will be shown how to embed a spatial diffusion process on segmentation labels within the Variational Bayes learning procedure so as to enforce spatial constraints among labels.