Academic literature on the topic 'Ensemble Kalman filter hyper-localized'

Abstract Ensemble square root filters can either assimilate all observations that are available at a given time at once, or assimilate the observations in batches or one at a time. For large-scale models, the filters are typically applied with a localized analysis step. This study demonstrates that the interaction of serial observation processing and localization can destabilize the analysis process, and it examines under which conditions the instability becomes significant. The instability results from a repeated inconsistent update of the state error covariance matrix that is caused by the localization. The inconsistency is present in all ensemble Kalman filters, except for the classical ensemble Kalman filter with perturbed observations. With serial observation processing, its effect is small in cases when the assimilation changes the ensemble of model states only slightly. However, when the assimilation has a strong effect on the state estimates, the interaction of localization and serial observation processing can significantly deteriorate the filter performance. In realistic large-scale applications, when the assimilation changes the states only slightly and when the distribution of the observations is irregular and changing over time, the instability is likely not significant.

Abstract The ensemble Kalman filter is typically implemented either by applying the localization on the background error covariance matrix (B-localization) or by inflating the observation error variances (R-localization). A mathematical demonstration suggests that for the same effective localization function, the background error covariance matrix from the B-localization method shows a higher rank than the R-localization method. The B-localization method is realized in the ensemble transform Kalman filter (ETKF) by extending the background ensemble perturbations through modulation (MP-localization). Specifically, the modulation functions are constructed from the leading eigenvalues and eigenvectors of the original B-localization matrix. Because of its higher rank than the classic R-localized ETKF, the B-/MP-localized ETKF is termed as the high-rank ETKF (HETKF). The performances of the HETKF and R-localized ETKF were compared through cycled data assimilation experiments using the Lorenz model II. The results show that the HETKF outperforms the R-localized ETKF especially for a small ensemble. The improved analysis in the HETKF is likely associated with the higher rank from the B-/MP-localization method, since its higher rank is expected to contribute more positively to alleviating the rank deficiency issue and thus improve the analysis for a small ensemble. The HETKF is less sensitive to the localization length scales and inflation factors. Furthermore, the experiments suggest that the above conclusion comparing the HETKF and R-localized ETKF does not depend on how the analyzed ensemble perturbations are subselected in the HETKF.

An important step when using some data assimilation methods, such as the ensemble Kalman filter and its variants, is to calibrate its parameters. Also called hyper-parameters, these include the model and observation errors, which have previously been shown to have a strong impact on the performance of the data assimilation method. Many metrics can be used to calibrate these hyper-parameters but may not all yield the same optimal set of values. The current study investigated the importance of the choice of metric used during the hyper-parameter calibration phase and its impact on discharge forecasts. The types of metrics used each focused on discharge accuracy, ensemble spread or observation-minus-background statistics. The calibration was performed for the ensemble square root Kalman filter over two catchments in Canada using two different hydrologic models per catchment. Results show that the optimal set of hyper-parameters depended heavily on the choice of metric used during the calibration phase, where data assimilation was applied. These sets of hyper-parameters in turn produced different hydrologic forecasts. This influence was reduced as the forecast lead time increased, because of not applying data assimilation in the forecast mode, and accordingly, convergence of model state ensembles produced in the calibration phase. However, the influence could remain considerable for a few days up to multiple weeks depending on the catchment and the model. As such, a preliminary analysis would be recommended for future studies to better understand the impact that metrics can have within and outside the bounds of hyper-parameter calibration.

Abstract A consistent hybrid ensemble filter (CHEF) for using hybrid forecast error covariance matrices that linearly combine aspects of both climatological and flow-dependent matrices within a nonvariational ensemble data assimilation scheme is described. The CHEF accommodates the ensemble data assimilation enhancements of (i) model space ensemble covariance localization for satellite data assimilation and (ii) Hodyss’s method for improving accuracy using ensemble skewness. Like the local ensemble transform Kalman filter (LETKF), the CHEF is computationally scalable because it updates local patches of the atmosphere independently of others. Like the sequential ensemble Kalman filter (EnKF), it serially assimilates batches of observations and uses perturbed observations to create ensembles of analyses. It differs from the deterministic (no perturbed observations) ensemble square root filter (ESRF) and the EnKF in that (i) its analysis correction is unaffected by the order in which observations are assimilated even when localization is required, (ii) it uses accurate high-rank solutions for the posterior error covariance matrix to serially assimilate observations, and (iii) it accommodates high-rank hybrid error covariance models. Experiments were performed to assess the effect on CHEF and ESRF analysis accuracy of these differences. In the case where both the CHEF and the ESRF used tuned localized ensemble covariances for the forecast error covariance model, the CHEF’s advantage over the ESRF increased with observational density. In the case where the CHEF used a hybrid error covariance model but the ESRF did not, the CHEF had a substantial advantage for all observational densities.

Abstract An ensemble Kalman filter (EnKF) estimates the error statistics of a model forecast using an ensemble of model forecasts. One use of an EnKF is data assimilation, resulting in the creation of an increment to the first-guess field at the observation time. Another use of an EnKF is to propagate error statistics of a model forecast forward in time, such as is done for optimizing the location of adaptive observations. Combining these two uses of an ensemble Kalman filter, a “preemptive forecast” can be generated. In a preemptive forecast, the increment to the first-guess field is, using ensembles, propagated to some future time and added to the future control forecast, resulting in a new forecast. This new forecast requires no more time to produce than the time needed to run a data assimilation scheme, as no model integration is necessary. In an observing system simulation experiment (OSSE), a barotropic vorticity model was run to produce a 300-day “nature run.” The same model, run with a different vorticity forcing scheme, served as the forecast model. The model produced 24- and 48-h forecasts for each of the 300 days. The model was initialized every 24 h by assimilating observations of the nature run using a hybrid ensemble Kalman filter–three-dimensional variational data assimilation (3DVAR) scheme. In addition to the control forecast, a 64-member forecast ensemble was generated for each of the 300 days. Every 24 h, given a set of observations, the 64-member ensemble, and the control run, an EnKF was used to create 24-h preemptive forecasts. The preemptive forecasts were more accurate than the unmodified, original 48-h forecasts, though not quite as accurate as the 24-h forecast obtained from a new model integration initialized by assimilating the same observations as were used in the preemptive forecasts. The accuracy of the preemptive forecasts improved significantly when 1) the ensemble-based error statistics used by the EnKF were localized using a Schur product and 2) a model error term was included in the background error covariance matrices.

The localized normal-score ensemble Kalman filter is shown to work for the characterization of non-multi-Gaussian distributed hydraulic conductivities by assimilating state observation data. The influence of type of flow regime, number of observation piezometers, and the prior model structure are evaluated in a synthetic aquifer. Steady-state observation data are not sufficient to identify the conductivity channels. Transient-state data are necessary for a good characterization of the hydraulic conductivity curvilinear patterns. Such characterization is very good with a dense network of observation data, and it deteriorates as the number of observation piezometers decreases. It is also remarkable that, even when the prior model structure is wrong, the localized normal-score ensemble Kalman filter can produce acceptable results for a sufficiently dense observation network.

Particle filters are well known in statistics. They have a long tradition in the framework of ensemble data assimilation (EDA) as well as Markov chain Monte Carlo (MCMC) methods. A key challenge today is to employ such methods in a high-dimensional environment, since the naïve application of the classical particle filter usually leads to filter divergence or filter collapse when applied within the very high dimension of many practical assimilation problems (known as the curse of dimensionality). The goal of this work is to develop a localized adaptive particle filter (LAPF), which follows closely the idea of the classical MCMC or bootstrap-type particle filter, but overcomes the problems of collapse and divergence based on localization in the spirit of the local ensemble transform Kalman filter (LETKF) and adaptivity with an adaptive Gaussian resampling or rejuvenation scheme in ensemble space. The particle filter has been implemented in the data assimilation system for the global forecast model ICON at Deutscher Wetterdienst (DWD). We carry out simulations over a period of 1 month with a global horizontal resolution of 52 km and 90 layers. With four variables analyzed per grid point, this leads to 6.6 × 106 degrees of freedom. The LAPF can be run stably and shows a reasonable performance. We compare its scores to the operational setup of the ICON LETKF.

Hybrid variational–ensemble data assimilation (hybrid DA) is widely used in research and operational systems, and it is considered the current state of the art for the initialization of numerical weather prediction models. However, hybrid DA requires a separate ensemble DA to estimate the uncertainty in the deterministic variational DA, which can be suboptimal both technically and scientifically. A new framework called the ensemble–variational integrated localized (EVIL) data assimilation addresses this inconvenience by updating the ensemble analyses using information from the variational deterministic system. The goal of EVIL is to encompass and generalize existing ensemble Kalman filter methods in a variational framework. Particular attention is devoted to the affordability and efficiency of the algorithm in preparation for operational applications.

Dans ce travail de thèse, des outils d’assimilation de données sont utilisés pour augmenter les performances de solveurs de mécanique des fluides dédiés à la simulation des grandes échelles. L’objectif est d’améliorer la prédiction et l’étude d’évènements marginaux nuisibles à l’intégrité des systèmes physiques. Bien que difficile à caractériser et à modéliser, la compréhension détaillée de ces phénomènes physiques complexes est essentielle pour le développement d’applications plus durables. Ces objectifs s’intègrent aux activités de recherche du projet ANR ALEKCIA dans lequel ce travail de thèse s’inscrit. Plus précisément, il s’agit de répondre au besoin de couplage de calculs de mécanique des fluides numérique avec un algorithme d’assimilation de données séquentiel. L’outil CONES (Coupling OpenFOAM with Numerical EnvironmentS),qui a été développé, permet d’y apporter une réponse en s’appuyant sur le logiciel OpenFOAM et le filtre de Kalman d’ensemble. Ce dernier permet à la fois le calibrage des paramètres physiques de la simulation numérique ainsi que l’inférence de champs physiques tels que le champ de vitesse. CONES est mis à contribution pour l’inférence de trois cas d’étude à la complexité grandissante. Le premier optimise les coefficients de fermeture de modèles de turbulence de type RANS pour un écoulement incompressible via l’assimilation de données expérimentales. La calibration de ces paramètres entraîne notamment une amélioration topologique des structures de recirculation de la géométrie. Le cas démontre également l’importance de la qualité des informations de sources hétérogènes observées plutôt que leur quantité. Dans une deuxième étude, la simulation des grandes échelles est utilisée pour fournir une prédiction des caractéristiques tridimensionnelles instationnaires d’un écoulement incompressible turbulent en canal. Outre l’optimisation du modèle Smagorinsky, le champ de vitesse est partiellement synchronisé avec les données observées pour favoriser la reconstruction des structures instationnaires. L’influence de certains hyper-paramètres tels que l’inflation est mise en lumière. Enfin, une variante de l’algorithme de Kalman, le filtre de Kalman d’ensemble hyper-localisé, est développée pour le dernier cas d’étude. Cette méthode permet notamment une diminution du coût de calcul. Elle est utilisée pour l’inférence d’une LES de l’écoulement compressible d’une géométrie simplifiée de moteur. La condition d’entrée de référence pulsée est correctement calibrée et le champ de vitesse est localement synchronisé sur les simulations inférées. La correction apportée par l’algorithme montre également une amélioration de la répartition énergétique de la région inférée en adéquation avec la répartition de référence. En conclusion, le potentiel du filtre de Kalman d’ensemble pour la calibration de paramètres physiques et la reconstruction de structures locales grâce à l’observation de données haute-fidélité d’un système réel a été démontré. Ceci permettrait l’étude d’évènements extrêmes pouvant nuire à l’intégrité du système physique grâce à la simulation numérique augmentée de ces phénomènes
In this thesis, data assimilation tools are used to increase the performance of fluid mechanics solvers dedicated to large eddy simulations. The aim is to improve the prediction and study of marginal events harmful to the integrity of physical systems. Although difficult to characterize and model, a detailed understanding of these complex physical phenomena is essential for the development of more sustainable applications. These objectives are in line with the research activities of the ANR ALEKCIA project, of which this thesis is a part. More specifically, the aim is to meet the need to couple numerical fluid mechanics calculations with a sequential data assimilation algorithm. CONES (Coupling OpenFOAM with Numerical EnvironmentS) tool has been developed to provide an answer to this need by using the OpenFOAM software and the ensemble Kalman filter. The latter can be used both to calibrate the physical parameters of the numerical simulation and to infer physical fields, such as the velocity field. CONES is used to infer three case studies of increasing complexity. The first optimizes the closure coefficients of RANS-type turbulence models for an incompressible flow by assimilating experimental data. The calibration of these parameters leads in particular to a topological improvement in the recirculation structures of the geometry. The case also demonstrates the importance of the quality of the heterogeneous source of information observed rather than its quantity. In a second study, large eddy simulation is used to provide a prediction of the unsteady three-dimensional characteristics of a turbulent incompressible flow in a channel. In addition to optimizing the Smagorinsky model, the velocity field is partially synchronized with the observed data to facilitate the reconstruction of the unsteady structures. The influence of hyper parameters such as inflation is highlighted. Finally, a variant of the Kalman algorithm, the hyper-localized ensemble Kalman filter, is developed for the last case study. In particular, this method reduces the computational cost.It is used to infer a LES of the compressible flow of a simplified engine geometry. The pulsed reference input condition is correctly calibrated and the velocity field of the inferred simulations is locally synchronized. The correction provided by the algorithm also shows an improvement in the energy distribution of the inferred region in line with the reference distribution. In conclusion, the potential of the ensemble Kalman filter for calibrating physical parameters and reconstructing local structures by observing high-fidelity data from the real system has been demonstrated. This could enable the study of extreme events that could damage the integrity of the physical system, thanks to the augmented numerical simulation of these phenomena