Дисертації з теми "High Dynamic Range images (HDR)"

In this thesis, I present an approach to a pipeline for time lapse photography using conventional digital images converted to HDR (High Dynamic Range) images (rather than conventional digital or film exposures). Using this method, it is possible to capture a greater level of detail and a different look than one would get from a conventional time lapse image sequence. With HDR images properly tone-mapped for display on standard devices, information in shadows and hot spots is not lost, and certain details are enhanced.

The limited dynamic range of digital images can be extended by composing different exposures of the same scene to produce HDR images. This thesis is composed of an overview of the state of the art techniques and three methods to tackle the image alignment and deghosting problems in the HDR imaging domain. The first method detects the areas affected by motion, registers the dynamic objects over a reference image, and combines low-dynamic range values to recover HDR values in the whole image. The second approach builds multiscopic HDR images from LDR multi-exposure images. It is based on a patch match algorithm which was adapted and improved to take advantage of epipolar geometry constraints of stereo images. The last method proposes to replace under/over exposed pixels in the reference image by using valid HDR values from other images in the multi-exposure LDR image sequence.
El limitado rango dinámico de las imágenes digitales puede ampliarse mezclando varias imágenes adquiridas con diferentes valores de exposición. Esta tesis incluye un detallado resumen del estado del arte y tres métodos diferentes para alinear las imágenes y corregir el efecto ’ghosting’ en imágenes HDR. El primer método está centrado en detectar las áreas afectadas por el movimiento y registrar los objetos dinámicos sobre una imagen de referencia de modo que se logre recuperar información a lo largo de toda la imagen. Nuestra segunda propuesta es un método para obtener imágenes HDR multiscópicas a partir de diferentes exposiciones LDR. Está basado en un algoritmo de ’patch match’ que ha sido adaptado para aprovechar las ventajas de las restricciones de la geometría epipolar de imágenes estéreo. Por último proponemos reemplazar los píxeles saturados en la imagen de referencia usando valores correctos de otras imágenes de la secuencia.

High speed video has been a significant tool for unraveling the quantitative and qualitative assessment of phenomena that is too fast to readily observe. It was first used in 1852 by William Henry Fox Talbot to settle a dispute with reference to the synchronous position of a horse's hooves while galloping. Since that time private industry, government, and enthusiasts have been measuring dynamic scenarios with high speed video. One challenge that faces the high speed video community is the dynamic range of the sensors. The dynamic range of the sensor is constrained to the bit depth of the analog to digital converter, the deep well capacity of the sensor site, and baseline noise. A typical high speed camera can span a 60 dB dynamic range, 1000:1, natively. More recently the dynamic range has been extended to about 80 dB utilizing different pixel acquisition methods. In this dissertation a method to extend the dynamic range will be presented and demonstrated to extend the dynamic range of a high speed camera system to over 170 dB, about 31,000,000:1. The proposed formation methodology is adaptable to any camera combination, and almost any needed dynamic range. The dramatic increase in the dynamic range is made possible through an adaptation of the current high dynamic range image formation methodologies. Due to the high cost of a high speed camera, a minimum number of cameras are desired to form a high dynamic range high speed video system. With a reduced number of cameras spanning a significant range, the errors on the formation process compound significantly relative to a normal high dynamic range image. The increase in uncertainty is created from the lack of relevant correlated information for final image formation, necessitating the development of a new formation methodology. In the proceeding text the problem statement and background information will be reviewed in depth. The development of a new weighting function, stochastic image formation process, tone map methodology, and optimized multi camera design will be presented. The proposed methodologies' effectiveness will be compared to current methods throughout the text and a final demonstration will be presented.
Ph. D.

Common cameras with a dynamic range of two orders cannot reproduce typical outdoor scenes with a radiance range of over five orders. Most high dynamic range (HDR) imaging techniques reconstruct the whole dynamic range from exposure bracketed low dynamic range (LDR) images. But the camera must be kept steady with no or small motion, which is not practical in many cases. Thus, we develop a more efficient framework for omnidirectional HDR imaging with a moving camera. The proposed framework is composed of three major stages: geometric calibration and rotational alignment, multi-view stereo correspondence and HDR composition. First, camera poses are determined and omnidirectional images are rotationally aligned. Second, the aligned images are fed into a spherical vision toolkit to find disparity maps. Third, enhanced disparity maps are used to warp differently exposed neighboring images to a target view and an HDR radiance map is obtained by fusing the registered images in radiance. We develop disparity-based forward and backward image warping algorithms for spherical stereo vision and implement them in GPU. We also explore some techniques for disparity map enhancement including a superpixel technique and a color model for outdoor scenes. We examine different factors such as exposure increment step size, sequence ordering, and the baseline between views. We demonstrate the success with indoor and outdoor scenes and compare our results with two state-of-the-art HDR imaging methods. The proposed HDR framework allows us to capture HDR radiance maps, disparity maps and an omnidirectional field of view, which has many applications such as HDR view synthesis and virtual navigation.

This thesis concerns with the introduction to the problematics of images with high dynamic range (HDR) and possibilities of HDR images compression options for display on devices with a low dynamic range (LDR). In the introduction is described historical evolution of recording of reality. It is focusing towards point of view of physics, human visual perception and digital recording. There are described the ways of generating and holding of HDR images. The thesis is corncerned to the techniques of HDR compression, it means the tone-mapping. The different techniques of tone-mapping are explained and specific aproach is targeted to the gradient domain high dynamic range compresion.

2007/2008
The intensity of natural light can span over 10 orders of magnitude from starlight to direct sunlight. Even in a single scene, the luminance of the bright areas can be thousands or millions of times greater than the luminance in the dark areas; the ratio between the maximum and the minimum luminance values is commonly known as dynamic range or contrast. The human visual system is able to operate in an extremely wide range of luminance conditions without saturation and at the same time it can perceive fine details which involve small luminance differences. Our eyes achieve this ability by modulating their response as a function of the local mean luminance with a process known as local adaptation. In particular, the visual sensation is not linked to the absolute luminance, but rather to its spatial and temporal variation. One consequence of the local adaptation capability of the eye is that the objects in a scene maintain their appearance even if the light source illuminating the scene changes significantly. On the other hand, the technologies used for the acquisition and reproduction of digital images are able to handle correctly a significantly smaller luminance range of 2 to 3 orders of magnitude at most. Therefore, a high dynamic range (HDR) image poses several challenges and requires the use of appropriate techniques. These elementary observations define the context in which the entire research work described in this Thesis has been performed. As indicated below, different fields have been considered; they range from the acquisition of HDR images to their display, from visual quality evaluation to medical applications, and include some developments on a recently proposed class of display equipment. An HDR image can be captured by taking multiple photographs with different exposure times or by using high dynamic range sensors; moreover, synthetic HDR images can be generated with computer graphics by means of physically-based algorithms which often involve advanced lighting simulations. An HDR image, although acquired correctly, can not be displayed on a conventional monitor. The white level of most devices is limited to a few hundred cd/m² by technological constraints, primarily linked to the power consumption and heat dissipation; the black level also has a non negligible luminance, in particular for devices based on the liquid crystal technology. However, thanks to the aforementioned properties of the human visual system, an exact reproduction of the luminance in the original scene is not strictly necessary in order to produce a similar sensation in the observer. For this purpose, dynamic range reduction algorithms have been developed which attenuate the large luminance variations in an image while preserving as far as possible the fine details. The most simple dynamic range reduction algorithms map each pixel individually with the same nonlinear function commonly known as tone mapping curve. One operator we propose, based on a modified logarithmic function, has a low computational cost and contains one single user-adjustable parameter. However, the methods belonging to this category can reduce the visibility of the details in some portions of the image. More advanced methods also take into account the pixel neighborhood. This approach can achieve a better preservation of the details, but the loss of one-to-one mapping from input luminances to display values can lead to the formation of gradient reversal effects, which typically appear as halos around the object boundaries. Different solutions to this problem have been attempted. One method we introduce is able to avoid the formation of halos and intrinsically prevents any clipping of the output display values. The method is formulated as a constrained optimization problem, which is solved efficiently by means of appropriate numerical methods. In specific applications, such as the medical one, the use of dynamic range reduction algorithms is discouraged because any artifacts introduced by the processing can lead to an incorrect diagnosis. In particular, a one-to-one mapping from the physical data (for instance, a tissue density in radiographic techniques) to the display value is often an essential requirement. For this purpose, high dynamic range displays, capable of reproducing images with a wide luminance range and possibly a higher bit depth, are under active development. Dual layer LCD displays, for instance, use two liquid crystal panels stacked one on top of the other over an enhanced backlight unit in order to achieve a dynamic range of 4 ÷ 5 orders of magnitude. The grayscale reproduction accuracy is also increased, although a “bit depth” can not be defined unambiguously because the luminance levels obtained by the combination of the two panels are partially overlapped and unevenly spaced. A dual layer LCD display, however, requires the use of complex splitting algorithms in order to generate the two images which drive the two liquid crystal panels. A splitting algorithm should compensate multiple sources of error, including the parallax introduced by the viewing angle, the gray-level clipping introduced by the limited dynamic range of the panels, the visibility of the reconstruction error, and glare effects introduced by an unwanted light scattering between the two panels. For these reasons, complex constrained optimization techniques are necessary. We propose an objective function which incorporates all the desired constraints and requirements and can be minimized efficiently by means of appropriate techniques based on multigrid methods. The quality assessment of high dynamic range images requires the development of appropriate techniques. By their own nature, dynamic range reduction algorithms change the luminance values of an image significantly and make most image fidelity metrics inapplicable. Some particular aspects of the methods can be quantified by means of appropriate operators; for instance, we introduce an expression which describes the detail attenuation introduced by a tone mapping curve. In general, a subjective quality assessment is preferably performed by means of appropriate psychophysical experiments. We conducted a set of experiments, targeted specifically at measuring the level of agreement between different users when adjusting the parameter of the modified logarithmic mapping method we propose. The experimental results show a strong correlation between the user-adjusted parameter and the image statistics, and suggest a simple technique for the automatic adjustment of this parameter. On the other hand, the quality assessment in the medical field is preferably performed by means of objective methods. In particular, task-based quality measures evaluate by means of appropriate observer studies the clinical validity of the image used to perform a specific diagnostic task. We conducted a set of observer studies following this approach, targeted specifically at measuring the clinical benefit introduced by a high dynamic range display based on the dual layer LCD technology over a conventional display with a low dynamic range and 8-bit quantization. Observer studies are often time consuming and difficult to organize; in order to increase the number of tests, the human observers can be partially replaced by appropriate software applications, known as model observers or computational observers, which simulate the diagnostic task by means of statistical classification techniques. This thesis is structured as follows. Chapter 1 contains a brief background of concepts related to the physiology of human vision and to the electronic reproduction of images. The description we make is by no means complete and is only intended to introduce some concepts which will be extensively used in the following. Chapter 2 describes the technique of high dynamic range image acquisition by means of multiple exposures. In Chapter 3 we introduce the dynamic range reduction algorithms, providing an overview of the state of the art and proposing some improvements and novel techniques. In Chapter 4 we address the topic of quality assessment in dynamic range reduction algorithms; in particular, we introduce an operator which describes the detail attenuation introduced by tone mapping curves and describe a set of psychophysical experiments we conducted for the adjustment of the parameter in the modified logarithmic mapping method we propose. In Chapter 5 we move to the topic of medical images and describe the techniques used to map the density data of radiographic images to display luminances. We point out some limitations of the current technical recommendation and propose an improvement. In Chapter 6 we describe in detail the dual layer LCD prototype and propose different splitting algorithms for the generation of the two images which drive the two liquid crystal panels. In Chapter 7 we propose one possible technique for the estimation of the equivalent bit depth of a dual layer LCD display, based on a statistical analysis of the quantization noise. Finally, in Chapter 8 we address the topic of objective quality assessment in medical images and describe a set of observer studies we conducted in order to quantify the clinical benefit introduced by a high dynamic range display. No general conclusions are offered; the breadth of the subjects has suggested to draw more focused comments at the end of the individual chapters.
XXI Ciclo
1982

Real-world radiance values can range over eight orders of magnitude from starlight to direct sunlight but few digital cameras capture more than three orders in a single Low Dynamic Range (LDR) image. We approach this problem using established High Dynamic Range (HDR) techniques in which multiple images are captured with different exposure times so that all portions of the scene are correctly exposed at least once. These images are then combined to create an HDR image capturing the full range of the scene. HDR capture introduces new challenges; movement in the scene creates faded copies of moving objects, referred to as ghosts. Many techniques have been introduced to handle ghosting, but typically they either address specific types of ghosting, or are computationally very expensive. We address ghosting by first detecting moving objects, then reducing their contribution to the final composite on a frame-by-frame basis. The detection of motion is addressed by performing change detection on exposure-normalized images. Additional special cases are developed based on a priori knowledge of the changing exposures; for example, if exposure is increasing every shot, then any decrease in intensity in the LDR images is a strong indicator of motion. Recent Superpixel over-segmentation techniques are used to refine the detection. We also propose a novel solution for areas that see motion throughout the capture, such as foliage blowing in the wind. Such areas are detected as always moving, and are replaced with information from a single input image, and the replacement of corrupted regions can be tailored to the scenario. We present our approach in the context of a panoramic tele-presence system. Tele-presence systems allow a user to experience a remote environment, aiming to create a realistic sense of "being there" and such a system should therefore provide a high quality visual rendition of the environment. Furthermore, panoramas, by virtue of capturing a greater proportion of a real-world scene, are often exposed to a greater dynamic range than standard photographs. Both facets of this system therefore stand to benefit from HDR imaging techniques. We demonstrate the success of our approach on multiple challenging ghosting scenarios, and compare our results with state-of-the-art methods previously proposed. We also demonstrate computational savings over these methods.

Tese de Doutoramento em Informática apresentada à Universidade de Trás-os-Montes e Alto Douro em 2013.
Nas duas últimas décadas, assistimos ao desenvolvimento de um número crescente de técnicas, designadas por tone mapping operators (TMO), para reprodução de imagens com elevada gama dinâmica (high dynamic range – HDR) em ecrãs tradicionais. Apesar de recentemente terem surgido TMO que têm em conta a característica específica de cada dispositivo de visualização, nenhum desses algoritmos foi desenvolvido especificamente para dispositivos com ecrã pequeno (DEP). Assim, nesta tese foi realizado um estudo sobre a visualização de imagens HDR em DEP, tendo como propósito a proposta de soluções para a melhoria da sua visualização. Inicialmente foi realizada uma avaliação dos TMO atualmente existentes que mostrou que os DEP com tamanho limitado, resolução e profundidade de cor, exigem uma investigação específica para encontrar ou criar uma solução adequada. Esse estudo permitiu, também, identificar um conjunto de características dos TMO que precisam ser enfatizadas para obter imagens mapeadas com uma maior fidelidade nos DEP, especialmente o realce dos detalhes. Com base nesse estudo foi proposta uma solução para visualização de imagens HDR em DEP, que tem por base a construção de um TMO híbrido. O TMO proposto, tendo por base um qualquer TMO, tenta melhorar os detalhes das imagens mapeadas com o TMO original. Através da realização de experiências psicofísicas foi demonstrado que este novo TMO produz melhores resultados em DEP que os obtidos com o TMO original. Apesar de ter sido desenvolvido um protótipo em J2ME do novo TMO, a sua atual implementação ainda é pouco eficiente para a realização de testes diretamente em DEP. Em conclusão, com este trabalho é identificada a problemática da visualização de imagens HDR em DEP, sendo apontadas sugestões de como melhorar esse processo e fica ainda a proposta de um novo TMO.

One of the main objectives in digital imaging is to mimic the capabilities of the human eye, and perhaps, go beyond in certain aspects. However, the human visual system is so versatile, complex, and only partially understood that no up-to-date imaging technology has been able to accurately reproduce the capabilities of the it. The extraordinary capabilities of the human eye have become a crucial shortcoming in digital imaging, since digital photography, video recording, and computer vision applications have continued to demand more realistic and accurate imaging reproduction and analytic capabilities. Over decades, researchers have tried to solve the colour constancy problem, as well as extending the dynamic range of digital imaging devices by proposing a number of algorithms and instrumentation approaches. Nevertheless, no unique solution has been identified; this is partially due to the wide range of computer vision applications that require colour constancy and high dynamic range imaging, and the complexity of the human visual system to achieve effective colour constancy and dynamic range capabilities. The aim of the research presented in this thesis is to enhance the overall image quality within an image signal processor of digital cameras by achieving colour constancy and extending dynamic range capabilities. This is achieved by developing a set of advanced image-processing algorithms that are robust to a number of practical challenges and feasible to be implemented within an image signal processor used in consumer electronics imaging devises. The experiments conducted in this research show that the proposed algorithms supersede state-of-the-art methods in the fields of dynamic range and colour constancy. Moreover, this unique set of image processing algorithms show that if they are used within an image signal processor, they enable digital camera devices to mimic the human visual system s dynamic range and colour constancy capabilities; the ultimate goal of any state-of-the-art technique, or commercial imaging device.

L’imagerie HDR et l’imagerie 3D sont deux domaines dont l’évolution simultanée mais indépendante n’a cessé de croître ces dernières années. D’une part, l’imagerie HDR (High Dynamic Range) permet d’étendre la gamme dynamique de couleur des images conventionnelles dites LDR (Low Dynamic Range). D’autre part, l’imagerie 3D propose une immersion dans le film projeté avec cette impression de faire partie de la scène tournée. Depuis peu, ces deux domaines sont conjugués pour proposer des images ou vidéos 3D HDR mais peu de solutions viables existent et aucune n’est accessible au grand public. Dans ce travail de thèse, nous proposons une méthode de génération d’images 3D HDR pour une visualisation sur écrans autostéréoscopiques en adaptant une caméra multi-points de vue à l’acquisition d’expositions multiples. Pour cela, des filtres à densité neutre sont fixés sur les objectifs de la caméra. Ensuite, un appareillement des pixels homologues permet l’agrégation des pixels représentant le même point dans la scène acquise. Finalement, l’attribution d’une valeur de radiance est calculée pour chaque pixel du jeud’images considéré par moyenne pondérée des valeurs LDR des pixels homologues. Une étape supplémentaire est nécessaire car certains pixels ont une radiance erronée. Nous proposons une méthode basée surla couleur des pixels voisins puis deux méthodes basées sur la correction de la disparité des pixels dontla radiance est erronée. La première est basée sur la disparité des pixels du voisinage et la seconde sur la disparité calculée indépendamment sur chaque composante couleur. Ce pipeline permet la générationd’une image HDR par point de vue. Un algorithme de tone-mapping est ensuite appliqué à chacune d’elles afin qu’elles puissent être composées avec les filtres correspondants à l’écran autostéréoscopique considéré pour permettre la visualisation de l’image 3D HDR
HDR imaging and 3D imaging are two areas in which the simultaneous but separate development has been growing in recent years. On the one hand, HDR (High Dynamic Range) imaging allows to extend the dynamic range of traditionnal images called LDR (Low Dynamic Range). On the other hand, 3Dimaging offers immersion in the shown film with the feeling to be part of the acquired scene. Recently, these two areas have been combined to provide 3D HDR images or videos but few viable solutions existand none of them is available to the public. In this thesis, we propose a method to generate 3D HDR images for autostereoscopic displays by adapting a multi-viewpoints camera to several exposures acquisition.To do that, neutral density filters are fixed on the objectives of the camera. Then, pixel matchingis applied to aggregate pixels that represent the same point in the acquired scene. Finally, radiance is calculated for each pixel of the set of images by using a weighted average of LDR values. An additiona lstep is necessary because some pixels have wrong radiance. We proposed a method based on the color of adjacent pixels and two methods based on the correction of the disparity of those pixels. The first method is based on the disparity of pixels of the neighborhood and the second method on the disparity independently calculated on each color channel. This pipeline allows the generation of 3D HDR image son each viewpoint. A tone-mapping algorithm is then applied on each of these images. Their composition with filters corresponding to the autostereoscopic screen used allows the visualization of the generated 3DHDR image

This work presents a framework for shading of virtual objects using high dynamic range (HDR) light probe sequences in real-time. The method is based on using HDR environment map of the scene which is captured in an on-line process by HDR video camera as light probes [32]. In each frame of the HDR video, an optimized CUDA kernel is used to project incident lighting into spherical harmonics in real time. Transfer coefficients are calculated in an offline process. Using precomputed radiance transfer the radiance calculation reduces to a low order dot product between lighting and transfer coefficients. We exploit temporal coherence between frames to further smooth lighting variation over time. Our results show that the framework can achieve the effects of consistent illumination in real-time with flexibility to respond to dynamic changes in the real environment. We are using low-order spherical harmonics for representing both lighting and transfer functions to avoid aliasing.

Orientador: Prof. Dr. André Guilherme Ribeiro Balan
Dissertação (mestrado) - Universidade Federal do ABC, Programa de Pós-Graduação em Ciência da Computação, 2014.
A fotografia é uma atividade em grande crescimento e desenvolvimento, não apenas entre prossionais, mas também para a sociedade como um todo. Espera-se que a imagem tomada de uma determinada cena seja tão real quanto possível, e, por sua vez, que os equipamentos existentes sejam capazes de obter e visualizar essas imagens, o mais el possível da cena que está sendo registrada. O trabalho desenvolvido e apresentado nesta dissertação busca fazer o levantamento e estudo de técnicas que manipulem imagens HDR (High Dynamic Range), ou seja, imagens que possuem grande quantidade de informações da cena que representa, a m de torná-las visualizáveis com todos os detalhes nela contidos de maneira mais real possível ou de forma artística. A manipulação necessária para tais imagens é realizada por meio do mapeamento das imagens HDR para imagens LDR (Low Dynamic Range). O mapeamento das imagens HDR pode ser realizado com operadores de tone mapping e, como abordado nesta dissertação, com a decomposição multiescala. A decomposição multiescala oferece resultados de alta qualidade, tornando se um método de grande importância para o área de Processamento de Imagens, pelo fato de dividir a imagem de entrada em camadas e manipulá-las individualmente, para depois restaurá-la. Neste trabalho são avaliados métodos de ltragem não linear e operadores de tone mapping que melhor se adequam ao processo de decomposição multiescala e ao método de decomposição multiescala juntamente com a aplicação de compressão das camadas obtidas no processo de decomposição, a m de obter imagens reais com aprimoramento e destaque de seus detalhes. Adicionalmente, é proposto um novo operador de tone mapping local baseado no operador local de Reinhard, com as mesmas características e, com ajuste de parâmetro, que obtém resultados mais robustos que o operador local de Reinhard. Com isso, novos parâmetros ou métodos são propostos para aumentar a qualidade das imagens obtidas.
Photography is an activity in huge growth and development, not only among professionals, but also to society as a whole. It is expected that an image taken of a certain scene be as real as possible and, for its turn, that the existing equipment could obtain and visualize those images as accurately as the scene being recorded. The work developed and presented in this dissertation seeks to do a survey and a study of techniques which manipulate HDR (High Dynamic Range) images, in other words, of images that have large amount of information of a scene that is represented, in order to turn the images viewable with all the details they have as accurately as possible or in an artistic format. The required manipulation of those images is held by the mapping from HDR images to LDR (Low Dynamic Range) images. The HDR images mapping can be done with tone mapping operators and, as discussed in this dissertation, with the multiscale decomposition.The multiscale decomposition oers high quality results, being a method of great relevance to the Image Processing area, by the fact that it divides the input image in layers and manipulates these individually, to restore the image, after that. In this work, the non-linear lter methods and tone mapping operators that best t the multiscale decomposition process and multiscale decomposition method along with the application of layers compression, are evaluated to obtain real images with improvement and details highlighted. Besides that, a new tone mapping operator is proposed, based on the Reinhard local operator, with the same characteristics and with parameter settings, which gives more robust results than the Reinhard local operator. Thus, new parameters or methods are suggested to increase the obtained image quality.

Vision has always been one of the most important cognitive tools of human beings. In this regard, the development of image sensors opens up the potential to view objects that our eyes cannot see. One of the most promising capability in some image sensors is their single-photon sensitivity that provides information at the ultimate fundamental limit of light. Time-resolved single-photon avalanche diode (SPAD) image sensors bring a new dimension as they measure the arrival time of incident photons with a precision in the order of hundred picoseconds. In addition to this characteristic, they can be fabricated in complementary metal-oxide-semiconductor (CMOS) technology enabling the integration of complex signal processing blocks at the pixel level. These unique features made CMOS SPAD sensors a prime candidate for a broad spectrum of applications. This thesis is dedicated to the optimization and characterization of quantum imagers based on the SPADs as part of the E.U. funded SUPERTWIN project to surpass the fundamental diffraction limit known as the Rayleigh limit by exploiting the spatio-temporal correlation of entangled photons. The first characterized sensor is a 32×32-pixel SPAD array, named “SuperEllen”, with in-pixel time-to-digital converters (TDC) that measure the spatial cross-correlation functions of a flux of entangled photons. Each pixel features 19.48% fill-factor (FF) in 44.64-μm pitch fabricated in a 150-nm CMOS standard technology. The sensor is fully characterized in several electro-optical experiments, in order to be used in quantum imaging measurements. Moreover, the chip is calibrated in terms of coincidence detection achieving the minimal coincidence window determined by the SPAD jitter. The second developed sensor in the context of SUPERTWIN project is a 224×272-pixel SPAD-based array called “SuperAlice”, a multi-functional image sensor fabricated in a 110-nm CMOS image sensor technology. SuperAlice can operate in multiple modes (time-resolving or photon counting or binary imaging mode). Thanks to the digital intrinsic nature of SPAD imagers, they have an inherent capability to achieve a high frame rate. However, running at high frame rate means high I/O power consumption and thus inefficient handling of the generated data, as SPAD arrays are employed for low light applications in which data are very sparse over time and space. Here, we present three zero-suppression mechanisms to increase the frame rate without adversely affecting power consumption. A row-skipping mechanism that is implemented in both SuperEllen and SuperAlice detects the absence of SPAD activity in a row to increase the duty cycle. A current-based mechanism implemented in SuperEllen ignores reading out a full frame when the number of triggered pixels is less than a user-defined value. A different zero-suppression technique is developed in the SuperAlice chip that is based on jumping through the non-zero pixels within one row. The acquisition of TDC-based SPAD imagers can be speeded up further by storing and processing events inside the chip without the need to read out all data. An on-chip histogramming architecture based on analog counters is developed in a 150-nm CMOS standard technology. The test structure is a 16-bin histogram with 9 bit depth for each bin. SPAD technology demonstrates its capability in other applications such as automotive that demands high dynamic range (HDR) imaging. We proposed two methods based on processing photon arrival times to create HDR images. The proposed methods are validated experimentally with SuperEllen obtaining >130 dB dynamic range within 30 ms of integration time and can be further extended by using a timestamping mechanism with a higher resolution.

Vision has always been one of the most important cognitive tools of human beings. In this regard, the development of image sensors opens up the potential to view objects that our eyes cannot see. One of the most promising capability in some image sensors is their single-photon sensitivity that provides information at the ultimate fundamental limit of light. Time-resolved single-photon avalanche diode (SPAD) image sensors bring a new dimension as they measure the arrival time of incident photons with a precision in the order of hundred picoseconds. In addition to this characteristic, they can be fabricated in complementary metal-oxide-semiconductor (CMOS) technology enabling the integration of complex signal processing blocks at the pixel level. These unique features made CMOS SPAD sensors a prime candidate for a broad spectrum of applications. This thesis is dedicated to the optimization and characterization of quantum imagers based on the SPADs as part of the E.U. funded SUPERTWIN project to surpass the fundamental diffraction limit known as the Rayleigh limit by exploiting the spatio-temporal correlation of entangled photons. The first characterized sensor is a 32×32-pixel SPAD array, named “SuperEllen”, with in-pixel time-to-digital converters (TDC) that measure the spatial cross-correlation functions of a flux of entangled photons. Each pixel features 19.48% fill-factor (FF) in 44.64-μm pitch fabricated in a 150-nm CMOS standard technology. The sensor is fully characterized in several electro-optical experiments, in order to be used in quantum imaging measurements. Moreover, the chip is calibrated in terms of coincidence detection achieving the minimal coincidence window determined by the SPAD jitter. The second developed sensor in the context of SUPERTWIN project is a 224×272-pixel SPAD-based array called “SuperAlice”, a multi-functional image sensor fabricated in a 110-nm CMOS image sensor technology. SuperAlice can operate in multiple modes (time-resolving or photon counting or binary imaging mode). Thanks to the digital intrinsic nature of SPAD imagers, they have an inherent capability to achieve a high frame rate. However, running at high frame rate means high I/O power consumption and thus inefficient handling of the generated data, as SPAD arrays are employed for low light applications in which data are very sparse over time and space. Here, we present three zero-suppression mechanisms to increase the frame rate without adversely affecting power consumption. A row-skipping mechanism that is implemented in both SuperEllen and SuperAlice detects the absence of SPAD activity in a row to increase the duty cycle. A current-based mechanism implemented in SuperEllen ignores reading out a full frame when the number of triggered pixels is less than a user-defined value. A different zero-suppression technique is developed in the SuperAlice chip that is based on jumping through the non-zero pixels within one row. The acquisition of TDC-based SPAD imagers can be speeded up further by storing and processing events inside the chip without the need to read out all data. An on-chip histogramming architecture based on analog counters is developed in a 150-nm CMOS standard technology. The test structure is a 16-bin histogram with 9 bit depth for each bin. SPAD technology demonstrates its capability in other applications such as automotive that demands high dynamic range (HDR) imaging. We proposed two methods based on processing photon arrival times to create HDR images. The proposed methods are validated experimentally with SuperEllen obtaining >130 dB dynamic range within 30 ms of integration time and can be further extended by using a timestamping mechanism with a higher resolution.

The continuous challenge of capturing and representing the high fidelity representation of the real-world scene on display devices to invoke true-to-life visual sensations has been the driving force behind the development of High Dynamic Range (HDR) imaging technologies. HDR imaging technologies enables the capture, storage and display of the light levels and colours inherent in the original capture of content. By doing so, it enables the HDR video to provide the sensation and quality which is closer to the seamless and immersive real-world appearance. However, existing conventional display devices being Low Dynamic Range (LDR) in nature and multimedia delivery technologies being designed and optimized for the 8-bit LDR video signal, does not support HDR video visualization and distribution, respectively. In this thesis, we explore how to exploit and utilize the existing multimedia delivery technologies for HDR video distribution keeping in view the need for backward-compatibility with already existing LDR displays. Unlike LDR video, HDR video utilizes higher bit-depth to faithfully preserve the luminance dynamic range encountered in the real-world scene and thus, cannot be distributed through current 8-bit video distribution infrastructure. Further, although HDR information can be utilized for rendering a closer to real-world LDR representation, this cannot be done by the LDR video player and display devices due to the incompatibility with HDR video. These challenges are addressed in our first contribution. We propose a two-layer scalable HDR video distribution method which is backward-compatible with the existing encoders and with ubiquitous LDR display devices. The base layer in the proposed method carries the LDR representation for LDR displays, generated from the HDR video. Whereas, the extension layer encodes the HDR information as a 8-bit video signal. An efficient mapping function is employed in the extension layer to improve the coding efficiency of the extension layer which in turn improves the HDR video quality in comparison to the other backward-compatible HDR video distribution methods. A review and performance evaluation of the state-of-the-art HDR video distribution methods is presented in our second contribution. This was done to categorize and critique the existing HDR video distribution methods in order to contribute to ongoing HDR video distribution standardization efforts which are presently impeded by the lack of comprehensive performance evaluation. 5 HDR video distribution methods, including ones in contention for being standardized as HDR video distribution method, were evaluated to present the performance assessment and behaviour of these methods for HDR video sequences having different characteristics. We showed that the performance of HDR video distribution methods is generally affected by the dynamic range and spatio-temporal characteristics of the HDR videos. Further, apart from identifying the most suitable distribution method for HDR video, we also highlighted the discrepancy in the prediction of HDR quality measured through the different HDR quality metrics employed. In our last contribution, we first presented the novel usage of the Perceptual Transfer Function (PTF) for improving and optimizing the compression performance of the state-of-the-art video compression standard; High Efficiency Video Coding (HEVC), for the HDR video compression. The PTF is utilized in the Rate-Distortion Optimization (RDO) process and results in an improved coding performance in terms of the Rate-Distortion efficiency for HDR video compression. We showed that using a PTF which is backward-compatible with the LDR transfer function can improve the HEVC compression efficiency which implies that HEVC needs to be optimized for the HDR video compression. Based on these observation, we further optimized the HEVC RateControl (RC) algorithm for the HDR video compression. A new λ-QP relationship is proposed which better estimates the relation between the HDR distortions and the bitrate utilized. As a result, intelligent bitrate allocation to the coding levels is achieved keeping in view the characteristics of the HDR content which results in an improved HDR quality at the same bitrate in comparison to the default RC algorithm for the HDR video.

L'un des buts principaux de l'imagerie numérique est d'une part la capture et d'autre part la reproduction de scènes réelles ou synthétiques sur des dispositifs d'affichage aux capacités restreintes. Les techniques d'imagerie traditionnelles sont limitées par la gamme de luminance qu'elles peuvent capturer et afficher. L'imagerie à grande gamme de luminance (High Dynamic Range – HDR) vise à dépasser cette limitation en capturant, représentant et affichant les quantités physique de la lumière présente dans une scène. Cependant, les technologies d'affichage existantes ne vont pas disparaitre instantanément, la compatibilité entre ces nouveaux contenus HDR et les contenus classiques est donc requise. Cette compatibilité est assurée par une opération de réduction des gammes de luminance (tone mapping) qui adapte les contenus HDR aux capacités restreintes des écrans. Bien que de nombreux opérateurs de tone mapping existent, ceux-ci se focalisent principalement sur les images fixes. Les verrous scientifiques associés au tone mapping de vidéo HDR sont plus complexes du fait de la dimension temporelle. Les travaux recherche menés dans la thèse se sont focalisés sur la préservation de la cohérence temporelle du vidéo tone mapping. Deux principaux axes de recherche ont été traités : la qualité subjective de contenus tone mappés et l'efficacité de la compression des vidéos HDR. En effet, tone mapper individuellement chaque image d'une séquence vidéo HDR engendre des artefacts temporels. Ces artefacts affectent la qualité visuelle de la vidéo tone mappée et il est donc nécessaire de les minimiser. Au travers de tests effectués sur des vidéos HDR avec différents opérateurs de tone mapping, nous avons proposé une classification des artefacts temporels en six catégories. Après avoir testé les opérateurs de tone mapping vidéo existants sur les différents types d'artefacts temporels, nous avons observé que seulement trois des six types d'artefacts étaient résolus. Nous avons donc créé une technique de post-traitement qui permet de réduire les 3 types d'artefacts non-considérés. Le deuxième aspect considéré dans la thèse concerne les relations entre compression et tone mapping. Jusque là, les travaux effectués sur le tone mapping et la vidéo compression se focalisaient sur l'optimisation du tone mapping de manière à atteindre des taux de compression élevés. Ces techniques modifient fortement le rendu, c'est à dire l'aspect de la vidéo, modifiant ainsi l'intention artistique initiale en amont dans la chaine de distribution (avant la compression). Dans ce contexte, nous avons proposé une technique qui permet de réduire l'entropie d'une vidéo tone mappée sans en modifier son rendu. Notre méthode adapte la quantification afin d'accroitre les corrélations entre images successives d'une vidéo
One of the main goals of digital imagery is to improve the capture and the reproduction of real or synthetic scenes on display devices with restricted capabilities. Standard imagery techniques are limited with respect to the dynamic range that they can capture and reproduce. High Dynamic Range (HDR) imagery aims at overcoming these limitations by capturing, representing and displaying the physical value of light measured in a scene. However, current commercial displays will not vanish instantly hence backward compatibility between HDR content and those displays is required. This compatibility is ensured through an operation called tone mapping that retargets the dynamic range of HDR content to the restricted dynamic range of a display device. Although many tone mapping operators exist, they focus mostly on still images. The challenges of tone mapping HDR videos are more complex than those of still images since the temporal dimensions is added. In this work, the focus was on the preservation of temporal coherency when performing video tone mapping. Two main research avenues are investigated: the subjective quality of tone mapped video content and their compression efficiency. Indeed, tone mapping independently each frame of a video sequence leads to temporal artifacts. Those artifacts impair the visual quality of the tone mapped video sequence and need to be reduced. Through experimentations with HDR videos and Tone Mapping Operators (TMOs), we categorized temporal artifacts into six categories. We tested video tone mapping operators (techniques that take into account more than a single frame) on the different types of temporal artifact and we observed that they could handle only three out of the six types. Consequently, we designed a post-processing technique that adapts to any tone mapping operator and reduces the three types of artifact not dealt with. A subjective evaluation reported that our technique always preserves or increases the subjective quality of tone mapped content for the sequences and TMOs tested. The second topic investigated was the compression of tone mapped video content. So far, work on tone mapping and video compression focused on optimizing a tone map curve to achieve high compression ratio. These techniques changed the rendering of the video to reduce its entropy hence removing any artistic intent or constraint on the final results. That is why, we proposed a technique that reduces the entropy of a tone mapped video without altering its rendering

High dynamic range (HDR) imaging is gaining widespread acceptance in computer graphics, photography and multimedia industry. Representing scenes with values corresponding to real-world light levels, HDR images and videos provide superior picture quality and more life-like visual experience than traditional 8-bit Low Dynamic Range (LDR) content. In this thesis, we present a few attempts to assess and improve the quality of HDR using subjective and objective approaches. We first conducted in-depth studies regarding HDR compression and HDR quality metrics. We show that High Efficiency Video Coding (HEVC) outperforms the previous version of compression standard on HDR content and could be used as a platform for HDR compression if provided with some necessary extensions. We also find that, compared to other quality metrics, the Visual Information Fidelity (VIF) quality metric has the highest correlation with subjective opinions on HDR videos. These findings contributed to the development of methods that optimize existing video compression standards for HDR applications. Next, the viewing experience of HDR content is evaluated both subjectively and objectively. The study shows a clear subjective preference for HDR content when individuals are given a choice between HDR and LDR displays. Eye tracking data were collected from individuals viewing HDR content in a free-viewing task. These eye tracking data collected are utilized in the development of a visual attention model for HDR content. Last but not least, we propose a computational approach to predict visual attention for HDR video content, the only one of its kind as all existing visual attention models are designed for HDR images. This proposed approach simulates the characteristics of the Human Visual System (HVS) and makes predictions by combining the spatial and temporal visual features. The analysis using eye tracking data affirms the effectiveness of the proposed model. Comparisons employing three well known quantitative metrics show that the proposed model substantially improves predictions of visual attention of HDR.
Applied Science, Faculty of
Electrical and Computer Engineering, Department of
Graduate

The target of this thesis is sumarize the theory of metods used for composition image sequence into single high dynamic range image. Selected methods and algorithms are described in theoretical part and implemented in practical part of this thesis. It was used C++ language with using OpenCV library.

Ces dernières années, les contenus vidéos ont évolué très rapidement. En effet, les télévisions (TV) ont rapidement évolué vers l’Ultra Haute résolution (UHD), la Haute Fréquence d’images (HFR) ou la stéréoscopie (3D). La tendance actuelle est à l’imagerie à Haute Dynamique de luminance (HDR). Ces technologies permettent de reproduire des images beaucoup plus lumineuses que celles des écrans actuels. Chacune de ces améliorations représente une augmentation du coût de stockage et nécessite la création de nouveaux standards de compression vidéo, toujours plus performant. La majorité des consommateurs est actuellement équipé de TV ayant une Dynamique Standard (SDR) qui ne supportent pas les contenus HDR et ils vont lentement renouveler leurs écrans pour un HDR. Il est donc important de délivrer un signal HDR qui puisse être décodé par ces deux types d’écrans. Cette rétro compatibilité est rendue possible par un outil appelé TMO (Tone Mapping Operator) qui transforme un contenu HDR en une version SDR. Au travers de cette thèse, nous explorons de nouvelles méthodes pour améliorer la compression HDR rétro compatible. Premièrement, nous concevons un TMO qui optimise les performances d’un schéma de compression scalable où une couche de base et d’amélioration sont envoyées pour reconstruire les contenus HDR et SDR. Il est démontré que le TMO optimal dépend seulement de la couche SDR de base et que le problème de minimisation peut être séparé en deux étapes consécutives. Pour ces raisons, nous proposons ensuite un autre TMO conçu pour optimiser les performances d’un schéma de compression utilisant uniquement une couche de base mais avec un modèle amélioré et plus précis. Ces deux travaux optimisent des TMO pour images fixes. Par la suite, la thèse se concentre sur l’optimisation de TMO spécifiques à la vidéo. Cependant, on y démontre que l’utilisation d’une prédiction pondérée pour la compression SDR est aussi bon voir meilleur que d’utiliser un TMO optimisé temporellement. Pour ces raisons, un nouvel algorithme et de nouveaux modes de prédictions pondérées sont proposés pour gérer plus efficacement la large diversité des changements lumineux dans les séquences vidéos
In recent years, video content evolved very quickly. Indeed, televisions (TV) quickly evolved to Ultra High Definition (UHD), High Frame Rate (HFR) or stereoscopy (3D). The recent trend is towards High Dynamic range (HDR). These new technologies allow the reproduction of much brighter images than for actual displays. Each of these improvements represents an increase in storage cost and therefore requires the creation of new video compression standards, always more efficient. The majority of consumers are currently equipped with Standard Dynamic Range (SDR) displays, that cannot handle HDR content. Consumers will slowly renew their display to an HDR one and it is therefore of great importance to deliver an HDR signal that can be decoded by both SDR and HDR displays. Such backward compatibility is provided by a tool called Tone Mapping Operator (TMO) which transforms an HDR content into an SDR version. In this thesis, we explore new methods to improve the backward compatible HDR compression. First, we design a Tone Mapping to optimize scalable compression scheme performances where a base and an enhancement layer are sent to reconstruct the SDR and HDR content. It is demonstrated that the optimum TMO only depends on the SDR base layer and that the minimization problem can be separated in two consecutive minimization steps. Based on these observations, we then propose another TMO designed to optimize the performances of compression schemes using only a base layer but with an enhanced and more precise model. Both of these works optimize TMO for still images. Thereafter, this thesis focuses on the optimization of video-specific TMO. However, we demonstrate that using a weighted prediction for the SDR compression is as good or even better than using a temporally optimized TMO. Therefore, we proposed a new weighted prediction algorithm and new weighted prediction modes to handle more efficiently the large diversity of brightness variations in video sequences

We describe the design and implementation of a high dynamic range (HDR) imaging system capable of capturing RGB color images with a dynamic range of 10,000,000 : 1 at 25 frames per second. We use a highly programmable camera unit with high throughput A/D conversion, data processing and data output. HDR acquisition is performed by multiple exposures in a continuous rolling shutter progression over the sensor. All the different exposures for one particular row of pixels are acquired head to tail within the frame time, which means that the time disparity between exposures is minimal, the entire frame time can be used for light integration and the longest expo- sure is almost the entire frame time. The system is highly configurable, and trade-offs are possible between dynamic range, precision, number of exposures, image resolution and frame rate.

La conversion d'une image à grande gamme dynamique (HDR) en une image à faible gamme dynamique est étudiée de façon à garantir un rendu visuel de cette dernière de bonne qualité. La première contribution concerne le rehaussement de contraste de l'image mappée en utilisant une fonction linéaire par morceaux pour que l'égalisation d'histogramme soit ajustée à la "s-courbe" d'adaptation du système visuel humain. la deuxième et troisième contributions portent sur la préservation des détails de l'image HDR. Des approches multirésolution séparables et non séparables, basées sur des stratégies non oscillatoires, prenant en compte les singularités de l'image HDR dans la dérivation du modèle mathématique, sont proposées. La quatrième contribution non seulement préserve les détails mais également améliore le contraste de l'image HDR mappée. Un schéma de lifting séparable "presque optimal" est proposé. Il s'appuie sur une étape de prédiction adaptative des coefficients. Cette dernière repose sur une combinaison linéaire pondérée des coefficients voisins pour extraire les détails pertinents sur l'image HDR à chaque niveau de résolution. Un mappage linéaire par morceaux est ensuite appliqué à la reconstruction grossière. les résultats de simulation fournissent de bonnes performances en termes de qualité visuelle et de métrique TMQI (Tone Mapped Quality Index) par rapport aux approches de mise en correspondance tonale classiques. l'impact des paramètres TMQI sur la qualité visuelle des images mappées est discuté. Les paramètres proposés montrent une forte corrélation entre la métrique modifiée et la note moyenne d'opinion
He conversion of High Dynamic Range (HDR) image into Low Dynamic Range (LDR) image is investigated so that the visual rendering of the latter is of good quality. The first contribution focused on the contrast enhancement of the tone mapped image using a piecewise linear function as a non-uniform histogram equalization adjustment to mode! the "s-shaped" curve of the human visual adaptation. The second and third contributions are concerned with the details preservation of the HDR image on the tone mapped image. Separable and non-separable multiresolution approaches based on essential non-oscillatory strategies, taking into account the HDR image singularities in the mathematical mode! derivation, are proposed. The fourth contribution not only preserves details but also enhances the contrast of the HDR tone mapped image. A separable "near optimal" lifting scheme using an adaptive powerful prediction step is proposed. The latter relies on a linear weighted combination depending on the neighbouring coefficients to extract the relevant fin est details on the HDR image at each resolution level. A piecewise linear mapping is then applied on the coarse reconstruction. Simulation results provide good performance, both in terms of visual quality and Tone Mapped Quality Index (TMQI) metric, compared to existing competitive tone mapping approaches. The impact of the TMQI parameters on the visual quality of the tone mapped images is discussed. The proposed parameters show a strong correlation between the modified metric and

Les technologies d'écran ont connu récemment une évolution rapide. De la télévision 3D à l'Ultra Haute Définition, la tendance est maintenant aux écrans HDR (pour ''High Dynamic Range'') permettant de reproduire une gamme de luminance bien plus élevée que les écrans classiques. L'émergence de cette technologie implique de nouveaux travaux de standardisation dans le domaine de la compression vidéo. Une question essentielle pour la distribution à grande échelle de contenu HDR est celle de la rétro-compatibilité. Tandis que la future génération d'écrans de télévision sera adaptée à ce nouveau format, il est nécessaire de permettre aux équipements plus anciens de décoder et afficher une version du même contenu dont la dynamique a été préalablement réduite par un procédé appelé ''tone mapping''. Cette thèse vise à explorer les schémas de compression HDR rétro-compatibles. Dans une première approche, un algorithme de tone mapping spécifié par l'encodeur est appliqué à l'image HDR. L'image générée, alors appelée LDR (pour ''Low Dynamic Range''), peut être encodée et décodée dans un format classique. L'encodeur transmet par ailleurs une quantité réduite d'information permettant à un décodeur HDR d'inverser l'opération de tone mapping et de reconstruire une version HDR. L'étude de ces schémas est axée sur la définition de méthodes de tone mapping optimisées pour les performances de compression. La suite de la thèse se concentre sur l'approche scalable dans laquelle les deux versions sont fournies à l'encodeur sans connaissance à priori sur l'opérateur de tone mapping utilisé. Le producteur garde donc le contrôle sur la création du contenu LDR. Cette version LDR est d'abord compressée comme une première couche. L'image reconstruite est utilisée par le codeur scalable pour compresser plus efficacement la couche HDR grâce à un mécanisme de prédiction inter-couches. Notre approche locale et non linéaire nous permet d'améliorer les performances de codage par rapport aux méthodes scalables existantes, en particulier dans le cas où un tone mapping complexe est utilisé pour générer la version LDR
In recent years, the display technologies have been rapidly evolving. From 3D television to Ultra High Definition, the trend is now towards High Dynamic Range (HDR) displays that can reproduce a luminance range far beyond the capabilities of conventional displays. The emergence of this technology involves new standardization effort in the field of video compression. In terms of large scale content distribution, the question of backward compatibility is critical. While the future generation of television displays will be adapted to this new format, it is necessary to enable the older equipment to decode and display a version of the same content whose dynamic range has been previously reduced by a process called “tone mapping”. This thesis aims at exploring the backward compatible HDR compression schemes. In a first approach, a tone mapping operator specified by the encoder is applied to the HDR image. The resulting image, called Low Dynamic Range (LDR), can then be encoded and decoded in a conventional format. The encoder additionally transmits a small amount of information enabling a HDR capable decoder to inverse the tone mapping operator and retrieve the HDR version. The study of these schemes is directed towards the definition of tone mapping operators optimized for the compression performance. We then focus on scalable approaches, where both versions are given to the encoder without prior knowledge on the tone mapping operator used. The producer thus keeps full control on the LDR content creation process. This LDR version is compressed as a first layer. The reconstructed image is used by the scalable encoder to compress the HDR layer efficiently by performing inter-layer predictions. Thanks to a local and non-linear approach, the proposed schemes improve the coding performance compared to the existing scalable methods, especially in the case where a complex tone mapping is used for generating the LDR version

This work presents a survey and a user evaluation of tone mapping operators (TMOs) for high dynamic range (HDR) video, i.e. TMOs that explicitly include a temporal model for processing of variations in the input HDR images in the time domain. The main motivations behind this work is that: robust tone mapping is one of the key aspects of HDR imaging [Reinhard et al. 2006]; recent developments in sensor and computing technologies have now made it possible to capture HDR-video, e.g. [Unger and Gustavson 2007; Tocci et al. 2011]; and, as shown by our survey, tone mapping for HDR video poses a set of completely new challenges compared to tone mapping for still HDR images. Furthermore, video tone mapping, though less studied, is highly important for a multitude of applications including gaming, cameras in mobile devices, adaptive display devices and movie post-processing. Our survey is meant to summarize the state-of-the-art in video tonemapping and, as exemplified in Figure 1 (right), analyze differences in their response to temporal variations. In contrast to other studies, we evaluate TMOs performance according to their actual intent, such as producing the image that best resembles the real world scene, that subjectively looks best to the viewer, or fulfills a certain artistic requirement. The unique strength of this work is that we use real high quality HDR video sequences, see Figure 1 (left), as opposed to synthetic images or footage generated from still HDR images.
VPS

Cílem této práce je demonstrovat možnost renderování srsti přímo do existujících obrazů bez toho, aby bylo po uživateli požadováno překreslení všech pixelů nebo dodání kompletní 3D geometrie a osvětlení. Srst je přidána na povrch objektů pomocí extrakce jejich přibližného tvaru a světelných informací z obrazu a takto získaný objekt je poté přerenderován. Tento přístup je nový v tom, že vysokoúrovňové úpravy obrazu (jako např. přidání srsti), mohou úspěšně vést k vizuálně korektním výsledkům a to i přes omezení nepřesnou geometrií a světelnými podmínkami. Relativně velká množina technik použitých v této práci zahrnuje obrazy s velkým dynamickým rozsahem, metody extrakce 3D tvaru z obrazu, výsledky výzkumu vnímání tvaru a osvětlení a fotorealistické renderování. Hlavním cílem práce je potvrdit koncept popsaný výše. Hlavním implementačním jazykem bylo C++ s použitím knihoven wxWidgets, OpenGL a libTIFF. Renderování bylo realizováno v software 3Delight kompatibilním se standardem Renderman, za pomoci množiny shaderů implementovaných v nativním jazyce Rendermanu.

The recent advances in sensor and display technologies have brought upon the High Dynamic Range (HDR) imaging capability. The modern multiple exposure HDR sensors can achieve the dynamic range of 100-120 dB and LED and OLED display devices have contrast ratios of 10^5:1 to 10^6:1. Despite the above advances in technology the image/video compression algorithms and associated hardware are yet based on Standard Dynamic Range (SDR) technology, i.e. they operate within an effective dynamic range of up to 70 dB for 8 bit gamma corrected images. Further the existing infrastructure for content distribution is also designed for SDR, which creates interoperability problems with true HDR capture and display equipment. The current solutions for the above problem include tone mapping the HDR content to fit SDR. However this approach leads to image quality associated problems, when strong dynamic range compression is applied. Even though some HDR-only solutions have been proposed in literature, they are not interoperable with current SDR infrastructure and are thus typically used in closed systems. Given the above observations a research gap was identified in the need for efficient algorithms for the compression of still images and video, which are capable of storing full dynamic range and colour gamut of HDR images and at the same time backward compatible with existing SDR infrastructure. To improve the usability of SDR content it is vital that any such algorithms should accommodate different tone mapping operators, including those that are spatially non-uniform. In the course of the research presented in this thesis a novel two layer CODEC architecture is introduced for both HDR image and video coding. Further a universal and computationally efficient approximation of the tone mapping operator is developed and presented. It is shown that the use of perceptually uniform colourspaces for internal representation of pixel data enables improved compression efficiency of the algorithms. Further proposed novel approaches to the compression of metadata for the tone mapping operator is shown to improve compression performance for low bitrate video content. Multiple compression algorithms are designed, implemented and compared and quality-complexity trade-offs are identified. Finally practical aspects of implementing the developed algorithms are explored by automating the design space exploration flow and integrating the high level systems design framework with domain specific tools for synthesis and simulation of multiprocessor systems. The directions for further work are also presented.

Master's thesis is focused on capturing of low dynamic range images using common devices such as camera and its multiple exposure. The main part of thesis is dedicated to composing these images to HDR image, inclusive sequence of images of static scenes, but also dynamic ones. Next part describes tone mapping used for display HDR image on LDR monitors. Moreover, there is given design and implementation of application solving problems mentioned earlier. In the end, the implemented application is evaluated and the possible continuation of this work is stated.

This thesis focuses on the high dynamic range (HDR) imaging problematic. It describes process of capturing and storage of HDR images in brief. Main part of thesis deals with techniques of how to display HDR images on common visual display devices, which are not able to represent them directly. This process is called tone-mapping. It is firstly described in general and than the text focuses on several techniques of tone-mapping, mainly on tone-mapping using bilateral filtering.

Au cours de la dernière décennie, la technologie de l’image et de la vidéo à haute gamme dynamique (High dynamic range - HDR) a attiré beaucoup d’attention, en particulier dans la communauté multimédia. Les progrés technologiques récents ont facilité l’acquisition, la compression et la reproduction du contenu HDR, ce qui a mené à la commercialisation des écrans HDR et à la popularisation du contenu HDR. Dans ce contexte, la mesure de la qualité du contenu HDR joue un rôle fondamental dans l’amélioration de la chaîne de distribution du contenu ainsi que des opérations qui la composent, telles que la compression et l’affichage. Cependant, l’évaluation de la qualité visuelle HDR présente de nouveaux défis par rapport au contenu à gamme dynamique standard (Standard dynamic range -SDR). Le premier défi concerne les nouvelles conditions introduites par la reproduction du contenu HDR, par ex. l’augmentation de la luminosité et du contraste. Même si une reproduction exacte de la luminance d’une scène n’est pas nécessaire pour la plupart des cas pratiques, une estimation précise de la luminance émise est cependant nécessaire pour les mesures d’évaluation objectives de la qualité HDR. Afin de comprendre les effets du rendu d’affichage sur la perception de la qualité, un algorithme permettant de reproduire très précisement une image HDR a été développé et une expérience subjective a été menée pour analyser l’impact de différents rendus sur l’évaluation subjective et objective de la qualité HDR. En outre, afin de comprendre l’impact de la couleur avec la luminosité accrue des écrans HDR, les effets des différents espaces de couleurs sur les performances de compression vidéo HDR ont également été analysés dans une autre étude subjective. Un autre défi consiste à estimer objectivement la qualité du contenu HDR, en utilisant des ordinateurs et des algorithmes. Afin de relever ce défi, la thèse procède à l’évaluation des performances des métriques de qualité d’image HDR avec référence (Full reference-FR). Les images HDR ont une plus grande plage de luminosité et des valeurs de contraste plus élevées. Etant donné que la plupart des métriques de qualité d’image sont développées pour les images SDR, elles doivent être adaptées afin d’estimer la qualité des images HDR. Différentes méthodes d’adaptation ont été utilisées pour les mesures SDR, et elles ont été comparées avec les métriques de qualité d’image existantes développées exclusivement pour les images HDR. De plus, nous proposons une nouvelle méthode d’évaluation des métriques objectives basée sur une nouvelle approche de classification. Enfin, nous comparons les scores de qualité subjectifs acquis en utilisant différentes méthodologies de test subjectives. L’évaluation subjective de la qualité est considérée comme le moyen le plus efficace et le plus fiable d’obtenir des scores de qualité «vérité-terrain» pour les stimuli sélectionnés, et les scores moyens d’opinion (Mean opinion scores-MOS) obtenus sont les valeurs auxquelles les métriques objectives sont entraînées pour correspondre. En fait, de fortes divergences peuvent facilement être rencontrés lorsque différentes bases de données de qualité multimédia sont considérées. Afin de comprendre la relation entre les valeurs de qualité acquises à l’aide de différentes méthodologies, la relation entre les valeurs MOS et les résultats des comparaisons par paires rééchellonés (Pairwise comparisons - PC) a été comparée. A cette fin, une série d’expériences ont été menées entre les méthodologies double stimulus impairment scale (DSIS) et des comparaisons par paires. Nous proposons d’inclure des comparaisons inter-contenu dans les expériences PC afin d’améliorer les performances de rééchelonnement et de réduire la variance inter-contenu ainsi que les intervalles de confiance. Les scores de PC rééchellonés peuvent également être utilisés pour des scénarios subjectifs d’évaluation de la qualité multimédia autres que le HDR
In the last decade, high dynamic range (HDR) image and video technology gained a lot of attention, especially within the multimedia community. Recent technological advancements made the acquisition, compression, and reproduction of HDR content easier, and that led to the commercialization of HDR displays and popularization of HDR content. In this context, measuring the quality of HDR content plays a fundamental role in improving the content distribution chain as well as individual parts of it, such as compression and display. However, HDR visual quality assessment presents new challenges with respect to the standard dynamic range (SDR) case. The first challenge is the new conditions introduced by the reproduction of HDR content, e.g. the increase in brightness and contrast. Even though accurate reproduction is not necessary for most of the practical cases, accurate estimation of the emitted luminance is necessary for the objective HDR quality assessment metrics. In order to understand the effects of display rendering on the quality perception, an accurate HDR frame reproduction algorithm was developed, and a subjective experiment was conducted to analyze the impact of different display renderings on subjective and objective HDR quality evaluation. Additionally, in order to understand the impact of color with the increased brightness of the HDR displays, the effects of different color spaces on the HDR video compression performance were also analyzed in another subjective study. Another challenge is to estimate the quality of HDR content objectively, using computers and algorithms. In order to address this challenge, the thesis proceeds with the performance evaluation of full-reference (FR) HDR image quality metrics. HDR images have a larger brightness range and higher contrast values. Since most of the image quality metrics are developed for SDR images, they need to be adapted in order to estimate the quality of HDR images. Different adaptation methods were used for SDR metrics, and they were compared with the existing image quality metrics developed exclusively for HDR images. Moreover, we propose a new method for the evaluation of metric discriminability based ona novel classification approach. Motivated by the need to fuse several different quality databases, in the third part of the thesis, we compare subjective quality scores acquired by using different subjective test methodologies. Subjective quality assessment is regarded as the most effective and reliable way of obtaining “ground-truth” quality scores for the selected stimuli, and the obtained mean opinion scores (MOS) are the values to which generally objective metrics are trained to match. In fact, strong discrepancies can easily be notified across databases when different multimedia quality databases are considered. In order to understand the relationship between the quality values acquired using different methodologies, the relationship between MOS values and pairwise comparisons (PC) scaling results were compared. For this purpose, a series of experiments were conducted using double stimulus impairment scale (DSIS) and pairwise comparisons subjective methodologies. We propose to include cross-content comparisons in the PC experiments in order to improve scaling performance and reduce cross-content variance as well as confidence intervals. The scaled PC scores can also be used for subjective multimedia quality assessment scenarios other than HDR

Glare in open-plan offices can negatively affect the productivity and well-being of office workers. Accurate glare prediction is challenging, as occupants' sensitivity to glare may differ under the same conditions. Developed as part of an ARC Linkage Project, this thesis challenges the limitations prevalent in current glare metrics by delivering a new model of predicting glare for open-plan offices. By utilising machine learning (ML) techniques, more accurate tools and methods are unlocked to assist architects and lighting engineers in the early stages of the design process. They ultimately enable more efficient daylit office designs with reduced glare discomfort in Australia.

Ces dernières années, les technologies d’écran se sont considérablement améliorées. Par exemple, le contraste des écrans à plage dynamique élevée (HDR) dépasse de loin la capacité d’un écran conventionnel. De plus, un écran à gamut de couleur étendu (WCG) peut couvrir un espace colorimétrique plus grand que jamais. L'évaluation de la qualité de ces nouveaux contenus est devenue un domaine de recherche actif, les métriques de qualité SDR classiques n'étant pas adaptées. Cependant, les études les plus récentes négligent souvent une caractéristique importante: les chrominances. En effet, les bases de données existantes contiennent des images HDR avec un gamut de couleur standard, négligeant ainsi l’augmentation de l’espace colorimétrique due au WCG et les artefacts chromatiques. La plupart des mesures de qualité HDR objectives non plus ne prennent pas en compte ces artefacts. Pour surmonter cette problématique, dans cette thèse, nous proposons deux nouvelles bases de données HDR/WCG annotés avec des scores subjectifs présentant des artefacts chromatique réaliste. En utilisant ces bases de données, nous explorons trois solutions pour créer des métriques HDR/WCG: l'adaptation des métrics de qualité SDR, l’extension colorimétrique d’une métrique HDR connue appelée HDR-VDP-2 et, enfin, la fusion de diverses métriques de qualité et de features colorimétriques. Cette dernière métrique présente de très bonnes performances pour prédire la qualité tout en étant sensible aux distorsions chromatiques
To improve their ability to display astonishing images, screen technologies have been greatly evolving. For example, the contrast of high dynamic range rendering systems far exceed the capacity of a conventional display. Moreover, a Wide Color gamut display can cover a bigger color space than ever. Assessing the quality of these new content has become an active field of research as classical SDR quality metrics are not adapted. However, state-of-the-art studies often neglect one important image characteristics: chrominances. Indeed, previous databases contain HDR images with a standard gamut thus neglecting the increase of color space due to WCG. Due to their gamut, these databases are less prone to contain chromatic artifacts than WCG content. Moreover, most existing HDR objective quality metrics only consider luminance and are not considering chromatic artifacts. To overcome this problematic, in this thesis, we have created two HDR / WCG databases with annotated subjective scores. We focus on the creation of a realistic chromatic artifacts that can arise during compression. In addition, using these databases, we explore three solutions to create HDR / WCG metrics. First, we propose a method to adapt SDR metrics to HDR / WCG content. Then, we proposed an extension of a well-known HDR metric called HDR-VDP-2. Finally, we create a new metric based on the merger of various quality metric and color features. This last metric presents very good performance to predict quality while being sensitive to chromatic distortion

The goal of this thesis is to improve graphical user interface of Tone Mapping Studio(TMS) program. This program is being developed on the Faculty of Information Technology(FIT), Brno University of Technology (BUT) by doc. Ing. Martin Čadík, PhD. The current program is using framework Qt3 , which is old and not compatible with modern libraries. This program has to be rewritten to support current version Qt5. I will analyze other programs in the area of working with High Dynamic Range (HDR) images and video. Changes for improving the interface will be proposed and UX tests will be done. Second part will consist of comparing plug-ins for converting images to grayscale that already exists in TMS.

Cílem práce je vytvoření podpory pro editovaní videa v HDR formátu. Pro editaci videa je zvolen program Adobe Premiere Pro a na dosažení požadovaného výsledku je vytvořen plugin do zmiňovaného softwaru, který poskytuje požadovanou funkcionalitu. V práci jsou vysvětleny principy vytváření, zobrazení a ukládáni obsahu z rozšíreným dynamickým rozsahem. Zároveň jsou vysvětleny principy vytváření pluginů pro Adobe Premiere Pro za pomoci SDK verze CS6. V praktické části této práce jsou vysvětleny detaily implementace, problémy, které byly řešené, a popis samotného pluginu. Návrh pluginů je vytvořen tak, aby byla možná další práce na tomto softwaru, přidaní další funkcionality a pro umožnění využití tohoto díla na rozvoji obsahu s rozšíreným dynamickým rozsahem.

High dynamic range (HDR) images and videos provide superior picture quality by allowing a larger range of brightness levels to be captured and reproduced than their traditional 8-bit low dynamic range (LDR) counterparts. Even with existing 8-bit displays, picture quality can be significantly improved if the content is first captured in HDR format and then converted to LDR format. This converting process is called tone-mapping. In this thesis, we address different aspects of tone-mapping. HDR video formats are unlikely to be broadly accepted without the backward-compatibility with LDR devices. We first consider the case where only the tone-mapped LDR content is transmitted and the HDR video is reconstructed at the receiver by inversely tone-mapping the encoded-decoded LDR video. We show that the appropriate choice of a tone-mapping operator can result in a reconstructed HDR video with good quality. We develop a statistical model of the distortion resulting from tone-mapping, compressing, de-compressing and inverse tone-mapping the HDR video. This model is used to formulate an optimization problem that finds the tone-curve that minimizes the distortion in the reconstructed HDR video. We also derive a simplified version of the model that leads to a closed-form solution for the optimization problem. Next, we consider the case where the HDR content is transmitted using an LDR and an enhancement layers. We formulate an optimization problem that minimizes the transmitted bit-rate of a video sequence and also results in the tone-mapped video that satisfies some desired perceptual appearance. The problem formulation also contains a constraint that suppresses temporal flickering artifacts. We also propose a technique that tone-maps an HDR video directly in a compression friendly color space (e.g., YCbCr) without the need to convert it to the RGB domain. We study the design of 3D HDR-LDR tone-mapping operators. To find the appropriate tone-mapping characteristics that contribute to good 3D representation, subjective psychophysical experiments are performed for i) evaluating existing tone-mapping operators on 3D HDR images and ii) investigating how the preferred level of brightness and details differ between 3D and 2D images. The results are analyzed to find out the desired attributes.

By utilizing a Research through Design approach this master thesis studies how technological changes might affect professionals working in the motion picture industry, specifically; how the advent of HDR (High Dynamic Range) affects the colorist. The research questions formulated are the following; (1) How can color grading in HDR be approached? (2) What effect can HDR have on visual modality? (3) What specific affordances can HDR offer the colorist? (4) How can HDR affect the creative space of the colorist? Three of the research questions are derived from the theoretical framework applied in this master thesis; starting with the social semiotic implementation of the term modality (models of reality), the Gibsonian term affordance (possibilities for action and meaning making) and its use in communications research, and lastly; the concept of creative space in motion picture production. Analytic autoethnography was used to generate primary data by documenting the process of color grading a 13-minute short film, and also performing semistructured interviews with four colorists. Amongst other findings, this study found that HDR offers a wider range of modality expression than SDR (Standard Dynamic Range); regarding several visual modality markers. Four HDR-specific affordances were formulated; (1) color expandability, (2) highlight differentiability, (3) tonal rangeability, (4) brightness disturbability. Relating to the concept of creative space; the colorists expressed a concern that they will have to create multiple versions when delivering HDR, but not get a bigger budget for it, therefore having less time to spend on other aspects of color grading.

In this thesis we focus on two different image processing challenges: high dynamic range (HDR) image/video generation, and colour matching. In both cases, we redefine these tasks by taking into account the prior knowledge from the different processes the camera performs when capturing the image. High dynamic range techniques have recently become very popular, thanks to the emergence of HDR recording and displaying technology. We propose two different approaches for HDR creation, one for still images, and one for video creation. In case of still images, most of previous methods follow a multiple exposure combination approach. These methods share a set of building assump-tions: i) the full dynamic range of the scene can be recovered, ii) colour channels are independent, and iii) the camera response function remains constant while varying the exposure time. First, we highlight how these assumptions do not apply in general for digital cameras, and then, we propose a method to im-prove multiple exposure combination. Our results outperform state-of-the-art multiple exposure techniques.In case of HDR video, we present a simple and affordable method to gen-erate high quality video from an HDR scene. Our input is an interlaced video alternating row pairs with different ISO values, as some DSLR camera models can provide. The proposed algorithm involves two main steps: i) the compu-tation of two single-ISO full-frame images (one for each ISO value) using an inpainting-based deinterlacing method, and ii) the linear combination of the full-frame ISO’s into a single HDR frame. Finally, the results are tone-mapped into an LDR frame ready for display. Results are free of ghosting artefacts and present little noise.Colour matching approaches attempt to transfer the colours of a reference image, to another source image. In this context, we focus on colour matching two images taken from the same scene. First, we propose a method that mod-ifies logarithmic encoded images, used in cinema for HDR content, in order to behave as gamma encoded images, used in most digital cameras. Then, we ex-tend a previous approach defined just for gamma encoded images. We redefine the transformation between the two images, by considering a projective trans-form, and we estimate the method parameters in a single optimisation step. The method outperforms the state-of-the-art methods and can handle real-life examples.
En aquesta tesi ens centrem en dos problemes de processament d’imatges diferents: generació d’imatge/vídeo d’alt rang dinàmic (HDR) i coloració. En tots dos casos, redefinim aquestes tasques tenint en compte el coneixement previ dels diferents processos que realitza la càmera en capturar la imatge. Actualment, les tèniques d’alt rang dinàmic s’han tornat molt populars, gràcies a l’aparició de la tecnologia per capturar i visualitzar HDR. Proposem dos enfocaments diferents per a la creació d’HDR, un per a imatges i un altre per a la creació de video. En el cas d’imatges, la majoria de mètodes combinen múltiples exposicions. Aquests mètodes comparteixen un conjunt d’hipòtesis: i) la recuperació del rang dinàmic complet de l’escena, ii) els canals de color són independents, i iii) la funció de resposta de la càmera es manté constant mentre es varia el temps d’exposició. En primer lloc, destaquem com aquestes suposicions no s’apliquen, en general, a les càmeres digitals, i després proposem un mètode per millorar aquesta tècnica. Els nostres resultats superen l’estat de l’art. En el cas de vídeo HDR, presentem un mètode senzill i assequible per generar vídeos d’alta qualitat d’una escena HDR. El nostre input és un vídeo entrellaçat alternant parells de fileres amb diferents valors d’ISO, com alguns models de càmeres DSLR poden proporcionar. L’algorisme inclou dos passos principals: i) el càlcul de dues imatges full-frame ISO (una per a cada valor d’ISO) utilizant un mètode de desentrella¸cat basat en inpainting, ii) la combinació lineal dels ISOs full-frame en un HDR únic. Finalment, els resultats es mapegen tonalment per obtenir un LDR per mostrar per pantalla. Els resultats no tenen artefactes de ghosting i presenten poc soroll. Els mètodes d’igualació de colors intenten transferir els colors d’una imatge de referència, a una altra imatge d’origen. En aquest context, ens centrem en el cas de dues imatges capturades a la mateixa escena. En primer lloc, proposem un mètode que modifica imatges codificades logarítmicament, utilitzades en el cinema per a continguts HDR, per tal de comportar-se com imatges gamma codificades, que s’utilitzen en la majoria de les càmeres digitals. A continuació, extenem un mètode definit prèviament només per imatges gamma codificades, redefinint la transformació entre les dues imatges, considerant una transformació projectiva i estimant els paràmetres del mètode en un únic pas d’optimització. El mètode supera l’estat de l’art i pot tractar exemples de la vida real.

This thesis presents a web-based approach to lighting three-dimensional geometry in a virtual scene. The use of High Dynamic Range (HDR) images for the lighting model makes it possible to convey a greater sense of photorealism than can be provided with a conventional computer generated three-point lighting setup. The use of QuickTime ™ Object Virtual Reality to display the three-dimensional geometry offers a sophisticated user experience and a convenient method for viewing virtual objects over the web. With this work, I generate original High Dynamic Range images for the purpose of image-based lighting and use the QuickTime ™ Object Virtual Reality framework to creatively alter the paradigm of object VR for use in object lighting. The result is two scenarios: one that allows for the virtual manipulation of an object within a lit scene, and another with the virtual manipulation of light around a static object. Future work might include the animation of High Dynamic Range image-based lighting, with emphasis on such features as depth of field and glare generation.

Today, infrared cameras are used especially for target tracking and surveillance operations. However, they have a high dynamic range output, and the standard display devices cannot handle them. In order to show them on common devices, the dynamic range is cropped. Thus, the contrast of the image is reduced. This is called as the High Dynamic Range (HDR) Compression. Although several algorithms have been proposed for preserving details during the HDR compression process, it cannot be used to enhance the local contrasts of image contents. In this thesis, we compare the performances of contrast enhancement techniques, which are suitable for real time applications. The methods experimented are generally histogram based methods. Some modifications are also proposed in order to reduce computational complexity of the process. Performances of these methods are compared with common objective quality metrics on different image sets.

Photo editing software allows digital images to be blurred, warped or re-colored at the touch of a button. However, it is not currently possible to change the material appearance of an object except by painstakingly painting over the appropriate pixels. Here we present a set of methods for automatically replacing one material with another, completely different material, starting with only a single high dynamic range image, and an alpha matte specifying the object. Our approach exploits the fact that human vision is surprisingly tolerant of certain (sometimes enormous) physical inaccuracies. Thus, it may be possible to produce a visually compelling illusion of material transformations, without fully reconstructing the lighting or geometry. We employ a range of algorithms depending on the target material. First, an approximate depth map is derived from the image intensities using bilateral filters. The resulting surface normals are then used to map data onto the surface of the object to specify its material appearance. To create transparent or translucent materials, the mapped data are derived from the object's background. To create textured materials, the mapped data are a texture map. The surface normals can also be used to apply arbitrary bidirectional reflectance distribution functions to the surface, allowing us to simulate a wide range of materials. To facilitate the process of material editing, we generate the HDR image with a novel algorithm, that is robust against noise in individual exposures. This ensures that any noise, which would possibly have affected the shape recovery of the objects adversely, will be removed. We also present an algorithm to automatically generate alpha mattes. This algorithm requires as input two images--one where the object is in focus, and one where the background is in focus--and then automatically produces an approximate matte, indicating which pixels belong to the object. The result is then improved by a second algorithm to generate an accurate alpha matte, which can be given as input to our material editing techniques.
Ph.D.
School of Electrical Engineering and Computer Science
Engineering and Computer Science
Computer Science

Dans cette thèse, nous discutons de quatre scénarios d’application différents qui peuvent être largement regroupés dans le cadre plus large de l’analyse et du traitement d’images à haute résolution à l’aide de techniques d’apprentissage approfondi. Les trois premiers chapitres portent sur le traitement des images de télédétection (RS) captées soit par avion, soit par satellite à des centaines de kilomètres de la Terre. Nous commençons par aborder un problème difficile lié à l’amélioration de la classification des scènes aériennes complexes par le biais d’un paradigme d’apprentissage profondément faiblement supervisé. Nous montrons comment en n’utilisant que les étiquettes de niveau d’image, nous pouvons localiser efficacement les régions les plus distinctives dans les scènes complexes et éliminer ainsi les ambiguïtés qui mènent à une meilleure performance de classification dans les scènes aériennes très complexes. Dans le deuxième chapitre, nous traiterons de l’affinement des étiquettes de segmentation des empreintes de pas des bâtiments dans les images aériennes. Pour ce faire, nous détectons d’abord les erreurs dans les masques de segmentation initiaux et corrigeons uniquement les pixels de segmentation où nous trouvons une forte probabilité d’erreurs. Les deux prochains chapitres de la thèse portent sur l’application des Réseaux Adversariatifs Génératifs. Dans le premier, nous construisons un modèle GAN nuageux efficace pour éliminer les couches minces de nuages dans l’imagerie Sentinel-2 en adoptant une perte de consistance cyclique. Ceci utilise une fonction de perte antagoniste pour mapper des images nuageuses avec des images non nuageuses d’une manière totalement non supervisée, où la perte cyclique aide à contraindre le réseau à produire une image sans nuage correspondant a` l’image nuageuse d’entrée et non à aucune image aléatoire dans le domaine cible. Enfin, le dernier chapitre traite d’un ensemble différent d’images `à haute résolution, ne provenant pas du domaine RS mais plutôt de l’application d’imagerie à gamme dynamique élevée (HDRI). Ce sont des images 32 bits qui capturent toute l’étendue de la luminance présente dans la scène. Notre objectif est de les quantifier en images LDR (Low Dynamic Range) de 8 bits afin qu’elles puissent être projetées efficacement sur nos écrans d’affichage normaux tout en conservant un contraste global et une qualité de perception similaires à ceux des images HDR. Nous adoptons un modèle GAN multi-échelle qui met l’accent à la fois sur les informations plus grossières et plus fines nécessaires aux images à haute résolution. Les sorties finales cartographiées par ton ont une haute qualité subjective sans artefacts perçus
In this thesis, we discuss four different application scenarios that can be broadly grouped under the larger umbrella of Analyzing and Processing high-resolution images using deep learning techniques. The first three chapters encompass processing remote-sensing (RS) images which are captured either from airplanes or satellites from hundreds of kilometers away from the Earth. We start by addressing a challenging problem related to improving the classification of complex aerial scenes through a deep weakly supervised learning paradigm. We showcase as to how by only using the image level labels we can effectively localize the most distinctive regions in complex scenes and thus remove ambiguities leading to enhanced classification performance in highly complex aerial scenes. In the second chapter, we deal with refining segmentation labels of Building footprints in aerial images. This we effectively perform by first detecting errors in the initial segmentation masks and correcting only those segmentation pixels where we find a high probability of errors. The next two chapters of the thesis are related to the application of Generative Adversarial Networks. In the first one, we build an effective Cloud-GAN model to remove thin films of clouds in Sentinel-2 imagery by adopting a cyclic consistency loss. This utilizes an adversarial lossfunction to map cloudy-images to non-cloudy images in a fully unsupervised fashion, where the cyclic-loss helps in constraining the network to output a cloud-free image corresponding to the input cloudy image and not any random image in the target domain. Finally, the last chapter addresses a different set of high-resolution images, not coming from the RS domain but instead from High Dynamic Range Imaging (HDRI) application. These are 32-bit imageswhich capture the full extent of luminance present in the scene. Our goal is to quantize them to 8-bit Low Dynamic Range (LDR) images so that they can be projected effectively on our normal display screens while keeping the overall contrast and perception quality similar to that found in HDR images. We adopt a Multi-scale GAN model that focuses on both coarser as well as finer-level information necessary for high-resolution images. The final tone-mapped outputs have a high subjective quality without any perceived artifacts

Visualization is the process of constructing methods, which are able to synthesize interesting and informative images from data sets, to simplify the process of interpreting the data. In this thesis a new approach to construct meteorological visualization methods using neural network technology is described. The methods are trained with examples instead of explicitely designing the appearance of the visualization.

This approach is exemplified using two applications. In the fist the problem to compute an image of the sky for dynamic weather, that is taking account of the current weather state, is addressed. It is a complicated problem to tie the appearance of the sky to a weather state. The method is trained with weather data sets and images of the sky to be able to synthesize a sky image for arbitrary weather conditions. The method has been trained with various kinds of weather and images data. The results show that this is a possible method to construct weather visaualizations, but more work remains in characterizing the weather state and further refinement is required before the full potential of the method can be explored. This approach would make it possible to synthesize sky images of dynamic weather using a fast and efficient empirical method.

In the second application the problem of computing synthetic satellite images form numerical forecast data sets is addressed. In this case a mode is trained with preclassified satellite images and forecast data sets to be able to synthesize a satellite image representing arbitrary conditions. The resulting method makes it possible to visualize data sets from numerical weather simulations using synthetic satellite images, but could also be the basis for algorithms based on a preliminary cloud classification.

Report code: LiU-Tek-Lic-2004:66.

The aim of the work was to put together the list of possible ways to use high resolution image in real estate practice. Photographies are created by stitching more than one image on itself. Further on was developed methodology of working with these images and was made an illustration with detailed description of the procedure for each individual case. For each procedure were proposed possibilities in practical using. For the thesis purposes was conducted real estate and public survey. Based on results of the survey were purposed draft measures to improve internet real estate advertising.

Les contributions de cette thèse sont divisées en trois parties principales. Dans la partie 1, nous proposons une méthode locale utilisant une distribution GGM pour approcher les distributions des images en les subdivisant en groupe de pixels que nous appelons dorénavant clusters. L'idée principale consiste à déterminer quelle caractéristique (couleur, luminance) est plus représentative pour une image donnée. Puis nous utilisons cette caractéristique pour subdiviser l'image en clusters. Quatre stratégies de mise en correspondance des clusters de l'image d'entrée avec ceux de l'image cible sont proposées. Ces stratégies ont pour but de produire des images photoréalistes dont le style ressemble à celui de l'image cible (dans notre cas le style d'une image est défini en termes de couleur et luminosité). Nous étendons le principe de transfert de couleur au transfert simultané de couleur et de gradient. Afin de pouvoir décrire las distributions de couleur et de gradient par une seule distribution, nous adoptons le modèle MGGD (multivariate generalized Gaussian distributions). Nous proposons une nouvelle transformation de distribution MGGD pour des applications de traitement d'image telles que le transfert multi-dimensionnel de caractéristiques d'image, de couleur, etc. De plus, nous adoptons aussi un modèle de distribution plus précis (distribution Beta bornée) pour représenter des distributions de couleur et de luminosité. Nous proposons une transformation de distribution Beta qui permet d'effectuer un transfert de couleur entre images et qui s'avère plus performante que celles basées sur les distributions Gaussiennes. Dans la partie 2, nous introduisons une nouvelle méthode permettant de créer des images HDR à partir d'une paire d'images, l'une prise avec flash et l'autre pas. Notre méthode consiste en l'utilisation d'une fonction de luminosité (brightness) simulant la fonction de réponse d'une caméra, et d'une nouvelle fonction d'adaptation de couleur (CAT), appelée CAT bi-locale (bi-local CAT), permettant de reproduire les détails de l'image flash. Cette approche évite toutes les limitations inhérentes aux méthodes classiques de création d'images HDR. Dans la partie 3, nous exploitons le potentiel de notre adaptation bi-locale CAT pour diverses applications d'édition d'image telles que la suppression de bruit (dé-bruitage), suppression de flou, transfert de texture, etc. Nous introduisons notre nouveau filtre guidé dans lequel nous incorporons l'adaptation bi-locale CAT dans la partie 3
This thesis addresses three main topics from the domain of image processing, i.e. color transfer, high-dynamic-range (HDR) imaging and guidance-based image filtering. The first part of this thesis is dedicated to color transfer between input and target images. We adopt cluster-based techniques and apply Gaussian mixture models to carry out a more precise color transfer. In addition, we propose four new mapping policies to robustly portray the target style in terms of two key features: color, and light. Furthermore, we exploit the properties of the multivariate generalized Gaussian distributions (MGGD). in order to transfer an ensemble of features between images simultaneously. The multi-feature transfer is carried out using our novel transformation of the MGGD. Despite the efficiency of the proposed MGGD transformation for multi-feature transfer, our experiments have shown that the bounded Beta distribution provides a much more precise model for the color and light distributions of images. To exploit this property of the Beta distribution, we propose a new color transfer method, where we model the color and light distributions by the Beta distribution and introduce a novel transformation of the Beta distribution. The second part of this thesis focuses on HDR imaging. We introduce a method for automatic creation of HDR images from only two images - flash and non-flash images. We mimic the camera response function by a brightness function and we recover details from the flash image using our new chromatic adaptation transform (CAT), called bi-local CAT. That way, we efficiently recover the dynamic range of the real-world scenes without compromising the quality of the HDR image (as our method is robust to misalignment). In the context of the HDR image creation, the bi-local CAT recovers details from the flash image, removes flash shadows and reflections. In the last part of this thesis, we exploit the potential of the bi-local CAT for various image editing applications such as image de-noising, image de-blurring, texture transfer, etc. We propose a novel guidance-based filter in which we embed the bi-local CAT. The proposed filter performs as good as (and for certain applications even better than) state-of-the art methods

This work will be a web application for image and video quality assessment, especially those with a high dynamic range (HDR). The metrics are used for the evaluation, the results of which will be used for scientific purposes. Broader public citizens will be acquainted with this issue. This app will have an assistant available for a simple understanding of the whole application. The application also tries to be autonomous and dynamic. For example, new metrics can be added from the administration environment and editing on the server side is not required. The administrator will be able to upload a metric, a local library, a startup script, and define the parameters. The parameters provide metrics necessary data from the processed image or video. Only installation of system applications and libraries will be required on the server side.

Fyzikálně přijatelné osvětlení v reálném čase je často dosaženo použitím aproximací. Současné metody často aproximují globální osvětlení v prostoru obrazu s využitím schopností moderních grafických karet. Dva techniky z této kategorie, screen-space ambient occlusion a screen-space directional occlusion jsou popsány detailněji v této práci. Screen-space directional occlusion je zobecněná verze screen-space ambient occlusion s podporou jednoho difúzního odrazu a závislostí na směrové informaci světla. Hlavním cílem projektu bylo experimentování s těmito metodami. Pro uniformní distribuci náhodných vzorek pro obě metody byla použita Halton sekvence. Pro potlačení šumu je použita bilaterální filtrace, která bere do úvahy geometrické vlastnosti scény. Metody jsou dál zrychleny použitím nižších rozlišení pro výpočet. Rekonstrukce výsledků do původní velikosti pro vytvoření konečného obrazu je realizována pomoci joint bilateral upsamplingu. Kromě metod globálního osvětlení byly v práci použity aj metody pro mapování stínů a HDR osvětlení.

Les tècniques d'imatge d'alt rang dinàmic (HDR) potencialment permeten la captura i l'emmagatzematge de tota la informació de llum en una escena. No obstant això, els dispositius comuns de visualització són limitats en termes de les seves capacitats de contrast i brillantor, per tant, les imatges HDR han de ser mapejades tonalment abans de presentar-les en un dispositiu de visualització per assegurar que es reprodueix l'aspecte original de l'escena. En aquesta tesi, es prenen dos enfocaments del problema de mapeig tonal. En primer lloc, es desenvolupa un marc general per a la millora de qualsevol imatge mapejada tonalment mitjançant la reducció de la distància a la corresponent imatge HDR en termes d'una mètrica perceptiva no local. La distància es redueix al mínim per mitjà d'un algoritme de descens de gradient. En segon lloc, es desenvolupa un operador de mapeig tonal (TMO) en temps real que s'adapta bé a les estadístiques d'escenes naturals, i concorda amb els nous descobriments psicofísics i dades neurofísiques. Determinem les correctes adaptacions no lineals necessàries per als nostres resultats de mapeig tonal per tal d'obtenir l'aparença òptima en diferents condicions de visualització, a través d'experiments psicofísics i desenvolupar un mètode automàtic per poder predir dades experimentals. El nostre TMO produeix resultats d'aspecte natural, sense cap tipus d'artefactes espacials o temporals. Els tests de preferència dels usuaris mostren que el nostre mètode obté millors resultats en comparació amb les tècniques més recents. El TMO és ràpid i podria ser implementat en el hardware de la càmera. Pot ser utilitzat per al monitoratge de càmeres HDR en pantalles regulars, com a substitut de la correcció gamma, i com una manera de proporcionar al colorista amb contingut que té alhora un aspecte natural i una aparença nítida i clara.
High Dynamic Range (HDR) imaging techniques potentially allow for the capture and storage of the full information of light in a scene. However, common display devices are limited in terms of their contrast and brightness capabilities, thus HDR images must be tone mapped before presentation on a display device to ensure that the original appearance of the scene is reproduced. In this thesis, we take two approaches to the tone mapping problem. First, we develop a general framework for improving any tone mapped image by reducing the distance with the corresponding HDR image in terms of a non-local perceptual metric. The distance is minimized by means of a gradient descent algorithm. Second, we develop a real-time Tone Mapping Operator (TMO) that is well suited to the statistics of natural scenes, and is in keeping with new psychophysical findings and neurophysical data. We determine the adequate non-linear adjustments needed for our tone mapping results to look best in different viewing conditions through a psychophysical experiment and develop an automatic method that can predict the experimental data. Our TMO produces results that look natural, without any spatio-temporal artifacts. User preference tests show that our method outperforms state of the art approaches. The TMO is fast and could be implemented on camera hardware. It can be used for on-set monitoring of HDR cameras on regular displays, as a substitute for gamma correction, and as a way of providing the colorist with content that is both natural looking and has a crisp and clear appearance.

The work of this thesis is motivated by the bene t of real-time HDR image processing which helps constructing better seeing aid devices for day-to-day uses. The seeing aids with HDR can achieve dynamic range greater than human eyes to capture visuals under various light conditions. Further, it helps range sensing devices to read the light codes better. This requires realtime HDR image processing running at 24 FPS. In this thesis, we demonstrate implementation of several HDR image compositing and tone mapping methods in CUDA to achieve realtime performance. In addition, a new tone mapping method termed PPEM is introduced as a novel tone mapping method by adjusting per pixel exposure. Finally, we apply relevant HDR methods to range sensing devices for robust reading.