Tesis sobre el tema "Face spoofing"

Computer vision is an image understanding discipline that studies how to reconstruct, interpret and understand a 3D scene from its 2D images. One of the goals is to automate the analysis of images through the use of computer software and hardware. Meanwhile, biometrics refer to the automated authentication process that rely on measureable physical characteristics such as individual’s unique fingerprints, iris, face, palmprint, gait and voice. Amongst these biometric identification schemes, face biometric is said to be the most popular where face authentication systems have been rapidly developed mainly for security reasons. However, the resistance of face biometric system to spoofing attack, which is an act to impersonate a valid user by placing fake face in front of the sensor to gain access, has become a critical issue. Thus, anti-spoofing technique is required to counter the attacks. Different materials have their own reflection properties. These reflection differences have been manipulated by researches for particular reasons such as in object classification. Many ways can be used to measure the reflection differences of each object. One of them is by using polarised light. Since none of the existing studies applied polarised light in face spoofing detection, therefore in this thesis, polarisation imaging technique was implemented to distinguish between genuine face and two types of spoofing attacks: printed photos and iPad displayed faces. From the investigations, several research findings can be listed. Firstly, unpolarised visible light could not be used in a polarisation imaging system to capture polarised images for designated purpose. Secondly, polarised light is able to differentiate between surface and subsurface reflections of real and fake faces. However, both of these reflections could not be used as one of the classification methods between real face and printed photos. Thirdly, polarised image could contribute to enhance the performance of face recognition system against spoofing attacks in which the newly proposed formula, SDOLP3F achieves higher accuracy rate. Next, near infrared (NIR) light in a polarisation imaging system do not provide significant differences between real face and the two face attacks. Apart from polarised spoofing face detection analysis, experiments to investigate the accuracy of depth data captured by three depth sensors was carried out. This investigation was conducted due to the concerns over the stability of the depth pixels involved in 3D spoofing face reconstruction in a publicly available spoofing face database known as 3DMAD. From the analysis, none of the three depth sensors which are the Kinect for Xbox 360, Kinect for Windows version 2.0 and Asus Xtion Pro Live are suitable for 3D face reconstruction for the purpose of spoofing detection due to the potential errors made by the fluctuated pixels. As a conclusion, polarisation imaging technique has the potential to protect face biometric system from printed photos and iPad displayed attacks. Further investigations using the same polarised light approach could be carried out on other future work as proposed at the end of this thesis.

Les systèmes d’identification faciale sont en plein essor et se retrouvent de plus en plus dans des produits grand public tels que les smartphones et les ordinateurs portables. Cependant, ces systèmes peuvent être facilement bernés par la présentation par exemple d’une photo imprimée de la personne ayant les droits d’accès au système. Cette thèse s’inscrit dans le cadre du projet ANR BIOFENCE qui vise à développer une certification des systèmes biométriques veine, iris et visage permettant aux industriels de faire valoir leurs innovations en termes de protection. L’objectif de cette thèse est double, d’abord il s’agit de développer des mesures de protection des systèmes 2D d’identification faciale vis à vis des attaques connues à ce jour (photos imprimées, photos ou vidéos sur un écran, masques) puis de les confronter à la méthodologie de certification développée au sein du projet ANR. Dans un premier temps, un état de l’art général des attaques et des contremesures est présenté en mettant en avant les méthodes algorithmiques (« software ») par rapport aux méthodes hardware. Ensuite, plusieurs axes sont approfondis au cours de ce travail. Le premier concerne le développement d’une contremesure basée sur une analyse de texture et le second concerne le développement d’une contre-mesure basée sur une analyse de mouvement. Ensuite, une modélisation du processus de recapture pour différencier un faux visage d’un vrai est proposée. Une nouvelle méthode de protection est développée sur ce concept en utilisant les données d'enrolment des utilisateurs et un premier pas est franchi dans la synthèse d'attaque pour un nouvel utilisateur à partir de sa donnée d'enrolment. Enfin, la méthodologie de certification développée pour les systèmes à empreintes digitales est évaluée pour les systèmes d'identification facial
Face identification systems are growing rapidly and invade the consumer market with security products in smartphones, computers and banking. However, these systems are easily fooled by presenting a picture of the person having legitimate access to the system. This thesis is part of the BIOFENCE project which aim to develop a certification of biometric systems in order for industrials to promote their innovations in terms of protection. Our goal is to develop new anti-spoofing countermeasures for 2D face biometric systems and to evaluate the certification methodology on protected systems. First, a general state of the art in face spoofing attack forgery and in anti-spoofing protection measures is presented. Then texture-based countermeasures and motion-based countermeasures are investigated leading to the development of two novel countermeasures. Then, the recapturing process is modelled and a new fake face detection approach is proposed based on this model. Taking advantage of enrolment samples from valid users, a first step toward the synthesis of spoofing attacks for new users is taken. Finally, the certification methodology originally developed for fingerprint technology is evaluated on face biometric systems

Orientador: José Mario De Martino
Dissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Elétrica e de Computação
Made available in DSpace on 2018-08-23T20:06:41Z (GMT). No. of bitstreams: 1 Pereira_TiagodeFreitas_M.pdf: 17638731 bytes, checksum: 15a8d07214e3b31accd3218e5bde20cb (MD5) Previous issue date: 2013
Resumo: O Resumo poderá ser visualizado no texto completo da tese digital
Abstract: The complete Abstract is available with the full electronic document.
Mestrado
Engenharia de Computação
Mestre em Engenharia Elétrica

Abstract Because of its natural and non-intrusive interaction, identity verification and recognition using facial information is among the most active areas in computer vision research. Unfortunately, it has been shown that conventional 2D face recognition techniques are vulnerable to spoofing attacks, where a person tries to masquerade as another one by falsifying biometric data and thereby gaining an illegitimate advantage. This thesis explores different directions for software-based face anti-spoofing. The proposed approaches are divided into two categories: first, low-level feature descriptors are applied for describing the static and dynamic characteristic differences between genuine faces and fake ones in general, and second, complementary attack-specific countermeasures are investigated in order to overcome the limitations of generic spoof detection schemes. The static face representation is based on a set of well-known feature descriptors, including local binary patterns, Gabor wavelet features and histogram of oriented gradients. The key idea is to capture the differences in quality, light reflection and shading by analysing the texture and gradient structure of the input face images. The approach is then extended to the spatiotemporal domain when both facial appearance and dynamics are exploited for spoof detection using local binary patterns from three orthogonal planes. It is reasonable to assume that no generic spoof detection scheme is able to detect all known, let alone unseen, attacks scenarios. In order to find out well-generalizing countermeasures, the problem of anti-spoofing is broken into two attack-specific sub-problems based on whether the spoofing medium can be detected in the provided view or not. The spoofing medium detection is performed by describing the discontinuities in the gradient structures around the detected face. If the display medium is concealed outside the view, a combination of face and background motion correlation measurement and texture analysis is applied. Furthermore, an open-source anti-spoofing fusion framework is introduced and its system-level performance is investigated more closely in order to gain insight on how to combine different anti-spoofing modules. The proposed spoof detection schemes are evaluated on the latest benchmark datasets. The main findings of the experiments are discussed in the thesis
Tiivistelmä Kasvokuvaan perustuvan henkilöllisyyden tunnistamisen etuja ovat luonnollinen vuorovaikutus ja etätunnistus, minkä takia aihe on ollut erittäin aktiivinen tutkimusalue konenäön tutkimuksessa. Valitettavasti tavanomaiset kasvontunnistustekniikat ovat osoittautuneet haavoittuvaisiksi hyökkäyksille, joissa kameralle esitetään jäljennös kohdehenkilön kasvoista positiivisen tunnistuksen toivossa. Tässä väitöskirjassa tutkitaan erilaisia ohjelmistopohjaisia ratkaisuja keinotekoisten kasvojen ilmaisuun petkuttamisen estämiseksi. Työn ensimmäisessä osassa käytetään erilaisia matalan tason piirteitä kuvaamaan aitojen ja keinotekoisten kasvojen luontaisia staattisia ja dynaamisia eroavaisuuksia. Työn toisessa osassa esitetään toisiaan täydentäviä hyökkäystyyppikohtaisia vastakeinoja, jotta yleispätevien menetelmien puutteet voitaisiin ratkaista ongelmaa rajaamalla. Kasvojen staattisten ominaisuuksien esitys perustuu yleisesti tunnettuihin matalan tason piirteisiin, kuten paikallisiin binäärikuvioihin, Gabor-tekstuureihin ja suunnattujen gradienttien histogrammeihin. Pääajatuksena on kuvata aitojen ja keinotekoisten kasvojen laadun, heijastumisen ja varjostumisen eroavaisuuksia tekstuuria ja gradienttirakenteita analysoimalla. Lähestymistapaa laajennetaan myös tila-aika-avaruuteen, jolloin hyödynnetään samanaikaisesti sekä kasvojen ulkonäköä ja dynamiikkaa irroittamalla paikallisia binäärikuvioita tila-aika-avaruuden kolmelta ortogonaaliselta tasolta. Voidaan olettaa, ettei ole olemassa yksittäistä yleispätevää vastakeinoa, joka kykenee ilmaisemaan jokaisen tunnetun hyökkäystyypin, saati tuntemattoman. Näin ollen työssä keskitytään tarkemmin kahteen hyökkäystilanteeseen. Ensimmäisessä tapauksessa huijausapuvälineen reunoja ilmaistaan analysoimalla gradienttirakenteiden epäjatkuvuuksia havaittujen kasvojen ympäristössä. Jos apuvälineen reunat on piilotettu kameran näkymän ulkopuolelle, petkuttamisen ilmaisu toteutetaan yhdistämällä kasvojen ja taustan liikkeen korrelaation mittausta ja kasvojen tekstuurianalyysiä. Lisäksi työssä esitellään vastakeinojen yhdistämiseen avoimen lähdekoodin ohjelmisto, jonka avulla tutkitaan lähemmin menetelmien fuusion vaikutuksia. Tutkimuksessa esitetyt menetelmät on kokeellisesti vahvistettu alan viimeisimmillä julkisesti saatavilla olevilla tietokannoilla. Tässä väitöskirjassa käydään läpi kokeiden päähavainnot

Orientador: Anderson de Rezende Rocha
Dissertação (mestrado) - Universidade Estadual de Campinas, Instituto de Computação
Made available in DSpace on 2018-08-23T22:22:57Z (GMT). No. of bitstreams: 1 Pinto_AllandaSilva_M.pdf: 47523880 bytes, checksum: 072eb0490c26631b80cdcc47d55a4817 (MD5) Previous issue date: 2013
Resumo: O resumo poderá ser visualizado no texto completo da tese digital
Abstract: The complete abstract is available with the full electronic document
Mestrado
Ciência da Computação
Mestre em Ciência da Computação

Abstract In the last decades, face recognition systems have evolved a lot in terms of performance. As a result, this technology is now considered as mature and is applied in many real world applications from border control to financial transactions and computer security. Yet, many studies show that these systems suffer from vulnerabilities to spoofing attacks, a weakness that may limit their usage in many cases. A face spoofing attack or presentation attack occurs when someone tries to masquerade as someone else by presenting a fake face in front of the face recognition camera. To protect the recognition systems against attacks of this kind, many face anti-spoofing methods have been proposed. These methods have shown good performances on the existing face anti-spoofing databases. However, their performances degrade drastically under real world variations (e.g., illumination and camera device variations). In this thesis, we concentrate on improving the generalization capabilities of the face anti-spoofing methods with a particular focus on the texture based techniques. In contrast to most existing texture based methods aiming at extracting texture features from gray-scale images, we propose a joint color-texture analysis. First, the face images are converted into different color spaces. Then, the feature histograms computed over each image band are concatenated and used for discriminating between real and fake face images. Our experiments conducted on three color spaces: RGB, HSV and YCbCr show that extracting the texture information from separated luminance chrominance color spaces (HSV and YCbCr) yields to better performances compared to gray-scale and RGB image representations. Moreover, to deal with the problem of illumination and image-resolution variations, we propose to extract this texture information from different scale images. In addition to representing the face images in different scales, the multi-scale filtering methods also act as pre-processing against factors such as noise and illumination. Although our obtained results are better than the state of the art, they are still far from the requirements of real world applications. Thus, to help in the development of robust face anti-spoofing methods, we collected a new challenging face anti-spoofing database using six camera devices in three different illumination and environmental conditions. Furthermore, we have organized a competition on the collected database where fourteen face anti-spoofing methods have been assessed and compared
Tiivistelmä Kasvontunnistusjärjestelmien suorituskyky on parantunut huomattavasti viime vuosina. Tästä syystä tätä teknologiaa pidetään nykyisin riittävän kypsänä ja käytetään jo useissa käytännön sovelluksissa kuten rajatarkastuksissa, rahansiirroissa ja tietoturvasovelluksissa. Monissa tutkimuksissa on kuitenkin havaittu, että nämä järjestelmät ovat myös haavoittuvia huijausyrityksille, joissa joku yrittää esiintyä jonakin toisena henkilönä esittämällä kameralle jäljennöksen kohdehenkilön kasvoista. Tämä haavoittuvuus rajoittaa kasvontunnistuksen laajempaa käyttöä monissa sovelluksissa. Tunnistusjärjestelmien turvaamiseksi on kehitetty lukuisia menetelmiä tällaisten hyökkäysten torjumiseksi. Nämä menetelmät ovat toimineet hyvin tätä tarkoitusta varten kehitetyillä kasvotietokannoilla, mutta niiden suorituskyky huononee dramaattisesti todellisissa käytännön olosuhteissa, esim. valaistuksen ja käytetyn kuvantamistekniikan variaatioista johtuen. Tässä työssä yritämme parantaa kasvontunnistuksen huijauksen estomenetelmien yleistämiskykyä keskittyen erityisesti tekstuuripohjaisiin menetelmiin. Toisin kuin useimmat olemassa olevat tekstuuripohjaiset menetelmät, joissa tekstuuripiirteitä irrotetaan harmaasävykuvista, ehdotamme väritekstuurianalyysiin pohjautuvaa ratkaisua. Ensin kasvokuvat muutetaan erilaisiin väriavaruuksiin. Sen jälkeen kuvan jokaiselta kanavalta erikseen lasketut piirrehistogrammit yhdistetään ja käytetään erottamaan aidot ja väärät kasvokuvat toisistaan. Kolmeen eri väriavaruuteen, RGB, HSV ja YCbCr, perustuvat testimme osoittavat, että tekstuuri-informaation irrottaminen HSV- ja YCbCr-väriavaruuksien erillisistä luminanssi- ja krominanssikuvista parantaa suorituskykyä kuvien harmaasävy- ja RGB-esitystapoihin verrattuna. Valaistuksen ja kuvaresoluution variaation takia ehdotamme myös tämän tekstuuri-informaation irrottamista eri tavoin skaalatuista kuvista. Sen lisäksi, että itse kasvot esitetään eri skaaloissa, useaan skaalaan perustuvat suodatusmenetelmät toimivat myös esikäsittelynä sellaisia suorituskykyä heikentäviä tekijöitä vastaan kuten kohina ja valaistus. Vaikka tässä tutkimuksessa saavutetut tulokset ovat parempia kuin uusinta tekniikkaa edustavat tulokset, ne ovat kuitenkin vielä riittämättömiä reaalimaailman sovelluksissa tarvittavaan suorituskykyyn. Sen takia edistääksemme uusien robustien kasvontunnistuksen huijaamisen ilmaisumenetelmien kehittämistä kokosimme uuden, haasteellisen huijauksenestotietokannan käyttäen kuutta kameraa kolmessa erilaisessa valaistus- ja ympäristöolosuhteessa. Järjestimme keräämällämme tietokannalla myös kansainvälisen kilpailun, jossa arvioitiin ja verrattiin neljäätoista kasvontunnistuksen huijaamisen ilmaisumenetelmää

Visage est l’une des meilleures biométries pour la reconnaissance de l’identité de personnes, car l’identification d’une personne par le visage est l’habitude instinctive humaine, et l’acquisition de données faciales est naturelle, non intrusive et bien acceptée par le public. Contrairement à la reconnaissance de visage par l’image 2D sur l’apparence, la reconnaissance de visage en 3D sur la forme est théoriquement plus stable et plus robuste à la variance d’éclairage, aux petits changements de pose de la tête et aux cosmétiques pour le visage. Spécifiquement, les courbures sont les plus importants attributs géométriques pour décrire la forme géométrique d’une surface. Elles sont bénéfiques à la caractérisation de la forme du visage qui permet de diminuer l’impact des variances environnementales. Cependant, les courbures traditionnelles ne sont définies que sur des surfaces lisses. Il est donc nécessaire de généraliser telles notions sur des surfaces discrètes, par exemple des visages 3D représenté par maillage triangulaire, et d’évaluer leurs performances en reconnaissance de visage 3D. En outre, même si un certain nombre d’algorithmes 3D FR avec une grande précision sont disponibles, le coût d’acquisition de telles données de haute résolution est difficilement acceptable pour les applications pratiques. Une question majeure est donc d’exploiter les algorithmes existants pour la reconnaissance de modèles à faible résolution collecté avec l’aide d’un nombre croissant de caméras consommateur de profondeur (Kinect). Le dernier problème, mais non le moindre, est la menace sur sécurité des systèmes de reconnaissance de visage 3D par les attaques de masque fabriqué. Cette thèse est consacrée à l’étude des attributs géométriques, des mesures de courbure principale, adaptées aux maillages triangulaires, et des schémas de reconnaissance de visage 3D impliquant des telles mesures de courbure principale. En plus, nous proposons aussi un schéma de vérification sur la reconnaissance de visage 3D collecté en comparant des modèles de résolutions hétérogènes équipement aux deux résolutions, et nous évaluons la performance anti-spoofing du système de RF 3D. Finalement, nous proposons une biométrie système complémentaire de reconnaissance veineuse de main basé sur la détection de vivacité et évaluons sa performance. Dans la reconnaissance de visage 3D par la forme géométrique, nous introduisons la généralisation des courbures principales conventionnelles et des directions principales aux cas des surfaces discrètes à maillage triangulaire, et présentons les concepts des mesures de courbure principale correspondants et des vecteurs de courbure principale. Utilisant ces courbures généralisées, nous élaborons deux descriptions de visage 3D et deux schémas de reconnaissance correspondent. Avec le premier descripteur de caractéristiques, appelé Local Principal Curvature Measures Pattern (LPCMP), nous générons trois images spéciales, appelée curvature faces, correspondant à trois mesures de courbure principale et encodons les curvature faces suivant la méthode de Local Binary Pattern. Il peut décrire la surface faciale de façon exhaustive par l’information de forme locale en concaténant un ensemble d’histogrammes calculés à partir de petits patchs dans les visages de courbure. Dans le deuxième système de reconnaissance de visage 3D sans enregistrement, appelée Principal Curvature Measures based meshSIFT descriptor (PCM-meshSIFT), les mesures de courbure principales sont d’abord calculées dans l’espace de l’échelle Gaussienne, et les extrèmes de la Différence de Courbure (DoC) sont définis comme les points de caractéristique. Ensuite, nous utilisons trois mesures de courbure principales et leurs vecteurs de courbure principaux correspondants pour construire trois descripteurs locaux pour chaque point caractéristique, qui sont invariants en rotation. [...]
Face is one of the best biometrics for person recognition related application, because identifying a person by face is human instinctive habit, and facial data acquisition is natural, non-intrusive, and socially well accepted. In contrast to traditional appearance-based 2D face recognition, shape-based 3D face recognition is theoretically more stable and robust to illumination variance, small head pose changes, and facial cosmetics. The curvatures are the most important geometric attributes to describe the shape of a smooth surface. They are beneficial to facial shape characterization which makes it possible to decrease the impact of environmental variances. However, exiting curvature measurements are only defined on smooth surface. It is required to generalize such notions to discrete meshed surface, e.g., 3D face scans, and to evaluate their performance in 3D face recognition. Furthermore, even though a number of 3D FR algorithms with high accuracy are available, they all require high-resolution 3D scans whose acquisition cost is too expensive to prevent them to be implemented in real-life applications. A major question is thus how to leverage the existing 3D FR algorithms and low-resolution 3D face scans which are readily available using an increasing number of depth-consumer cameras, e.g., Kinect. The last but not least problem is the security threat from spoofing attacks on 3D face recognition system. This thesis is dedicated to study the geometric attributes, principal curvature measures, suitable to triangle meshes, and the 3D face recognition schemes involving principal curvature measures. Meanwhile, based on these approaches, we propose a heterogeneous cross-resolution 3D FR scheme, evaluate the anti-spoofing performance of shape-analysis based 3D face recognition system, and design a supplementary hand-dorsa vein recognition system based on liveness detection with discriminative power. In 3D shape-based face recognition, we introduce the generalization of the conventional point-wise principal curvatures and principal directions for fitting triangle mesh case, and present the concepts of principal curvature measures and principal curvature vectors. Based on these generalized curvatures, we design two 3D face descriptions and recognition frameworks. With the first feature description, named as Local Principal Curvature Measures Pattern descriptor (LPCMP), we generate three curvature faces corresponding to three principal curvature measures, and encode the curvature faces following Local Binary Pattern method. It can comprehensively describe the local shape information of 3D facial surface by concatenating a set of histograms calculated from small patches in the encoded curvature faces. In the second registration-free feature description, named as Principal Curvature Measures based meshSIFT descriptor (PCM-meshSIFT), the principal curvature measures are firstly computed in the Gaussian scale space, and the extremum of Difference of Curvautre (DoC) is defined as keypoints. Then we employ three principal curvature measures and their corresponding principal curvature vectors to build three rotation-invariant local 3D shape descriptors for each keypoint, and adopt the sparse representation-based classifier for keypoint matching. The comprehensive experimental results based on FRGCv2 database and Bosphorus database demonstrate that our proposed 3D face recognition scheme are effective for face recognition and robust to poses and occlusions variations. Besides, the combination of the complementary shape-based information described by three principal curvature measures significantly improves the recognition ability of system. To deal with the problem towards heterogeneous cross-resolution 3D FR, we continuous to adopt the PCM-meshSIFT based feature descriptor to perform the related 3D face recognition. [...]

Abstract The face plays an important role in our social interactions as it conveys rich sources of information. We can read a lot from one face image, but there is also information we cannot perceive without special devices. The thesis concerns using computer vision methodologies to analyse two kinds of subtle facial information that can hardly be perceived by naked eyes: the micro-expression (ME), and the heart rate (HR). MEs are rapid, involuntary facial expressions which reveal emotions people do not intend to show. It is difficult for people to perceive MEs as they are too fast and subtle, thus automatic ME analysis is valuable work which may lead to important applications. In the thesis, the progresses of ME studies are reviewed, and four parts of work are described. 1) We introduce the first spontaneous ME database, the SMIC. The lacking of data is hindering ME analysis research, as it is difficult to collect spontaneous MEs. The protocol for inducing and annotating SMIC is introduced to help future ME collections. 2) A framework including three features and a video magnification process is introduced for ME recognition, which outperforms other state-of-the-art methods on two ME databases. 3) An ME spotting method based on feature difference analysis is described, which can spot MEs from spontaneous long videos. 4) An automatic ME analysis system (MESR) was proposed for firstly spotting and then recognising MEs. The HR is an important indicator of our health and emotional status. Traditional HR measurements require skin-contact which cannot be applied remotely. We propose a method which can counter for illumination changes and head motions and measure HR remotely from color facial videos. We also apply the method for solving the face anti-spoofing problem. We show that the pulse-based feature is more robust than traditional texture-based features against unseen mask spoofs. We also show that the proposed pulse-based feature can be combined with other features to build a cascade system for detecting multiple types of attacks. At last, we summarize the contributions of the work, and propose future plans about ME and HR studies based on limitations of the current work. It is also planned to combine the ME and HR (maybe also other subtle signals from face) to build a multimodal system for affective status analysis
Tiivistelmä Kasvot ovat monipuolinen informaatiolähde ja keskeinen ihmisten välisessä vuorovaikutuksessa. Pystymme päättelemään paljon yhdestäkin kasvokuvasta, mutta kasvoissa on paljon tietoa, jota ei pysty irrottamaan ilman erityiskeinoja. Tässä työssä analysoidaan konenäöllä ihmiselle vaikeasti havaittavaa tietoa: mikroilmeitä ja sydämen sykettä. Tahdosta riippumattomat mikroilmeet paljastavat tunteita, joita ihmiset pyrkivät piilottamaan. Mikroilmeiden havaitseminen on vaikeaa niiden nopeuden ja pienuuden vuoksi, joten automaattinen analyysi voi johtaa uusiin merkittäviin sovelluksiin. Tämä työ tarkastelee mikroilmetutkimuksen edistysaskeleita ja sisältää neljä uutta tulosta. 1) Spontaanien mikroilmeiden tietokanta (Spontaneous MIcroexpression Corpus, SMIC). Spontaanien mikroilmeiden aiheuttaminen datan saamiseksi on oma haasteensa. SMIC:n keräämisessä ja mikroilmeiden annotoinnissa käytetty menettely on kuvattu myöhemmän datan keruun ohjeistukseksi. 2) Aiempia mikroilmeiden tunnistusmenetelmiä paremmaksi kahden testitietokannan avulla todennettu ratkaisu, joka käyttää kolmea eri piirrettä ja videon suurennusta. 3) Piirre-eroanalyysiin perustuva mikroilmeiden havaitsemismenetelmä, joka havaitsee ne pitkistä realistisista videoista. 4) Automaattinen analyysijärjestelmä (Micro-Expression Spotting and Recognition, MESR), jossa mikroilmeet havaitaan ja tunnistetaan. Sydämen syke on tärkeä terveyden ja tunteiden indikoija. Perinteiset sykkeenmittausmenetelmät vaativat ihokontaktia, eivätkä siten toimii etäältä. Tässä työssä esitetään sykkeen videolta pienistä värimuutoksista mittaava menetelmä, joka sietää valaistusmuutoksia ja sallii pään liikkeet. Menetelmä on monikäyttöinen ja sen sovelluksena kuvataan todellisten kasvojen varmentaminen sykemittauksella. Tulokset osoittavat sykepiirteiden toimivan perinteisiä tekstuuripiirteitä paremmin uudenlaisia naamarihuijauksia vastaan. Syketietoa voidaan myös käyttää osana sarjatyyppisissä ratkaisuissa havaitsemaan useanlaisia huijausyrityksiä. Työn yhteenveto keskittyy suunnitelmiin parantaa mikroilmeiden ja sydämen sykkeen analyysimenetelmiä nykyisen tutkimuksen rajoitteiden pohjalta. Tavoitteena on yhdistää mikroilmeiden ja sydämen sykkeen analyysit, sekä mahdollisesti muuta kasvoista saatavaa tietoa, multimodaaliseksi affektiivisen tilan määrittäväksi ratkaisuksi

The dissertation investigates the promises and challenges for application of cardiac signals in biometrics and affective computing, and noninvasive measurement of cardiac signals. We have mainly discussed two major cardiac signals: electrocardiogram (ECG), and photoplethysmogram (PPG). ECG and PPG signals hold strong potential for biometric authentications and identifications. We have shown that by mapping each cardiac beat from time domain to an angular domain using a limit cycle, intra-class variability can be significantly minimized. This is in contrary to conventional time domain analysis. Our experiments with both ECG and PPG signal shows that the proposed method eliminates the effect of instantaneous heart rate on the shape morphology and improves authentication accuracy. For noninvasive measurement of PPG beats, we have developed a systematic algorithm to extract pulse rate from face video in diverse situations using video magnification. We have extracted signals from skin patches and then used frequency domain correlation to filter out non-cardiac signals. We have developed a novel entropy based method to automatically select skin patches from face. We report beat-to-beat accuracy of remote PPG (rPPG) in comparison to conventional average heart rate. The beat-to-beat accuracy is required for applications related to heart rate variability (HRV) and affective computing. The algorithm has been tested on two datasets, one with static illumination condition and the other with unrestricted ambient illumination condition. Automatic skin detection is an intermediate step for rPPG. Existing methods always depend on color information to detect human skin. We have developed a novel standalone skin detection method to show that it is not necessary to have color cues for skin detection. We have used LBP lacunarity based micro-textures features and a region growing algorithm to find skin pixels in an image. Our experiment shows that the proposed method is applicable universally to any image including near infra-red images. This finding helps to extend the domain of many application including rPPG. To the best of our knowledge, this is first such method that is independent of color cues.
Ph. D.

L'identité des personnes sur Internet devient un problème de sécurité majeur. Depuis les accords de Bale, les institutions bancaires ont intégré la vérification de l'identité des personnes ou Know Your Customer (KYC) dans leur processus d'inscription. Avec la dématérialisation des banques, cette procédure est devenue l'e-KYC ou KYC à distance qui fonctionne à distance via le smartphone de l'utilisateur. De même, la vérification d'identité à distance est devenue la norme pour l'inscription aux outils de signature électronique. De nouvelles réglementations émergent pour sécuriser cette approche, par exemple, en France, le cadre PVID encadre l'acquisition à distance des documents d'identité et du visage des personnes dans le cadre du règlement eIDAS. Cela est nécessaire, car on assiste à l'émergence d'un nouveau type de criminalité numérique : l'usurpation d'identité profonde. Grâce aux nouveaux outils d'apprentissage profond, les imposteurs peuvent modifier leur apparence pour ressembler à quelqu'un d'autre en temps réel. Les imposteurs peuvent alors accomplir toutes les actions courantes requises lors d'une inscription à distance sans être détectés par les algorithmes de vérification d'identité. Aujourd'hui, il existe des applications sur smartphone et des outils destinés à un public plus limité qui permettent aux imposteurs de transformer facilement leur apparence en temps réel. Il existe même des méthodes pour usurper une identité à partir d'une seule image du visage de la victime. L'objectif de cette thèse est d'étudier les vulnérabilités des systèmes d'authentification d'identité à distance face aux nouvelles attaques
The identity of people on the Internet is becoming a major security issue. Since the Bale agreements, banking institutions have integrated the verification of people's identity or Know Your Customer (KYC) in their registration process. With the dematerialization of banks, this procedure has become e-KYC or remote KYC which works remotely through the user's smartphone. Similarly, remote identity verification has become the standard for enrollment in electronic signature tools. New regulations are emerging to secure this approach, for example, in France, the PVID framework regulates the remote acquisition of identity documents and people's faces under the eIDAS regulation. This is required because a new type of digital crime is emerging: deep identity theft. With new deep learning tools, imposters can change their appearance to look like someone else in real time. Imposters can then perform all the common actions required in a remote registration without being detected by identity verification algorithms. Today, smartphone applications and tools for a more limited audience exist allowing imposters to easily transform their appearance in real time. There are even methods to spoof an identity based on a single image of the victim's face. The objective of this thesis is to study the vulnerabilities of remote identity authentication systems against new attacks in order to propose solutions based on deep learning to make the systems more robust

Les systèmes biométriques vocaux sont utilisés dans diverses applications pour une authentification sécurisée. Toutefois, ces systèmes sont vulnérables aux attaques par usurpation d'identité. Il est donc nécessaire de disposer de techniques de détection plus robustes. Cette thèse propose de nouvelles techniques de détection fiables et efficaces contre les attaques invisibles. La première contribution est un ensemble non linéaire de classificateurs de sous-bandes utilisant chacun un modèle de mélange gaussien. Des résultats compétitifs montrent que les modèles qui apprennent des indices discriminants spécifiques à la sous-bande peuvent être nettement plus performants que les modèles entraînés sur des signaux à bande complète. Étant donné que les DNN sont plus puissants et peuvent effectuer à la fois l'extraction de caractéristiques et la classification, la deuxième contribution est un modèle RawNet2. Il s'agit d'un modèle de bout en bout qui apprend les caractéristiques directement à partir de la forme d'onde brute. La troisième contribution comprend la première utilisation de réseaux neuronaux graphiques (GNN) avec un mécanisme d'attention pour modéliser la relation complexe entre les indices d'usurpation présents dans les domaines spectral et temporel. Nous proposons un réseau d'attention spectro-temporel E2E appelé RawGAT-ST. Il est ensuite étendu à un réseau d'attention spectro-temporel intégré, appelé AASIST, qui exploite la relation entre les graphes spectraux et temporels hétérogènes. Enfin, cette thèse propose une nouvelle technique d'augmentation des données appelée RawBoost et utilise un modèle vocal auto-supervisé et pré-entraîné pour améliorer la généralisation
Voice biometric systems are being used in various applications for secure user authentication using automatic speaker verification technology. However, these systems are vulnerable to spoofing attacks, which have become even more challenging with recent advances in artificial intelligence algorithms. There is hence a need for more robust, and efficient detection techniques. This thesis proposes novel detection algorithms which are designed to perform reliably in the face of the highest-quality attacks. The first contribution is a non-linear ensemble of sub-band classifiers each of which uses a Gaussian mixture model. Competitive results show that models which learn sub-band specific discriminative information can substantially outperform models trained on full-band signals. Given that deep neural networks are more powerful and can perform both feature extraction and classification, the second contribution is a RawNet2 model. It is an end-to-end (E2E) model which learns features directly from raw waveform. The third contribution includes the first use of graph neural networks (GNNs) with an attention mechanism to model the complex relationship between spoofing cues present in spectral and temporal domains. We propose an E2E spectro-temporal graph attention network called RawGAT-ST. RawGAT-ST model is further extended to an integrated spectro-temporal graph attention network, named AASIST which exploits the relationship between heterogeneous spectral and temporal graphs. Finally, this thesis proposes a novel data augmentation technique called RawBoost and uses a self-supervised, pre-trained speech model as a front-end to improve generalisation in the wild conditions

A biometric system is essentially a pattern recognition system being used in ad-versarial environment. Since, biometric system like any conventional security system is exposed to malicious adversaries, who can manipulate data to make the system ineffective by compromising its integrity. Current theory and de- sign methods of biometric systems do not take into account the vulnerability to such adversary attacks. Therefore, evaluation of classical design methods is an open problem to investigate whether they lead to design secure systems. In order to make biometric systems secure it is necessary to understand and evalu-ate the threats and to thus develop effective countermeasures and robust system designs, both technical and procedural, if necessary. Accordingly, the extension of theory and design methods of biometric systems is mandatory to safeguard the security and reliability of biometric systems in adversarial environments. In this thesis, we provide some contributions towards this direction. Among all the potential attacks discussed in the literature, spoof attacks are one of the main threats against the security of biometric systems for identity recognition. Multimodal biometric systems are commonly believed to be in-trinsically more robust to spoof attacks than systems based on a single biomet-ric trait, as they combine information coming from different biometric traits. However, recent works have question such belief and shown that multimodal systems can be misled by an attacker (impostor) even by spoofing only one of the biometric traits. Therefore, we first provide a detailed review of state-of-the-art works in multimodal biometric systems against spoof attacks. The scope ofstate-of-the-art results is very limited, since they were obtained under a very restrictive “worst-case” hypothesis, where the attacker is assumed to be able to fabricate a perfect replica of a biometric trait whose matching score distribu-tion is identical to the one of genuine traits. Thus, we argue and investigate the validity of “worst-case” hypothesis using large set of real spoof attacks and provide empirical evidence that “worst-case” scenario can not be representa- ixtive of real spoof attacks: its suitability may depend on the specific biometric trait, the matching algorithm, and the techniques used to counterfeit the spoofed traits. Then, we propose a security evaluation methodology of biometric systems against spoof attacks that can be used in real applications, as it does not require fabricating fake biometric traits, it allows the designer to take into account the different possible qualities of fake traits used by different attackers, and it exploits only information on genuine and impostor samples which is col- lected for the training of a biometric system. Our methodology evaluates the performances under a simulated spoof attack using model of the fake score distribution that takes into account explicitly different degrees of the quality of fake biometric traits. In particular, we propose two models of the match score distribution of fake traits that take into account all different factors which can affect the match score distribution of fake traits like the particular spoofed biometric, the sensor, the algorithm for matching score computation, the technique used to construct fake biometrics, and the skills of the attacker. All these factors are summarized in a single parameter, that we call “attack strength”. Further, we propose extension of our security evaluation method to rank several biometric score fusion rules according to their relative robustness against spoof attacks. This method allows the designer to choose the most robust rule according to the method prediction. We then present empirical analysis, using data sets of face and fingerprints including real spoofed traits, to show that our proposed models provide a good approximation of fake traits’ score distribution and our method thus providing an adequate estimation of the security1 of biometric systems against spoof attacks. We also use our method to show how to evaluate the security of different multimodal systems on publicly available benchmark data sets without spoof attacks. Our experimental results show that robustness of multimodal biometric systems to spoof attacks strongly depends on the particular matching algorithm, the score fusion rule, and the attack strength of fake traits. We eventually present evidence, considering a multimodal system based on face and fingerprint biometrics, that the proposed methodology to rank score fusion rules is capable of providing correct ranking of score fusion rules under spoof attacks.

碩士
國立臺灣大學
生醫電子與資訊學研究所
107
Our proposed method is capable of authenticating the input image is from real user or spoofing attack, including paper photograph, digital photograph, and video, using only the Red, Green, Blue (RGB) frontal camera of common smart phone, without the help of depth camera or infrared thermal sensor. We first capture live faces in each frame of input video streams by single shot multi-box detector then feed into our designed convolution neural network after certain data augmentation and finally obtain a well-trained spoof face classifier.

碩士
國立中央大學
資訊工程學系
107
In this thesis, we propose a face classification system based on deep learning algorithm. This system is capable of distinguishing real and fake faces from RGB images taken by a normal camera. To do that, we have built a system of 4 parts: RGB image processing, HSV image processing, YCrCb image processing, and classification. With the first 3 parts of image processing, the model will have different viewpoints of the object to be considered so that the classification can make the most accurate conclusion possible. In addition, in order to achieve optimal processing performance, we include encoder and decoder structure models, which eliminate unnecessary components and help the model focus only on the components it gives. is important, and most importantly, this structure helps reduce the complexity of the model. In the process of experimentation, we found some problems arising in the processing of data, namely that the research data does not match the actual data. In order to create a model for good results on actual research and operational data, we have applied a number of special tweaks to the data before being put into training. Experimental results indicate that our system gives a very high result on public databases.

碩士
元智大學
電機工程學系
106
There are many ways to do authentication, but most of the systems verification are still based on passwords. Passwords are very valuable to hackers, and there is endless news that involving hackers stealing passwords and obtaining user information for illegal purposes. In order to solve this problem, people gradually turn their attention to the biometric authentication system with high security. With the great evolution of deep convolutional neural networks in recent years, deep convolutional features with high robustness and adaptability has been utilized as features in the liveness detection mechanism. However, a large amount of parameters and high computational complexity are less suitable for portable mobile device with offline operation. In this paper, we use a lightweight local binary pattern based deep convolutional network to analyze real faces and fake faces. In order to evaluate our performance, we also utilized the CASIA-FASD database, REPLAY-ATTACK database as our benchmark database. Empirically, our proposed architecture not only shows that can improve the overall performance, but also significantly reduce amount of parameters in the relevant neural network method.

碩士
國立臺灣大學
工程科學及海洋工程學研究所
107
With the increasing requirements for face recognition in many authentication systems, how to prevent intruders from accessing the permission via Face Anti-Spoofing（FAS） techniques has become an important research area in biometrics. After the endeavors over the past few years, researchers around the world have achieved acceptable FAS detection accuracy in the same training and testing dataset. However, it is still problematic when the model trained on one dataset is tested on some other datasets. The detection error rate increases dramatically when this kind of cross-dataset evaluation arises. To address this issue, this thesis introduces the unique techniques of transfer learning and unsupervised learning to increase the generalization ability for cross-dataset evaluation. Specifically, we develop a pre-trained deep learning model to extract the high dimension features of the attack and bona fide images, and the extracted features are clustered into two subsets after the dimension is reduced. One particular characteristic of this strategy is that the dataset that being used to train the pre-trained model is not necessarily in the FAS domain, which makes our framework naturally cross-data oriented. This is quite different from other existing transfer learning methods, which mostly utilize the labeled data of the target domain to fine-tune the model parameters. Based on benchmark dataset experiments, our FAS classifier achieved lower average classification error rate (ACER) scores than state-of-the-art methods by 3%. We believe that the proposed semi-supervised learning model is of potential to overcome this challenging FAS task in biometrics.

碩士
國立交通大學
電控工程研究所
104
In the recent years, there are many liveness detection methods proposed to against photograph spoofing through analyzing the fundamental differences between human faces and printed faces. The differences in surface between human faces and printed faces are distinctive specular reflections and shades because a human face is a complex 3D object whereas a printed face can be seen as a planar object. Because an image recaptured from a photograph has the twice reflection, the gradient direction histogram of the image is different. Furthermore, the HSV color space can be more perfectly to deal with some information which human’s eyes cannot get and closer to the perception of humans. Therefore, we present to analyze facial image textures using multi-scale LBP and gradient direction analysis by Sobel operators on the illumination component of HSV color space for detecting whether there is a live person or a printed face in front of the camera. From our experiment results, we can see that the proposed feature can improve the face spoofing detection performance.

碩士
國立交通大學
電控工程研究所
105
In recent years, there are many spoofing face detection methods proposed to against 2D face spoofing attacks including printed photos and screen displayed photos through analyzing the differences between human faces and fake faces. The difference in light reflection between human faces and 2D fake faces mainly comes from distinctive shades and specular reflections because a human face is a complex 3D object whereas a 2D fake face is a flat surface. Besides, the color distribution of a fake face image is quite different to a real face due to the qualities of spoofing mediums such as a printer and a screen. For example, a printed face image usually has lower color contrast than a real face while a screen displayed face image has higher color contrast. For these different properties, this thesis proposes an approach by using the combination of two texture features and a color feature to determine whether there is a live person or a spoofing face in front of the camera. The two texture features include multi-scale LBP and the R-G deviated texture proposed in this thesis, while the color feature adopts the color moment, which is a measurement of the color distribution of an image. From the experimental results, the proposed approach indeed improves the performance of spoofing face detection.

碩士
國立臺灣科技大學
資訊工程系
107
With the development of science and technology, face recognition is now an important technology for authentication in various access control applications, especially used in mobile devices. Unlocking by face has gradually replaced fingerprint identification in some scenarios, which becomes one of the major biometric authentication technology of mobile phones. In a common camera, due to the lack of depth information, it is easy to make fake face images to crack the identification system (e.g., paper printing and screen display) compared with other biological features such as fingerprints and palm prints. Therefore, face liveness detection against spoofing attack using 2D image distortion analysis will be a very important issue in the field of information security. By virtue of the different features between real faces and fake faces, this thesis adopts local binary pattern and 2D image distortion analysis to extract texture information of images, which are used for developing our face liveness detection system against spoofing attack to distinguish fake faces from real faces by a deep neural network. The system employs only a single image captured from a common camera to discriminant real faces and fake faces. In the experiments, three kinds of face spoofing databases are used as subjects of cross-validation. The methods and dataset made by ourselves presented in this thesis can effectively classify the authenticity of human faces. The accuracy of the inside test reaches 99.55%, while that of the outside test attains 95.13%. The experimental results show that our face liveness detection system has high accuracy and generality.