Academic literature on the topic 'ReLU neural networks'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the lists of relevant articles, books, theses, conference reports, and other scholarly sources on the topic 'ReLU neural networks.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Journal articles on the topic "ReLU neural networks"
1
Liang, XingLong, and Jun Xu. "Biased ReLU neural networks." Neurocomputing 423 (January 2021): 71–79. http://dx.doi.org/10.1016/j.neucom.2020.09.050.
Full text
2
Huang, Changcun. "ReLU Networks Are Universal Approximators via Piecewise Linear or Constant Functions." Neural Computation 32, no. 11 (November 2020): 2249–78. http://dx.doi.org/10.1162/neco_a_01316.
Full textAbstract:
This letter proves that a ReLU network can approximate any continuous function with arbitrary precision by means of piecewise linear or constant approximations. For univariate function [Formula: see text], we use the composite of ReLUs to produce a line segment; all of the subnetworks of line segments comprise a ReLU network, which is a piecewise linear approximation to [Formula: see text]. For multivariate function [Formula: see text], ReLU networks are constructed to approximate a piecewise linear function derived from triangulation methods approximating [Formula: see text]. A neural unit called TRLU is designed by a ReLU network; the piecewise constant approximation, such as Haar wavelets, is implemented by rectifying the linear output of a ReLU network via TRLUs. New interpretations of deep layers, as well as some other results, are also presented.
3
Kulathunga, Nalinda, Nishath Rajiv Ranasinghe, Daniel Vrinceanu, Zackary Kinsman, Lei Huang, and Yunjiao Wang. "Effects of Nonlinearity and Network Architecture on the Performance of Supervised Neural Networks." Algorithms 14, no. 2 (February 5, 2021): 51. http://dx.doi.org/10.3390/a14020051.
Full textAbstract:
The nonlinearity of activation functions used in deep learning models is crucial for the success of predictive models. Several simple nonlinear functions, including Rectified Linear Unit (ReLU) and Leaky-ReLU (L-ReLU) are commonly used in neural networks to impose the nonlinearity. In practice, these functions remarkably enhance the model accuracy. However, there is limited insight into the effects of nonlinearity in neural networks on their performance. Here, we investigate the performance of neural network models as a function of nonlinearity using ReLU and L-ReLU activation functions in the context of different model architectures and data domains. We use entropy as a measurement of the randomness, to quantify the effects of nonlinearity in different architecture shapes on the performance of neural networks. We show that the ReLU nonliearity is a better choice for activation function mostly when the network has sufficient number of parameters. However, we found that the image classification models with transfer learning seem to perform well with L-ReLU in fully connected layers. We show that the entropy of hidden layer outputs in neural networks can fairly represent the fluctuations in information loss as a function of nonlinearity. Furthermore, we investigate the entropy profile of shallow neural networks as a way of representing their hidden layer dynamics.
4
Dung, D., V. K. Nguyen, and M. X. Thao. "ON COMPUTATION COMPLEXITY OF HIGH-DIMENSIONAL APPROXIMATION BY DEEP ReLU NEURAL NETWORKS." BULLETIN of L.N. Gumilyov Eurasian National University. MATHEMATICS. COMPUTER SCIENCE. MECHANICS Series 133, no. 4 (2020): 8–18. http://dx.doi.org/10.32523/2616-7182/2020-133-4-8-18.
Full textAbstract:
We investigate computation complexity of deep ReLU neural networks for approximating functions in H\"older-Nikol'skii spaces of mixed smoothness $\Lad$ on the unit cube $\IId:=[0,1]^d$. For any function $f\in \Lad$, we explicitly construct nonadaptive and adaptive deep ReLU neural networks having an output that approximates $f$ with a prescribed accuracy $\varepsilon$, and prove dimension-dependent bounds for the computation complexity of this approximation, characterized by the size and depth of this deep ReLU neural network, explicitly in $d$ and $\varepsilon$. Our results show the advantage of the adaptive method of approximation by deep ReLU neural networks over nonadaptive one.
5
Gühring, Ingo, Gitta Kutyniok, and Philipp Petersen. "Error bounds for approximations with deep ReLU neural networks in Ws,p norms." Analysis and Applications 18, no. 05 (September 19, 2019): 803–59. http://dx.doi.org/10.1142/s0219530519410021.
Full textAbstract:
We analyze to what extent deep Rectified Linear Unit (ReLU) neural networks can efficiently approximate Sobolev regular functions if the approximation error is measured with respect to weaker Sobolev norms. In this context, we first establish upper approximation bounds by ReLU neural networks for Sobolev regular functions by explicitly constructing the approximate ReLU neural networks. Then, we establish lower approximation bounds for the same type of function classes. A trade-off between the regularity used in the approximation norm and the complexity of the neural network can be observed in upper and lower bounds. Our results extend recent advances in the approximation theory of ReLU networks to the regime that is most relevant for applications in the numerical analysis of partial differential equations.
6
Dũng, Dinh, Van Kien Nguyen, and Mai Xuan Thao. "COMPUTATION COMPLEXITY OF DEEP RELU NEURAL NETWORKS IN HIGH-DIMENSIONAL APPROXIMATION." Journal of Computer Science and Cybernetics 37, no. 3 (September 28, 2021): 291–320. http://dx.doi.org/10.15625/1813-9663/37/3/15902.
Full textAbstract:
The purpose of the present paper is to study the computation complexity of deep ReLU neural networks to approximate functions in H\"older-Nikol'skii spaces of mixed smoothness $H_\infty^\alpha(\mathbb{I}^d)$ on the unit cube $\mathbb{I}^d:=[0,1]^d$. In this context, for any function $f\in H_\infty^\alpha(\mathbb{I}^d)$, we explicitly construct nonadaptive and adaptive deep ReLU neural networks having an output that approximates $f$ with a prescribed accuracy $\varepsilon$, and prove dimension-dependent bounds for the computation complexity of this approximation, characterized by the size and the depth of this deep ReLU neural network, explicitly in $d$ and $\varepsilon$. Our results show the advantage of the adaptive method of approximation by deep ReLU neural networks over nonadaptive one.
7
Полковникова, Н. А., Е. В. Тузинкевич, and А. Н. Попов. "Application of convolutional neural networks for monitoring of marine objects." MORSKIE INTELLEKTUAL`NYE TEHNOLOGII), no. 4(50) (December 17, 2020): 53–61. http://dx.doi.org/10.37220/mit.2020.50.4.097.
Full textAbstract:
В статье рассмотрены технологии компьютерного зрения на основе глубоких свёрточных нейронных сетей. Применение нейронных сетей особенно эффективно для решения трудно формализуемых задач. Разработана архитектура свёрточной нейронной сети применительно к задаче распознавания и классификации морских объектов на изображениях. В ходе исследования выполнен ретроспективный анализ технологий компьютерного зрения и выявлен ряд проблем, связанных с применением нейронных сетей: «исчезающий» градиент, переобучение и вычислительная сложность. При разработке архитектуры нейросети предложено использовать функцию активации RELU, обучение некоторых случайно выбранных нейронов и нормализацию с целью упрощения архитектуры нейросети. Сравнение используемых в нейросети функций активации ReLU, LeakyReLU, Exponential ReLU и SOFTMAX выполнено в среде Matlab R2020a. На основе свёрточной нейронной сети разработана программа на языке программирования Visual C# в среде MS Visual Studio для распознавания морских объектов. Программапредназначена для автоматизированной идентификации морских объектов, производит детектирование (нахождение объектов на изображении) и распознавание объектов с высокой вероятностью обнаружения. The article considers computer vision technologies based on deep convolutional neural networks. Application of neural networks is particularly effective for solving difficult formalized problems. As a result convolutional neural network architecture to the problem of recognition and classification of marine objects on images is implemented. In the research process a retrospective analysis of computer vision technologies was performed and a number of problems associated with the use of neural networks were identified: vanishing gradient, overfitting and computational complexity. To solve these problems in neural network architecture development, it was proposed to use RELU activation function, training some randomly selected neurons and normalization for simplification of neural network architecture. Comparison of ReLU, LeakyReLU, Exponential ReLU, and SOFTMAX activation functions used in the neural network implemented in Matlab R2020a.The computer program based on convolutional neural network for marine objects recognition implemented in Visual C# programming language in MS Visual Studio integrated development environment. The program is designed for automated identification of marine objects, produces detection (i.e., presence of objects on image), and objects recognition with high probability of detection.
8
Gao, Hongyang, Lei Cai, and Shuiwang Ji. "Adaptive Convolutional ReLUs." Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 04 (April 3, 2020): 3914–21. http://dx.doi.org/10.1609/aaai.v34i04.5805.
Full textAbstract:
Rectified linear units (ReLUs) are currently the most popular activation function used in neural networks. Although ReLUs can solve the gradient vanishing problem and accelerate training convergence, it suffers from the dying ReLU problem in which some neurons are never activated if the weights are not updated properly. In this work, we propose a novel activation function, known as the adaptive convolutional ReLU (ConvReLU), that can better mimic brain neuron activation behaviors and overcome the dying ReLU problem. With our novel parameter sharing scheme, ConvReLUs can be applied to convolution layers that allow each input neuron to be activated by different trainable thresholds without involving a large number of extra parameters. We employ the zero initialization scheme in ConvReLU to encourage trainable thresholds to be close to zero. Finally, we develop a partial replacement strategy that only replaces the ReLUs in the early layers of the network. This resolves the dying ReLU problem and retains sparse representations for linear classifiers. Experimental results demonstrate that our proposed ConvReLU has consistently better performance compared to ReLU, LeakyReLU, and PReLU. In addition, the partial replacement strategy is shown to be effective not only for our ConvReLU but also for LeakyReLU and PReLU.
9
Petzka, Henning, Martin Trimmel, and Cristian Sminchisescu. "Notes on the Symmetries of 2-Layer ReLU-Networks." Proceedings of the Northern Lights Deep Learning Workshop 1 (February 6, 2020): 6. http://dx.doi.org/10.7557/18.5150.
Full textAbstract:
Symmetries in neural networks allow different weight configurations leading to the same network function. For odd activation functions, the set of transformations mapping between such configurations have been studied extensively, but less is known for neural networks with ReLU activation functions. We give a complete characterization for fully-connected networks with two layers. Apart from two well-known transformations, only degenerated situations allow additional transformations that leave the network function unchanged. Reduction steps can remove only part of the degenerated cases. Finally, we present a non-degenerate situation for deep neural networks leading to new transformations leaving the network function intact.
10
Zheng, Shuxin, Qi Meng, Huishuai Zhang, Wei Chen, Nenghai Yu, and Tie-Yan Liu. "Capacity Control of ReLU Neural Networks by Basis-Path Norm." Proceedings of the AAAI Conference on Artificial Intelligence 33 (July 17, 2019): 5925–32. http://dx.doi.org/10.1609/aaai.v33i01.33015925.
Full textAbstract:
Recently, path norm was proposed as a new capacity measure for neural networks with Rectified Linear Unit (ReLU) activation function, which takes the rescaling-invariant property of ReLU into account. It has been shown that the generalization error bound in terms of the path norm explains the empirical generalization behaviors of the ReLU neural networks better than that of other capacity measures. Moreover, optimization algorithms which take path norm as the regularization term to the loss function, like Path-SGD, have been shown to achieve better generalization performance. However, the path norm counts the values of all paths, and hence the capacity measure based on path norm could be improperly influenced by the dependency among different paths. It is also known that each path of a ReLU network can be represented by a small group of linearly independent basis paths with multiplication and division operation, which indicates that the generalization behavior of the network only depends on only a few basis paths. Motivated by this, we propose a new norm Basis-path Norm based on a group of linearly independent paths to measure the capacity of neural networks more accurately. We establish a generalization error bound based on this basis path norm, and show it explains the generalization behaviors of ReLU networks more accurately than previous capacity measures via extensive experiments. In addition, we develop optimization algorithms which minimize the empirical risk regularized by the basis-path norm. Our experiments on benchmark datasets demonstrate that the proposed regularization method achieves clearly better performance on the test set than the previous regularization approaches.
Dissertations / Theses on the topic "ReLU neural networks"
1
Wang, Hao. "A new scheme for training ReLU-based multi-layer feedforward neural networks." Thesis, KTH, Skolan för datavetenskap och kommunikation (CSC), 2017. http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-217384.
Full textAbstract:
A new scheme for training Rectified Linear Unit (ReLU) based feedforward neural networks is examined in this thesis. The project starts with the row-by-row updating strategy designed for Single-hidden Layer Feedforward neural Networks (SLFNs). This strategy exploits the properties held by ReLUs and optimizes each row in the input weight matrix individually, under the common optimization scheme. Then the Direct Updating Strategy (DUS), which has two different versions: Vector-Based Method (VBM) and Matrix-Based Method (MBM), is proposed to optimize the input weight matrix as a whole. Finally DUS is extended to Multi-hidden Layer Feedforward neural Networks (MLFNs). Since the extension, for general ReLU-based MLFNs, faces an initialization dilemma, a special structure MLFN is presented. Verification experiments are conducted on six benchmark multi-class classification datasets. The results confirm that MBM algorithm for SLFNs improves the performance of neural networks, compared to its competitor, regularized extreme learning machine. For most datasets involved, MLFNs with the proposed special structure perform better when adding extra hidden layers.Ett nytt schema för träning av rektifierad linjär enhet (ReLU)-baserade och framkopplade neurala nätverk undersöks i denna avhandling. Projektet börjar med en rad-för-rad-uppdateringsstrategi designad för framkopplade neurala nätverk med ett dolt lager (SLFNs). Denna strategi utnyttjar egenskaper i ReLUs och optimerar varje rad i inmatningsviktmatrisen individuellt, enligt en gemensam optimeringsmetod. Därefter föreslås den direkta uppdateringsstrategin (DUS), som har två olika versioner: vektorbaserad metod (VBM) respektive matrisbaserad metod (MBM), för att optimera ingångsviktmatrisen som helhet. Slutli- gen utvidgas DUS till framkopplade neurala nätverk med flera lager (MLFN). Eftersom utvidgningen för generella ReLU-baserade MLFN står inför ett initieringsdilemma presenteras därför en MLFN med en speciell struktur. Verifieringsexperiment utförs på sex datamängder för klassificering av flera klasser. Resultaten bekräftar att MBM-algoritmen för SLFN förbättrar prestanda hos neurala nätverk, jämfört med konkurrenten, den regulariserade extrema inlärningsmaskinen. För de flesta använda dataset, fungerar MLFNs med den föreslagna speciella strukturen bättre när man lägger till extra dolda lager.
2
Strandqvist, Jonas. "Attractors of autoencoders : Memorization in neural networks." Thesis, Linnéuniversitetet, Institutionen för datavetenskap och medieteknik (DM), 2020. http://urn.kb.se/resolve?urn=urn:nbn:se:lnu:diva-97746.
Full textAbstract:
It is an important question in machine learning to understand how neural networks learn. This thesis sheds further light onto this by studying autoencoder neural networks which can memorize data by storing it as attractors.What this means is that an autoencoder can learn a training set and later produce parts or all of this training set even when using other inputs not belonging to this set. We seek out to illuminate the effect on how ReLU networks handle memorization when trained with different setups: with and without bias, for different widths and depths, and using two different types of training images -- from the CIFAR10 dataset and randomly generated. For this, we created controlled experiments in which we train autoencoders and compute the eigenvalues of their Jacobian matrices to discern the number of data points stored as attractors.We also manually verify and analyze these results for patterns and behavior. With this thesis we broaden the understanding of ReLU autoencoders: We find that the structure of the data has an impact on the number of attractors. For instance, we produced autoencoders where every training image became an attractor when we trained with random pictures but not with CIFAR10. Changes to depth and width on these two types of data also show different behaviour.Moreover, we observe that loss has less of an impact than expected on attractors of trained autoencoders.
3
Nery, Kaio Cesar Mendes da Silva. "Redu??o do n?mero de parcelas para modelagem da prognose do volume de floresta." UFVJM, 2016. http://acervo.ufvjm.edu.br/jspui/handle/1/1067.
Full textAbstract:
?rea de concentra??o: Silvicultura e Manejo Florestal.Submitted by Jos? Henrique Henrique (jose.neves@ufvjm.edu.br) on 2016-12-15T17:35:57Z No. of bitstreams: 2 license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) kaio_cesar_mendes_da_silva_nery.pdf: 1845996 bytes, checksum: 5d05a5ba4a2dd512a413bc72377348a4 (MD5)
Approved for entry into archive by Rodrigo Martins Cruz (rodrigo.cruz@ufvjm.edu.br) on 2016-12-16T15:43:43Z (GMT) No. of bitstreams: 2 license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) kaio_cesar_mendes_da_silva_nery.pdf: 1845996 bytes, checksum: 5d05a5ba4a2dd512a413bc72377348a4 (MD5)
Made available in DSpace on 2016-12-16T15:43:43Z (GMT). No. of bitstreams: 2 license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) kaio_cesar_mendes_da_silva_nery.pdf: 1845996 bytes, checksum: 5d05a5ba4a2dd512a413bc72377348a4 (MD5) Previous issue date: 2016
O objetivo deste estudo foi avaliar o efeito da redu??o de parcelas permanentes no custo da realiza??o do invent?rio florestal cont?nuo e realizar a prognose do volume de floresta comparando o emprego das Redes Neurais Artificiais ao modelo tradicionalmente utilizado proposto por Clutter (1963). Os dados utilizados foram provenientes de povoamentos localizados no litoral norte da Bahia, totalizando cerca de 3.000 hectares de floresta. Foram propostas duas metodologias para auxiliar na redu??o das parcelas. Para a metodologia proposta no estudo 1, os dados foram divididos aleatoriamente em dois grupos: treinamento (10, 20, 30, 40, 50, 60, 70, 80, e 90%) e generaliza??o (90, 80, 70, 60, 50, 40, 30, 20, 10%). Os dados do treinamento foram utilizados para gerar as redes neurais artificias enquanto que os dados da generaliza??o serviram para validar a capacidade das redes em gerar resultados precisos para dados desconhecidos. A metodologia proposta no estudo 2 dividiu aleatoriamente os dados em dois grupos: treinamento a escolha fixa de quantidades de parcelas pr?-estabelecidas nas tr?s classes de s?tio (10, 20, 30, 40, 50 e 60) totalizando 30, 60, 90, 120, 150 e 180 parcelas utilizadas para o treino das redes neurais e os demais dados foram utilizados para validar a capacidade das redes. A estimativa da vari?vel de estudo foi gerada no sistema computacional Neuroforest 3.3. A aplica??o das Redes Neurais Artificiais apresentaram resultados satisfat?rios bem como a aplica??o de ambas metodologias permitiram reduzir consideravelmente o custo para a realiza??o do invent?rio florestal.
Disserta??o (Mestrado) ? Programa de P?s-Gradua??o em Ci?ncia Florestal, Universidade Federal dos Vales do Jequitinhonha e Mucuri, 2016.
This study aimed to evaluate the effects of permanent plots in the cost of carrying out the continuous forest inventory and to perform prognosis of forest production comparing the use of Artificial Neural Networks to the traditional model proposed by Clutter (1963). Data were obtained from municipalities located in the northern region of Bahia state yielding 3,000 hectares of forest. Two different methodologies were proposed to reduce the number of parcels. The methodology proposed for the study 1 involved the random division of the data into two groups consisting of random reducing portions of 10, 20, 30, 40, 50, 60, 70, 80, and 90% with these percentages used for training and the remaining 90, 80, 70, 60, 50, 40, 30, 20, 10% used for validation. The methodology proposed for study 2 consisted of random reduction in parcels with a fixed parcel per site 10, 20, 30, 40, 50 and 60 in each class (30, 60, 90, 120, 150 and 180 parcels). The estimates of the study variables were generated in the computer system Neuroforest 3.3. The applications of Artificial Neural Networks showed satisfactory results and the application of both methodologies have considerably reduced the cost for conducting the forest inventory.
4
Segatto, Ênio Carlos. "Relé diferencial para transformadores de potência utilizando ferramentas inteligentes." Universidade de São Paulo, 2005. http://www.teses.usp.br/teses/disponiveis/18/18133/tde-19022016-144637/.
Full textAbstract:
Este trabalho apresenta a proposta de um sistema completo de proteção diferencial de transformadores de potência, aplicando-se as técnicas de Redes Neurais Artificiais (RNAs). O esquema proposto busca a classificação do sistema de proteção como um problema de reconhecimento e reconstrução de padrões, representando um método alternativo aos algoritmos convencionais. Vários fatores como, por exemplo, as situações de energização do transformador e a saturação dos transformadores de corrente, podem causar uma má operação do dispositivo de proteção. Com o objetivo de melhoramento na proteção digital de transformadores de potência, desenvolveu-se um sistema de proteção diferencial, incluindo dispositivos com base em RNAs, em substituição à filtragem harmônica de sinais existente no algoritmo convencional. Em complementação, esquemas de reconstrução das ondas distorcidas provenientes da saturação dos TCs são também propostos e adicionados ao algoritmo final de proteção, sendo esses comparados ao algoritmo convencional de proteção diferencial de transformadores. Com a referida adição de ferramentas de inteligência artificial a um algoritmo completo de proteção diferencial de transformadores, obteve-se uma solução bastante precisa e eficiente, capaz de responder em um tempo reduzido, se comparada aos métodos convencionais.This work proposes a complete differential protection system for power transformers, applying the Artificial Neural Network (ANN) theory. The proposed approach treat the classification of the protection system as a problem of pattern recognition and as an alternative method to the conventional algorithms. Several factors such as, for example, transformer energization and CT saturation can cause an inadequate operation of the protection relay. With the objective of improving the power transformer digital protection, a complete protection system was developed, including an ANN-based device in substitution to harmonic filters, in use in the conventional algorithm. Some approaches concerning the reconstruction of the distorted signals caused by the CTs saturation are also proposed. These routines are added to the final protection algorithm and they are compared to the conventional algorithm for power transformer protection. With the use of artificial intelligence tools in a complete power transformer protection algorithm, one intends to obtain a very precise, fast and efficient solution, if compared to the conventional methods.
5
Penido, Tamires Mousslech Andrade. "Modelagem da produ??o de povoamentos de eucalipto utilizando diferentes metodologias." UFVJM, 2017. http://acervo.ufvjm.edu.br/jspui/handle/1/1460.
Full textAbstract:
Data de aprova??o retirada da vers?o impressa do trabalho.Submitted by Jos? Henrique Henrique (jose.neves@ufvjm.edu.br) on 2017-09-05T18:36:05Z No. of bitstreams: 2 license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) tamires_mousslech_andrade_penido.pdf: 7601696 bytes, checksum: 5db79c506834a030e0682976f71d2692 (MD5)
Approved for entry into archive by Rodrigo Martins Cruz (rodrigo.cruz@ufvjm.edu.br) on 2017-09-18T17:02:44Z (GMT) No. of bitstreams: 2 license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) tamires_mousslech_andrade_penido.pdf: 7601696 bytes, checksum: 5db79c506834a030e0682976f71d2692 (MD5)
Made available in DSpace on 2017-09-18T17:02:44Z (GMT). No. of bitstreams: 2 license_rdf: 0 bytes, checksum: d41d8cd98f00b204e9800998ecf8427e (MD5) tamires_mousslech_andrade_penido.pdf: 7601696 bytes, checksum: 5db79c506834a030e0682976f71d2692 (MD5) Previous issue date: 2017
Coordena??o de Aperfei?oamento de Pessoal de N?vel Superior (CAPES)
A modelagem ? um procedimento estat?stico empregado por gestores florestais para esbo?ar o desenvolvimento vegetal com precis?o. Informa??es confi?veis do crescimento e da produ??o s?o essenciais para predizer e quantificar a estrutura futura do povoamento. O presente trabalho foi dividido em dois cap?tulos. Os objetivos foram avaliar a efici?ncia de se estimar a altura empregando diferentes modelos hipsom?tricos, crit?rios de estratifica??o e m?todos de ajuste, al?m de comparar tr?s categorias de modelos de crescimento e produ??o (MCP) em planta??es comerciais de eucalipto. Foram definidas quatro unidades de manejo florestal, totalizando 293,43 ha. O invent?rio florestal cont?nuo foi realizado em 34 parcelas permanentes de 400 m2. O espa?amento de plantio foi de 3,0 x 2,5 m. Avaliou-se a precis?o do ajuste de treze modelos hipsom?tricos. Foram treinadas RNA empregando as mesmas vari?veis de resposta e preditoras adotadas nas equa??es selecionadas. As categorias de MCP testadas foram: em n?vel de povoamento (MP), pelo sistema de equa??es simult?neas de Clutter; de distribui??o diam?trica (MDD), pelo ajuste de fun??o densidade de probabilidade de Weibul-2P e de ?rvores individuais (MAI), pelo modelo de Pienaar e Schiver. As equa??es provenientes do modelo de altura em fun??o do di?metro e da altura dominante forneceram estimativas confi?veis da altura para diferentes crit?rios de estratifica??o, demonstrando superioridade em rela??o aos modelos locais. A modelagem por regress?o e redes demonstraram-se adequadas para estimar a altura, com ou sem estratifica??o do banco de dados. A estratifica??o ? um procedimento que pode melhorar a qualidade das estimativas de altura obtidas por regress?o e RNA. As tr?s categorias de modelo proporcionaram estimativas confi?veis da produ??o em volume com casca, aos 36, 48, 60 e 72 meses, para as unidades de manejo estudadas. O MAI foi a categoria mais precisa e consistente na estimativa do volume por hectare. As proje??es com MP e MDD podem gerar estimativas similares de volume para idades al?m daquelas em que se realizou o invent?rio florestal.
Disserta??o (Mestrado) ? Programa de P?s-Gradua??o em Ci?ncia Florestal, Universidade Federal dos Vales do Jequitinhonha e Mucuri, 2017.
Modeling is a statistical procedure employed by forest managers to sketch plant development with precision. Reliable growth and production information are essential to predict and quantify the future stand structure. The present work was divided in two chapters. The objectives were to evaluate the efficiency of height estimation using different hypsometric models, stratification criteria and adjustment methods, beside to evaluate and compare three categories of growth and yield models (MCP) in commercial eucalypt plantations. Four forest management units were defined, totaling 293.43 ha. The continuous forest inventory was realized in 34 permanent plots of 400 m2. The planting spacing was 3.0 x 2.5 m. The accuracy of the fit of thirteen hypsometric models was evaluated. ANN were trained using the same response e predictive variables adopted in the selected equations. The MCP categories tested were: in level of stand (MP), using Clutter?s simultaneous equations; diameter distribution model (MDD), by adjustment of the Weibull-2P?s probability density function and individual trees (MAI), by Pienaar and Schiver model. The equations from the height model according to the diameter and the dominant height provided reliable height estimates for different stratification criteria, showing superiority in relation to local models. Regression and networks modelling were suitable for estimating height, with or without stratification of the database. Stratification is a procedure that can improve the quality of the estimates obtained by regression and ANN. The three model categories provided reliable estimates of the volume with bark production at 36, 48, 60 and 72 months for the management units studied. MAI was the most accurate and consistent category in estimating volume per hectare. Projections with MP and MDD can generate similar estimates of volume for ages beyond those in which the forest inventory was carried out.
6
Silva, Lazaro Eduardo da. "Tecnologias WEB aplicadas aos sistemas elétricos de potência." Universidade de São Paulo, 2010. http://www.teses.usp.br/teses/disponiveis/18/18154/tde-04102010-153801/.
Full textAbstract:
A utilização de equipamentos digitais e de meios de comunicação de dados, na conexão entre os dispositivos de uma subestação de energia elétrica oportunizam o uso de tecnologias para aprimorar e elevar as potencialidades de controle e supervisão do sistema elétrico de potência. As concessionárias de energia elétrica demandam altos investimentos na construção e implantação de sistemas de supervisão, controle e aquisição de dados. A partir destes sistemas é possível visualizar em um mapa da rede elétrica, os pontos de instalação dos relés e os status do seu funcionamento, possibilitando a concretização de um diagnóstico e o controle destes equipamentos. A World Wide Web se tornou um meio conveniente para acesso às informações devido ao fato dos browsers serem capazes de integrar diferentes serviços de rede em uma única e amigável interface com o usuário. A implementação de um sistema de supervisão e controle utilizando tecnologias web gratuitas pode, com custo competitivo, agregar dados que estejam dispostos geograficamente, em uma plataforma computacional ergonômica, coesa e flexível, provendo acesso aos mesmos a partir de qualquer dispositivo conectado a Internet. Como resultado deste trabalho, foi implementado um sistema Web de controle e supervisão de um bay típico de saída de linha com interface simples, geração de relatórios pertinentes e anunciação de eventos. Foi implementado um segundo sistema Web de supervisão de uma lógica de corte e restabelecimento de cargas conhecido como ERAC (Esquema Regional de Alívio de Carga) que, utilizando recursos distribuídos, exibe em uma interface intuitiva as regiões de atuação dos estágios de cortes, anunciando os eventos e gerando relatórios de interesse. Por fim, foi desenvolvido um sistema de aquisição de dados de frequência e tensão de um medidor de qualidade da energia elétrica, construção de gráficos com os dados variantes no tempo, exportação dos dados de acordo com um período selecionado e implementação de uma rede neural perceptron multicamadas Time Delay Neural Network para predição dos valores de frequência futuros. As aplicações de supervisão e controle do sistema elétrico foram desenvolvidas com tecnologias Web e testadas em uma rede Intranet, de forma a avaliar a pertinência da implementação de aplicações para o sistema elétrico de potência, que podem ser acessadas de um browser padrão. Cabe ressaltar que todo sistema desenvolvido foi testado em equipamentos reais que fazem parte de uma estrutura laboratorial disponível, aos alunos de pós-graduação da Universidade de São Paulo, Escola de Engenharia de São Carlos, Laboratório de Sistemas de Energia Elétrica onde duas subestações com relés digitais comerciais são simuladas em bastidores interligados por um backbone de fibra óptica, os quais estão conectados a rede local do laboratório, permitindo o acesso remoto aos equipamentos. Tais sistemas, por estarem em plataforma Web, podem agregar recursos, que estão distribuídos, na construção de interfaces amigáveis e intuitivas, além da disponibilidade de acesso a informação de qualquer ponto conectado à Internet que uma aplicação Web oportuniza.The use of digital equipment and data communication for the connection of the devices in an electric power substation provides the implementation of new technologies to improve and upgrade the potentialities of control and supervision of an electrical power system. The utilities demand high investments in the establishment and implementation of supervision, control and data acquisition systems. Based on these systems, it is possible to visualize a map of the electrical network, the installation points of digital relays and the status of their functions, making it possible the diagnosis and control of such equipment. The World Wide Web started a convenient access to information due to the friendly interface with the user. The implementation of a supervising and control system using free web technologies can, in a competitive cost, join data which are geographically available to an ergonomic computational platform and also provides access to the users from any device connected to the web. As a result of this work, a supervision Web control System of a typical bay with a simple interface was set up, as well as the development of reports and announcement of events. A second network supervision system for cutting load and for power reestablishment, known as regional load relief scheme, was implemented. Finally, a voltage and frequency data acquisition system from a power quality equipment with time varying graphics was developed as well as a implementation of a multiple neural perception net (Time Delay Neural Network) in order to predict future values of the frequency. The supervision and control applications were developed with web technologies and tested using an intranet in order assure the application on the electrical power system, which can be accessed by a standard browser. It is relevant to mention that the whole system was tested in actual equipment which belongs to a laboratorial structure available to the post graduation students at the University of São Paulo, School of Engineering of São Carlos, Power Systems Laboratory where two substations with commercial digital relays are simulated and interconnected by a fiber optics backbone and also connected to the local laboratory network which allows remote access to the equipment. Such system as set up in a web platform that can add resources which are distributed in the development of a friendly and intuitive interface in addition to the availability of access to the information from any point connected to the web.
7
Njima, Wafa. "Méthodes de localisation de capteurs dans le contexte de l'Internet des Objets." Electronic Thesis or Diss., Paris, CNAM, 2019. http://www.theses.fr/2019CNAM1264.
Full textAbstract:
Grace à l'émergence croissante de l'Internet des Objets et à l'importance de l'information de position dans ce contexte, lalocalisation attire de plus en plus d'attention dans la communauté des chercheurs. La localisation en extérieur est assuréepar le GPS qui n'est pas adapté aux environnements intérieurs. Plusieurs techniques de localisation en intérieur existent,mais il n'existe pas encore un standard. L'objectif de cette thèse est d'améliorer les techniques de localisation existantestout en maintenant un niveau de localisation satisfaisant avec une faible complexité de calcul. Afin de surmonter lesinconvénients des techniques de localisation existantes, nous avons proposé des approches alternatives. Dans un contexte de communication distribuée, la trilatération a été combinée avec un processus d'optimisation qui vise à compléter la matrice de distances inter nœuds à partir des données partiellement connues en se basant sur des algorithmes d’optimisation avancés. Ainsi une solution de localisation pour une architecture distribuée a été proposée. Pour une architecture centralisée utilisant la technique du fingerprinting contenant les puissances reçues, plusieurs stratégies ont été étudiées. Une étude comparative entre les différentes métriques d'évaluation de similarité a été développée. Cette étude a été suivie par le développement d'un modèle linéaire entre le fingerprint de test et les fingerprints d'une base de données générant une relation mathématique qui relie les puissances du signal reçues par un objet à ses coordonnées. Cela aide à diminuer la complexité de calcul en ligne et ainsi mieux s'adapter à un système temps réel. Enfin, la relation entre les puissances reçues et les coordonnées a été confiée à un réseau de neurones convolutif (CNN) qui traite le problème de localisation comme un problème de classification d'images radio. Les performances de toutes les approches proposées ont été évaluées et discutées. Ces résultats montrent bien l'amélioration des performances des techniques basiques en termes de précision de localisation et de complexitéWith the growing emergence of the Internet of Things and the importance of position information in this context, localization is attracting more and more attention in the researchers' community. The outdoor location is provided by GPS which is not suitable for indoors environments. Several indoor localization techniques exist, but there is not yet a standard.Existing methods are mainly based on trilateration or fingerprinting. Trilateration is a geometric method that exploits thedistances between an object and reference points to locate it. This method only works when we have at least 3 access points detected and is strongly affected by multi paths. In order to overcome these disadvantages, the fingerprinting methodcompares the fingerprint associated to the object to be located to a fingerprints' database constructed on offline. The estimated position is a combination of the selected training positions. This method is of great interest. However, it requiressignificant computing and storage capabilities. The aim of this thesis is to improve the existing localization techniqueswhile maintaining a satisfying localization accuracy with low computational complexity. In order to overcome the disadvantages of these two classes of localization techniques, we propose alternative approaches. For trilateration, it hasbeen combined with an optimization process that aims at completing the inter-node distance matrix from partially knowndata. Advanced optimization algorithms have been used in developing the mathematical equation corresponding to eachone. Using this method, we came up with a localization solution for a distributed IoT architecture. As for fingerprinting, we have exploited it to develop localization systems for a centralized IoT architecture. A comparative study between different metrics of similarity evaluation is conducted. This study was followed by the development of a linear model generating a mathematical relation that links the powers of the signal received by an object to its coordinates. This helps to reduce the online complexity of and adapts our system to real time. This is also ensured by the development of a CNN model which deal with the localization problem as radio images classification problem. The performances of all proposed approaches are evaluated and discussed. These results show the improvement of the performances of basic techniques in terms of localization accuracy and complexity
8
Alsubaihi, Salman. "Studying Perturbations on the Input of Two-Layer Neural Networks with ReLU Activation." Thesis, 2019. http://hdl.handle.net/10754/655886.
Full textAbstract:
Neural networks was shown to be very susceptible to small and imperceptible
perturbations on its input. In this thesis, we study perturbations on two-layer
piecewise linear networks. Such studies are essential in training neural networks
that are robust to noisy input. One type of perturbations we consider is `1 norm
bounded perturbations. Training Deep Neural Networks (DNNs) that are robust
to norm bounded perturbations, or adversarial attacks, remains an elusive problem.
While verification based methods are generally too expensive to robustly train large
networks, it was demonstrated in [1] that bounded input intervals can be inexpensively
propagated per layer through large networks. This interval bound propagation (IBP)
approach lead to high robustness and was the first to be employed on large networks.
However, due to the very loose nature of the IBP bounds, particularly for large
networks, the required training procedure is complex and involved. In this work, we
closely examine the bounds of a block of layers composed of an affine layer followed
by a ReLU nonlinearity followed by another affine layer. In doing so, we propose
probabilistic bounds, true bounds with overwhelming probability, that are provably
tighter than IBP bounds in expectation. We then extend this result to deeper networks
through blockwise propagation and show that we can achieve orders of magnitudes
tighter bounds compared to IBP. With such tight bounds, we demonstrate that a
simple standard training procedure can achieve the best robustness-accuracy tradeoff
across several architectures on both MNIST and CIFAR10. We, also, consider
Gaussian perturbations, where we build on a previous work that derives the first
and second output moments of a two-layer piecewise linear network [2]. In this work,
we derive an exact expression for the second moment, by dropping the zero mean
assumption in [2].
9
Bibi, Adel. "Understanding a Block of Layers in Deep Neural Networks: Optimization, Probabilistic and Tropical Geometric Perspectives." Diss., 2020. http://hdl.handle.net/10754/662589.
Full textAbstract:
This dissertation aims at theoretically studying a block of layers that is common in al- most all deep learning models. The block of layers of interest is the composition of an affine layer followed by a nonlinear activation that is followed by another affine layer. We study this block from three perspectives. (i) An Optimization Perspective. Is it possible that the output of the forward pass through this block is an optimal solution to a certain convex optimization problem? We show an equivalency between the forward pass through this block and a single iteration of deterministic and stochastic algorithms solving a ten- sor formulated convex optimization problem. As consequence, we derive for the first time a formula for computing the singular values of convolutional layers surpassing the need for the prohibitive construction of the underlying linear operator. Thereafter, we show that several deep networks can have this block replaced with the corresponding optimiza- tion algorithm predicted by our theory resulting in networks with improved generalization performance. (ii) A Probabilistic Perspective. Is it possible to analytically analyze the output of a deep network upon subjecting the input to Gaussian noise? To that regard, we derive analytical formulas for the first and second moments of this block under Gaussian input noise. We demonstrate that the derived expressions can be used to efficiently analyze the output of an arbitrary deep network in addition to constructing Gaussian adversarial attacks surpassing any need for prohibitive data augmentation procedures. (iii) A Tropi- cal Geometry Perspective. Is it possible to characterize the decision boundaries of this block as a geometric structure representing a solution set to a certain class of polynomials
(tropical polynomials)? If so, then, is it possible to utilize this geometric representation of the decision boundaries for novel reformulations to classical computer vision and machine learning tasks on arbitrary deep networks? We show that the decision boundaries of this block are a subset of a tropical hypersurface, which is intimately related to a the polytope that is the convex hull of two zonotopes. We utilize this geometric characterization to shed lights on new perspectives of network pruning.
Book chapters on the topic "ReLU neural networks"
1
Cui, Jia-le, Shuang Qiu, Ming-yang Jiang, Zhi-li Pei, and Yi-nan Lu. "Text Classification Based on ReLU Activation Function of SAE Algorithm." In Advances in Neural Networks - ISNN 2017, 44–50. Cham: Springer International Publishing, 2017. http://dx.doi.org/10.1007/978-3-319-59072-1_6.
Full text
2
Hagiwara, Katsuyuki. "On a Fitting of a Heaviside Function by Deep ReLU Neural Networks." In Neural Information Processing, 59–69. Cham: Springer International Publishing, 2018. http://dx.doi.org/10.1007/978-3-030-04167-0_6.
Full text
3
Bak, Stanley, Hoang-Dung Tran, Kerianne Hobbs, and Taylor T. Johnson. "Improved Geometric Path Enumeration for Verifying ReLU Neural Networks." In Computer Aided Verification, 66–96. Cham: Springer International Publishing, 2020. http://dx.doi.org/10.1007/978-3-030-53288-8_4.
Full text
4
Ponomarchuk, Anton, Christoph Koutschan, and Bernhard Moser. "Unboundedness of Linear Regions of Deep ReLU Neural Networks." In Communications in Computer and Information Science, 3–10. Cham: Springer International Publishing, 2022. http://dx.doi.org/10.1007/978-3-031-14343-4_1.
Full text
5
Goubault, Eric, Sébastien Palumby, Sylvie Putot, Louis Rustenholz, and Sriram Sankaranarayanan. "Static Analysis of ReLU Neural Networks with Tropical Polyhedra." In Static Analysis, 166–90. Cham: Springer International Publishing, 2021. http://dx.doi.org/10.1007/978-3-030-88806-0_8.
Full text
6
Kleine Büning, Marko, Philipp Kern, and Carsten Sinz. "Verifying Equivalence Properties of Neural Networks with ReLU Activation Functions." In Lecture Notes in Computer Science, 868–84. Cham: Springer International Publishing, 2020. http://dx.doi.org/10.1007/978-3-030-58475-7_50.
Full text
7
Bak, Stanley. "nnenum: Verification of ReLU Neural Networks with Optimized Abstraction Refinement." In Lecture Notes in Computer Science, 19–36. Cham: Springer International Publishing, 2021. http://dx.doi.org/10.1007/978-3-030-76384-8_2.
Full text
8
Tran, Hoang-Dung, Neelanjana Pal, Patrick Musau, Diego Manzanas Lopez, Nathaniel Hamilton, Xiaodong Yang, Stanley Bak, and Taylor T. Johnson. "Robustness Verification of Semantic Segmentation Neural Networks Using Relaxed Reachability." In Computer Aided Verification, 263–86. Cham: Springer International Publishing, 2021. http://dx.doi.org/10.1007/978-3-030-81685-8_12.
Full textAbstract:
AbstractThis paper introduces robustness verification for semantic segmentation neural networks (in short, semantic segmentation networks [SSNs]), building on and extending recent approaches for robustness verification of image classification neural networks. Despite recent progress in developing verification methods for specifications such as local adversarial robustness in deep neural networks (DNNs) in terms of scalability, precision, and applicability to different network architectures, layers, and activation functions, robustness verification of semantic segmentation has not yet been considered. We address this limitation by developing and applying new robustness analysis methods for several segmentation neural network architectures, specifically by addressing reachability analysis of up-sampling layers, such as transposed convolution and dilated convolution. We consider several definitions of robustness for segmentation, such as the percentage of pixels in the output that can be proven robust under different adversarial perturbations, and a robust variant of intersection-over-union (IoU), the typical performance evaluation measure for segmentation tasks. Our approach is based on a new relaxed reachability method, allowing users to select the percentage of a number of linear programming problems (LPs) to solve when constructing the reachable set, through a relaxation factor percentage. The approach is implemented within NNV, then applied and evaluated on segmentation datasets, such as a multi-digit variant of MNIST known as M2NIST. Thorough experiments show that by using transposed convolution for up-sampling and average-pooling for down-sampling, combined with minimizing the number of ReLU layers in the SSNs, we can obtain SSNs with not only high accuracy (IoU), but also that are more robust to adversarial attacks and amenable to verification. Additionally, using our new relaxed reachability method, we can significantly reduce the verification time for neural networks whose ReLU layers dominate the total analysis time, even in classification tasks.
9
Hashemi, Vahid, Panagiotis Kouvaros, and Alessio Lomuscio. "OSIP: Tightened Bound Propagation for the Verification of ReLU Neural Networks." In Software Engineering and Formal Methods, 463–80. Cham: Springer International Publishing, 2021. http://dx.doi.org/10.1007/978-3-030-92124-8_26.
Full text
10
Khedr, Haitham, James Ferlez, and Yasser Shoukry. "PEREGRiNN: Penalized-Relaxation Greedy Neural Network Verifier." In Computer Aided Verification, 287–300. Cham: Springer International Publishing, 2021. http://dx.doi.org/10.1007/978-3-030-81685-8_13.
Full textAbstract:
AbstractNeural Networks (NNs) have increasingly apparent safety implications commensurate with their proliferation in real-world applications: both unanticipated as well as adversarial misclassifications can result in fatal outcomes. As a consequence, techniques of formal verification have been recognized as crucial to the design and deployment of safe NNs. In this paper, we introduce a new approach to formally verify the most commonly considered safety specifications for ReLU NNs – i.e. polytopic specifications on the input and output of the network. Like some other approaches, ours uses a relaxed convex program to mitigate the combinatorial complexity of the problem. However, unique in our approach is the way we use a convex solver not only as a linear feasibility checker, but also as a means of penalizing the amount of relaxation allowed in solutions. In particular, we encode each ReLU by means of the usual linear constraints, and combine this with a convex objective function that penalizes the discrepancy between the output of each neuron and its relaxation. This convex function is further structured to force the largest relaxations to appear closest to the input layer; this provides the further benefit that the most “problematic” neurons are conditioned as early as possible, when conditioning layer by layer. This paradigm can be leveraged to create a verification algorithm that is not only faster in general than competing approaches, but is also able to verify considerably more safety properties; we evaluated PEREGRiNN on a standard MNIST robustness verification suite to substantiate these claims.
Conference papers on the topic "ReLU neural networks"
1
Moser, Bernhard A., Michal Lewandowski, Somayeh Kargaran, Werner Zellinger, Battista Biggio, and Christoph Koutschan. "Tessellation-Filtering ReLU Neural Networks." In Thirty-First International Joint Conference on Artificial Intelligence {IJCAI-22}. California: International Joint Conferences on Artificial Intelligence Organization, 2022. http://dx.doi.org/10.24963/ijcai.2022/463.
Full textAbstract:
We identify tessellation-filtering ReLU neural networks that, when composed with another ReLU network, keep its non-redundant tessellation unchanged or reduce it.The additional network complexity modifies the shape of the decision surface without increasing the number of linear regions. We provide a mathematical understanding of the related additional expressiveness by means of a novel measure of shape complexity by counting deviations from convexity which results in a Boolean algebraic characterization of this special class. A local representation theorem gives rise to novel approaches for pruning and decision surface analysis.
2
Kouvaros, Panagiotis, and Alessio Lomuscio. "Towards Scalable Complete Verification of Relu Neural Networks via Dependency-based Branching." In Thirtieth International Joint Conference on Artificial Intelligence {IJCAI-21}. California: International Joint Conferences on Artificial Intelligence Organization, 2021. http://dx.doi.org/10.24963/ijcai.2021/364.
Full textAbstract:
We introduce an efficient method for the complete verification of ReLU-based feed-forward neural networks. The method implements branching on the ReLU states on the basis of a notion of dependency between the nodes. This results in dividing the original verification problem into a set of sub-problems whose MILP formulations require fewer integrality constraints. We evaluate the method on all of the ReLU-based fully connected networks from the first competition for neural network verification. The experimental results obtained show 145% performance gains over the present state-of-the-art in complete verification.
3
Aftab, Arya, Alireza Morsali, and Shahrokh Ghaemmaghami. "Multi-Head Relu Implicit Neural Representation Networks." In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022. http://dx.doi.org/10.1109/icassp43922.2022.9747352.
Full text
4
Salman, Shaeke, Canlin Zhang, Xiuwen Liu, and Washington Mio. "Towards Quantifying Intrinsic Generalization of Deep ReLU Networks." In 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020. http://dx.doi.org/10.1109/ijcnn48605.2020.9206619.
Full text
5
Zhu, Yiwei, Feng Wang, Wenjie Wan, and Min Zhang. "Attack-Guided Efficient Robustness Verification of ReLU Neural Networks." In 2021 International Joint Conference on Neural Networks (IJCNN). IEEE, 2021. http://dx.doi.org/10.1109/ijcnn52387.2021.9534410.
Full text
6
Wang, Yuan. "Estimation and Comparison of Linear Regions for ReLU Networks." In Thirty-First International Joint Conference on Artificial Intelligence {IJCAI-22}. California: International Joint Conferences on Artificial Intelligence Organization, 2022. http://dx.doi.org/10.24963/ijcai.2022/492.
Full textAbstract:
We study the relationship between the arrangement of neurons and the complexity of the ReLU-activated neural networks measured by the number of linear regions. More specifically, we provide both theoretical and empirical evidence for the point of view that shallow networks tend to have higher complexity than deep ones when the total number of neurons is fixed. In the theoretical part, we prove that this is the case for networks whose neurons in the hidden layers are arranged in the forms of 1x2n, 2xn and nx2; in the empirical part, we implement an algorithm that precisely tracks (hence counts) all the linear regions, and run it on networks with various structures. Although the time complexity of the algorithm is quite high, we verify that the problem of calculating the number of linear regions of a ReLU network is itself NP-hard. So currently there is no surprisingly efficient way to solve it. Roughly speaking, in the algorithm we divide the linear regions into subregions called the "activation regions", which are convex and easy to propagate through the network. The relationship between the number of the linear regions and that of the activation regions is also discussed.
7
Daróczy, Bálint, and Dániel Rácz. "Gradient representations in ReLU networks as similarity functions." In ESANN 2021 - European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. Louvain-la-Neuve (Belgium): Ciaco - i6doc.com, 2021. http://dx.doi.org/10.14428/esann/2021.es2021-153.
Full text
8
Voigtlaender, Felix, and Philipp Petersen. "Approximation in Lp(µ) with deep ReLU neural networks." In 2019 13th International conference on Sampling Theory and Applications (SampTA). IEEE, 2019. http://dx.doi.org/10.1109/sampta45681.2019.9030992.
Full text
9
Ide, Hidenori, and Takio Kurita. "Improvement of learning for CNN with ReLU activation by sparse regularization." In 2017 International Joint Conference on Neural Networks (IJCNN). IEEE, 2017. http://dx.doi.org/10.1109/ijcnn.2017.7966185.
Full text
10
Li, Gordon H. Y., Ryoto Sekine, Rajveer Nehra, Robert M. Gray, Luis Ledezma, Qiushi Guo, and Alireza Marandi. "All-optical, ultrafast energy-efficient ReLU function for nanophotonic neural networks." In CLEO: Science and Innovations. Washington, D.C.: Optica Publishing Group, 2022. http://dx.doi.org/10.1364/cleo_si.2022.sth5g.6.
Full textAbstract:
We introduce and experimentally demonstrate an all-optical ReLU nonlinear activation function based on the strong quadratic nonlinearity of lithium niobate nanophotonic waveguides and achieve a record-breaking energy-time product per activation of 1.2 × 10 − 27 J · s to overcome the nonlinearity bottleneck in photonic neural networks.