Journal articles on the topic 'Neural Network Pruning'
To see the other types of publications on this topic, follow the link: Neural Network Pruning.
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the top 50 journal articles for your research on the topic 'Neural Network Pruning.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Browse journal articles on a wide variety of disciplines and organise your bibliography correctly.
1
JORGENSEN, THOMAS D., BARRY P. HAYNES, and CHARLOTTE C. F. NORLUND. "PRUNING ARTIFICIAL NEURAL NETWORKS USING NEURAL COMPLEXITY MEASURES." International Journal of Neural Systems 18, no. 05 (October 2008): 389–403. http://dx.doi.org/10.1142/s012906570800166x.
Abstract:
This paper describes a new method for pruning artificial neural networks, using a measure of the neural complexity of the neural network. This measure is used to determine the connections that should be pruned. The measure computes the information-theoretic complexity of a neural network, which is similar to, yet different from previous research on pruning. The method proposed here shows how overly large and complex networks can be reduced in size, whilst retaining learnt behaviour and fitness. The technique proposed here helps to discover a network topology that matches the complexity of the problem it is meant to solve. This novel pruning technique is tested in a robot control domain, simulating a racecar. It is shown, that the proposed pruning method is a significant improvement over the most commonly used pruning method Magnitude Based Pruning. Furthermore, some of the pruned networks prove to be faster learners than the benchmark network that they originate from. This means that this pruning method can also help to unleash hidden potential in a network, because the learning time decreases substantially for a pruned a network, due to the reduction of dimensionality of the network.
2
Ganguli, Tushar, and Edwin K. P. Chong. "Activation-Based Pruning of Neural Networks." Algorithms 17, no. 1 (January 21, 2024): 48. http://dx.doi.org/10.3390/a17010048.
Abstract:
We present a novel technique for pruning called activation-based pruning to effectively prune fully connected feedforward neural networks for multi-object classification. Our technique is based on the number of times each neuron is activated during model training. We compare the performance of activation-based pruning with a popular pruning method: magnitude-based pruning. Further analysis demonstrated that activation-based pruning can be considered a dimensionality reduction technique, as it leads to a sparse low-rank matrix approximation for each hidden layer of the neural network. We also demonstrate that the rank-reduced neural network generated using activation-based pruning has better accuracy than a rank-reduced network using principal component analysis. We provide empirical results to show that, after each successive pruning, the amount of reduction in the magnitude of singular values of each matrix representing the hidden layers of the network is equivalent to introducing the sum of singular values of the hidden layers as a regularization parameter to the objective function.
3
Koene, Randal A., and Yoshio Takane. "Discriminant Component Pruning: Regularization and Interpretation of Multilayered Backpropagation Networks." Neural Computation 11, no. 3 (April 1, 1999): 783–802. http://dx.doi.org/10.1162/089976699300016665.
Abstract:
Neural networks are often employed as tools in classification tasks. The use of large networks increases the likelihood of the task's being learned, although it may also lead to increased complexity. Pruning is an effective way of reducing the complexity of large networks. We present discriminant components pruning (DCP), a method of pruning matrices of summed contributions between layers of a neural network. Attempting to interpret the underlying functions learned by the network can be aided by pruning the network. Generalization performance should be maintained at its optimal level following pruning. We demonstrate DCP's effectiveness at maintaining generalization performance, applicability to a wider range of problems, and the usefulness of such pruning for network interpretation. Possible enhancements are discussed for the identification of the optimal reduced rank and inclusion of nonlinear neural activation functions in the pruning algorithm.
4
Ling, Xing. "Summary of Deep Neural Network Pruning Algorithms." Applied and Computational Engineering 8, no. 1 (August 1, 2023): 352–61. http://dx.doi.org/10.54254/2755-2721/8/20230182.
Abstract:
As deep learning has rapidly progressed in the 21st century, artificial neural networks have been continuously enhanced with deeper structures and larger parameter sets to tackle increasingly complex problems. However, this development also brings about the drawbacks of high computational and storage costs, which limit the application of neural networks in some practical scenarios. As a result, in recent years, more researchers have suggested and implemented network pruning techniques to decrease neural networks' computational and storage expenses while retaining the same level of accuracy. This paper reviews the research progress of network pruning techniques and categorizes them into unstructured and structured pruning. Finally, the shortcomings of current pruning techniques and possible future development directions are pointed out.
5
Gong, Ziyi, Huifu Zhang, Hao Yang, Fangjun Liu, and Fan Luo. "A Review of Neural Network Lightweighting Techniques." Innovation & Technology Advances 1, no. 2 (January 16, 2024): 1–16. http://dx.doi.org/10.61187/ita.v1i2.36.
Abstract:
The application of portable devices based on deep learning has become increasingly widespread, which has made the deployment of complex neural networks on embedded devices a hot research topic. Neural network lightweighting is one of the key technologies for applying neural networks to embedded devices. This paper elaborates and analyzes neural network lightweighting techniques from two aspects: model pruning and network structure design. For model pruning, a comparison of methods from different periods is conducted, highlighting their advantages and limitations. Regarding network structure design, the principles of four classical lightweight network designs are described from a mathematical perspective, and the latest optimization methods for these networks are reviewed. Finally, potential research directions for lightweight neural network pruning and structure design optimization are discussed.
6
Guo, Changyi, and Ping Li. "Hybrid Pruning Method Based on Convolutional Neural Network Sensitivity and Statistical Threshold." Journal of Physics: Conference Series 2171, no. 1 (January 1, 2022): 012055. http://dx.doi.org/10.1088/1742-6596/2171/1/012055.
Abstract:
Abstract The hybrid pruning algorithm can not only ensure the precision of the network, but also achieve a good balance between pruning ratio and computation. However, traditional pruning algorithms use coarse-grained or fine-grained pruning networks, which have the tradeoff problem between pruning rate and computation amount. To this end, this paper presents. A hybrid pruning method of sensitivity and statistical threshold. Firstly, coarse-grained pruning is carried out on the network, and a fast sensitivity test is conducted on the convolutional layer of the network to determine the channels that need pruning within the tolerance range of network precision decline. Then, fine-grained pruning is performed on the network. Count the weights of the pruned network, calculate the thresholds of the weights of each layer, and delete the weights less than the thresholds so as to further reduce the size of the network and the amount of calculation. Hybrid pruning performs very well in AlexNet and Resnet networks. In particular, the method proposed in this paper is used in CIFAR-10 dataset, and the compression of FLOPs is 60%, while the compression of parameter number is nearly 80%. Compared with single pruning method, Hybrid pruning is better.
7
Zou, Yunhuan. "Research On Pruning Methods for Mobilenet Convolutional Neural Network." Highlights in Science, Engineering and Technology 81 (January 26, 2024): 232–36. http://dx.doi.org/10.54097/a742e326.
Abstract:
This paper comprehensively reviews pruning methods for MobileNet convolutional neural networks. MobileNet is a lightweight convolutional neural network suitable for resource-constrained environments such as mobile devices.Various pruning methods can be applied to reduce the model's storage space and computational complexity, including channel pruning, kernel pruning, and weight pruning. Channel pruning removes unimportant channels to reduce redundant parameters and computations in the model, while kernel pruning reduces redundant calculations by pruning convolutional kernels. Weight pruning involves setting small-weighted elements to zero to remove unimportant weights. These pruning methods can be used individually or in combination. After pruning, fine-tuning is necessary to restore the model's performance. Factors such as pruning rate, pruning order, and pruning location need to be considered to achieve a balance between reducing model size and computational complexity while minimizing performance loss. Pruning methods based on MobileNet convolutional neural networks reduce the parameter count and computational complexity, improving model lightweightness and inference efficiency. These methods are of significant value in resource-constrained environments such as mobile devices. This review provides insights into pruning methods for MobileNet convolutional neural networks and their applications in lightweight and efficient model deployment. Further advancements such as automated pruning methods driven by reinforcement learning algorithms can enhance the pruning process to achieve optimal model compression effects. Future research should focus on adapting and optimizing these pruning methods for specific problem domains and achieving even higher compression ratios and computational speedups.
8
Liang, Ling, Lei Deng, Yueling Zeng, Xing Hu, Yu Ji, Xin Ma, Guoqi Li, and Yuan Xie. "Crossbar-Aware Neural Network Pruning." IEEE Access 6 (2018): 58324–37. http://dx.doi.org/10.1109/access.2018.2874823.
9
Tsai, Feng-Sheng, Yi-Li Shih, Chin-Tzong Pang, and Sheng-Yi Hsu. "Formulation of Pruning Maps with Rhythmic Neural Firing." Mathematics 7, no. 12 (December 17, 2019): 1247. http://dx.doi.org/10.3390/math7121247.
Abstract:
Rhythmic neural firing is thought to underlie the operation of neural function. This triggers the construction of dynamical network models to investigate how the rhythms interact with each other. Recently, an approach concerning neural path pruning has been proposed in a dynamical network system, in which critical neuronal connections are identified and adjusted according to the pruning maps, enabling neurons to produce rhythmic, oscillatory activity in simulation. Here, we construct a sort of homomorphic functions based on different rhythms of neural firing in network dynamics. Armed with the homomorphic functions, the pruning maps can be simply expressed in terms of interactive rhythms of neural firing and allow a concrete analysis of coupling operators to control network dynamics. Such formulation of pruning maps is applied to probe the consolidation of rhythmic patterns between layers of neurons in feedforward neural networks.
10
Wang, Miao, Xu Yang, Yunchong Qian, Yunlin Lei, Jian Cai, Ziyi Huan, Xialv Lin, and Hao Dong. "Adaptive Neural Network Structure Optimization Algorithm Based on Dynamic Nodes." Current Issues in Molecular Biology 44, no. 2 (February 7, 2022): 817–32. http://dx.doi.org/10.3390/cimb44020056.
Abstract:
Large-scale artificial neural networks have many redundant structures, making the network fall into the issue of local optimization and extended training time. Moreover, existing neural network topology optimization algorithms have the disadvantage of many calculations and complex network structure modeling. We propose a Dynamic Node-based neural network Structure optimization algorithm (DNS) to handle these issues. DNS consists of two steps: the generation step and the pruning step. In the generation step, the network generates hidden layers layer by layer until accuracy reaches the threshold. Then, the network uses a pruning algorithm based on Hebb’s rule or Pearson’s correlation for adaptation in the pruning step. In addition, we combine genetic algorithm to optimize DNS (GA-DNS). Experimental results show that compared with traditional neural network topology optimization algorithms, GA-DNS can generate neural networks with higher construction efficiency, lower structure complexity, and higher classification accuracy.
11
Ao, Ren, Zhang Tao, Wang Yuhao, Lin Sheng, Dong Peiyan, Chen Yen-kuang, Xie Yuan, and Wang Yanzhi. "DARB: A Density-Adaptive Regular-Block Pruning for Deep Neural Networks." Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 04 (April 3, 2020): 5495–502. http://dx.doi.org/10.1609/aaai.v34i04.6000.
Abstract:
The rapidly growing parameter volume of deep neural networks (DNNs) hinders the artificial intelligence applications on resource constrained devices, such as mobile and wearable devices. Neural network pruning, as one of the mainstream model compression techniques, is under extensive study to reduce the model size and thus the amount of computation. And thereby, the state-of-the-art DNNs are able to be deployed on those devices with high runtime energy efficiency. In contrast to irregular pruning that incurs high index storage and decoding overhead, structured pruning techniques have been proposed as the promising solutions. However, prior studies on structured pruning tackle the problem mainly from the perspective of facilitating hardware implementation, without diving into the deep to analyze the characteristics of sparse neural networks. The neglect on the study of sparse neural networks causes inefficient trade-off between regularity and pruning ratio. Consequently, the potential of structurally pruning neural networks is not sufficiently mined.In this work, we examine the structural characteristics of the irregularly pruned weight matrices, such as the diverse redundancy of different rows, the sensitivity of different rows to pruning, and the position characteristics of retained weights. By leveraging the gained insights as a guidance, we first propose the novel block-max weight masking (BMWM) method, which can effectively retain the salient weights while imposing high regularity to the weight matrix. As a further optimization, we propose a density-adaptive regular-block (DARB) pruning that can effectively take advantage of the intrinsic characteristics of neural networks, and thereby outperform prior structured pruning work with high pruning ratio and decoding efficiency. Our experimental results show that DARB can achieve 13× to 25× pruning ratio, which are 2.8× to 4.3× improvements than the state-of-the-art counterparts on multiple neural network models and tasks. Moreover, DARB can achieve 14.3× decoding efficiency than block pruning with higher pruning ratio.
12
Lee, Donghyeon, Eunho Lee, and Youngbae Hwang. "Lossless Reconstruction of Convolutional Neural Network for Channel-Based Network Pruning." Sensors 23, no. 4 (February 13, 2023): 2102. http://dx.doi.org/10.3390/s23042102.
Abstract:
Network pruning reduces the number of parameters and computational costs of convolutional neural networks while maintaining high performance. Although existing pruning methods have achieved excellent results, they do not consider reconstruction after pruning in order to apply the network to actual devices. This study proposes a reconstruction process for channel-based network pruning. For lossless reconstruction, we focus on three components of the network: the residual block, skip connection, and convolution layer. Union operation and index alignment are applied to the residual block and skip connection, respectively. Furthermore, we reconstruct a compressed convolution layer by considering batch normalization. We apply our method to existing channel-based pruning methods for downstream tasks such as image classification, object detection, and semantic segmentation. Experimental results show that compressing a large model has a 1.93% higher accuracy in image classification, 2.2 higher mean Intersection over Union (mIoU) in semantic segmentation, and 0.054 higher mean Average Precision (mAP) in object detection than well-designed small models. Moreover, we demonstrate that our method can reduce the actual latency by 8.15× and 5.29× on Raspberry Pi and Jetson Nano, respectively.
13
Pei, Songwen, Yusheng Wu, Jin Guo, and Meikang Qiu. "Neural Network Pruning by Recurrent Weights for Finance Market." ACM Transactions on Internet Technology 22, no. 3 (August 31, 2022): 1–23. http://dx.doi.org/10.1145/3433547.
Abstract:
Convolutional Neural Networks (CNNs) and deep learning technology are applied in current financial market to rapidly promote the development of finance market and Internet economy. The continuous development of neural networks with more hidden layers improves the performance but increases the computational complexity. Generally, channel pruning methods are useful to compact neural networks. However, typical channel pruning methods would remove layers by mistake due to the static pruning ratio of manual setting, which could destroy the whole structure of neural networks. It is difficult to improve the ratio of compressing neural networks only by pruning channels while maintaining good network structures. Therefore, we propose a novel neural Networks Pruning by Recurrent Weights ( NPRW ) that can repeatedly evaluate the significance of weights and adaptively adjust them to compress neural networks within acceptable loss of accuracy. The recurrent weights with low sensitivity are compulsorily set to zero by evaluating the magnitude of weights, and pruned network only uses a few significant weights. Then, we add the regularization to the scaling factors on neural networks, in which recurrent weights with high sensitivity can be dynamically updated and weights of low sensitivity stay at zero invariably. By this way, the significance of channels can be quantitatively evaluated by recurrent weights. It has been verified with typical neural networks of LeNet, VGGNet, and ResNet on multiple benchmark datasets involving stock index futures, digital recognition, and image classification. The pruned LeNet-5 achieves the 58.9% reduction amount of parameters with 0.29% loss of total accuracy for Shanghai and Shenzhen 300 stock index futures. As for the CIFAR-10, the pruned VGG-19 reduces more than 50% FLOPs, and the decrease of network accuracy is less than 0.5%. In addition, the pruned ResNet-164 tested on the SVHN reduces more than 58% FLOPs with relative improvement on accuracy by 0.11%.
14
Scholl, Carolin, Michael E. Rule, and Matthias H. Hennig. "The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules." PLOS Computational Biology 17, no. 10 (October 11, 2021): e1009458. http://dx.doi.org/10.1371/journal.pcbi.1009458.
Abstract:
During development, biological neural networks produce more synapses and neurons than needed. Many of these synapses and neurons are later removed in a process known as neural pruning. Why networks should initially be over-populated, and the processes that determine which synapses and neurons are ultimately pruned, remains unclear. We study the mechanisms and significance of neural pruning in model neural networks. In a deep Boltzmann machine model of sensory encoding, we find that (1) synaptic pruning is necessary to learn efficient network architectures that retain computationally-relevant connections, (2) pruning by synaptic weight alone does not optimize network size and (3) pruning based on a locally-available measure of importance based on Fisher information allows the network to identify structurally important vs. unimportant connections and neurons. This locally-available measure of importance has a biological interpretation in terms of the correlations between presynaptic and postsynaptic neurons, and implies an efficient activity-driven pruning rule. Overall, we show how local activity-dependent synaptic pruning can solve the global problem of optimizing a network architecture. We relate these findings to biology as follows: (I) Synaptic over-production is necessary for activity-dependent connectivity optimization. (II) In networks that have more neurons than needed, cells compete for activity, and only the most important and selective neurons are retained. (III) Cells may also be pruned due to a loss of synapses on their axons. This occurs when the information they convey is not relevant to the target population.
15
Wu, Tao, Jiao Shi, Deyun Zhou, Xiaolong Zheng, and Na Li. "Evolutionary Multi-Objective One-Shot Filter Pruning for Designing Lightweight Convolutional Neural Network." Sensors 21, no. 17 (September 2, 2021): 5901. http://dx.doi.org/10.3390/s21175901.
Abstract:
Deep neural networks have achieved significant development and wide applications for their amazing performance. However, their complex structure, high computation and storage resource limit their applications in mobile or embedding devices such as sensor platforms. Neural network pruning is an efficient way to design a lightweight model from a well-trained complex deep neural network. In this paper, we propose an evolutionary multi-objective one-shot filter pruning method for designing a lightweight convolutional neural network. Firstly, unlike some famous iterative pruning methods, a one-shot pruning framework only needs to perform filter pruning and model fine-tuning once. Moreover, we built a constraint multi-objective filter pruning problem in which two objectives represent the filter pruning ratio and the accuracy of the pruned convolutional neural network, respectively. A non-dominated sorting-based evolutionary multi-objective algorithm was used to solve the filter pruning problem, and it provides a set of Pareto solutions which consists of a series of different trade-off pruned models. Finally, some models are uniformly selected from the set of Pareto solutions to be fine-tuned as the output of our method. The effectiveness of our method was demonstrated in experimental studies on four designed models, LeNet and AlexNet. Our method can prune over 85%, 82%, 75%, 65%, 91% and 68% filters with little accuracy loss on four designed models, LeNet and AlexNet, respectively.
16
Wang, Jielei, Zongyong Cui, Zhipeng Zang, Xiangjie Meng, and Zongjie Cao. "Absorption Pruning of Deep Neural Network for Object Detection in Remote Sensing Imagery." Remote Sensing 14, no. 24 (December 9, 2022): 6245. http://dx.doi.org/10.3390/rs14246245.
Abstract:
In recent years, deep convolutional neural networks (DCNNs) have been widely used for object detection tasks in remote sensing images. However, the over-parametrization problem of DCNNs hinders their application in resource-constrained remote sensing devices. In order to solve this problem, we propose a network pruning method (named absorption pruning) to compress the remote sensing object detection network. Unlike the classical iterative three-stage pruning pipeline used in existing methods, absorption pruning is designed as a four-stage pruning pipeline that only needs to be executed once, which differentiates it from existing methods. Furthermore, the absorption pruning no longer identifies unimportant filters, as in existing pruning methods, but instead selects filters that are easy to learn. In addition, we design a method for pruning ratio adjustment based on the object characteristics in remote sensing images, which can help absorption pruning to better compress deep neural networks for remote sensing image processing. The experimental results on two typical remote sensing data sets—SSDD and RSOD—demonstrate that the absorption pruning method not only can remove 60% of the filter parameters from CenterNet101 harmlessly but also eliminate the over-fitting problem of the pre-trained network.
17
Xiao, Penghao, Teng Xu, Xiayang Xiao, Weisong Li, and Haipeng Wang. "Distillation Sparsity Training Algorithm for Accelerating Convolutional Neural Networks in Embedded Systems." Remote Sensing 15, no. 10 (May 17, 2023): 2609. http://dx.doi.org/10.3390/rs15102609.
Abstract:
The rapid development of neural networks has come at the cost of increased computational complexity. Neural networks are both computationally intensive and memory intensive; as such, the minimal energy and computing power of satellites pose a challenge for automatic target recognition (ATR). Knowledge distillation (KD) can distill knowledge from a cumbersome teacher network to a lightweight student network, transferring the essential information learned by the teacher network. Thus, the concept of KD can be used to improve the accuracy of student networks. Even when learning from a teacher network, there is still redundancy in the student network. Traditional networks fix the structure before training, such that training does not improve the situation. This paper proposes a distillation sparsity training (DST) algorithm based on KD and network pruning to address the above limitations. We first improve the accuracy of the student network through KD, and then through network pruning, allowing the student network to learn which connections are essential. DST allows the teacher network to teach the pruned student network directly. The proposed algorithm was tested on the CIFAR-100, MSTAR, and FUSAR-Ship data sets, with a 50% sparsity setting. First, a new loss function for the teacher-pruned student was proposed, and the pruned student network showed a performance close to that of the teacher network. Second, a new sparsity model (uniformity half-pruning UHP) was designed to solve the problem that unstructured pruning does not facilitate the implementation of general-purpose hardware acceleration and storage. Compared with traditional unstructured pruning, UHP can double the speed of neural networks.
18
Wu, Tingting, Chunhe Song, Peng Zeng, and Changqing Xia. "Cluster-Based Structural Redundancy Identification for Neural Network Compression." Entropy 25, no. 1 (December 21, 2022): 9. http://dx.doi.org/10.3390/e25010009.
Abstract:
The increasingly large structure of neural networks makes it difficult to deploy on edge devices with limited computing resources. Network pruning has become one of the most successful model compression methods in recent years. Existing works typically compress models based on importance, removing unimportant filters. This paper reconsiders model pruning from the perspective of structural redundancy, claiming that identifying functionally similar filters plays a more important role, and proposes a model pruning framework for clustering-based redundancy identification. First, we perform cluster analysis on the filters of each layer to generate similar sets with different functions. We then propose a criterion for identifying redundant filters within similar sets. Finally, we propose a pruning scheme that automatically determines the pruning rate of each layer. Extensive experiments on various benchmark network architectures and datasets demonstrate the effectiveness of our proposed framework.
19
Duckro, Donald E., Dennis W. Quinn, and Samuel J. Gardner. "Neural Network Pruning with Tukey-Kramer Multiple Comparison Procedure." Neural Computation 14, no. 5 (May 1, 2002): 1149–68. http://dx.doi.org/10.1162/089976602753633420.
Abstract:
Reducing a neural network's complexity improves the ability of the network to generalize future examples. Like an overfitted regression function, neural networks may miss their target because of the excessive degrees of freedom stored up in unnecessary parameters. Over the past decade, the subject of pruning networks produced nonstatistical algorithms like Skeletonization, Optimal Brain Damage, and Optimal Brain Surgeon as methods to remove connections with the least salience. The method proposed here uses the bootstrap algorithm to estimate the distribution of the model parameter saliences. Statistical multiple comparison procedures are then used to make pruning decisions. We show this method compares well with Optimal Brain Surgeon in terms of ability to prune and the resulting network performance.
20
Qin, Tian, Jiang Zhang, and Xihua Zhu. "Analysis of Pruning Optimization Technology Based on Deep Learning." Highlights in Science, Engineering and Technology 4 (July 26, 2022): 332–38. http://dx.doi.org/10.54097/hset.v4i.921.
Abstract:
Neural network models have become a major topic in many areas in recent years due to their excellent problem-solving ability. However, during the training process in the real application, neural network models usually have relatively high requirements on the performance of the platform, which limits their applications, especially in battery-driven devices or embedded platforms with relatively weak performance. Therefore, the importance of optimizing and pruning neural network models is very important. This paper analyzes four categories of pruning techniques, including channel pruning, neuron pruning, weight pruning and layer pruning. This paper investigates its main ideas and particular pruning methods and compares the optimised effects of these pruning techniques in different neural network models through experiments. This paper discusses neuron pruning on the VGG-16 model, weight pruning on ResNet and AlexNet, and layer pruning on the YOLOV3 model, respectively. The parameter amount and inference speed before and after pruning are compared, and the accuracies drop before and after pruning are verified on data sets such as CIFAR-10.
21
Wang, Jiajun. "Research on pruning optimization techniques for neural networks." Applied and Computational Engineering 19, no. 1 (October 23, 2023): 152–58. http://dx.doi.org/10.54254/2755-2721/19/20231025.
Abstract:
Large deep neural networks have been deploying in more and more application scenarios due to their success in multiple application scenarios. However, deep neural networks are difficult to apply to devices with fewer resources, as the large models and huge demand for computing resources make this difficult. Pruning optimization, as a critical model compression method, has become an essential part of the deployment process of deep neural networks and has extreme significance. This article summarizes the methods of deep neural network pruning optimization technology, sorts out the current research status of pruning optimization technology, analyzes different fine-grained pruning optimization technologies based on the different fine-grained levels of pruning optimization technology, and comparing the characteristics of different fine-grained pruning optimization techniques. This article also introduces the development process and current development direction of different fine-grained pruning optimization technologies, compares the effectiveness differences between different fine-grained pruning optimization technologies and looks forward to the combination of pruning optimization technology and model quantification technology. The end of the paper summarizes pruning optimization techniques and provides prospects.
22
Wang, Shuang, and Zhaogong Zhang. "ScoringNet: A Neural Network Based Pruning Criteria for Structured Pruning." Scientific Programming 2023 (April 14, 2023): 1–9. http://dx.doi.org/10.1155/2023/9983781.
Abstract:
Convolutional neural networks (CNNs) have shown their great power in multiple computer vision tasks. However, many recent works improve their performance by adding more layers and parameters, which lead to computational redundancy in many application scenarios, making it harder to implement on low-end devices. To solve this problem, model pruning methods are proposed, which aim to lower the computational and memory requirements of CNNs. In this paper, we propose ScoringNet, a neural network (NN) based pruning criteria for structured pruning procedure. ScoringNet generates a set of scores for each output channel in a model, which is used to reconstruct a pruned model later in a structured pruning way. ScoringNet can also use the gradient information to generate better scores, making the pruned model perform better. By using NNs, there are fewer hyperparameters, making it easier to implement. Experiment results demonstrate that the proposed ScoringNet can outperform or achieve competitive results compared to many state-of-the-art methods in both postpruning and pruning-at-initialization setups.
23
Thodberg, Hans Henrik. "IMPROVING GENERALIZATION OF NEURAL NETWORKS THROUGH PRUNING." International Journal of Neural Systems 01, no. 04 (January 1991): 317–26. http://dx.doi.org/10.1142/s0129065791000352.
Abstract:
A technique for constructing neural network architectures with better ability to generalize is presented under the name Ockham's Razor: several networks are trained and then pruned by removing connections one by one and retraining. The networks which achieve fewest connections generalize best. The method is tested on a classification of bit strings (the contiguity problem): the optimal architecture emerges, resulting in perfect generalization. The internal representation of the network changes substantially during the retraining, and this distinguishes the method from previous pruning studies.
24
Lu, Sheng. "Study on pruning optimization based on HRank pruning method." Applied and Computational Engineering 6, no. 1 (June 14, 2023): 1204–11. http://dx.doi.org/10.54254/2755-2721/6/20230600.
Abstract:
Convolutional neural network (CNN), as one of the most important pillars of deep learning technology, has been paid increased attention to by researchers. However, by the expansion of the application of convolutional neural network, it faces more and more complex problems and more and more diverse situations. In order to solve these problems more appropriately, the number of parameters of convolutional neural networks continuously increases. This limits its potential to be deployed to distal devices with relatively low computing power and memory space. To improve this situation, this paper studies the compression of convolutional neural network model with acceptable sacrifice of accuracy. Based on Pytorch, Resnet-56 on CIFAR-10 is pruned. By using HRank pruning method, convolution kernels of each convolution layers are ordered according to the size of their determinant rank. Through deleting low-rank, less importance convolution kernels and reserving high-rank, more important convolution kernels, the pruning purpose is achieved. By changing the default compression ratio of different convolution layer, more important ones are applied with lower compression rates and less important ones are applied with higher compression rates, the paper achieves higher compression rate with the acceptable loss of accuracy. At the same time, the paper tests the efficiency of pruning under different learning rates. Finally, the paper finds that when the preset compression rate of the intermediate convolutional layer is changed slightly, the accuracy maintains at the level of 92.720% but the compress rate of Params reaches to 44.7% with 0.47M left and the compress rate of Flops reaches 51.1% with 61.39M left. In addition, the paper finds that when the learning rate is 0.05, the learning efficiency reaches its optimum.
25
Jeczmionek, Ernest, and Piotr A. Kowalski. "Flattening Layer Pruning in Convolutional Neural Networks." Symmetry 13, no. 7 (June 27, 2021): 1147. http://dx.doi.org/10.3390/sym13071147.
Abstract:
The rapid growth of performance in the field of neural networks has also increased their sizes. Pruning methods are getting more and more attention in order to overcome the problem of non-impactful parameters and overgrowth of neurons. In this article, the application of Global Sensitivity Analysis (GSA) methods demonstrates the impact of input variables on the model’s output variables. GSA gives the ability to mark out the least meaningful arguments and build reduction algorithms on these. Using several popular datasets, the study shows how different levels of pruning correlate to network accuracy and how levels of reduction negligibly impact accuracy. In doing so, pre- and post-reduction sizes of neural networks are compared. This paper shows how Sobol and FAST methods with common norms can largely decrease the size of a network, while keeping accuracy relatively high. On the basis of the obtained results, it is possible to create a thesis about the asymmetry between the elements removed from the network topology and the quality of the neural network.
26
KAMMA, Koji, Yuki ISODA, Sarimu INOUE, and Toshikazu WADA. "Neural Behavior-Based Approach for Neural Network Pruning." IEICE Transactions on Information and Systems E103.D, no. 5 (May 1, 2020): 1135–43. http://dx.doi.org/10.1587/transinf.2019edp7177.
27
Cheng, Hanjing, Zidong Wang, Lifeng Ma, Xiaohui Liu, and Zhihui Wei. "Multi-task Pruning via Filter Index Sharing: A Many-Objective Optimization Approach." Cognitive Computation 13, no. 4 (June 25, 2021): 1070–84. http://dx.doi.org/10.1007/s12559-021-09894-x.
Abstract:
AbstractState-of-the-art deep neural network plays an increasingly important role in artificial intelligence, while the huge number of parameters in networks brings high memory cost and computational complexity. To solve this problem, filter pruning is widely used for neural network compression and acceleration. However, existing algorithms focus mainly on pruning single model, and few results are available to multi-task pruning that is capable of pruning multi-model and promoting the learning performance. By utilizing the filter sharing technique, this paper aimed to establish a multi-task pruning framework for simultaneously pruning and merging filters in multi-task networks. An optimization problem of selecting the important filters is solved by developing a many-objective optimization algorithm where three criteria are adopted as objectives for the many-objective optimization problem. With the purpose of keeping the network structure, an index matrix is introduced to regulate the information sharing during multi-task training. The proposed multi-task pruning algorithm is quite flexible that can be performed with either adaptive or pre-specified pruning rates. Extensive experiments are performed to verify the applicability and superiority of the proposed method on both single-task and multi-task pruning.
28
Liu, Yu, Yong Wang, Haojin Qi, and Xiaoming Ju. "SuperPruner: Automatic Neural Network Pruning via Super Network." Scientific Programming 2021 (September 13, 2021): 1–11. http://dx.doi.org/10.1155/2021/9971669.
Abstract:
Most network pruning methods rely on rule-of-thumb for human experts to prune the unimportant channels. This is time-consuming and can lead to suboptimal pruning. In this paper, we propose an effective SuperPruner algorithm, which aims to find optimal pruned structure instead of pruning unimportant channels. We first train a VerifyNet, a kind of super network, which is able to roughly evaluate the performance of any given network structure. The particle swarm optimization algorithm is then used to search for optimal network structure. Lastly, the weights in the VerifyNet are used as the initial weights of the optimal pruned structure to make fine-tuning. VerifyNet is a network performance evaluation; our algorithm can quickly prune the network under any hardware constraints. Our algorithm can be applied in multiple fields such as object recognition and semantic segmentation. Extensive experiment results demonstrate the effectiveness of SuperPruner. For example, on CIFAR-10, the pruned VGG16 achieves 93.18% Top-1 accuracy and reduces 74.19% of FLOPs and 89.25% of parameters. Compared with state-of-the-art methods, our algorithm can achieve higher pruned ratio with less accuracy cost.
29
Gou, Longxiang, Ziyi Han, and Zhimeng Yuan. "An analysis of different methods for deep neural network pruning." Applied and Computational Engineering 52, no. 1 (March 27, 2024): 81–86. http://dx.doi.org/10.54254/2755-2721/52/20241292.
Abstract:
Neural network pruning, the process of removing unnecessary weights or neurons from a neural network model, has become an essential technique for reducing computational cost and increasing processing speed, thereby improving overall performance. This article has grouped current pruning methods into three classeschannel pruning, filter pruning, and parameter sparsificationand discussed how each method works. Each approach has its own strengths: channel pruning is particularly useful for reducing model depth and width, filter pruning is more suitable for maintaining model depth while decreasing storage requirements, and parameter sparsification can be applied across various network architectures to achieve both storage and computational efficiency. This work will delve into how each method works and highlight key related works of each category. In the future, it is expected that future research in neural network pruning could focus on developing more sophisticated techniques that can automatically identify important weights or neurons within a network.
30
Ding, Yunlong, and Di-Rong Chen. "Optimization Based Layer-Wise Pruning Threshold Method for Accelerating Convolutional Neural Networks." Mathematics 11, no. 15 (July 27, 2023): 3311. http://dx.doi.org/10.3390/math11153311.
Abstract:
Among various network compression methods, network pruning has developed rapidly due to its superior compression performance. However, the trivial pruning threshold limits the compression performance of pruning. Most conventional pruning threshold methods are based on well-known hard or soft techniques that rely on time-consuming handcrafted tests or domain experience. To mitigate these issues, we propose a simple yet effective general pruning threshold method from an optimization point of view. Specifically, the pruning threshold problem is formulated as a constrained optimization program that minimizes the size of each layer. More importantly, our pruning threshold method together with conventional pruning works achieves a better performance across various pruning scenarios on many advanced benchmarks. Notably, for the L1-norm pruning algorithm with VGG-16, our method achieves higher FLOPs reductions without utilizing time-consuming sensibility analysis. The compression ratio boosts from 34% to 53%, which is a huge improvement. Similar experiments with ResNet-56 reveal that, even for compact networks, our method achieves competitive compression performance even without skipping any sensitive layers.
31
Tessier, Hugo, Vincent Gripon, Mathieu Léonardon, Matthieu Arzel, Thomas Hannagan, and David Bertrand. "Rethinking Weight Decay for Efficient Neural Network Pruning." Journal of Imaging 8, no. 3 (March 4, 2022): 64. http://dx.doi.org/10.3390/jimaging8030064.
Abstract:
Introduced in the late 1980s for generalization purposes, pruning has now become a staple for compressing deep neural networks. Despite many innovations in recent decades, pruning approaches still face core issues that hinder their performance or scalability. Drawing inspiration from early work in the field, and especially the use of weight decay to achieve sparsity, we introduce Selective Weight Decay (SWD), which carries out efficient, continuous pruning throughout training. Our approach, theoretically grounded on Lagrangian smoothing, is versatile and can be applied to multiple tasks, networks, and pruning structures. We show that SWD compares favorably to state-of-the-art approaches, in terms of performance-to-parameters ratio, on the CIFAR-10, Cora, and ImageNet ILSVRC2012 datasets.
32
Gangopadhyay, Briti, Pallab Dasgupta, and Soumyajit Dey. "Safety Aware Neural Pruning for Deep Reinforcement Learning (Student Abstract)." Proceedings of the AAAI Conference on Artificial Intelligence 37, no. 13 (June 26, 2023): 16212–13. http://dx.doi.org/10.1609/aaai.v37i13.26966.
Abstract:
Neural network pruning is a technique of network compression by removing weights of lower importance from an optimized neural network. Often, pruned networks are compared in terms of accuracy, which is realized in terms of rewards for Deep Reinforcement Learning (DRL) networks. However, networks that estimate control actions for safety-critical tasks, must also adhere to safety requirements along with obtaining rewards. We propose a methodology to iteratively refine the weights of a pruned neural network such that we get a sparse high-performance network without significant side effects on safety.
33
Jakob Krzyston, Rajib Bhattacharjea, and Andrew Stark. "Neural network compression with feedback magnitude pruning for automatic modulation classification." ITU Journal on Future and Evolving Technologies 3, no. 2 (July 13, 2022): 157–64. http://dx.doi.org/10.52953/eujf4214.
Abstract:
In the past few years, there have been numerous demonstrations of neural networks outperforming traditional signal processing methods in communications, notably for Automatic Modulation Classification (AMC). Despite the increase in accuracy, these algorithms are notoriously infeasible for integrating into edge computing applications. In this work, we propose an enhanced version of a simple neural network pruning technique, Iterative Magnitude Pruning (IMP), called Feedback Magnitude Pruning (FMP) and demonstrate its effectiveness for the "Lightning-Fast Modulation Classification with Hardware-Effficient Neural Network" 2021 AI for Good: Machine Learning in 5G Challenge hosted by the International Telecommunications Union (ITU) and Xilinx. IMP achieved a compression ratio of 9.313, while our proposed FMP achieved a compression ratio of 831 and normalized cost of 0.0419. Our FMP result was awarded second place, demonstrating the compression and classification accuracy benefits of pruning with feedback.
34
Gong, Wei. "A Neural Networks Pruning and Data Fusion Based Intrusion Detection Model." Applied Mechanics and Materials 651-653 (September 2014): 1772–75. http://dx.doi.org/10.4028/www.scientific.net/amm.651-653.1772.
Abstract:
The abilities of summarization, learning and self-fitting and inner-parallel computing make artificial neural networks suitable for intrusion detection. On the other hand, data fusion based IDS has been used to solve the problem of distorting rate and failing-to-report rate and improve its performance. However, multi-sensor input-data makes the IDS lose its efficiency. The research of neural network based data fusion IDS tries to combine the strong process ability of neural network with the advantages of data fusion IDS. A neural network is designed to realize the data fusion and intrusion analysis and Pruning algorithm of neural networks is used for filtering information from multi-sensors. In the process of intrusion analysis pruning algorithm of neural networks is used for filtering information from multi-sensors so as to increase its performance and save the bandwidth of networks.
35
Lei, Yu, Dayu Wang, Shenghui Yang, Jiao Shi, Dayong Tian, and Lingtong Min. "Network Collaborative Pruning Method for Hyperspectral Image Classification Based on Evolutionary Multi-Task Optimization." Remote Sensing 15, no. 12 (June 13, 2023): 3084. http://dx.doi.org/10.3390/rs15123084.
Abstract:
Neural network models for hyperspectral images classification are complex and therefore difficult to deploy directly onto mobile platforms. Neural network model compression methods can effectively optimize the storage space and inference time of the model while maintaining the accuracy. Although automated pruning methods can avoid designing pruning rules, they face the problem of search efficiency when optimizing complex networks. In this paper, a network collaborative pruning method is proposed for hyperspectral image classification based on evolutionary multi-task optimization. The proposed method allows classification networks to perform the model pruning task on multiple hyperspectral images simultaneously. Knowledge (the important local sparse structure of the network) is automatically searched and updated by using knowledge transfer between different tasks. The self-adaptive knowledge transfer strategy based on historical information and dormancy mechanism is designed to avoid possible negative transfer and unnecessary consumption of computing resources. The pruned networks can achieve high classification accuracy on hyperspectral data with limited labeled samples. Experiments on multiple hyperspectral images show that the proposed method can effectively realize the compression of the network model and the classification of hyperspectral images.
36
Ai, Fang Ju. "An Improved Pruning Algorithm for Fuzzy Neural Network." Applied Mechanics and Materials 411-414 (September 2013): 2031–36. http://dx.doi.org/10.4028/www.scientific.net/amm.411-414.2031.
Abstract:
The number of fuzzy rules directly determines the complexity and efficiency of Fuzzy Neural Network (FNN). The Iterative Pruning (IP) algorithm belongs to the Pruning Method, and it spends much time computing adjusting factors of the remaining weights. So the Improved Iterative Pruning (IIP) algorithm is put forward, which adopts dividing blocks strategy and uses the Generalized Inverse Matrix (GIM) algorithm to replace the Conjugate Gradient Precondition Normal Equation (CGPCNE) algorithm for updating the remaining weights. The IIP algorithm is applied in the rule-reasoning layer of FNN to simplify its rules and structure in a great extent and preserve a good level of accuracy and generalization ability without retraining after pruning. The simulation results demonstrate the effectiveness and the feasibility of the algorithm.
37
Camacho, Jose David, Carlos Villaseñor, Carlos Lopez-Franco, and Nancy Arana-Daniel. "Neuroplasticity-Based Pruning Method for Deep Convolutional Neural Networks." Applied Sciences 12, no. 10 (May 13, 2022): 4945. http://dx.doi.org/10.3390/app12104945.
Abstract:
In this paper, a new pruning strategy based on the neuroplasticity of biological neural networks is presented. The novel pruning algorithm proposed is inspired by the knowledge remapping ability after injuries in the cerebral cortex. Thus, it is proposed to simulate induced injuries into the network by pruning full convolutional layers or entire blocks, assuming that the knowledge from the removed segments of the network may be remapped and compressed during the recovery (retraining) process. To reconnect the remaining segments of the network, a translator block is introduced. The translator is composed of a pooling layer and a convolutional layer. The pooling layer is optional and placed to ensure that the spatial dimension of the feature maps matches across the pruned segments. After that, a convolutional layer (simulating the intact cortex) is placed to ensure that the depth of the feature maps matches and is used to remap the removed knowledge. As a result, lightweight, efficient and accurate sub-networks are created from the base models. Comparison analysis shows that in our approach is not necessary to define a threshold or metric as the criterion to prune the network in contrast to other pruning methods. Instead, only the origin and destination of the prune and reconnection points must be determined for the translator connection.
38
Ge, Yisu, Shufang Lu, and Fei Gao. "Small Network for Lightweight Task in Computer Vision: A Pruning Method Based on Feature Representation." Computational Intelligence and Neuroscience 2021 (April 17, 2021): 1–12. http://dx.doi.org/10.1155/2021/5531023.
Abstract:
Many current convolutional neural networks are hard to meet the practical application requirement because of the enormous network parameters. For accelerating the inference speed of networks, more and more attention has been paid to network compression. Network pruning is one of the most efficient and simplest ways to compress and speed up the networks. In this paper, a pruning algorithm for the lightweight task is proposed, and a pruning strategy based on feature representation is investigated. Different from other pruning approaches, the proposed strategy is guided by the practical task and eliminates the irrelevant filters in the network. After pruning, the network is compacted to a smaller size and is easy to recover accuracy with fine-tuning. The performance of the proposed pruning algorithm is validated on the acknowledged image datasets, and the experimental results prove that the proposed algorithm is more suitable to prune the irrelevant filters for the fine-tuning dataset.
39
Grau, M. Mar Abad, and L. Daniel Hernandez Molinero. "Local Representation Neural Networks for Feature Selection." Journal of Advanced Computational Intelligence and Intelligent Informatics 3, no. 4 (August 20, 1999): 326–31. http://dx.doi.org/10.20965/jaciii.1999.p0326.
Abstract:
Pruning methods for feature selection in neural networks start out from the idea that the representation of the data must evolve from a distributed representation of the information to a more localised representation which will represent the skeleton of the network, needing long training times imposed by the back propagation algorithm. Even the quasi-Newton algorithm spent a long computation time. We propose a three-layer network based on local representation with a step-threshold function and an algorithm called Direct Method for Structural Learning, both allow a very fast pruning of superfluous attributes.
40
Zhang, Chaoyan, Cheng Li, Baolong Guo, and Nannan Liao. "Neural Network Compression via Low Frequency Preference." Remote Sensing 15, no. 12 (June 16, 2023): 3144. http://dx.doi.org/10.3390/rs15123144.
Abstract:
Network pruning has been widely used in model compression techniques, and offers a promising prospect for deploying models on devices with limited resources. Nevertheless, existing pruning methods merely consider the importance of feature maps and filters in the spatial domain. In this paper, we re-consider the model characteristics and propose a novel filter pruning method that corresponds to the human visual system, termed Low Frequency Preference (LFP), in the frequency domain. It is essentially an indicator that determines the importance of a filter based on the relative low-frequency components across channels, which can be intuitively understood as a measurement of the “low-frequency components”. When the feature map of a filter has more low-frequency components than the other feature maps, it is considered more crucial and should be preserved during the pruning process. We conduct the proposed LFP on three different scales of datasets through several models and achieve superior performances. The experimental results obtained on the CIFAR datasets and ImageNet dataset demonstrate that our method significantly reduces the model size and FLOPs. The results on the UC Merced dataset show that our approach is also significant for remote sensing image classification.
41
Bondarenko, Andrey, Arkady Borisov, and Ludmila Alekseeva. "Neurons vs Weights Pruning in Artificial Neural Networks." Environment. Technology. Resources. Proceedings of the International Scientific and Practical Conference 3 (June 16, 2015): 22. http://dx.doi.org/10.17770/etr2015vol3.166.
Abstract:
<p class="R-AbstractKeywords">Artificial neural networks (ANN) are well known for their good classification abilities. Recent advances in deep learning imposed second ANN renaissance. But neural networks possesses some problems like choosing hyper parameters such as neuron layers count and sizes which can greatly influence classification rate. Thus pruning techniques were developed that can reduce network sizes, increase its generalization abilities and overcome overfitting. Pruning approaches, in contrast to growing neural networks approach, assume that sufficiently large ANN is already trained and can be simplified with acceptable classification accuracy loss.</p><p class="R-AbstractKeywords">Current paper compares nodes vs weights pruning algorithms and gives experimental results for pruned networks accuracy rates versus their non-pruned counterparts. We conclude that nodes pruning is more preferable solution, with some sidenotes.</p>
42
Guo, Wenzhe, Hasan Erdem Yantır, Mohammed E. Fouda, Ahmed M. Eltawil, and Khaled Nabil Salama. "Towards Efficient Neuromorphic Hardware: Unsupervised Adaptive Neuron Pruning." Electronics 9, no. 7 (June 27, 2020): 1059. http://dx.doi.org/10.3390/electronics9071059.
Abstract:
To solve real-time challenges, neuromorphic systems generally require deep and complex network structures. Thus, it is crucial to search for effective solutions that can reduce network complexity, improve energy efficiency, and maintain high accuracy. To this end, we propose unsupervised pruning strategies that are focused on pruning neurons while training in spiking neural networks (SNNs) by utilizing network dynamics. The importance of neurons is determined by the fact that neurons that fire more spikes contribute more to network performance. Based on these criteria, we demonstrate that pruning with an adaptive spike count threshold provides a simple and effective approach that can reduce network size significantly and maintain high classification accuracy. The online adaptive pruning shows potential for developing energy-efficient training techniques due to less memory access and less weight-update computation. Furthermore, a parallel digital implementation scheme is proposed to implement spiking neural networks (SNNs) on field programmable gate array (FPGA). Notably, our proposed pruning strategies preserve the dense format of weight matrices, so the implementation architecture remains the same after network compression. The adaptive pruning strategy enables 2.3× reduction in memory size and 2.8× improvement on energy efficiency when 400 neurons are pruned from an 800-neuron network, while the loss of classification accuracy is 1.69%. And the best choice of pruning percentage depends on the trade-off among accuracy, memory, and energy. Therefore, this work offers a promising solution for effective network compression and energy-efficient hardware implementation of neuromorphic systems in real-time applications.
43
Alshahrani, Mona, Othman Soufan, Arturo Magana-Mora, and Vladimir B. Bajic. "DANNP: an efficient artificial neural network pruning tool." PeerJ Computer Science 3 (November 6, 2017): e137. http://dx.doi.org/10.7717/peerj-cs.137.
Abstract:
Background Artificial neural networks (ANNs) are a robust class of machine learning models and are a frequent choice for solving classification problems. However, determining the structure of the ANNs is not trivial as a large number of weights (connection links) may lead to overfitting the training data. Although several ANN pruning algorithms have been proposed for the simplification of ANNs, these algorithms are not able to efficiently cope with intricate ANN structures required for complex classification problems. Methods We developed DANNP, a web-based tool, that implements parallelized versions of several ANN pruning algorithms. The DANNP tool uses a modified version of the Fast Compressed Neural Network software implemented in C++ to considerably enhance the running time of the ANN pruning algorithms we implemented. In addition to the performance evaluation of the pruned ANNs, we systematically compared the set of features that remained in the pruned ANN with those obtained by different state-of-the-art feature selection (FS) methods. Results Although the ANN pruning algorithms are not entirely parallelizable, DANNP was able to speed up the ANN pruning up to eight times on a 32-core machine, compared to the serial implementations. To assess the impact of the ANN pruning by DANNP tool, we used 16 datasets from different domains. In eight out of the 16 datasets, DANNP significantly reduced the number of weights by 70%–99%, while maintaining a competitive or better model performance compared to the unpruned ANN. Finally, we used a naïve Bayes classifier derived with the features selected as a byproduct of the ANN pruning and demonstrated that its accuracy is comparable to those obtained by the classifiers trained with the features selected by several state-of-the-art FS methods. The FS ranking methodology proposed in this study allows the users to identify the most discriminant features of the problem at hand. To the best of our knowledge, DANNP (publicly available at www.cbrc.kaust.edu.sa/dannp) is the only available and on-line accessible tool that provides multiple parallelized ANN pruning options. Datasets and DANNP code can be obtained at www.cbrc.kaust.edu.sa/dannp/data.php and https://doi.org/10.5281/zenodo.1001086.
44
Naeem, Saad, Noreen Jamil, Habib Ullah Khan, and Shah Nazir. "Complexity of Deep Convolutional Neural Networks in Mobile Computing." Complexity 2020 (September 17, 2020): 1–8. http://dx.doi.org/10.1155/2020/3853780.
Abstract:
Neural networks employ massive interconnection of simple computing units called neurons to compute the problems that are highly nonlinear and could not be hard coded into a program. These neural networks are computation-intensive, and training them requires a lot of training data. Each training example requires heavy computations. We look at different ways in which we can reduce the heavy computation requirement and possibly make them work on mobile devices. In this paper, we survey various techniques that can be matched and combined in order to improve the training time of neural networks. Additionally, we also review some extra recommendations to make the process work for mobile devices as well. We finally survey deep compression technique that tries to solve the problem by network pruning, quantization, and encoding the network weights. Deep compression reduces the time required for training the network by first pruning the irrelevant connections, i.e., the pruning stage, which is then followed by quantizing the network weights via choosing centroids for each layer. Finally, at the third stage, it employs Huffman encoding algorithm to deal with the storage issue of the remaining weights.
45
JEARANAITANAKIJ, KIETIKUL, and OUEN PINNGERN. "SPARTAN SIMPLICITY: A PRUNING ALGORITHM FOR NEURAL NETS." Journal of Circuits, Systems and Computers 17, no. 04 (August 2008): 569–96. http://dx.doi.org/10.1142/s0218126608004514.
Abstract:
Having more hidden units than necessary can produce a neural network that has a poor generalization. This paper proposes a new algorithm for pruning unnecessary hidden units away from the single-hidden layer feedforward neural networks, resulting in a Spartan network. Our approach is simple and easy to implement, yet produces a very good result. The idea is to train the network until it begins to lose its generalization. Then the algorithm measures the sensitivity and automatically prunes away the most irrelevant unit. We define this sensitivity as the absolute difference between the desirable output and the output of the pruned network. Unlike other pruning methods, our algorithm is distinct in calculating the sensitivity from the validation set, instead of the training set, without increasing the asymptotic time complexity of the back-propagation algorithm. In addition, for a classification problem, we raise a point that the sensitivities of some well-known pruning algorithms may still underestimate the irrelevance of hidden unit even though the validation set is used in measuring the sensitivity. We resolve this problem by considering the number of misclassified patterns as the main concern. The Spartan simplicity algorithm is applied to three artificial and seven standard benchmarks. In most problems, the algorithm can produce a compact-sized network with high generalization ability in comparison with other pruning algorithms.
46
Huang, Junhao, Weize Sun, and Lei Huang. "Joint Structure and Parameter Optimization of Multiobjective Sparse Neural Network." Neural Computation 33, no. 4 (2021): 1113–43. http://dx.doi.org/10.1162/neco_a_01368.
Abstract:
This work addresses the problem of network pruning and proposes a novel joint training method based on a multiobjective optimization model. Most of the state-of-the-art pruning methods rely on user experience for selecting the sparsity ratio of the weight matrices or tensors, and thus suffer from severe performance reduction with inappropriate user-defined parameters. Moreover, networks might be inferior due to the inefficient connecting architecture search, especially when it is highly sparse. It is revealed in this work that the network model might maintain sparse characteristic in the early stage of the backpropagation (BP) training process, and evolutionary computation-based algorithms can accurately discover the connecting architecture with satisfying network performance. In particular, we establish a multiobjective sparse model for network pruning and propose an efficient approach that combines BP training and two modified multiobjective evolutionary algorithms (MOEAs). The BP algorithm converges quickly, and the two MOEAs can search for the optimal sparse structure and refine the weights, respectively. Experiments are also included to prove the benefits of the proposed algorithm. We show that the proposed method can obtain a desired Pareto front (PF), leading to a better pruning result comparing to the state-of-the-art methods, especially when the network structure is highly sparse.
47
Мельниченко, А. В., and К. А. Здор. "INCORPORATING ATTENTION SCORE TO IMPROVE FORESIGHT PRUNING ON TRANSFORMER MODELS." Visnyk of Zaporizhzhya National University Physical and Mathematical Sciences, no. 2 (December 19, 2023): 22–28. http://dx.doi.org/10.26661/2786-6254-2023-2-03.
Abstract:
With rapid development of technologies and growing number of application of neural networks, the problem of optimization arises. Among other methods to optimize training and inference time, neural network pruning has attracted attention in recent years. The main goal of pruning is to reduce the computational complexity of neural network models while retaining performance metrics on desired level. Among the various approaches to pruning, Single-shot Network Pruning (SNIP) methods was designed as a straightforward and effective approach to optimize number of parameters before training. However, as neural network architectures have evolved, particularly with the growing popularity of transformers, a need to reevaluate traditional pruning methods arises. This paper aims to revisit SNIP pruning method, evaluate its performance on transformer model, and introduce an enhanced version of SNIP, specifically designed for transformer architectures. The paper outlines the mathematical framework of SNIP algorithm, and proposes a modification, based on specifics of transformers models. Transformer models achieved impressive results because of their attention mechanisms for a multitude of tasks such as language modeling, translation, computer vision tasks and many others. The proposed modification takes into account this unique feature and combines this information with traditional loss gradients. Traditional method calculates importance score for weights of the network using only gradients from loss function, in the case of enhanced algorithm. In the enhanced version, the importance score is a composite metric that incorporates not only the gradient from the loss function but also from the attention activations. To evaluate the efficiency of proposed modifications, a series of experiments were conducted on image classification task, using Linformer variation of transformer architectures. The results of experiments demonstrate the efficiency of incorporating attention scores in pruning. Conducted experiments show that model pruned by modified algorithm outperforms model pruned by original SNIP by 34% in validation accuracy, confirming the validity of the improvements introduced.
48
Ameen, Salem, and Sunil Vadera. "Pruning Neural Networks Using Multi-Armed Bandits." Computer Journal 63, no. 7 (September 26, 2019): 1099–108. http://dx.doi.org/10.1093/comjnl/bxz078.
Abstract:
Abstract The successful application of deep learning has led to increasing expectations of their use in embedded systems. This, in turn, has created the need to find ways of reducing the size of neural networks. Decreasing the size of a neural network requires deciding which weights should be removed without compromising accuracy, which is analogous to the kind of problems addressed by multi-armed bandits (MABs). Hence, this paper explores the use of MABs for reducing the number of parameters of a neural network. Different MAB algorithms, namely $\epsilon $-greedy, win-stay, lose-shift, UCB1, KL-UCB, BayesUCB, UGapEb, successive rejects and Thompson sampling are evaluated and their performance compared to existing approaches. The results show that MAB pruning methods, especially those based on UCB, outperform other pruning methods.
49
Cai, Mingzhuo, Yihong Su, Binyu Wang, and Tianyu Zhang. "Research on compression pruning methods based on deep learning." Journal of Physics: Conference Series 2580, no. 1 (September 1, 2023): 012060. http://dx.doi.org/10.1088/1742-6596/2580/1/012060.
Abstract:
Abstract In the past decade or so, deep neural networks have continuously refreshed the best performance in tasks such as computer vision and natural language processing, and have become the most concerned research directions. Although deep neural networks have significant performance, they are difficult to deploy on hardware-constrained embedded and mobile devices due to their huge number of parameters, computational and storage costs. Therefore, in recent years, the lightweight design of network structure has gradually become a cutting-edge and popular direction, improving the operation speed and optimizing the storage space under the premise of maintaining the accuracy of the neural network. In this paper, the results achieved by domestic and foreign scholars in deep learning model compression are sorted and classified, and the methods of network pruning, model quantification, knowledge distillation, and lightweight network design are summarized, and the compression effect of related methods on known public models is summarized. Finally, the possible directions, application directions and development trends of future neural network model lightweight research are prospected.
50
Zhong, Xudong. "Convolutional Neural Network Structure Optimization based on Network Pruning." Highlights in Science, Engineering and Technology 24 (December 27, 2022): 125–30. http://dx.doi.org/10.54097/hset.v24i.3904.
Abstract:
Recently, the number of layers of neural network model is deeper and deeper, the number of parameters is more and more, and the calculation scale is also larger and larger. This improves the use conditions of some excellent models, which is not conducive to the wide application of deep learning methods in more fields. In view of this trend of increasing the size of neural network models, in this paper, we optimize the structure of a convolutional neural network model for image super-resolution, which reduces the size of the model. The model structure optimization method we use is network pruning, which simplifies the number of layers and parameters of the model, improves the effect of the model and reduces the computational consumption of the model. The key insight of network pruning is to remove the relatively redundant and unimportant parts of the network to make the original network sparser and more streamlined. And the simplified model can keep the original performance. The original model used a cascade structure for multiple sampling, resulting in the increase of the scale of the neural network. By removing the redundant sampling structure and retaining only one sampling process, the number of layers of the model is reduced to 1/3 of the original. Under the same data set (BSD300) training, the PSNR (evaluation index of model effect) of the model is improved from 24.471 db to 24.490 db, and the training time is reduced by 13.8% of the original.