Abstract—Reliable malware family classification remains a major cybersecurity challenge, particularly for malware families that exhibit high intra-class variability and strong visual similarity to other families. Although Convolutional Neural Networks (CNNs) based on image representations of malware have demonstrated strong performance, they often struggle to capture complex feature dependencies between overlapping and visually similar classes. This paper proposes the Adaptive Hybrid Residual Attention Block (A-HRAB), an attention module integrated into residual convolutional blocks to enhance feature representation. The proposed module computes channel and spatial attention maps and combines them through a learnable adaptive fusion mechanism that dynamically balances their relative contributions based on the input features. Experiments conducted on the Malimg and MaleVis datasets demonstrate the effectiveness of the proposed approach. On the Malimg dataset, the proposed CNN with A-HRAB achieves 99.4% accuracy and a Macro-F1 score of 98.6%, wheras on the MaleVis dataset, it attains 96.1% accuracy and a Macro-F1 score of 96.0%. Furthermore, the proposed model consistently outperforms the baseline CNN and conventional attention modules, including Squeeze-and-Excitation (SE) and Convolutional Block Attention Module (CBAM). A class-wise analysis further confirms substantial performance improvements for challenging malware families characterized by high intra-class variability and strong inter-class similarity.
 
Keywords—malware classification, image-based malware analysis, Convolutional Neural Network (CNN), attention mechanisms, deep learning for cybersecurity
 
Copyright © 2026 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).