Abstract

Breast tumor segmentation in mammographic images is challenging because of variations in the tumor size, contrast, and shape. This study introduces a Transfer Learning based Ensemble Net (TL-ESMNet), a novel framework that incorporates adaptive segmentation with attention mechanisms within the Ensemble Net (ESM-Net) stage to address these complexities. Utilizing Transfer Learning (PreTL) with pre-trained models, such as ResNet and VGG16, TL-ESMNet extracts robust feature representations that are critical for precise segmentation. The attention mechanism of the framework, which combines spatial and channel attention, dynamically focuses on relevant regions and channels, enhancing its adaptability to small, low-contrast, or irregularly shaped tumors. Segmentation was performed using U-Net or DeepLabV3, leveraging multi-scale feature extraction to ensure accurate tumor boundary delineation. A dynamic fusion mechanism within ESM-Net merges outputs from specialized models based on image characteristics, providing robustness across diverse tumor presentations. TL-ESMNet is rigorously evaluated on the Digital Database for Screening Mammography (DDSM) and INbreast datasets, achieving Dice Coefficients (DCE) of 0.91 and 0.88, respectively, and demonstrating strong performance across additional metrics, including Precision of 0.92 and 0.89, and Recall of 0.89 and 0.86 on DDSM and INbreast datasets, respectively. Attention mechanisms significantly enhance the segmentation accuracy, particularly in complex cases involving small- or low-contrast tumors. This scalable and efficient framework provides a reliable solution for automated breast tumor segmentation, supporting improved clinical decision making in breast cancer diagnosis.