Optimized Deep Learning for Feature Extraction and Detection of Oral Cancer from Histopathological Images
DOI:
https://doi.org/10.31185/wjcms.542Keywords:
Deep learning, Feature Extraction, EfficientNet-B5, DenseNet-201, Whale Optimization Algorithm, Oral Cancer, Histopathological Images.Abstract
This study aims to develop an efficient, powerful deep learning model that accurately classifies oral cancer from high-resolution histopathology images, especially in cases with high intra-class variability and extreme class imbalance. We propose a novel Ensemble Deep Learning framework combined with two streams: CNN1(EfficientNet-B5), CNN2 (DenseNet-201), and a new Hybrid -Spatial-Channel Attention module to extract and fuse the cytological and structural features. We addressed class imbalance using a cost-sensitive learning approach and data augmentation techniques. The Whale Optimization Algorithm (WOA) was used for optimizing hyperparameters. The model is based on late-fusion of CNN1 (fine-grained features) and CNN2 (architectural features). The procedure that was followed was to utilize seven classes of 12425 high-resolution images from (ORCHID and UFES-NDB) and diagnose the seven specific histopathological classes into three primary diagnostic groups: normal, pre-cancer, and cancer. The dataset was divided into Training (80%), Validation (10%), and Testing (10%) to prevent data leakage. It is found that the overall accuracy of the proposed architecture is very robust at 90.35% on the test set. Using five-fold cross-validation. The model achieved the highest Matthews Correlation Coefficient (MCC), particularly in highly imbalanced classes. The proposed method can effectively improve the representation of features and the precision of classification for complex medical image analysis by achieving high metrics in Accuracy, Precision, Recall, Specificity, and F1-Score, which will be a powerful computational tool for automated histopathology analysis. Optimized ensemble learning enhances the detection and classification of Oral Squamous Cell Carcinoma (OSCC) and precancerous lesions by merging fine-grained features and architectural features. Future models should be validated in independent patient populations on external datasets to make it ready for full clinical use and investigated regarding the incorporation of multimodal patient information.
Downloads
References
[1] B. Ilhan, K. Lin, P. Guneri, and P. Wilder-Smith, “Improving Oral Cancer Outcomes with Imaging and Artificial Intelligence,” J. Dent. Res., vol. 99, no. 3, pp. 241–248, 2020, doi: 10.1177/0022034520902128.
[2] N. Hoda, M. Aastha, B. Akshay, A., and S. K., S., “Artificial Intelligence Based Assessment and Application of Imaging Techniques for Early Diagnosis in Oral Cancers,” Int. Surg. J., vol. 11, no. 2, pp. 318–22, 2024, doi: 10.18203/2349-2902.isj20.
[3] R. Dharani and K. Danesh, “Optimized deep learning ensemble for accurate oral cancer detection using CNNs and metaheuristic tuning,” Intell. Med., vol. 11, p. 100258, 2025, doi: 10.1016/j.ibmed.2025.100258.
[4] D. Sezer, Y. Esra, Kavalcı, A. Kemal, and A. Erdinç, “Performance of vision transformer and swin transformer models for lemon quality classification in fruit juice factories,” Eur. Food Res. Technol., vol. 250, no. 9, pp. 2291–2302, 2024, doi: 10.1007/s00217-024-04537-5.
[5] J. Aayush, G. Neha, S. Dilbag, K. Vijay, and K. Manjit, “Classification of the COVID-19 infected patients using DenseNet201 based deep transfer learning,” J. Biomol. Struct. Dyn., vol. 39, no. 15, pp. 5682–5689, 2021, doi: 10.1080/07391102.2020.1788642.
[6] Z. Liang, S. Ting, and D. Zuohua, “A Novel Improved Whale Optimization Algorithm for Global Optimization and Engineering Applications,” Mathematics, vol. 12, no. 5, p. 636, 2024, doi: 10.3390/math12050636.
[7] N. Ali, Fahem, A. Ahmad, Shaker, M. Mohammed, Ibrahim, and R. Khan, Ihtiram, “Feature Fusion for Improved Skin Cancer Diagnosis Using Support Vector Machines,” Wasit J. Comput. Math. Sci., vol. 4, no. 1, pp. 53–58, 2025, doi: 10.31185/wjcms.382.
[8] N. Ahmed, Adil, I. Mohammed, Salah, S. Mustafa, Muslih, A.-K. Kibriya, A. Hiba, Rashid, and A.-A. Mohammed, “Deep Learning Algorithm for Lung Cancer Detection Using EfficientNet-B3,” Wasit J. Comput. Math. Sci., vol. 2, no. 4, pp. 68–76, 2023, doi: 10.31185/wjcms.209.
[9] G. Akilandasowmya, G. Nirmaladevi, S. Suganthi, and A. Aishwariya, “Skin cancer diagnosis: Leveraging deep hidden features and ensemble classifiers for early detection and classification,” Biomed. Signal Process. Control, vol. 88, no. part c, p. 105306, 2024, doi: 10.1016/j.bspc.2023.105306.
[10] R. Dharani Danesh and K. Danesh, “Oral Cancer Segmentation and Identification System Based on Histopathological Images using MaskMeanShiftCNN and SV-OnionNet,” Intell. Med., vol. 10, no. 2, p. 100185, 2024, doi: 10.1016/j.ibmed.
[11] P. Sunil Kumar and R. Harikumar, “Performance analysis of linear layer neural networks for oral cancer classification,” in 2017 6th ICT International Student Project Conference (ICT-ISPC), Johor, Malaysia, 2017, pp. 1–4. doi: 10.1109/ICT-ISPC.2017.8075357.
[12] J. Fahed, A. Omar, M. Dimitrios, A. M. Samara, S. Yusser, and H. Yazan, “A Novel Lightweight Deep Convolutional Neural Network for early detection of oral cancer,” Oral Dis., vol. 28, no. 4, pp. 1123–1130, 2022, doi: 10.1111/odi.13825.
[13] S. Camalan et al., “Convolutional Neural Network-Based Clinical Predictors of Oral Dysplasia: Class Activation Map Analysis of Deep Learning Results,” Cancers (Basel), vol. 13, no. 6, p. 1291, 2021, doi: 10.3390/cancers13061291.
[14] J. Pandia, Rajan and N. Edward, Rajan, Samuel, “Computer-assisted medical image classification for early diagnosis of oral cancer employing deep learning algorithm,” J. Cancer Res. Clin. Oncol., vol. 145, pp. 829–837, 2019, doi: 10.1007/s00432-018-02834-7.
[15] Z. Hu, A. Abeer, M. Paul, P. W. C. Prasad, A. Salih, and A. Elchouemic, “Early stage oral cavity cancer detection: Anisotropic pre-processing and fuzzy C-means segmentation,” in 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA, 2018, pp. 714–719. doi: 10.1109/CCWC.2018.8301673.
[16] G. Yunus, Emre, “Ensemble deep learning framework integrating deep image features and statistical descriptors for robust tumor diagnosis,” Biomed. Signal Process. Control, vol. 118, p. 109645, 2026, doi: 10.1016/j.bspc.2026.109645.
[17] M. Rajakani, R. Kavitha, and B. Kannan, “Metaheuristic Optimization Based Deep Learning Model for Multispectral Image Classification,” Math Oper Res, vol. 3, no. 2, pp. 45–58, 2023, doi: 10.21203/rs.3.rs-2731247/v1.
[18] M. Ali, Ali and M. Mazin, Abed, “Optimized Cancer Subtype Classification and Clustering Using Cat Swarm Optimization and Support Vector Machine Approach for Multi-Omics Data,” J. Soft Comput. Data Min., vol. 5, no. 2, pp. 223–44, 2024, doi: Ali, Ali Mahmoud, and Mazin Abed Mohammed.
[19] A. Mohammed, Mazin and M. Ali, Ali, “Enhanced cancer subclassification using multi-omics clustering and quantum Cat swarm optimization,” Iraqi J. Comput. Sci. Math., vol. 5, no. 3, pp. 552–582, 2024, doi: 10.52866/ijcsm.2024.05.03.035.
[20] T. Zixuan et al., “A literature review of artificial intelligence (AI) for medical image segmentation: from AI and explainable AI to trustworthy AI,” Quant Imaging Med Surg., vol. 14, no. 12, pp. 9620–9652, 2024, doi: 10.21037/qims-24-723.
[21] N. Chaudhary et al., “High-resolution AI image dataset for diagnosing oral submucous fibrosis and squamous cell carcinoma,” Zenodo Data, V1, 2024. https://zenodo.org/records/12636426?utm_
[22] R. Maria, Clara, Falcão et al., “NDB-UFES: An oral cancer and leukoplakia dataset composed of histopathological images and patient data,” Mendeley Data, V4, 2023. https://data.mendeley.com/datasets/bbmmm4wgr8/4
[23] U. Themes, “Oral mucosa: Physiological and physicochemical aspects,” Internet, 2026. https://basicmedicalkey.com/oral-mucosa-physiological-and-physicochemical-aspects/
[24] V. Thakur, S. Jassal, A. Kumar, S. Malik, P. Sharma, and S. Sahi, “A short review on OSMF: oral submucous fibrosis,” J Curr Med Res Opin, vol. 3, no. 9, pp. 619–624, 2020, doi: 10.15520/jcmro.v3i09.337.
[25] S. Ivan, J., “Oral cavity & oropharynx Potentially malignant & dysplasia/ Leukoplakia,” Internet, 2021. https://share.google/ghoCGIhyy9NOEZY2M
[26] D. Olinici et al., “The ultrastructural features of the premalignant oral lesions,” Rom J Morphol Embryol, vol. 59, no. 1, pp. 243–248, 2018, [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/29940634/
[27] M. Pereira, D. Oliveira, G. Landman, and L. Kowalski, “Histologic subtypes of oral squamous cell carcinoma: prognostic relevance,” J Can Dent Assoc, vol. 73, no. 4, pp. 339–44, 2007, [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/17484800/
[28] T. Mingxing and L. Quoc, V., “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” in Proceedings of the 36th International Conference on Machine Learning (ICML), 2019, pp. 6105–6114. doi: 10.48550/arXiv.1905.11946.
[29] P. Padhi and M. M. Das, “Hand gesture recognition using denseNet201- mediapipe hybrid modelling,” in International Conference on Automation, Computing and Renewable Systems (ICACRS), Pudukkottai, India: IEEE, 2022, pp. 995–999. doi: 10. 1109/ICACR S55517. 2022. 10029 038.
[30] A. Abadicio et al., “Ground-level Post-disaster image classification using DenseNet201 for disaster damage assessment,” in 2023 International Conference On Cyber Management And Engineering (CyMaEn), 2023, pp. 132–137. doi: 10.1109/CyMaEn57228.2023.10050981.
[31] L. Maodong, X. Guanghui, L. Qiang, and C. Jie, “A chaotic strategy-based quadratic Opposition-Based Learning adaptive variable-speed whale optimization algorithm,” Math. Comput. Simul., vol. 193, pp. 71–99, 2022, doi: 10.1016/j.matcom.2021.10.003.
[32] D. Chicco and G. Jurman, “The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation,” BMC Genomics, vol. 21, no. 1, p. 6, 2020, doi: 10.1186/s12864-019-6413-7.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Tamara Afif Anai, Mays Afif Anaee, Samah Mohamed
This work is licensed under a Creative Commons Attribution 4.0 International License.
