Zero-training: Pretrained Foundation Model for Limited Healthcare Tabular Datasets
Main Article Content
Abstract
Tabular data is central to many machine learning applications, as most enterprises use databases to store their data as structured or semi-structured data. While traditional machine learning models such as gradient boosting trees, Random Forest models dominate this domain, pre-trained tabular models like Tabular Prior-data Fitted Network (TabPFN) have shown ability to build models using limited training data or no training data. The Study shows transfer learning can be deployed in tabular data. Implemented models are evaluated using F1-score to ensure the balance between recall and precision. Results show that TabPFN outperforms models from 100 to 10k sample sizes datasets, TabPFN achieved (76.0% F1-score) in 5k sample size. However, as the training size increases, the performance gap decreased. At 10k sample size XGBoost eventually matches TabPFN and then starts outperforming TabPFN in above 10k sample sizes. These findings highlight pretrained foundation models in building machine learning models in domains that lack enough labeled tabular datasets.
Article Details
Section
This work is licensed under a Creative Commons Attribution 4.0 International License.
How to Cite
References
[1] H. O’Brien Quinn, M. Sedky, J. Francis, and M. Streeton, “Literature Review of Explainable Tabular Data Analysis,” Oct. 01, 2024, Multidisciplinary Digital Publishing Institute (MDPI). doi: 10.3390/electronics13193806.
[2] R. Marcinkevičs and J. E. Vogt, “Interpretable and explainable machine learning: A methods-centric overview with concrete examples,” Wiley Interdiscip. Rev. Data Min. Knowl. Discov., vol. 13, no. 3, May 2023, doi: 10.1002/widm.1493.
[3] R. Miotto, F. Wang, S. Wang, X. Jiang, and J. T. Dudley, “Deep learning for healthcare: review, opportunities and challenges,” Brief. Bioinform., vol. 19, no. 6, pp. 1236–1246, Nov. 2018, doi: 10.1093/BIB/BBX044.
[4] N. Hollmann et al., “Accurate predictions on small data with a tabular foundation model,” Nature, vol. 637, no. 8045, pp. 319–326, Jan. 2025, doi: 10.1038/s41586-024-08328-6.
[5] S. Somvanshi, S. Aaqib Javed, G. Antariksa, A. Hossain, and S. Das, “A Survey on Deep Tabular Learning,” 2024. A Survey on Deep Tabular Learning, vol. 1, no. 1, p. 43, 2024, doi: 10.48550/arXiv.2410.12034.
[6] A. A. Adegun, S. Viriri, and J. R. Tapamo, “Review of deep learning methods for remote sensing satellite images classification: experimental survey and comparative analysis,” J. Big Data, vol. 10, no. 1, Dec. 2023, doi: 10.1186/s40537-023-00772-x.
[7] L. Grinsztajn, K. Flöge, O. Key, F. Birkel, P. Jund, B. Roof, B. Jäger, D. Safaric, S. Alessi, A. Hayler, M. Manium, R. Yu, F. Jablonski, S. B. Hoo, A. Garg, J. Robertson, M. Bühler, V. Moroshan, L. Purucker, C. Cornu, L. C. Wehrhahn, A. Bonetto, B. Schölkopf, S. Gambhir, N. Hollmann, and F. Hutter, "TabPFN-2.5: Advancing the state of the art in tabular foundation models," arXiv preprint arXiv:2511.08667, 2025.
[8] S. Sercan¨, S. Arık, and T. Pfister, “TabNet: Attentive Interpretable Tabular Learning,” 2021. [Online]. Available: www.aaai.org
[9] A. Al-Nafjan, A. Aljuhani, A. Alshebel, A. Alharbi, and A. Alshehri, “Artificial Intelligence in Predictive Healthcare: A Systematic Review,” Oct. 01, 2025, Multidisciplinary Digital Publishing Institute (MDPI). doi: 10.3390/jcm14196752.
[10] R. Shwartz-Ziv and A. Armon, "Tabular data: Deep learning is not all you need," Information Fusion, vol. 81, pp. 84–90, 2022, doi: 10.1016/j.inffus.2021.11.011..
[11] H.-J. Ye, S.-Y. Liu, H.-R. Cai, Q.-L. Zhou, and D.-C. Zhan, "A closer look at deep learning methods on tabular datasets," arXiv preprint arXiv:2407.00956, 2024.
[12] M. Y. Ng et al., “Perceptions of Data Set Experts on Important Characteristics of Health Data Sets Ready for Machine Learning: A Qualitative Study,” JAMA Netw. Open, vol. 6, no. 12, p. E2345892, Dec. 2023, doi: 10.1001/jamanetworkopen.2023.45892.
[13] Q. W. Khan, K. Iqbal, R. Ahmad, A. Rizwan, A. N. Khan, and D. H. Kim, “An intelligent diabetes classification and perception framework based on ensemble and deep learning method,” PeerJ Comput. Sci., vol. 10, 2024, doi: 10.7717/peerj-cs.1914.
[14] J. Gao, Y. Lu, N. Ashrafi, I. Domingo, K. Alaei, and M. Pishgar, “Prediction of sepsis mortality in ICU patients using machine learning methods,” BMC Med. Inform. Decis. Mak., vol. 24, no. 1, Dec. 2024, doi: 10.1186/s12911-024-02630-z.
[15] A. Nazir, A. Hussain, M. Singh, and A. Assad, “Deep learning in medicine: advancing healthcare with intelligent solutions and the future of holography imaging in early diagnosis,” Multimed. Tools Appl., vol. 84, no. 17, pp. 17677–17740, May 2025, doi: 10.1007/s11042-024-19694-8.
[16] M. Chaieb, M. Azzouz, M. Ben Refifa, and M. Fraj, “Deep learning-driven prediction in healthcare systems: Applying advanced CNNs for enhanced breast cancer detection,” Comput. Biol. Med., vol. 189, p. 109858, May 2025, doi: 10.1016/j.compbiomed.2025.109858.
[17] K. Avazov et al., “Bridging the Gap Between Accuracy and Efficiency in AI-Based Breast Cancer Diagnosis from Histopathological Data,” Cancers (Basel)., vol. 17, no. 13, Jul. 2025, doi: 10.3390/cancers17132159.
[18] S. Sharma, R. K. Dudeja, G. S. Aujla, R. S. Bali, and N. Kumar, “DeTrAs: deep learning-based healthcare framework for IoT-based assistance of Alzheimer patients,” Neural Comput. Appl., vol. 37, no. 28, pp. 23017–23029, Oct. 2025, doi: 10.1007/s00521-020-05327-2.
[19] A. Alzahrani, “Early Detection of Lung Cancer Using Predictive Modeling Incorporating CTGAN Features and Tree-Based Learning,” IEEE Access, vol. 13, pp. 34321–34333, 2025, doi: 10.1109/ACCESS.2025.3543215.
[20] S. Bang, M. O. Aarvold, W. J. Hartvig, N. O. E. Olsson, and A. Rauzy, “APPLICATION OF MACHINE LEARNING TO LIMITED DATASETS: PREDICTION OF PROJECT SUCCESS.,” Journal of Information Technology in Construction, vol. 27, p. 732, Jan. 2022, doi: 10.36680/J.ITCON.2022.036.
[21] C. Varghese, E. Habermann, K. Hanson, A. Choudhary, H. Salehinejad, and C. Thiels, “Tabular foundation models as a new portable standard in local surgical risk prediction,” Surgery, vol. 192, p. 110078, Apr. 2026, doi: 10.1016/j.surg.2025.110078.
[22] P. García, J. de Curtò, I. de Zarzà, J. C. Cano, and C. T. Calafate, “Foundation Models for Cybersecurity: A Comprehensive Multi-Modal Evaluation of TabPFN and TabICL for Tabular Intrusion Detection,” Electronics (Switzerland), vol. 14, no. 19, Oct. 2025, doi: 10.3390/electronics14193792.