e-ISSN: Applied
editor@ijaida.com | editor.ijaida@gmail.com

AI-Based Concrete Defect Assessment through Prompt Engineering and Multimodal Learning

Abstract

The safety and durability of civil infrastructure depend heavily on the timely detection and assessment of structural defects in concrete elements. Traditional inspection methods rely on manual visual assessment, which is often time-consuming, subjective, labor-intensive, and prone to human error. Recent advancements in Artificial Intelligence (AI), Computer Vision, and Large Language Models (LLMs) have enabled the development of intelligent systems capable of automating structural health monitoring tasks. In particular, Multimodal Large Language Models (MLLMs) have emerged as powerful tools capable of processing and reasoning across multiple data modalities, including images, text, and sensor information. This paper presents a Prompt-Driven Multimodal Large Language Model framework for concrete defect identification. The proposed system combines image-based defect detection with prompt engineering techniques to enable contextual understanding and automated interpretation of structural defects such as cracks, spalling, delamination, honeycombing, corrosion staining, and surface deterioration. The framework integrates image acquisition systems, multimodal feature extraction, prompt generation mechanisms, and large language model reasoning to provide accurate defect classification and maintenance recommendations. The study discusses the architecture, implementation methodology, hardware requirements, experimental evaluation, advantages, limitations, and future research directions. Results indicate that prompt-driven multimodal learning significantly improves defect recognition accuracy, interpretability, and decision support capabilities compared to conventional computer vision approaches.

Keywords

Multimodal Large Language Models, Concrete Defect Detection, Structural Health Monitoring, Prompt Engineering, Computer Vision, Civil Infrastructure, Artificial Intelligence, Crack Detection

Citation of this Article

Monal Sharma, & Deepika Agarwal. (2026). AI-Based Concrete Defect Assessment through Prompt Engineering and Multimodal Learning. International Journal of Artificial Intelligence and Data Analytics, 1(1), 15-24. DOI: https://doi.org/10.47001/IJAIDA/2026.101003

References

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." NAACL-HLT Proceedings, 4171–4186.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2021). "An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale." International Conference on Learning Representations (ICLR).
Brown, T. B., Mann, B., Ryder, N., et al. (2020). "Language Models are Few-Shot Learners." Advances in Neural Information Processing Systems, 33, 1877–1901.
Cha, Y. J., Choi, W., & Büyüköztürk, O. (2017). "Deep Learning-Based Crack Damage Detection Using Convolutional Neural Networks." Computer-Aided Civil and Infrastructure Engineering, 32(5), 361–378.
Dorafshan, S., Thomas, R. J., & Maguire, M. (2018). "Comparison of Deep Convolutional Neural Networks and Edge Detectors for Image-Based Crack Detection." Construction and Building Materials, 186, 1031–1045.
Li, S., Zhao, X., & Zhou, G. (2020). "Automatic Crack Detection and Measurement Based on Deep Learning." Automation in Construction, 115, 103173.
Yeum, C. M., & Dyke, S. J. (2015). "Vision-Based Automated Crack Detection for Bridge Inspection." Computer-Aided Civil and Infrastructure Engineering, 30(10), 759–770.
Heo, S. (2025). "Prompt Driven Multimodal Large Language Models for Concrete Defect Identification." IEEE Access. https://doi.org/10.1109/ACCESS.2025.3605263.
Jin, S., Song, H., Kang, J., Yu, B., & Park, S. (2026). "Concrete Crack Reasoning: Explainable Defect Diagnosis Incorporating Generative Pretrained Transformer 4 and Multimodal Nondestructive Testing Data." Automation in Construction, 181, 106661. https://doi.org/10.1016/j.autcon.2025.106661.
Wang, X., Yue, Q., & Liu, X. (2025). "Crack Image Classification and Information Extraction in Steel Bridges Using Multimodal Large Language Models." Automation in Construction, 105995. https://doi.org/10.1016/j.autcon.2025.105995.
Zhang, H., Pan, X., Asgarinejad, M., & Song, T. (2026). "Multimodal Large Language Model-Driven Framework for Road Crack Assessment." Automation in Construction, 185, 106872. https://doi.org/10.1016/j.autcon.2026.106872.
Mazni, M., Husain, A. R., Shapiai, M. I., Ibrahim, I. S., Zulkifli, R., & Anggara, D. W. (2024). "Identification of Concrete Cracks Using Deep Learning Models: A Systematic Review." Applications of Modelling and Simulation.
König, J., Jenkins, M., Mannion, M., Barrie, P., & Morison, G. (2022). "What's Cracking? A Review and Analysis of Deep Learning Methods for Structural Crack Segmentation, Detection and Quantification." arXiv preprint arXiv:2202.03714.
Ali, R., Chuah, J. H., Talip, M. S. A., Mokhtar, N., & Shoaib, M. A. (2022). "Structural Crack Detection Using Deep Convolutional Neural Networks." Automation in Construction, 133, 103989.
Cha, Y.-J., Choi, W., & Büyüköztürk, O. (2017). "Deep Learning-Based Crack Damage Detection Using Convolutional Neural Networks." Computer-Aided Civil and Infrastructure Engineering, 32(5), 361–378.
Dorafshan, S., Thomas, R. J., & Maguire, M. (2018). "Comparison of Deep Convolutional Neural Networks and Edge Detectors for Image-Based Crack Detection in Concrete." Construction and Building Materials, 186, 1031–1045.
Dung, C. V., & Anh, L. D. (2019). "Autonomous Concrete Crack Detection Using Deep Fully Convolutional Neural Network." Automation in Construction, 99, 52–58.
Li, S., Zhao, X., & Zhou, G. (2019). "Automatic Pixel-Level Multiple Damage Detection of Concrete Structure Using Fully Convolutional Network." Computer-Aided Civil and Infrastructure Engineering, 34(7), 616–634.
Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). "Road Crack Detection Using Deep Convolutional Neural Network." IEEE International Conference on Image Processing (ICIP), 3708–3712.
Wang, K. C. P. (2017). "Deep Learning for Identifying Pavement Cracks." International Journal of Pavement Engineering, 18(10), 1–6.
Li, V. C., & Li, M. (2017). "Exploring the Potential of Artificial Intelligence in Concrete Infrastructure Inspection." Journal of Civil Structural Health Monitoring. Relevant AI-based inspection research.
Kim, B., Cho, S., & others. (2022). "Deep Learning-Based Visual Inspection Approaches for Concrete Infrastructure Damage Detection." Construction and Building Materials.
Liao, Y., Huang, C., & Soliman, A.-H. (2025). "A Review of Concrete Bridge Surface Defect Detection Based on Deep Learning." Optoelectronics Letters, 21(9), 562–576. https://doi.org/10.1007/s11801-025-4116-7.
Wang, J., Wu, W., Niu, H., Shi, B., Wu, N., Hao, X., Zhang, C., & Fu, R. (2025). "Review of Concrete Defect Detection Methods Based on Deep Learning." Computer Science, 52(6A).
Zhang, X., Yang, F., Zhang, Y., & Zhu, Y. (2016). "Road Crack Detection Using Deep Convolutional Neural Networks." IEEE International Conference on Image Processing (ICIP), 3708–3712.
Minaee, S., Boykov, Y., Porikli, F., Plaza, A., Kehtarnavaz, N., & Terzopoulos, D. (2021). "Image Segmentation Using Deep Learning: A Survey." IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7), 3523–3542.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2021). "An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale." International Conference on Learning Representations (ICLR).
Liu, Y., Ott, M., Goyal, N., et al. (2019). "RoBERTa: A Robustly Optimized BERT Pretraining Approach." arXiv preprint arXiv:1907.11692.
Brown, T. B., Mann, B., Ryder, N., et al. (2020). "Language Models Are Few-Shot Learners." Advances in Neural Information Processing Systems, 33, 1877–1901.
OpenAI. (2023). "GPT-4 Technical Report." arXiv preprint arXiv:2303.08774.
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2024). "Visual Instruction Tuning." Advances in Neural Information Processing Systems.
Li, J., Li, D., Savarese, S., & Hoi, S. (2023). "BLIP-2: Bootstrapping Language-Image Pre-Training with Frozen Image Encoders and Large Language Models." Proceedings of the 40th International Conference on Machine Learning, 19730–19742.
License: This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0).