Journal of Instrumentation Technology & Innovations
Automatic Chest X-ray Report Generation Using Machine Learning
Abstract
In this study, a deep neural network is suggested for the automatic creation of precise radiologist reports from chest X-ray pictures. The proposed network responds to the need for medical image captioning by learning to extract key features from the image and creating tag embeddings for each patient's X-ray images. Medical image captioning demands coherence and high accuracy in identifying abnormalities and extracting information. For a finer representation, the network encodes picture and tag features with self-attention using transformers for self and cross-attention. The generated findings are obtained by applying cross-attention between the image and tag features and the input sequence, while the impressions are generated by applying cross-attention between the generated findings and the input sequence. Evaluation using a publicly available dataset shows that the proposed network can generate a readable radiology report with a higher BLEU score compared to the state-of-the-art approach. This study demonstrates the potential of deep neural networks in automating radiology report generation and improving healthcare facilities.References (25)
- Ba, J., Mnih, V., & Kavukcuoglu, K. (2015). Multiple object recognition with visual attention. In International Conference on Learning Representations (ICLR).
- Delrue, L., Gosselin, R., Ilsen, B., Van Landeghem, A., de Mey, J., & Duyck, P. (2011). Difficulties in the interpretation of chest radiography. In Comparative Interpretation of CT and Standard Radiography of the Chest (pp. 27–49). Springer.
- Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2015). Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2), 304 –
- 4. Denkowski, M., & Lavie, A. (2014). Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation (pp. 376–380).
- Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2015). Long -term recu rrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2625–2634).
- Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., Gao, J ., He, X., Mitchell, M., Platt, J. C., et al. (2015). From captions to visual concepts and back. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1473–1482).
- Gong, Y., Jia, Y., Leung, T., Toshev, A., & Ioffe, S. (2013). Deep convolutional ranking for multilabel image annotation. In International Conference on Learning Representations (ICLR).
- Guillaumin, M., Mensink, T., Verbeek, J., & Schmid, C. (2009). Tagprop: Discriminative metric learning in nearest neighbor models for image auto -annotation. In IEEE 12th International Conference on Computer Vision (ICCV) (pp. 309–316).
- Hochreiter, S., & Schmidhuber, J. (1997). Long short -term memory. Neural Computation, 9(8), 1735–1780.
- Karpathy, A., & Fei -Fei, L. (2015). D eep visual -semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 3128–3137).
- Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv pr eprint arXiv:1412.6980.
- Kisilev, P., Walach, E., Barkan, E., Ophir, B., Alpert, S., & Hashoul, S. Y. (2015). From medical image to automatic medical report generation. IBM Journal of Research and Development, 59(2/3).
- Xiaodan Liang, Zhiting Hu, Hao Zhang , Chuang Gan, and Eric P. Xing. (2017). Recurrent topic - transition GAN for visual paragraph generation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). Journal of Instrumentation Technology & Innovations Volume 13, Issue 1 ISSN: 2249-4731 (Online), ISSN: 2347-7261 (Print) © STM Journals 2023. All Rights Reserved 39
- Chin-Yew Lin. (2004). ROUGE: A package for automatic evaluation of summa ries. In Text summarization branches out: Proceedings of the ACL-04 workshop, Volume 8. Barcelona, Spain.
- Pengfei Liu, Xipeng Qiu, Xinchi Chen, Shiyu Wu, and Xuanjing Huang. (2015). Multi -timescale long short-term memory neural network for modeling sentences and documents. In Proceedings of the 2015 conference on empirical methods in natural language processing (EMNLP), pages 2326– 2335.
- Eric Martin and Chris Cundy. (2018). Parallelizing linear recurrent neural nets over sequence length. In International Conference on Learning Representations (ICLR).
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei -Jing Zhu. (2002). BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics (ACL), pages 311 –318. Association for Computational Linguistics.
- Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. (2014). How to construct deep recurrent neural networks. In International Conference on Learning Representations (ICLR).
- Hoo-Chang Shin, Kirk Roberts, Le Lu, Dina Demner -Fushman, Jianhua Yao, and Ronald M Summers. (2016). Learning to read chest x -rays: Recurrent neural cascade model for automated image annotation. In Proceedings of the IEEE Conferenc e on Computer Vision and Pattern Recognition (CVPR), pages 2497–2506.
- K. Simonyan and A. Zisserman. (2014). Very deep convolutional networks for large -scale image recognition. CoRR, abs/1409.1556.
- Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. (2015). CIDEr: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 4566–4575.
- Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. (2015). Show an d tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 3156–3164.
- Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. (2015). Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning (ICML), pages 2048–2057.
- Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. (2016). Image captio ning with semantic attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4651–4659.
- Zizhao Zhang, Yuanpu Xie, Fuyong Xing, Mason McGough, and Lin Yang. (2017). MDNet: A semantically and visually interpre table medical image diagnosis network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6428–6436.