Multi-disease Recognition Method for Fundus Images Based On Text Enhancement
Author:
Affiliation:

1.Southern University of Science and Technology;2.Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences

Funding:

Science and Technology Research Funding of Shenzhen (JSGG20220831105002004)

Ethical statement:

  • Article
  • |
  • Figures
  • |
  • Metrics
  • |
  • Reference
  • |
  • Related
  • |
  • Cited by
  • |
  • Materials
    Abstract:

    In this work, a new paradigm of visual language modeling is introduced in ophthalmic image disease recognition. And a multi-disease recognition algorithm based on a pre-trained model of contrasting language images is proposed. First, a new multi-labeled fundus image dataset MDFCD8 containing 8 categories is constructed based on several publicly available fundus image datasets. Then, the generative artificial intelligence GPT-4 is utilized to generate expert knowledge describing the fine-grained pathological features of fundus images, which solves the problem of the lack of text labels in fundus image datasets. The experimental results showed that, the proposed method outperforms the traditional convolutional neural network and Transformer network by 4.8% and 3.2%, respectively. This study also conducted ablation experiments on each module to validate the effectiveness of the method, and also demonstrated the potential of visual language modeling in ophthalmic disease research.

    Reference
    Related
    Cited by
Get Citation
Share
Article Metrics
  • Abstract:
  • PDF:
  • HTML:
History
  • Received:April 22,2024
  • Revised:April 22,2024
  • Adopted:
  • Online: June 11,2024
  • Published:
Baidu
map