Multi-Modal Vision-Language Model for Cancer Diagnosis
Cross-modal pipeline correlating high-dimensional medical imaging with unstructured biomedical literature, with a Grad-CAM + Gradio explainability interface for clinical interpretability.
Pipeline
The problem
Tumour assessment reads imaging and clinical narrative separately, and single-modality deep models return a label without showing which region of the scan drove it — which clinicians cannot act on.
Architecture
End-to-end flow, stage by stage.
- 01
MRI imaging
- 02
Vision Transformer
- 03
BioBERT fusion
- 04
Grad-CAM
- 05
Gradio UI
How it works
Single-modality baselines first
Separate notebooks establish reference points: BioBERT and ClinicalBERT on the clinical report corpus, and a Vision Transformer on the MRI images, so cross-modal gains can be attributed rather than assumed.
Cross-modal fusion
Image features and clinical text representations — including synthetic text experiments — are combined in a cross-modal training notebook, with dedicated inference and evaluation notebooks reporting accuracy, F1 and AUROC on cleaned train/validation splits.
Explainability in the loop
Grad-CAM heatmaps are surfaced through a Gradio interface so a reviewer can see the image regions behind each prediction instead of trusting a bare score, reaching 80% validation accuracy on the MRI tumour task.
Reproducible data contract
Raw and cleaned CSVs are kept side by side with an explicit input/output contract, so notebook runs are repeatable and the pipeline extends to other cancer types, modalities and text sources.
- Fused Vision Transformer features from MRI scans with BioBERT and Clinical BERT representations of clinical text to support cross-modal diagnostic reasoning.
- Reached 80% validation accuracy on MRI tumor data and exposed Grad-CAM heatmaps through a Gradio interface so clinicians could inspect the image regions influencing each prediction.
- Designed the inference pipeline to extend across additional cancer types, imaging modalities, and biomedical text sources rather than remain tied to one dataset.
By the numbers
From text-only to explainable cross-modal
Validation accuracy (%)
higher is betterModalities used
higher is betterModel families benchmarked
What it ships with
- BioBERT and ClinicalBERT text baselines
- Vision Transformer imaging baseline
- Cross-modal training and inference notebooks
- Grad-CAM explainability via Gradio
- Cleaned, versioned train/validation metadata
- Text-only reports→80%
- Validation accuracy on MRI data
- Black-box→Grad-CAM
- Per-prediction explainability
Stack
More projects
Agentic AI
Voyager AI
Agentic travel planning platform on Next.js and Vercel with topic scoping, hallucination prevention, and structured itinerary output across the full agent workflow.
Agentic AI
Zone Scout
Multi-agent lead intelligence system with an automated enrichment layer over Google Maps Places API — verified contacts, reviews, business metadata, and social profiles per validated lead.