Dec 2025 – Feb 2026
RACE
RAG-Optimized Clinical Reasoning Engine
Clinical language models hallucinate facts that can endanger patients, and the models capable of real reasoning are too large to run anywhere near the point of care.
I take models from research notebooks to constrained hardware — and I engineer against hallucination as a first-class risk, not an afterthought.
Pipeline
End-to-end flow, stage by stage.
- 01
Ingest
- 02
Chunk
- 03
Embed
- 04
Retrieve Top-K
- 05
Reason
- 06
Cite evidence
How it works
Retrieval layer grounds every answer
Medical knowledge bases and clinical guidelines are chunked and embedded with sentence-transformers (all-MiniLM-L6) into a persistent ChromaDB collection. Each query runs a semantic top-K search first, so generation always starts from retrieved evidence rather than model memory, and the returned passages are shown alongside the answer.
Domain fine-tuning for clinical reasoning
Llama-3-8B is supervised fine-tuned on a medical reasoning SFT dataset using QLoRA adapters (rank 16) through TRL's SFTTrainer. Only about 0.1% of parameters — roughly 8M of 8B — are trained, which keeps the run affordable while teaching the model step-by-step clinical chain-of-thought instead of one-line verdicts.
Memory-efficient deployment on consumer GPUs
4-bit NF4 quantization through bitsandbytes cuts the footprint from roughly 32GB to 5.5GB, and gradient checkpointing plus paged optimizers prevent out-of-memory failures during training. The result runs inference on an 8GB card such as an RTX 3050 or T4 rather than requiring A100-class hardware.
Interactive evidence-first interface
A Streamlit front end holds the quantized model in cache, so the first query pays the 30–60s load cost once and follow-up questions return in roughly 5–10s with retrieval completing in under a second.
Engineering decisions
- Fine-tuned Llama-3-8B with QLoRA (rank-16 adapters on q_proj/v_proj — 8.4M trainable parameters, 0.1% of the model) on chain-of-thought medical reasoning data.
- Applied 4-bit NF4 quantization with double quantization via bitsandbytes, compressing the model from 32 GB to 5.5 GB for consumer 8 GB VRAM GPUs.
- Grounded every answer in retrieved evidence: RecursiveCharacterTextSplitter chunking, all-MiniLM-L6-v2 embeddings (384-dim), persisted in ChromaDB with Top-K similarity search.
- Kept ingestion, training, and inference as separate modules, served through a Streamlit interface with cached model loading.
- Prevented OOM during training with gradient checkpointing and the paged_adamw_8bit optimizer.
By the numbers
Efficiency of the compound design
Quantization and adapter training against the full-precision, fully fine-tuned baseline.
Model footprint (GB)
lower is betterTrained parameters (% of model)
lower is betterQuery latency, warm model (s)
lower is betterMinimum VRAM to serve (GB)
lower is betterWhat it ships with
- Evidence passages returned with every generated answer
- QLoRA adapter training pipeline included in the repo
- 4-bit quantized inference for 8GB VRAM machines
- Persistent ChromaDB vector store for repeat sessions
- Chain-of-thought clinical reasoning prompts
- Streamlit UI with cached model loading
Outcomes
Before → After- 32 GB→5.5 GB
- Compressed model size
- ~3–5s→<1s
- Evidence retrieval latency
- 24 GB+→8 GB
- VRAM deployment target met
- 100%→0.1%
- Parameters trained via QLoRA