Gemini for Enterprise: Building a Multi-Modal Knowledge Base for a Hospital Network

Medical records are the most important data in healthcare — and often the most inaccessible. When a patient arrives at a hospital's emergency room, their records from another branch of the same network can take 15 minutes or more to locate and review. This is the story of how a Vietnamese private hospital network used Gemini 1.5 Pro to unify 50,000+ documents across 12 locations into a searchable knowledge base that doctors query in natural language.

The Data Problem: Four Systems, Zero Interoperability

The hospital group had grown by acquisition over 8 years. Each acquired facility brought its own record-keeping system:

• System A: Scanned PDFs of handwritten admission forms (1990s legacy)
• System B: Digital lab results in HL7 format
• System C: X-ray and imaging files (DICOM) with radiologist annotations
• System D: Modern EHR system for recent outpatient records (2020+)

A cross-location patient lookup required a doctor to check all four systems manually. For complex cases with years of history, this took 15+ minutes — and critical information was routinely missed because the search was incomplete.

The specific challenges:

• Handwritten Vietnamese text (with regional dialect variations and medical abbreviations)
• Mixed document quality — faded faxes, low-resolution scans, inconsistent layouts
• No unified patient identifier across all four systems
• Privacy constraints: data could not leave on-premise infrastructure

Why Gemini, Not a Traditional OCR + Search Pipeline

The conventional approach — OCR → text extraction → search index — fails for medical records because:

1. Vietnamese handwriting is highly variable; generic OCR models struggle with medical abbreviations
2. Extracting meaning from an X-ray requires visual understanding, not just metadata
3. Lab results need to be interpreted in the context of the narrative clinical notes

Gemini 1.5 Pro's native multi-modal understanding processes text, images, and handwriting together in a single inference call. This eliminated the need for separate OCR, NLP, and image analysis pipelines.

Architecture

Document Ingestion → Gemini Processing → Unified Index → Doctor Query Interface

Step 1: Document Ingestion and Classification

import google.generativeai as genai
from pathlib import Path

genai.configure(api_key=GEMINI_API_KEY)
model = genai.GenerativeModel('gemini-1.5-pro')

def process_medical_document(file_path: str, doc_type: str) -> dict:
    """Process any medical document type through Gemini."""
    path = Path(file_path)

    # Load document (PDF, image, or DICOM preview)
    with open(path, 'rb') as f:
        doc_bytes = f.read()

    prompt = f"""
    You are a medical document analyst for a Vietnamese hospital.
    Document type: {doc_type}

    Extract the following from this document:
    1. Patient identifiers (name, DOB, ID number, local patient codes from all systems)
    2. Date of record
    3. Clinical summary (diagnoses, symptoms, key findings)
    4. Medications mentioned
    5. Test results and values (with units and reference ranges if present)
    6. Doctor/specialist notes
    7. Follow-up instructions

    For handwritten content: transcribe verbatim, then provide a cleaned interpretation.
    For X-rays/images: describe key findings in clinical language.

    Return structured JSON. Language: preserve Vietnamese text as-is.
    """

    response = model.generate_content([
        {'mime_type': 'application/pdf', 'data': doc_bytes},
        prompt
    ])

    return parse_medical_json(response.text)

Step 2: Patient Identity Resolution

The hardest problem: a patient called "Nguyen Van An" in System A is "NVA-20451" in System B and "PT-0091234" in System C. Gemini helped resolve this:

def resolve_patient_identity(records: list[dict]) -> dict:
    """Use Gemini to match records belonging to the same patient."""

    prompt = f"""
    These {len(records)} medical records may belong to the same patient.
    Compare name variations (including Vietnamese name ordering differences),
    dates of birth, phone numbers, and any other identifiers.

    Records:
    {json.dumps(records, ensure_ascii=False, indent=2)}

    Group records that definitely belong to the same patient.
    Express confidence as percentage. Flag ambiguous cases for human review.
    Return: {{"groups": [...], "ambiguous": [...]}}
    """

    response = model.generate_content(prompt)
    return json.loads(response.text)

Step 3: Vector Index with Multi-Modal Embeddings

Processed records were indexed using embeddings that capture both textual and visual content:

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

qdrant = QdrantClient(host='localhost', port=6333)

# Create collection with Gemini embedding dimensions
qdrant.create_collection(
    collection_name='medical_records',
    vectors_config=VectorParams(size=768, distance=Distance.COSINE)
)

def index_processed_record(record_id: str, document: dict, original_bytes: bytes):
    """Index a processed medical record with multi-modal embedding."""

    # Create rich text representation for embedding
    text_repr = f"""
    Bệnh nhân: {document['patient']['name']}
    Ngày sinh: {document['patient']['dob']}
    Ngày khám: {document['date']}
    Chẩn đoán: {', '.join(document['diagnoses'])}
    Thuốc: {', '.join(document['medications'])}
    Ghi chú bác sĩ: {document['doctor_notes']}
    Kết quả xét nghiệm: {format_lab_results(document['lab_results'])}
    """

    # Embed with Gemini
    embedding_response = genai.embed_content(
        model='models/embedding-001',
        content=text_repr,
        task_type='retrieval_document'
    )

    qdrant.upsert(
        collection_name='medical_records',
        points=[PointStruct(
            id=record_id,
            vector=embedding_response['embedding'],
            payload={
                'patient_id': document['patient']['unified_id'],
                'date': document['date'],
                'source_system': document['source'],
                'location': document['hospital_location'],
                'diagnoses': document['diagnoses'],
                'summary': document['clinical_summary'],
                'has_imaging': document['has_imaging'],
            }
        )]
    )

Step 4: Natural Language Query Interface

Doctors query the system in Vietnamese natural language:

def doctor_query(query: str, patient_context: dict | None = None) -> dict:
    """Handle a doctor's natural language query across all medical records."""

    # Embed the query
    query_embedding = genai.embed_content(
        model='models/embedding-001',
        content=query,
        task_type='retrieval_query'
    )['embedding']

    # Build filter (patient-specific or global)
    search_filter = None
    if patient_context:
        search_filter = Filter(
            must=[FieldCondition(
                key='patient_id',
                match=MatchValue(value=patient_context['unified_id'])
            )]
        )

    # Retrieve relevant records
    results = qdrant.search(
        collection_name='medical_records',
        query_vector=query_embedding,
        query_filter=search_filter,
        limit=8,
        score_threshold=0.72
    )

    # Generate answer with Gemini
    context = '\n\n'.join([
        f"[{r.payload['date']} - {r.payload['location']}]\n{r.payload['summary']}"
        for r in results
    ])

    answer_prompt = f"""
    Bạn là trợ lý bác sĩ tại hệ thống bệnh viện.
    Câu hỏi của bác sĩ: {query}

    Hồ sơ bệnh án liên quan:
    {context}

    Trả lời ngắn gọn, chính xác, trích dẫn ngày và địa điểm của hồ sơ.
    Nếu không đủ thông tin, nói rõ.
    """

    response = model.generate_content(answer_prompt)
    return {
        'answer': response.text,
        'source_records': [r.payload for r in results],
        'confidence': min(r.score for r in results) if results else 0
    }

Example queries doctors use:

• "Bệnh nhân này có dị ứng thuốc gì không?"
• "Kết quả HbA1c của bệnh nhân từ năm ngoái đến nay?"
• "Có phim X-quang ngực nào trong 6 tháng qua không?"

Results

| Metric | Before | After | |--------|--------|-------| | Diagnosis lookup time | 15+ minutes | 30 seconds | | Documents indexed | 0 (siloed) | 50,000+ | | Cross-location access | Manual request | Instant | | Handwriting OCR accuracy | 60% (generic OCR) | 94% (Gemini) | | Missed cross-reference rate | ~35% | < 3% | | Pipeline complexity | 4 separate systems | 1 unified API |

Key Insights

Gemini's Native Multi-Modal Eliminates Pipeline Complexity

Traditional approaches require separate services: OCR engine → text cleaner → NLP extractor → image classifier → search index. Each step has its own error rate, maintenance burden, and latency. Gemini's native multi-modal understanding processes all document types in one call. This is not a marginal improvement — it fundamentally changes what is buildable with a small team.

Vietnamese Handwriting Needs Domain Prompting

Generic OCR models achieve ~60% accuracy on Vietnamese medical handwriting. Gemini with a well-crafted system prompt (including examples of common medical abbreviations and regional name patterns) achieved 94%. Prompt engineering is the key lever.

Patient Identity Resolution is the Hardest Problem

The technical AI work was straightforward compared to resolving patient identities across four systems with different conventions. Gemini accelerated this — but the team still built a human review queue for ambiguous cases. Do not fully automate identity resolution in healthcare contexts.

On-Premise Deployment is Non-Negotiable for Healthcare

The hospital required all data to remain on-premise. We deployed Gemini via the Vertex AI API with VPC Service Controls, ensuring no patient data left the hospital's infrastructure. Cloud AI does not mean cloud data.

Conclusion

Healthcare AI's promise — instant access to complete patient history — has historically been blocked by data fragmentation and document diversity. Gemini 1.5 Pro's multi-modal understanding makes this achievable without a 3-year data engineering project.

The hospital now has a query system that any doctor can use in natural language, with answers synthesized from 50,000+ records across 12 locations, in under 30 seconds.

Contact Ventra Rocket to discuss AI knowledge base solutions for your healthcare or enterprise organization.