Medical Intake Agent

Automate patient intake form processing with VLM extraction and local database storage

Source Code: src/gaia/agents/emr/ · src/gaia/agents/emr/dashboard/

Demonstration Application: This is a proof-of-concept demo showcasing AMD Ryzen AI capabilities. Not intended for production use with real patient data. Do not use with actual PHI (Protected Health Information).

The GAIA Medical Intake Agent demonstrates automated patient intake form processing using Vision Language Models (VLM). Drop a scanned intake form into the watch folder, and within seconds the agent extracts patient demographics, insurance information, medical history, and more—storing everything in a searchable SQLite database.

All processing happens 100% locally on AMD Ryzen AI hardware. No cloud APIs, no data leaving your machine—critical for healthcare scenarios where patient privacy is paramount. The agent includes a real-time dashboard for monitoring processing status, viewing patient records, and querying the database using natural language.

Want to learn how it works? See the EMR Agent Playbook for a step-by-step guide to building this agent from scratch.

How It Works

The EMR agent combines three AI models in a sophisticated pipeline that runs entirely on your local hardware:

1. Vision Language Model (VLM) - The Qwen3-VL-4B model “sees” the intake form image and extracts text using a carefully crafted prompt that guides it to identify specific fields (name, DOB, allergies, medications, etc.). Unlike traditional OCR, the VLM understands context—it knows that “DOB” means date of birth and can handle handwritten entries, checkboxes, and varied form layouts.

2. LLM Validation & Querying - The Qwen3-Coder-30B model (a Mixture-of-Experts architecture that activates only 3B parameters per inference) validates extracted data, handles natural language queries, and generates SQL to search the patient database. When you ask “Which patients have penicillin allergies?”, the LLM translates this to proper SQL.

3. Embedding Model - The nomic-embed model creates vector embeddings for semantic similarity search, enabling fuzzy matching when looking up returning patients or finding related records.

Why Local Matters: Running on-device with AMD Ryzen AI means sub-second inference latency, no per-request API costs, and complete data sovereignty. A typical intake form processes in 10-15 seconds on AMD Ryzen AI MAX+ hardware.

Key Features

• Automatic file watching - Monitors a directory for new intake forms
• Drag-and-drop upload - Drop files directly into the Watch Folder panel
• VLM-powered extraction - Uses Qwen3-VL-4B-Instruct for OCR and data extraction
• Local database storage - SQLite with full patient record schema
• New/returning detection - Identifies returning patients and flags changes
• Critical alerts - Automatic alerts for allergies and missing fields
• Web dashboard - Real-time monitoring with SSE updates
• Cumulative efficiency metrics - Track total time saved across all processed forms

Required Models

The EMR agent uses three models, downloaded automatically on first run via gaia-emr init:

Model	Size	Purpose
Qwen3-Coder-30B-A3B-Instruct-GGUF	18.6 GB	LLM for chat queries and patient search
Qwen3-VL-4B-Instruct-GGUF	3.3 GB	Vision language model for form extraction
nomic-embed-text-v2-moe-GGUF	522 MB	Embedding model for similarity search

Disk Space: Ensure you have at least 25 GB of free disk space for model downloads.

Prerequisites

Windows

Installs amd-gaia from PyPI. Recommended for most users on Windows.

Step 1: Create Project Directory

Open PowerShell and run:

mkdir my-emr-project

cd my-emr-project

Step 2: Create Virtual Environment

uv venv .venv --python 3.12

uv will automatically download Python 3.12 if not already installed.

Step 3: Activate the Environment

.\.venv\Scripts\Activate.ps1

If you see a script execution error, run this once:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Then retry the activation command.

You should see (.venv) in your terminal prompt when activated.

Step 4: Install GAIA with EMR Dependencies

uv pip install "amd-gaia[api,rag]"

Step 5: Verify Installation

gaia-emr --help

Linux

Installs amd-gaia from PyPI. Recommended for most users on Linux.

Step 1: Create Project Directory

Open a terminal and run:

mkdir my-emr-project

cd my-emr-project

Step 2: Create Virtual Environment

uv venv .venv --python 3.12

uv will automatically download Python 3.12 if not already installed.

Step 3: Activate the Environment

source .venv/bin/activate

You should see (.venv) in your terminal prompt when activated.

Step 4: Install GAIA with EMR Dependencies

uv pip install "amd-gaia[api,rag]"

Step 5: Verify Installation

gaia-emr --help

Windows (Dev)

For contributors. Clones amd/gaia from GitHub.

Requires: Git for Windows must be installed.

Step 1: Clone the Repository

Open PowerShell and run:

git clone https://github.com/amd/gaia.git

cd gaia

Step 2: Create Virtual Environment

uv venv .venv --python 3.12

Step 3: Activate the Environment

.\.venv\Scripts\Activate.ps1

If you see a script execution error, run this once:

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Then retry the activation command.

You should see (.venv) in your prompt when activated.

Step 4: Install in Editable Mode

uv pip install -e ".[dev,api,rag]"

Step 5: Verify Installation

gaia-emr --help

Changes to GAIA source code will take effect immediately without reinstalling.

Linux (Dev)

For contributors. Clones amd/gaia from GitHub.

Requires: Git must be installed (sudo apt install git).

Step 1: Clone the Repository

Open a terminal and run:

git clone https://github.com/amd/gaia.git

cd gaia

Step 2: Create Virtual Environment

uv venv .venv --python 3.12

Step 3: Activate the Environment

source .venv/bin/activate

You should see (.venv) in your prompt when activated.

Step 4: Install in Editable Mode

uv pip install -e ".[dev,api,rag]"

Step 5: Verify Installation

gaia-emr --help

Changes to GAIA source code will take effect immediately without reinstalling.

The api extra provides FastAPI and uvicorn for the web dashboard. The rag extra provides PyMuPDF for PDF processing.

Having issues? Check the Troubleshooting guide, create an issue on GitHub, or contact us at [e-mail].

---

Quick Start

First time here? Complete the Setup guide first to install Lemonade Server and uv.

Step 1: Initialize (First Time Only)

Download and load all required models before first use:

gaia-emr init

This command:

• Checks Lemonade server is running and context size is configured
• Downloads and loads all required models:
  • VLM: Qwen3-VL-4B-Instruct-GGUF (form extraction)
  • LLM: Qwen3-Coder-30B-A3B-Instruct-GGUF (chat/query processing)
  • Embedding: nomic-embed-text-v2-moe-GGUF (similarity search)
• Verifies all models are loaded and ready

Context Size: For best results, set Lemonade context size to 32768. Right-click the Lemonade tray icon → Settings → Context Size → 32768.

Partial Success: If the LLM fails to download but VLM succeeds, form extraction will still work. Chat queries and natural language patient search require the LLM. Run gaia-emr init again to retry failed downloads.

Step 2: Launch

CLI App

Download sample forms

Download sample intake forms from GitHub and save them to a local directory (e.g., ./intake-forms/).

Dev Install: Sample forms are already included in the repository at data/img/intake-forms/.

Start watching for forms

gaia-emr watch --watch-dir ./intake-forms

The agent will process the sample forms and display extracted patient data.

Example Output

+-----------------------------------+
| Medical Intake Agent              |
| Automatic Patient Form Processing |
+-----------------------------------+
  Watch folder: ./intake-forms
  Database:     ./data/patients.db

  File            Size      Hash          Status
  --------------  --------  ------------  ------
  IMG_2992.jpg    1.7 MB    eaabe23e...   new
  IMG_2993.jpg    887.6 KB  03f2391e...   new
  IMG_2995.jpg    2.3 MB    348071a9...   new
  IMG_2996.jpg    2.1 MB    7a73ea84...   new

Processing Pipeline (7 steps)
  File: IMG_2992.jpg
  [5/7] Extracting patient data
  VLM extracting from image...
  Extracted 1844 chars in 14.12s
Pipeline complete in 14.3s -> Alice Williams

Extracted Fields
  Identity
    first_name           Alice
    last_name            Williams
    date_of_birth        1980-04-04
  Contact
    phone                [phone]
    email                [e-mail]
  Insurance
    insurance_provider   Medicaid Demo
  ...

34 fields extracted

Query patients

Once processing completes, you can query the database using natural language. The agent uses tool calling to translate your questions into SQL queries and retrieve results from the SQLite database.

Which patients have allergies?

Show me all patients processed today

Summarize today's intake forms

Type quit or press Ctrl+C to stop.

CLI Commands Reference

init

gaia-emr init

Command	Description
init	Download required models
watch	Watch folder and process forms
dashboard	Launch web dashboard
query	One-shot database query
reset	Delete database and start fresh
-h	Full command reference

Desktop App

The recommended way to use the EMR Dashboard is via the native Electron desktop app.

Launch the desktop app

gaia-emr dashboard

The dashboard opens automatically in a native desktop window.

First run will install Electron dependencies automatically. The default database is ./data/patients.db.

Add intake forms

Drag and drop intake form images or PDFs directly onto the Watch Folder panel in the dashboard. The agent will process them automatically.

Download sample intake forms from GitHub to test the agent.

Dev Install: Build the frontend (first time only)

If you installed from source (Windows/Linux Dev tabs), you need to build the frontend first:

cd src/gaia/agents/emr/dashboard/frontend

npm install

npm run build

Then return to the repository root:

cd ../../../../../..

Dashboard Features

The dashboard includes four main views:

• Dashboard - Real-time stats, cumulative efficiency metrics, and live processing feed
• Patient Database - Searchable patient list with detailed records
• Chat - Natural language queries about patients
• Settings - Configure watch directory, upload files, and manage database

Watch Folder Panel: The left column displays a Watch Folder panel with status indicators:

• Green dot - Processed files
• Red flashing dot - Currently processing
• Orange dot - Queued for processing

Drag-and-Drop: Drop intake form images or PDFs directly onto the Watch Folder panel to upload and process them instantly.

Command Options

default

gaia-emr dashboard

Option	Default	Description
--watch-dir	./intake_forms	Directory to monitor for intake forms
--db	./data/patients.db	SQLite database path
--port	8080	Server port
--browser	off	Open in browser instead of Electron
--no-open	off	Don’t auto-open, server only

If Electron/Node.js is not available, the dashboard automatically falls back to opening in your default web browser.

---

Supported Intake Form Formats

The agent accepts scanned or photographed intake forms in these formats:

Extension	Processing
.png, .jpg, .jpeg	Direct image processing
.pdf	Converted to image via PyMuPDF
.tiff, .bmp	Direct image processing

---

Under the Hood

Image Preprocessing Pipeline

Before reaching the VLM, intake forms go through an optimization pipeline:

1. EXIF Orientation - Auto-rotates images based on camera metadata (critical for phone photos)
2. Resolution Scaling - Resizes to max 1024px while preserving aspect ratio (balances quality vs. token count)
3. JPEG Compression - Reduces file size with quality=85 for faster transmission to the model
4. Token Estimation - Calculates expected image tokens to verify they fit within context window

Returning Patient Detection

The agent uses a multi-signal approach to identify returning patients:

• Exact Match - Name + DOB combination lookup in SQLite
• Fuzzy Match - Levenshtein distance for misspellings (“Jon Smith” → “John Smith”)
• Embedding Similarity - Vector search using nomic-embed for semantic matching

When a returning patient is detected, the agent highlights any changes from their previous record (new allergies, updated insurance, etc.).

Real-Time Dashboard Architecture

The dashboard uses Server-Sent Events (SSE) for real-time updates without polling:

• FastAPI backend streams processing events as they occur
• React frontend subscribes to /api/events endpoint
• Sub-100ms latency from file detection to UI update
• No WebSocket complexity—SSE is simpler and works through proxies

Tool Calling for Database Queries

When you ask a natural language question, the LLM uses function calling (tool use) to interact with the database:

This pattern ensures the LLM never directly writes SQL—it calls predefined, validated tools that safely construct queries.

---

Troubleshooting

PyMuPDF Required

ERROR: PyMuPDF required for PDF processing

Solution: pip install "amd-gaia[rag]"

JSON Parse Failed

WARNING: Failed to parse extraction for: form.jpg

Solution: Check image quality and form layout clarity.

Database Locked

ERROR: database is locked

Solution: Only one agent should access the database at a time.

Slow First Run

First run downloads models (LLM, VLM, embeddings) which may take several minutes. Subsequent runs start immediately.

Model Download Failed

Failed to download file: [model files]
Download succeeded but failed to rename file: The process cannot access the file

Solution: Run gaia-emr init again to resume the download. If the error persists:

1. Close any applications that may be using the model files
2. Delete partial downloads in Lemonade’s model cache directory
3. Run gaia-emr init again

Context Size Too Small

Context size too small! Image requires 4203 tokens but model context is only 4096.

Solution: Increase Lemonade context size:

1. Right-click Lemonade tray icon → Settings
2. Set Context Size to 32768
3. Restart the model

Corrupted Model / Init Fails

If gaia-emr init fails repeatedly or the agent won’t start due to model errors:

Solution: Delete the corrupted model from Lemonade’s cache and restart:

1. Close Lemonade Server (right-click tray icon → Exit)
2. Navigate to the model cache directory:
   • Windows: %LOCALAPPDATA%\AMD\LemonadeModels\
   • Linux: ~/.local/share/lemonade/models/
3. Delete the corrupted model folder (e.g., Qwen3-VL-4B-Instruct-GGUF/)
4. Restart Lemonade Server
5. Run gaia-emr init again to re-download

---

Learn More

Part 1: Getting Started

Build this agent from scratch and understand the core components

Part 2: Dashboard & API

Deep dive into the web dashboard and REST API endpoints

Part 3: Architecture

Database schema, processing pipeline, and system design

---