The Future of Personal Health Assistant
Multimodal Health Data Management Agents
About
Your watch knows your heart rate. Your doctor knows your medications. Your phone knows what you ate yesterday. None of these systems talk to each other, and there is no good way for a patient (or a clinician) to ask a question that spans them. In this course you build one of the agents that could change that: what it needs to know about health data, how it reasons over that data with LLMs, and how to put it in front of a real person without making things worse. Each week you add one piece, and your team ends the term with a working prototype you demo at a poster showcase.
Staff
TAs support weekly labs, office hours, Slack, grading of labs and project milestones, and the final showcase.
Schedule
Before Week 4, students may give an optional 2-minute project pitch to help form teams (up to 1% extra credit).
| Week | Topic | Learning objective | Readings | Slides |
|---|---|---|---|---|
| Week 1 · Sep 11 | Kickoff: Personal Health Assistants & Health Data Foundations | Get the big picture of personal health assistants and why health data is so fragmented, set up your course Fitbit, and start working with your own data. | Deep Medicine (intro) | |
| Week 2 · Sep 18 | Health Data Sources I: Clinical, Self-Report & Consumer Data | Learn how clinical records, wearables, and self-report data are structured, and how FHIR and medical codes let a computer read a patient's record. | ||
| Week 3 · Sep 25 | Health Data Sources II: Sensors, Signals & Derived Metrics | See how raw sensor signals become steps, heart rate, and sleep, and learn why the numbers on your dashboard are estimates, not ground truth. | ||
| Week 4 · Oct 2 | Team Formation + Project Scoping Studio | Team up, shape your project idea, and learn what separates health AI that works in the real world from projects that quietly fail. | ||
| Week 5 · Oct 9 | Data Management I: Relational, Document & Clinical Data Stores | Build databases that hold clinical records and notes, and learn to pull structured facts out of messy medical text. | ||
| Week 6 · Oct 16 | Data Management II: Time-Series, Multimodal Data & Feature Extraction | Handle high-frequency wearable data at scale and turn weeks of raw signal into features your assistant can actually use. | ||
| Week 7 · Oct 23 | Data Management III: Embeddings, Vector Search & RAG | Prompt LLMs for health tasks and see where they go wrong, then close out data management: index your health data for search by meaning, and ground LLM answers in it with retrieval (RAG). Lab time is reserved for catching up on the data-management stack. | ||
| Week 8 · Oct 30 | HCI for Personal Health Assistants: Interface, Trust & Uncertainty | Design how your assistant talks to a real person: showing uncertainty, knowing when to say 'I don't know', and earning trust without overstepping. The design studio produces the storyboard, user flow, and low-fidelity prototype you bring to next week's proposal. | ||
| Week 9 · Nov 6 | Project Proposal Presentations | Pitch and defend your project proposal, bringing something concrete to the table (a prototype, a hand-drawn interface, a system sketch), and sharpen the plan with structured feedback from peers and instructors. | ||
| Week 10 · Nov 13 | Agents I: Tool Use, Function Calling & API Boundaries | Turn a chatbot into an agent that can call tools, query your data, and act, while keeping it safe when it does. | ||
| Week 11 · Nov 20 | Agents II: Planning, Memory & Multi-Step Health Questions | Build agents that break a hard health question into steps and remember context, and learn to debug them when their reasoning goes wrong. | ||
| Week 12 · Nov 27 | Thanksgiving Break (no class) | |||
| Week 13 · Dec 4 | Privacy, Safety, Ethics, Governance & Integration | Cover the privacy, safety, and bias rules around health data, try to break each other's agents, then wire your own system together end to end. | ||
| Week 14 · Dec 11 | Final Showcase | Show off your working personal health assistant with a lightning pitch, a live demo, and a poster. | ||
Who's a fit
This is a project-based course for students who want to build AI systems with real health data constraints. You do not need to be an expert in both computer science and health, but you should bring strength in at least one side and be willing to learn from teammates on the other.
You are likely a good fit if you have at least one of the following backgrounds:
- CS / data systems / AI: You can write Python, use pandas or similar tools, call APIs, and are comfortable debugging code with documentation and AI assistance.
- Health / biomedical informatics / public health / clinical research: You understand health data, clinical workflows, patient-facing or clinician-facing problems, or biomedical research contexts, and have some experience with data analysis in Python or R.
- HCI / design / product / evaluation: You can prototype interfaces, think carefully about user needs, evaluate systems with users, and communicate design tradeoffs.
These are recommendations, not hard requirements. If you have the underlying skills from other coursework, research, or industry experience, just email us with a short note about what you’ve done and we’ll talk.
Interested in taking the course? A short pre-course survey helps us understand your background before registration. Fill it out here. Responses are due Friday, September 11, 11:59 PM ET.
Recommended background
These Columbia courses cover the skills we expect. They are a useful reference, not a checklist. Equivalent skills from elsewhere count just as much.
- COMS 4111 · Introduction to Databases Relational data, SQL, schema design.
- COMS 4701 · Artificial Intelligence Search, planning, basic ML.
- COMS 4705 · Natural Language Processing Tokens, embeddings, transformer basics.
- BINF G4000 · Methods I in Biomedical Informatics Clinical informatics, data sources, terminologies.
- BINF G4002 · Methods II in Biomedical Informatics Standards (FHIR, ICD, RxNorm), clinical workflow.
- PHIN P8105 · Data Science I (Mailman) Reproducible data analysis in R/Python.
- COMS 4170 · User Interface Design Prototyping, usability evaluation, interaction design.
Final project
Each team of 3–4 builds a working Personal Health Assistant. It pulls in data from at least one wearable, self-report, or behavioral source (a synthetic clinical record is optional), stores and retrieves it, reasons over it through an LLM/RAG/agent component, and shows the result through a simple interface a non-technical person could actually use. You will design for safety, privacy, and evaluation as you go, not bolt them on at the end. The term ends with a poster and demo showcase.
The first graded milestone is the Week 9 proposal, and teams are expected to arrive with more than an idea: bring something concrete reviewers can react to — a rough prototype, a hand-drawn interface, a system sketch, or another tangible design artifact. The Week 8 design studio produces exactly these. Milestones, presentation formats, deliverables, and grading are spelled out on the project details page.
Project details: milestones, formats, deliverables & grading →
Grading
| Component | Weight |
|---|---|
| Weekly lab (in-class submission) | 10% |
| Weekly lab (take-home submission) | 10% |
| Project proposal + presentation | 15% |
| Final implementation | 30% |
| Final demo / showcase | 20% |
| Final technical writeup + reflection | 15% |
| Extra credit | +1% |
| Total | 100% (+1%) |
Policies
- AI use
-
You should use LLMs in this course. They are part of the stack you are learning. The labs are written so that pasting a prompt into ChatGPT gets you started but not to the actual answer. You still have to inspect your own data, debug, and decide what is interesting. Each take-home lab submission includes a short reflection: what you tried, what failed, what you learned, and how you used AI tools. Be honest there, cite any AI assistance, and never paste sensitive or personal health data into public tools.
- Privacy of student data
-
You opt in to share your own Fitbit and self-tracking data for class exercises. Synthetic data is always available if you would rather not use your own. Week 1 hands out a course consent form, and you can opt out at any time without penalty.
- Fitbit and devices
-
Each student gets a loaner Fitbit in Week 1 and returns it at the end of the term. There is no out-of-pocket cost. If you already own a compatible wearable you can use that instead.
- Collaboration
-
Weekly lab work is done individually, but you are welcome to collaborate with your teammates as you work through it. The final project is a team of three or four, and we form mixed teams that pair CS strengths with health and clinical strengths. Individual reflections are your own work.
- Accommodations
-
If you need accommodations, contact Disability Services and the instructors during the first week so we can sort things out early.
- Late work
-
Each student gets three “grace days” across the term, no questions asked. Beyond that, late work is graded down 10% per day unless arranged in advance.
