Overview
IIITLBachat is a personal and family finance platform built for multilingual users. It combines voice-first expense entry using Sarvam AI's Hindi-English models, collaborative family expense circles, AI-powered receipt extraction, anomaly detection on spending patterns, and a Gemini-powered finance chatbot — all surfaced through spending heatmaps and budget tracking dashboards.
The Problem
Personal finance apps designed for Indian users often miss voice-first, multilingual expense entry — a key usability requirement when many household finance decisions happen in Hindi. The goal was a platform where expense logging feels as natural as speaking, with AI-assisted extraction for receipts and bills.
My Contribution
Built the full-stack application including voice expense pipeline using Sarvam AI, receipt and PDF extraction with confidence scoring, collaborative expense circle logic, anomaly detection for overspend patterns, Gemini-based finance chatbot, and the Chart.js visualization layer for heatmaps and budget tracking.
Architecture
Voice Expense Entry
Audio input is sent to Sarvam AI's speech-to-text models for Hindi and English. Extracted text is parsed using Gemini to identify amount, category, and merchant — returning a structured expense object with a confidence score.
Receipt & PDF Extraction
Uploaded receipts and PDFs are processed using Gemini's multimodal capabilities. Each extracted line item includes a confidence score. Low-confidence items are flagged for user review before being committed to the ledger.
Family Expense Circles
A circle is a shared expense group. Members can log expenses against the circle, split costs, and view each member's contribution. Aggregation runs at request time from the shared ledger.
Anomaly Detection
Weekly budget baselines are established from historical spending per category. Significant deviations trigger an overspend alert surfaced in the dashboard.
Finance Chatbot
Gemini has read-only access to the user's expense history. The chatbot answers questions about spending patterns, budget status, and category breakdowns in natural language.
Engineering Decisions & Tradeoffs
Sarvam AI for Hindi speech recognition over generic STT
Rationale: Sarvam AI's models are specifically optimized for Indian languages, providing better accuracy on Hindi expense utterances than generic English STT models with Hindi input.
Tradeoff: Dependency on an external API; if Sarvam AI is unavailable, voice entry falls back to manual text entry.
Confidence scoring on extracted items
Rationale: Preventing incorrect auto-committed expenses is critical for a finance application. Confidence scoring allows high-confidence extractions to be auto-committed while routing uncertain items to user review.
Tradeoff: Adds a review step that interrupts the voice-first flow for ambiguous inputs.
Challenges & Solutions
Parsing informal Hindi expense utterances
After Sarvam AI transcribes the audio, Gemini normalizes informal language patterns (e.g., '200 rupay chai ke liye') into structured expense objects with category classification.
Keeping chatbot responses grounded to actual user data
The chatbot receives a structured summary of the user's expense history as context — it does not have database write access and cannot invent spending data.
Evidence & Results
- ›Voice expense entry supporting Hindi and English input
- ›Receipt extraction with per-item confidence scoring
- ›Collaborative family expense circles with split tracking
- ›Overspend anomaly detection against weekly budget baselines
- ›Finance chatbot with read-only access to personal expense history
- ›Note: the app is accessible at https://iiitl-bachat.vercel.app/