systems / recap-budget
ReCap Budget
A seven-developer startup and a receipt classifier built before function calling
Read the receipt better than anyone, then give advice you can defend.
- Period
- December 2021 – September 2023
- My role
- CEO and Machine Learning Engineer. Hiring and firing, budget, architecture, and the ML.
- State
- Concluded September 2023. $46,000 cash plus about $60,000 in-kind.
- $46KCash raisedlargely via the Stumberg Venture Competition
- ≈$60KIn-kind prizes
- 7Developers ledwith hiring and firing authority
- 100+ categoriesClassifierover 85% reported accuracy
- AWS Lambda
- S3
- Textract
- Comprehend
- FastAPI
- GPT-3
- PyTorch
- Python
Diagram nodes, in flow order: User photo; S3 (object event); CNN gate (is this a receipt?); Textract (forms and lines); SKU lookup (Target / Walmart); Fine-tuned GPT-3 (short name + price → category); Spending ledger; Advice model (fine-tuned).
The problem
Every budgeting app knew you spent $84.12 at H-E-B. None of them knew what you bought. If you could read the line items, you could give advice specific enough to matter.
The pipeline
The app dropped a receipt image into S3. That triggered a Lambda that first ran a small CNN to confirm the image was a receipt, then sent it to Textract for form and line extraction. If the store was Target or Walmart, we resolved the SKU to a category by scraping. Otherwise the line went to our classifier.
The classifier was a fine-tuned GPT-3. We curated 40,000 product name, price, and category records, used Textract to build a thousand-item mapping from receipt abbreviations to full names, ran Comprehend to find brand names and keywords, and trained an SVM to shorten full names into receipt-style names. That gave us a dataset like "Colgate Premium Toothpaste Whitening" → "Colgate Toothpaste" → "CLGT TP + $2.89" → Oral Hygiene. The multiclassifier learned more than a hundred categories from short name and price at over 85% accuracy.
The advice layer
Because we knew spending at that granularity, the advice model could respond to specific choices. We built a framework to populate JSON from LLM output before function calling was a thing, so responses fit our API schema. I did bias-mitigation work on the fine-tuned advice model, because a model that tells people how to spend their money is exactly where bias shows up.
The company
Seven developers, serverless on more than twenty-five AWS services. I hired, fired, raised the money, and wrote the sixty-page business plan using a document-QA tool I built with OPT-1.3B because ChatGPT didn't exist yet. We won the competition.
ReCap Budget, in twenty-five seconds
ReCap Budget read the whole receipt: a CNN gate, Textract, and a GPT-3 classifier fine-tuned on 40,000 items turned 'CLGT TP $2.89' into Oral Hygiene, so the advice could be about what you actually bought.