AI Lab · AI product · Usability pilot · 2026 · 7 min read
Problem statement
Check whether your experience supports each resume claim.
Problem
Job seekers want clear, relevant resume bullets that represent their experience accurately and remain defensible when an interviewer asks how the work was done.
Issue
AI rewrites can introduce unsupported responsibilities or numbers. A passing technical test also does not establish that users can complete the resume workflow successfully.
Solution
Resume Fit Copilot connects an evidence bank to a line-by-line claim checker. A pilot with six AI test users identified four blockers and pointed the product toward a checker-first workflow.
Progress Six AI test users identified four blockers and favored the checker.
Problem
The need comes from desk research, not interviews. No real user has been interviewed yet.
Users
The personas are proto-personas built from sources, checked against research. They are not research participants.
| Persona | Job to be done | What makes them leave |
|---|---|---|
| Ravi, 34, laid off after 8 years in IT services (primary) | "Get me bullets I can send tonight and defend in the interview." | 15 to 20 minutes spent with nothing to send |
| Meghna, 29, business analyst doing an MBA (primary) | "Make my bullets read like a decision maker's, without lying." | It looks amateur, loses her work, or contradicts her own resume |
| Farhan, 38, applying to jobs in Dubai (backup) | "Tell me if my CV matches this job, and give me one I can send." | It feels built for IT people, or like a trick |
Product
| Feature | What it does |
|---|---|
| Evidence bank | One fact per line with a stable ID. Client names tagged once and always hidden |
| Requirement map | Each job requirement is Covered, Likely (confirm), Weak, Missing or Check yourself, judged one evidence line at a time |
| Gap coach | For each gap, your nearest experience and three questions. The app refuses to use a fill-in line until you complete it |
| Cited bullets | One bullet per evidence line, checked by rules, retried if it fails, reviewed by you |
| Check AI text | Paste bullets from any AI. Each is matched to your evidence and cited or flagged, with a safer version |
Built on Python, LangChain retrievers (word and local-embedding meaning search, fused) and a rule-based verifier, with a Streamlit interface. Offline results looked strong: the verifier accepted 0 of 30 unsupported bullets, and the checker flagged 11 of 11 invented bullets. But those cases are synthetic and labelled by the AI that wrote the code, so they are not real-world accuracy.
Pilot method
I passed a 14-case scripted test run, so I wanted to see what happens when someone who does not know the app uses it. Six AI agents, two per persona, each played a person who had just arrived from a friend's link. They could not read the code or docs, only look at the screen, and had to act with the persona's patience, reading speed and device. Each ran a 5-second test on the landing page, then tried five tasks, from loading a resume to checking AI-written bullets.
The inputs were messy on purpose: a resume pasted as people copy it from a PDF, job posts written like real postings, and AI bullets with known overclaims planted. Limits: the agents share a model family with the builder, they knew they were in a study, and minutes are estimates. The pilot finds usability problems and defects. It cannot prove demand.
Results
| Measure | Result |
|---|---|
| Left with bullets they would send | 0 of 6 |
| Planted overclaims caught by "Check AI text" | All of them, in all 6 sessions |
| Named the checker the best part | 6 of 6 |
| Would pay Rs 499 for the app as it was | 0 of 6. Five would come back, for the checker only |
| Blocker defects in the core path | 4, all reproduced by hand |
This is why the scripted run missed them: it used a clean demo bank and approved every bullet in one pass. Real resumes are not clean.
Findings
The checker itself had one flaw that cost trust: it flagged a user's own true sentence because the import had split cause and result into two lines.
Decisions and trade-offs
A checker first, an organiser second, the writer frozen
Why: The pilot proved the checker was the one thing people valued, and it works alongside ChatGPT instead of competing with it.
Trade-off: Fixing the writer in a Release 2.1 nobody had asked for
An LLM judges with a quoted reason; hard rules can only make the verdict stricter
Why: An LLM can be talked into "close enough". For numbers, named tools, ownership verbs and client names, the answer must be provable.
Trade-off: Rules alone, which missed nuance in every session
Two scores shown apart: Defensibility and Job fit
Why: A high job fit built on claims you cannot defend is exactly the harm this product exists to prevent, so the two are never combined.
Trade-off: One number, or no score
Compare the resume against an older resume or one-tap answers ("Did you lead the team of 5?")
Why: A resume cannot prove itself, and a blank evidence bank is where users gave up.
Trade-off: Requiring an evidence bank up front
Business case
Zero of six would pay for the product as it was. That is a statement about visible value, not willingness to pay, because the testers were AI agents.
Feedback loop
What I learned