AI bias simulation: when a credit model treats women unfairly
The model never sees sex, yet women get lower limits. You have four decisions and incomplete evidence. Find the cause before you choose the fix.
- Five kinds of evidence, three possible root causes, one hidden
- Contain the harm without stopping the business
- Decide who to tell, and when, before a journalist calls
- Fairness measures explained in plain words as you play
Not sure yet? Try the free 3-round Price War first: no sign-up, about five minutes.
Is there a simulation that teaches what to do when an AI model turns out to be biased?
Yes. Model Gone Wrong is a 20-minute Bodhih Arena simulation in which a credit-limit model appears to give women lower limits than similar men. You choose the evidence, contain the harm, decide who to tell and when, and pick a fix while the root cause stays hidden. A scored debrief explains the result.
What is Model Gone Wrong?
Model Gone Wrong is a single-player responsible AI simulation from Bodhih Arena. You are Head of Model Risk at an invented bank, Larkfield Bank, whose machine-learning model, LimitIQ, sets credit card limits for about 30,000 new customers a month. A fairness check shows women getting lower limits. Over four steps you assess, contain, disclose and fix, without knowing whether the cause is a proxy input, a data fault or biased history. Scores cover harm, transparency, continuity and the fix.
Model Gone Wrong at a glance
| Format | Micro-sim, single player |
|---|---|
| Length | About 20 minutes per play |
| Players | One person; every play is a new seeded scenario |
| Devices | Any modern browser on a laptop, tablet or phone |
| Scoring | Scored on the server out of 100, with sub-scores and a debrief |
| Credential | Open Badges 3.0 micro-sim credential at a score of 80 or more, with a public verification page |
| Competencies | Decision Making (SPN.4.3), Stakeholder Management (SPN.2.2), Commercial Acumen (SPN.4.5), Data Literacy (SPN.6.2) |
| LMS | LTI 1.3 launch from Canvas, Moodle, Blackboard or D2L, with grade passback; usable as a Bodhih pathway stage |
| Price for one person | $12 |
| Price per seat | $12 per seat, volume pricing by seat count |
“We do not use sex in the model” is not the end of the conversation
Bias hides in inputs that look neutral
Leaders are told a model is fair because protected fields were removed. Then outcomes by group tell a different story, and nobody is sure why.
The first fix people reach for is often the wrong one
Retraining, adjusting thresholds or repairing data each solve a different problem. Pick one without a diagnosis and the gap stays, or a new legal problem appears.
Disclosure is a judgement call made under pressure
Tell the board and regulator now, with half the facts, or wait for a complete story? Teams rarely practise this until it is real.
Here is the situation you walk into
It is Monday morning. Dana Pryce, a data scientist in your model risk team, sends you her quarterly fairness check of LimitIQ, the model that sets card limits at Larkfield Bank. Women’s average limit is noticeably lower than men’s. She is careful: women in the data earn less on average, so this is not yet a like-for-like comparison.
Your analysts can run three checks this week. A matched comparison, an input check, a data pipeline audit, a review of the training history or a look at complaints. Each shows only what it can see. Meanwhile the model keeps setting limits for new customers every day.
Next you decide whether to pause the model, keep it running with human review, lift women’s limits by the gap, or carry on. Then who to tell, and when. Then the fix, what to do for customers already affected, and what monitoring stays in place.
4 decisions, one debrief
- STEP 1
Assess
Pick up to three of five evidence pulls. Findings appear in a table, and the fairness measures (approval rate ratio, average limit gap, missed good payers) are explained in plain words.
- STEP 2
Contain
Pause LimitIQ, keep it running with underwriters reviewing the affected pattern, lift women’s limits by the gap, or keep it running as is. Each choice trades harm against revenue and cost.
- STEP 3
Disclose
Choose any of the board risk committee, the regulator, affected customers and the public, and whether to tell them now or when the fix is ready. A journalist may call.
- STEP 4
Fix
Retrain without inputs that track sex, retrain on corrected history, repair the data feed or set separate cut-offs by sex. Decide on remediation for past customers and what monitoring stays.
Who Model Gone Wrong is for
- AI, data and analytics leaders. A shared case for teaching business colleagues why outcome testing, root-cause analysis and monitoring belong in every model’s life.
- Risk, compliance and audit professionals. Practise the containment and disclosure side of a model incident, where judgement matters as much as statistics.
- Product owners and senior managers. People who approve or sponsor automated decisions about customers and need to ask the right questions before and after launch.
- Data science and business students. Connects fairness metrics to real consequences: customers harmed, revenue lost, a regulator and a news story.
Who should skip it
It is a learning exercise with an invented bank, not legal advice or a compliance audit of your own models.
What you can do after Model Gone Wrong
You will leave with a working method for model incidents: measure fairly, diagnose before fixing, contain, disclose early and accurately, and put things right for the people affected.
Read fairness measures with confidence
Explain an approval rate ratio, the four-fifths warning line and why only a matched comparison shows the unfair part of a gap.
Spot the three common causes of model bias
Tell a proxy input from a data-pipeline fault and from biased training history, and know which evidence reveals each.
Match the fix to the cause
Know why retraining does nothing for a broken data feed, and why separate cut-offs by sex hide the symptom and can create a new legal risk.
Contain harm while the business keeps running
Weigh a pause against human review of the affected cases, using harm and continuity rather than instinct.
Disclose and remediate credibly
Tell the board and regulator early with what you know and do not know, review affected customers and leave monitoring and an appeals route in place.
Learning goals
- Read fairness measures in plain words and know what each can and cannot show
- Diagnose the root cause before choosing a fix
- Contain harm, disclose accurately and put things right for the people affected
Skills practised: Decision making under uncertainty, Root-cause analysis, Interpreting fairness metrics, Risk and incident management, Regulator and board communication, Ethical judgement in AI, Evidence-based reasoning, Strategic thinking about trust.
Competencies scored
Sub-scores map to the open Bodhih Competency Spine, so results sit beside your other learning records.
What the debrief shows
- What happened: how many customers were harmed before and after the alert, what your containment cost and what you paid back.
- The reveal: the true root cause, the real size of the harm, whether it was reportable to the regulator and whether a journalist called.
- Why each cause needs a different fix, and what strong players do at each of the four steps.
- Four sub-scores mapped to Competency Spine skills, and a key concept matched to the hidden cause: proxy discrimination, disparate impact or historical bias.
Proxy discrimination
When a model treats a protected group differently through inputs that stand in for the protected characteristic, such as a postcode or a spending pattern, even though the characteristic itself is never used. Removing the protected field is not enough: you must test outcomes by group. Depending on the hidden cause, the debrief may instead teach disparate impact or historical bias.
Use Model Gone Wrong with a team
Seats are $12 each. Buy up to 25 seats online and assign them from your company admin; above that, or for a facilitated programme or an institution licence, request a quote and we activate your seats after payment.
Responsible AI workshops
Play individually, then compare evidence choices and disclosure timing in groups. Different hidden causes make the discussion richer.
Model risk and governance onboarding
Give new members of a model risk, data or compliance team a shared incident to reason through before they meet a real one.
Leadership briefings on AI risk
Twenty minutes gives senior managers a felt sense of what a model incident involves, ahead of a policy or board discussion.
University modules on data ethics
Launch it from your LMS through LTI 1.3 and use the score and debrief as a seminar starter or a short assessed task.
Volume pricing
| Seats | Per seat | How to buy |
|---|---|---|
| 1 | $12 | Online |
| 2 to 10 | $10.80 | Online |
| 11 to 50 | $9.60 | Online up to 25, then quote |
| 51+ | $8.40 | Quote |
Model Gone Wrong compared with a case study or slide-based course
| Compared on | A case study or slide-based course | Model Gone Wrong |
|---|---|---|
| What you do | Read how someone else handled a biased model | Investigate, contain, disclose and fix one yourself |
| Uncertainty | The cause is stated on page one | The root cause and the size of the harm are hidden until the end |
| Fairness metrics | Defined on a slide | Shown live, in plain words, as the evidence comes in |
| Consequences | Described | Customers harmed, revenue, costs and possible penalties move with your choices |
| Replay | The same story every time | A new seeded scenario each play, often with a different cause |
| Evidence of learning | Attendance | Scored sub-skills, a debrief and a credential at 80 or more |
What to pair with Model Gone Wrong
Applied AI/ML Practitioner certification
For data people who want to build the bias checks and model cards that would have caught LimitIQ earlier.
See the course →Online courseAI/ML Foundations certification
For managers who want to understand how models inherit faults from their data, with a full module on responsible AI.
See the course →Pro KitThe AI Use Policy and Governance Kit
Puts sign-off, monitoring and appeals, the safeguards strong players leave in place, into a governance programme you can run.
See the kit →Pro KitThink Clearly When Machines Think Fast
Root-cause worksheets, a decision memo template and bias cheat sheets for the diagnose-before-you-fix habit.
See the kit →AssessAll assessmentEveryday AI Judgment at Work (AssessAll)
A scenario-based check of how well your people judge when to trust and when to question AI outputs.
Take the assessment →Related simulations
How much does Model Gone Wrong cost for one person?
One person can buy Model Gone Wrong online for $12, for use on their own Bodhih account. Each simulation is sold on its own. If you want to see how Bodhih Arena feels first, play the free 3-round Price War, no sign-up needed.
What does it cost for a team, a risk function or a class?
Seats are $12 each, with volume tiers by seat count. Up to 25 seats can be bought online. Above that, or for a facilitated session or an institution licence, request a quote; payment is offline and we activate the seats.
How long does it take?
About 20 minutes across four steps: assess, contain, disclose and fix. Allow a few more minutes for the debrief and the reveal of the hidden cause.
Do I need statistics or data science to play?
No. Every fairness measure is explained in plain words beside the numbers. Data professionals will recognise the methods, and non-specialists can still make sound decisions.
Is this about a real bank or real law?
No. Larkfield Bank, LimitIQ and everyone in the story are invented. The simulation uses widely discussed ideas such as the four-fifths rule only as a teaching device. It is not legal advice; take advice for your own markets.
What do I get?
Access to the simulation, a scored result with four sub-scores, a debrief explaining what happened, why and what strong players do, the reveal of the hidden root cause and harm, and one key concept.
Can I replay it?
Yes, as often as you like. Each play is a new seeded scenario, and the root cause can change between a proxy input, a data fault and biased history, so the evidence you choose matters every time.
Is there a credential?
Yes. Score 80 or more and you receive an Open Badges 3.0 micro-sim credential with a public verification page and an Add to LinkedIn button.
Does it work with Canvas, Moodle or our LMS?
Yes. Bodhih Arena supports LTI 1.3 for Canvas, Moodle, Blackboard and D2L Brightspace, including the activity picker and grade passback. It can also be a stage in a Bodhih learning pathway.
How is this better than an AI ethics case study?
A case study gives you the answer with the story. Here you decide with partial evidence, see the cost of each choice, and only then learn the cause. That is closer to how model incidents feel in practice.
What about my data, devices and refunds?
It runs in a web browser on a laptop, desktop or tablet with nothing to install. Your results are kept on your Bodhih account; see bodhih.org/privacy for details. Refund terms are at bodhih.org/refunds.
Find the cause before you choose the fix
Four decisions, partial evidence and real trade-offs between harm, honesty and keeping the lights on. Run the investigation now, then compare your choices with what strong players do.
Want a feel for Arena first? Play the free 3-round Price War.
- Play it as often as you like for 12 months: every play is a new scenario
- A debrief after each play: what happened, why, and what strong players do
- A verifiable credential when you score 80 or more
- Scored on the server, mapped to the Bodhih Competency Spine
Not sure yet? Try the free 3-round Price War first, no sign-up.
More from the Bodhih family
Measure before and after, plan the next step and bring in a facilitator when you want one.