Is math biased? Can the formula we choose decide whether AI or humans come out on top?
People say an AI is "more accurate" than humans as if accuracy were one number. It isn't: every error formula notices some mistakes and ignores others. This site lets you test whether the choice of formula can change who looks like the winner.
The problem
AI systems are judged by numbers like "95% accurate" or "error of 2.3", and those numbers decide whether people trust an AI over a person. But the number depends on which formula was used. If the formula hides the kind of mistake that matters, a harmful AI can look excellent on paper.
The mathematics we use
Every measure here starts from one idea: error = prediction − correct answer.
- Mean error — shows direction, but positive and negative errors cancel.
- Absolute value (MAE) — removes the sign so errors cannot cancel.
- Squares and square roots (RMSE) — make big misses count much more.
- Median and maximum — the typical miss versus the worst miss.
The answer in 30 seconds
The correct answers are 10, 20, 30, 40, 50, 60. The AI is perfect five times and misses once by 10. The humans are off by exactly 2 every time.
| Formula used | What it asks | AI | Humans | Winner |
|---|---|---|---|---|
| Median absolute error | How big is a typical miss? | 0 | 2 | AI |
| Mean absolute error (MAE) | How big is the average miss? | 1.67 | 2 | AI |
| Root mean squared error (RMSE) | Are there big misses? | 4.08 | 2 | Humans |
| Maximum error | How bad is the worst miss? | 10 | 2 | Humans |
Same data, opposite verdicts. Every number is calculated correctly. The bias is not in the arithmetic; it is in choosing which formula to report.
How this connects to AI
AI models are trained by making one chosen error formula as small as possible, and tested with the same formulas before they are used. So the formula shapes what an AI learns to care about, and decides whether it is called "good enough".
Two meanings of "bias" on this site
Statistical bias is a number: the mean error. It shows whether predictions lean too high or too low.
Bias in the judging is our Big Question: a formula that quietly favours one side or hides one kind of mistake. A formula can show zero statistical bias and still be a biased way to judge.
How to explore this site
- Analyser — drag a point and watch every measure change.
- Compare — the same measures for the AI and for humans, side by side.
- Experiments — start with Pick the Winner and the Fairness Lab.
- Metric Guide — what each formula answers, plus an 8-question check.
- Research Data — load answers from people and an AI and compare them.
Use Simple (top right) for the five main measures, or Advanced for all measures and the full statistics.
AI use disclosure
We used AI assistants, including Claude by Anthropic, to help write and debug the website's code and to draft parts of the explanations. Our group chose the Big Question, decided which formulas to include, designed the experiments, tested the site, and checked the calculations ourselves. The example numbers on this site are invented for teaching and are not real AI results.
About this project
Group members: [Member 1], [Member 2], [Member 3], [Member 4]
Course: MATH 10 · [Section] · [School]
Instructor: [Instructor's name]
MATH+AI Futures Project · "Mathematics and AI: Shaping a Responsible and Inclusive Future"
Prediction workspace
Type values, load a preset, or drag points on the graphs.
Live measures — AI predictions
Error = prediction − correct answer. The humans' figures are on the tab.
Data summary
Accuracy within tolerance
Prediction data
Presets
Correct vs. predicted
Residual plot
Target MSE
Target mean error / bias
Error shift and scale
Nothing changes until you press Apply.
Worst offenders
The observations with the largest errors, biggest first.
What the Math Says
Updates with your data. Open "Show the math" for the working.
AI vs humans, side by side
The same measures computed twice. Smaller is better for error measures; closer to zero is better for bias; higher is better for correlation and R².
Error size, head to head
What the comparison says
Observation by observation
Who got closer on each one, and whether their errors ran the same way.
| Point | Correct | AI error | Human error | Same direction? | Closer | AI − human error |
|---|
Do they miss the same way?
Error direction
Experiments
Change one thing at a time and watch the measures react. Experiments replace the Analyser data; "Restore starting data" brings yours back.
Pick the Winner
Observe the results
What to look for
Metric Guide
What each formula measures, and what it cannot see.
Check your understanding — 8 quick questions
The numbers in these questions are invented for practice.
What these questions cover
- Why errors can cancel.
- Squared versus absolute errors.
- The effect of one outlier.
- Why MAPE fails at zero.
- Correlation versus agreement.
- Choosing a formula for a real task.
- Who receives the errors.
References
- Willmott, C. J., & Matsuura, K. (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate Research, 30(1), 79–82. https://doi.org/10.3354/cr030079
- Chai, T., & Draxler, R. R. (2014). Root mean square error (RMSE) or mean absolute error (MAE)? – Arguments against avoiding RMSE in the literature. Geoscientific Model Development, 7, 1247–1250. https://doi.org/10.5194/gmd-7-1247-2014
These two papers argue opposite sides of which formula scientists should report. Even experts treat the choice of metric as a judgement, not a fact.
Research data
Privacy: files and rows you load here stay in your own browser. Nothing is uploaded or saved by this site.
Load right-or-wrong answers from people and from an AI, check them, and read the summary. Every figure is calculated from the data you load.
1 · Import
Expected columns
source is human or ai. correct is 1 or 0 (yes/no also works). Human rows need a participant ID; AI rows may leave it blank. A file with only AI rows is fine.
2 · Check
Problems in the data are listed before anything is calculated.
3 · Counts
Accuracy
Accuracy by category
Human accuracy across participants
Accuracy by source
Accuracy by category — answer counts
Coverage — what is present and what is missing
Humans vs AI — where the numbers come from
"Correct" means the final answer matched the expected answer exactly, under a scoring rule fixed before collecting. It tests exact correctness, not closeness.
Accuracy by problem
Which problems were hardest, and whether the AI and the humans struggled on the same ones.Which problems the AI got right and wrong.
| Problem | Category | AI | Human correct | Human rate | AI vs human rate |
|---|
Participants, by anonymous ID
| Participant | Answered | Correct | Accuracy | Per-problem result |
|---|
What these figures can and cannot show
4 · Export
Download these tables as CSV for your report.