When Air France 447's autopilot failed over the Atlantic in 2009, the pilots had three and a half minutes to fly the plane manually. They couldn't. Not because the automation had failed them, but because it had worked so well that the pilots’ underlying skills had eroded.

That accident has become the defining case study in automation paradox: the gradual erosion of human capability that happens when automated systems work too well for too long. The same dynamic is now playing out across lending operations.

AI is doing exactly what it should be doing

AI is now handling some of the most time-consuming parts of lending, from document capture to customer onboarding and KYC checks, and doing it faster and more consistently than any team could manage.

But what is happening to the people overseeing these systems?

A credit analyst who manually reviews financial statements for years isn't just crunching numbers; they're developing a gut feeling for hidden risks, learning to spot when clean figures might be misleading. This intuition grows from sheer experience: seeing customer files, understanding what's normal, and learning from past mistakes.

AI is doing its job and doing it well. The real concern is that we haven't built systems to proactively address the human skills that vanish as a result.

Research on AI-driven skill erosion describes this as “capacity-hostile environments.” Essentially, when AI takes over tasks, it removes the hands-on experiences humans need to develop and maintain their skills.

A human in the loop is not the same as a capable human in the loop

Keeping a human in the loop is the right instinct, but it does not solve the problem on its own. When a reviewer is presented with a recommendation from a system that is usually right, accepting it is the path of least resistance.

Studies on automation bias show two common problems:

• People either act on incorrect AI suggestions because they don't question them enough,

• Or they miss real risks because the AI didn't flag anything, assuming no flag means no problem.

• A small business with borderline financials but a strong business model, where the team lacks the confidence to overrule the AI's hesitation.

• A loan where collateral is contested, requiring someone with prior experience in similar disputes.

• A complex, large loan that falls outside the model's training, demanding a credit manager who can handle the nuances without AI guidance.

And neither of these issues becomes clear until it's too late.

If a reviewer spends months just approving AI outputs instead of doing their own analysis, they're not maintaining the judgment skills that make their review meaningful. And it goes beyond simply forming an independent view. Even when they do look at the AI's suggestion, the instinct to pause, question, and stress-test it has to be a deliberate, conscious effort.

MIT researcher Renée Richardson Gosline puts it plainly: sometimes slowing people down is the best way to serve them. For example, a credit manager taking a beat to assess risks before seeing the AI's recommendation, or an underwriter jotting down their own observations before the model's score pops up. These short, deliberate pauses before accepting an AI's output force a genuine re-engagement of judgment.

They shift the process from mindlessly approving to actively thinking, making a small but crucial difference: ensuring analytical skills stay sharp over time, instead of quietly fading away.

The problem shows up when it's too late

Day to day, these issues are invisible. Files flow smoothly, performance metrics look good, but the decline in a credit team’s independent judgment is not being tracked.

A study of an accounting firm found exactly this: reliance on automation eroded staff competence so gradually that nobody noticed until the system was removed, and employees could no longer perform core tasks.

While not resulting in dramatic system failures, here's how this problem can show up in lending:

These are the make-or-break moments, separating strong lending institutions from vulnerable ones, especially when a team can't rely on its own judgment.

Where the RBI's framework stands

The RBI's FREE-AI framework states that regulated entities are accountable for AI-assisted decisions regardless of the system's autonomy. For that accountability to hold up, the humans in the decision chain must be capable enough to look at a decision, understand it fully, and stand behind it.

An organisation where experienced staff approve AI outputs without the skill to evaluate them falls short of what a reliable lending system requires: that the person accountable for a decision can read a file, weigh the risks, and take ownership of the call independently if needed. This often gets overlooked when discussing the RBI framework, which tends to focus on things like audit trails and model governance. Those are important, but a fundamental question remains: are the people accountable for these decisions capable of making them?

Building platforms that keep humans capable

The answer is smarter, more intentional design. We need to send complex cases to seasoned experts instead of just automating everything. The review processes should empower humans to make their own independent assessments. More than just tracking speed, we need to measure if decision-makers are maintaining the deep judgment these choices demand.

As one research framework puts it, a sustainable approach requires humans to remain cognitively active and adaptive alongside the automation. Rather than just being present in the workflow, they need to engage with it.

It is perhaps the only way to build an AI-led system that holds up under pressure.

Until next time,

Srijan
Co-founder
FinBox