Venality, Incompetence, and Cowardice

This essay originally appeared in issue #52 of The Crux, PivotNine’s weekly newsletter about technology. Subscribe to get early access to exclusive research and opinion from PivotNine analysts.

The Robodebt Royal Commission published its report into the “massive failure of public administration” that was Australia’s Robodebt on 7 July 2023.

It’s worth reading.

I’m going to be a little self-indulgent here and write about Robodebt, because it’s not every day you get called “heroic IT expert Justin Warren” in the newspaper.

Mr O’Donovan flatters me, perhaps too much, but it would be false modesty to shy away from my contribution altogether. I was but one of many, which is the more important point to note. Asher Wolf nerd-sniped me early on and I can be stubborn at times. I still have some matters on foot that I can’t talk about yet, but hopefully I will be able to soon.

I think that Robodebt is an important case study. Systems just like it are getting built all around us, right now. Some of you may be involved in building them, or may know people who are.

They are not inevitable, and you can take steps to stop them.

Background

Robodebt was a plan by the Department of Human Services (DHS) to use computers to find when the government had mistakenly paid people more than they were supposed to be paid, and then require them to pay it back. Often years after the original ’mistake’ was allegedly made. DHS used data people are required to give to the Australian Tax Office as annual income data and averaged it over the whole year. Then, if the fortnightly average so derived didn’t match what DHS thought people had declared (despite often mismatching datasets) DHS would raise a debt against them.

The people targeted were mostly those on income support. Poor people, essentially, not rich people who could afford scary lawyers. DHS thought they were an easy target to make some money that could go into the government budget and make it look like the books were balanced.

But it’s illegal to do that because you can’t average annual income and match it against fortnightly payment entitlements. Any halfway competent accountant could have told them that, or anyone who’d actually read the legislation, which most of the incompetent buffoons running the thing never bothered to do.

The input modelling was laughably wrong, but it gave the answer that powerful people wanted to hear. Those wrong assumptions then used the leverage of automation to make massive mistakes at scale. When the scale of the failure became too obvious to simply ignore, they tried to make minor cosmetic changes to cover up the gaping holes.

And what a cover-up it was.

Outcome

The Commission found that, from the outset, this was “a scheme that would unlawfully take money from social security recipients on a massive scale, and ultimately fail to come anywhere close to the savings which had been promised in order to justify its adoption.”

The Commission found that those who brought the scheme into action knew it was unlawful and did it anyway. Then, when it was fucking up in obvious, predictable, and predicted ways, they tried to cover it up.

And succeeded, for years.

Much of the commentary I’ve read thus far has tiptoed around what I see as the central pillar of the whole saga: A group of powerful people in government decided to steal money from people and were allowed to do it for a very long time.

The various “checks and balances” that were supposed to stop this sort of thing happening in the first place all failed. The mechanisms that were supposed to stop it soon after it started all failed. The mechanisms that were supposed to stop it after it was very, very obvious how illegal and broken the whole thing was failed for much, much longer than they should have.

It took the concerted efforts of a loosely coordinated group of ordinary people, mostly volunteers, to stop this thing.

Lessons

The purpose of a system is what it does, and systems are built by people who have names and addresses. You may be building a system right now.

Some people are misguided or simply incompetent. Some are actively, purposefully malicious. Sufficiently advanced incompetence is indistinguishable from malice, so it doesn’t actually matter much which it is if you’re a person the system is aimed at. Your money still gets stolen by your government.

And once it’s obvious that the system is broken and hurting people, lots of the people who are supposed to stop it simply won’t.

They will trust but won’t verify. They will allow themselves to be mislead even when provided with “as dishonest a document as the Commission has seen.” They will start an assessment “more than five months after field work was originally scheduled to begin, and more than two and a half years after the Scheme commenced.” They will write a report that is used to perpetuate the cover-up; giving “those wanting the Scheme to continue unexamined a shield against criticism from advocacy groups, the media and political opponents.”

The people who are supposed to help simply won’t, but others will. Other people—generally outside the structures that lay claim to power—will help one another in myriad ways. They will work to fill in the yawning gaps in the official frameworks to do what is needed; without compensation and often without acknowledgement or thanks. They will band together, relentlessly pushing the official structures to change, however little and however unwillingly, until they do.

Because they have no choice.

Aftermath

To say that I am angry about this whole debacle somewhat understates how I feel. I have passed well beyond livid into the kind of controlled and focused rage that can fuel a person for years. I hate it so much it gives me energy.

At this point I doubt I will ever stop being angry about Robodebt. The intensity may ebb and flow, but it was such a clear violation of every ethical and moral principle I hold dear that I can’t see myself every getting over it.

My government set out to very deliberately hurt its own citizens (and not only citizens) on a massive scale. It not only succeeded in doing so, it was cheered on by a lot of people who scream bloody murder if the government dares to try to take a tiny portion of their vast riches to make other people’s lives better.

And all of those people are still out there, many of them in positions where they can do it all again.

Consequences

There must be consequences. Not just for those individuals who put the scheme into motion and sustained it over years, but also for those who were complicit, whose “venality, incompetence and cowardice” allowed the whole thing to happen and keep on happening. The people who looked away, and the people who cheered it on.

The consequences need not be permanent, but they should be significant. I fail to see how any of those involved can be trusted to hold positions of trust or authority, and I challenge their superiors to justify why they believe they should. There must be atonement before there can be forgiveness.

There must also be consequences for systems: they must change. The systems that allowed this to happen will resist; stable systems resist change because that’s what stable means. That resistance must be overcome, and it is the responsibility of those with power over these systems to do that work.

It is not up to me to change the culture and structure of these departments because I am not the responsible Minister or Department Secretary. It is their responsibility, and theirs alone. If they aren’t interested in doing that work, they should get out of the way and let someone else do the job. And if they refuse, consequences should be visited also upon them.

Finally, I invite you to reflect on your own position. If you’re in charge of systems, are you making them resistant to malicious actors? Are you making it easy to expose corruption and abuse? What values and morals are you embedding into the things you design and build?

Are you doing a good job or will I need to come for you next?