The welfare agency Försäkringskassan (FK) of Sweden has found itself in controversy after journalist Gabriel Geiger brought to the public attention a lot of biases contained in its fraud AI system. Another driver of such suspicions is that the fraud detection system was designed and trained using morphological tools that allow for mathematical models and legal analysis.
The Investigation: Connecting the Dots of Systemic Bias Through FOIA Requests
It all started with Geiger sending over a hundred emails and 12 FOIA (Freedom of Information Act) requests over three years, which bore no fruit. The agency would often refer to the need for confidentiality regarding the specifics of the system concerning fraud detection. As Geiger stated on Twitter:
How many people receive a risk score? Confidential. How many people has the system flagged? Confidential. How much did the system cost to build? Confidential. How do you define fraud? Confidential
— Gabriel Geiger (@gabriels_geiger) November 27, 2024
Geiger was ultimately able to obtain key datasets through the process of requesting another Swedish Agency, but not to Försäkringskassan. These datasets uncovered some disturbing patterns in fraud cases. For instance, 63% of the cases flagged were women, 4.2 times more immigrants compared to Swedish natives were flagged and 72% of the people who were flagged were low-income ones although little less than half of applicants on welfare were low-income.
Key Findings: The Algorithm’s Disparate Treatment
Working with the academics, Geiger’s team validated six fairness definitions that are frequently employed in the field of algorithmic ethics. The results indicated that:
Women: Flagging cases, 15% more women than men, innocent women to be more specific, were flagged.
Immigrants: Despite low levels in the welfare applications, 22% of the applicants, immigrants were 54% of the flagged cases.
Education Level: Those who do not hold a university degree in particular education or area of expertise are 2.3 more times likely to be marked for petty frauds, without bearing in mind whether their credentials allow them to be in such a position.
Outcome Accuracy: Only 27% amount of fraud cases which were flagged were established to be legitimate fraud cases which means that 72 percent of the cases that were investigated were not even serious.
Geiger tweeted the following:
With support from academics, we tested six different statistical fairness definitions. The results are disturbing. Women, immigrants, low-income earners and people without a university education who were innocent more likely to be *wrongly* selected for investigation
— Gabriel Geiger (@gabriels_geiger) November 27, 2024
The ‘Fairness Procedure’: An Example Of Bad Practice?
For criticisms of discrimination, Försäkringskassan also adopted a “fairness procedure” in the AI system. However, the evaluation of this procedure brought some fundamental critique which found serious problems. For example, it permitted ‘militarizing’ up to four immigrants for each Swedish native. Or even if the algorithm did not meet fairness criteria, the agency could, in principle, ignore it where legal consent was obtained.
“Fairness procedure permits immigrants selection of four for every Swede without testing if this is at all warranted. The model could also fail these ‘generous’ thresholds, there’s even an exit strategy: It’s ok if the legal is ok,” Geiger tweeted.
But there was a catch. The agency said it had created a new fairness procedure to prevent discrimination. We obtained it via FOIA. When we tested it against our data, all the problems re: discrimination seemed to disappear
— Gabriel Geiger (@gabriels_geiger) November 27, 2024
The agency held the results aggregated of the procedure of the particular aspect to be the consistent remark that if shared fraudsters will have ‘they won the game in the system’.
Economic Perspectives: Costs and Inefficiencies
As reported, the development cost of the welfare fraud detection system is about lakhs of Euros, however, it seems questionable. It was deduced that up to 73% of cases that were flagged were false positives, thus operational expenses of the system which included staff being engaged in the unnecessary investigations extended the burden by another Euros 1.8 million per year. Some, however, oppose this position and say that welfare resources could be used for these areas.
Comparative Context: Bias In AI Systems – A Global Perspective
It is also pertinent to note that the Swedish welfare case is not anomalous. In 2023, the Algorithmic Justice League carried out a study in Europe that suggested that for almost 65 percent of people in administration, AI systems required support, and operated with bias. Similar stories have also come in the UK and the Netherlands. In these welfare schemes, majority and low-income groups were caught allegedly and there were several litigations and public protests.
The Terrible Reality Of Algorithmic Accountability And The Missing Open Floor To The Problems
The disaster that has occurred in Japan and the anger of the people that arose from the consequential alleged misuse of AI should serve as the last warning to US authorities and other nations well wisdom would suggest that like – AI should be antagonized. Gabriels Geigers’ research has highlighted the serious consequences of unregulated AI systems, abusive to the people, which have been deployed in public services.
The case exemplifies the deformity design of algorithms that perpetuates inequality manifesting in biased policies against women, immigrants, and low-income earners for these policies are predicated on the algorithm.
Traditionally seen as a model of social justice, Sweden will have to face this uncomfortable paradox. The facts require stronger control, recurrent checks, and accountability in AI’s use. Countries in all parts of the world should use this case as a lesson, ensuring that their systems encourage equity and efficiency as well as safeguard the people’s liberties.
Geiger’s employment makes one realize the potential of investigative journalism and its role in holding institutions accountable, helping in advancing technology that is fairer towards all in the provision of social services. As he pointed out:
“This investigation shows not only the failures of the particular system but the problems which arise when a biased AI is integrated into a central administrative system. Compliance is not relative – it affords a human rights point.”
That’s because @fk_media ‘s fairness procedure allows, for example, 4 immigrants to be selected for every Swede without having to test if this is at all justified. Even if the model fails these 'generous' thresholds there’s an escape hatch: It's okay if legal says it's okay pic.twitter.com/EQhYkWJvhK
— Gabriel Geiger (@gabriels_geiger) November 27, 2024
The saga of Försäkringskassan raises a helpful reminder about the absence of ethical AI in practice: it is possible to use technology and improve operational effectiveness, but there is no place for abuse of justice or fairness in the process.
Last Updated on by Icy Tales Team