An investigation into how the platform that trains AI judges the humans doing the training, and why workers say the system is designed to leave them in the dark
There is a particular cruelty in being graded by a system you cannot see, for reasons you will never be told, with no appeal and no recourse. It is the cruelty of the algorithm made bureaucratic. Dressed up in the language of science and quality assurance, but functioning, in practice, as something closer to a trap.
I have spent several weeks reading through hundreds of forum posts, Glassdoor reviews, Discord threads, and platform documentation, and combing through the company’s public-facing technical literature. What I found is a scoring apparatus that is, by design, opaque to the people it judges. A machine that generates verdicts without explaining them. And a workforce of tens of thousands of highly educated people, doctors, engineers, lawyers, linguists, who are building the foundational datasets for some of the most powerful AI systems on earth, while operating under conditions of almost total informational darkness about their own standing.
This is a story about that darkness. What it is. Why it exists. Who benefits from it.
The Promise
Labelbox is not some fly-by-night operation. Founded in San Francisco in 2018, it has raised over $188 million in venture capital from SoftBank, Andreessen Horowitz, and Kleiner Perkins, among others. The company is valued at roughly $3.2 billion. Its clients include Google, ElevenLabs, and what its marketing materials describe vaguely as “leading AI labs.” The pitch is simple: Labelbox1 operates what it calls a “data factory for AI”, producing the labeled training data that makes large language models and computer vision systems function.
The contractor-facing arm of this operation is called Alignerr2. The pitch to workers, advertising on LinkedIn, Twitter, and AI enthusiast communities, is generous and intelligently calibrated to attract educated, ambitious people. Flexible hours. Remote work. Intellectually stimulating tasks. Pay rates between $15 and $150 per hour, depending on expertise. The Alignerr website describes its community as “a powerhouse of global experts shaping cutting-edge models.” The acceptance rate, per the company’s own materials, is lower than Harvard’s.
This framing is not accidental. When you need humans to evaluate AI outputs, rank competing model responses, write complex code solutions, or assess technical accuracy, you want people who believe they are doing skilled work, not clicking boxes. The more educated and motivated the contractor, the better the training signal for the model. The value proposition is real on both sides.
The problem begins when you try to understand how that value is being measured.
The Machinery of Judgment
To understand why workers find themselves confused and frustrated by Labelbox’s scoring systems, you first need to understand what those systems are actually doing. At least, what the company is willing to say publicly.
Labelbox’s3 quality analysis documentation describes two primary mechanisms for evaluating annotator performance: benchmark scoring and consensus scoring. In benchmark scoring, a set of “gold standard” labels, designated correct answers, is distributed to annotators. Their work is then compared against these benchmarks mathematically. For image annotation tasks, the platform uses Intersection over Union (IoU), a geometric measure of overlap between the annotator’s bounding boxes and the gold standard. For text tasks, it uses BLEU scores and other NLP similarity metrics.
Consensus scoring, meanwhile, compares annotations on the same data row by different labelers to determine a consensus winner. The algorithm finds annotation pairs that maximize total IoU scores, assigns a zero value to unmatched annotations, and averages the results. A 100% benchmark score means complete match with the gold standard; 0% means no match at all.
For the text and reasoning tasks that dominate Alignerr’s higher-paying work, Labelbox has layered in what it calls LLM-assisted quality control. This is where a fine-tuned language model acts as a judge, generating similarity scores between a contractor’s response and a ground truth answer. The LLM flags responses below a certain similarity threshold for human review. The company describes this system as having advantages of “scalability” and “consistency.” Unlike human reviewers, the LLM “applies the same criteria consistently across all evaluations.”
Sitting on top of all of this is what Labelbox calls the “Alignerr trust score.” According to the Labelbox’s4 own description, this is a “multidimensional score” that “incorporates various factors such as historical accuracy, consistency, task completion rate, and the ability to handle complex assignments.” High-trust workers are prioritized for more critical or complex tasks. Low-trust workers get more oversight or are effectively demoted within the pipeline.
This is the score that controls your career on Alignerr. The score that determines which tasks you see, how much you earn, and whether you continue working at all.
And it is the one score workers are never allowed to see in full.
What Workers Actually Experience
Read enough Glassdoor reviews5, and a pattern emerges with striking consistency. One reviewer writes:
“Payment only processed for ‘approved’ work, not completed work. Projects can pause indefinitely, leaving submitted work unpaid but retained by company. Approval standards are subjective and inconsistent.”
Another describes being removed from a project after completing sixty-two tasks with a 4.4 out of 5 quality rating, losing $2,100 in earnings that were never paid. The company’s response, repeated in review after review with near-identical wording:
“As per our records you have been removed from the Alignerr program. This decision is final, and we will not be reviewing any requests for reinstatement. Please note that Support is unable to assist with matters related to program removal.”
A third reviewer on Glassdoor explains:
“My rating was above average (4.0, the average was 3.5), and most of the reviewers said that I did my work perfectly… The support doesn’t allow contacting them and provides an automatic response that the ‘payment will be withheld due to community rules violations.’ No violations were stated.”
One Glassdoor post from a former worker describes passing the assessment, joining all required platforms, completing the onboarding, and then waiting nine months without a single task assigned, before being removed without explanation.
Then there is the question of the assessments themselves. According to Real Ways to Earn6, the Alignerr assessment process is intentionally ruthless, with some workers reporting that a 95 percent score still counts as a fail and locks them out of that role permanently. The site aitrainer.work7 notes that a failed assessment is often permanent for a specific role, comparing the experience to a final exam with no retake permitted.
This is not a minor grievance about difficult tests. It is an architectural choice. And it matters because the assessment is not merely a hiring hurdle. It sets the initial value of a worker’s trust score, a number they will carry with them through every subsequent task, shaping what work they see and how their contributions are valued, with no ability to review the reasoning behind the original judgment.
The Circular Logic of the Gold Standard
The central problem with Labelbox’s benchmark system. And it is a problem that the company’s own documentation hints at, without fully confronting, is the circularity of the “gold standard” label.
Labelbox tells clients that a benchmark label can be created by the client themselves or chosen from a “perfect label” done by the labeling team. In practice, this means that the ground truth against which a contractor’s work is measured was itself produced by a human. One who may have had their own interpretive biases, their own reading of ambiguous instructions, their own aesthetic preferences when it comes to free-text tasks.
For image annotation tasks with clear geometric boundaries, IoU scoring is relatively defensible. A bounding box either overlaps the object or it doesn’t. The math is clean. But for the RLHF tasks that Alignerr specializes in evaluating AI responses for quality, writing code solutions, and ranking competing model outputs. There is no clean math. There is only judgment. And when you pit one person’s judgment against a “gold standard” that was itself the product of someone else’s judgment, you are not measuring quality. You are measuring conformity.
The Labelbox documentation acknowledges this obliquely, noting that “low scores on an objective task may point towards gaps in labeler training or can allude to ambiguous wording in labeling instructions, leaving room for different interpretations.”
But this acknowledgment is aimed at clients, at the people managing annotation projects. Not at the workers being scored. Workers themselves receive a number. They do not receive the reasoning. They cannot challenge the benchmark. They cannot see it. They cannot know whether the gold standard against which their work was judged was itself accurate.
Now add the LLM layer. Labelbox’s fine-tuned language model assesses similarity between contractor responses and ground truth. It then flags low-similarity responses for human review, or, in some cases, scores them automatically. This means that in a significant portion of cases, a contractor’s work is being judged not by a human, but by an AI trained on previous human judgments, measuring deviation from a standard that the contractor has never been shown. The output of this process feeds into the trust score. The master metric that governs the contractor’s livelihood.
This is not transparency. This is the appearance of transparency. A system dressed in numbers and methodology that, from the inside, functions exactly like a black box.
The Forum Evidence
If you want to understand what workers actually think about all this, you have to go where workers go when they are frustrated and anonymous. Reddit. Glassdoor. Discord servers. The forums of AI trainer communities where displaced annotators gather to compare notes.
What you find is a consistent vocabulary. Words like “arbitrary.” “Unexplained.” “Final.” Workers who describe completing tasks well, receiving positive feedback from human reviewers, and then being deactivated anyway. Workers who describe scoring above the stated threshold on an assessment and still being told they failed. Workers who describe submitting work, having it accepted by reviewers, and then finding their accounts suspended just before the payment processing window. A pattern documented so consistently across reviews that it has become something of a dark joke in the community.
A Glassdoor8 reviewer recounts being told by a support representative to stop “spamming” when asking about withheld payment of $200, and being told that terminations are permanent, with no payment forthcoming.
“They also lied and said that their guidelines state that deactivated accounts won’t receive payments. This is a lie, as I’ve read the guidelines, which clearly states that payment will be withheld until a human reviews the account.”
That last sentence deserves attention. The worker had read the guidelines. They were citing them precisely. And they were being told, by a representative of the platform, that the guidelines said something different from what they actually said. This is not just opacity. This is opacity backed by misinformation.
The Broader Problem: Algorithmic Management at Scale
What Labelbox is doing is not unique to Labelbox. It is the operating model of the gig economy, applied with particular sophistication to a workforce that is too educated and too dispersed to organize effectively.
Privacy International9 has documented that data labelers are “carrying the AI supply chain on their backs,” and yet their status in the gig economy means they “can and will be exploited by black box algorithms and unreasonable working conditions.” Academic research published through SAGE Journals10 in 2025 notes that “algorithmic evaluations of worker performance can be rife with their own biases, which can be persistent and harmful to worker rights and dignity, given the black-boxed nature of many AI-based platforms.”
Research from the ACM11 has explored how workers in platform economies can attempt to audit algorithmic management through what researchers call “worker-centric tools,” essentially, attempts to reverse-engineer the systems that govern them. The challenge is that these efforts require exactly the kind of coordinated action that platform companies are structurally incentivized to prevent. When workers organize on Reddit to share experiences, posts disappear. When they rate the platform on Glassdoor, the reviews accumulate, but the company’s response is the same boilerplate: “This decision is final.”
European labor organizations have started to push back. The European Trade Union Confederation published a manual in late 2025 on “breaking open the black box of algorithmic management,” and the EU Platform Work Directive, set to become law in member states by December 2026, includes explicit rights for workers to receive explanations of automated decisions and to have their representatives consulted on changes to AI systems. These are rights that Alignerr workers in the United States and the Global South do not currently have.
For workers outside Europe, the situation is more precarious. An Oxford Economics report12 on the data annotation industry published in December 2025 estimated that the global market for data collection and annotation has grown dramatically, driven by what it calls the AI revolution. But the report is largely silent on the labor conditions of the people doing the annotating. The value chain is described; the humans at the bottom of it are not.
The Trust Score Paradox
There is something almost poetic about the central contradiction here. The Alignerr trust score exists, per Labelbox’s own explanation, to “optimize workflow and maintain high data quality standards.” The system is meant to ensure that the humans training AI models are reliably producing good work. High-trust workers are prioritized for more critical or complex tasks; the trust score “serves as a valuable feedback mechanism, providing AI trainers with insights into their performance.”
But here is the problem: for the trust score to function as a feedback mechanism, workers must be able to see it, understand it, and respond to it. A feedback mechanism that doesn’t communicate feedback isn’t a feedback mechanism. It is a sorting machine.
And it becomes a particularly vexed sorting machine when the score is multidimensional and composed of inputs that the worker cannot directly influence because they don’t know what those inputs are. If my trust score drops because my historical accuracy slipped on a batch of tasks where the gold standard was, in my judgment, itself incorrect. I can’t contest that. If my completion rate metric is being penalized because a project was paused without warning, and I had no tasks to complete, I can’t address that. If an LLM similarity score flagged my response as low-quality despite a human reviewer approving it, I don’t know what to think about that.
Labelbox boasts that its platform processed over 50 million annotations in a single month, representing over 200,000 human hours. At that scale, individual workers are, inevitably, abstracted into data points. The system does not have time for nuance. The algorithm makes a call. The worker receives a verdict. The platform moves on.
Labelbox is not, as some frustrated workers have claimed online, straightforwardly a scam. It is a real company with real clients doing real work that matters for AI development. The technical infrastructure it has built, the benchmark scoring, the consensus mechanisms, and the LLM-as-judge pipeline, is sophisticated and, for the enterprise clients who purchase it, genuinely useful. The platform offers role-based permissions, project dashboards, and activity logs, and from the client’s perspective, the control and transparency are excellent.
That is precisely the problem. The transparency is pointed entirely in one direction. Clients can see everything. Workers can see almost nothing.
The company claims to provide workers with “insights into their performance” through the trust score. But insights require explanations, and explanations are exactly what the boilerplate termination messages refuse to provide. “This decision is final” is not an explanation. It is a door slamming.
The legal architecture that enables this is the independent contractor classification. Because Alignerr workers are contractors, not employees, they lack the protections that employment law provides. Minimum wage guarantees in many jurisdictions, protections against wrongful termination, rights to explanation for adverse employment actions. The company’s own contractor agreement includes a provision, “Section 5,” that allows for termination without cause and without notice. Workers who signed this in good faith, excited by the prospect of intellectually stimulating work at competitive rates, find themselves without recourse when that clause is invoked.
As Privacy International13 observes, it took years for the European Parliament to even consider enshrining the rights of platform workers into law; the ILO has a report on decent work in the platform economy that will be discussed at 2025 and 2026 International Labour Conferences. For the workers in Manila, Nairobi, and rural Ohio who are currently annotating data for the next generation of AI models, that calendar moves very slowly.
What Better Would Look Like
The technical mechanisms for greater transparency already exist. They are not particularly difficult to implement. If Labelbox wanted to show workers their trust scores, not just tell them the score exists, it could. If it wanted to explain which components of a trust score declined after a batch of tasks, and provide the actual benchmark labels that the worker’s annotations were compared against, it could. If it wanted to build an appeal mechanism for automated quality decisions, particularly LLM-generated ones, which are known to have systematic biases, it could.
The reason these mechanisms don’t exist is not technical. It is commercial. Opacity is cheaper than transparency. Automated termination is cheaper than case-by-case review. An appeal mechanism creates liability exposure. A visible trust score invites argument.
The counterargument, and Labelbox would undoubtedly make it, is that the gold standard labels and benchmarking criteria are proprietary. Revealing them would allow workers to game the system rather than produce genuinely good work. This is not an entirely frivolous concern. But it is also not an either/or situation. The identity of a benchmark label can remain private while the reasoning behind a quality decision can still be communicated. “Your bounding boxes were an average of 23% below the IoU threshold on this task type” is actionable feedback that does not reveal proprietary information. “This decision is final” is nothing but a shutdown.
The humans training AI models are, by the nature of the work, building systems that will increasingly be used to judge the quality of human work. They are teaching language models to evaluate text, rank responses, detect errors. They are, in a real sense, training their own evaluators.
And they are doing this while being evaluated by systems they cannot see, whose criteria they cannot access, whose verdicts they cannot appeal. There is an irony in the fact that the most sophisticated quality evaluation infrastructure in the history of data annotation, the benchmark scoring, the consensus algorithms, the LLM-as-judge pipelines, the multidimensional trust scores, is oriented entirely toward the enterprise client and almost entirely away from the worker.
What gets measured matters. What gets measured and shown to you shapes your behavior. What gets measured, hidden from you, and used to terminate your contract without explanation is something else entirely. It is control dressed as science. And the workers building the AI future deserve better than that.
Sources
- “Inside the data factory: How Labelbox produces the highest quality data at scale” labelbox.com/blog/inside-the-data-factory-how-labelbox-produces-the-highest-quality-data-at-scale/. Accessed 28 May 2026. ↩︎
- Alignerr, www.alignerr.com/. Accessed 28 May 2026. ↩︎
- “Quality analysis” Labelbox, docs.labelbox.com/docs/quality-analysis. Accessed 28 May 2026. ↩︎
- “Inside the data factory: How Labelbox produces the highest quality data at scale” labelbox.com/blog/inside-the-data-factory-how-labelbox-produces-the-highest-quality-data-at-scale/. Accessed 28 May 2026. ↩︎
- Glassdoor, www.glassdoor.com/Reviews/Employee-Review-Alignerr-E10042963-RVW100054641.htm. Accessed 28 May 2026. ↩︎
- Thurman, Anna. “Alignerr Review — Get Paid to Train AI From Home” Real Ways to Earn, 12 Apr. 2026, realwaystoearnmoneyonline.com/alignerr-review/. Accessed 28 May 2026. ↩︎
- AI Trainer, aitrainer.work/guides/alignerr-review/. Accessed 28 May 2026. ↩︎
- Glassdoor, www.glassdoor.co.in/Reviews/Employee-Review-Alignerr-E10042963-RVW98006198.htm. Accessed 28 May 2026. ↩︎
- “Humans in the AI loop: the data labelers behind some of the most powerful LLMs’ training datasets” Privacy International, 15 Aug. 2024, privacyinternational.org/explainer/5357/humans-ai-loop-data-labelers-behind-some-most-powerful-llms-training-datasets. Accessed 28 May 2026. ↩︎
- Sage Journals, journals.sagepub.com/doi/10.1177/01708406241282125. Accessed 28 May 2026. ↩︎
- ACM, dl.acm.org/doi/10.1145/3570601. Accessed 29 May 2026. ↩︎
- Wilse-Samson, Laurence. “The Economic Impact of the Data Annotation Industry” 8 Dec. 2025, static.scale.com/uploads/6691558a94899f2f65a87a75/static.scale.com/uploads/6691558a94899f2f65a87a75/Oxford%20Economics_%20Impact%20of%20Data%20Annotation%20%20Dec%202025.pdf. Accessed 29 May 2026. ↩︎
- “Humans in the AI loop: the data labelers behind some of the most powerful LLMs’ training datasets” Privacy International, 15 Aug. 2024, privacyinternational.org/explainer/5357/humans-ai-loop-data-labelers-behind-some-most-powerful-llms-training-datasets. Accessed 29 May 2026. ↩︎
