Şimal Yücel AAIH Editorial Writer | Opinion article for the Alliance for AI & Humanity

Imagine a scenario where a company develops an algorithm to screen job applications to determine which candidates should proceed to an interview. The company wants the algorithm to be fair, so they design it to ignore demographic information such as race and gender. They believe this “blindness” to sensitive attributes will prevent bias and promote fairness. However, while some might view this approach as fair because it treats all applications identically, others might argue it is unfair because it fails to address historical inequalities and systemic bias in hiring. Some might prioritize treating all candidates the same, while others might prioritize compensating for societal inequalities to equalize opportunities.

This is the problem at the heart of algorithmic fairness: we do not always agree on what fairness means. And yet automated systems increasingly influence significant decisions, such as who gets a job, who gets approved for a loan, or who receives medical treatment. When those systems make consequential choices, whose understanding of fairness should guide them?

Fairness Is More Than a Formula.  When we talk about making algorithms fair, we quickly find out that people have different ideas about what fairness means. Distributive fairness focuses on the outcomes of decisions: how benefits and burdens are distributed among individuals. In the context of algorithms, it might involve ensuring that a loan approval system does not disproportionately deny loans to certain racial or economic groups. Procedural fairness, by contrast, concerns how decisions are made and whether the processes are transparent and just. A hiring algorithm, for example, should provide understandable reasons for its choices and allow applicants to challenge an evaluation they consider unfair.

These differences are not merely philosophical. Researchers have proposed mathematical definitions of fairness, such as demographic parity, which compares selection rates across groups, and equalized odds, which compares certain error rates. But the choice of a metric matters. A system can perform well according to one definition and poorly according to another. Kleinberg, Mullainathan, and Raghavan (2016) show that some desirable fairness conditions cannot generally be satisfied simultaneously when groups have different underlying outcome rates and predictions are imperfect.

This illustrates the need for trade-offs, where prioritizing one aspect of fairness may lead to compromises in another. The idea that algorithms are value-free and objective can mask deeper issues like historical injustices and power imbalances. Simply removing racial or gender identifiers from data does not eliminate bias if the underlying data reflects systemic inequalities. As Birhane (2021) argues, technical fixes alone may fail to address the experiences of communities disproportionately affected by these systems.

Who Makes the Ethical Choices? Algorithms are created by humans, and humans have biases. These biases can unintentionally be built into algorithms through the data they are trained on or the way they are designed. If a hiring system learns from historical company decisions that disadvantage certain groups, it may continue to favor the same kinds of candidates. But fairness involves more than correcting training data: it also requires deciding which outcomes we value, which mistakes matter most, and whose interests receive priority. Developers and organizations behind these algorithms make decisions during design, implementation, and deployment that carry significant ethical implications. For example, a developer could choose to weigh certain objectives more heavily to produce an “optimal” algorithm. That choice makes the system value-laden and can affect its fairness. Johnson (2023) questions the assumption that algorithms operate independently of human values. In high-risk situations, the consequences of choosing the wrong priorities can fall on people who never had a say in making those choices.

If there is no single, universally accepted understanding of fairness, is it better to rely on a limited number of people to make choices that can affect our lives? Or should we rely on a more diverse and deliberative environment that includes a broader range of perspectives?

Co-Creation: Giving Affected People a Voice I argue that involving a diverse group of stakeholders in designing algorithms is crucial for managing different opinions about what is fair, especially in a democratic society. The approach I propose is co-creation: a process where relevant stakeholders collaborate from the beginning of a project, actively contributing to its design and development. Those affected by decisions should have a say in the process—not only after a system is deployed, but while its goals and standards are still being defined.

This argument is grounded in the Principle of All Affected Interests: people whose interests are significantly affected by a decision ought to have an appropriate opportunity to influence it. Algorithmic decisions, particularly in high-stakes contexts, should therefore be shaped by those who are impacted by them. This does not mean every stakeholder must become a programmer. It means their experiences and judgments should inform what the system is trying to achieve and what counts as an acceptable outcome.

There are already approaches that integrate human input into algorithm development. Human-in-the-loop machine learning, for example, can involve experts who label data or correct predictions. Value-sensitive design seeks to incorporate stakeholder values. These approaches offer valuable insights, but human involvement is not automatically the same as shared decision-making. Expert feedback aimed at improving technical accuracy is different from allowing affected communities to question the goals, criteria, and consequences of a system.

Consider the difference in a hiring system. In a consultation, a company might ask applicants whether they trust an algorithm after its selection criteria have already been fixed. In co-creation, applicants and representatives of affected groups would help debate those criteria before they are finalized, see how their input shaped the design, and have ways to challenge the system later. Co-creation does not require every participant to control every technical choice. It requires genuine influence, clear explanations of decisions, and accountability when suggestions are rejected.

Co-creation requires continuous engagement and deliberation. Societal norms, ethical standards, and user needs can change over time. A one-time survey or consultation is not enough if stakeholders cannot influence the decisions that follow. Participants need opportunities to discuss disagreements, understand trade-offs, question decisions, and influence revisions. Wong’s (2020) discussion of democratizing algorithmic fairness offers a useful guide: decisions and their reasons, should be public, justifiable to stakeholders, open to appeal and revision, and subject to oversight.

What Would This Look Like in Practice? Consider a university designing an algorithm to assess scholarship applications. Administrators may want to reward academic achievement, while students from disadvantaged backgrounds may argue that grades alone fail to reflect unequal access to resources. Developers may focus on predictive accuracy, while student representatives may prioritize opportunities for those who have faced structural barriers. None of these concerns can be resolved simply by choosing the most accurate model. A co-creation process would bring these groups into the conversation before the system is built, while decisions about scholarship criteria are still open. Unlike a consultation that simply collects opinions, participants could help compare alternative criteria, identify overlooked harms, and recommend changes that administrators must publicly address. It would include individuals directly impacted by the algorithm’s outcomes, especially those who have experienced unfair treatment. Transparent recruitment, community outreach, and collaboration with organizations representing underrepresented groups could help ensure participation is meaningful rather than symbolic. Support such as accessible meeting formats and compensation can reduce barriers to taking part.

Stakeholders do not need to understand every technical detail to contribute effectively, but a foundational understanding is important. Clear, non-technical explanations of the algorithm’s purpose, possible consequences, and competing fairness measures would allow them to engage in informed discussion. When disagreements arise, structured dialogue can help participants explain their priorities and reflect on the assumptions behind them. Consensus may not always be possible; decision-makers should still explain whose concerns were prioritized and why.

Meaningful participation also requires an account of what changed: if the university rejects a student recommendation, it should explain why rather than treating the meeting as proof of inclusion. Participation should continue after deployment. Surveys, public feedback, advisory committees, and transparent records of changes can help identify problems and make revisions possible. Crucially, institutions must remain accountable for final decisions. Co-creation is not a guarantee that an algorithm will be fair, nor is it a substitute for independent oversight. Its value lies in making difficult choices more inclusive, visible, and open to challenge.

Fairness Requires Participation. Making algorithms fair is not just about solving technical problems; it is also about understanding what fairness means to different people and making difficult choices about the technology we create. Because those choices can shape access to employment, finance, healthcare, and opportunity, they should not be left exclusively to a small group of developers or institutions. Co-creation offers a way to bring broader societal values and lived experiences into algorithm design. The goal is not to pretend that every disagreement can disappear, but to make the process of negotiating fairness more democratic and accountable. Ultimately, the question is not only whether we can build fairer algorithms. It is also who gets to decide what “fair” means—and who gets heard when that decision is made.