THE founding principle of affirmative action was fairness. After years of oppression, it seemed folly to judge blacks by the same measures as whites.

“You do not take a person who, for years, has been hobbled by chains and liberate him, bring him up to the starting line of a race,” President Lyndon B. Johnson said in a 1965 speech that laid the groundwork for affirmative action, “and then say, ‘You are free to compete with all the others,’ and still justly believe that you have been completely fair.”

Given this history, it was striking to watch the 80 minutes of Supreme Court oral arguments about affirmative action on Wednesday. With the courtroom overflowing, filled with people who have spent their careers fighting for or against affirmative action, only one side talked about fairness. And it was not the side defending affirmative action.

The lawyer for Abigail Fisher, a young white woman rejected by the University of Texas, argued that she had been denied equal treatment. The conservative justices, sympathetic to Ms. Fisher’s case, expressed particular concern that affluent black students were receiving preferential treatment.

Nobody on the other side — not the university’s lawyer, not the Obama administration’s, not the liberal justices — responded by talking about the obstacles that black and Latino students must overcome. The defenders of affirmative action spoke instead about the value of diversity. Without diverse college classes, they argued, students will learn less and society will lack for future leaders.

The decision to emphasize diversity over fairness is one that affirmative-action proponents made long before Wednesday, and it is a big reason they find themselves in such a vulnerable position today.

Americans value diversity. But they value fairness more. Most people oppose a college’s or employer’s rejecting an applicant who appears qualified for the sake of creating a group that demographically resembles the country.

With affirmative action boiled down to a diversity program, it finds itself in retreat. Five of the six states that have held referendums on racial preferences have banned them, including California and Florida. The Supreme Court limited the legal forms of preferences in 2003 and suggested that they had only 25 years left. Based on last week’s oral arguments, and the fact that Justice Anthony Kennedy has never voted to uphold preferences, the court may restrict them further or forbid them.

Yet supporters of affirmative action do not necessarily need to despair. They still have a path open to them, one that remains legal and popular. It involves resurrecting Johnson’s vision of an affirmative action program based on fairness, which the Rev. Dr. Martin Luther King Jr. also favored.

The crucial choice that affirmative-action proponents made long ago was to focus the program on race rather than more broadly on disadvantage.

There were some obvious reasons to do so. Americans have never been comfortable talking about class. It reeks of the social order the country rejected at its founding (Britain’s) and of the economic system the country spent decades fighting (communism). But race was an undeniably American problem, from slavery to civil rights to the discrimination that, according to voluminous social-science research, lingers.

By forgoing a broader view of disadvantage, colleges lost the ability to claim that their overriding goal was meritocracy. “That was the key moment, when they forfeited fairness,” Richard D. Kahlenberg of the Century Foundation, who has written a book about affirmative action, told me.

Institutions using affirmative action could not claim to be bringing everybody — rich and poor, white and black, native and immigrant — up to the same starting line, in Mr. Johnson’s formulation. They instead were creating a system that depended on racial categories.

From a legal perspective, the decision made the supporters’ task harder. The very laws intended to address the country’s racial history set a high bar for any race-based system. In its first major affirmative-action ruling, the Bakke case of 1978, the Supreme Court rejected the notion that society-wide discrimination justified preferences for individuals. The court reaffirmed that finding in 2003, while also reaffirming that diversity was a legitimate rationale.

It is impossible to know whether affirmative action could have had a more enduring foundation were it based on a broader equal-opportunity approach. Proponents never tried this alternative. Courts, however, have consistently upheld socioeconomic preferences. Had black and Latino students been benefiting from those preferences, as many would, at least some portion of affirmative action might be in less peril.

But the liberals behind the great successes of the civil rights and women’s movements never showed as much interest in economic diversity. On college campuses, administrators have insisted for years that they care about disadvantage, beyond race, but they have done relatively little about it. They have preferred a version of diversity focused on elites from every race.

Black and Latino college applicants, as well as athletes and so-called legacies, receive large preferences — the equivalent of 150 to 300 SAT points. Low-income students, controlling for race, receive either no preference or a modest one, depending on which study you believe. At the country’s 200 most selective colleges, a mere 5 percent of students come from the bottom 25 percent of the income spectrum, according to Anthony P. Carnevale of Georgetown. In court on Wednesday, Justice Samuel A. Alito Jr. attacked the political underbelly of this system. The University of Texas argued that diversity within racial groups was also important, citing “the African-American or Hispanic child of successful professionals in Dallas.” Skeptically, Justice Alito asked the university’s lawyer, “They deserve a leg up against, let’s say, an Asian or a white applicant whose parents are absolutely average?”

Justice Kennedy followed up by telling the lawyer, in one of the most quoted lines of the day, “So what you’re saying is that what counts is race above all.”

Even in California, which has banned racial preferences, race can still dominate the debate. Richard H. Sander, a law professor at the University of California, Los Angeles, has found some hard-to-explain patterns in U.C.L.A.’s undergraduate admissions. The college has accepted a significantly higher percentage of blacks and Latinos than whites and Asians with the same “holistic score,” a number the admissions office gives to every applicant, based on test scores, grades, extracurricular activities and obstacles overcome. U.C.L.A. officials say that the holistic scores do not fully capture the obstacles some students face.

Back in the 1960s, Dr. King understood the vulnerability of today’s affirmative action. “Many white workers whose economic condition is not too far removed from the economic condition of his black brother will find it difficult to accept,” he wrote in a private letter, “special consideration to the Negro in the context of unemployment, joblessness, etc. and does not take into sufficient account their plight (that of the white worker).”

If the courts and voters continue to restrict racial preferences, supporters will have three options. They can give up, which is unlikely. They can quietly subvert the law, as some critics, like Mr. Sander, believe is happening in California. Or they can attempt an overhaul of affirmative action.

The economic argument for a different version has only become stronger over time. Outright racism certainly exists, and colleges would have a hard time taking it into account if race-based affirmative action became illegal. But simple discrimination seems to have become a relatively smaller obstacle over the last few decades, while socioeconomic disadvantage has become a larger one.

The title of a recent paper by Roland G. Fryer Jr., a Harvard economist, summarizes the trends: “Racial inequality in the 21st century: The declining significance of discrimination.”

Racial gaps remain large enough that colleges would struggle to recruit as many black and Latino students without explicitly taking race into account. But some experts, like Mr. Kahlenberg, think they could come close. To do so, they would need to consider not just income, but also wealth, family structure and neighborhood poverty. Those factors disproportionately afflict black and Latino students — and hold back children from life’s starting line.

Mr. Kahlenberg argues that wealth is especially defensible, because it can capture discrimination’s intergenerational effects. Some universities in states where racial preferences are banned, including California, have begun taking small steps to consider class more fully.

Until the Supreme Court rules, sometime next year, the focus will be on its decision. And its decision matters. Yet the choices that universities make matter, too. You wouldn’t have known it from sitting in the courtroom, but there is a version of affirmative action — legal, generally popular and arguably more meritocratic — that higher education has not yet even tried.