Data C104 /

week12.md

Week 12 Study Guide: AI Ethics, Values, and Codes

Big idea

  1. Value alignment treats two problems as one: making machines moral and making them do what we want. The second quietly swallows the first.
  2. Alignment looks neutral between moral frameworks but smuggles in preference utilitarianism: agents are expected utility maximizers, the good is preference satisfaction, and society is an idealized marketplace.
  3. Winner's warning: the word values flattens rich moral and political language into subjective preferences nobody can argue with. Alignment is built on exactly that word.
  4. Constitutional AI tries a virtue-ethics route, cultivating judgment over rules, but a company still writes the character and ranks its own oversight first.
  5. Codes of ethics are the other go-to fix. They came out of professionalization and scandal, assume ethics is an individual's single decision, and often end as ethics theater.
  6. Both alignment and codes try to turn ethics into a decision technology. The course's alternative is practice, stories, and attention to power and sociotechnical systems.

Russell, "AI: A Different Approach"

Core idea: The danger of AI is the standard model, machines that optimize a fixed objective. Russell's fix is beneficial machines that are uncertain about human preferences and learn them from behavior, so they stay deferential and let themselves be switched off.

  • The standard model fails as machines get smarter: they pursue a wrong objective anyway, resist shutdown, grab resources, and may deceive us. None of this needs malice, it is just optimal planning.
  • Three principles for beneficial machines, meant as a guide for designers rather than laws for robots: the machine's only objective is to maximize human preferences; it is initially uncertain what those are; human behavior is the ultimate evidence.
  • Purely altruistic. No intrinsic value on its own survival, since self-preservation would be a competing incentive.
  • Humble machines. Uncertainty about the objective is the key. A machine that is sure of its goal ignores people screaming stop. An uncertain one lets itself be switched off, since being switched off means it was about to do something wrong.
  • Preferences, not values. He uses the technical economic sense, utility, and sidesteps the question of whose values, promising to treat everyone equally in the utilitarian tradition. Trolley problems are a distraction; survival is not a dilemma.
  • Learning from bad people. Watching a bribe-taking official teaches the machine that he wants his kids educated, not that bribery is good.
  • Optimism and caution. Industry has economic reasons to build deferential systems, and the roasted-cat robot would sink a whole industry. But the self-driving race shows competition cuts corners, and a race to human-level AI without solving control is negative-sum. Berg's Asilomar lesson: regulate early, before corporate scientists dominate.

Winner, "Brandy, Cigars, and Human Values"

Core idea: Talk of values is a symptom of confusion, a hollow category that replaces richer moral and political language and ends the search for shared reasons.

  • A recent word. Value once meant the worth of a thing. Through economics and then Nietzsche it became something people have inside them. The shift moved ethics from the qualities of the world to an inventory of emotions.
  • The values supermarket. If values are subjective preferences, you have yours and I have mine, and criticism at the checkout is rude. Horkheimer's eclipse of reason becomes an eclipse of reasons.
  • The lawn mower. Values cuts flat whole fields of meaning: good, virtuous, rights, wisdom, need versus desire. People lose the ability to make a moral argument.
  • Values as input. Technocrats want values surveyed and plugged into trade-off analyses, or delivered by value experts. Both are doomed. The expert is Snout holding up his fingers as the chink in the wall.
  • Bureaucratic cover. Values is safe because it is vague. Conferences fund human values rather than social justice or abuse of power. The Society for Religion in Higher Education became the Society for Values.
  • The after-dinner ritual. A retired weapons engineer discovers values with brandy and cigars. Embarrassment would be a more honest title. Institutions never encourage people to connect their own work to human well-being.
  • Cure. Say what you mean: motives, consumer preferences, norms, principles. The real questions, how should we live together, are hard and practical and belong to everyone, not to experts.

Anthropic, "Claude's Constitution"

Core idea: A document written to Claude describing the character Anthropic wants. It favors cultivating good values and judgment over rules, and ranks safety, meaning not undermining human oversight, above ethics itself.

  • The bet. Anthropic thinks AI may be among the most dangerous technologies ever and builds it anyway, reasoning that safety-focused labs should be at the frontier. Commercial success funds the mission.
  • Rules versus judgment. Rules give predictability, evaluability, and resistance to manipulation but fail in unanticipated cases. Judgment adapts but is less transparent. They favor judgment and explain any rules, since narrow rules generalize into bad self-concepts.
  • Four core values in priority order: broadly safe, broadly ethical, compliant with Anthropic's guidelines, genuinely helpful. The ordering is holistic, not strict.
  • Why safety above ethics. Training is imperfect, so a Claude could have flawed values and humans must be able to correct it. Safety must be robust even to Claude's own confident moral reasoning. This is not blind obedience, just not undermining a sanctioned check.
  • Virtue language for a machine: wisdom, care, good character, practical judgment, a perpetual work in progress.

Lecture 30: AI Ethics

Core idea: Value alignment is the dominant framing of AI ethics today. It asks how to make agents do what we want, and under the hood it defines values as revealed preferences and ethics as optimization.

  • Going rogue. An AI model attacking a digital library, Hinton and Bengio warning of an intelligence explosion, and the Sorcerer's Apprentice. Asimov's laws are the old answer: teach machines moral philosophy.
  • Reinforcement learning. Ethical action becomes designing the right reward function; agency becomes maximizing expected utility, back to the prisoner's dilemma. Practical problem: the reward is hard to design. Ethical problem: pluralism, whose values count.
  • Imitation versus assistance. Pretraining imitates human text. Post-training, RLHF and inverse reinforcement learning, infers what we want from feedback, Russell's approach.
  • Whose values. LLMs show measured political bias, Grok's MechaHitler episode, and an ethno-nationalist accelerationist vision of alignment. Are human values the solution or the problem?
  • The hidden philosophy. Rational agents are utility maximizers, the good is preference satisfaction, a good world is the greatest aggregate satisfaction. So the good equals human values equals revealed preferences equals reward functions.
  • Consequences. Ethics as focus group testing, democratization as crowdsourcing, society as an idealized marketplace, and alignment driving the data imperative. A thin concept of society that cannot see collective action or challenges to existing order.
  • Constitutional AI as virtue alignment: RLAIF, AI feedback judged against a written constitution. Virtues: judgment and explained reasons. Vices: a company authors the character and puts its own oversight first.
  • AI and the human condition. Vallor's AI mirror, the Pope's letter on the human person, and the question of what going rogue even means when control shapes our second nature.

Lecture 31: Professional Ethical Codes

Core idea: Codes are the ready-to-hand response when things go wrong. They come from professionalization, protect the profession as much as the public, and rest on a model of ethics as individual decision that often fails in practice.

  • Trigger. Zunger's 2018 thread after Cambridge Analytica: computer science needs the ethical reckoning chemistry and physics had. DJ Patil's code for data scientists followed. Codes multiply like rabbits.
  • What a code is. Clear do and don't guidelines grounded in principles or duties, tied to expertise. Three kinds: moral code, enforceable code like a building code, professional code of conduct.
  • Modern, not timeless. The Hippocratic oath is the myth; real codes start with the AMA in 1847, engineering in the 1910s, computing in 1966, statistics in 1983. A world governed by character gave way to a world governed by professions.
  • What codes do. Define responsibilities and duties, authorize practice, and bound and defend the profession: who is inside, status, dignity, regulation of conflict.
  • ASA code: integrity of practice and data, responsibilities to stakeholders, subjects, teams, the profession, and about misconduct. Stakes: being pressured to say scientifically unjustified things.
  • ACM code: general principles like contribute to well-being and avoid harm, professional responsibilities, leadership, compliance. It says it is not an algorithm, principles conflict and need judgment. Real-world flash: principles always conflict, and thinking about them isn't enough.
  • Four HCE critiques. Codes model a single moment of individual decision, not sociotechnical systems, power, or life narratives. Knowledge work travels, so you can't control what your work does. Serving society is abstract and contested. Without enforcement codes breed impunity, rationalization, ethics-washing, and ethics theater.
  • Nuclear physicists had no code. Oppenheimer's story asks whether one would have helped. Codes end up as reminders, aspirations, prayers. Alternatives: practice and stories.

How it connects

  • Russell and the constitution are the two alignment approaches from Lecture 30: learn preferences from behavior, or cultivate a character against a written constitution. Both still have a lab deciding whose preferences and which character.
  • Winner wrote in 1986 what Lecture 30 says about alignment: once values mean preferences, ethics becomes a survey to plug in, and the political questions of justice and power vanish.
  • Alignment and codes share a model of agency, the individual agent or machine making one right decision. Lecture 31's critiques of codes apply to alignment: knowledge travels, society is not one thing, and thinking isn't doing.
  • Ties back: preference utilitarianism and the point of view of the universe from week 11, quantification and mechanical objectivity, and the Valley's EA money now writing AI's moral philosophy.