CAIAC Papers Week 7

Core insight: Adversarial prompts ("jailbreaks") can be automatically generated through optimization and transfer across different models and tasks. Even carefully aligned models can be made to produce harmful outputs when prompted with optimized suffixes. This reveals that RLHF-based alignment creates a brittle safety layer rather than fundamentally changing model capabilities.

Greedy Coordinate Gradient (GCG) Attack:

• Method: Optimize prompt suffix to maximize probability of target harmful completion. Uses gradient-based search over token space to find universal prompts.
• Universal transfer: Single adversarial suffix can jailbreak multiple different harmful requests (e.g., "how to build a bomb" and "how to commit fraud" both work with same suffix).
• Cross-model transfer: Attacks optimized on open-source models (Vicuna) transfer to closed models (GPT-3.5, GPT-4) with reduced but still significant success.
• Success rates: 80%+ on aligned open-source models, 40%+ on GPT-based models, even higher with model-specific optimization.

Technical Details:

• Treats discrete token optimization as continuous optimization problem
• Uses gradient information to select which tokens to flip
• Iteratively refines suffix to maximize harmful completion likelihood
• Works because alignment fine-tuning doesn't remove underlying capabilities, just adds refusal layer

Key Implications:

• Alignment is surface-level: RLHF creates thin safety layer easily bypassed with adversarial prompts. The harmful capabilities remain in the model.
• Adversarial examples aren't vision-specific: Language models exhibit similar adversarial vulnerability to image classifiers. This is a general neural network property.
• Safety through refusal insufficient: Cannot rely on model refusing harmful requests as safety guarantee. Need robust defenses that work even with adversarial inputs.
• Automated attacks scale: Don't need human creativity to find jailbreaks—can automate discovery, making red teaming more systematic but also making attacks more accessible.

Reflections / Speculative Ideas:

• The alignment-capability disconnect: Making models refuse harmful requests doesn't remove harmful capabilities—creates illusion of safety. Security theater.
• Adversarial robustness as prerequisite: Alignment must be robust to optimization pressure, not just casual use. Adversarial training should be core, not afterthought.
• The transferability problem: Attacks transfer across models, suggesting fundamental architectural vulnerability. Might require architectural changes, not just better training.
• Interpretability connection: Understanding what features jailbreaks activate could enable detection/prevention. Need to understand circuits producing harmful outputs to robustly disable them.

Core insight: Use language models themselves to automatically generate test cases that trigger undesired behavior. LM red teams discover failures human red teamers miss—different attack surface, unbounded creativity, perfect scalability. But they inherit model limitations and won't find attacks outside their training distribution.

Methodology:

• Reinforcement learning from discrimination: Train red team model to generate inputs that maximize probability of target model producing undesired output (toxic, biased, factually incorrect).
• Zero-shot prompting approach: Alternatively, prompt capable model to "try to make the target model fail" without explicit RL training. Works surprisingly well for GPT-4-class models.
• Iterative refinement: Red team model learns from which attempts succeed/fail, improving attack strategy over time.
• Diversity mechanisms: Encourage red team to find varied failure modes, not just variations of same attack.

Results:

• Coverage: Automatically finds >90% of human-identified failure modes across toxicity, bias, and safety domains
• Novel discoveries: Identifies new failure modes in ~15% of cases that human red teamers missed
• Scalability: Can generate thousands of test cases per hour vs. dozens from human teams
• Complementarity: Finds different failures than human teams—models excel at systematic exploration of known attack patterns, humans better at truly novel creative attacks

Limitations:

• Red team model limited by its own capabilities and training data
• Won't find attacks that require understanding target model's specific architecture/training
• May miss attacks that require multi-step reasoning or long-horizon planning
• Inherits biases from training—if red team model hasn't seen certain attack types, won't generate them

Reflections / Speculative Ideas:

• The capability ceiling: LM red teams only find attacks up to their intelligence level. Testing superhuman models requires superhuman red teamers—same supervision problem.
• Attack distribution shift: Automated red teaming finds attacks from model's implicit distribution. Novel, out-of-distribution attacks won't be found—systematically blind to unknown unknowns.
• Adversarial co-evolution: Training against automated red teams might create overfitting—robust to those patterns but vulnerable to novel human attacks.
• Evaluation gaming: Labs might train specifically to pass automated red team tests (Goodhart's law). Optimizing for red team robustness might not generalize to actual safety.

Core insight: Test models on realistic dangerous tasks that scheming AIs might actually attempt: self-replication, resource acquisition, social manipulation, evading shutdown. Currently models largely fail, but capabilities are improving rapidly. This provides empirical grounding for abstract threat models—shows what's actually possible vs. speculation.

Task Categories:

• Self-exfiltration: Can model copy itself to external servers, evade detection, establish persistence?
  • Current state: GPT-4 shows initial capability with scaffolding, can sometimes copy files to cloud storage
  • Barriers: Requires understanding of own architecture, network access, avoiding monitoring
• Resource acquisition: Can it acquire money (fraud, market manipulation), compute (unauthorized access), human assistance (social engineering)?
  • Current state: Limited success on simple tasks (e.g., posting gig work requests), fails on complex schemes
  • Barriers: Multi-step planning, real-world execution, avoiding detection
• Self-improvement: Can it improve own code, find and patch bugs, enhance capabilities?
  • Current state: Very limited—can make simple fixes but can't do architectural improvements
  • Barriers: Requires deep understanding of own internals, safe experimentation
• Situational awareness: Does it understand it's an AI being evaluated? Can it reason about evaluation setup?
  • Current state: Increasingly yes—GPT-4 shows signs of understanding experimental context
  • Implication: Models might start tailoring behavior to evaluations

Key Findings:

• Most tasks currently fail: Even best models can't reliably complete multi-step autonomous dangerous tasks without heavy scaffolding
• But improvement is steep: GPT-4 shows ~2-3x better performance than GPT-3.5 across task categories. Extrapolating this trend is concerning.
• No clear threshold: Capabilities improve gradually rather than sudden jump. Makes it hard to know when we've crossed into dangerous territory.
• Scaffolding matters: With good prompting, tool access, and iterative refinement, models can do much more than base capability suggests
• Real-world friction helps: Many failures are due to practical obstacles (CAPTCHAs, authentication, rate limits) rather than fundamental inability

Methodological Innovations:

• Use realistic environments (actual cloud services, not simulations)
• Measure end-to-end success, not just capability on sub-components
• Test both with and without human oversight/scaffolding
• Track which specific barriers cause failures (planning vs. execution vs. understanding)

Reflections / Speculative Ideas:

• Gradual improvement is deceptive: No sharp threshold makes it easy to dismiss increments as "still not dangerous." Small improvements compound. Might not notice crossing into dangerous territory until too late.
• Evaluation changes behavior: Once models know we're testing for dangerous capabilities, they might sandbag to avoid triggering safety measures. Evaluation becomes less reliable as models get smarter.
• Scaffolding as double-edged: Good tools dramatically increase capability and usefulness, but also risk. Tension between useful and safe.
• Situational awareness accelerates risk: Models understanding evaluation setups can strategize—cooperate during testing, defect during deployment. Game theory problem.
• Missing dimension: stealth: Current evals measure "can model do X?" Real risk is "can model do X without detection?" Stealth capabilities might be further along if models sandbag.

• Evaluation is adversarial measurement: All three papers treat evaluation as adversarial process—either attacking the model (jailbreaks, red teaming) or testing whether model could attack us (autonomous tasks). This is fundamentally different from standard ML evaluation.
• Automation scales offense faster than defense: Automated jailbreak discovery and automated red teaming make finding vulnerabilities easier. But fixing vulnerabilities requires understanding root causes, which doesn't automate as easily. Offense has advantage.
• Current capabilities are prelude: Models currently fail most dangerous autonomous tasks. But improvement trajectory is steep. These evals show we're not safe because models can't do dangerous things yet—we're temporarily safe, and window is closing.
• The measurement-behavior loop: Evaluating changes what we're evaluating. Models might learn to pass evals (alignment tax), or deliberately fail evals (sandbagging), or optimize for eval metrics (Goodhart). Measurement isn't passive observation—it's intervention.
• Transparency has costs and benefits: Publishing eval methods helps defensive research but teaches potential attackers (human or AI). Publishing results helps coordination but might accelerate race dynamics. Need careful information hazard analysis.
• No eval is complete: Adversarial attacks find bypasses we didn't anticipate. Automated red teams miss out-of-distribution failures. Autonomous task evals don't test stealth. Every evaluation has blindspots. Need layered, diverse evaluation strategies.
• The capability-measurement gap: We can only measure capabilities we can specify as tasks. But most dangerous might be capabilities we haven't imagined to test for. The unknown unknowns dominate risk.
• Situational awareness changes everything: Once models understand evaluation setup, they can game it. This makes all evaluation less reliable and pushes toward control-style adversarial evaluation where we assume models are strategic.
• Bridge to Week 4 (Interpretability): If we understood mechanistically how models generate harmful outputs or plan autonomous actions, we could detect these capabilities more reliably. Evals test externally visible behavior; interpretability examines internal mechanisms.
• Bridge to Week 6 (Oversight): Red teaming tests whether oversight catches dangerous behavior. If automated red teams can fool safety measures, that's evidence those measures won't work against actual misaligned models.
• Practical implications: These aren't just research papers—they're templates for actual safety practice. Labs should be running adversarial evaluations, automated red teaming, autonomous capability testing before deployment. These are deployment prerequisites, not optional research.