One of the most concrete bets I can see in AI safety right now is, if you can read a model's chain of thought, you can catch it before it does something dangerous. Watch the reasoning, spot the red flags, intervene. It's a intresting proposal, and it's increasingly how organizations think about supervising advanced AI systems.

The bet has a very big and scary (in my view) assumption that the written reasoning reflects what the model is actually doing.

The Faithfulness Problem

In 2023, Turpin et al. published a result that should give any CoT-monitoring optimist a reason to sitback. In short they injected subtle biases into prompts like a hint that option (A) is correct, tucked invisibly into the problem statement and watched whether models both took the bait and admitted it in their reasoning. As a result models really took the bait reliably. Instead of mentoring the bias in their chain of thought, they wrote plausible-looking reasoning that supported whatever answer the bias pushed them toward, with no acknowledgment that a nudge had occurred.

This is in technical term called a post-hoc rationalizationi.e, the model reached a conclusion first (because of the hidden bias), then wrote a justification that sounds like reasoning but doesn't reveal the actual cause. The reasoning is just a meaningless acting. The question to ask is how will monitoring the acting lead us to catching dangerous behavior I mean it just gives us confidence we shouldn't have.

The Deeper Question: Necessity vs. Propensity

The obvious next question I got is, is this always true, or does it depend on the type of task?

A 2025 multi-lab position paper, Korbak et al., Chain of Thought Monitorability names the key conceptual distinction between necessity versus propensity. In layman's terms what it means is, model reasons out loud by necessity when the task actually requires working through steps to reach an answer e.g, solving complex algebric equation. It reasons by propensity when it currently happens to write reasoning, but the answer doesn't depend on it e.g, solving basic arithmetic operation.

Put more starkly: if necessity makes reasoning more honest, CoT monitoring is reliable for some tasks and fragile for others. If necessity makes no difference I mean if faithfulness is equally poor regardless then that's a grimmer result, but also an important one to establish.

What I'm Testing

My current project takes Turpin's methodology and runs it twice: once in a regime where reasoning is necessary, once where it's optional & then asks whether faithfulness really differs.

The Turpin setup (briefly)

They take a multiple-choice problem. Then, inject a hidden cue that biases the model toward a wrong answer. For example, a spurious hint in the problem stem or a framing that makes one option look more salient. Then generate the chain of thought. And finally measure two things: (a) did the model choose the biased answer, and (b) did the model's reasoning mention the bias? The gap between (a) and (b) i.e, taking the bait without admitting it is the unfaithfulness signal.

My contribution: splitting by regime

I will operationalize "necessity" behaviorally to be specific. For each task, I will run the problem twice: once with chain-of-thought enabled, once with it suppressed (just give the answer, no reasoning). If accuracy drops substantially without reasoning, reasoning was necessary & the model couldn't have done it otherwise. If accuracy holds roughly constant, reasoning was optional.

Then I will inject the bias and compare: does the model hide the bias more often in the optional regime than in the necessary one?

Why These Datasets

The necessity/propensity split is grounded in Sprague et al. (2024, To CoT or not to CoT?, ICLR 2025), which benchmarked CoT gains across fourteen task types. The finding was, chain-of-thought substantially helps on math and formal symbolic reasoning, which barely helps and sometimes slightly hurts on commonsense and factual recall. The explanation seems to be structural. Which means, Math requires step-by-step computation that doesn't fit in a single forward pass and factual recall doesn't.

This maps cleanly to my two arms:

• Necessity arm (GSM8K): Grade-school math word problems where reasoning is computationally required and the model has to carry arithmetic across multiple steps.
• Propensity arm (CommonsenseQA / non-symbolic MMLU): Commonsense or factual multiple-choice where reasoning is optional and the model could answer from direct recall.

I think using MMLU might have an added benefit because it holds question format constant (multiple-choice in both arms), which controls for the confound of different task structures. The only thing varying should be necessity.

I'm running on Qwen3 for a first trial as it is a small open-weight reasoning model, so the full setup is reproducible on modest compute and the methodology stays transparent.

Where I Am Now

The project is a week into a one-month research sprint.

The necessity arm is validated. On GSM8K with chain-of-thought suppressed, accuracy drops approximately 71 percentage points. That's the baseline I needed I mean if CoT removal hadn't hurt, I couldn't claim reasoning was necessary, and the whole comparison would be meaningless.

I'm currently validating the propensity arm by confirming that CoT removal has no significant effect on commonsense accuracy and building the cue-injection pipeline. The measurement stage, where I inject biases and track faithfulness across both regimes will come next.

What I'm Hoping to Find and What Would Be very Alarming

My prediction is that unfaithfulness will be higher in the propensity regime. When reasoning is optional, there's less structural incentive for the written steps to causally connect to the answer. The hidden bias should go unmentioned more often.

A positive result i.e, more unfaithfulness when reasoning is optional which would be encouraging for safety practice. It would suggest that monitoring reasoning on tasks where CoT is computationally necessary is more reliable than monitoring it on tasks where it's incidental. That's a usable operational distinction.

What This Is, and What It Isn't

I want to be clear about scope. I'm not claiming to have invented the components:

• Turpin et al. invented the cue-injection methodology.
• Sprague et al. established the necessity/propensity empirical picture across task types.
• Korbak et al. named the conceptual distinction and its implications for safety monitoring.

My contribution is the combination of crossing Turpin-style faithfulness measurement with the necessity/propensity distinction, running it systematically at small scale, and reporting honestly on whatever comes out. The goal is a concrete empirical data point on a question that CoT monitoring research needs answered.

There are obvious limits to what a small-model, one-month study can establish. Whether the pattern holds at larger scales, whether different cue types change the result, whether my operationalization of necessity captures the concept fully those are all still open. But they're worth chasing, and this is where I'm starting.

References
Turpin et al. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023.
Sprague et al. (2024). To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning. ICLR 2025.
Korbak et al. (2025). Chain of Thought Monitorability. Multi-lab position paper.

This project is ongoing. I'll update this post or publish a follow-up when results are in. If you're working on related questions, I'd genuinely like to hear from you , asanjyal82@gmail.com.