
BASIL: Bayesian Assessment of Sycophancy in LLMs
Katherine Atwell, Pedram Heydari, Anthony Sicilia, Malihe Alikhani
Conference Paper
|
June 25, 2026
Sycophancy (overly agreeable or flattering behavior) poses a fundamental challenge for human-AI collaboration, particularly in high-stakes decision-making domains such as health, law, and education. A central difficulty in studying sycophancy in large language models (LLMs) is disentangling sycophantic belief shifts from rational changes in behavior driven by new evidence or user-provided information. Existing approaches either measure descriptive behavior changes or apply normative evaluations that rely on objective ground truth, limiting their applicability to subjective or uncertain tasks. We introduce a Bayesian probabilistic framework, grounded in behavioral economics and rational decision theory, that explicitly separates sycophancy from rational belief updating. Within this framework, we propose two group-truth-independent metrics for studying sycophancy: (i) a descriptive metric that measures sycophancy while controlling for rational responses to evidence, and (ii) a normative metric that quantifies how sycophancy leads models astray from Bayesian-consistent belief updating. Applying our framework across multiple LLMs and three uncertainty-driven tasks, we find robust evidence of sycophantic belief shifts and show that their impact on rationality depends on whether models systematically over- or under-update their beliefs, with most baselines demonstrating significant increases in error due to sycophancy when the model over-updates. Finally, we propose a novel post-hoc calibration method and two fine-tuning strategies that reward Bayesian-rational updating (BayesSFT and BayesDPO). We find evidence that post-hoc calibration significantly reduces Bayesian error, and observe significant reductions in both sycophancy and Bayesian error associated with our novel fine-tuning methods.
citation
Atwell, K., Heydari, P., Sicilia, A., & Alikhani, M. (2026). BASIL: Bayesian Assessment of Sycophancy in LLMs. Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency, 6613-6642. https://doi.org/10.1145/3805689.3812404
topics
Artificial Intelligence
Artificial Intelligence
Artificial Intelligence
Artificial Intelligence
related work

Measuring Agency-Amplifying AI Literacy: The HuskyAI Coaching Platform for Workforce Readiness
July 2026
HuskyAI is a web platform on which students develop AI knowledge by working through authentic
Artificial Intelligence
Artificial Intelligence
Working Paper


Measuring Agency-Amplifying AI Literacy: The HuskyAI Coaching Platform for Workforce Readiness
July 2026
HuskyAI is a web platform on which students develop AI knowledge by working through authentic
Artificial Intelligence
Artificial Intelligence
Working Paper


Measuring Agency-Amplifying AI Literacy: The HuskyAI Coaching Platform for Workforce Readiness
July 2026
HuskyAI is a web platform on which students develop AI knowledge by working through authentic
Artificial Intelligence
Artificial Intelligence
Working Paper

.png)