top of page
Wireframe - 8.png

Asking the Right Questions: Can expert-prompted LLMs reformulate legal queries from non-experts?

Katherine Atwell, Morgan A. Gray, Jaromir Savelka, Len Rial, Sera Linardi, Malihe Alikhani

Conference Paper

|

July 3, 2026

Large language models are widely used by everyday users, and can be asked to perform tasks that require specialized expertise, such as interpreting contractual terms and conditions, filing personal taxes, or diagnosing medical symptoms. Although these tools should not be used in place of professional advice, they can be useful starting points for users seeking professional help, improving users’ access and interactions with professionals. In this vein, this paper introduces a legal question reformulation task to assist non-experts in their interactions with lawyers. This has the potential to streamline discussions between lawyers and clients, who may not know the correct legal language to communicate their needs. Using a novel evaluation framework informed by legal expertise, we investigate the quality of model-generated legal question reformulations on in-the-wild data from non-experts seeking legal advice. Our findings indicate that LLMs have significant potential in legal reasoning, but some unexpected safety concerns may emerge. Further, adding linguisticallyaligned in-domain text samples can improve performance for smaller models, even when the samples are not aligned factually with the given question.

citation

Atwell, K., Gray, M. A., Savelka, J., Rial, L., Linardi, S., & Alikhani, M. (2026). Asking the Right Questions: Can expert-prompted LLMs reformulate legal queries from non-experts? Proceedings of the Second Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U), 167-181. https://doi.org/10.18653/v1/2026.customnlp4u-1.15

topics

Artificial Intelligence

Artificial Intelligence

Artificial Intelligence

Artificial Intelligence

related work

Rectangle 35

Measuring Agency-Amplifying AI Literacy: The HuskyAI Coaching Platform for Workforce Readiness

July 2026

HuskyAI is a web platform on which students develop AI knowledge by working through authentic

Artificial Intelligence

Artificial Intelligence

Working Paper

Rectangle 35

Measuring Agency-Amplifying AI Literacy: The HuskyAI Coaching Platform for Workforce Readiness

July 2026

HuskyAI is a web platform on which students develop AI knowledge by working through authentic

Artificial Intelligence

Artificial Intelligence

Working Paper

Rectangle 35

Measuring Agency-Amplifying AI Literacy: The HuskyAI Coaching Platform for Workforce Readiness

July 2026

HuskyAI is a web platform on which students develop AI knowledge by working through authentic

Artificial Intelligence

Artificial Intelligence

Working Paper

bottom of page