Rule enforcement in LLMs: a parameter efficient fine-tuning approach with self-generated training dataset

Large Language Models (LLMs) often have implicit knowledge of domain-specific rules, such as age requirements for obtaining a driver’s license, but may not consistently apply this knowledge in conversations. In this paper, we explore a method for fine-tuning LLMs using datasets generated by the LLM itself. The goal is to explicitly enforce specific rules, such as declaring ineligibility if the age requirement is not met, within a defined context. We evaluate whether this fine-tuning approach enables the model to recognize the need to apply relevant knowledge in other contexts, such as marriage eligibility, where the LLM already has knowledge of the underlying criteria. Our results show that after fine-tuning, the LLM not only applies the rule in the training contexts, but also generalizes this behavior to enforce the rule in different domains. This suggests that fine-tuning, even with self-generated datasets, can improve the ability of the LLM to apply its knowledge more consistently, leading to more reliable performance in rule-based scenarios.

reference talk

 Rule enforcement in LLMs: a parameter efficient fine-tuning approach with self-generated training dataset (AIxHMI 2024 @ AIxIA 2024, 26/11/2024) — Daniele Franch (Daniele Franch, Pierluigi Roberti, Enrico Blanzieri)

origin event

 AIxHMI 2024 @ AIxIA 2024

journal or series

 CEUR Workshop Proceedings (CEUR-WS.org)

funding project

 ENGINES — ENGineering INtElligent Systems around intelligent agent technologies (28/09/2023–27/02/2026)

reference publication for talk

 Rule enforcement in LLMs: a parameter efficient fine-tuning approach with self-generated training dataset (AIxHMI 2024 @ AIxIA 2024, 26/11/2024) — Daniele Franch (Daniele Franch, Pierluigi Roberti, Enrico Blanzieri)