Rule enforcement in LLMs: a parameter efficient fine-tuning approach with self-generated training dataset

Large Language Models (LLMs) often have implicit knowledge of domain-specific rules, such as age requirements for obtaining a driver’s license, but may not consistently apply this knowledge in conversations. In this paper, we explore a method for fine-tuning LLMs using datasets generated by the LLM itself. The goal is to explicitly enforce specific rules, such as declaring ineligibility if the age requirement is not met, within a defined context. We evaluate whether this fine-tuning approach enables the model to recognize the need to apply relevant knowledge in other contexts, such as marriage eligibility, where the LLM already has knowledge of the underlying criteria. Our results show that after fine-tuning, the LLM not only applies the rule in the training contexts, but also generalizes this behavior to enforce the rule in different domains. This suggests that fine-tuning, even with self-generated datasets, can improve the ability of the LLM to apply its knowledge more consistently, leading to more reliable performance in rule-based scenarios.