Avaliação de Grandes Modelos de Linguagem para Raciocínio em Direito Tributário

Abstract

Tax law is essential for regulating relationships between the State and taxpayers, being crucial for tax collection and maintaining public functions. The complexity and constant evolution of tax laws make their interpretation an ongoing challenge for legal professionals. Although Natural Language Processing (NLP) has become a promising technology in the legal field, its application in brazilian tax law, especially for legal entities, remains a relatively unexplored area. This work evaluates the use of Large Language Models (LLMs) in Brazilian tax law covering federal tax aspects, analyzing their ability to process questions and generate answers in Portuguese for legal entities’ queries. For this purpose, we built an original dataset composed of real questions and answers provided by experts, allowing us to evaluate the ability of both proprietary and open-source LLMs to generate legally valid answers. The research uses quantitative and qualitative metrics to measure the accuracy and relevance of generated answers, capturing aspects of legal reasoning and semantic coherence. As contributions, this work presents a dataset specific to the tax law domain, a detailed evaluation of different LLMs’ performance in legal reasoning tasks, and an evaluation approach that combines quantitative and qualitative metrics, thus advancing the application of artificial intelligence in the analysis of tax laws and regulations.