Threat detection in security logs: a comparison between a rules-based approach and analysis with LLM

Abstract

This paper aims to systematically compare two approaches for detecting and classifying malicious payloads in HTTP parameters: the classical rule-based approach using regular expressions, and the use of large language models (LLMs). This study used the HttpParamsDataset, which contains 31,067 records divided into five classes: normal traffic, SQL injection, Cross-Site Scripting, path traversal, and command injection. Seven models from two providers were tested — Anthropic (Claude Haiku 4.5, Sonnet 4.6, and Opus 4.6) and OpenAI (GPT-4o-mini, GPT-4.1-mini, GPT-4.1, and GPT-5.4) — in two modalities: pure textual analysis and dynamic regex script generation. The baseline was a static engine with 40 regex rules inspired by the OWASP ModSecurity Core Rule Set. Experiments were run over 30 sub-samples of 500 records each, with paired statistical tests for validation. Results show that LLM textual analysis outperforms the rule-based approach in almost every scenario: Claude Haiku 4.5 reached Macro-F1 of 0.967 against 0.867 for the rule engine (p < 0.0001), at a cost of US$ 1.96 for the 30 sub-samples. When LLMs were asked to generate regex scripts, performance dropped below the hand-written engine across all tested models. Total cost of the 15 experiments was US$ 35.81, with 9.8 hours of aggregate computation time. Since the experiments are statistically independent, they were run in parallel (4 concurrent processes for Anthropic and 2 for OpenAI, with both providers running in parallel to each other), reducing the effective wall-clock time to approximately 3.5 hours. The findings suggest that, for payload classification, the direct use of LLMs as textual classifiers offers better cost-effectiveness than approaches based on static rules or automatic regex generation, indicating that hybrid systems — with regex rules as a first filtering layer and LLMs for ambiguous cases — represent a promising direction for production environments.