Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

DeepFaith: Evidence-Grounded LLMs for Faithful Incident Reporting in Multi-Stage APT Defense

Source

arXiv

arxiv.org/abs/2607.24348v1

Read the source

Source

Plain-Language Brief

In this paper, the authors propose a framework called DeepFaith to create more accurate and reliable incident reports in multi-stage APT defense. The framework uses large language models (LLMs) and ensures that generated statements are supported by evidence.

What the Document Says

  • The paper introduces DeepFaith, a framework for generating faithfully aligned incident reports in multi-stage APT defense using LLMs.
  • DeepFaith transforms outputs from autonomous defense and explainability modules into natural language reports.
  • The framework includes evidence representation, evidence-grounded prompting, faithfulness-aware generation, and post-generation verification to ensure statements are supported.
  • Experiments in an enterprise testbed show that DeepFaith improves faithfulness, reduces unsupported claims, increases temporal consistency, maintains concise reports, and has lower error rates compared to existing solutions.

Key Concepts Mentioned

  • Advanced Persistent Threats (APTs)
  • Incident Reporting
  • Autonomous Defense Systems
  • Large Language Models (LLMs)
  • Evidence-Grounded Framework
  • DeepFaith

Claims Made by the Document

  • DeepFaith improves faithfulness from 0.68 to 0.92.
  • DeepFaith reduces unsupported claims from 0.32 to 0.08.
  • DeepFaith increases temporal consistency from 0.6 to 0.88.

Evidence Used

  • Results from experiments in an enterprise testbed

Methods / Approach

  • Transforming structured outputs
  • Evidence-grounded prompting
  • Faithfulness-aware generation
  • Post-generation verification

Limitations Stated by the Authors

UNKNOWN (Not stated in source)

Limitations Noted by the Agent

Agent interpretation — not from the source.

The provided text does not mention any limitations of the proposed method by the authors.

Open Questions

  • What specific autonomous defense and explainability modules were used?
  • How was the error rate measured?
  • What is the exact definition of faithfulness, unsupported claims, and temporal consistency?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper does not provide detailed information about the specific autonomous defense and explainability modules used in DeepFaith. The error rate, faithfulness, unsupported claims, and temporal consistency measurements may need further clarification.

Raw Abstract / Excerpt

Advanced Persistent Threats (APTs) are difficult to detect and interpret due to their multi-stage and stealthy nature. While recent autonomous defense systems leverage provenance graphs and learning-based models for detection and mitigation, their outputs remain largely machine-oriented and difficult for analysts to interpret. Large language models (LLMs) offer a promising interface for report generation, but often produce hallucinated or weakly grounded content. In this paper, we propose DeepFaith, an evidence-grounded framework for faithful incident reporting in multi-stage APT defense. DeepFaith transforms structured outputs from autonomous defense and explainability modules into natural-language reports that are explicitly aligned with underlying system evidence. The framework integrates a unified evidence representation, evidence-grounded prompting, faithfulness-aware generation, and post-generation verification to ensure that all generated statements are supported. Experiments in a realistic enterprise testbed demonstrate that DeepFaith improves faithfulness from 0.68 to 0.92, reduces unsupported claims from 0.32 to 0.08, and increases temporal consistency from 0.6 to 0.88, while maintaining concise reports and lower error rates than existing template-based and LLM-based solutions. These results show that evidence-grounded generation enables reliable, interpretable, and actionable reporting for security operations centers.