Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Agentic Method for Deterministic Validation of Legacy Code Migration

Source

arXiv

arxiv.org/abs/2607.28271v1

Read the source

Source

Plain-Language Brief

The paper proposes a new method called 'Locksmith Loop' for testing the migration of COBOL programs to Java. This method is designed to improve coverage beyond input-search plateaus, reaching near complete coverage in two open-source programs and 91.90% branch coverage in an internal production-like COBOL program. The method uses mocks, parity-preserving mutations, and a deterministic oracle for validation.

What the Document Says

  • The paper discusses the migration of COBOL programs to Java and the challenges of ensuring correct functionality due to lack of test data and difficulty in validating all corner cases.
  • A new method called 'Locksmith Loop' is proposed for overcoming these challenges.
  • The method involves preparing two runtime environments, instrumenting each with mocks, and executing off-mainframe on commodity hardware.
  • An iterative agentic loop performs Witness Search and parity-preserving mutations.
  • When routing boundaries are reached, an analyzer identifies a Locked Paragraph.
  • The method was tested across three COBOL-Java case studies.
  • Locksmith improved coverage beyond input-search plateaus in all case studies.
  • Near complete coverage was achieved on two open-source programs and 91.90% branch coverage on the internal production-like COBOL program.
  • The generated Java matched the COBOL reference under deterministic parity checks in all accepted test cases.
  • The method is claimed to be a novel approach for validating agentic coding output using a deterministic oracle.

Key Concepts Mentioned

  • Legacy Code Migration
  • COBOL programs
  • Java
  • Testing
  • Test Data
  • Corner Cases
  • Witness Search
  • Parity-Preserving Mutations
  • Deterministic Oracle

Claims Made by the Document

  • Locksmith improves coverage beyond input-search plateaus.
  • Locksmith reaches near complete coverage on open-source programs.
  • Locksmith achieves 91.90% branch coverage on internal production-like COBOL program.
  • Generated Java matches COBOL reference under deterministic parity checks.

Evidence Used

  • Improved coverage results from three COBOL-Java case studies were provided.

Methods / Approach

  • Locksmith Loop
  • Witness Search
  • Parity-Preserving Mutations

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The paper does not discuss potential limitations or drawbacks of the Locksmith Loop method.

Open Questions

  • Are there any known limitations or drawbacks to the Locksmith Loop method?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper does not provide information about the scalability, efficiency, or applicability of the Locksmith Loop method beyond COBOL-Java migrations.

Raw Abstract / Excerpt

Migration of legacy COBOL programs to Java requires extensive testing to ensure correct functionality. This effort is often complicated by the lack of test data and the difficulty of validating all corner cases. In this paper we propose a novel agentic test-synthesis method, the 'Locksmith Loop,' which is initiated by preparing two runtime environments: the COBOL source and the generated Java target are each instrumented with mocks and executed off-mainframe on commodity hardware, then an iterative agentic loop performs Witness Search over input mocks to penetrate program branches, followed by parity-preserving mutations. When routing boundaries are reached, an analyzer identifies a Locked Paragraph: a condition preventing deeper exploration. Across three COBOL-Java case studies, spanning two open-source programs and one internal production-like COBOL program and ranging from 430 to 4,114 source lines, Locksmith consistently improved coverage beyond input-search plateaus, reaching nearly complete coverage on the two open-source programs and 91.90% branch coverage on the internal production-like COBOL program. The generated Java matched the COBOL reference under deterministic parity checks in all accepted test cases. Through these findings we demonstrate, to the best of our knowledge, a novel approach for validating agentic coding output using a deterministic oracle.