AutoMutate++: Explaining AV/EDR Detection Through Controlled Mutation

Across the repository’s 180-round selector comparison, genetic search was the only strategy to produce successful classification changes, while coverage guidance delivered the most trustworthy executions. The central result was counter-intuitive: more mutation was not better.

Problem

Endpoint Detection and Response products expose a verdict, but rarely the artifact or runtime evidence behind it. For defenders and researchers, understanding why a detection changed often means manually rebuilding a sample, rerunning it in a lab, comparing telemetry, and trying to separate a meaningful signal from a broken artifact or unstable execution.

That workflow is slow, difficult to reproduce, and vulnerable to false conclusions. A failed build, an instrumented trace, or a flaky worker can look like a detection change unless the experiment treats reliability as a first-class result.

Research objective

AutoMutate++ automates controlled experiments that ask which observable software characteristics correlate with a detection change. It does not stop at producing a changed verdict. The system builds and executes each candidate in isolation, collects runtime and EDR telemetry, classifies the outcome through differential testing, extracts normalized behavioural tokens, and feeds the evidence into later experiments.

This work is authorised, lab-only defensive research. The evaluated artifacts are non-operational but behaviourally representative; implementation details that would turn the case study into an evasion recipe are intentionally omitted.

My contribution

I designed and implemented the closed-loop framework during my Master’s thesis at armasuisse’s Cyber-Defence Campus:

• the Rust controller and orchestration model, exposed through five typed gRPC services and 25 RPCs;
• the cross-compilation and artifact pipeline spanning source AST, LLVM IR, and PE binaries;
• isolated Windows worker execution, behavioural monitoring, and telemetry collection;
• the three-run differential classifier that separates meaningful outcomes from broken or unstable executions;
• normalized token extraction, correlation scoring, and feedback-driven mutation selection;
• the evaluation and validation infrastructure used to compare four selector strategies.

System architecture

The controller coordinates a builder, isolated Windows workers, and an Elasticsearch-backed experiment store. A UI or CLI submits campaign configuration over gRPC; the builder assembles and instruments artifacts; workers execute them under Hyper-V or remote isolation; the controller triages results and records runs, rounds, telemetry, tokens, and artifacts for later analysis in Kibana.

Three-layer mutation pipeline

The mutation engine operates at three different abstractions, allowing an experiment to vary source structure, compiler representation, or final executable structure without conflating them:

Reliable experimentation

Every candidate is evaluated with a three-run differential protocol. The sequence distinguishes environmental or instrumentation effects from a candidate that genuinely changes the observed outcome:

Workers execute the artifact through ten controlled phases while six telemetry sources capture execution, EDR, and system evidence. Persistent experiment records make every classification traceable back to its configuration, artifacts, and raw observations.

Feedback-guided search

Telemetry is normalized into nine behavioural token categories. The scoring layer measures each token’s detection correlation and confidence, then converts sufficiently strong evidence into avoid or seek guidance for later candidates.

I evaluated four ways of choosing the next mutation recipe:

• Coverage-guided selection prioritizes under-explored transformations.
• Genetic selection evolves higher-scoring candidate recipes.
• Token-guided selection follows correlations learned from earlier telemetry.
• Random selection provides an unguided baseline.

Results

The repository’s selector-comparison evaluation contains 180 rounds: 39 coverage-guided, 51 genetic, 40 token-guided, and 50 random. The strategies exposed a clear trade-off between effectiveness and experimental reliability.

The genetic strategy was the only selector to produce successful classification changes: 6 across 51 rounds, an 11.8% rate, with a maximum evaluation score of 0.899. Coverage guidance did not produce the same outcome, but 87.2% of its rounds were trustworthy and only 10.3% failed during mutation.

Mutation complexity explained the reliability gap

Uniform random search selected far larger recipes than the guided approaches. Its average of 7.3 AST transformations per round coincided with a 68% mutation-failure rate. Genetic search found the only successful candidates with an average recipe size of just 1.9 transformations.

The result is more useful than a simple winner ranking. Genetic search was more effective; coverage guidance was more reliable; random search was both complex and fragile. It shows that a productive experimental loop must optimize for interpretable, executable candidates, not merely maximize the number of transformations.

Engineering quality

Beyond the selector comparison, the framework was validated through 42 integration tests across 13 test files and six component-level evaluation experiments. The tests cover the build pipeline, differential classification, telemetry handling, token extraction, and selector behaviour.

Typed gRPC boundaries, deterministic campaign configuration, isolated execution, persistent experiment records, and reproducible datasets make the results auditable instead of reducing them to a one-off demonstration.

Limitations

• The 180-round comparison is a bounded repository evaluation, not proof that one selector is universally superior.
• Results depend on the evaluated artifacts, configurations, telemetry visibility, and product versions.
• Combining several transformations can make causal attribution difficult, even when a candidate changes classification.
• Stochastic execution and worker instability reduce the number of trustworthy observations.
• The available material does not establish that every selector-comparison round was a live execution against a named commercial product, so the results are presented as controlled evaluation outcomes rather than product-specific bypass claims.
• Ethical and operational constraints limit which implementation details and artifacts can be published.

The source repository contains the framework, evaluation documentation, and sanitised result material.