Mismatch between fuzz_targets/{trial}.fuzz_target and logs/xxx/{trial}.log as experimental results

[Once2gain]

Summary

There is a file mismatch between the generated fuzz target and the execution log in ss-fuzz-gen/results/output-xxx/. Specifically, the file fuzz_targets/{result.trial:02d}.fuzz_target does not correspond to the log file logs/run/{result.trial:02d}.log. This can cause confusion when analyzing results, as the log points to a crash from a fuzz target that has been overwritten by a later, non-compiling version.

This occurs when the first cycle successfully produces and runs a fuzz target, but the analysis stage fails, triggering a second cycle. If the second cycle fails to produce a compilable fuzz target, it overwrites the original fuzz target with the final, broken attempt, while the execution log from the successful first cycle is retained.

Sequence of Events

Cycle 1:

1. Writing Stage:

   • The OnePromptPrototyper successfully generates a compilable fuzz target.
   • This fuzz target is written to fuzz_targets/{result.trial:02d}.fuzz_target.
   • (See: writing_stage.py#L70)

2. Execution Stage:

   • The fuzz target is successfully compiled and executed.
   • The build and run logs are written to logs/build/{result.trial:02d}.log and logs/run/{result.trial:02d}.log respectively. The run log may contain crash information.
   • (See: builder_runner.py#L579 and builder_runner.py#L626)

3. Analysis Stage:

   • The analysis of the execution results fails for some reason, triggering a second cycle.
   • (See: pipeline.py#L88)

Cycle 2:

1. Writing Stage:

   • The OnePromptEnhancer attempts to improve the fuzz target but fails to produce a compilable version after exhausting the maximum number of turns.
   • The last, non-compilable fuzz target attempt from the enhancer overwrites the original, working fuzz target at fuzz_targets/{result.trial:02d}.fuzz_target.
   • (See: writing_stage.py#L70)

2. Execution Stage:

   • The execution stage is skipped because the new fuzz target does not compile.

Resulting Mismatch

This process leads to the following inconsistent state:

• The file fuzz_targets/{result.trial:02d}.fuzz_target contains the final, non-compilable code from the failed attempt in Cycle 2.
• The log file logs/run/{result.trial:02d}.log and logs/build/{result.trial:02d}.log contains the execution and crash logs from the successful run in Cycle 1.

Proposed Solution / Expected Behavior

The artifacts from a given trial should remain consistent. When a subsequent cycle fails to produce a valid and compilable fuzz target, it should not overwrite the artifacts from a previously successful run. The final fuzz_target file should correspond to the code that generated the results in the logs/run/ directory.

[moizulfiqar]

Hi! Is this issue resolved yet?