XRLFuzz: Fuzzing Binaries Guided by Format Information Based on Deep Reinforcement Learning

Abstract

Fuzzing is a popular and effective automatic vulnerability mining method. More and more fuzzing techniques are starting to integrate reinforcement learning. But traditional fuzzing based on reinforcement learning is blind in sample mutation due to the lack of format information and efficient mutation algorithms. As a result, it is challenging to achieve higher coverage and the number of effective fuzzing is limited, leading to low utilization. To mitigate these problems, this paper proposes a new fuzzer named XRLFuzz, a format information guided fuzzing based on deep reinforcement learning for binaries. We use dynamic instrumentation techniques to provide runtime information, And then we use these information to perform format division and extract keywords based on the invalid mutation reuse algorithm. Format information is incorporated into the action dimension of deep reinforcement learning to guide the selection of mutation strategy. The experimental results show that the format division technology and keyword extraction technology both improve the efficiency of fuzzing, and XRLFuzz achieves code coverage of 106% to 276% of AFL.