ASan Root Cause Parser

Address Sanitizer (ASan) is a memory corruption detection mechanism built into both clang and gcc. It is capable of detecting the following conditions: use after free, heap buffer overflow, stack buffer overflow, global buffer overflow, use after return, use after scope, initialization order, and memory leaks. It is often combined with fuzzing techniques in order to alert on bugs that may not have otherwised crashed the target application. While targeting large applications, it is common to end up with hundreds to thousands of crash reports. Depending on your fuzzing framework, many of these may be duplicates. This python script will parse ASan crash reports and group them based on the backtrace information.

In order to examine how this works, we have a deliberately insecure C program that’s easy to crash in two very specific places in the code. This will allow us to utilize the ASan analyzer to group crashes by root cause.

Vulnerable Program

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char *argv[])
{
        char greeting[25];
        char name[25];

        if (argc != 3) {
                fprintf(stderr, "Usage: %s <greeting> <name>\n", argv[0]);
                exit(1);
        }

        strcpy(greeting, argv[1]);
        strcpy(name, argv[2]);

        printf("%s %s\n", greeting, name);

        return 0;
}

Compile

Address Sanitizer is configured with the flag -fsanitize=address. There are other sanitizers available, including memory sanitizer, thread sanitizer, undefind behavior sanitizer, dataflow sanitizer, and more. For a full list, refer to clang’s documentation. For simplicity sake, we will compile the test program using gcc as it is most likely to be present on your system. We’ll also include debug symbols for verbosity sake.

gcc test.c -o test -fsanitize=address -ggdb

We are primarily concerned with being able to identify the root cause based on the backtrace, so we need to go ahead and disable ASLR before running any test cases that cause crashes. If we do not, then each crash will appear to be unique based on the memory addresses displayed.

sudo bash -c 'echo 0 > /proc/sys/kernel/randomize_va_space'

Example Report

The following is an example report generated by Address Sanitizer using a gcc compiled binary. In this case, we have a stack based buffer overflow write of 26 bytes into a 25 byte buffer. This one byte overwrite would not typically crash the application in this specific instance due to extra bytes surrounding the variable. ASan allows us to detect the non-crashing error that is indeed quite serious.

=================================================================
==11570==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fffffffdd19 at pc 0x7ffff6ecc709 bp 0x7fffffffdc80 sp 0x7fffffffd428
WRITE of size 26 at 0x7fffffffdd19 thread T0
    #0 0x7ffff6ecc708  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x62708)
    #1 0x400c2c in main /tmp/scratch/test.c:16
    #2 0x7ffff6ac082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #3 0x4009a8 in _start (/tmp/scratch/test+0x4009a8)

Address 0x7fffffffdd19 is located in stack of thread T0 at offset 121 in frame
    #0 0x400a85 in main /tmp/scratch/test.c:6

  This frame has 2 object(s):
    [32, 57) 'greeting'
    [96, 121) 'name' <== Memory access at offset 121 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow ??:0 ??
Shadow bytes around the buggy address:
  0x10007fff7b50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff7b60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff7b70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff7b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff7b90: 00 00 00 00 f1 f1 f1 f1 00 00 00 01 f2 f2 f2 f2
=>0x10007fff7ba0: 00 00 00[01]f3 f3 f3 f3 f3 f3 f3 f3 00 00 00 00
  0x10007fff7bb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff7bc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff7bd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff7be0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff7bf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==11570==ABORTING

Trigger multiple root cause crashes

  • ./test $(python -c 'print "A"*25') test 2> asan.log.1
  • ./test $(python -c 'print "A"*25') test 2> asan.log.2
  • ./test hello $(python -c 'print "A"*25') 2> asan.log.3

Grab ASan Analyzer

Analyze the crashes

./asanalyzer.py 'asan.log.*'
[-] Unique stack (asan.log.2)
        Description: stack-buffer-overflow on address 0x7fffffffdcd9 at pc 0x7ffff6ecc709 bp 0x7fffffffdc80 sp 0x7fffffffd428
        Duplicates (1)
                asan.log.1
[-] Unique stack (asan.log.3)
        Description: stack-buffer-overflow on address 0x7fffffffdd19 at pc 0x7ffff6ecc709 bp 0x7fffffffdc80 sp 0x7fffffffd428
        Duplicates (0)
comments powered by Disqus