Valgrind, Perl, and XS

[Marvin Humphrey]

Greets,

Valgrind has just helped me track down a nasty segfault-producing bug
in a test app. This time, it turns out it was an invalid read in an
XS dependency and not something in in my own distro, but I'd like to
continue to use Valgrind as a troubleshooting aid going forward with
my expanding C and XS code base.

At this moment I have ready access to three systems:

1. A G4 laptop running OS X Tiger. Valgrind isn't available
for OS X, so that's out.
2. A Pentium 4 running FreeBSD 5.3. Perl and Valgrind don't
play well together on this system. "valgrind perl -e '1'"
segfaults and dies. I tried upgrading to Perl 5.8.7 from the
5.8.5 install, but it didn't make a difference.
3. A dual Xeon running RedHat 9 with Perl 5.8.6 installed. This
works, but I have some questions.

When I run "valgrind --leak-check=full perl -e '1'" on the RedHat
system (full output below), a number of leaks get reported. Maybe
it's nothing to worry about, maybe it's a problem with this
particular Perl installation... I'm not sure. In any case, it makes
troubleshooting XS extensions more complex, since it's hard to see
where flaws begin and end.

The problem is compounded by the fact that all of my modules are
using Perl's own memory management functions: New, Newz, Safefree,
etc. I've discovered that if I use malloc, valgrind is able to
isolate where a memory leak is occurring down to the particular XS
function, which is great. But with New, you can't tell what's going
on. (Again, full output below.)

A little XS distro with the the following functions is available here:

http://www.rectangular.com/downloads/MemLeak-0.01.tar.gz

void
leak_memory_via_malloc()
PREINIT:
char* foo;
PPCODE:
foo = malloc(2);

void
leak_memory_via_New()
PREINIT:
char* foo;
PPCODE:
New(1, foo, 3, char);

If you have Valgrind and you want to try to duplicate my results,
download and decompress, then...

perl Build.PL
./Build
valgrind --leak-check=full perl -Mblib t/00-leak.t

I see from the list archives that Valgrind has been discussed here in
the past. If any of you can share your methodology for
troubleshooting memory problems with valgrind (or without), I would
be grateful.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/



$ valgrind --leak-check=full perl -e '1'
==2577== Memcheck, a memory error detector.
==2577== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et al.
==2577== Using LibVEX rev 1471, a library for dynamic binary
translation.
==2577== Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP.
==2577== Using valgrind-3.1.0, a dynamic binary instrumentation
framework.
==2577== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.
==2577== For more details, rerun with: -v
==2577==
==2577==
==2577== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 21 from 1)
==2577== malloc/free: in use at exit: 46,079 bytes in 544 blocks.
==2577== malloc/free: 648 allocs, 104 frees, 53,100 bytes allocated.
==2577== For counts of detected errors, rerun with: -v
==2577== searching for pointers to 544 not-freed blocks.
==2577== checked 325,504 bytes.
==2577==
==2577== 1,186 (100 direct, 1,086 indirect) bytes in 1 blocks are
definitely lost in loss record 1 of 3
==2577== at 0x401A619: malloc (vg_replace_malloc.c:149)
==2577== by 0x8098B14: Perl_safesysmalloc (in /usr/local/bin/perl)
==2577== by 0x809AA63: Perl_my_setenv (in /usr/local/bin/perl)
==2577== by 0x8060011: perl_parse (in /usr/local/bin/perl)
==2577== by 0x805E417: main (in /usr/local/bin/perl)
==2577==
==2577== LEAK SUMMARY:
==2577== definitely lost: 100 bytes in 1 blocks.
==2577== indirectly lost: 1,086 bytes in 23 blocks.
==2577== possibly lost: 0 bytes in 0 blocks.
==2577== still reachable: 44,893 bytes in 520 blocks.
==2577== suppressed: 0 bytes in 0 blocks.
==2577== Reachable blocks (those to which a pointer was found) are
not shown.
==2577== To see them, rerun with: --show-reachable=yes


$ valgrind --leak-check=full perl -Mblib t/00-leak.t
==2917== Memcheck, a memory error detector.
==2917== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et al.
==2917== Using LibVEX rev 1471, a library for dynamic binary
translation.
==2917== Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP.
==2917== Using valgrind-3.1.0, a dynamic binary instrumentation
framework.
==2917== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.
==2917== For more details, rerun with: -v
==2917==
1..1
ok 1 - use MemLeak;
==2917==
==2917== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 25 from 1)
==2917== malloc/free: in use at exit: 1,052,222 bytes in 23,751 blocks.
==2917== malloc/free: 41,110 allocs, 17,359 frees, 13,634,866 bytes
allocated.
==2917== For counts of detected errors, rerun with: -v
==2917== searching for pointers to 23,751 not-freed blocks.
==2917== checked 1,310,408 bytes.
==2917==
==2917== 50 bytes in 25 blocks are definitely lost in loss record 1
of 10
==2917== at 0x401A619: malloc (vg_replace_malloc.c:149)
==2917== by 0x40218D3: XS_MemLeak_leak_memory_via_malloc (in /home/
xxxx/devlib/MemLeak/blib/arch/auto/MemLeak/MemLeak.so)
==2917== by 0x80AC1B7: Perl_pp_entersub (in /usr/local/bin/perl)
==2917== by 0x80A5DFF: Perl_runops_standard (in /usr/local/bin/perl)
==2917== by 0x806115D: S_run_body (in /usr/local/bin/perl)
==2917== by 0x8060F55: perl_run (in /usr/local/bin/perl)
==2917== by 0x805E42C: main (in /usr/local/bin/perl)
==2917==
==2917==
==2917== 1,285 (175 direct, 1,110 indirect) bytes in 26 blocks are
definitely lost in loss record 6 of 10
==2917== at 0x401A619: malloc (vg_replace_malloc.c:149)
==2917== by 0x8098B14: Perl_safesysmalloc (in /usr/local/bin/perl)
==2917== by 0x809AA63: Perl_my_setenv (in /usr/local/bin/perl)
==2917== by 0x8060011: perl_parse (in /usr/local/bin/perl)
==2917== by 0x805E417: main (in /usr/local/bin/perl)
==2917==
==2917== LEAK SUMMARY:
==2917== definitely lost: 225 bytes in 51 blocks.
==2917== indirectly lost: 1,110 bytes in 23 blocks.
==2917== possibly lost: 0 bytes in 0 blocks.
==2917== still reachable: 1,050,887 bytes in 23,677 blocks.
==2917== suppressed: 0 bytes in 0 blocks.
==2917== Reachable blocks (those to which a pointer was found) are
not shown.
==2917== To see them, rerun with: --show-reachable=yes

[Jeremy White]

Valgrind has just helped me track down a nasty segfault-producing bug in a
test app. This time, it turns out it was an invalid read in an XS
dependency and not something in in my own distro, but I'd like to continue
to use Valgrind as a troubleshooting aid going forward with my expanding C
and XS code base.

Valgrind is a fantastic tool, and has helped me track down some really nasty
bugs within my XS/C code. It also found problems that I wasn't even aware
off:)

I've considered making my module run under Valgrind as part of the test
suite, but haven't done enough analysis on the issues involved.

I see from the list archives that Valgrind has been discussed here in the
past. If any of you can share your methodology for troubleshooting memory
problems with valgrind (or without), I would be grateful.

Although not because of Valgrind, I ended up moving all of my memory
allocation into separate global functions. So I now have:

mymodule_malloc
mymodule_free
mymodule_realloc

This made tracking memory issues with Valgrind very simple.

Cheers,

jez.

[Marvin Humphrey]

Post by Jeremy White
Although not because of Valgrind, I ended up moving all of my
mymodule_malloc
mymodule_free
mymodule_realloc
This made tracking memory issues with Valgrind very simple.

It seems like you could do this with macros, no? Throw in an #ifdef
DEBUG that defines mymodule_malloc as a wrapper function, with an
#else clause that defines it as New.

I'm having a bit of trouble figuring out how to pass around the 4th
arg to New, the cast. What do your memory allocation functions look
like?

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

[Marvin Humphrey]

Post by Marvin Humphrey
It seems like you could do this with macros, no?

No. This doesn't do the trick; the place where the memory gets lost
is still hidden.

#ifdef MEM_LEAK_DEBUG
#define MemLeak_New(x,v,n,t) \
(v = (t*)MemLeak_New_void(x,(n*sizeof(t))))
#else
#define MemLeak_New(x,v,n,t) New(x,v,n,t)
#endif

void*
MemLeak_New_void(int x, int num) {
char* ptr;
New(x, ptr, num, char);
return (void*)ptr;
}

Are you using the Perl malloc wrappers, Jeremy?

According to perldebguts, Perl's -DL switch is obsolete since 5.6 and
programs like Valgrind and Purify are preferred. If that's true, how
do the p5p folks hunt down memory leaks when the information Valgrind
spits out is so non-specific?

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

[Jeremy White]

It seems like you could do this with macros, no? Throw in an #ifdef DEBUG
that defines mymodule_malloc as a wrapper function, with an #else clause
that defines it as New.
I'm having a bit of trouble figuring out how to pass around the 4th arg to
New, the cast. What do your memory allocation functions look like?

#undef malloc
#undef free
#undef realloc

void* MyModule_Malloc (long size) {
void* memory=malloc(size);
return memory;
}

:) I'm using the OS memory management functions rather than perl's.

Cheers,

jez.