Memory...growing, and growing

[a***@alfarrabio.di.uminho.pt]

Hi

I am trying to debug some code that is using too much memory. I am not
sure if it is from the XS code, but if you say it isn't at least I know
I should look elsewhere.

Here it is (resumed):

array = newAV();
av_push(array, newSVuv(dictionary_get_occ(D, wid)));

for (j = 0; j < MAXENTRY; j++) {
twid = 0;
prob = 0.0;
twid = dictionary_get_id(D, wid, j);
if (twid) {
prob = dictionary_get_val(D, wid, j);
# tword can't be freed...
tword = word_list_get_by_id(T, twid);
if (!tword) { fprintf(stderr, "Id: %d\n", twid); }
av_push(array, newSVpvn(tword, strlen(tword)));
av_push(array, newSVnv((double)prob));
}
}

RETVAL = newRV_noinc((SV*)array);
OUTPUT:
RETVAL

Am I doing something wrong? Maybe because of refcounts and such?

Thanks
ambs

[Marvin Humphrey]

Post by a***@alfarrabio.di.uminho.pt
array = newAV();
av_push(array, newSVuv(dictionary_get_occ(D, wid)));
for (j = 0; j < MAXENTRY; j++) {
twid = 0;
prob = 0.0;
twid = dictionary_get_id(D, wid, j);
if (twid) {
prob = dictionary_get_val(D, wid, j);
# tword can't be freed...
tword = word_list_get_by_id(T, twid);
if (!tword) { fprintf(stderr, "Id: %d\n", twid); }
av_push(array, newSVpvn(tword, strlen(tword)));
av_push(array, newSVnv((double)prob));
}
}
RETVAL = newRV_noinc((SV*)array);
RETVAL

Your handling of the perl array looks correct. You create a new AV
with a refcount of one. You add a bunch of scalars to the array,
each of which is created with a refcount of 1. Crucially, you handle
the arrayref correctly, creating a single reference which does not
increase the refcount of the AV. This one arrayref ends up on the
stack, mortalized via the a call to sv_2mortal from the OUTPUT/RETVAL
apparatus. Assuming for the sake of simplicity that we don't copy
anything out of @_, when that mortalized ref comes due, your array
and all of its members should fall like dominoes.

I think you'll need to look elsewhere. Coincidentally, I'm debugging
memory leaks right this moment using valgrind. If you have access to
a Linux system, I'd strongly recommend you give it a try.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

[a***@alfarrabio.di.uminho.pt]

Your handling of the perl array looks correct. You create a new AV with
a refcount of one. You add a bunch of scalars to the array, each of
which is created with a refcount of 1. Crucially, you handle the
arrayref correctly, creating a single reference which does not increase
the refcount of the AV. This one arrayref ends up on the stack,
mortalized via the a call to sv_2mortal from the OUTPUT/RETVAL
apparatus. Assuming for the sake of simplicity that we don't copy
all of its members should fall like dominoes.

Thank you. One less place to look at ;)

I think you'll need to look elsewhere. Coincidentally, I'm debugging
memory leaks right this moment using valgrind. If you have access to a
Linux system, I'd strongly recommend you give it a try.

Unfortunately, I think I will need to do that ;)

Cheers
ambs

[Dan Sully]

* Marvin Humphrey shaped the electrons to say...

Post by Marvin Humphrey
I think you'll need to look elsewhere. Coincidentally, I'm debugging
memory leaks right this moment using valgrind. If you have access to
a Linux system, I'd strongly recommend you give it a try.

Any suggestions for using valgrind with Perl/XS? I'm debugging a memory leak
right now too:

long bio_write_cb(struct bio_st *bm, int m, const char *ptr, int l, long x, long y) {

if (m == BIO_CB_WRITE) {
SV *sv = (SV *) BIO_get_callback_arg(bm);
sv_catpvn(sv, ptr, l);
}

if (m == BIO_CB_PUTS) {
SV *sv = (SV *) BIO_get_callback_arg(bm);
l = strlen(ptr);
sv_catpvn(sv, ptr, l);
}

return l;
}

static BIO* sv_bio_create(void) {

SV *sv = newSVpvn("", 0);

/* create an in-memory BIO abstraction and callbacks */
BIO *bio = BIO_new(BIO_s_mem());

BIO_set_callback(bio, bio_write_cb);
BIO_set_callback_arg(bio, (void *)sv);

return bio;
}

static SV* sv_bio_final(BIO *bio) {

SV* sv;

BIO_flush(bio);
sv = (SV *)BIO_get_callback_arg(bio);
BIO_free_all(bio);

if (!sv) {
sv = &PL_sv_undef;
}

return sv;
}

MODULE = Crypt::OpenSSL::X509 PACKAGE = Crypt::OpenSSL::X509

PROTOTYPES: DISABLE

.... snip ....

SV*
pubkey(x509)
Crypt::OpenSSL::X509 x509;

PREINIT:
EVP_PKEY *pkey;
BIO *bio;

CODE:

pkey = X509_get_pubkey(x509);
bio = sv_bio_create();

if (pkey == NULL) {

BIO_free_all(bio);
EVP_PKEY_free(pkey);
croak("Public Key is unavailable\n");
}

if (pkey->type == EVP_PKEY_RSA) {

PEM_write_bio_RSAPublicKey(bio, pkey->pkey.rsa);

} else if (pkey->type == EVP_PKEY_DSA) {

PEM_write_bio_DSA_PUBKEY(bio, pkey->pkey.dsa);

} else {

BIO_free_all(bio);
EVP_PKEY_free(pkey);
croak("Wrong Algorithm type\n");
}

EVP_PKEY_free(pkey);

RETVAL = sv_bio_final(bio);

OUTPUT:
RETVAL


Repeated calls to $x509->pubkey cause a noticiable memory leak.

-D
--
Adobe Photoshop - When you want the truth. Real bad.

[Dan Sully]

* Marvin Humphrey shaped the electrons to say...

Post by Dan Sully
if (!sv) {
sv = &PL_sv_undef;
}

That was just in the for debugging, I commented it out and still have the
leak behavior. valgrind reports this:

==30258== 436 (112 direct, 324 indirect) bytes in 1 blocks are definitely lost in loss record 11 of 16
==30258== at 0x4A2080E: malloc (vg_replace_malloc.c:149)
==30258== by 0x56CC911: CRYPTO_malloc (in /usr/lib/libcrypto.so.0.9.8)
==30258==
==30258==
==30258== 788 bytes in 53 blocks are definitely lost in loss record 13 of 16
==30258== at 0x4A2080E: malloc (vg_replace_malloc.c:149)
==30258== by 0x499A85: Perl_safesysmalloc (util.c:78)
==30258== by 0x49B7DF: Perl_savepvn (util.c:789)
==30258== by 0x42D8C7: Perl_gv_fetchpv (gv.c:754)
==30258== by 0x427C2C: S_init_main_stash (perl.c:3510)
==30258== by 0x42210A: S_parse_body (perl.c:1657)
==30258== by 0x421DE1: perl_parse (perl.c:1598)
==30258== by 0x41D674: main (perlmain.c:97)

If I comment out some of the openssl calls, the CRYPTO_malloc leak goes away,
but the perl one is still there. How would I find out what part of my XS code
is causing that leak?

-D
--
Adobe Photoshop - When you want the truth. Real bad.