preparing perl-interpreters

[Michael Ganz]

I don't know if this is the right list, but I didn't found a better one for
the problem.



At the moment I'am developing an application-server with different services
in C++ on Linux, Windows and Solaris.

The server-admin can implement own servicefunctions by configuring and
implementing perl-scripts using XS-functions for interaction with server
(XML-DOM, call other servicefunctions, logging, etc.). This works great, but
there are memory leaks and sometimes the server crashed in the
perl-interpreter depended on the used modules (I guess a multithreaded
issue).



So, I decided to implement a special perl-corba-server for running the
scripts.

Each perl script is parsed and stored in an interpreter instance using a
stl-map.



For faster running, I decided to parse the scripts at startup of the
perlserver.

To make it stable I fork the server process.



I saw that every interpreter-instance will allocate about 1MB of RAM
(Linux).

So dealing with about 250 scripts, this allocates about 250 MB of RAM.



Now my questions:

Did I understood or implemented something wrong (interpreter
per script - one interpreter for all) ?

Why does an Interpreter-instance not running, uses this memory amount
(environment) ?

Is it possible to compile and store the scripts without an
complete interpreter instance ?

.



Regards Michael Ganz

[Alexandre Jousset]

Hi,

Post by Michael Ganz
I saw that every interpreter-instance will allocate about 1MB of RAM
(Linux).
So dealing with about 250 scripts, this allocates about 250 MB of RAM.

I may be wrong but if I understand correctly, it's 250MB of *virtual* memory. Not real. libperl.so is loaded once and shared between multiple instances.

My 2c,

[Rob Janes]

it's possible perl is statically linked. ldd `type -p perl` will show
the libperl.so shared library if it's dynamically linked.

valgrind will show actual memory usage.

pmap will show shared library segments. you can assume libperl.so
code segments are shared. i don't think the data segments are shared.

ps is deceptive in that it does not report shared memory savings.

http://www.linuxquestions.org/linux/articles/Technical/Understanding_memory_usage_on_Linux
http://stackoverflow.com/questions/131303/linux-how-to-measure-actual-memory-usage-of-an-application-or-process

-rob

       Hi,

Post by Michael Ganz
I saw that every interpreter-instance will allocate about 1MB of RAM
(Linux).
So dealing with about 250 scripts, this allocates about 250 MB of RAM.

       I may be wrong but if I understand correctly, it's 250MB of *virtual*
memory. Not real. libperl.so is loaded once and shared between multiple
instances.
       My 2c,
--
--      \^/                                            --
--    -/ O \---------------------------------------    --
--   | |/ \|      Alexandre (Midnite) Jousset      |   --
--    -|___|---------------------------------------    --

[Michael Ganz]

-----Ursprüngliche Nachricht-----
Gesendet: Dienstag, 7. September 2010 18:45
An: 'Rob Janes'
Betreff: AW: preparing perl-interpreters

-----Ursprüngliche Nachricht-----
Gesendet: Dienstag, 7. September 2010 17:16
Betreff: Re: preparing perl-interpreters
it's possible perl is statically linked. ldd `type -p perl` will

show

the libperl.so shared library if it's dynamically linked.

It's linked shared, ldd shows an entry to libperl.so (Perl 5.12.1).

valgrind will show actual memory usage.

I have not yet valgrinded. OpenSuse 11.2 has an error here. But I try
this on another box.

pmap will show shared library segments. you can assume libperl.so
code segments are shared. i don't think the data segments are

shared.
pmap shows a lot of anon blocks with size of 2048K.
00002ad3ff039000 2048K 8K 4K 8K 0K rw-p [anon]
00002ad3ff239000 4K 0K 0K 0K 0K ---p
[anon]00002ad404000000 131072K 130936K 11903K 130936K 0K rw-p
[anon]
...
00002ad410000000 655360K 654952K 59541K 654952K 0K rw-p [anon]
00002ad43c000000 168068K 167964K 15285K 167964K 0K rw-p [anon]

ps is deceptive in that it does not report shared memory savings.

I looked to the processes by top
Now loading about 1000 perl-scripts and having 10 forked children of my
perl-server.
The Perl-Interpreter Instances are created in the perl-server
top - 18:35:35 up 13 days, 6:17, 2 users, load average: 0.00, 0.00,
0.00
Tasks: 227 total, 1 running, 226 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.1%sy, 0.0%ni, 99.6%id, 0.1%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 8184844k total, 6689408k used, 1495436k free, 554284k
buffers
Swap: 16779852k total, 0k used, 16779852k free, 3988704k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18790 mganz 20 0 1459m 1.0g 16m S 0 12.8 0:10.70 perlServer
19292 mganz 20 0 1459m 1.0g 3744 S 0 12.7 0:00.13 perlServer
19295 mganz 20 0 1459m 1.0g 3264 S 0 12.7 0:00.10 perlServer
19298 mganz 20 0 1459m 1.0g 3264 S 0 12.7 0:00.10 perlServer
19301 mganz 20 0 1459m 1.0g 3264 S 0 12.7 0:00.10 perlServer
19304 mganz 20 0 1459m 1.0g 3272 S 0 12.7 0:00.15 perlServer
19307 mganz 20 0 1459m 1.0g 3264 S 0 12.7 0:00.13 perlServer
19310 mganz 20 0 1459m 1.0g 3264 S 0 12.7 0:00.08 perlServer
19313 mganz 20 0 1459m 1.0g 3264 S 0 12.7 0:00.18 perlServer
19316 mganz 20 0 1459m 1.0g 3272 S 0 12.7 0:00.10 perlServer
19319 mganz 20 0 1459m 1.0g 3272 S 0 12.7 0:00.12 perlServer
http://www.linuxquestions.org/linux/articles/Technical/Understanding_me

mory_usage_on_Linux
http://stackoverflow.com/questions/131303/linux-how-to-measure-

actual-

memory-usage-of-an-application-or-process
-rob

       Hi,

Post by Michael Ganz
I saw that every interpreter-instance will allocate about 1MB of

RAM

Post by Michael Ganz
(Linux).
So dealing with about 250 scripts, this allocates about 250 MB of

RAM.

       I may be wrong but if I understand correctly, it's 250MB of

*virtual*

memory. Not real. libperl.so is loaded once and shared between

multiple

instances.
       My 2c,
--
--      \^/                                            --
--    -/ O \---------------------------------------    --
--   | |/ \|      Alexandre (Midnite) Jousset      |   --
--    -|___|---------------------------------------    --

[Rob Janes]

the rw--- lines in the pmap output are the data segments. the r---
lines are code.

also have a look at the output of the "free" command. free is easier than top.

-rob

-----Ursprüngliche Nachricht-----
Gesendet: Dienstag, 7. September 2010 18:45
An: 'Rob Janes'
Betreff: AW: preparing perl-interpreters

-----Ursprüngliche Nachricht-----
Gesendet: Dienstag, 7. September 2010 17:16
Betreff: Re: preparing perl-interpreters
it's possible perl is statically linked.  ldd `type -p perl` will

show

the libperl.so shared library if it's dynamically linked.

It's linked shared, ldd shows an entry to libperl.so (Perl 5.12.1).

valgrind will show actual memory usage.

I have not yet valgrinded. OpenSuse 11.2 has an error here. But I try
this on another box.

pmap will show shared library segments.  you can assume libperl.so
code segments are shared.  i don't think the data segments are

shared.
pmap shows a lot of anon blocks with size of 2048K.
00002ad3ff039000   2048K      8K      4K      8K      0K rw-p [anon]
00002ad3ff239000      4K      0K      0K      0K      0K ---p
[anon]00002ad404000000 131072K 130936K  11903K 130936K      0K rw-p
[anon]
...
00002ad410000000 655360K 654952K  59541K 654952K      0K rw-p [anon]
00002ad43c000000 168068K 167964K  15285K 167964K      0K rw-p [anon]

ps is deceptive in that it does not report shared memory savings.

I looked to the processes by top
Now loading about 1000 perl-scripts and having 10 forked children of my
perl-server.
The Perl-Interpreter Instances are created in the perl-server
top - 18:35:35 up 13 days,  6:17,  2 users,  load average: 0.00, 0.00,
0.00
Tasks: 227 total,   1 running, 226 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.1%sy,  0.0%ni, 99.6%id,  0.1%wa,  0.0%hi,  0.0%si,
0.0%st
Mem:   8184844k total,  6689408k used,  1495436k free,   554284k
buffers
Swap: 16779852k total,        0k used, 16779852k free,  3988704k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
18790 mganz     20   0 1459m 1.0g  16m S    0 12.8   0:10.70 perlServer
19292 mganz     20   0 1459m 1.0g 3744 S    0 12.7   0:00.13 perlServer
19295 mganz     20   0 1459m 1.0g 3264 S    0 12.7   0:00.10 perlServer
19298 mganz     20   0 1459m 1.0g 3264 S    0 12.7   0:00.10 perlServer
19301 mganz     20   0 1459m 1.0g 3264 S    0 12.7   0:00.10 perlServer
19304 mganz     20   0 1459m 1.0g 3272 S    0 12.7   0:00.15 perlServer
19307 mganz     20   0 1459m 1.0g 3264 S    0 12.7   0:00.13 perlServer
19310 mganz     20   0 1459m 1.0g 3264 S    0 12.7   0:00.08 perlServer
19313 mganz     20   0 1459m 1.0g 3264 S    0 12.7   0:00.18 perlServer
19316 mganz     20   0 1459m 1.0g 3272 S    0 12.7   0:00.10 perlServer
19319 mganz     20   0 1459m 1.0g 3272 S    0 12.7   0:00.12 perlServer
http://www.linuxquestions.org/linux/articles/Technical/Understanding_me

mory_usage_on_Linux
http://stackoverflow.com/questions/131303/linux-how-to-measure-

actual-

memory-usage-of-an-application-or-process
-rob

       Hi,

Post by Michael Ganz
I saw that every interpreter-instance will allocate about 1MB of

RAM

Post by Michael Ganz
(Linux).
So dealing with about 250 scripts, this allocates about 250 MB of

RAM.

       I may be wrong but if I understand correctly, it's 250MB of

*virtual*

memory. Not real. libperl.so is loaded once and shared between

multiple

instances.
       My 2c,
--
--      \^/                                            --
--    -/ O \---------------------------------------    --
--   | |/ \|      Alexandre (Midnite) Jousset      |   --
--    -|___|---------------------------------------    --