Adding .c and .h files to XS distro

[Marvin Humphrey]

Greets,

I'm working on a huge CPAN distro, 50 modules and counting, of which
around 10 or so use XS code at present. The XS code is integrated
into the distro in an unconventional way -- it's at the end of
the .pm file after an __XS__ token. A routine built into Makefile.PL
walks the file hierarchy and extracts all the code, writing a single
KinoSearch.xs file on the fly just before WriteMakefile gets called.
There are a couple other tricks I'll skip for brevity's sake that are
necessary to make this scheme fly, but on the whole, it seems to be
working out pretty well.

I have a new problem now: in addition to the XS code, there's a bunch
of straight up C code which I've been stuffing in at the top of
the .xs file. That C codebase is growing large enough that it needs
to be broken out into separate files. However, I don't understand
the MakeMaker docs on how to add in .c and .h files.

The bulk of my C programming has been via Inline::C and XS, so I've
never had to beat my head against make. It seems to me that trying
to pick apart a wrapper as complex as MakeMaker is the wrong way to
go about learning a tool like make. I'd be content to put off those
lessons a little longer if another solution presented itself, but the
docs for the obvious candidate, Module::Build indicate that XS
support is in alpha, and sure enough, "./Build test" can't even find
the compiled libraries on my system. :\

I'll need to learn make eventually, and I'm happy to study, I'd just
like to 1) solve my current problem sooner rather than later, and 2)
go about learning systematically. Any advice?

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

[Sisyphus]

----- Original Message -----
From: "Marvin Humphrey" <***@rectangular.com>
To: <perl-***@perl.org>
Sent: Wednesday, November 09, 2005 1:47 PM
Subject: Adding .c and .h files to XS distro
.
.

Post by Marvin Humphrey
I have a new problem now: in addition to the XS code, there's a bunch
of straight up C code which I've been stuffing in at the top of
the .xs file. That C codebase is growing large enough that it needs
to be broken out into separate files. However, I don't understand
the MakeMaker docs on how to add in .c and .h files.

Perhaps the section "EXAMPLE 4" in 'perldoc perlxstut' can help out here.

Cheers,
Rob

[Muppet]

Post by Marvin Humphrey
I have a new problem now: in addition to the XS code, there's a
bunch of straight up C code which I've been stuffing in at the top
of the .xs file. That C codebase is growing large enough that it
needs to be broken out into separate files. However, I don't
understand the MakeMaker docs on how to add in .c and .h files.

Various approaches:


dirty: write your extra C and H files. include them in MANIFEST.
#include them from your xs code's verbatim C section.


proper: follow the example of EXAMPLE 4 in perlxstut, and build a
little library for yourself, which you link in. c.f. L<perlxstut>


quick and just as good: add your extra files' names to MANIFEST so
that they get dirstributed, and add extra object files to the OBJECT
key to WriteMakefile, like so:

WriteMakefile (
...
OBJECT => q[$(BASE_EXT)$(OBJ_EXT) myextracfile$(OBJ_EXT)],
...
);

Note that you are overwriting the default for OBJECT, so we supply
that default manually -- that's what $(BASE_EXT)$(OBJ_EXT) is. You
specify only the object file version of your filename -- for
myextracfile.c you'd specify myextracfile$(OBJ_EXT), which gets
expanded to myextracfile.o or myextracfile.obj or whatever is correct
for the platform; the make rules can figure out that this needs to be
built from a c file with the same name. See L<ExtUtils::MakeMaker>
for more info -- search for OBJECT.

Post by Marvin Humphrey
I'll need to learn make eventually, and I'm happy to study, I'd
just like to 1) solve my current problem sooner rather than later,
and 2) go about learning systematically. Any advice?

Make is actually quite easy, once you understand how it does
bookkeeping. The harder part is writing portable Make and getting it
to be happy with MakeMaker.


--
Examples really shouldn't include unexploded ordnance.
-- Joe Smith, referring to an example program i wrote.

[Marvin Humphrey]

Thanks to Sisyphus and muppet for the pointer to perlxstut Example
#4. It's funny, I went through that tutorial back in May or June,
even typing in all the example code. But it seems I didn't retain
the bits I didn't understand.

Post by Muppet
dirty: write your extra C and H files. include them in MANIFEST.
#include them from your xs code's verbatim C section.

Most of the code I'm talking about needs to be #include-d anyway:

1) Wrapper functions around PerlIO_xxxxx, e.g. KinoIO_read_vint below
2) typedefs and init/manip/destroy routines for the new types
3) global constants, e.g. "#define A_GLOBAL_CONSTANT 255"
4) Utilities, e.g. Kino_confess which allows Carp::confess to be
invoked from C a la the sprintf croak/warn.

All that stuff is supposed to be accessible throughout the
amalgamated KinoSearch.xs -- it's dirty by design. But it makes
sense to break it up into a few C modules, e.g. KinoIO.h/KinoIO.c.

... [ time passes while Marvin experiments ] ...

Hey wait a minute... you're going to need to #include at least one of
the .h files in the .xs file regardless, aren't you? Otherwise
there's no way to get at the stuff in the compiled libraries from
Perl. So I take it what you're calling "dirty" is the *absence* of
more rigorous steps, not the mere existence of an #include directive
in the verbatim C section of the .xs file.

Post by Muppet
proper: follow the example of EXAMPLE 4 in perlxstut, and build a
little library for yourself, which you link in. c.f. L<perlxstut>

OK, ultimately, this is what I settled on. But I had to pass through
the other stages before I understood enough to make it work. Didn't
know what make did, didn't know what ar or ranlib did... I could do
decent enough memory management to write a multi-field mergesort
algo with a bunch of pointer acrobatics and not leak, but I knew
next to nothing about compiling! That's where starting off with
Inline::C gets ya.

Post by Muppet
quick and just as good: add your extra files' names to MANIFEST so
that they get dirstributed, and add extra object files to the
WriteMakefile (
...
OBJECT => q[$(BASE_EXT)$(OBJ_EXT) myextracfile$(OBJ_EXT)],
...
);

Worked like a charm, once I got all my .c/.h files sorted out and
figured out how to use include guards to stop those redefinitions...
that is, until I tried to tidy up and move the new files into a src/
directory. boom!

Post by Muppet
Note that you are overwriting the default for OBJECT, so we supply
that default manually -- that's what $(BASE_EXT)$(OBJ_EXT) is. You
specify only the object file version of your filename -- for
myextracfile.c you'd specify myextracfile$(OBJ_EXT), which gets
expanded to myextracfile.o or myextracfile.obj or whatever is
correct for the platform; the make rules can figure out that this
needs to be built from a c file with the same name. See
L<ExtUtils::MakeMaker> for more info -- search for OBJECT.

An excellent explanation. Thanks.

Post by Muppet
Make is actually quite easy, once you understand how it does
bookkeeping.

This was useful to know. Between Make's rep as "evil" due to the tab/
space issue and hearing Michael Schwern (maintainer of MakeMaker) let
off steam about MakeMaker at Portland Perl Mongers meetings, I had
gotten the impression that it was much more complex than it actually
turns out to be.

This was the best of the various overviews I stumbled across while
googling:

http://www.cprogramming.com/tutorial/makefiles.html

Cheers,

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

/**************************************************************/

/* read in a Variable INTeger, stored in 1-5 bytes */
U32
KinoIO_read_vint (PerlIO *fh) {
unsigned char aUChar;
int bitshift;
int check_val;
U32 aU32;

/* start by reading one byte; use the lower 7 bits */
check_val = PerlIO_read(fh, &aUChar, 1);
if (check_val < 1)
Kino_confess("KinoIO_read_vint error: %d", check_val);

aU32 = aUChar & 0x7f;

/* keep reading and shifting as long as the high bit is set */
for (bitshift = 7; (aUChar & 0x80) != 0; bitshift += 7) {
check_val = PerlIO_read(fh, &aUChar, 1);
if (check_val < 1)
Kino_confess("KinoIO_read_vint error: %d", check_val);
aU32 |= (aUChar & 0x7f) << bitshift;
}
return aU32;
}

[Marvin Humphrey]

Hi folks,

Turns out that things are quite hunky-dory yet. We're compiling, at
least -- but the behavior isn't what I thought it was, and it's not
optimum.

Right now, the auxiliary .c and .h files are in src/ along with a
Makefile.PL. Here's a simplified manifest:

KinoSearch.xs # built on the fly, but MM doesn't know that
lib/KinoSearch.pm
Makefile.PL
src/KinoIO.c
src/KinoIO.h
src/KinoUtil.c
src/KinoUtil.h
src/Makefile.PL
t/00-load.t

OBJECT is a multi-item list in the main Makefile.PL:

OBJECT => '$(BASEEXT)$(OBJ_EXT) src/KinoIO$(OBJ_EXT) '
. 'src/KinoUtil$(OBJ_EXT)',

The auxiliary src/Makefile.PL is essentially a copy of perlxstut
Example #4. (The important parts of both Makefile.PL files are
included below my sig.) *However*, the main Makefile.PL does *not*
use a MYEXTLIB or a MY::postamble, as per the perlxstut example -- I
took them away because with this line in the main Makefile.PL...

MYEXTLIB => 'src/KinoAuxLibs$(LIB_EXT)',

... this error appears:

make: *** No rule to make target `src/KinoAuxLibs.a', needed
by `subdirs'. Stop.

OK. Well, things were working without it, so we'll leave it out,
right? Fine... Then, I edit KinoSearch.xs, and invoke "make test"...
The auxiliary .c and .h files haven't changed, but they get
recompiled into .o files anyway. In fact, they get recompiled into
the main directory of the distro -- which didn't happen when starting
fresh with "make distclean; perl Makefile.PL; make". And if I invoke
"make test" again and again, the files keep getting needlessly
recompiled into '.' -- and isn't the very point of Make to thwart
such waste? :\

I got to wondering if the second Makefile.PL was actually doing
anything... yes, it is: if it goes away, the auxiliary .c files get
compiled into '.' upon "make distclean; perl Makefile.PL; make", and
these errors appear...

powerpc-apple-darwin8-gcc-4.0.0: src/KinoIO.o: No such file or
directory
powerpc-apple-darwin8-gcc-4.0.0: src/KinoUtil.o: No such file or
directory

That makes sense -- they aren't in src/, they're in '.'. It's a
little weird that they only get compiled to one place or the other,
but that's life, I guess. Interestingly, if I change OBJECT in the
main Makefile.PL, removing each "src/"...

OBJECT => '$(BASEEXT)$(OBJ_EXT) KinoIO$(OBJ_EXT) '
. 'KinoUtil$(OBJ_EXT)';

... gcc can't find the .o files either...

powerpc-apple-darwin8-gcc-4.0.0: KinoIO.o: No such file or
directory
powerpc-apple-darwin8-gcc-4.0.0: KinoUtil.o: No such file or
directory

... buf for a different reason: this time, nobody ever found the .c
or .h files, and so the .o files were never created in *either* '.'
or src/. There's an obvious solution: move all the .c and .h files
out of src/ and into '.'. That works. But it defeats the whole
purpose of moving stuff into subdirectories in the first place.

So... with my current setup, things are working, but only by accident.

1. On "make distclean; perl Makefile.PL; make", the
auxiliary src/Makefile causes the .c/.h files to get
compiled within src/ -- and gcc can find them because of
the way OBJECT is constructed in ./Makefile. For whatever
reason, the auxiliary .c/.h files don't get compiled twice.

2. On "make test", src/Makefile seems to do what I think it ought
to do: it notes that the .c/.h files haven't changed, and
doesn't do anything. But the main Makefile isn't
behaving as it should -- there's some sort of disconnect
between where it looks and where it compiles to -- and so
it triggers a recompile.

It seems to me that adding a bunch of extra files to OBJECT only
works if the .c/.h files are in the root dir of the distro, no?

Returning to where we started, the GOOD solution is to link in the .a
file. And that's where I'm stuck. What will it take to get rid of
that first error?

For reference, this is the MYEXTLIB line from perlxstut Example #4...

'MYEXTLIB' => 'mylib/libmylib$(LIB_EXT)',

... and this is the postamble:

sub MY::postamble {
'
$(MYEXTLIB): mylib/Makefile
cd mylib && $(MAKE) $(PASSTHRU)
';
}

Cheers,

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

#######################################################################
### excerpt from ./Makefile.PL
#######################################################################

WriteMakefile(
NAME => 'KinoSearch',
AUTHOR => 'Marvin Humphrey <marvin at rectangular dot
com>',
VERSION_FROM => 'lib/KinoSearch.pm',
ABSTRACT_FROM => 'lib/KinoSearch.pm',
PREREQ_PM => { 'Test::More' => 0, },
INC => "-Isrc",
OBJECT => $OBJECT_string,
PERL_MALLOC_OK => 1,
# MYEXTLIB => 'src/KinoAuxLibs$(LIB_EXT)',
dist => { COMPRESS => 'gzip -9f', SUFFIX => 'gz', },
# Note: KinoSearch.xs file gets cleaned
clean => { FILES => 'KinoSearch-* KinoSearch.xs invindex' },
);

#sub MY::postamble {
#'
# $(MYEXTLIB): src/Makefile
# cd src && $(MAKE) $(PASSTHRU)
#'
#}


#######################################################################
### excerpt from ./src/Makefile.PL
#######################################################################

WriteMakefile(
NAME => 'KinoAuxLibs',
SKIP => [qw(all static static_lib dynamic dynamic_lib)],
);

sub MY::top_targets {
'
all :: static

pure_all :: static

static :: KinoAuxLibs$(LIB_EXT)

KinoAuxLibs$(LIB_EXT): $(O_FILES)
$(AR) cr KinoAuxLibs$(LIB_EXT) $(O_FILES)
$(RANLIB) KinoAuxLibs$(LIB_EXT)

';
}

[Muppet]

Post by Marvin Humphrey
Turns out that things are quite hunky-dory yet. We're compiling,
at least -- but the behavior isn't what I thought it was, and it's
not optimum.

[snip informative description]

I think it boils down to conflict between the two Makefile.PLs. In
reality, you only need the top one. For example, Gtk2 has a shedload
of xs files in a subdirectory, but only one Makefile.PL, in the top
directory.

IIRC, the contents are OBJECT are used as the default for LDFROM,
which is used as the list of objects to link into the object that
will be loaded by your module. That is, you should be able to do this:

WriteMakefile(
NAME => 'MyModule',
...
INC => '-Isrc', # so we can see the extra headers
OBJECT => 'MyModule$(OBJ_EXT) src/extrafile1$(OBJ_EXT) src/
extrafile2$(OBJ_EXT)',
...
);

and it should all Just Work. To clarify, I would expect the example
above to build MyModule.so from MyModule.o, src/extrafile1.o, and src/
extrafile2.o after building the .o files from the .c files, and after
building MyModule.c from MyModule.xs. No fiddling with extra libs or
extra makefiles should be necessary.


Gtk2 uses this scheme to build a single Gtk2.so from almost two
hundred xs/*.xs files. We just dump a whole slew of filenames into
OBJECT and let MakeMaker and Make do the rest.

On win32 there is(was?) an issue with MakeMaker that resulted in some
pain and suffering when building Gtk2 --- the source files in the
subdirectory would be compiled into objects in the current directory,
so the link command would fail. A bit of extra magic fixed that; you
can find it as the const_ccmd() sub in the MY package in
Glib::MakeHelper. Look near the end: http://search.cpan.org/src/TSCH/
Glib-1.101/MakeHelper.pm


Hope that helps...


--
How come hair colors for women take an hour, but "wash out the gray"
stuff for men only five minutes? This is so unfair!
-- Elysse, complaining about commercials

[Marvin Humphrey]

Jus a quick question: Have you tried Module::Build? It might magically
solve all your problems - or not but it might be worth a try and if
not,
send a bug report to the Build guys to improve it :)

I'd tried Module::Build earlier, before even breaking out the .c/.h
files, and it had failed. But it turns out, not for the reason I'd
thought.

I assembled a reduced test case and posted to the module-build-
general list. The configuration error was quickly isolated:

- xs_files => { 'TestXS.xs' => 'TestXS.xs' },
+ xs_files => { 'TestXS.xs' => 'lib/TestXS.xs' },

With that change, my test case worked properly. As an aside, you
don't need the specify the xs_files param at all if you keep the XS
file in lib/ where it is to be "installed" -- however, Module::Build
won't find a .xs file in the root directory of the distro, so you
need to do one or the other.

After making that tweak to the KinoSearch Build.PL file, two more
modifications were required. Build.PL had to be told about the src/
directory...

c_source => 'src',

... and I had to move ppport.h out of the distro's root directory and
into src/.

Now, everything works! All the individual parts get compiled, and
only when they need to be. If I regenerate the KinoSearch.xs file by
running my modified Build.PL script, Build is smart enough to figure
out that the .c/.h files haven't been modified, and it uses the
existing .o files. If a single .c file gets changed, only that one
file gets recompiled. Perfect!

Thanks to everyone for the help.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

[Tels]

-----BEGIN PGP SIGNED MESSAGE-----

Moin Marvin,

Post by Marvin Humphrey
Hi folks,
Turns out that things are quite hunky-dory yet. We're compiling, at
least -- but the behavior isn't what I thought it was, and it's not
optimum.
Right now, the auxiliary .c and .h files are in src/ along with a

Jus a quick question: Have you tried Module::Build? It might magically
solve all your problems - or not but it might be worth a try and if not,
send a bug report to the Build guys to improve it :)

Best wishes,

Tels

- --
Signed on Wed Nov 16 11:38:23 2005 with key 0x93B84C15.
Visit my photo gallery at http://bloodgate.com/photos/
PGP key on http://bloodgate.com/tels.asc or per email.

"TT: If I go to Blockbuster and rent a movie and watch it, am I a bad
person? Is that bad?
JV: No, you're not a bad person. But you don't have any right.
TT: But I rented the movie. Why should it be illegal?
JV: Well then, you have to get a machine that's licensed to show it. " -
Keith J. Winstein (TT) vs. Jack Valenty (JV) in http://tinyurl.com/2y65n