standard project structure?

[Xavier Noria]

Say a distribution has a regular Perl module

Foo::Bar::Baz

and another Perl module, used by the former, that is a loader:

Foo::Bar::Baz::XSModule

so we have something like this:

lib/Foo/Bar/Baz.pm
lib/Foo/Bar/Baz/XSModule.pm
lib/Foo/Bar/Baz/XSModule.xs
Makefile.PL
MANIFEST
...

With that structure make does not compile anything. Makefile.PL has
no XS specific stuff right now (I tried to use the XS parameter of
WriteMakefile without luck).

Which is the standard way to do that? Where do you put the .xs? What
are MODULE and PACKAGE set to?

-- fxn

[Marvin Humphrey]

Post by Xavier Noria
Say a distribution has a regular Perl module
Foo::Bar::Baz
Foo::Bar::Baz::XSModule
lib/Foo/Bar/Baz.pm
lib/Foo/Bar/Baz/XSModule.pm
lib/Foo/Bar/Baz/XSModule.xs
Makefile.PL
MANIFEST
...

Try it with the .xs file in the root directory of the distro -- the
same directory as Makefile.PL.

Post by Xavier Noria
What are MODULE and PACKAGE set to?

For the above config, set...

MODULE = Foo::Bar::Baz::XSModule PACKAGE = Foo::Bar::Baz

Foo::Bar::Baz::XSModule should contain the XSLoader::load command.
Foo::Bar::Baz should "use Foo::Bar::Baz::XSModule;", but should *not*
contain an XSLoader::load command, or you'll get redefinition
warnings. The effect is the same as if Foo::Bar::Baz::XSModule
looked like this:

package Foo::Bar::Baz::XSModule;
use strict;
use warnings;

our $VERSION = 0.01;

package Foo::Bar::Baz
use strict;
use warnings;

sub pollute { }
sub the { }
sub foo_bar_baz { }
sub namespace { }

1;

Best,

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

[Xavier Noria]

Post by Marvin Humphrey
Try it with the .xs file in the root directory of the distro -- the
same directory as Makefile.PL.

Post by Xavier Noria
What are MODULE and PACKAGE set to?

For the above config, set...
MODULE = Foo::Bar::Baz::XSModule PACKAGE = Foo::Bar::Baz
Foo::Bar::Baz::XSModule should contain the XSLoader::load command.
Foo::Bar::Baz should "use Foo::Bar::Baz::XSModule;", but should
*not* contain an XSLoader::load command, or you'll get redefinition
warnings.

Thank you! I didn't manage to get it right yet, though:

% make
cp lib/Foo/Bar/Baz.pm blib/lib/Foo/Bar/Baz.pm
cp lib/Foo/Bar/Baz/XSModule.pm blib/lib/Foo/Bar/Baz/XSModule.pm
Running Mkbootstrap for Foo::Bar::Baz ()
chmod 644 Baz.bs
rm -f blib/arch/auto/Foo/Bar/Baz/Baz.bundle
env MACOSX_DEPLOYMENT_TARGET=10.3 cc -bundle -undefined
dynamic_lookup -L/usr/local/lib Baz.o -o blib/arch/auto/Foo/Bar/Baz/
Baz.bundle \
\

powerpc-apple-darwin8-gcc-4.0.1: Baz.o: No such file or directory
powerpc-apple-darwin8-gcc-4.0.1: no input files
make: *** [blib/arch/auto/Foo/Bar/Baz/Baz.bundle] Error 1

Playing around with MODULE and PACKAGE wasn't fruitful. I uploaded a
tarball with the test in case you'd like to see the actual files:

http://www.hashref.com/tmp/Foo-Bar-Baz.tar.gz

-- fxn

[Marvin Humphrey]

Xavier,

Changing this line in Makefile.PL...

NAME => 'Foo::Bar::Baz',

to this...

NAME => 'Foo::Bar::Baz::XSModule',

... triggers compilation for me.

I'm curious why you're doing things this way. I have a similar
scheme going on for my 50+ module distro, mainly because I want to
have the XS code and the Perl code in the same file to make editing
and revision tracking easier. To elaborate on posts from earlier
this week, this is how it works:

Before Makefile.PL invokes WriteMakefile, it generates KinoSearch.xs,
from scratch, on the fly. It starts with a little bit of header code
in a here-doc:

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

#define NEED_sv_2pv_nolen
#include "ppport.h"

#include "KinoGlobals.h"
#include "KinoDataTypes.h"
#include "KinoIO.h"

MODULE = KinoSearch PACKAGE = KinoSearch

PROTOTYPES: disable

SV*
_dummy_function()
CODE:
RETVAL = newSVpv("Hello World", 0);
OUTPUT:
RETVAL

(If _dummy_function isn't there, compilation fails -- I think xsubpp
doesn't know what to do with a blank MODULE/PACKAGE section.) Then
it walks the file hierarchy in lib/ opening up .pm modules and
looking for code like this:

1;
__END__
__XS__

MODULE = KinoSearch PACKAGE = KinoSearch::Store::InStream

[...]

__POD__

=head1 NAME

It adds everything between the __XS__ token and the next line-
starting-double-underscore-token to KinoSearch.xs. Only one C shared
library gets built -- the one loaded by KinoSearch.pm. Here's
KinoSearch.pm, minus the docs:

package KinoSearch;
use strict;
use warnings;

use 5.008003;

our $VERSION = '0.05_03';
use constant K_DEBUG => 0;

use base qw( Exporter );

use XSLoader;
# This loads a large number of disparate subs.
# See the docs for KinoSearch::Util::ToolSet.
XSLoader::load( 'KinoSearch', $KinoSearch::VERSION );

our @EXPORT_OK = qw( K_DEBUG );

1;

Though it's done via a ToolSet (see the new ToolSet module on CPAN),
effectively every module in the KinoSearch suite has "use
KinoSearch;" at the top, which causes the C library to load. That's
all KinoSearch.pm does, actually -- it has no public UI.

To come back to where we started... the line in my Makefile.PL which
corresponds to yours is:

NAME => 'KinoSearch',

That's also the name of the distro. I'm not sure about the simplest
way to make things work if you want to distribute Foo::Bar::Baz and
not Foo::Bar::Baz::XSModule; a second Makefile.PL in another
directory would probably do the trick, but there may be an easier way.

HTH,

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

Post by Xavier Noria

Post by Marvin Humphrey
Try it with the .xs file in the root directory of the distro --
the same directory as Makefile.PL.

Post by Xavier Noria
What are MODULE and PACKAGE set to?

For the above config, set...
MODULE = Foo::Bar::Baz::XSModule PACKAGE = Foo::Bar::Baz
Foo::Bar::Baz::XSModule should contain the XSLoader::load
command. Foo::Bar::Baz should "use Foo::Bar::Baz::XSModule;", but
should *not* contain an XSLoader::load command, or you'll get
redefinition warnings.

% make
cp lib/Foo/Bar/Baz.pm blib/lib/Foo/Bar/Baz.pm
cp lib/Foo/Bar/Baz/XSModule.pm blib/lib/Foo/Bar/Baz/XSModule.pm
Running Mkbootstrap for Foo::Bar::Baz ()
chmod 644 Baz.bs
rm -f blib/arch/auto/Foo/Bar/Baz/Baz.bundle
env MACOSX_DEPLOYMENT_TARGET=10.3 cc -bundle -undefined
dynamic_lookup -L/usr/local/lib Baz.o -o blib/arch/auto/Foo/Bar/
Baz/Baz.bundle \
\
powerpc-apple-darwin8-gcc-4.0.1: Baz.o: No such file or directory
powerpc-apple-darwin8-gcc-4.0.1: no input files
make: *** [blib/arch/auto/Foo/Bar/Baz/Baz.bundle] Error 1
Playing around with MODULE and PACKAGE wasn't fruitful. I uploaded
http://www.hashref.com/tmp/Foo-Bar-Baz.tar.gz
-- fxn

[Xavier Noria]

Post by Marvin Humphrey
Xavier,
Changing this line in Makefile.PL...
NAME => 'Foo::Bar::Baz',
to this...
NAME => 'Foo::Bar::Baz::XSModule',
... triggers compilation for me.
I'm curious why you're doing things this way.

Because the main module in the distribution is a pure Perl module.
Some of it is factored out for speed to C, and I wanted to put that
in a second, auxiliar module. Maybe this is just not the natural way
to prepare a distro with XS (I am a newbie here).

Post by Marvin Humphrey
I have a similar scheme going on for my 50+ module distro, mainly
because I want to have the XS code and the Perl code in the same
file to make editing and revision tracking easier. To elaborate on

Thanks for such a detailed explanation.

-- fxn

[Marvin Humphrey]

Post by Xavier Noria
Because the main module in the distribution is a pure Perl module.
Some of it is factored out for speed to C, and I wanted to put that
in a second, auxiliar module. Maybe this is just not the natural
way to prepare a distro with XS (I am a newbie here).

Readonly and Readonly::XS may serve as a useful model. Those are
distributed separately. NAME isn't an issue if you make 2 distros.
If you want people to be able to use the pure Perl version on its own
even if they can't use the XS version, I'd do that.

If the module's functionality is part Perl and part XS, the standard
way of doing things is to put everything in the same package. In
fact, it's usually a good idea for maintenance reasons do everything
you can in Perl and only implement stuff in XS that 1) can't be done
any other way and/or 2) has been identified as a performance
bottleneck and can justify the extra maintenance burden of XS.

Post by Xavier Noria
Thanks for such a detailed explanation.

It's time for me to give back after having gotten a bunch of good
advice here recently. :)

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

[Salvador Fandino]

Post by Marvin Humphrey
Xavier,
Changing this line in Makefile.PL...
NAME => 'Foo::Bar::Baz',
to this...
NAME => 'Foo::Bar::Baz::XSModule',
... triggers compilation for me.
I'm curious why you're doing things this way.

Because the main module in the distribution is a pure Perl module. Some
of it is factored out for speed to C, and I wanted to put that in a
second, auxiliar module. Maybe this is just not the natural way to
prepare a distro with XS (I am a newbie here).

as Marvin has already toll you, the usual way is to put everything on
the same package, but if you want to divide it on two it is also
possible because you can include packages inside other packages and they
will be recursively build.

For instance, create both packages skeletons with h2xs

$ h2xs -AX -n Foo # no XS on this one
$ h2xs -A -n Foo::LowXS

Then move the XS module directory inside its parent:

$ mv Foo-XS Foo/LowXS

Recreate the Foo/MANIFEST and put the actual code in place, and that's all.

Though you should remember that Foo and Foo::LowXS are different
modules, you will probably need to "use Foo::LowXS" from Foo.

Cheers,

- Salva