Writing Python Extensions in C++ with PyCXX

Barry Scott barry@barrys-emacs.org

Contents

Acknowledgments

Thank you to Geoffrey Furnish for patiently teaching me the finer points of C++ and its template facility, and his critique of PyCXX in particular. With version 4 I welcome Barry Scott as co-author. -- Paul Dubois

Paul is no longer contributing to PyCXX. Thanks for all your great work on PyCXX Paul. -- Barry Scott.

Overview

PyCXX is designed to make it easier to extend Python with C++

PyCXX Version 6.1 and later supports both Python 2 and Python 3.

PyCXX Version 7.1 and later support the Python 3 limited API (PEP-384).

PyCXX is a set of C++ facilities to make it easier to write Python extensions. The chief way in which PyCXX makes it easier to write Python extensions is that it greatly increases the probability that your program will not make a reference-counting error and will not have to continually check error returns from the Python C API. PyCXX integrates Python with C++ in these ways:

Download and Installation

Download PyCXX from http://sourceforge.net/projects/cxx/.

The distribution layout is:

DirectoryDescription
.Makefile for Unix and Windows, Release documentation
./CXXHeader files
./SrcSource files
./DocDocumentation
./DemoTesting and Demonstartion files

To use PyCXX you use its include files and add its source routines to your module.

Installation:

The header file CXX/config.h may need to be adjusted for the compiler you use. As of this writing, only a fairly obscure reference to part of the standard library needs this adjustment. Unlike prior releases, PyCXX now assumes namespace support and a standard C++ library.

Example code

The Demo directory of the distribution contains examples of how to use many of the facilities in PyCXX.

In the top level folder of the PyCXX distributions are a number of makefiles that can be used to build and run the example code.

For example on Linux example_linux_py30.mak will build and run the example code.

make -f example_linux_py30.mak clean test
SourceDescription
Demo/test_simple.py, Demo/simple.cxxThe simplest code needed to create a module and classes in C++.
Demo/test_example.py, Demo/example.cxxThe original PyCXX example code. It is now the main test suite for PyCXX.
Demo/range.hxx, Demo/range.cxxImpliments the range object use by example.cxx.

PyCXX - Supporting Python 3 limited API (PEP-384)

Starting with Python 3.4 and PyCXX 7.1.0 it is possible to create extensions that use the Python limited API. (It was not possible to support Python 3.3)

Choose the oldest version of python that you want support and the binary extension will run in that version and all the newer versions.

Define Py_LIMITED_API when compiling all your code and the PyCXX code.

The value of Py_LIMITED_API is the first python version that you want to support.

Note: Some of the PyCXX API cannot be supported in the Py_LIMITED_API mode. The header files only include classes and functions that can be supported.

PyCXX - Supporting Python 2 and Python 3

It is possible to have common code that can be compiled to work with Python 2 or Python 3.

Define PYCXX_PYTHON_2TO3 to turn on the compatibility support. When compiling against Python 2 this means faking up the Python 3 API and when compiling against Python 3 faking the old Python 2 API.

The changes from Python 2 to Python 3 that require code changes are:

This means that you will need to:

Use of namespaces

All PyCXX assets are in namespace "Py". You need to include the Py:: prefix when referring to them, or include the statement:

using namespace Py;

Wrapper for standard objects: <CXX/Objects.hxx>

Header file CXX/Objects.hxx requires adding file Src/cxxsupport.cxx and Src/IndirectPythonInterface.cxx to your module sources. CXX/Objects.hxx provides a set of wrapper classes that allow you access to most of the Python C API using a C++ notation that closely resembles Python. For example, this Python:

d = {}
d[ "a" ] = 1
d[ "b" ] = 2
alist = d.keys()
print alist

Can be written in C++:

Py::Dict d;
Py::List alist;
d[ "a" ] = Py::Long( 1 );
d[ "b" ] = Py::Long( 2 );
alist = d.keys();
std::cout << alist << std::endl;

You can optionally use the CXX/Extensions.hxx facility described later to define Python extension modules and extension objects.

We avoid programming with Python object pointers

The essential idea is that we avoid, as much as possible, programming with pointers to Python objects, that is, variables of type PyObject *. Instead, we use instances of a family of C++ classes that represent the usual Python objects. This family is easily extendible to include new kinds of Python objects.

For example, consider the case in which we wish to write a method, taking a single integer argument, that will create a Python dict and insert into it that the integer plus one under the key value. In C we might do that as follows:

static PyObject *mymodule_addvalue( PyObject *args )
{
    PyObject *d;
    PyObject *f;
    int k;
    PyArgs_ParseTuple( args, "i", &k );
    d = PyDict_New();
    if( !d )
        return NULL;

    f = PyInt_NEW( k+1 );
    if( !f )
    {
        Py_DECREF( d ); /* have to get rid of d first */
        return NULL;
    }
    if( PyDict_SetItemString( d, "value", f ) == -1 )
    {
        Py_DECREF( f );
        Py_DECREF( d );
        return NULL;
    }

    return d;
}

If you have written a significant Python extension, this tedium looks all too familiar. The vast bulk of the coding is error checking and cleanup. Now compare the same thing written in C++ using CXX/Objects.hxx. The things with Python-like names ( Long, Dict, Tuple ) are from CXX/Objects.hxx.

static PyObject *mymodule_addvalue( PyObject *pargs )
{ 
    try
    {
        Tuple args( pargs ); 
        args.verify_length( 1 ); 

        Dict d; 
        Long k = args[0]; 
        d["value"] = k + 1;

        return new_reference_to( d ); 
    } 
    catch( const PyException & )
    { 
        return NULL;
    }
}

If there are not the right number of arguments or the argument is not an integer, an exception is thrown. In this case we choose to catch it and convert it into a Python exception. The C++ exception handling mechanism takes care all the cleanup.

Note that the creation of the Long k got the first argument and verified that it is an Long.

Just to peek ahead, if you wrote this method in an ExtensionModule-derived module of your own, it would be a method and it could be written as:

Object addvalue( const Tuple &args )
{
    args.verify_length( 1 );
    Dict d;
    Long k = args[0];
    d["value"] = k + 1;
    return d;
}

The basic concept is to wrap Python pointers

The basic concept of CXX/Objects.hxx is to create a wrapper around each PyObject * so that the reference counting can be done automatically, thus eliminating the most frequent source of errors. In addition, we can then add methods and operators so that Python objects can be manipulated in C++ much like you would in Python.

Each Object contains a PyObject * to which it owns a reference. When an Object is destroyed, it releases its ownership on the pointer. Since C++ calls the destructors on objects that are about to go out of scope, we are guaranteed that we will keep the reference counts right even if we unexpectedly leave a routine with an exception.

As a matter of philosophy, CXX/Objects.hxx prevents the creation of instances of its classes unless the instance will be a valid instance of its class. When an attempt is made to create an object that will not be valid, an exception is thrown.

Class Object represents the most general kind of Python object. The rest of the classes that represent Python objects inherit from it.

Object
Type
Float
Long
Complex
Char
Sequence -> SeqBase<T>
    String
    Tuple
    List
Mapping -> MapBase<T>
    Dict
Callable
Module

There are several constructors for each of these classes. For example, you can create an Long from an integer as in

Long s( 3 )

However, you can also create an instance of one of these classes using any PyObject * or another Object. If the corresponding Python object does not actually have the type desired, an exception is thrown. This is accomplished as follows. Class Object defines a virtual function accepts:

virtual bool accepts( PyObject *p )

The base class version of accepts returns true for any pointer p except 0. This means we can create an Object using any PyObject *, or from any other Object. However, if we attempt to create an Long from a PyObject *, the overridding version of accepts in class Long will only accept pointers that correspond to Python ints. Therefore if we have a Tuple t and we wish to get the first element and be sure it is an Long, we do

Long first_element = t[0]

This will not only accomplish the goal of extracting the first element of the Tuple t, but it will ensure that the result is an Long. If not, an exception is thrown. The exception mechanism is discussed later.

global methods

global methods
Returns Name( signature ) Comment
ObjectasObject( PyObject *p )Convert an owned Python pointer into a PyCXX Object
PyObject *Null()return (PyObject *)NULL
PyObject *new_reference_to( PyObject *p )Increment the reference count of p
PyObject *new_reference_to( const Object &g )Increment the reference count of g
ObjectNone()Return the Python None opject
ObjectTrue()Return the Python True opject
ObjectFalse()Return the Python False opject

Comparisons

PyCXX implements a full set of comparison operators (==, !=, >=, >, < <=) for the PyCXX objects.

BetweenAnd
ObjectObject
LongLong
Longlong
Longint
LongPY_LONG_LONG
FloatFloat
Floatdouble
SeqBasea<T>::iteratorSeqBase<T>::iterator
SeqBase<T>::const_iteratorSeqBase<T>::const_iterator
Sequence::iteratorSequence::iterator
Sequence::const_iteratorSequence::const_iterator
MapBase<T>::iteratorMapBase<T>::iterator
MapBase<T>::const_iteratorMapBase<T>::const_iterator
Mapping::iteratorMapping::iterator
Mapping::const_iteratorMapping::const_iterator

Class Object

Class Object serves as the base class for the other classes. Its default constructor constructs a Py_None, the unique object of Python type None. The interface to Object consists of a large number of methods corresponding to the operations that are defined for every Python object. In each case, the methods throw an exception if anything goes wrong.

There is no method corresponding to PyObject_SetItem with an arbitrary Python object as a key. Instead, create an instance of a more specific child of Object and use the appropriate facilities.

The comparison operators use the Python comparison function to compare values. The method is is available to test for absolute identity.

A conversion to standard library string type std::string is supplied using method as_string. Stream output of PyCXX Object instances uses this conversion, which in turn uses the Python object's str() representation.

All the numeric operators are defined on all possible combinations of Object, long, and double. These use the corresponding Python operators, and should the operation fail for some reason, an exception is thrown.

Dealing with pointers returned by the Python C API

Often, PyObject * pointers are acquired from some function, particularly functions in the Python C API. If you wish to make an object from the pointer returned by such a function, you need to know if the function returns you an owned or unowned reference. Unowned references are unusual but there are some cases where unowned references are returned.

Usually, Object and its children acquire a new reference when constructed from a PyObject *. This is usually not the right behavior if the reference comes from one of the Python C API calls.

If p is an owned reference, you can add the boolean true as an extra argument in the creation routine, Object( p, true ), or use the function asObject( p ) which returns an Object created using the owned reference. For example, the routine PyString_FromString returns an owned reference to a Python string object. You could write:

Object w = asObject( PyString_FromString( "my string" ) );

or using the constructor,

Object w( PyString_FromString( "my string" ), true );

In fact, you would never do this, since PyCXX has a class String and you can just say:

String w( "my string" );

Indeed, since most of the Python C API is similarly embodied in Object and its descendents, you probably will not use asObject all that often.

Class Object
Returns Name( signature ) Comment
Basic Methods
explicit Object( PyObject *pyob=Py_None, bool owned=false ) Construct from pointer.
explicit Object( const Object &ob ) Copycons; acquires an owned reference.
Object & operator=( const Object &rhs ) Acquires an owned reference.
Object & operator=( PyObject *rhsp ) Acquires an owned reference.
virtual ~Object() Releases the reference.
void increment_reference_count() Explicitly increment the count
void decrement_reference_count() Explicitly decrement count but not to zero
PyObject* operator*() const Lends the pointer
PyObject* ptr() const Lends the pointer
virtual bool accepts( PyObject *pyob ) const Would assignment of pyob to this object succeed?
std::string as_string() const str() representation
Python API Interface
int reference_count() const reference count
Type type() const associated type object
String str() const str() representation
String repr() const repr() representation
bool hasAttr( const std::string &s ) const hasattr( this, s )
Object getAttr( const std::string &s ) const getattr( this, s )
Object getItem( const Object &key ) const getitem( this, key )
long hashValue() const hash( this )
void setAttr( const std::string &s,
const Object &value )
this.s = value
void delAttr( const std::string &s ) del this.s
void delItem( const Object &key ) del this[key]
bool isCallable() const does this have callable behavior?
bool isList() const is this a Python list?
bool isMapping() const does this have mapping behaviors?
bool isNumeric() const does this have numeric behaviors?
bool isSequence() const does this have sequence behaviors?
bool isTrue() const is this true in the Python sense?
bool isType( const Type &t ) const is type( this ) == t?
bool isTuple() const is this a Python tuple?
bool isString() const is this a Python string?
bool isUnicode() const is this a Python Unicode string?
bool isDict() const is this a Python dictionary?
Comparison Operators
bool is( PyObject *pother ) const test for identity
bool is( const Object &other ) const test for identity
bool operator==( const Object &o2 ) const Comparisons use Python rich compare
bool operator!=( const Object &o2 ) const Comparisons use Python rich compare
bool operator>=( const Object &o2 ) const Comparisons use Python rich compare
bool operator<=( const Object &o2 ) const Comparisons use Python rich compare
bool operator<( const Object &o2 ) const Comparisons use Python rich compare
bool operator>( const Object &o2 ) const Comparisons use Python rich compare

The Basic Types

Corresponding to each of the basic Python types is a class that inherits from Object. Here are the interfaces for those types. Each of them inherits from Object and therefore has all of the inherited methods listed for Object. Where a virtual function is overridden in a class, the name is underlined.

Class Type

Class Type corresponds to Python type objects. There is no default constructor.

class Type
Returns Name and Signature Comments
explicit Type( PyObject *pyob, bool owned = false ) Constructor
explicit Type( const Object &ob ) Constructor
explicit Type( const Type &t ) Copycons
Type& operator=( const Object &rhs ) Assignment
Type& operator=( PyObject *rhsp ) Assignment
virtual bool accepts( PyObject *pyob ) const Uses PyType_Check

Class Long

Class Long, derived publically from Object, corresponds to Python type long. In Python, a long is an integer type of unlimited size. Implicit conversions to both double and long are provided, although the latter may of course fail if the number is actually too big. All constructors are explicit. The default constructor produces a Python long zero.

class Long
Returns Name and Signature Comments
explicit Long( PyObject *pyob, bool owned = false ) Constructor
explicit Long( const Long &ob ) Constructor
explicit Long( long v = 0L ) Construct from long
explicit Long( int v ) Contruct from int
explicit Long( const Object &ob ) Copycons
Long & operator=( const Object &rhs ) Assignment
Long & operator=( PyObject *rhsp ) Assignment
virtual bool accepts( PyObject *pyob ) const Based on PyLong_Check
double operator double() const Implicit conversion to double
long operator long() const Implicit conversion to long
Long & operator=( int v ) Assign from int
Long & operator=( long v ) Assign from long

Class Float

Class Float corresponds to Python floats, which in turn correspond to C double. The default constructor produces the Python float 0.0.

class Float
Returns Name and Signature Comments
explicit Float( PyObject *pyob, bool owned = false ) Constructor
Float( const Float &f )   Construct from float
explicit Float( double v=0.0 ) Construct from double
explicit Float( const Object &ob ) Copycons
Float& operator=( const Object &rhs ) Assignment
Float& operator=( PyObject *rhsp ) Assignment
virtual bool accepts( PyObject *pyob ) const Based on PyFloat_Check
double operator double() const Implicit conversion to double
Float & operator=( double v ) Assign from double
Float & operator=( int v ) Assign from int
Float & operator=( long v ) Assign from long
Float & operator=( const Long &iob ) Assign from Long

Sequences

PyCXX implements a quite sophisticated wrapper class for Python sequences. While every effort has been made to disguise the sophistication, it may pop up in the form of obscure compiler error messages, so in this documentation we will first detail normal usage and then discuss what is under the hood.

The basic idea is that we would like the subscript operator [] to work properly, and to be able to use STL-style iterators and STL algorithms across the elements of the sequence.

Sequences are implemented in terms of a templated base class, SeqBase<T>. The parameter T is the answer to the question, sequence of what? For Lists, for example, T is Object, because the most specific thing we know about an element of a List is simply that it is an Object.( Class List is defined below; it is a descendent of Object that holds a pointer to a Python list ). For strings, T is Char, which is a wrapper in turn of Python strings whose length is one.

For convenience, the word Sequence is a typedef of SeqBase<Object>.

General sequences

Suppose you are writing an extension module method that expects the first argument to be any kind of Python sequence, and you wish to return the length of that sequence. You might write:

static PyObject*
my_module_seqlen( PyObject *args ) {
try
    {
    Tuple t( args );       // set up a Tuple pointing to the arguments.
    if( t.length() != 1 ) 
         throw PyException( "Incorrect number of arguments to seqlen." );
    Sequence s = t[0];   // get argument and be sure it is a sequence
    return new_reference_to( Long( s.length() ) );
    }
catch( const PyException  &)
    {
    return Py_Null;
    }
}

As we will explain later, the try/catch structure converts any errors, such as the first argument not being a sequence, into a Python exception.

Subscripting

When a sequence is subscripted, the value returned is a special kind of object which serves as a proxy object. The general idea of proxy objects is discussed in Scott Meyers' book, "More Effective C++". Proxy objects are necessary because when one subscripts a sequence it is not clear whether the value is to be used or the location assigned to. Our proxy object is even more complicated than normal because a sequence reference such as s[i] is not a direct reference to the i'th object of s.

In normal use, you are not supposed to notice this magic going on behind your back. You write:

Object t;
Sequence s;
s[2] = t + s[1]

and here is what happens: s[1] returns a proxy object. Since there is no addition operator in Object that takes a proxy as an argument, the compiler decides to invoke an automatic conversion of the proxy to an Object, which returns the desired component of s. The addition takes place, and then there is an assignment operator in the proxy class created by the s[2], and that assignment operator stuffs the result into the 2 component of s.

It is possible to fool this mechanism and end up with a compiler failing to admit that a s[i] is an Object. If that happens, you can work around it by writing Object( s[i] ), which makes the desired implicit conversion, explicit.

Iterators

Each sequence class provides the following interface. The class seqref<T> is the proxy class. We omit the details of the iterator, const_iterator, and seqref<T> here. See CXX/Objects.hxx if necessary. The purpose of most of this interface is to satisfy requirements of the STL.

The SeqBase<T> Interface

SeqBase<T> inherits from Object.

class SeqBase>T<
Type Name
typedef int size_type
typedef seqref<T> reference
typedef T const_reference
typedef seqref<T>* pointer
typedef int difference_type
virtual size_type max_size() const
virtual size_type capacity() const;
virtual void swap( SeqBase<T> &c );
virtual size_type size() const;
explicit SeqBase<T>();
explicit SeqBase<T>( PyObject *pyob, bool owned = false );
explicit SeqBase<T>( const Object &ob );
SeqBase<T> & operator=( const Object &rhs );
SeqBase<T> & operator=( PyObject *rhsp );
virtual bool accepts( PyObject *pyob ) const;
size_type length() const ;
const T operator[]( size_type index ) const;
seqref<T> operator[]( size_type index );
virtual T getItem( size_type i ) const;
virtual void setItem( size_type i, const T &ob );
SeqBase<T> repeat( int count ) const;
SeqBase<T> concat( const SeqBase<T> &other ) const ;
const T front() const;
seqref<T> front();
const T back() const;
seqref<T> back();
void verify_length( size_type required_size );
void verify_length( size_type min_size, size_type max_size );
class iterator;
iterator begin();
iterator end();
class const_iterator;
const_iterator begin() const;
const_iterator end() const;

Any heir of class Object that has a sequence behavior should inherit from class SeqBase<T>, where T is specified as the type of object that represents the individual elements of the sequence. The requirements on T are that it has a constructor that takes a PyObject *as an argument, that it has a default constructor, a copy constructor, and an assignment operator. In short, any properly defined heir of Object will work.

Classes Char and String

Python strings are unusual in that they are immutable sequences of characters. However, there is no character type per se; rather, when subscripted strings return a string of length one. To simulate this, we define two classes Char and String. The Char class represents a Python string object of length one. The String class represents a Python string, and its elements make up a sequence of Char's.

The user interface for Char is limited. Unlike String, for example, it is not a sequence.

The Char interface

Char inherits from Object. Char holds a single Unicode character.

unicodestring is a typedef for std::basic_string<Py_UNICODE>

class Char
Type Name
explicit Char( PyObject *pyob, bool owned = false )
Char( const Object &ob )
Char( int v )
Char( Py_UNICODE v )
Char( const unicodestring &v )
Char & operator=( const Object &o )
Char & operator=( PyObject *p )
Char & operator=( int v )
Char & operator=( Py_UNICODE v )
Char & operator=( unicodestring &v )
operator String() const

The String Interface

String inherits from SeqBase<Char>. String holds a sequence of Unicode characters.

The following is taken from Pythons's unicode.h.

Many of these APIs take two arguments encoding and errors. These parameters encoding and errors have the same semantics as the ones of the builtin unicode() API.

Setting encoding to NULL causes the default encoding to be used.

Error handling is set by errors which may also be set to NULL meaning to use the default handling defined for the codec. Default error handling for all builtin codecs is "strict" (ValueErrors are raised).

The codecs all use a similar interface. Only deviation from the generic ones are documented.

class String
Type Name
explicit String( PyObject *pyob, bool owned = false )
String( const Object &ob )
String()
String( const char *latin1 )
String( const char *latin1, Py_ssize_t size )
String( const std::string &latin1 )
String( const std::string &v, const char *encoding, const char *error=NULL )
String( const char *s, const char *encoding, const char *error=NULL )
String( const char *s, Py_ssize_t len, const char *encoding, const char *error=NULL )
String & operator=( const Object &o )
String & operator=( PyObject *p )
String & operator=( const unicodestring &v )
size_type size() const
size_type capacity() const
unicodestring as_unicodestring() const
std::string operator std::string() const
String encode( const char *encoding, const char *error="strict" )
std::string as_std_string( const char *encoding=NULL, const char *error="strict" ) const

Classes Byte and Bytes

Bytes corresponds to the Python type byte.

The Byte interface

Byte inherits from Object. Byte holds a single 8-bit byte of data.

class Byte
Type Name
explicit Byte( PyObject *pyob, bool owned = false )
Byte( const Object &ob )
Byte( const std::string &v )
Byte( char v )
Byte & operator=( const Object &o )
Byte & operator=( PyObject *p )
Byte & operator=( const std::string &v )
Byte & operator=( char v )
operator Bytes() const

The Bytes Interface

Bytes inherits from SeqBase<Byte>. Bytes holds a sequence of 8-bit data.

class Bytes
Type Name
explicit Bytes( PyObject *pyob, bool owned = false )
Bytes( const Object &ob )
Bytes()
Bytes( const char *v )
Bytes( const char *v, Py_ssize_t size )
Bytes( const std::string &v )
Bytes( const std::string &v, Py_ssize_t size )
Bytes( const char *v )
Bytes( const char *v, Py_ssize_t size )
Bytes & operator=( const Object &o )
Bytes & operator=( PyObject *p )
Bytes & operator=( const std::string &v )
size_type size() const
size_type capacity() const
String decode( const char *encoding, const char *error="strict" )
std::string operator std::string() const
Bytes encode( const char *encoding, const char *error="strict" )
std::string as_std_string() const

Class Tuple

Class Tuple represents Python tuples. A Tuple is a Sequence. There are two kinds of constructors: one takes a PyObject *as usual, the other takes an integer number as an argument and returns a Tuple of that length, each component initialized to Py_None. The default constructor produces an empty Tuple.

Tuples are not immutable, but attempts to assign to their components will fail if the reference count is not 1. That is, it is safe to set the elements of a Tuple you have just made, but not thereafter.

Example: create a Tuple containing( 1, 2, 4 )

Tuple t( 3 );
t[0] = Long( 1 );
t[1] = Long( 2 );
t[2] = Long( 4 );

Example: create a Tuple from a list:

Dict d
...
Tuple t( d.keys() )

Tuple inherits from Sequence.. Special run-time checks prevent modification if the reference count is greater than one.

class Tuple
Type Name Comment
virtual void setItem( int offset, const Object &ob ) setItem is overridden to handle tuples properly.
explicit Tuple( PyObject *pyob, bool owned = false )
Tuple( const Object &ob )
explicit Tuple( int size = 0 ) Create a tuple of the given size. Items initialize to Py_None. Default is an empty tuple.
explicit Tuple( const Sequence &s ) Create a tuple from any sequence.
Tuple& operator=( const Object &rhs )
Tuple& operator=( PyObject *rhsp )
Tuple getSlice( int i, int j ) const Equivalent to python's t[i:j]

Class TupleN

Class TupleN is an easy way to make a Tuple of N items.

Example: create a Tuple containing( 1, 2, 4 )

TupleN t3( Long( 1 ), Long( 2 ), Long( 3 ) );

Example: create a Tuple containing( "Hello", "World" )

TupleN t2( String( "Hello" ), String( "Hello" ) );
class TupleN
Type Name Comment
  TupleN() Tuple of 0 elements
  TupleN( const Object &ob1 ) Tuple of 1 element
  TupleN( const Object &ob1, const Object &ob2 ) Tuple of 2 elements
  TupleN( const Object &ob1, const Object &ob2, const Object &ob3 ) Tuple of 3 elements
  TupleN( const Object &ob1, const Object &ob2, const Object &ob3,
        const Object &ob4 )
Tuple of 4 elements
  TupleN( const Object &ob1, const Object &ob2, const Object &ob3,
        const Object &ob4, const Object &ob5 )
Tuple of 5 elements
  TupleN( const Object &ob1, const Object &ob2, const Object &ob3,
        const Object &ob4, const Object &ob5, const Object &ob6 )
Tuple of 6 elements
  TupleN( const Object &ob1, const Object &ob2, const Object &ob3,
        const Object &ob4, const Object &ob5, const Object &ob6,
        const Object &ob7 )
Tuple of 7 elements
  TupleN( const Object &ob1, const Object &ob2, const Object &ob3,
        const Object &ob4, const Object &ob5, const Object &ob6,
        const Object &ob7, const Object &ob8 )
Tuple of 8 elements
  TupleN( const Object &ob1, const Object &ob2, const Object &ob3,
        const Object &ob4, const Object &ob5, const Object &ob6,
        const Object &ob7, const Object &ob8, const Object &ob9 )
Tuple of 9 elements

Class List

Class List represents a Python list, and the methods available faithfully reproduce the Python API for lists. A List is a Sequence.

List inherits from Sequence.

class List
Type Name Comment
explicit List( PyObject *pyob, bool owned = false )
List( const Object &ob )
List( int size = 0 ) Create a list of the given size. Items initialized to Py_None. Default is an empty list.
List( const Sequence &s ) Create a list from any sequence.
List& operator=( const Object &rhs )
List& operator=( PyObject *rhsp )
List getSlice( int i, int j ) const
void setSlice( int i, int j, const Object &v )
void append( const Object &ob )
void insert( int i, const Object &ob )
void sort() Sorts the list in place, using Python's member function. You can also use the STL sort function on any List instance.
void reverse() Reverses the list in place, using Python's member function.

Mappings

A class MapBase<T> is used as the base class for Python objects with a mapping behavior. The key behavior of this class is the ability to set and use items by subscripting with strings. A proxy class mapref<T> is defined to produce the correct behavior for both use and assignment.

For convenience, Mapping is a typedef for MapBase<Object>.

The MapBase<T> interface

MapBase<T> inherits from Object. T should be chosen to reflect the kind of element returned by the mapping.

class MapBase<T>
Type Name Comment
T operator[]( const std::string &key ) const
mapref<T> operator[]( const std::string &key )
int length() const Number of entries.
int hasKey( const std::string &s ) const Is m[s] defined?
T getItem( const std::string &s ) const m[s]
virtual void setItem( const std::string &s, const Object &ob ) m[s] = ob
void delItem( const std::string &s ) del m[s]
void delItem( const Object &s )
List keys() const A list of the keys.
List values() const A list of the values.
List items() const Each item is a key-value pair.

Class Dict

Class Dict represents Python dictionarys. A Dict is a Mapping. Assignment to subscripts can be used to set the components.

Dict d
d["Paul Dubois"] = "( 925 )-422-5426"

Dict inherits from MapBase<Object>.

class Dict
Type Name Comment
explicit Dict( PyObject *pyob, bool owned = false )
Dict( const Dict &ob )
Dict() Creates an empty dictionary
Dict& operator=( const Object &rhs )
Dict& operator=( PyObject *rhsp )

Clsss Callable.

Class Callable provides an interface to those Python objects that support a call method. Class Module holds a pointer to a module. If you want to create an extension module, however, see the extension facility. There is a large set of numeric operators.

class Callable
TypeNameComment
explicitCallable( PyObject *pyob, bool owned = false )
Callable &operator=( const Object &rhs )
Callable &operator=( PyObject *rhsp )
Objectapply( const Tuple &args ) constCall the object with the given positional arguments
Objectapply( const Tuple &args, const Dict &kwd ) constCall the object with the given positional and keyword arguments
Objectapply( PyObject *pargs = 0 ) const Call the object with args as the arguments. Checks that pargs is a tuple.

Interface to class Module

class Module
Type Name Comment
explicit Module( PyObject *pyob, bool owned = false )
explicit Module( const std::string name ) Construct from name of module; does the import if needed.
Module( const Module &ob ) Copy constructor
Module& operator=( const Object &rhs ) Assignment
Module& operator=( PyObject *rhsp ) Assignment

Numeric interface

Unary operators for plus and minus, and binary operators +, -, *, /, and % are defined for pairs of objects and for objects with scalar integers or doubles( in either order ). Functions abs( ob ) and coerce( o1, o2 ) are also defined.

The signature for coerce is:

inline std::pair<Object,Object> coerce( const Object &a, const Object &b )

Unlike the C API function, this simply returns the pair after coercion.

Stream I/O

Any object can be printed using stream I/O, using std::ostream &operator<< ( std::ostream &os, const Object &ob ). The object's str() representation is converted to a standard string which is passed to std::ostream &operator<< ( std::ostream &os, const std::string &).

Exceptions

All the standard python exceptions have a C++ equivilent that can to caught and thrown.

In addition new exceptions can be defined using PyCXX as well.

Exceptions thrown from C++ will be converted into Python exceptions when returning to Python code.

Python exceptions are converted into C++ exceptions when returning from Python code back into C++ code.

class BaseException and its children

All the Python standard exceptions are provided as C++ classes. The C++ class hierarchy mirrors the Python class hierarchy. The base of the exception hierarchy is class BaseException.

The derived exception class, such as IndexError, RuntimeError and ValueError, has a constructor which takes an explanatory string as an argument, and is used in a throw statement such as:

throw IndexError( "Index too large in MyObject access." );

You cannot throw BaseException, but you can catch it.

See the Python documentation for a list of all the standard exceptions.

List of Exceptions

The exception hierarchy mirrors the Python exception hierarchy. The concrete exception classes are shown here. With ValueError being the pattern for all the other expections.

BaseException
Type Interface for class Exception Comment
explicit BaseException()
BaseException( const std::string &reason )
BaseException( PyObject *exception, const std::string &reason )
void clear() Clear the exception.
Object errorType() Returns the type of the exception
Object errorValue() Returns the value of the exception
Python Standard Exceptions
Type Interface for class Exception
void python-standard-exception( const std::string &reason )

Using Exceptions in extension methods

The exception facility allows you to integrate the C++ and Python exception mechanisms. To do this, you must use the style described below when writing module methods in the old C style.

Note: If using the ExtensionModule or PythonExtension mechanisms described below, the method handlers include exception handling so that you only need to use exceptions explicitly in unusual cases.

Catching Exceptions from the Python API or PyCXX.

In the example, some_method, any expections raise from the C++ will be automatically converted into Python exceptions.

Object
some_method( Object &args )
{
    Tuple a( args ); // we know args is a Tuple

    if( a.length() != 2 )
    {
        throw AttributeError( "2 arguments expected" );
    }

    // do something useful
    // and return a result

    return result;
}

And in this example the call_python method is calling back into python and handling ArithmeticError from the "func". Any other expections will be passed back to Python.

The exceptions error type and error value can be obtained with the errorType() and errorValue() functions.

Object
cal_python( Object &args )
{
    Tuple a( args ); // we know args is a Tuple
    Callable func( a[0] );   // first arg expected to be a python callable

    ...
    Tuple call_args( 1 );
    call_args[0] = Long( 42 );
    try
    {
        Object result = func.apply( call_args );
    }
    catch( ArithmeticError &e )
    {
        Object err_type = e.errorType();
        Object err_value = e.errorValue();

        // get the text of the error for reporting
        String message = err_value.str();

        // handle error
        e.clear();
    }

    return result;
}

How to clear an Exception

If you anticipate that an Exception may be thrown and wish to recover from it then add a try/catch block for the expected exception. Then use the method clear() to tell python that the expection has been handled.

catch( ValueError &e )
{
    // handle expection
    e.clear();
}

Extension Facilities

CXX/Extensions.hxx provides facilities for:

These facilities use CXX/Objects.hxx and its support file cxxsupport.cxx.

If you use CXX/Extensions.hxx you must also include source files cxxextensions.c and cxx_extensions.cxx

Creating an Python extension module

The usual method of creating a Python extension module is to declare and initialize its method table in C. This requires knowledge of the correct form for the table and the order in which entries are to be made into it, and requires casts to get everything to compile without warning. The PyCXX header file CXX/Extensions.h offers a simpler method. Here is a sample usage, in which a module named "example" is created. Note that two details are necessary:

To create an extension module, you inherit from class ExtensionModule templated on yourself: In the constructor, you make calls to register methods of this class with Python as extension module methods. In this example, two methods are added( this is a simplified form of the example in Demo/example.cxx ):

class example_module : public ExtensionModule<example_module>
{
public:
    example_module()
    : ExtensionModule<example_module>( "example" )
    {
        add_varargs_method( "sum", &example_module::ex_sum, "sum( arglist ) = sum of arguments" );
        add_varargs_method( "test", &example_module::ex_test, "test( arglist ) runs a test suite" );

        initialize( "documentation for the example module" );
    }

    virtual ~example_module() {}

private:
    Object ex_sum( const Tuple &a ) { ... }
    Object ex_test( const Tuple &a ) { ... }
};

To initialize the extension, you just instantiate one static instance( static so it does not destroy itself! ):

void initexample()
{
    static example_module* example = new example_module;
}

The methods can be written to take Tuples as arguments and return Objects. If exceptions occur they are trapped for you and a Python exception is generated. So, for example, the implementation of ex_sum might be:

Object ex_sum( const Tuple &a )
{
    Float f( 0.0 );
    for( int i = 0; i < a.length(); i++ )
    {
        Float g( a[i] );
        f = f + g;
    }
    return f;
}

class ExtensionModule contains methods to return itself as a Module object, or to return its dictionary.

class ExtensionModule
Type Name Comment
explicit ExtensionModule( char* name ) Create an extension module named "name"
virtual ~ExtensionModule() Destructor
Dict moduleDictionary() const Returns the module dictionary; module must be initialized.
Module module() const This module as a Module.
void add_varargs_method( char *name, method_varargs_function_t method, char *documentation="" ) Add a method to the module.
void add_keyword_method( char *name, method_keyword_function_t method, char *documentation="" Add a method that takes keywords
void initialize()( protected, call from constructor ) Initialize the module once all methods have been added.

The signatures above are:

typedef Object( T::*method_varargs_function_t )( const Tuple &args );
typedef Object( T::*method_keyword_function_t )( const Tuple &args, const Dict &kws
 );

That is, the methods take a Tuple or a Tuple and a Dict, and return an Object. The example below has an &in front of the name of the method; we found one compiler that needed this.

Creating a Python extension type

One of the great things about Python is the way you can create your own object types and have Python welcome them as first-class citizens. Unfortunately, part of the way you have to do this is not great. Key to the process is the creation of a Python "type object". All instances of this type must share a reference to this one unique type object. The type object itself has a multitude of "slots" into which the addresses of functions can be added in order to give the object the desired behavior.

Creating extension objects is of course harder since you must specify how the object behaves and give it methods. This is shown in some detail in the example range.h and range.cxx, with the test routine rangetest.cxx, in directory Demo. If you have never created a Python extension before, you should read the Extension manual first and be very familiar with Python's "special class methods". Then what follows will make more sense.

The basic idea is to inherit from PythonExtension templated on your self

class MyObject: public PythonExtension<MyObject> {...}

As a consequence:

Sample usage of PythonExtension

Here is a brief overview. You create a class that inherits from PythonExtension templated upon itself. You override various methods from PythonExtension to implement behaviors, such as getattr, sequence_item, etc. You can also add methods to the object that are usable from Python using a similar scheme as for module methods above.

One of the consequences of inheriting from PythonExtension is that you are inheriting from PyObject itself. So your class is-a PyObject and instances of it can be passed to the Python C API. Note: this example uses the namespace feature of PyCXX.

Hint: You can avoid needing to specify the Py:: prefix if you include the C++ statement using Py; at the top of your files.

class range: public Py::PythonExtension<range>
{
public:
    ... constructors, data, etc.
    ... methods not callable from Python
    // initializer, see below
    static void init_type();
    // override functions from PythonExtension
    virtual Py::Object repr();
    virtual Py::Object getattr( const char *name );

    virtual int sequence_length();
    virtual Py::Object sequence_item( int i );
    virtual Py::Object sequence_concat( const Py::Object &j );
    virtual Py::Object sequence_slice( int i, int j );

    // define python methods of this object
    Py::Object amethod( const Py::Tuple &args );
    Py::Object value( const Py::Tuple &args );
    Py::Object assign( const Py::Tuple &args ); 
};

To initialize the type we provide a static method that we can call from some module's initializer. We set the name, doc string, and indicate which behaviors range objects support. Then we adds the methods.

void range::init_type()
{
    behaviors().name( "range" );
    behaviors().doc( "range objects: start, stop, step" );
    behaviors().supportRepr();
    behaviors().supportGetattr();
    behaviors().supportSequenceType();

    add_varargs_method( "amethod", &range::amethod, "demonstrate how to document amethod" );
    add_varargs_method( "assign", &range::assign );
    add_varargs_method( "value", &range::value );

    behaviors().readyType();
}

Do not forget to add the call range::init_type() to some module's init function. You will want a method in some module that can create range objects, too.

Your extension class T inherits PythonExtension<T>.

class PythonExtension<T>
Type Name Comment
virtual ~PythonExtension<T>() Destructor
PyTypeObject* type_object() const Returns the object type object.
int check( PyObject *p ) Is p a T?
Protected
void add_varargs_method( char *name, method_keyword_function_t method, char *documentation="" Add a method that takes arguments
void add_keyword_method( char *name, method_keyword_function_t method, char *documentation="" Add a method that takes keywords
static PythonType& behaviors() The type object
void initialize()( protected, call from constructor ) Initialize the module once all methods have been added.

As before the signatures for the methods are Object mymethod( const Tuple &args ) and Object mykeywordmethod( const Tuple &args, const Dict &keys ). In this case, the methods must be methods of T.

To set the behaviors of the object you override some or all of these methods from PythonExtension<T>:

virtual int print( FILE *, int );
virtual Object getattr( const char * );
virtual int setattr( const char *, const Object &);
virtual Object getattro( const Object &);
virtual int setattro( const Object &, const Object &);
virtual int compare( const Object &);
virtual Object repr();
virtual Object str();
virtual long hash();
virtual Object call( const Object &, const Object &);

// Sequence methods
virtual int sequence_length();
virtual Object sequence_concat( const Object &);
virtual Object sequence_repeat( int );
virtual Object sequence_item( int );
virtual Object sequence_slice( int, int );
virtual int sequence_ass_item( int, const Object &);
virtual int sequence_ass_slice( int, int, const Object &);

// Mapping
virtual int mapping_length();
virtual Object mapping_subscript( const Object &);
virtual int mapping_ass_subscript( const Object &, const Object &);

// Number
virtual int number_nonzero();
virtual Object number_negative();
virtual Object number_positive();
virtual Object number_absolute();
virtual Object number_invert();
virtual Object number_int();
virtual Object number_float();
virtual Object number_long();
virtual Object number_oct();
virtual Object number_hex();
virtual Object number_add( const Object &);
virtual Object number_subtract( const Object &);
virtual Object number_multiply( const Object &);
virtual Object number_divide( const Object &);
virtual Object number_remainder( const Object &);
virtual Object number_divmod( const Object &);
virtual Object number_lshift( const Object &);
virtual Object number_rshift( const Object &);
virtual Object number_and( const Object &);
virtual Object number_xor( const Object &);
virtual Object number_or( const Object &);
virtual Object number_power( const Object &, const Object &);

// Buffer
virtual int buffer_getreadbuffer( int, void** );
virtual int buffer_getwritebuffer( int, void** );
virtual int buffer_getsegcount( int* );

Note that dealloc is not one of the functions you can override. That is what your destructor is for. As noted below, dealloc behavior is provided for you by PythonExtension.

Type initialization

To initialize your type, supply a static public member function that can be called from the extension module. In that function, obtain the PythonType object by calling behaviors() and apply appropriate "support" methods from PythonType to turn on the support for that behavior or set of behaviors.

void supportPrint( void );
void supportGetattr( void );
void supportSetattr( void );
void supportGetattro( void );
void supportSetattro( void );
void supportCompare( void );
void supportRepr( void );
void supportStr( void );
void supportHash( void );
void supportCall( void );

void supportSequenceType( int methods_to_support=
                        support_sequence_length |
                        support_sequence_repeat |
                        support_sequence_item |
                        support_sequence_slice |
                        support_sequence_concat );
void supportMappingType( int methods_to_support=
                        support_mapping_length |
                        support_mapping_subscript );
void supportNumberType( int methods_to_support=
                support_number_add |
                support_number_subtract |
                support_number_multiply |
                support_number_remainder |
                support_number_divmod |
                support_number_power |
                support_number_negative |
                support_number_positive |
                support_number_absolute |
                support_number_invert |
                support_number_lshift |
                support_number_rshift |
                support_number_and |
                support_number_xor |
                support_number_or |
                support_number_int |
                support_number_float,
            int inplace_methods_to_support=0
             );
void supportBufferType(  int methods_to_support=
                    support_buffer_getbuffer |
                    support_buffer_releasebuffer );

Then call add_varargs_method or add_keyword_method to add any methods desired to the object.

Notes on memory management and extension objects

Normal Python objects exist only on the heap. That is unfortunate, as object creation and destruction can be relatively expensive. Class PythonExtension allows creation of both local and heap-based objects.

If an extension object is created using operator new, as in:

range* my_r_ref = new range( 1, 20, 3 )

then the entity my_r_ref can be thought of as "owning" the reference created in the new object. Thus, the object will never have a reference count of zero. If the creator wishes to delete this object, they should either make sure the reference count is 1 and then do delete my_r_ref, or decrement the reference with Py_DECREF( my_r_ref ).

Should my_r_ref give up ownership by being used in an Object constructor, all will still be well. When the Object goes out of scope its destructor will be called, and that will decrement the reference count, which in turn will trigger the special dealloc routine that calls the destructor and deletes the pointer.

If the object is created with automatic scope, as in:

range my_r( 1, 20, 3 )

then my_r can be thought of as owning the reference, and when my_r goes out of scope the object will be destroyed. Of course, care must be taken not to have kept any permanent reference to this object. Fortunately, in the case of an exception, the C++ exception facility will call the destructor of my_r. Naturally, care must be taken not to end up with a dangling reference, but such objects can be created and destroyed more efficiently than heap-based PyObjects.