Assignment To Non-Lvalue Cython

This is declaring a C array of 500 C arrays of 500 doubles. That's 500 * 500 packed double values (stored on the stack in this case, unless Cython does something funky) without any indirection, which aids performance and cache utilization, but obviously adds severe restrictions. Maybe you want this, but you should learn enough C to know what that means first. By the way, one restriction is that the size must be a compile-time constant (depending on the C version; C99 and C10 allow it), which is what the first error message is about.

If you do use arrays, you don't initialize the way you did, because that doesn't make any sense. is already 500x500 double variables, and arrays as a whole can't be assigned to (which is what the latter error message is trying to tell you). In particular, list comprehension creates a fully-blown Python list object (which you could also use from Cython, see below) containing fully-blown "boxed" Python objects ( objects, in this case). A list is not compatible with a C array. Use a nested loop with item assignment for the initialization. Lastly, such an array takes 500 * 500 * 8 byte, which is almost 2 MiB. On some systems, that's larger than the entire stack, and on all other systems, it's such a large portion of the stack that it's a bad idea. You should heap-allocate that array.

If you use a Python list, be warned that you won't get much improvement in performance and memory use (assuming your code will mostly be manipulating that list), though you may gain back some convenience in return. You could just leave off the , or use as the type ( should work too, but you gain nothing from it, so you might as well omit it).

A NumPy array may be faster, more memory-efficient, and more convenient to use. If you can implement the performance-critical parts of your algorithm in terms of NumPy's operations, you may gain the desired speedup without using Cython at all.

Overview¶

Cython has native support for most of the C++ language. Specifically:

  • C++ objects can now be dynamically allocated with and keywords.
  • C++ objects can be stack-allocated.
  • C++ classes can be declared with the new keyword .
  • Templated classes and functions are supported.
  • Overloaded functions are supported.
  • Overloading of C++ operators (such as operator+, operator[],…) is supported.

Procedure Overview¶

The general procedure for wrapping a C++ file can now be described as follows:

  • Specify C++ language in script or locally in a source file.
  • Create one or more .pxd files with blocks and (if existing) the C++ namespace name. In these blocks,
    • declare classes as blocks
    • declare public names (variables, methods and constructors)
  • Write an extension modules, from the .pxd file and use the declarations.

A simple Tutorial¶

An example C++ API¶

Here is a tiny C++ API which we will use as an example throughout this document. Let’s assume it will be in a header file called :

and the implementation in the file called :

This is pretty dumb, but should suffice to demonstrate the steps involved.

namespaceshapes{classRectangle{public:intx0,y0,x1,y1;Rectangle();Rectangle(intx0,inty0,intx1,inty1);~Rectangle();intgetArea();voidgetSize(int*width,int*height);voidmove(intdx,intdy);};}
#include"Rectangle.h"namespaceshapes{Rectangle::Rectangle(){}Rectangle::Rectangle(intX0,intY0,intX1,intY1){x0=X0;y0=Y0;x1=X1;y1=Y1;}Rectangle::~Rectangle(){}intRectangle::getArea(){return(x1-x0)*(y1-y0);}voidRectangle::getSize(int*width,int*height){(*width)=x1-x0;(*height)=y1-y0;}voidRectangle::move(intdx,intdy){x0+=dx;y0+=dy;x1+=dx;y1+=dy;}}

Specify C++ language in setup.py¶

The best way to build Cython code from scripts is the function. To make Cython generate and compile C++ code with distutils, you just need to pass the option :

Cython will generate and compile the file (from the ), then it will compile (implementation of the class) and link both objects files together into , which you can then import in Python using (if you forget to link the , you will get missing symbols while importing the library in Python).

Note that the option has no effect on user provided Extension objects that are passed into . It is only used for modules found by file name (as in the example above).

The function in Cython versions up to 0.21 does not recognize the option and it needs to be specified as an option to an that describes your extension and that is then handled by as follows:

The options can also be passed directly from the source file, which is often preferable (and overrides any global option). Starting with version 0.17, Cython also allows to pass external source files into the command this way. Here is a simplified setup.py file:

And in the .pyx source file, write this into the first comment block, before any source code, to compile it in C++ mode and link it statically against the code file:

To compile manually (e.g. using ), the command-line utility can be used to generate a C++ file, and then compile it into a python extension. C++ mode for the command is turned on with the option.

fromdistutils.coreimportsetupfromCython.Buildimportcythonizesetup(ext_modules=cythonize("rect.pyx",# our Cython sourcesources=["Rectangle.cpp"],# additional source file(s)language="c++",# generate C++ code))
fromdistutils.coreimportsetup,ExtensionfromCython.Buildimportcythonizesetup(ext_modules=cythonize(Extension("rect",# the extension namesources=["rect.pyx","Rectangle.cpp"],# the Cython source and# additional C++ source fileslanguage="c++",# generate and compile C++ code)))
fromdistutils.coreimportsetupfromCython.Buildimportcythonizesetup(name="rectangleapp",ext_modules=cythonize('*.pyx'),)
# distutils: language = c++# distutils: sources = Rectangle.cpp

Declaring a C++ class interface¶

The procedure for wrapping a C++ class is quite similar to that for wrapping normal C structs, with a couple of additions. Let’s start here by creating the basic block:

This will make the C++ class def for Rectangle available. Note the namespace declaration. Namespaces are simply used to make the fully qualified name of the object, and can be nested (e.g. ) or even refer to classes (e.g. to declare static members on MyClass).

Declare class with cdef cppclass¶

Now, let’s add the Rectangle class to this extern from block - just copy the class name from Rectangle.h and adjust for Cython syntax, so now it becomes:

cdefexternfrom"Rectangle.h"namespace"shapes":cdefcppclassRectangle:

Add public attributes¶

We now need to declare the attributes and methods for use on Cython:

Note that the constructor is declared as “except +”. If the C++ code or the initial memory allocation raises an exception due to a failure, this will let Cython safely raise an appropriate Python exception instead (see below). Without this declaration, C++ exceptions originating from the constructor will not be handled by Cython.

cdefexternfrom"Rectangle.h"namespace"shapes":cdefcppclassRectangle:Rectangle()except+Rectangle(int,int,int,int)except+intx0,y0,x1,y1intgetArea()voidgetSize(int*width,int*height)voidmove(int,int)

Declare a var with the wrapped C++ class¶

Now, we use cdef to declare a var of the class with the C++ statement:

It’s also possible to declare a stack allocated object, as long as it has a “default” constructor:

Note that, like C++, if the class has only one constructor and it is a nullary one, it’s not necessary to declare it.

rec_ptr=newRectangle(1,2,3,4)try:recArea=rec_ptr.getArea()...finally:delrec_ptr# delete heap allocated object
cdefexternfrom"Foo.h":cdefcppclassFoo:Foo()deffunc():cdefFoofoo...
cdefexternfrom"Rectangle.h"namespace"shapes":

Create Cython wrapper class¶

At this point, we have exposed into our pyx file’s namespace the interface of the C++ Rectangle type. Now, we need to make this accessible from external Python code (which is our whole point).

Common programming practice is to create a Cython extension type which holds a C++ instance as an attribute and create a bunch of forwarding methods. So we can implement the Python extension type as:

And there we have it. From a Python perspective, this extension type will look and feel just like a natively defined Rectangle class. It should be noted that

If you want to give attribute access, you could just implement some properties:

Cython initializes C++ class attributes of a cdef class using the nullary constructor. If the class you’re wrapping does not have a nullary constructor, you must store a pointer to the wrapped class and manually allocate and deallocate it. A convenient and safe place to do so is in the __cinit__ and __dealloc__ methods which are guaranteed to be called exactly once upon creation and deletion of the Python instance.

If you prefer giving the same name to the wrapper as the C++ class, see the section on resolving naming conflicts.

cdefclassPyRectangle:cdefRectanglec_rect# hold a C++ instance which we're wrappingdef__cinit__(self,intx0,inty0,intx1,inty1):self.c_rect=Rectangle(x0,y0,x1,y1)defget_area(self):returnself.c_rect.getArea()defget_size(self):cdefintwidth,heightself.c_rect.getSize(&width,&height)returnwidth,heightdefmove(self,dx,dy):self.c_rect.move(dx,dy)
@propertydefx0(self):returnself.c_rect.x0@x0.setterdefx0(self):def__set__(self,x0):self.c_rect.x0=x0...
cdefclassPyRectangle:cdefRectangle* c_rect# hold a pointer to the C++ instance which we're wrappingdef__cinit__(self,intx0,inty0,intx1,inty1):self.c_rect=newRectangle(x0,y0,x1,y1)def__dealloc__(self):delself.c_rect...

Advanced C++ features¶

We describe here all the C++ features that were not discussed in the above tutorial.

Overloading¶

Overloading is very simple. Just declare the method with different parameters and use any of them:

cdefexternfrom"Foo.h":cdefcppclassFoo:Foo(int)Foo(bool)Foo(int,bool)Foo(int,int)

Overloading operators¶

Cython uses C++ naming for overloading operators:

Note that if one has pointers to C++ objects, dereferencing must be done to avoid doing pointer arithmetic rather than arithmetic on the objects themselves:

cdefexternfrom"foo.h":cdefcppclassFoo:Foo()Foooperator+(Foo)Foooperator-(Foo)intoperator*(Foo)intoperator/(int)cdefFoofoo=newFoo()foo2=foo+foofoo2=foo-foox=foo*foo2x=foo/1
cdefFoo* foo_ptr=newFoo()foo=foo_ptr[0]+foo_ptr[0]x=foo_ptr[0]/2delfoo_ptr

Nested class declarations¶

C++ allows nested class declaration. Class declarations can also be nested in Cython:

Note that the nested class is declared with a but without a .

cdefexternfrom"<vector>"namespace"std":cdefcppclassvector[T]:cppclassiterator:Toperator*()iteratoroperator++()bintoperator==(iterator)bintoperator!=(iterator)vector()voidpush_back(T&)T&operator[](int)T&at(int)iteratorbegin()iteratorend()cdefvector[int].iteratoriter#iter is declared as being of type vector<int>::iterator

C++ operators not compatible with Python syntax¶

Cython try to keep a syntax as close as possible to standard Python. Because of this, certain C++ operators, like the preincrement or the dereferencing operator cannot be used with the same syntax as C++. Cython provides functions replacing these operators in a special module . The functions provided are:

  • for dereferencing. will produce the C++ code
  • for pre-incrementation. will produce the C++ code . Similarly for , and .
  • for the comma operator. will produce the C++ code .

These functions need to be cimported. Of course, one can use a to have shorter and more readable functions. For example: .

For completeness, it’s also worth mentioning which can also be written .

Templates¶

Cython uses a bracket syntax for templating. A simple example for wrapping C++ vector:

Multiple template parameters can be defined as a list, such as or . Optional template parameters can be indicated by writing . In the event that Cython needs to explicitly reference the type of a default template parameter for an incomplete template instantiation, it will write , so if the class provides a typedef for its template parameters it is preferable to use that name here.

Template functions are defined similarly to class templates, with the template parameter list following the function name:

# import dereference and increment operatorsfromcython.operatorcimportdereferenceasderef,preincrementasinccdefexternfrom"<vector>"namespace"std":cdefcppclassvector[T]:cppclassiterator:Toperator*()iteratoroperator++()bintoperator==(iterator)bintoperator!=(iterator)vector()voidpush_back(T&)T&operator[](int)T&at(int)iteratorbegin()iteratorend()cdefvector[int] *v=newvector[int]()cdefintiforiinrange(10):v.push_back(i)cdefvector[int].iteratorit=v.begin()whileit!=v.end():printderef(it)inc(it)delv
cdefexternfrom"<algorithm>"namespace"std":Tmax[T](Ta,Tb)printmax[long](3,4)printmax(1.5,2.5)# simple template argument deduction

Standard library¶

Most of the containers of the C++ Standard Library have been declared in pxd files located in . These containers are: deque, list, map, pair, queue, set, stack, vector.

For example:

The pxd files in also work as good examples on how to declare C++ classes.

Since Cython 0.17, the STL containers coerce from and to the corresponding Python builtin types. The conversion is triggered either by an assignment to a typed variable (including typed function arguments) or by an explicit cast, e.g.:

The following coercions are available:

Python type =>C++ type=> Python type
bytesstd::stringbytes
iterablestd::vectorlist
iterablestd::listlist
iterablestd::setset
iterable (len 2)std::pairtuple (len 2)

All conversions create a new container and copy the data into it. The items in the containers are converted to a corresponding type automatically, which includes recursively converting containers inside of containers, e.g. a C++ vector of maps of strings.

Iteration over stl containers (or indeed any class with and methods returning an object supporting incrementing, dereferencing, and comparison) is supported via the syntax (including in list comprehensions). For example, one can write:

If the loop target variable is unspecified, an assignment from type is used for type inference.

fromlibcpp.vectorcimportvectorcdefvector[int] vectcdefintiforiinrange(10):vect.push_back(i)foriinrange(10):printvect[i]
fromlibcpp.stringcimportstringfromlibcpp.vectorcimportvectorcdefstrings=py_bytes_objectprint(s)cpp_string=<string>py_unicode_object.encode('utf-8')cdefvector[int] vect=xrange(1,10,2)print(vect)# [1, 3, 5, 7, 9]cdefvector[string] cpp_strings=b'ab cd ef gh'.split()print(cpp_strings[1])# b'cd'
cdefvector[int] v=...forvalueinv:f(value)return[x*xforxinvifx%2==0]

Simplified wrapping with default constructor¶

If your extension type instantiates a wrapped C++ class using the default constructor (not passing any arguments), you may be able to simplify the lifecycle handling by tying it directly to the lifetime of the Python wrapper object. Instead of a pointer attribute, you can declare an instance:

Cython will automatically generate code that instantiates the C++ object instance when the Python object is created and deletes it when the Python object is garbage collected.

cdefclassVectorStack:cdefvector[int] vdefpush(self,x):self.v.push_back(x)defpop(self):ifself.v.empty():raiseIndexError()x=self.v.back()self.v.pop_back()returnx

Exceptions¶

Cython cannot throw C++ exceptions, or catch them with a try-except statement, but it is possible to declare a function as potentially raising an C++ exception and converting it into a Python exception. For example,

This will translate try and the C++ error into an appropriate Python exception. The translation is performed according to the following table (the prefix is omitted from the C++ identifiers):

C++Python
(all others)

The message, if any, is preserved. Note that a C++ can denote EOF, but does not carry enough information for Cython to discern that, so watch out with exception masks on IO streams.

This will catch any C++ error and raise a Python MemoryError in its place. (Any Python exception is valid here.)

If something_dangerous raises a C++ exception then raise_py_error will be called, which allows one to do custom C++ to Python error “translations.” If raise_py_error does not actually raise an exception a RuntimeError will be raised.

cdefexternfrom"some_file.h":cdefintfoo()except+
cdefintbar()except+MemoryError
cdefintraise_py_error()cdefintsomething_dangerous()except+raise_py_error

Static member method¶

If the Rectangle class has a static member:

you can declare it using the Python @staticmethod decorator, i.e.:

namespaceshapes{classRectangle{...public:staticvoiddo_something();};}
cdefexternfrom"Rectangle.h"namespace"shapes":cdefcppclassRectangle:...@staticmethodvoiddo_something()

Declaring/Using References¶

Cython supports declaring lvalue references using the standard syntax. Note, however, that it is unnecessary to declare the arguments of extern functions as references (const or otherwise) as it has no impact on the caller’s syntax.

Keyword¶

Though Cython does not have an keyword, Cython local variables not explicitly typed with are deduced from the types of the right hand side of all their assignments (see the compiler directive). This is particularly handy when dealing with functions that return complicated, nested, templated types, e.g.:

(Though of course the syntax is preferred for objects supporting the iteration protocol.)

cdefvector[int] v=...it=v.begin()

RTTI and typeid()¶

Cython has support for the operator.

from cython.operator cimport typeid

The operator returns an object of the type .

If you want to store a type_info value in a C variable, you will need to store it as a pointer rather than a reference:

from libcpp.typeinfo cimport type_info cdef const type_info* info = &typeid(MyClass)

If an invalid type is passed to , it will throw an exception which is converted into a exception in Python.

An additional C++11-only RTTI-related class, , is available in .

Caveats and Limitations¶

Access to C-only functions¶

Whenever generating C++ code, Cython generates declarations of and calls to functions assuming these functions are C++ (ie, not declared as . This is ok if the C functions have C++ entry points, but if they’re C only, you will hit a roadblock. If you have a C++ Cython module needing to make calls to pure-C functions, you will need to write a small C++ shim module which:

  • includes the needed C headers in an extern “C” block
  • contains minimal forwarding functions in C++, each of which calls the respective pure-C function

C++ left-values¶

C++ allows functions returning a reference to be left-values. This is currently not supported in Cython. is also not considered a left-value.

0 thoughts on “Assignment To Non-Lvalue Cython

Leave a Reply

Your email address will not be published. Required fields are marked *