Calling Rust From Python

11 minute read Published:

Introduction

Hello! This is a detailed example of exposing Rust code to other languages (in this case, Python). Most articles I’ve seen that cover this topic uses really trivial example functions, skipping over a lot of the complexity. Even the better ones out there typically don’t have a pre-existing, reasonably complex program to work with. I’m going to start with trivial functions and build my way up to being able to define a scene for my raytracer in Python using a series of calls to Rust, then render it and return the resulting image data back to Python. If you want to know more about the raytracer, I wrote a series of posts on it here, but it won’t be necessary; I’ll explain parts of the raytracer here as we need them. Hopefully this will give a more complete picture of how to incorporate complex Rust code into Python.

I’ve never written any sort of Python/C interop before, so this should be another learning experience all around. I’m going to arbitrarily choose CFFI as the Python interop library. It’s portable across interpreters and seems nicer to use than ctypes. I expect the main concepts will be broadly applicable to other libraries (and other languages such as Ruby). Let get started!

Calling Functions

The first thing to do is to define a Rust function we want to call from Python.

We’re actually defining a function for Rust’s C foreign-function interface. The basic idea here is that we write a wrapper in Python that knows how to call C functions, and a wrapper in Rust that exposes C functions and translates them to regular function calls in Rust. It’s sort of like we’re calling from Python into C into Rust. The no_mangle attribute and extern "C" above instruct rustc not to change the name of the function (otherwise CFFI wouldn’t be able to find it later) and to emit a function that can be called as if it were written in C. We’ll need both for all functions that we want to expose to C.

Now we need to instruct Cargo to build this library as a dynamic library (“dylib” in Cargo terms). I’m writing this on a Windows PC, so Cargo produces a raytracer_ffi.dll file. I tested it on Linux as well and it created libraytracer_ffi.so.

Then we need some Python code to load and call this shared library:

Let’s break this down a bit. First we import the cffi module and create an FFI object. Then we call cdef and pass it some text - this text is a C function signature matching the double function in Rust. CFFI parses this function signature in order to determine how to call the function. We’ll need to do this for all of the functions and structs we want to expose to Python. Then we open the DLL file with dlopen. Finally, we call the double function as if it were a regular Python function and print the result.

And now we should be able to call double from Python:

$ python.exe test.py
18

Side note: I wasn’t able to get this working with PyPy on 64-bit Windows. I didn’t find out why, but I assume it has something to do with how PyPy only provides 32-bit binaries. PyPy worked fine for me on Linux, but I had to use 64-bit CPython on Windows.

Passing Structures

Now, if I’m going to be able to define a scene in Python, I’ll need to be able to call functions and pass in structs as arguments. I’ll keep working with this toy program a bit longer, but instead of simply doubling an integer, let’s try and get it to calculate the length of a vector using vector::Vector3::length.

First, I’ll need to tell rustc that Vector3 should be laid out like a C struct.

It appears that CFFI doesn’t have any way to call functions with stack-allocated structures. Using the stack for small, copyable structures like Vector3 is pretty common in Rust, but I guess it isn’t in C? So instead, our Rust function will have to accept a pointer to a Vector3.

Here we define an extern function which accepts a raw pointer to a Vector3. Dereferencing raw pointers is unsafe, so we use an unsafe block to convert the raw pointer to a Rust reference. Finally, we call length() and return the value.

Back in Python-land, we define a structure type matching Vector3 and the signature of the length function. Now we need to allocate a new vector_t object, which is done with the ffi.new() function. We need to pay attention to ownership here - the memory for the vector_t is allocated by Python and it will have to be freed by Python. In this case, it will be freed when the vector object gets garbage collected so we don’t need to worry about it, but we’ll need to be more careful about ownership later.

$ python.exe test.py
1.73205080757

Returning References Back To Python

Now we’ll start the process of building our actual FFI code. We’ll start with the Scene structure. I don’t especially want to expose all the complexity of the Scene structure to Python, so instead we’ll use another C idiom and return an opaque pointer.

Notice that we use Box::new to heap-allocate the structure, and Box::into_raw to convert it into a raw pointer to return. The corresponding Python code is:

I’m not actually sure void* is the right way to go here, but I don’t know any other way to do opaque pointers in this situation. If you know more about this, let me know. CFFI seems to understand uint32_t all on its own, and presumably will call the Rust function with the appropriate integer width.

$ python.exe raytracer.py
From Rust: Scene { width: 800, height: 600, fov: 45, elements: [],
    lights: [], shadow_bias: 0.0000000000001, max_recursion_depth: 10 }
From Python: <cdata 'void*' 0x000000000155B260>

Sharp readers might have noticed that we’re leaking Scene objects - we’re allocating some memory on the heap for the boxed Scene and never freeing it. For this trivial example, it doesn’t matter much because it will be cleaned up when the process terminates, but it is rather inelegant, so let’s fix that.

Disposing Of Allocated Objects

This goes back to the brief discussion of ownership earlier. Previously, Python owned the allocated Vector3 object, so we could trust that it would be safely freed when it was garbage-collected. Now, we have an object allocated by Rust, but owned by a pointer in Python. Python doesn’t know how to deallocate an object owned by Rust, so we’ll have to return ownership of the pointer to Rust and allow Rust to free the memory.

Freeing the memory is actually quite simple - we use Box::from_raw to convert the raw pointer back into a box, and then just let it fall out of scope. Rust will automatically clean everything up for us.

Right now, there’s nothing to stop us from freeing the scene more than once, or continuing to use that pointer after the scene has been freed. There’s nothing we can do about that from the Rust side, but in Python we can at least build a safe wrapper to work with.

Here, we define a Python class to represent our Scene. It defines the __enter__ and __exit__ methods necessary to act as a Context Manager, which allows us to use it with the with statement at the end. Running this file confirms that the scene object is being freed:

$ python.exe raytracer.py
Freeing the scene

Enums

Before we begin constructing our scene in Python, however, there’s one more bit of complexity to tackle first. Every object in this raytracer contains a Material structure to define what color the surface is, whether it’s reflective or transparent, etc. This is defined in Rust using some enums and a struct:

Rust’s enums have no equivalent in C, and even if they did that DynamicImage type certainly doesn’t. We’ll have to create C-compatible wrappers for these types that we can expose to Python. I’ll focus on the Coloration enum for now, the SurfaceType enum will work the same way.

We’ll start by defining another enum:

I know, I just said we can’t do enums in C. Instead, we’ll define a couple of functions to create CColoration values on the heap and return opaque pointers to them like we did with the Scene.

First, the simple case of a solid color:

Then, the more complex case of a path to a texture file.

Here we take a pointer to a null-terminated character array (a C-style string) and convert it to a Rust string, which has a length and is encoded in UTF-8. This conversion could fail, if the C string isn’t valid UTF-8. Notice that we need to be very careful not to panic. We can’t just unwrap the result of converting the CStr to a regular string, because panicking across FFI boundaries is undefined behavior. Instead, we return a null pointer on all error conditions. A more serious project would probably want to have more robust error handling, but this is sufficient for now.

The corresponding Python should be relatively familiar by now:

The SurfaceType enum works basically the same way as above, so I’ll spare you the details.

All those with’s are kind of ugly, but that’s the price we pay for safety. We can verify that everything is being freed as expected:

$ python.exe raytracer.py
Freeing surface type
Freeing surface type
Freeing surface type
Freeing coloration
Freeing coloration
Freeing coloration
Freeing coloration
Freeing coloration

Constructing the Scene

Finally, we’re ready to start constructing the scene. I’ll focus on the case of adding a Sphere to the scene. The code to define other objects is pretty much the same.

First, we need a new struct to represent Material:

And a function to add a sphere to a scene:

Most of this is the now-familiar C foreign-function boilerplate. The material.to_rust() method works pretty much as you’d expect - it constructs a Material value from a CMaterial value, potentially loading the texture contained in the CColoration. More noteworthy is the way we convert the scene Box back into a raw pointer at the end of the method. This prevents Rust from deallocating our scene.

You might reasonably ask why I chose to have one function that creates and adds the sphere directly to the scene. This does, after all, make it impossible for me to return a Sphere to Python. The answer is that since I don’t really want to manipulate Spheres in Python, there’s not much point in going to all that extra effort. You can go ahead and do that if you like.

Now that we have all of that, we can call it from Python as before:

$ python.exe raytracer.py
Sphere { center: Point { x: 0, y: 0, z: -5 }, radius: 1, material:
  Material { coloration: Texture, albedo: 0.18, surface:
  Reflective { reflectivity: 0.7 } } }

Rendering and Returning the Image To Python

Now that we can define a scene in Python, we need a way to render it and return the resulting image. We can’t just return a byte array, because Python can’t handle stack-allocated objects, and anyway it would overflow the stack. We could return a pointer/length pair, but then we have to pass it back to Rust to free it. Instead, we’ll follow the C convention and have the caller provide a buffer to render the image into.

After the usual boilerplate, we convert the C-style byte array into a mutable slice with the slice::from_raw_parts_mut function, then wrap that into an ImageBuffer and pass it to the raytracer for rendering. Slices in Rust don’t own their contents, so we don’t need to do anything special to prevent Rust from trying to free the buffer.

In Python, we need to save the dimensions of the image so that we can allocate an appropriate buffer. The raytracer uses 4-byte RGBA pixels, so we calculate the buffer size as 4 * width * height, allocate an appropriate buffer, and render the image into it. Then we call ffi.buffer to wrap it into a convenient Python object. Finally, we pass that to the Pillow library to be wrapped into an Image object that we can save out to disk or do further processing on.

Rendered By Python Click to see high-resolution image

Conclusion

Overall, this turned out to be easier than I’d expected. CFFI’s user-friendly interface helped a lot, I think, though the Rust side has a lot of boilerplate. I expect some macros or something could help with that. I’d like to thank Jake Goulding and co. for the Rust FFI Omnibus, which covers all of the basic techniques listed above (and provides examples for a number of other languages, if you’d like to compare).

As usual, if you want to try playing around with the code yourself, you can check out the GitHub Repository. If you do, though, be careful with the complexity of the scene you try to render. It’s very easy to reach multi-hour rendering times when you’re defining scenes programmatically. Otherwise, enjoy!