A Be API FFI - Part 1
The Be/Haiku API is notoriously difficult to use from other languages other than its native C++. The internet is littered with examples of failed and abandoned attempts to wrap it into Python, Rust, Lua, and many other languages. Maybe it’s just impossible. Though if I believed that, I wouldn’t be writing this, so I must have some plan.
This is the first in a series that presents a way to once and for all allow an API library to be written in ANY language that has reasonable foreign function interface (FFI) support. Doing it properly requires some modifications to the internals, but each change is incremental, independently useful to an FFI, and most provide correctness, safety, testing, or other benefits to the Haiku codebase even without considering FFIs. Most of the changes are small, and the larger ones can be done as a series of small incremental changes. There are some tedious parts, but they can mostly be handled by a couple of fairly simple scripts. The changes are also completely source and binary compatible with all existing applications.
The technical inspiration for this primarily came from a few sleepless nights reading the source of the (possibly abandoned?) haiku-rs crate and the Haiku app_server.
Interested? Read on!
Why Is It So Hard?
Why is the Be API so hard to wrap into an FFI? The usual answer is that it is a multithreaded, message passing, C++ based API and that makes it far too hard to deal with or keep up to date.
But why is that hard to deal with, at least for languages that have an FFI system? Let’s take a closer look.
C++ Wrapper Difficulty Tiers
When wrapping a C++ codebase via an FFI system, it can be useful to think about a few difficulty tiers.
- Skipable - These are things that just aren’t worth it, typically because they are obsolete, not that important, or the target language ecosystem provides a native equivalent. The difficult part here is usually political: Who decides what is obsolete, important, or should be left out?
- C, basically - This is code that is basically just C in a .cpp file: Free functions,
constvalues,enum,struct,#define, etc. Any language that has an FFI can interface with C. Some true C++ features can also live here, such as namespaces. Most FFI tools can create a very useable interface with a few hours to a few days work. The main difficulties here are threading models, reentrancy, and maintaining a stable ABI. Some languages have very specific ways that they want to deal with things like threading and callbacks, so it is generally best if the other language handles those things and calls the C functions when it wants. Additionally, if anenumgets reordered or astructgets rearranged things will break. This is particularly an issue for things that are considered internal implementation details but are actually useful or necessary for the FFI. These are all pretty easy details to work with. - Concrete classes - These are classes that are instantiated and their public members used but they are not really meant to be a parent class and they don’t have virtual functions. FFI tools can generally generate a reasonably usable, if slightly awkward, interface with not too much more difficulty than tier 1, though it varies.
- Base classes - These are classes with virtual functions that are generally not meant to be used directly, but inherited from every time you use them. They may be a true ABC or not, but once inheritance hits things get difficult. A few languages have FFIs that can handle this, but it’s awkward in the best case and near impossible in the worst. You can try to do things with virtual table trampolines and similar techniques, but it gets ugly fast.
In addition, things like templates and polymorphism can each add somewhere between 0 and 1 tier each, depending on the target language and FFI tools available.
Be API Difficulty Tiers
The Be API contains important interfaces at every difficulty tier, with most kits having important APIs at tier 2 and 3. This is why FFI projects seem to start rapidly and then stall out. Tier 1 is handled quickly, tier 2 takes a bit longer, tier 3 stalls, and then an enum or const gets a new value and suddenly the library is broken, the developer is discouraged and the project stops.
Let’s take a look at some of the kits and their tiers to see why. This isn’t an exhaustive analysis of every API or class, but a rough estimate of the parts that most applications will want to use.
| Kit | Tier | Notes |
|---|---|---|
| Application | 3 | Rough start! Most of this kit, including BApplication and BLooper are tier 3, though BMessage and a few other things are tier 2. |
| Interface | 3 | Another rough one! Almost everything here is meant to be subclassed, and the Layout API is templated. |
| Kernel | 1 | Oh good, an easy one. Ports, areas, semaphores, etc, are all basic C. |
| Media | 1-3 | For simple media playback this is tier 1 or 2. For all the fancy stuff, this is tier 3 |
| Network | 0-2 | Arguably tier 0 since most languages have a solid networking library and libcurl exists, though technically tier 2 |
| Storage | 1-2 | There is a tier 1 API in the Kernel kit and a separate tier 2 API that wraps the tier 1. Odd, but handy. |
| Support | 0-3 | A grab-bag kit with APIs at every tier. BArchivable and BFlattenable are the important tier 3 classes |
| Translation | 2-3 | A cool kit that should be used more. Often works in the background of other APIs. Tier 2 to use directly, tier 3 to create a new translator. |
And that’s why its so hard. Sure, many kits are tier 1 or 2 so they can be wrapped with a reasonable amount of work, but with so much of the Application and Interface kits at tier 3, whats the point?
Of course, if I really believed that I wouldn’t be writing this. So what gives?
Until next time!