The general state of FFI in Go

can be expressed well with a story about when I had tried to get mattn/go-sqlite3 drivers to work on a Windows machine a couple years ago around version 1.17, and CGO would not build properly because my $GOPATH or $CC was in C:\Program Files\ and it split the path on the whitespace. Although I personally find the use of whitespace in system paths especially (and just windows in general) to be a tragedy, I was shocked that this was a real problem.

Other than dealing with some random cross platform issues that teammates would have before we switched to using the modernc/sqlite3 driver, I didn't have any experience writing CGO. I'll admit that I was maybe a bit ignorant to the details of exactly why it sucked, but the bad user experiences and many anecdotes I'd heard were enough for me to look elsewhere for solutions when it came time to needing to call foreign symbols in dynamically linked libraries from a Go program.

So why does CGO suck?

==============================

There have been many posts and videos that document this far better than I could, probably the most popular being this post from Dave Cheney. A quick summary would be:

  1. For performance, Go using CGO is going to be closer to Python than Go. (which doesn't mean it's not still significantly faster than python)
  2. CGO is slow and often still painful to build cross platform.
  3. Go is no longer able to be built into a single static binary.

This being said, if I can find a solution that solves even 1 of these three things, I would consider that a W.

The program in question:

==============================

Limbo is an open source, modern Rust reimplementation of sqlite/libsql that I have become heavily involved in during the last few months in my spare time.

One of my primary contributions has been developing the extension library, where I am focused on providing (by way of many procedural macros) the most intuitive and simple API possible for users who wish to extend limbo's functionality with things like functions, virtual tables and virtual filesystems in safe Rust (and eventually other languages), without users having to write any of the ugly and confusing unsafe/FFI code that is currently required to extend sqlite3 in any way.

Since limbo is written in Rust and the extensions need to be able to be loaded dynamically at runtime, even though both extension library and core are both in Rust, unfortunately this means that Rust must fall back to adhering to C ABI to call itself, and most of the goodies (traits, smart pointers, etc) and memory safety that Rust gives us by default are all out the window.

However with Rust, aside from that unfortunate situation, once you get used to it, FFI in general is a rather pleasant experience that I can equate to writing a better C with lots of features, or I can imagine something similar to Zig (shot in the dark, I haven't yet written zig).

What this has to do with Go:

==============================

A github issue for limbo was created, asking if a Go driver would be available. The most common theme between almost all the relevant comments was: avoid CGO (including my own, due to my previous interactions with it).

As a Go developer by profession, after a couple weeks of no activity on the issue, I decided that I would take on that task as well.

Besides a jank, half finished, speed-run programming language, I have almost exclusively used Go to write backends and web services, which basically everyone agrees is where it really shines. Most of the systems or general programming outside of web that I have done, has all been in Rust or C/C++ (mostly Rust). I say that to highlight that my knowledge and experience with Go and the ecosystem outside of web was/is minimal, so I was more or less starting from scratch.

All I knew at first was that I didn't want to use CGO, and I had heard of other packages that were supposedly 'Pure Go' implementations. So I naively figured there must be a way to just ignore whatever API the Go team wants you to use, and perhaps there is some 'unsafe' platform specific os package that lets you dlopen that people just avoid for one reason or another.

I started to look at how other drivers managed this, and realized that all of Go's current options for sqlite3 drivers, fall into one of the following categories:

(note: although there are a few more drivers out there that I have not named, they would be still included in one of the below categories).

  1. CGO: github.com/mattn/go-sqlite3

The defacto-standard and the first search result on Google for 'go sqlite driver'.

  1. Code-gen: modernc.org/sqlite

An extremely impressive project that generates Go code for the entire sqlite codebase, as well as some supporting libc code, for many different platforms.

  1. WASM: github.com/ncruces/go-sqlite3

Uses a wasm build of sqlite3 and packages the wazero runtime, as well as a Go vfs implementation wrapping sqlite3's OS interface. Has some minor caveats, but is production ready and well implemented.

  1. Pipes: github.com/cvilsmeier/sqinn-go

"reads requests from stdin, forwards the request to SQLite, and writes a response to stdout."

Why can't I just dlopen()?

When I was unable to find anything that remotely resembles dlopen in the Go standard library (except for windows.LoadLibrary() surprisingly, but more on that in my next post), I recalled a conversation I had with my boss/CTO at work.

She has been writing Go since the first open source release, and that is one of the things we originally bonded over, was the desire to move the company from PHP to Go. I remembered her telling me about a Game engine written in Go that wrote it's own CGO replacement because of how annoyed they were with CGO itself. I was now curious and decided to check it out.

Enter: purego

A library that allows you to dlopen()/dlsym() in Go without CGO. After looking at the API, I quickly realized that this is exactly what I was looking for.

But how did they do it? 

syscall.a1, syscall.a2, _ = syscall_syscall15X(cfn, sysargs[0], sysargs[1], sysargs[2], sysargs[3], sysargs[4],
    sysargs[5], sysargs[6], sysargs[7], sysargs[8], sysargs[9], sysargs[10], sysargs[11],
    sysargs[12], sysargs[13], sysargs[14])
syscall.f1 = syscall.a2 // on amd64 a2 stores the float return. On 32bit platforms floats aren't support

rawdogging syscalls, apparently...

Calling Rust from cursed Go

==============================

Purego makes registering foreign symbols very simple. When the driver is registered, I dlopen() the library and

// Adapted for brevity/demonstration

var (
	libOnce           sync.Once
	limboLib          uintptr
	dbOpen            func(string) uintptr
	dbClose           func(uintptr) uintptr
	connPrepare       func(uintptr, string) uintptr
// ... all the function pointers at global scope
)

// Register all the symbols on library load
func ensureLibLoaded() error {
	libOnce.Do(func() {
         limboLib, err := purego.Dlopen(libPath, purego.RTLD_NOW|purego.RTLD_GLOBAL)
        if err != nil {
            return
        }
                     // functionPointer, uintptr, symbol name string
		purego.RegisterLibFunc(&dbOpen, limboLib, FfiDbOpen)
		purego.RegisterLibFunc(&dbClose, limboLib, FfiDbClose)
		purego.RegisterLibFunc(&connPrepare, limboLib, FfiDbPrepare)
		purego.RegisterLibFunc(&connGetError, limboLib, FfiDbGetError)
        // ...

After playing around with it a bit, and deciphering what my types needed to look like to properly pass values back and forth from Rust, I ended up with something like this:


type valueType int32

const (
	intVal  valueType = 0
	textVal valueType = 1
	blobVal valueType = 2
	realVal valueType = 3
	nullVal valueType = 4
)

// struct to send values over FFI
type limboValue struct {
	Type  valueType
	_     [4]byte // padding
	Value [8]byte
}

type Blob struct {
	Data uintptr
	Len  int64
}

I had to use a byte slice instead of a uintptr if I wanted to represent a union and possibly interpret those bytes as an int64 or float64, as well as a pointer to a TextValue or BlobValue struct that stores the bytes + length

With the accompanying Rust type:


#[repr(C)]
pub enum ValueType {
    Integer = 0,
    Text = 1,
    Blob = 2,
    Real = 3,
    Null = 4,
}

#[repr(C)]
pub struct LimboValue {
    value_type: ValueType,
    value: ValueUnion,
}

#[repr(C)]
union ValueUnion {
    int_val: i64,
    real_val: f64,
    text_ptr: *const c_char,
    blob_ptr: *const c_void,
}

This is how, for example, I convert the slice of driver.Args that in order to implement statement.Query from database/sql/driver.

// convert a Go slice of driver.Value to a slice of limboValue that can be sent over FFI
// for Blob types, we have to pin them so they are not garbage collected before they can be copied
// into a buffer on the Rust side, so we return a function to unpin them that can be deferred after this call
func buildArgs(args []driver.Value) ([]limboValue, func(), error) {
	pinner := new(runtime.Pinner)                 
    // I was unaware that runtime.Pinner was a thing, prior to this
	argSlice := make([]limboValue, len(args))
	for i, v := range args {
		limboVal := limboValue{}
		switch val := v.(type) {
		case nil:
			limboVal.Type = nullVal
		case int64:
			limboVal.Type = intVal
			limboVal.Value = *(*[8]byte)(unsafe.Pointer(&val))
		case float64:
			limboVal.Type = realVal
			limboVal.Value = *(*[8]byte)(unsafe.Pointer(&val))
		case string:
			limboVal.Type = textVal
			cstr := CString(val)
			pinner.Pin(cstr)
			*(*uintptr)(unsafe.Pointer(&limboVal.Value)) = uintptr(unsafe.Pointer(cstr))
		case []byte:
			limboVal.Type = blobVal
			blob := makeBlob(val)
			pinner.Pin(blob)
			*(*uintptr)(unsafe.Pointer(&limboVal.Value)) = uintptr(unsafe.Pointer(blob))
		default:
			return nil, pinner.Unpin, fmt.Errorf("unsupported type: %T", v)
		}
		argSlice[i] = limboVal
	}
	return argSlice, pinner.Unpin, nil
}

// convert a byte slice to a Blob type that can be sent over FFI
func makeBlob(b []byte) *Blob {
	if len(b) == 0 {
		return nil
	}
	return &Blob{
		Data: uintptr(unsafe.Pointer(&b[0])),
		Len:  int64(len(b)),
	}
}

Looking at purego's source code gave me some inspiration and helped me get the general idea of how to manipulate and work with pointers and types received over FFI. For instance, this is the function they use to convert a Go string to a C String:

/*
Credit (Apache2 License) to:
      https://github.com/ebitengine/purego/blob/main/internal/strings/strings.go
*/
func CString(name string) *byte {
	if hasSuffix(name, "\x00") {
		return &(*(*[]byte)(unsafe.Pointer(&name)))[0]
	}
	b := make([]byte, len(name)+1)
	copy(b, name)
	return &b[0]
}

And I was able to adapt everything else around these concepts. There were a few things that I wasn't super pleased with that I still have to figure out. For instance, sending back an array of strings from Rust was such a pain in the ass that rows.Columns() method calls this function:

#[no_mangle]
pub extern "C" fn rows_get_columns(rows_ptr: *mut c_void) -> i32 {
    if rows_ptr.is_null() {
        return -1;
    }
    let rows = LimboRows::from_ptr(rows_ptr);
    rows.stmt.num_columns() as i32
}

to get the number of result columns for the prepared statement, then calls rows_get_column_name with the index of the column name to return.

It's all pretty cursed huh?

================================

But it works, and so far there is a decent start to bindings that don't fit into any of the above categories with any of the existing sqlite drivers :)

I'll follow this post up soon with another explaining some of the caveats, the workarounds for them, whether or not we actually solved any of the issues that we described with CGO, and maybe run some benchmarks.

But for now, thanks for reading.