pthorpe92.dev

The absolute beginners guide to databasemaxxing

Unknown — Sun, 22 Mar 2026 00:00:00 +0000

So, you are interested in getting started learning databases?</h2>

I get a lot of emails from people asking me how they can begin to learn the vast world of databases, and whether they are far enough along on their programming journey to bother trying to start to learn this sub-genre of CS. This post is meant to be my authoritative answer to those questions... Sort of a database specific version of: this</a> post about programming in general.

First, lets answer the question of: Is it the right time to start learning database internals?...</code> or am I far enough along to begin?</code>

I would say that if you want to start learning DBMS internals, you should have completed at least a couple non-trivial projects, and you should be familiar with low level OS concepts, interacting with libc/POSIX API's. I highly recommend finishing an interpreter or compiler project before moving on to databases, because a compiler and a database are much more similar than you might imagine at first... Both are taking some code, parsing it into an AST and transforming it into some output (in fact SQLite uses a bytecode VM internally, very similar to what you would see in an interpreter).

You should also be deeply familiar with using databases and writing SQL/understanding JOIN's and schemas and the relational model first... Don't try to skip this step, as it will make everything that much more difficult later on. Some of this information may already be obvious to you, if you are an advanced user of databases, but will be explained anyway for brevity.

The rest of this post will assume you are roughly this far along, so I will not be defining things like AST</code> because I will assume you are familiar with this at this point.

What if I am already familiar with DBMS internals? </blockquote> Then this post is not for you: it is over-simplified in a way that attempts to make everything as easy to understand as possible for absolute beginners.

Common terminology</h3>
Let's go over some terms that you might not have heard if you have only ever been on the user end of database systems: Not just their definition, but we'll talk about in plain english what people mean when they use these terms. You will hear pretty much all these terms used frequently in the database world.
Remember these are somewhat simplified explanations, for their complete definitions, please refer to CMU Database Group's courses and read the Database Internals book by Alex Petrov.

Databasemaxxing</code>: This is the brain-rot way of saying: "living a Database-centric lifestyle". To dedicate one's entire life to the study and practice of engineering database systems, choosing to live by the relational model. </li>
OLTP</code> or OLTP system</code>: Simply means a transactional database system like Postgres, MySQL or SQLite. These systems are generally focused on short, transactional workloads with frequent writes and reads as quickly as possible while providing all the ACID guarantees we expect from database systems. These systems store values on disk in groups of Rows, much like you would imagine they would if you have ever used one. </li>
OLAP</code> or OLAP system</code>: Conversely, this is an analytical database system like DuckDB, Snowflake or Clickhouse. These systems are focused on performing highly complex queries (think huge multi JOIN graphs with lots of aggregations), and write performance is generally far less of a concern than with OLTP systems: the performance focus is instead shifted to the query planner/optimizer, to better handle extremely large, complex joins over many large tables where the data cannot fit in memory (or might not even be on the same computer/node/data-center). These database systems typically (but not always) store the underlying values by groups of Column instead of by Row... This is (partially) because it makes it easier to compress the data on disk, which is not something typically done by OLTP database systems. </li>
DDL</code>: Data Definition Language / any SQL statements that define the schema, so: CREATE TABLE</code>, CREATE INDEX</code>, DROP TABLE</code> etc. </li>
DML</code>: Data Manipulation Language, so think INSERT</code> / UPDATE</code> / DELETE</code> statements... Anything manipulating existing data in your database system. </li>
DQL</code>: Data Query Language: Obviously this would be your SELECT</code> queries. </li>
Tuple</code>: is just database-speak for a row</code>.. I imagine it's called this because a Row is very similar to the tuple you might know from your favorite programming language: one or more grouped values of various types. </li>
Scan</code> or Full scan</code>: The process of, or the reading of every single row in a table: So think, "the query does a full scan of 't'", means that if 't' has N rows, the query had to read all N rows in order to return the result (and that result could be anywhere from 0..=N rows). </li>
Row ID</code>: An internal identifier for a row/tuple. Many storage engines use an integer row identifier (or something equivalent) which indexes store so the database can locate the row quickly. Some systems will use a declared INTEGER PRIMARY KEY as this ID: this is what's referred to as a "row id alias", because it is both a user facing column value, and also an internal identifier. </li>
ETL</code>: This stands for: Extract, Transform, Load and typically this means a job which pulls data (usually out of an OLTP system), it might do things like create new indexes that weren't needed in the OLTP system but will be necessary or helpful to run analytical queries, and/or otherwise makes the raw data more 'structured'... Then loads it into an OLAP system to have analytical queries ran against it. </li>
Access method</code>: The way in which we are going to retrieve values from a given table... For example: if we have WHERE indexed_column = 123</code>, then our access method would be an index seek on indexed_column</code>... but if our query is SELECT * from t;</code>: our access method would be a table scan, because our query requires us to fetch all the rows. </li>
Seek</code> or Index seek</code>: A fast lookup using an Index, which allows the database system to traverse quickly to the value or tuple in question. (You can think of this as essentially the opposite of a Scan... at least generally: Scan = bad/slow, Seek = good/fast) </li>

Binding</code>: The process of making AST nodes aware of your schema: So lets say we have SELECT a, b FROM t;</code>... This initially parses into something like: </li> </ul>

    Stmt</span>::</span>Select</span>{</span>cols</span>:</span>[</span>Expr</span>::</span>Ident</span>(</span>"a"</span>),</span> Expr</span>::</span>Ident</span>(</span>"b"</span>)]}</span> // ...</span></span></code></pre>
After the binding process, we would end up with something like:</p>
   Stmt</span>::</span>Select</span>{</span></span>
   cols</span>:</span> Expr</span>::</span>Column</span>(</span>ColumnRef</span>{</span>table</span>:</span> TableRef</span>{</span>name</span>:</span>"t"</span>,</span>id</span>:</span> 0</span>},</span> name</span>:</span> "a"</span>,</span> id</span>:</span> 0</span>},</span> </span></span>
         Expr</span>::</span>Column</span>(</span>ColumnRef</span>{</span>table</span>:</span> TableRef</span>{</span>name</span>:</span>"t"</span>,</span>id</span>:</span> 0</span>},</span> name</span>:</span> "b"</span>,</span> id</span>:</span> 1</span>} ) ))</span></span></code></pre>
and we have additional schema information encoded in the AST now, we know that "t" is a real table and it has columns "a" and "b" with internal ID's of 0 and 1 (internal ID's may just be the index of the col in the table)... It's now more than just an Identifier</code> and it's aware of our actual schema. More info on this later.</p>

WAL</code>: Write-Ahead-Log: for several reasons, database systems usually do not write data directly to the main database file(s) during a transaction... Instead, writes are done to a separate, append-only file that contains the new/changed data and metadata about the operation (or in the case of SQLite: the new version of each of the 4kb pages touched). WAL's exist for several reasons:</li>
</ul>

Doing all your writes as append-only keeps data written sequentially, which is much faster than doing scattered writes across 1 or more files on each write transaction.</li>
You now have a log of all transactions/changes, which can be replayed in the event of a crash.. If you write directly to the DB file(s), you will have no idea on startup after your crashed, whether or not your active writes ever finished or were somehow corrupted. Since the WAL is only ever truncated after</strong> the contents are copied to the database file(s), we know that it's contents have 100% safely been transferred over.</li>
</ol>


Snapshot isolation</code>: When transaction t1</code> begins their transaction at time t</code>, either implicitly or explicitly, they should only ever see the state of the database as it exists at time t</code>: Meaning if they are scanning a table which gets inserted into by some transaction t2</code> during the scan, it will not see t2</code>'s rows. You can think of it like freezing the database right at the start of your transaction, each transaction should only ever see that frozen snapshot while that transaction is going on. (Not all databases use snapshot isolation by default, but many modern ones do. There are other levels of transaction isolation, such as "strictly serializable")</p>
</li>

Catalog</code>: You can somewhat interchangeably use this with Schema</code>, as it's generally referring to an internal representation of the database system's in-memory schema. E.g. the Binding phase resolves identifiers to columns/tables using the Schema or Catalog.</p>
</li>

Predicate</code>: This is typically referring to the WHERE</code> clause of a query, essentially meaning the constraints we are putting on our query which will filter the resulting output.
In SELECT * FROM t WHERE a = 7</code>: t.a = 7</code> is our predicate.</p>
</li>

Push-Down</code> or Predicate Push-Down</code>: This is referring to applying a predicate (meaning essentially: filtering the resulting output) at a lower layer...
Now this one is going to seem incredibly obvious, but let's imagine this query:</p>
</li>
</ul>
SELECT</span> *</span></span>
FROM</span> users u</span></span>
JOIN</span> orders o </span>ON</span> u.id</span> =</span> o.user_id</span></span>
WHERE</span> u.last_name</span> =</span> 'Peters'</span>;</span></span></code></pre>
Now theoretically we could:</p>
Read all rows from users</span></span>
Join them with all rows from orders</span></span>
Filter the result where last_name = 'Peters'</span></span></code></pre>
This obviously works but is very dumb, because we are joining a ton of rows that we are just immediately throwing away.</p>
A better plan is:</p>
Use the predicate last_name = 'Peters' while reading users</span></span>
Only keep matching rows</span></span>
Join those rows with orders</span></span></code></pre>
This is called predicate pushdown, because we pushed the filter down closer to the data source.</p>


Checkpointing</code>: The process of updating the actual database files with data from transactions which were written to the Write-Ahead-Log. This is typically done when the WAL reaches a certain size, or can be done manually, usually with a PRAGMA command.</p>
</li>

MVCC</code> (Multi-Version Concurrency Control): A technique used by many databases to allow multiple transactions to run at the same time. Instead of overwriting rows, the database keeps multiple versions so that each transaction can see the correct snapshot.</p>
</li>

Selectivity</code>: How many rows a predicate is expected to match, relative to the total number of rows in the table.</p>
For example, if a table has 1,000,000 rows:</p>
WHERE id = 5</p>
is very selective, because it will probably return 1 row.</p>
WHERE is_active = true</p>
is not very selective, because it might return half the table.</p>
Selectivity is extremely important for the query planner, because it helps decide things like:</p>

Should we use an index or do a scan?</li>
Which table should we read first in a join?</li>
Which join order will be fastest?</li>
</ul>
In general:
High selectivity = few rows = good for index lookups

Low selectivity = many rows = scan might be faster*</p>
</li>
</ul>
The life-cycle of a SQL query:</h3>
What happens under the hood when you execute a query?</p>


Startup: when our database system first starts up, it must load the current Schema|Catalog into memory. All this information/metadata is all just tables reserved for internal use, which describe the state of the database (all the tables, indexes, views, sequences). There is typically either a header on the first page of each file, or a dedicated file, which stores DB specific metadata such as PRAGMA values, settings, ANALYZE information, etc.</p>
</li>

Parser: the string is ran through a SQL parser, which transforms the raw string/text into an AST, which we will use directly in our database system. For each type of ast::Statement</code>, we have some function which takes each statement node variant, and builds a query plan based it after traversing it, and sometimes rewriting parts of it.</p>
</li>

Binding: as described above, the AST is rewritten to include Schema/Catalog aware nodes... In addition to rejecting any queries which might reference invalid/nonexistent tables/columns, because the AST is traversed in this step, some systems might also try to evaluate any constant expressions/optimize in this step as well, e.g. by replacing stuff like (12 * 12)</code> with (144)</code>.</p>
</li>

Query planning: Now that we have an AST which is actually useful to us, we need to figure out how we will execute it.</p>
</li>
</ol>
Example query: SELECT a, b from t WHERE a BETWEEN 1 and 100;</code></p>
Essentially, the way we need to think about executing queries: is we pretty much need to imagine that the underlying table has like a billion+ rows (obviously not always, and often times we may actually know roughly many rows each table has). What I mean by this, is that we always want to avoid doing the dumbest thing... Which in this case would be:</p>
1. Open table `t`</span></span>
2. Go row-by-row, check the value of `a`</span></span>
    If `a >= 1` && `a <= 100`: output the row</span></span>
3. Close table `t`</span></span></code></pre>
That is definitely the most straight forward thing to do... But what if there is an index on a</code>?</p>
Well let's think about what this means, and what is an index, really? We might already know that indexes make lookups faster, but why</em> and how</em>?</p>
First: Do you know why</em> our first example is the "dumb" thing to do?... Well as you probably guessed, it's because a</code> is not going to be in sorted order, so we cannot simply skip directly to the values which are >= 1 and then output rows, stopping when the value of a</code> is > 100, so we have</em> to check all</em> the tuples in the entire table to guarantee that we return all values which satisfy our predicate.</p>
Lets imagine we have an index on users.last_name</code>... We want to return all users whose last_name = 'Peters'. So instead of scanning the entire table, we can do a faster Seek on the index and then use the RowID's from any matching index values... That allows us to then do a Seek on the users</code> table directly because we now have the sorted value (the RowID or PRIMARY KEY). Any time we have the PRIMARY KEY or sometimes called the "RowID alias" in our predicate, this allows us to use a Seek operation instead of a scan.</p>

The Index Seek then becomes our access method</em> for our users</code> table on this query.</p>
*NOTE: Above, in our explanation of Selectivity</em>, we said this:</p>
In general:</span></span>
High selectivity = few rows = good for index lookups  </span></span>
Low selectivity = many rows = scan might be faster*</span></span></code></pre>
How/why could a scan</em> be faster?</strong></p>
Think about it for a minute and see if you can guess...</p>
The reason is because every time you find a matching row while searching your index, you now have to grab the RowID and then do an additional Seek into the underlying table to fetch the actual result column values. This is why not all index lookups are actually faster, if the predicate is determined to have very low selectivity, it often can make more sense to just do a full scan of the table, because you no longer are having to jump around between several B+Trees/data structures to fetch all the needed values.</p>
But let's say we have an index on (a, b)</code> and the query is SELECT a, b FROM t WHERE a > 100;</code>... Now our index contains both</em> the values we need to satisfy the query. This is what's known as a Covering</em> index, and this is an extremely ideal situation in query planning. No extra Row ID Seek is needed to fetch the underlying row, you simply Seek to the index and return the values while they match.</p>
So what if we have a join in our query?</p>
Example:</p>
SELECT</span> *</span></span>
FROM</span> users u</span></span>
JOIN</span> orders o </span>ON</span> u.id</span> =</span> o.user_id</span></span>
WHERE</span> u.last_name</span> =</span> 'Peters'</span>;</span></span></code></pre>
Now things get more complicated, because we are no longer just choosing an access method for one table… we also need to decide:</p>

Which table do we read first?</li>
What access method do we use for each table?</li>
In what order do we perform the joins?</li>
</ol>
This is called selecting the join order</strong> and the access methods</strong>, and this is one of the primary and most difficult jobs of the query planner / optimizer.</p>
We could execute this query in several different ways.</p>
One possible plan:</p>
1. Scan users</span></span>
2. For each user, lookup matching rows in orders</span></span></code></pre>
Another possible plan:</p>
1. Use index on users.last_name to find 'Peters'</span></span>
2. For each matching user, lookup orders by user_id</span></span></code></pre>
Another possible plan:</p>
1. Scan orders</span></span>
2. For each order, lookup user by primary key (RowID)</span></span></code></pre>
All of these are logically correct, but some of them might be 1000x slower</strong> than others depending on how much data is in each table and what indexes exist.</p>
This is why the optimizer must estimate things like:</p>

How many rows are in each table</li>
How selective a predicate is</li>
Whether an index can be used</li>
How expensive each join order would be</li>
</ul>
Even for a simple query with 3 tables, there may already be dozens of possible join orders. Choosing the correct one is one of the hardest parts of building a database system, Inthe general case, optimal join ordering is known to be NP-hard, which is why real databases rely on a mixture of heuristics and cost estimation.</p>

Execution: now that we have chosen a query plan, we actually need to run it.</li>
</ol>
The query planner does not directly run the query... Instead, it produces some kind of internal representation of the plan, which is made up of operators</strong>.</p>
You can think of operators as small building blocks that each do one simple thing, for example:</p>

Scan operator -> reads rows from a table</li>
Index seek operator -> looks up rows using an index</li>
Filter operator -> applies the WHERE predicate</li>
Join operator -> combines rows from two tables</li>
Sort operator -> orders rows</li>
Aggregate operator -> computes things like COUNT / SUM / GROUP BY</li>
</ul>
A query plan is basically a tree of these operators.</p>
For example, a simple query like:</p>
SELECT *
FROM users
WHERE last_name = 'Peters';</p>
might turn into a plan like:</p>
Filter(last_name = 'Peters')</span></span>
    IndexSeek(users_last_name_idx)</span></span></code></pre>
Or for a join:</p>
Join(users.id = orders.user_id)</span></span>
    IndexSeek(users_last_name_idx)</span></span>
    Scan(orders)</span></span></code></pre>
Each operator produces rows, and passes them to the operator above it.</p>
This means execution usually happens row-by-row, not all at once.</p>
The scan reads a row →

the filter checks it →

the join combines it →

the result is returned.</p>
Some databases compile either the AST directly, or the query plan into bytecode, which is stepped through by a virtual machine (SQLite and tursodb). Many execute the plan using an iterator-style execution engine (like Postgres), and some might even generate native code (I believe some OLAP engines do this).</p>
But the idea is generally the same: The planner decides what to do</em> and the execution engine actually does it</em>.</p>
At some point during execution, the database actually needs to read and write data on disk, and this is handled by the storage engine</strong>. Most relational databases store tables and indexes using B-Trees (usually B+Trees, but it varies), which organize rows into fixed-size blocks called pages</strong> (often 4KB–16KB each). The database never reads a single row directly from disk, it always reads at least one entire page at a time.</p>
A component, often called the Pager</em> or Buffer Pool manager</em> (names vary by database), is responsible for loading these pages from disk, writing them back when they change, and then keeping track of which pages are currently in memory. Since disk access is extremely</em> slow compared to RAM, the database system keeps a page cache</strong>, which stores these recently used pages in memory. A huge part of database performance is just managing this cache correctly: deciding which pages to keep, which to evict, and when they need to be flushed back to disk.</p>
How rows are actually stored on disk</h3>
Earlier we mentioned that tables and indexes are stored using B-Trees, and that the database reads and writes fixed-size pages instead of individual rows. Let's go a little deeper into what that actually means.</p>
Each table is usually stored as a B+Tree, where each node of the tree is one page on disk. The pages at the bottom of the tree (called leaf pages) contain the actual rows, while the pages above them contain keys that help the database find the correct leaf page quickly. In fact if we SELECT</code> some metadata about tables in SQLite: we can see that for each table in the Catalog, we store the "root page" of the table: </p>
Inside a page, rows are not just stored one after another in an array type of situation... The page has a small header and a list of pointers (often called slots or cell pointers), and then the actual row data somewhere else in the page. This makes it possible to insert and delete rows without having to rewrite the entire page every time.</p>
When the database wants to read rows from a table, it uses something usually called a cursor</strong>. A cursor is basically an object that remembers where we currently are in a given B-Tree. It knows which page we are on, which slot inside the page we are looking at, and how to move to the next or previous row.</p>
To find a specific row, the database starts at the root of the B-Tree, compares keys, follows pointers down through the internal pages, and eventually reaches the correct leaf page. Once it is on the leaf page, the cursor can move from row to row without having to search the tree again.</p>
Indexes work the same way (at least B+Tree indexes do... There are other forms of indexes, primarily hash table indexes but we will leave that out for now), except as we stated earlier, instead of storing the full row they usually store the indexed value alongside a reference to the row in the table (often the RowID or primary key).</p>
Sometimes a row is too large to fit entirely inside one page. In that case, the database stores part of the row on the page and the rest in one or more overflow pages</strong>. The main page contains a pointer to the overflow pages, and the storage engine follows those pointers to read the full value. This is commonly used for large text or blob values.</p>
Writing rows works by finding the correct leaf page using the B-Tree, modifying the page in memory, and then marking it 'dirty' in the page cache. The pager later can write the updated page back to disk (usually through the WAL first, if the database is using one).</p>
All of this might sound complicated (and I assure you, it is..), but at least the important idea is fairly simple:</p>
Rows live on pages, the pages live in B-Trees and we use cursors to walk the tree to find the rows... So above, (in the simplest possible terms) when we say that we need to fetch rows from the underlying table, what we mean is we create a new BTree Cursor, and our pager will know from the root page, which page (and the page can just be an offset into a database file) to fetch from the pager, and then we follow pointers from there to possibly fetch other pages and then eventually read the tuples on those pages.</p>

===============================</p>
I hope this is helpful to someone out there, I tried to compile a list of things I wish I would have known at the very beginning of my journey. Maybe I will do a part 2 at some point. If you have any suggestions for improvements or comments, feel free to reach out on X</a> or by email</a></p>



You can do better
Unknown — Fri, 13 Mar 2026 00:00:00 +0000
Lately on X</h2>
I have noticed a trend of posts that I imagine are probably the natural response to 'LinkedIn-brain' posts, which might be the result of the poor job market right now:</p>
"If hard work paid off, the donkey would own the farm"</code></p>
And if you were to come straight from the slop filled cringe fest that is LinkedIn these days, I can totally understand why you might have this attitude... But I can't help but find myself wanting to tweet about this, before realizing that I don't pay for more than 160 characters. This is why I have decided to write a quick blog post about it.</p>
The argument for working hard</h2>
Instead of saying some generic motivational jargon, I'll try to give you a few examples and explain specifically what I mean.</p>
Let's say you work on a team</strong>, and your job generally consists of working on tickets that are assigned to you, and you spend some time every day pushing PR's that get reviewed by others and your time is allotted by them: You notice that your CI runs take ~25-30 mins each, and this is mildly annoying but definitely something every developer notices, but nothing anyone does anything about.</p>
When you look at the DAG, you notice that there is no cacheing of build artifacts or dependencies and all your jobs are running serially.</p>
You could spend the ~1hr or so setting up sccache and matrix runners so everything runs in parallel and cuts your CI/CD build and deployment pipeline down to ~3-4 mins, or you could just mind your business and only do what is assigned to you... Maybe someone will think of that, and make a Jira ticket for you to pick up one day.</p>
But if you go out of your way to fix this on your own, every single person on the team, without a doubt, will notice that their build times got cut by 80% and their dev experience improved.</p>
Now you repeat this behavior over the course of a couple years, and it's time for some layoffs at your company. Seems a lot less likely they are going to choose you, if you spent your time making objectively beneficial decisions for the team, instead of only focusing on what you were tasked with doing.</p>
Same idea when you come up for a promotion or a raise... EVERYONE notices this, and choosing to go above and beyond makes you a better teammate, who others will want to work with.</p>
Take initiative on your own. Often times we developers are the 'boots on the ground' and have insight into things like cloud cost savings, potential productivity increases and workflow improvements that PM's and admin won't. In a world where we LLM's are becoming what seems like a threat to this kind of work, don't be ashamed of separating yourself from others by taking initiative and providing value to your team and your company.</p>
I'm not saying you have to work 9-9-6 and have no social life, I'm just saying that it's rewarding to take personal responsibility and ownership over the products you ship and the things you work on. There is absolutely nothing wrong with working hard and taking pride in what you do.</p>
Thank you for your attention to this matter.</p>

P</li>
</ul>


Rewriting SQLite: Adding a new join algorithm
Unknown — Sat, 22 Nov 2025 00:00:00 +0000
I am not sure that there exists a group of bigger SQLite nerds than those of us at Turso.</h1>
We use SQLite for everything</em>.. Including for OLAP workloads where we should be using duckdb</code>, or for services which we would love</em> to be totally stateless pods,
instead we have statefulsets and PVC's with EBS volumes containing SQLite .db files. Every one of us has probably read just about the entire 265k LOC amalgamation
sqlite3.c source file more than once all the way through.</p>
So if anyone has an insight into some of the pain points that will arise from using SQLite in production, surely we are at least good candidates.</p>
But if you ask most Database nerds, what the fundamental limitations are of SQLite, I would guess that almost all will mention at least one of the following:</p>


Exclusive write lock.</p>
The architecture and design around how SQLite manages transaction isolation, as well as their journaling design (which we will cover a bit in #2 as well),
means that the writer</code> lock is exclusive and no two connections can obtain a write transaction concurrently. (This is something that tursodb</code> is attempting to fix
with MVCC mode.)</p>
</li>

Write performance.</p>
At least in WAL mode, SQLite appends all (usually 4kb) pages that are touched as part of a write transaction, referred to as WAL frames, to the log... This includes
any and all pages which are shuffled around due to B-Tree balancing, so in theory an INSERT of just a single byte could cause many 4kb pages to be written to the log.
Also, when checkpointing (moving these frames from the log to the DB file), SQLite does not use any cached pages it might already have in-memory, it reads each page from disk
during the backfilling process and performs N 4kb reads and N 4kb writes where N is the total number of frames needing backfilling. This can cause substantial write
amplification and is known to be problematic for write-heavy workloads. For applications which are even moderately write-heavy, which are not gracefully shutting down periodically, typically you have to setup some kind of a background process which monitors the WAL size and periodically forcing a blocking TRUNCATE checkpoint due to the possibility of the WAL growing unbounded.</p>
</li>

Checksum/integrity verification.</p>
See this blog post</a> for more details</p>
tl;dr SQLite can 'silently' allow corruption and the user will not find out till much later on.</p>
</li>

Read performance for any OLAP style queries.</p>
SQLite’s query engine is extremely good for what it is but its physical join toolbox is intentionally small. By design, SQLite relies on nested loop joins as the core join strategy.
That choice keeps the planner simple and predictable, but it comes with some hard ceilings once you push into any analytic-style workloads.
This is what we will be talking about today in this blog post.</p>
</li>
</ol>
Join Algorithms: A Brief Tour</h2>
Most databases implement a small family of join algorithms:</p>


Nested Loop Join (NLJ):


The simplest: iterate over table A and, for each row, search for matches in table B.</p>
</li>

Merge Join:


Requires both inputs to be ordered on the join key. Walk them together like merging two sorted lists.</p>
</li>

Hash Join:


Build an in-memory hash table of one input keyed by the join column(s), then probe it with rows from the other input.</p>
</li>
</ul>
There are lots of refinements (index nested loops, block nested loops, hybrid/grace hash joins, etc.), but almost everything is a variant of these.</p>
SQLite fundamentally lives in the NLJ world, making some optimizations.</p>
How Nested Loop Joins Work (and how SQLite makes them fast)</h3>
The nested loop join really is as simple as it sounds. The outer loop iterates over one table, and for each row, the inner loop searches the other for matches.</p>
SQLite does add several layers of optimization on top of this:</p>


Using an index on the join column to avoid full scans of table B
(so the inner loop becomes an index lookup/Seek rather than a linear scan).</p>
</li>

Reordering tables to minimize work</p>
</li>

Automatic “ephemeral” indexes when no suitable index exists.
SQLite may build a temporary B-tree (we see OpenAutoindex</code> in the plan) to avoid repeated full scans on the inner table.</p>
</li>
</ul>
In SQLite’s VDBE bytecode you can see the basic NLJ shape:</p>
OpenRead cursor for employees</span></span>
</span>
OpenRead cursor for departments</span></span>
</span>
For each row in employees (Rewind/Next loop)</span></span>
</span>
Seek into departments (using persistent or ephemeral index)</span></span>
</span>
Check predicate</span></span>
</span>
Emit result</span></span></code></pre>
Oversimplified pseudocode that includes an ephemeral index looks soemthing like:</p>
tempIndex</span> =</span> new</span> BTreeIndex</span>()</span></span>
for</span> rowA</span> in</span> tableA</span>:</span></span>
    tempIndex</span>.</span>insert</span>(</span>rowA</span>.</span>key</span>,</span> rowA</span>.</span>RowID</span>)</span></span>
</span>
for</span> rowB</span> in</span> tableB</span>:</span></span>
    rowID</span> =</span> tempIndex</span>.</span>get</span>(</span>rowB</span>.</span>key</span>)</span></span>
    if</span> rowID</span> is not</span> None</span>:</span></span>
        rowA</span> =</span> tableA</span>.</span>seek</span>(</span>rowID</span>)</span></span>
        output</span>(</span>rowA</span>,</span> rowB</span>)</span></span></code></pre>
This works extremely well for small datasets or OLTP-style queries where joins are selective and tables are modest in size.
(NOTE: SQLite will sometimes also build a bloom filter as a further optimization, so this can be probed to determine whether or not a Seek
should even be done on that join key in the inner loop, but because tursodb</code> does not yet implement this optimization we will pretty much omit this for now)</p>
It also works surprisingly</em> well for a lot of medium-sized workloads, which is why SQLite has done so well with only NLJ.</p>
How can we improve on these optimizations.</h2>
For our first improvement, let's consider the following:</p>
SELECT</span> *</span></span>
FROM</span> large_table_1 </span>AS</span> a</span></span>
JOIN</span> large_table_2 </span>AS</span> b</span></span>
ON</span> substr(a.category, </span>1</span>, </span>3</span>) </span>=</span> substr(b.category, </span>1</span>, </span>3</span>);</span></span></code></pre>
Logically this is an equality join, but it’s on expressions rather than raw columns. If there is no index specifically on substr(b.category, 1, 3)</code>,</p>
SQLite's query planner has a much harder time constructing an efficient ephemeral index here, so it likely will be a full naive nested loop with scans.</p>
If both tables have million+ rows, it can very quickly turn into tens of billions of comparisons and page touches, even if the page cache saves from reading everything from disk each time.</p>
So overall, ideally we would like to improve on this in two different places:</p>


Use a more performant data structure, to improve on the O(log n) insert/seeks for each outer table row.</p>
</li>

Improve the query planner to properly use non 'plain column' expressions as part of our equijoin predicate,
and try our best not to fall back to nested scans.</p>
</li>
</ol>
Behold the hash join:</h1>
With a similar plan as using an ephemeral index, we can instead use a hash table to get near constant time lookups, at the cost of some additional memory and CPU cycles for hashing.</p>
The general idea is to:</p>


Choose one table as the build input (preferably the smaller one).</p>
</li>

Make a single pass over it, building an in-memory hash table keyed by the join column(s).</p>
</li>

Then scan the larger probe input, and for each row, do a ~constant-time hash lookup.</p>
</li>
</ul>
If we can choose a hash function that will perform well and avoid collisions as much as possible, we can get fast hashing speed and near O(1) lookups when probing. We can either store the
materialized values directly in the hash table, or for more complex/fallback scenarios we can store the rowID of the underlying row, along with the join key (for that case, will cost an additional SeekRowID when a probe matches) and we are able to emit or use the result columns.</p>
The crucial differences vs an ephemeral B-tree:</p>
lookups are ~O(1) instead of O(log N), inserts should be cheaper because they don't require MakeRecord</code> opcode to build the index record with the join key,
which in the case of the ephemeral index needs to be saved to compare directly to the probe key, along with the underlying rowid. Also for the common case where
we cache the row's payload, we are also saving the additional BTree seek required to fetch the result columns from the row itself, we can store them during the build
phase and get the purely constant time lookup.</p>
the hash table is structurally more amenable to optimizations:</p>

partitioning</li>
spilling individual partitions to disk(grace/hybrid hash join)</li>
eventually things like parallelizing build/probe, vectorizing probes, etc.</li>
</ul>
We think we could improve on the default SQLite features by adding hash joins to tursodb</code>.</p>
Implementing Hash joins in tursodb</h1>
Because SQLite (and so of course, tursodb) is implemented as essentially a virtual machine with a storage engine attached, and query plans are just emitted bytecode, adding a new join
algorithm will need to consist of creating new Opcodes which we emit at plan time and will be stepped through at runtime and executed by the VDBE (Virtual database engine).</p>
So we are going to need Opcodes to:</p>


HashBuild - feed a build-side row into the hash table.</p>
</li>

HashBuildFinalize - finalize the build, potentially spilling partitions.</p>
</li>

HashProbe - hash and probe a key, yielding the build-side rowid(s).</p>
</li>

HashNext - iterate through additional matches in the same bucket.</p>
</li>

HashClose - free the hash table.</p>
</li>
</ul>
Instead of simply using a different data structure for the join, we will also make an attempt to improve the optimizer/query planner.</p>
Remember the example query:</p>
select</span> users.first_name, products.id</span> from</span> users </span></span>
join</span> products </span>ON</span> substr(users.first_name,</span>1</span>,</span>2</span>) </span>=</span> substr(products.name, </span>1</span>,</span>2</span>);</span></span></code></pre>
Because neither side of the binary equality expression is a plain Column, and in this case both are expressions, the ast looks something like:


BinaryExpression{lhs: Expr::Function, operator: Operator::Eq, rhs: Expr::Function}</code>

</p>


We know SQLite actually will not build an ephemeral index for this query, instead it will fall back to the most naive nested loop join with a full O(n * m)
scan. We will improve on this by walking the expressions and tracking the indices into them for each expression that matches the `build` and `probe` tables,
this way we can determine that for the following:


pub struct</span> HashJoinOp</span> {</span></span>
    /// Index of the build table in the join order</span></span>
    pub</span> build_table_idx</span>:</span> usize</span>,</span></span>
    /// Index of the probe table in the join order</span></span>
    pub</span> probe_table_idx</span>:</span> usize</span>,</span></span>
    /// Join key references, each entry points to an equality condition in the [WhereTerm]</span></span>
    /// and indicates which side of the equality belongs to the build table.</span></span>
    pub</span> join_keys</span>:</span> Vec</span><</span>HashJoinKey</span>></span>,</span></span>
    /// Memory budget for hash table (used for cost based optimizations)</span></span>
    pub</span> mem_budget</span>:</span> usize</span>,</span></span>
}</span></span>
</span>
pub struct</span> HashJoinKey</span> {</span></span>
    /// Index into the predicate array</span></span>
    pub</span> where_clause_idx</span>:</span> usize</span>,</span></span>
    /// Which side of the binary equality expression belongs to the build table.</span></span>
    /// The other side implicitly belongs to the probe table, because we will only</span></span>
    /// ever select equijoin conditions for hash joins.</span></span>
    pub</span> build_side</span>:</span> BinaryExprSide</span>,</span></span>
}</span></span></code></pre>
The expression will look something like this:</p>
ast</span>::</span>BinaryExpression</span>{</span> </span></span>
    lhs</span>:</span> Expr</span>::</span>Function</span>{</span>func</span>:</span> Func</span>::</span>Substr</span>,</span> args</span>:</span> [</span>ast</span>::</span>Column</span>{</span>users</span>.</span>first_name</span>,</span> ..</span>},</span> ..</span>],</span> </span></span>
    operator</span>:</span> Operator</span>::</span>Eq</span>,</span> </span></span>
    rhs</span>:</span> Expr</span>::</span>Function</span>{</span>func</span>:</span> Func</span>::</span>Substr</span>,</span> args</span>:</span> [</span>ast</span>::</span>Column</span>{</span>products</span>.</span>name</span>}</span>..</span>}] }</span></span>
}</span></span></code></pre>
We would walk the ast::Expr, and track that build_side</code> has BinaryExprSide::Left</code> since we are going to use users</code> as the build table, then given the
predicate we will be able to translate the expression itself and load the value from the output register directly into the hash table, so the resulting value is hashed
and the probe side expression can be compared against it.</p>
Otherwise, we would simply be calling emit_column_or_rowid</code> directly, and instead tracking the column index.. Here we are tracking the expr</code> index and calling translate_expr</code>
into the resulting register.</p>
-- simplified</span></span>
addr  opcode             p1    p2    p3  p4        p5  comment</span></span>
----  -----------------  ----  ----  --  --------- --  -------</span></span>
1</span>     OpenRead           </span>0     2     0             0</span>   table</span>=</span>users, </span>root</span>=</span>2</span>, iDb</span>=</span>0</span></span>
2</span>     OpenRead           </span>1     3     0             0</span>   table</span>=</span>products, </span>root</span>=</span>3</span>, iDb</span>=</span>0</span></span>
3</span>     Rewind</span>             0     8     0             0</span>   Rewind table</span> users</span></span>
4</span>       Column           </span>0     1     5             0</span>   r[5]</span>=</span>users.first_name</span></span>
5</span>       Function</span>         0     5     4</span>   substr    </span>0</span>   r[4]</span>=</span>func(r[5..7]) </span>-- evaluate the function with the col argument -> HashBuild</span></span>
6</span>       HashBuild        </span>0     4     1             0</span>   </span></span>
7</span>     Next</span>               0     4     0             0</span>   </span></span>
8</span>     HashBuildFinalize  </span>3     0     0             0</span>   </span></span>
9</span>     Rewind</span>             1     20    0             0</span>   Rewind table</span> products</span></span>
10</span>      Column           </span>1     1     9             0</span>   r[9]</span>=</span>products.name</span></span>
11</span>      Function</span>         0     9     8</span>   substr    </span>0</span>   r[8]</span>=</span>func(r[9..11])</span></span>
12</span>      HashProbe        </span>3     8     1</span>   r[12]</span>=</span>19  0</span>   </span></span>
13</span>      SeekRowid        </span>0     12    19            0</span>   if</span> (r[12]</span>!=</span>cursor</span> 0</span> for table</span> users.rowid) </span>goto</span> 19</span></span>
14</span>      Column           </span>0     1     1             0</span>   r[1]</span>=</span>users.first_name</span></span>
15</span>      RowId            </span>1     2     0             0</span>   r[2]</span>=</span>products.rowid</span></span>
16</span>      ResultRow        </span>1     2     0             0</span>   output</span>=</span>r[1..2]</span></span>
17</span>      HashNext         </span>3     12    19            0</span>   </span></span>
18</span>      Goto</span>             0     13    0             0</span>   </span></span>
19</span>    Next</span>               1     10    0             0</span>   </span></span>
20</span>    HashClose          </span>3     0     0             0</span>   </span></span>
21</span>    Halt               </span>0     0     0             0</span>   </span></span></code></pre>
The fun part is of course inside HashBuild / HashBuildFinalize / HashProbe: the hash table implementation and how we make it safe for large joins.</p>
Which hash function should we use?</h2>
After asking on X</code>, I received some great suggestions from DB veterans, such as XXHash</code> (@_Felipe</a>), and GxHash</code> or rapidhash</code> (@AlexMillerDB</a>)
and went down the smhasher</a> rabbit hole, which is pretty much a one stop shop for comparing hash functions.</p>
After a couple of naive benchmarks, I went with rapidhash</code> for the MVP. In the future we may choose different
hash functions depending on the key type and size, but “one good, fast hash” gets us very far.</p>
We also fold SQLite-style collations into the hash:</p>
For COLLATE NOCASE text, we hash the to_lowercase() representation.</p>
For COLLATE RTRIM, we trim trailing whitespace before hashing.</p>
For BINARY we hash the raw bytes.</p>
That way equality and hashing line up with the comparison rules the rest of the engine uses.</p>
Spillin’: making hash joins behave under memory pressure (the hard part).</h2>
Hash tables are obviously very fast but they can also be incredibly memory-hungry. In a traditional RDBMS, the hash table lives inside the same buffer manager
that already knows how to spill to disk and re-read pages. Because SQLite doesn't include the hash table in their toolbox, for tursodb, the hash join is an extra
piece of machinery, outside of the pager; so it would be very easy to accidentally blow through RAM on a big join if everything is naively kept in memory.</p>
pub struct</span> HashTable</span> {</span></span>
    /// The hash buckets (used when not spilled).</span></span>
    buckets</span>:</span> Vec</span><</span>HashBucket</span>></span>,</span></span>
    /// Number of entries in the table.</span></span>
    num_entries</span>:</span> usize</span>,</span></span>
    /// Current memory usage in bytes.</span></span>
    mem_used</span>:</span> usize</span>,</span></span>
    /// Memory budget in bytes.</span></span>
    mem_budget</span>:</span> usize</span>,</span></span>
    /// Number of join keys.</span></span>
    num_keys</span>:</span> usize</span>,</span></span>
    /// Collation sequences for each join key.</span></span>
    collations</span>:</span> Vec</span><</span>CollationSeq</span>></span>,</span></span>
    /// Current state of the hash table.</span></span>
    state</span>:</span> HashTableState</span>,</span></span>
    /// IO object for disk operations.</span></span>
    io</span>:</span> Arc</span><</span>dyn</span> IO</span>></span>,</span></span>
    /// Current probe position bucket index.</span></span>
    probe_bucket_idx</span>:</span> usize</span>,</span></span>
    /// Current probe entry index within bucket.</span></span>
    probe_entry_idx</span>:</span> usize</span>,</span></span>
    /// Cached hash of current probe keys to avoid recomputing</span></span>
    current_probe_hash</span>:</span> Option</span><</span>u64</span>></span>,</span></span>
    /// Current probe key values being searched.</span></span>
    current_probe_keys</span>:</span> Option</span><</span>Vec</span><</span>Value</span>>></span>,</span></span>
    spill_state</span>:</span> Option</span><</span>SpillState</span>></span>,</span></span>
    /// Index of current spilled partition being probed</span></span>
    current_spill_partition_idx</span>:</span> usize</span>,</span></span>
}</span></span>
pub struct</span> HashEntry</span> {</span></span>
    pub</span> hash</span>:</span> u64</span>,</span></span>
    pub</span> key_values</span>:</span> Vec</span><</span>Value</span>></span>,</span></span>
    pub</span> rowid</span>:</span> i64</span>,</span></span>
    /// For most cases, to avoid an additional SeekRowID/btree seek on a match, we can</span></span>
    /// pre-cache the needed result columns in the hash table directly.</span></span>
    pub</span> payload_values</span>:</span> Vec</span><</span>Value</span>></span>,</span></span>
}</span></span>
#</span>[</span>derive</span>(</span>Debug</span>,</span> Clone</span>)]</span></span>
pub struct</span> HashBucket</span> {</span></span>
    entries</span>:</span> Vec</span><</span>HashEntry</span>></span>,</span></span>
    // ...</span></span>
}</span></span></code></pre>
The hash table tracks:</p>
mem_used</code>: approximate bytes for all resident entries in memory.</p>
mem_budget</code>: a soft per-join limit (eventually will be configurable, but will start with 64 MiB by default in release builds).</p>
state</code>: Building, Probing, Spilled, Closed.</p>
During the build phase:</p>
We insert entries and increment mem_used</code> by entry.size_bytes().</p>
If mem_used</code> + entry_size</code> would exceed mem_budget, we:</p>


Initialize a SpillState (if this is the first overflow).</p>
</li>

Redistribute existing buckets into NUM_PARTITIONS in-memory partition buffers using the hash value.</p>
</li>

Mark the table as Spilled.</p>
</li>

Pick one or more partition buffers to spill entirely to disk, freeing their memory.</p>
</li>
</ul>
The spill path uses a very simple on-disk layout:</p>
Each partition has one or more SpillChunk{file_offset, size_bytes, num_entries}</code>.</p>
Each entry is serialized as:</p>
[hash:8][rowid:8][num_keys:varint][key1_type:1][key1_len:varint][key1_data]...</span></span></code></pre>
We append a length-prefixed entry blob for each HashEntry</code> in that partition.</p>
After writing a partition:</p>


That partition’s buffer is cleared.</p>
</li>

mem_used</code> is reduced by the buffer’s mem_used</code>.</p>
</li>

We remember the chunk metadata inside SpilledPartition</code>.</p>
</li>
</ul>
At the end of the build phase, finalize_build does one more pass:</p>
For partitions that have already spilled, any remaining entries are serialized and appended as new chunks.</p>
For partitions that never spilled, we keep them purely in memory:</p>

We consume their PartitionBuffer, build buckets, and mark them as PartitionState::InMemory.

These are always resident and only counted in mem_used (no disk chunks, no LRU).</li>
</ul>
</li>
</ul>
At that point we transition to Probing and the build side is “frozen”.</p>
Hybrid Grace hash join and partition reloading.</h3>
This algorithm behaves like a hybrid grace hash join, in that we do not hash/partition both the build + probe tables.
We attempt to do the whole thing in memory, but if we do spill, than we begin to partition the table and spill only
the necessary partitions to disk.</p>
Any partitions that we did</em> have the memory budget for during build time are kept memory resident throughout
the entire join and only partitions which did</em> spill are cycled to and from disk and tracked via LRU. We treat the mem_limit</code> for the table as a
soft limit and simply try to avoid both drastically exceeding this limit, and thrashing of parititons to/from disk.</p>

Overview</strong></p>
The hash function also determinus a partition index:</p>
fn</span> partition_from_hash</span>(</span>hash</span>:</span> u64</span>)</span> -></span> usize</span> {</span></span>
    ((</span>hash</span> >></span> (</span>64</span> -</span> PARTITION_BITS</span>))</span> as usize</span>)</span> &</span> (</span>NUM_PARTITIONS</span> -</span> 1</span>)</span></span>
}</span></span></code></pre>
For partitions that never spilled (PartitionState::InMemory):</strong></p>
There’s nothing special to do; they live in RAM for the lifetime of the join.</em></p>
For partitions that did spill:</strong></p>
We only load them on demand when the probe hits that partition.</em></p>
Loading is done via a re-entrant load_spilled_partition(partition_idx)</code>:</p>
If the partition is OnDisk, we:</p>


Schedule asynchronous reads for each SpillChunk (we use our own I/O model with hand-rolled state machines and event loop, no async/await).</p>
</li>

Accumulate all bytes into a read_buffer.</p>
</li>

Once all chunks are read, we parse them into HashEntrys and build buckets.</p>
</li>

We estimate the resident memory of those buckets and update accounting.</p>
</li>
</ul>
If the partition is already Loaded, we simply mark it as recently-used in an LRU.</p>
If reads are still in flight, we return an IOResult::IO</code> and the VM yields.</p>
Keeping probe-time memory under control (and avoiding thrash)</h3>
pub fn</span> probe_partition</span>(</span></span>
    &</span>mut</span> self</span>,</span></span>
    partition_idx</span>:</span> usize</span>,</span></span>
    probe_keys</span>: &</span>[</span>Value</span>],</span></span>
)</span> -></span> Option</span><</span>&</span>HashEntry</span>></span> {</span></span>
    let</span> hash</span> =</span> hash_join_key</span>(</span>...</span>);</span></span>
    let</span> partition</span> =</span> spill_state</span>.</span>find_partition</span>(</span>partition_idx</span>)</span>?</span>;</span></span>
    if</span> !</span>partition</span>.</span>is_loaded</span>() {</span> return</span> None</span>; }</span></span>
</span>
    let</span> bucket_idx</span> =</span> (</span>hash</span> as usize</span>)</span> %</span> partition</span>.</span>buckets</span>.</span>len</span>();</span></span>
    let</span> bucket</span> = &</span>partition</span>.</span>buckets</span>[</span>bucket_idx</span>];</span></span>
    // scan bucket entries, compare keys, emit matches</span></span>
}</span></span></code></pre>
If we stopped here, probe-time behavior could still be ugly. Imagine the probe side alternates between two “cold” partitions:</p>
We would repeatedly load partition A, then evict it and load B, then evict it and load A again, etc.</p>
To keep things sane, the hash table maintains a small, in-memory LRU cache of loaded spilled partitions:</p>
loaded_partitions_lru</em>: VecDeque<usize></code> partition indices in LRU order.</p>
loaded_partitions_mem</em>: usize</code> approximate memory used by the loaded spilled partitions’ buckets.</p>
Important distinctions:</p>
mem_used</code> tracks the base in-memory footprint:</p>
The original non-spilled buckets.</p>
Any never-spilled partitions that were materialized in memory.</p>
loaded_partitions_mem</code> tracks only the additional cost of bringing spilled partitions back into RAM.</p>
When we load or parse a spilled partition, we estimate:</p>
fn</span> partition_bucket_mem</span>(</span>buckets</span>: &</span>[</span>HashBucket</span>])</span> -></span> usize</span> {</span></span>
    buckets</span>.</span>iter</span>()</span>.</span>map</span>(</span>|</span>b</span>|</span> b</span>.</span>size_bytes</span>())</span>.</span>sum</span>()</span></span>
}</span></span></code></pre>
And before committing to keeping it resident, we call:</p>
evict_partitions_to_fit</span>(</span>resident_mem</span>,</span> protect_idx</span>)</span></span></code></pre>
which:</p>


While mem_used + loaded_partitions_mem + incoming_mem > mem_budget:</p>
</li>

Picks the least-recently-used partition that is Loaded, and has backing spill chunks (!chunks.is_empty()).</p>
</li>

Clears its buckets, resets it to OnDisk, and subtracts its resident_mem from loaded_partitions_mem.</p>
</li>
</ul>
Then we record the new resident partition:</p>
record_partition_resident</span>(</span>partition_idx</span>,</span> resident_mem</span>);</span></span></code></pre>
which swaps in the new resident_mem value for that partition + adjusts loaded_partitions_mem</code> and moves the partition to the back of the LRU as most-recently-used.</p>
The effect in practice:</strong></p>
The hash join respects mem_budget</code> during build (by spilling whole partitions).</p>
During probe, it keeps at most a small working set of spilled partitions resident at once.</p>
Typical workloads where only a few partitions are hot see greatly reduced thrash: once a partition is in the LRU, repeated probes for that partition reuse the same buckets.</p>
Pathological workloads where the probe touches more distinct partitions than fit in memory still work; they just spill/reload more often, but at least not entirely arbitrarily/uncontrolled.
Eventually if necessary, we can improve on this algorithm further down the road, but when we have more metadata avialable and we are tracking things like
total size of each table, that will allow us to make better choices in the query planner, as even choosing the proper table for the build</code> side makes a tremendous
difference in performance, as we could avoid spilling to disk entirely in many situations.</p>
When does the planner pick a hash join?</h3>
Right now, tursodb is conservative:</p>
We only consider hash joins for queries where SQLite would otherwise build an ephemeral index on the inner side of a nested loop.</p>
We avoid hash joins when a good persistent index already exists and can be used directly, using a real index is always</em> cheaper than building a hash table.</p>
The planner will restrict choosing hashjoins only for INNER equijoins.</p>
We must identify which side should be the build side (this is tough at the moment because we don't track metadata about the size of tables).</p>
Then we generate bytecode that materializes the build input by creating a loop of Rewind/HashBuild/Next</code> for the build table, with any predicates needed from the
WHERE clause which might limit the amount of necessary rows needed to hash. (e.g. SELECT t.a, t2.b FROM t JOIN t2 ON t2.c = t.c WHERE t.a > 100</code>, we only have
to hash the join keys for rows in t where a > 100), only if the build table is the only table reference in that particular predicate so we can mark it consumed.</p>
Final outcome:</h1>
At the risk of this blog post getting far too long, let's look at the best case scenario with our improvements we have made:</p>
Query:</h2>
SELECT</span> users.first_name, products.id</span> FROM</span> users </span></span>
JOIN</span> products </span>ON</span> substr(users.first_name,</span>1</span>,</span>2</span>) </span>=</span> substr(products.name, </span>1</span>,</span>2</span>);</span></span></code></pre>
No indexes

15k rows per table</p>
Times:

SQLite: 22.772 seconds</code>  -- nested loop 

tursodb: 2.334 seconds</code>  -- hash join</p>
Because SQLite doesn't create an ephemeral btree index here, we are able to get 10x speedup on this query (and we would likely have been able to get similar speedup by just
improving the planner and choosing to still build btree index with two expressions).</p>
And a much more fair comparison:</strong></p>
Query:</h2>
select</span> users.first_name, products.name</span> from</span> users </span></span>
join</span> products </span>on</span> products.name</span> =</span> users.first_name;</span></span></code></pre>
No indexes 

15k rows per table</p>
Times:
SQLite: 20ms</code>  -- ephemeral index 

tursodb: 15ms</code> -- hash join</p>
You can check out the PR here</a></p>
Thanks for reading! If you are interested or a fellow database/systems nerd, stop by the repo</a> or the discord, and join our open source community.</p>


Checkpointing performance in Write-Ahead-Logs
Unknown — Wed, 13 Aug 2025 00:00:00 +0000
Making Checkpointing fast</h3>
Since I began working for turso officially in May of this year (relevant post</a>), I had been spending most of my time learning the inner workings of the cloud platform, and familiarizing myself with the other codebases and libraries. When I was told a few weeks ago that I could begin spending 80% of my time working on the database itself, I was more than thrilled, as for the past 10 months, this was the project I would spend time on for fun, to give myself a break from work.</p>
One of my first tasks was performance, specifically around our async I/O implementation (io_uring</code>). Benchmarks and flamegraphs showed that under write-heavy workloads performance would degrade sharply after enough writes. Digging in, this wasn’t just an I/O problem, our Write-Ahead Log (WAL) needed some dire attention. The early WAL implementation was sufficient for bootstrapping correctness, but it was now time for a full revamp.</p>
One of several design choices that turso</code> made, is to use the WAL journal mode exclusively. Until MVCC support is finished, this will remain our primary method of transaction isolation, as well as for the usual benefit of journaling, providing durability.</p>
A checkpoint</code> is the process of moving committed frames from the log back into the main database file. There are different 'modes' of checkpointing, a few of which are: Restart</code> and Truncate</code>, which both restart the log over from the beginning (with truncate</code> then going on to physically truncate the WAL file afterwards), and Passive</code>, which is the default mode that will be triggered periodically by default during the pager's cache flush method.</p>
There are a number of locks that are shared between all connections, which must be held either exclusively by a writer or a checkpointer, or by readers to signify that there is an active read transaction going on, and what the maximum frame is for that read transaction.</p>
The checkpoint operation, sort of like a read transaction, also has it's own min and maximum frames that it needs to backfill</code> (backfilling is the process of writing pages from the log to the database). All the current readers are evaluated and depending on the checkpoint's Mode, either the checkpoint's max_frame is decreased, or a Busy error is returned, to prevent moving frames which are not in the reader's snapshot to the database file, where the reader might see them before they should.</p>
The (simplified) checkpoint flow</h2>
Trigger:</strong></p>
Either:
a. At the end of a write transaction, during cacheflush process the log will check if the amount of frames in the cache is greater than a specific threshold (typically 1000 frames), and auto-trigger PASSIVE checkpoint.


b. An explicit PRAGMA wal_checkpoint(mode);</code> is executed.</p>
Process outline:</strong></p>

Locking:</strong> Acquire the checkpoint locks needed for the chosen mode (e.g., checkpoint_lock</code> and read-mark 0</code> for all modes, and modes that require “all frames” also hold the exclusive writer lock).</li>
Safe bound:</strong> Compute the checkpoint's max_frame by examining active readers, capping it to the lowest pinned frame among them. The previous checkpoint’s max is stored as nBackfills</code>, which denotes the current checkpoint's min_frame.</li>
Worklist:</strong> Build the list of (page_id → latest_frame ≤ max_frame)</code> candidates to backfill.</li>
Copy:</strong> For each selected frame, issue a read from the WAL and write the page into the main DB file at its offset.</li>
Finalize:</strong> If all</strong> possible frames were backfilled, fsync</code> and truncate the DB as needed, and depending on mode, restart/truncate the WAL; advance nBackfill</code>. Release locks and return counts.</li>
</ol>
So what are the performance issues here?</p>
Copied frames one at a time, with strictly serialized reads and writes.</p>
Issued writes in no particular page order (whatever order they came from the frame cache in).</p>
Never looked at the in-memory page cache, so even already-hot pages were re-read from the WAL every time.</p>
Symptoms:</strong></strong></p>
Lots</strong> of syscalls, poor locality.</p>
Average checkpoint time was roughly 8-13 ms on a representative test on my machine.</p>
CPU stalled on I/O and cache misses, even for warm pages.</p>
Correct, but not ideal.</p>
One of the most exciting features we are introducing in tursodb</code> not found in SQLite is Async I/O (via io_uring</code>), This allows us to do things concurrently, so the previous setup of having all writes fully serialized awaiting the reads, is not exactly ideal here.</p>
I had recently introduced using vectored I/O for any contiguous pages that are written to the DB file, so we could read in batches of pages, and then submit single pwritev</code> syscalls with a provided array of buffers, for any groups of page ID's that were sequential. This helped increase performance, but the serialization of the reads was still a bottleneck.</p>
Old flow:</strong></p>
  Start</span></span>
  ├─ acquire proper locks</span></span>
  ├─ compute mxSafeFrame</span></span>
  ├─ build (page_id, latest_frame≤mxSafeFrame) list</span></span>
  └─ → ReadPage</span></span>
</span>
ReadPage</span></span>
  ├─ Issue `pread` for current page</span></span>
  └─ → WaitReadPage</span></span>
</span>
WaitReadPage</span></span>
  ├─ Check if current page still locked (awaiting I/O)</span></span>
  └─ Locked? if pages =< batch_len -> FlushBatch : return I/O</span></span>
</span>
FlushBatch</span></span>
  ├─ Collect buffers from current batch of read pages</span></span>
  ├─ Issue a `pwritev` for each contiguous run.</span></span>
  ├─ Store marker to track completion of run.</span></span>
  └─ → WaitFlushBatch</span></span>
</span>
WaitFlushBatch</span></span>
  ├─ If Batch complete:</span></span>
  ├─── More reads ? ReadPage : Done</span></span>
  └─ Else? return I/O</span></span>
Done</span></span>
  ├─ fsync db file</span></span>
  ├─ advance nBackfill and increment checkpoint_seq</span></span>
  ├─ (Restart/Truncate modes) if all frames backfilled, reset the log</span></span>
  └─ release locks, return counts</span></span>
</span></code></pre>
New proposed flow</strong>:</p>
  Start</span></span>
  ├─ acquire proper locks</span></span>
  ├─ compute mxSafeFrame</span></span>
  ├─ build (page_id, latest_frame≤mxSafeFrame) list</span></span>
  └─ → Processing</span></span>
</span>
Processing</span></span>
  ├─ drain completed reads into a WriteBatch</span></span>
  ├─ flush WriteBatch when “good enough” (based on run-length heuristics)</span></span>
  ├─ issue more reads until limits are hit</span></span>
  ├─ loop until nothing is left</span></span>
  └─ → Done</span></span>
</span>
Done</span></span>
  ├─ fsync db file</span></span>
  ├─ advance nBackfill and increment checkpoint_seq</span></span>
  ├─ (Restart/Truncate modes) if all frames backfilled, reset the log</span></span>
  └─ release locks, return counts</span></span></code></pre>
How can we trust that the cached pages are the exact frames we need? SQLite reads each page from the log during checkpoint, so why diverge from this?</p>
We encode an 'epoch' and store it on the page object in memory, a packed combination of the frame_id</code>, and the checkpoint_seq</code>, which is an integer stored on the WAL header that is incremented at each successful checkpoint. So in addition to ensuring that the page is not dirty, we make sure it matches the exact frame and generation. To be even more secure, when debug assertions are enabled, we still read the page from the log and we assert that it has the same contents of any page we would have accepted from the cache.</p>
This results in excellent percentage of cache hits for write benchmarks or any write heavy loads. It is likely to be less during any read heavy workloads, as we won't have all the relevant frames in memory due to being evicted to make room for the read pages, but this still a tremendous win compared to a guaranteed 2 syscalls per-page.</p>
So Why doesn't SQLite do this?</strong></strong></p>
SQLite tends to be conservative in terms of performance optimizations, and is more likely to keep things simple. Also, since SQLite allows multi-process connections, there are many instances where the page cache will either be empty, or not shared between connections, making this optimization not nearly as useful.</p>
New Benchmarks?:</strong></strong>
(Average time spent checkpointing during write benchmark)</p>
Before: 7-12ms</code></p>
After: 2-5ms</code></p>


Learn systems, not langauges
Unknown — Sat, 09 Aug 2025 00:00:00 +0000
Due to the nature of my story and the attention that it's received, as mentioned in my last post</a>, I frequently get emails from developers, college kids, or other people with troubled pasts that are looking for advice on either how to learn or how to get into tech. This blog post will be a compilation of things I have typed out probably a hundred times now, which I hope might help anyone who seeks such advice, and which I can point to that effectively sums up my insight and learnings.</p>
I went from not having written any code in 15+ years, to programming for ~16 hours a day for about three and a half years now. One of the things I am most thankful for, is my past experience programming, while it did not leave me with much remaining skill or experience, it did at least get me in the right direction, and I at least knew what</em> I should be spending my time on.</p>
At the start, I was only able to get the python interpreter and use the bundled 'IDLE' Tkinter python IDE (on a heavily locked-down Windows 10 laptop with 8GB of RAM btw), and there was no internet except at the Recreation building which we had limited hours in, only on weekdays. So I had to save wikipedia articles and copy + paste code from github and stack overflow into .txt files, because there was definitely no access to git at that time. Eventually started writing C and Go and building my own library of data structures before I got my first job a couple years ago. Everything was so fascinating to me, and I sat and absorbed every piece of tech culture, history and lore that I could find. A lot of things were coming back to me, and perhaps the nostalgia is what had so much of an impact on me, is that it reminded me of a time before I had thrown my life away, and things were much more simple.</p>
Roughly twenty thousand hours of programming later, I now spend my work days hacking on database internals.</p>
The first thing I would like to mention about this, is that there are zero short-cuts, and no substitute for all those hours spent. The best thing you can hope for is that the hours you spend will be as rewarding and fruitful as possible. You could easily</em> spend the same amount of time misguided and unaware of what to learn, and end up squandering a significant portion of it.</p>

It's called Tutorial hell because Tutorials are the devil.</p>
</blockquote>
There are great resources out there, like Harvard's CS50 or I'm sure some of the code academy lessons are great, and https://boot.dev</code> (shout-out to Lane, is a really good guy and although I never took any courses myself, I had recommended my 'protege' to sign up and use it to study and learn in between our weekly meetings and it really does a great job of teaching a lot of fundamentals in a structured and gamified way) is pretty great.</p>
But the youtube tutorials</code> that claim you can learn X in Y minutes, or they have some project you follow along with, is a complete and utter waste of your time.</p>
My biggest piece of advice can be boiled down to this:</p>

Learn systems</strong>, not languages or frameworks.</p>
</blockquote>
One thing that confuses most new (and not so new) learners, is mistaking a language or framework itself, with the important aspect of 'building software'.</p>
This applies especially for the demographic that I seem to hear from most frequently. People who, like myself, came from (usually back-end) Web, who maybe learned some Rust or Zig and now want to look for Rust jobs and are having a hard time. This is not uncommon because these languages have some considerable hype around them and are generally challenging and an exciting thing to learn.</p>
The lesson here, is that most</strong> of companies hiring for Rust jobs are not looking for 'rust developers', they are usually looking for devs with a strong systems background. Usually the language will end up being in C++ or C, but that is almost completely irrelevant here. What they want is someone who deeply understands POSIX/OS API's, filesystem and networking I/O and protocols, memory, atomics and threads, etc, etc.</p>
Learn the languages as a side effect</em> of practicing and studying building software. Learn and experience for yourself what problems Rust solves before you spend time learning it. You cannot truly appreciate the ownership/borrowing model unless you have dealt with the frustration of keeping track of which poorly documented C libraries mutate or free</code> resources, or had to scan all your code to try to determine what the lifetime</code> of your own objects are after you have passed their pointers to libraries you wrote a while ago.</p>
It's not any different if your goals are just to get into tech in general.. It may be tempting to spend all your time learning NextJS, React and tailwindcss, because those seem to be what companies are looking for. But the truth is that now that LLM's are quite good at generating this type of code, none of those things are actually</strong> valuable to a company by themselves in the long run, if you don't deeply understand the architecture of HTTP servers and web technology, how scalable web apps are built and shipped.</p>
If you are looking to get into web, make sure you are gaining a fundamental understanding of the HTTP protocol, basic networking, containers and the docker/OCI ecosystem, databases, message queue's, learn SQL even if you always use an ORM. Learn (and experience for yourself) what problems React solves by building a reactive UI with just vanilla JS and the DOM API.</p>
Prefer Go to Django, Laravel or NextJS, use net/http</code> to build your own</em> web framework and understand how they work. Abstractions are great and come in very handy when you are just trying to ship things and you don't want to deal with building a session table and your own middleware to be able to call request.UserID()</code> or request.IsAdmin()/IsAuthenticated()</code> in your handlers, but if you don't understand how</em> that framework knows who sent that request, and how they are logged in, then getting good at that framework is not really providing you any real long term value, I would argue that you are doing yourself a disservice by using it.</p>

Ditch the file-explorer. Embrace the CLI.</p>
</blockquote>
I'm not saying that everyone needs to be a vim user, but I highly</strong> recommend it. You will</strong> thank me in a couple months if you spend the time to learn at least the bindings. Working out of the CLI makes you a better programmer, I don't care what anyone says. You may know some ultra cracked dev who clicks around file explorers and some light-mode IDE and knows or uses zero keybindings for anything... but that is the exception, not the rule. Computers take some input -> process it -> produce some output, and CLI tools help you think in this paradigm. I'm not saying you are wrong for using a GUI for everything, I'm saying learning to be good at the CLI is a major career hack if you want to be a programmer.</p>

Install and configure a minimal installation of Arch Linux.</p>
</blockquote>
You want to be deeply familiar with your system, write and edit your config files yourself. If you need a cronjob setup or to write a SystemD service file, it should be second nature.</p>
Write shell scripts to automate even normal things. I have hand written shell scripts, some of which are years old, that do stupid things like change my desktop wallpaper, update discord, prune my docker cache, build and deploy my blog, removed unused files from my ~/Downloads directory, initialize key re-mapping on startup, I have a slack bot that listens to a channel and plays the sound of police sirens when we have an incident that needs acknowledgement. These are all small but valuable things that create a better understanding of how all the software you interact with operates.</p>

Build. Build. Build.</p>
</blockquote>
There is no substitute for building software if you want to get good at building software.</p>
The following is a list of projects that I recommend. These are geared towards anyone interested in systems, but imho there should not be a distinction. Learn computer science if you want to be a good programmer.</p>
NOTE: Most of these are not easy or beginner projects, these are for after you have learned to program and you are comfortably writing web back-ends.</p>
All of these are really preferable to be written in either C, C++, Rust, or Zig. But C is preferred if you haven't written any of these languages.</p>
hey, didn't he just say 'learn systems, not languages.. </span></span>
then proceeds to tell you specifically which language to use?</span></span></code></pre>
touche</em>.. HN commentor... touche</p>


Data structures:</p>
Implement your own library for common things like a string type, dynamic array and hashmap, all from scratch. It's preferred that this is used in future projects.</p>
</li>

coreutils:</p>
cat/less, ls, mv/cp, and find/grep. For grep, you don't have to implement a whole regex engine, you can just do glob</code> or have fun designing a more simple engine with custom rules. Otherwise you can make them as simple or as feature complete as you want</p>
</li>

shell:</p>
Lots of fun, and a good warm up for an interpreter or compiler. Learn about environments, subprocesses, signals, traps, inheriting env vars, string interpolation and variable expansion. Learn why cd</code> is the only shell command that works when you yeet your $PATH</code>, and how fork</code> and exec</code> work, what even is POSIX and why do people care when they complain that your favorite shell isn't compliant? The general point here is to be as familiar as possible with libc and POSIX/linux api's and system calls.</p>
</li>

http1 server:</p>
Use raw linux sockets and bind/accept/send/recv syscalls, the HTTP parsing doesn't have to be completely up to spec of course, but it should handle basic requests. You can handle each connection on it's own thread, but it is really preferable to learn epoll</code> and your accept</code> should be non-blocking. Bonus points for connection pooling with an LRU cache.</p>
</li>

Interpreter/Compiler:</p>
This is a lot of fun, can be any language. writing an interpreter in Go</code> by Thorsten Ball is a good book (I haven't actually read it but I have heard really good things and I have skimmed some of it, plus the author is a gigachad 11x programmer). Demystify what is happening when you write and run code, you don't have to write an optimizing x86-64 back-end (I sure as hell didn't) to get a ton</em> of value from this project. If this seems very advanced, maybe start by writing the super tiny compiler</a> in your language of choice, or at the very least learn to write recursive descent parsers. Designing your own bytecode and VM is particularly fun, and its super valuable to learn about labels and jump tables and how loops and branches work at the assembly or bytecode level.</p>
</li>

Memory allocator (C):</p>
Write a library that mmap's a large chunk of memory on initialization and learn about how to manage the book-keeping and fragmentation, and what trade-offs are made by different kinds of allocators. Ginger Bill has a fantastic series on allocators found here</a> that I highly recommend.</p>
</li>

Emulator:</p>
The Chip8 is a great first emulator project. The 8080, and 6502 are much more advanced, but all are very valuable projects to understand particularly CPU internals, memory addressing and probably bit manipulation more than anything.</p>
</li>
</ul>
Some others are a Cache, a basic TUI Text Editor (bummer I have never got around to this one), Stackful coroutine based Async I/O library (this was a lot of fun in Rust, basically implementing goroutines, backed by io_uring).</p>
This post is getting long and out of hand, maybe I'll refactor this out into a separate page of just the project recommendations and gather up some more useful links. But thanks for reading and I hope this is able to help someone.</p>


whois pthorpe92.dev
Unknown — Thu, 19 Jun 2025 00:00:00 +0000
Preston Thorpe</h1>

  
    
  </figure>
</div>
senior engineer @ Turso</a></p>

building the modern evolution of sqlite</a></p>
</blockquote>
formerly:</p>
principal engineer @ Unlocked Labs</a></p>
open source:</h3>
maintainer @ eza</a></p>
maintainer @ UnlockEdv2</a></p>
=====================================================================</p>
Currently incarcerated at MVCF in Charleston, ME. (story</a>)


cli, linux, and rust nerd.


open source enthusiast


systems/databases/cloud/devops</p>
I love Rust, Go, CLI/TUI/all things tty</code>, databases, cloud/infra, building developer tools,
systems programming, the Open source community, and I enjoy teaching. I'm a long time linux user
(I've had a ubuntu tattoo since '09), some of my favorite tools are DWM/awesomewm, Neovim, Tmux, Ghostty and Alacritty.</p>
I am an advocate for FOSS, privacy, security, criminal justice/prison reform,
but most of all for hard work and discipline. I enjoy working on anything challenging, and I consider myself extremely fortunate that
I have a career that doubles as a hobby.</p>
I believe that I'm a living example of why education is the solution to mass incarceration. Access to
education and technology while incarcerated changed my life completely and I want to be a part of helping give
more people the opportunity I had, to study and devote time to building the skills necessary to have
a meaningful life and career when they re-enter society.</p>
These are some developers I look up to, or projects that have inspired me that I feel are worth sharing: (all unrelated to me)</p>
https://github.com/awesomekling</a></p>
https://github.com/apavlo</a></p>
https://github.com/dtolnay</a></p>
https://github.com/andrewrk</a></p>
https://github.com/artempyanykh</a></p>
https://github.com/tjdevries</a></p>
https://github.com/fasterthanlime</a></p>
https://github.com/tokio-rs/tokio</a></p>
https://github.com/LadyBirdBrowser/ladybird</a></p>
https://github.com/ziglang/zig</a></p>
https://github.com/neovim/neovim</a></p>
https://github.com/ratatui-org/ratatui</a></p>
https://github.com/uutils/coreutils</a></p>
https://github.com/mrnugget</a></p>
https://github.com/orhun</a></p>
https://github.com/penberg</a></p>
https://github.com/sunfishcode</a></p>
https://github.com/mitchellh</a></p>


Working on databases from prison, How I got here, part 2.
Unknown — Mon, 16 Jun 2025 00:00:00 +0000
How I got here pt. 2</h2>
Posted on turso's blog</a></p>


Calling Rust from cursed Go
Unknown — Wed, 26 Feb 2025 00:00:00 +0000
The general state of FFI in Go</h1>
can be expressed well with a story about when I had tried to get mattn/go-sqlite3</code> drivers
to work on a Windows machine a couple years ago around version 1.17</code>, and CGO would not build properly because my $GOPATH
or $CC was in C:\Program Files\</code> and it split the path on the whitespace</a>.
Although I personally find the use of whitespace in system paths especially (and just windows in general) to be a tragedy, I was shocked that this was a real problem.</p>
Other than dealing with some random cross platform issues that teammates would have before we switched
to using the modernc/sqlite3</code> driver, I didn't have any experience writing</em> CGO. I'll admit that
I was maybe a bit ignorant to the details of exactly why</em> it sucked, but the bad user experiences and
many anecdotes I'd heard were enough for me to look elsewhere for solutions when it came time to needing to
call foreign symbols in dynamically linked libraries from a Go program.</p>
So why does CGO suck?</h3>
==============================</p>
There have been many posts and videos that document this far better than I could, probably the most popular being this</a> post from Dave Cheney. A quick summary would be:</p>

For performance, Go using CGO is going to be closer to Python than Go. (which doesn't mean it's not still significantly faster than python)</li>
CGO is slow and often still painful to build cross platform.</li>
Go is no longer able to be built into a single static binary.</li>
</ol>
This being said, if I can find a solution that solves even 1 of these three things, I would consider that a W.</p>
The program in question:</h3>
==============================</p>
Limbo</a> is an open source, modern Rust reimplementation of sqlite/libsql</a> that I have become heavily involved in during the last few months in my spare time.</p>
One of my primary contributions has been developing the extension library, where I am focused on providing (by way of many procedural macros)
the most intuitive and simple API possible for users who wish to extend limbo's functionality with things like functions,
virtual tables and virtual filesystems in safe Rust (and eventually other languages), without users having to write any of the ugly and confusing
unsafe/FFI code that is currently required to extend sqlite3 in any way.</p>
Since limbo</code> is written in Rust and the extensions need to be able to be loaded dynamically at runtime, even though both extension library and core
are both in Rust, unfortunately this means that Rust must fall back to adhering to C</code> ABI to call itself, and most of the goodies (traits,
smart pointers, etc) and memory safety that Rust gives us by default are all out the window.</p>
However with Rust, aside from that unfortunate situation, once you get used to it, FFI</code> in general is a rather pleasant experience that I can
equate to writing a better C with lots of features, or I can imagine something similar to Zig (shot in the dark, I haven't yet written zig).</p>
What this has to do with Go:</h3>
==============================</p>
A github issue</a> for limbo was created, asking if a Go driver would be available. The most common theme
between almost all the relevant comments was: avoid CGO</code> (including my own, due to my previous interactions with it).</p>
As a Go developer by profession, after a couple weeks of no activity on the issue, I decided that I would take on that task as well.</p>
Besides a jank, half finished, speed-run programming language</a>, I have almost exclusively used Go to write backends and web services,
which basically everyone agrees is where it really shines. Most of the systems or general programming outside of web that I have done, has all been in
Rust or C/C++ (mostly Rust). I say that to highlight that my knowledge and experience with Go and the ecosystem outside of web was/is minimal, so I was more
or less starting from scratch.</p>
All I knew at first was that I didn't want to use CGO, and I had heard of other packages that were supposedly 'Pure Go' implementations.
So I naively figured there must be a way to just ignore whatever API the Go team wants you to use, and perhaps there is some 'unsafe' platform
specific os package that lets you dlopen</code> that people just avoid for one reason or another.</p>
I started to look at how other drivers managed this, and realized that all of Go's current options for sqlite3 drivers, fall into one of the
following categories:</p>
(note: although there are a few more drivers out there that I have not named, they would be still included in one of the below categories).</p>

CGO:</strong> github.com/mattn/go-sqlite3</code></li>
</ol>

The defacto-standard and the first search result on Google for 'go sqlite driver'.</p>
</blockquote>

Code-gen:</strong> modernc.org/sqlite</code></li>
</ol>

An extremely impressive project that generates Go code for the entire sqlite codebase, as well as some supporting libc code,
for many different platforms.</p>
</blockquote>

WASM:</strong> github.com/ncruces/go-sqlite3</code></li>
</ol>

Uses a wasm build of sqlite3 and packages the wazero</a> runtime, as well as a Go vfs implementation wrapping
sqlite3's OS interface. Has some minor caveats, but is production ready and well implemented.</p>
</blockquote>

Pipes:</strong> github.com/cvilsmeier/sqinn-go</code></li>
</ol>

"reads requests from stdin, forwards the request to SQLite, and writes a response to stdout."</p>
</blockquote>
Why can't I just dlopen()</code>?</em></h3>
When I was unable to find anything that remotely resembles dlopen</code> in the Go standard library (except for windows.LoadLibrary()</code> surprisingly,
but more on that in my next post), I recalled a conversation I had with my boss/CTO at work.</p>
She has been writing Go since the first open source release, and that is one of the things we originally bonded over,
was the desire to move the company from PHP to Go. I remembered her telling me about a Game engine written in Go that wrote it's own CGO
replacement because of how annoyed they were with CGO itself. I was now curious and decided to check it out.</p>
Enter: purego</a></p>
A library that allows you to dlopen()</code>/dlsym()</code> in Go without CGO. After looking at the API, I quickly realized that this is exactly
what I was looking for.</p>
But how did they do it?</em>
</p>
syscall</span>.</span>a1</span>,</span> syscall</span>.</span>a2</span>,</span> _</span> =</span> syscall_syscall15X</span>(</span>cfn</span>,</span> sysargs</span>[</span>0</span>],</span> sysargs</span>[</span>1</span>],</span> sysargs</span>[</span>2</span>],</span> sysargs</span>[</span>3</span>],</span> sysargs</span>[</span>4</span>],</span></span>
    sysargs</span>[</span>5</span>],</span> sysargs</span>[</span>6</span>],</span> sysargs</span>[</span>7</span>],</span> sysargs</span>[</span>8</span>],</span> sysargs</span>[</span>9</span>],</span> sysargs</span>[</span>10</span>],</span> sysargs</span>[</span>11</span>],</span></span>
    sysargs</span>[</span>12</span>],</span> sysargs</span>[</span>13</span>],</span> sysargs</span>[</span>14</span>])</span></span>
syscall</span>.</span>f1</span> =</span> syscall</span>.</span>a2</span> // on amd64 a2 stores the float return. On 32bit platforms floats aren't support</span></span></code></pre>
rawdogging syscalls, apparently...</p>
Calling Rust from cursed Go</h2>
==============================</p>
Purego makes registering foreign symbols very simple. When the driver is registered, I dlopen()</code> the library and</p>
// Adapted for brevity/demonstration</span></span>
</span>
var</span> (</span></span>
	libOnce</span>           sync</span>.</span>Once</span></span>
	limboLib</span>          uintptr</span></span>
	dbOpen</span>            func</span>(</span>string</span>)</span> uintptr</span></span>
	dbClose</span>           func</span>(</span>uintptr</span>)</span> uintptr</span></span>
	connPrepare</span>       func</span>(</span>uintptr</span>,</span> string</span>)</span> uintptr</span></span>
// ... all the function pointers at global scope</span></span>
)</span></span>
</span>
// Register all the symbols on library load</span></span>
func</span> ensureLibLoaded</span>()</span> error</span> {</span></span>
	libOnce</span>.</span>Do</span>(</span>func</span>() {</span></span>
         limboLib</span>,</span> err</span> :=</span> purego</span>.</span>Dlopen</span>(</span>libPath</span>,</span> purego</span>.</span>RTLD_NOW</span>|</span>purego</span>.</span>RTLD_GLOBAL</span>)</span></span>
        if</span> err</span> !=</span> nil</span> {</span></span>
            return</span></span>
        }</span></span>
                     // functionPointer, uintptr, symbol name string</span></span>
		purego</span>.</span>RegisterLibFunc</span>(</span>&</span>dbOpen</span>,</span> limboLib</span>,</span> FfiDbOpen</span>)</span></span>
		purego</span>.</span>RegisterLibFunc</span>(</span>&</span>dbClose</span>,</span> limboLib</span>,</span> FfiDbClose</span>)</span></span>
		purego</span>.</span>RegisterLibFunc</span>(</span>&</span>connPrepare</span>,</span> limboLib</span>,</span> FfiDbPrepare</span>)</span></span>
		purego</span>.</span>RegisterLibFunc</span>(</span>&</span>connGetError</span>,</span> limboLib</span>,</span> FfiDbGetError</span>)</span></span>
        // ...</span></span></code></pre>
After playing around with it a bit, and deciphering what my types needed to look like to properly pass values back and forth
from Rust, I ended up with something like this:</p>
</span>
type</span> valueType</span> int32</span></span>
</span>
const</span> (</span></span>
	intVal</span>  valueType</span> =</span> 0</span></span>
	textVal</span> valueType</span> =</span> 1</span></span>
	blobVal</span> valueType</span> =</span> 2</span></span>
	realVal</span> valueType</span> =</span> 3</span></span>
	nullVal</span> valueType</span> =</span> 4</span></span>
)</span></span>
</span>
// struct to send values over FFI</span></span>
type</span> limboValue</span> struct</span> {</span></span>
	Type</span>  valueType</span></span>
	_</span>     [</span>4</span>]</span>byte</span> // padding</span></span>
	Value</span> [</span>8</span>]</span>byte</span></span>
}</span></span>
</span>
type</span> Blob</span> struct</span> {</span></span>
	Data</span> uintptr</span></span>
	Len</span>  int64</span></span>
}</span></span></code></pre>
I had to use a byte slice instead of a uintptr</code> if I wanted to represent a union and possibly interpret those bytes as an
int64 or float64, as well as a pointer to a TextValue or BlobValue struct that stores the bytes + length</p>
With the accompanying Rust type:</p>
</span>
#</span>[</span>repr</span>(</span>C</span>)]</span></span>
pub enum</span> ValueType</span> {</span></span>
    Integer</span> =</span> 0</span>,</span></span>
    Text</span> =</span> 1</span>,</span></span>
    Blob</span> =</span> 2</span>,</span></span>
    Real</span> =</span> 3</span>,</span></span>
    Null</span> =</span> 4</span>,</span></span>
}</span></span>
</span>
#</span>[</span>repr</span>(</span>C</span>)]</span></span>
pub struct</span> LimboValue</span> {</span></span>
    value_type</span>:</span> ValueType</span>,</span></span>
    value</span>:</span> ValueUnion</span>,</span></span>
}</span></span>
</span>
#</span>[</span>repr</span>(</span>C</span>)]</span></span>
union</span> ValueUnion</span> {</span></span>
    int_val</span>:</span> i64</span>,</span></span>
    real_val</span>:</span> f64</span>,</span></span>
    text_ptr</span>: *</span>const</span> c_char</span>,</span></span>
    blob_ptr</span>: *</span>const</span> c_void</span>,</span></span>
}</span></span></code></pre>
This is how, for example, I convert the slice of driver.Args</code> that in order to implement statement.Query</code>
from database/sql/driver</code>.</p>
// convert a Go slice of driver.Value to a slice of limboValue that can be sent over FFI</span></span>
// for Blob types, we have to pin them so they are not garbage collected before they can be copied</span></span>
// into a buffer on the Rust side, so we return a function to unpin them that can be deferred after this call</span></span>
func</span> buildArgs</span>(</span>args</span> []</span>driver</span>.</span>Value</span>) ([]</span>limboValue</span>,</span> func</span>(),</span> error</span>) {</span></span>
	pinner</span> :=</span> new</span>(</span>runtime</span>.</span>Pinner</span>)</span>                 </span></span>
    // I was unaware that runtime.Pinner was a thing, prior to this</span></span>
	argSlice</span> :=</span> make</span>([]</span>limboValue</span>,</span> len</span>(</span>args</span>))</span></span>
	for</span> i</span>,</span> v</span> :=</span> range</span> args</span> {</span></span>
		limboVal</span> :=</span> limboValue</span>{}</span></span>
		switch</span> val</span> :=</span> v</span>.(</span>type</span>) {</span></span>
		case</span> nil</span>:</span></span>
			limboVal</span>.</span>Type</span> =</span> nullVal</span></span>
		case int64</span>:</span></span>
			limboVal</span>.</span>Type</span> =</span> intVal</span></span>
			limboVal</span>.</span>Value</span> = *</span>(</span>*</span>[</span>8</span>]</span>byte</span>)(</span>unsafe</span>.</span>Pointer</span>(</span>&</span>val</span>))</span></span>
		case float64</span>:</span></span>
			limboVal</span>.</span>Type</span> =</span> realVal</span></span>
			limboVal</span>.</span>Value</span> = *</span>(</span>*</span>[</span>8</span>]</span>byte</span>)(</span>unsafe</span>.</span>Pointer</span>(</span>&</span>val</span>))</span></span>
		case string</span>:</span></span>
			limboVal</span>.</span>Type</span> =</span> textVal</span></span>
			cstr</span> :=</span> CString</span>(</span>val</span>)</span></span>
			pinner</span>.</span>Pin</span>(</span>cstr</span>)</span></span>
			*</span>(</span>*</span>uintptr</span>)(</span>unsafe</span>.</span>Pointer</span>(</span>&</span>limboVal</span>.</span>Value</span>))</span> =</span> uintptr</span>(</span>unsafe</span>.</span>Pointer</span>(</span>cstr</span>))</span></span>
		case</span> []</span>byte</span>:</span></span>
			limboVal</span>.</span>Type</span> =</span> blobVal</span></span>
			blob</span> :=</span> makeBlob</span>(</span>val</span>)</span></span>
			pinner</span>.</span>Pin</span>(</span>blob</span>)</span></span>
			*</span>(</span>*</span>uintptr</span>)(</span>unsafe</span>.</span>Pointer</span>(</span>&</span>limboVal</span>.</span>Value</span>))</span> =</span> uintptr</span>(</span>unsafe</span>.</span>Pointer</span>(</span>blob</span>))</span></span>
		default</span>:</span></span>
			return</span> nil</span>,</span> pinner</span>.</span>Unpin</span>,</span> fmt</span>.</span>Errorf</span>(</span>"unsupported type: %T"</span>,</span> v</span>)</span></span>
		}</span></span>
		argSlice</span>[</span>i</span>]</span> =</span> limboVal</span></span>
	}</span></span>
	return</span> argSlice</span>,</span> pinner</span>.</span>Unpin</span>,</span> nil</span></span>
}</span></span>
</span>
// convert a byte slice to a Blob type that can be sent over FFI</span></span>
func</span> makeBlob</span>(</span>b</span> []</span>byte</span>)</span> *</span>Blob</span> {</span></span>
	if</span> len</span>(</span>b</span>)</span> ==</span> 0</span> {</span></span>
		return</span> nil</span></span>
	}</span></span>
	return</span> &</span>Blob</span>{</span></span>
		Data</span>:</span> uintptr</span>(</span>unsafe</span>.</span>Pointer</span>(</span>&</span>b</span>[</span>0</span>])),</span></span>
		Len</span>:</span>  int64</span>(</span>len</span>(</span>b</span>)),</span></span>
	}</span></span>
}</span></span></code></pre>
Looking at purego's source code gave me some inspiration and helped me get the general idea of how to manipulate
and work with pointers and types received over FFI. For instance, this is the function they use to convert a Go
string to a C String:</p>
/*</span></span>
Credit (Apache2 License) to:</span></span>
      https://github.com/ebitengine/purego/blob/main/internal/strings/strings.go</span></span>
*/</span></span>
func</span> CString</span>(</span>name</span> string</span>)</span> *</span>byte</span> {</span></span>
	if</span> hasSuffix</span>(</span>name</span>,</span> "</span>\x00</span>"</span>) {</span></span>
		return</span> &</span>(</span>*</span>(</span>*</span>[]</span>byte</span>)(</span>unsafe</span>.</span>Pointer</span>(</span>&</span>name</span>)))[</span>0</span>]</span></span>
	}</span></span>
	b</span> :=</span> make</span>([]</span>byte</span>,</span> len</span>(</span>name</span>)</span>+</span>1</span>)</span></span>
	copy</span>(</span>b</span>,</span> name</span>)</span></span>
	return</span> &</span>b</span>[</span>0</span>]</span></span>
}</span></span>
</span></code></pre>
And I was able to adapt everything else around these concepts. There were a few things that I wasn't super pleased with that I still
have to figure out. For instance, sending back an array of strings from Rust was such a pain in the ass that rows.Columns()</code> method
calls this function:</p>
#</span>[</span>no_mangle</span>]</span></span>
pub extern</span> "C"</span> fn</span> rows_get_columns</span>(</span>rows_ptr</span>: *</span>mut</span> c_void</span>)</span> -></span> i32</span> {</span></span>
    if</span> rows_ptr</span>.</span>is_null</span>() {</span></span>
        return</span> -</span>1</span>;</span></span>
    }</span></span>
    let</span> rows</span> =</span> LimboRows</span>::</span>from_ptr</span>(</span>rows_ptr</span>);</span></span>
    rows</span>.</span>stmt</span>.</span>num_columns</span>()</span> as i32</span></span>
}</span></span></code></pre>
to get the number of result columns for the prepared statement, then calls rows_get_column_name</code> with the index of the column
name to return.</p>
It's all pretty cursed huh?</h3>
================================</p>
But it works, and so far there is a decent start to bindings that don't fit into any of the above
categories with any of the existing sqlite drivers :)</p>
I'll follow this post up soon with another explaining some of the caveats, the workarounds for them, whether or not we actually
solved any of the issues that we described with CGO, and maybe run some benchmarks.</p>
But for now, thanks for reading.</p>

PThorpe92</a></li>
</ul>


Magic isn't real
Unknown — Thu, 03 Oct 2024 00:00:00 +0000
Any sufficiently advanced technology is indistinguishable from magic</code>.</p>

Arthur C. Clarke</li>
</ul>
This quote applies just as much to developers as it does non-tech people, sometimes more. I remember towards the beginning of
my programming journey (both the first time I learned 18+ years ago, and again ~15 years later), the root cause of the feeling
responsible for what they call tutorial hell</code> (I personally loathe tutorials, and always chose to instead try to build
things myself, and I attribute a great deal of the relative success I have achieved to this).</p>
The situation:</p>
You feel like you understand perfectly how to properly swing a hammer, lay brick, frame drywall, and you learned the right way to measure
and cut beams with a saw, yet you still look at buildings and architecture and stand completely baffled that those tools you have learned
were the same ones used to build these great structures. With no idea where to start, you stare at your tools, supplies and materials
wondering if they must have some kind of special equipment or secret freemason knowledge that you don't have access to. You don't
know how someone ended up with that result, using the same tools you see in front of you, and you definitely cannot imagine cutting
the first board or laying the first brick.</p>
Many know that this is the exact feeling of learning how to program, and fully grasping the concepts of loops, variables, data structures,
trees, stacks, linked-lists, arrays, control flow, etc, etc, etc... then looking at a compiler, a video game, an operating system,
web browser and thinking yeah right...</code>. Those devs must all have started programming C and x86 assembly while they were in diapers, and
all attended Stanford where they were taught secret knowledge passed down from Ken Thompson, by Brian Kernaghan himself.</p>
Assuming you don't take the strict path of the JS frameworker vercel user: eventually after enough time, you start to recognize patterns.
You 'go to definition' on enough methods from libraries you use to see how they are implemented and you build enough side projects and
watch enough 'tsoding daily', 'sphaerophoria', and 'awesomekling' to begin to demystify at least how things like web/network protocols,
or image/video encodings, syscalls/file IO operations work at some level. You no longer would feel completely lost if you had to write
a shell or a lisp</code> interpreter: you would at the very least know that to begin, you would probably have to read the source file into memory and break it up into tokens before trying to parse it to build the syntax tree needed so you can traverse and analyze it before stepping through it to execute the code. Previously, what now feels so obvious to you, would have seemed some kind of sorcery reserved only for the
aforementioned programming elite.</p>
I'm sure I'm not alone, in that each time you pull the curtain off a piece of 'magic</em>', you have the same thought:</p>
Oooooh yeah. I mean, well duh.. how else would you do that? I can't believe I couldn't see it</code>.</p>
As time goes on, there are less and less things I run into where I cannot mentally parse at least from a very broad and high level, what an
implementation might look like. Now I definitely</em> don't claim to know how kernel internals, 3d rendering, or GPU drivers work, but what I
mean is most things have lost the shadowy mystique, and feel more like something I can get excited to learn about, rather than a scary
forbidden knowledge I will never be allowed to possess. Although for those things, that may as well be the case ;)</p>
The other day, after a long day's work managing synchronizing different environments/k8s clusters, I decided to browse HN as I normally do
at that time. I ran into a post referencing comptime</code> for Go, that linked to a github repo</a>. It immediately caught
my attention, as although I have not written Zig myself, Andrew Kelly is one of my programming idols and I definitely follow zig's development.
Comptime is one of Zig's most envied language features, and although it is achievable via metaprogramming or constexpr</code> in other languages,
zig's straightforward procedural approach/API makes it particularly unique and admired.</p>
This was when I came upon that familiar feeling:</p>


  </div>




  </div>


^^^^^^
Me if you had told me I had to implement comptime in go without touching the compiler
So I decided that I had to know how this was done, and I had a few hours to spare so I decided I would maybe try to contribute, or at least
add some kind of feature of any level of value, just to force myself to understand what was going on here.</p>
Then after a brief peruse through the code...</p>
Turns out, you can use the source file information you get through this flag you can pass at build time in Go called -toolexec</code>
which allows you to invoke toolchain programs, in this case the prep</code> binary, which is called with the absolute path of the program
and by using a combination of another one of the author's packages, goinject</code>, and the yaegi: (yet another elegant Go interpreter)</code>
library: you can get the AST, file decorator and import restorer, by implementing Modifier</code>, which then allows you to collect the variables
from the relevant function in the tree, output them each to a temporary file, on which you run the interpreter, giving you the
computed results of foo in prep.Comptime(foo())</code>, which you then use to replace the values in the DST by the Modify</code> pass.
viola, you have achieved compile time computations.</p>
Oh, well yeah. That makes perfect sense. I mean how else did I think it was gonna work?</p>
After a couple hours, I had added variable scoping, and global const declarations which I concluded was actually
not a useful feature at all, because each function is evaluated on it's own, leaving essentially a 0% chance of
actual naming/scope conflicts. But the point is, I didn't discover that until I had finished writing it with
some tests, and although the 'feature' is useless, the whole process was a very valuable learning experience and
all around good use of my time.</p>
This is just a reminder to everyone at different levels of their developer journey, that the "magic" is not real and the overwhelming
majority of the time, you are simply lacking the necessary context and it will likely make perfect sense to you as soon as you have it.</p>
It's always</em> worth your time to learn parts of the stack that you might not work in daily. As you build your fundamental understanding,
it demystifies other pieces of the puzzle that you would never would have put together otherwise. Even if it doesn't feel
important now, I guarantee the knowledge pays off at some point in the future.</p>
Keep learning every day, strive for deeper understanding, and spend time building or hacking on even things that are considered
'solved problems'. Even if you are only paid to write React, it is very much of value to you and your career to understand how
the internals work, or how your one-click 'serverless' auto-scaling deployments work...</p>
(hint: servers)</p>


PHP/Laravel: Why we rewrote our app in Go
Unknown — Sun, 14 Jul 2024 00:00:00 +0000
(Modern) PHP: Does it really suck?</h3>
Like many, many developers out there over the age of 30, I basically started my programming journey with PHP (and perl).
Circa ~'05, PHP was the</em> go-to language for the web, and Adobe Dreamweaver supported it out of the box! Most HTML was
hand-written, most connections were absent TLS, and lots of form data was still being stuffed raw-dog right into SQL queries.</p>
This, I estimate, is what is responsible for 69.420% of the hate PHP gets today. (The rest comes from JS andys who have never
seen or written a line of it, who likely just want to deflect some of the flak)</p>
When I was hired at my current job, due to my situation of being currently incarcerated see my other post</a>,
I was happy to find any dev work, and in absolutely no position to be picky about the tech stack. I learned that we would be using Laravel</code>, and PHP 8.2 for the
backend, and although I had less than enthusiastic feelings about PHP itself, I had heard good things about Laravel and dove in head-first.</p>
Background info:</code>

We are a non-profit tech startup building an education-focused platform to aggregate data from various integrations (Learning management systems),
in which incarcerated students can access their coursework and grades, and correctional facilities can track their progress across
multiple providers. The original product was developed in the MO state prison system by a handful of residents, and was written in PHP 5.6,
with no internet access, no stack overflow, and no real-world development experience. It was truly impressive to see the result of this,
and I am surprised to this day how much they were able to accomplish with such limited resources. This was the reason PHP was chosen as the
primary language for version 2 of the platform.</p>
The Good:</code>

Since I had been away from the tech world for so long, I missed out on most 'frameworks'. Rails was just blowing up when I was programming last,
so I never got much experience with the concept of an HTTP server being abstracted away from the developer. Back then a lot of the PHP was inline in the HTML, and you just thought about the whole process differently. I soon realized that nearly every single thing you might want to do, Laravel not only had thought of it, but seemed to have a command that generates the necessary code to do it.
If I had realized this sooner, I would have had a much more smooth experience with the platform. Unfortunately I spent the first few months trying to solve
a lot of the common problems by hand first, before realizing that Laravel had a highly opinionated way of doing things, and that it was best to just
follow the conventions. I'm not against an opinionated approach, in fact it's one of the things I like most about Go.</p>
Overall, Laravel, for me, is much more accessible than a framework like Django, which completely abstracts away any concept of a server
(if you aren't a django developer, and you were to see a django codebase for the first time, I'd be willing to bet pretty much whatever that
unless you studied it pretty thoroughly and did some google'ing, you couldn't tell me it's an http server). Laravel has the straight forward and easily identifiable concepts you would expect, the ORM is actually pretty good, assuming you can configure your LSP and linter
to properly recognize the methods, and the built-in migration system is easy to use. All in all, as far as frameworks go,
Laravel is surprisingly good, and for a blog or simple e-commerce website, I'd take it over NextJS or Django any day (I'm not a Ruby dev so I can't speak on Rails).</p>
Modern PHP is very</em> different from the PHP of old. It has some surprisingly good language features like Traits and match statements, however their enums are in competition with typescript for being the worst of any language I have seen. Modern PHP manages to take a rather straight-forward object oriented approach, at a bit more than the level of python. My personal preferences lean much heavier into the functional realm, so I was pleased to find that PHP now also provides some basic functional features too, like their 'iterator' package.</p>
The Bad:</em></p>
First lets go with the obvious... C++ and JS share this issue as well: backwards compatibility has just kept so much suspect nonsense in the language, that you are stuck trying to figure out which of the 4 ways to do X thing you should be using. PHP's problem in particular, is that they seem to have switched the naming convention each time they designed one of these bad features, so you will often find a lower_snake, Pascal, camelCase, ... versions of a slightly different named function and be stuck with that mess for a while. I don't know if they switched the global naming convention every 5 years, but damn is there some inconsistency there.</p>
Another fairly obvious one: I am heavily on team "strongly and staticly typed languages are better", I write Rust in my free time on all my hobby or contract projects. Now,
after having worked with PHP and Laravel for about 8 months, I can say that I am even more convinced. You may be able to write plenty of larger,
more complex applications in Laravel with no problems, however
if one day you need to perform a non-trivial refactor, you will be in for a rough time.</p>
Some of the most common issues I have had with Laravel are the fact that because
Laravel takes care of so many common things for you, you will define things like Requests</code> and Resources</code>, that come with methods on them validate the
JSON bodies or serialize the returned database rows into a JSON response. This is all well and good, but many applications will have several Request and
Resource types for each controller, and if you have multiple rows/fields change on each table, you will be having to edit files like these across many parts
of your application. Anything you may miss, static typing will easily catch at compile time.</p>
"Dynamic types are fine, testing fixes this problem..."
</p>
PHP's linter and LSP are simply sub-par compared to what you
are going to get with Gopls</code> or Rust-Analyzer</code>, or even something like TSServer</code>. This isn't necessarily a PHP or even an issue with Laravel, but
the introspection tools are just not able to handle the complex types that you are working with in Laravel. Simply setting up a good linter
took hours because it would throw errors on lots of valid code, and I had to outright disable it for certain files. (skill issue, I know. but I want the linter to just work)</p>
Yes, PHP 8+ has some type hinting, but there is no compile time to enforce them at, so much of the usefulness is lost. I have had handlers where there was (declare(strict_types = 1)), declared:
that didn't throw errors until that specific handler had been called.</p>
Now, Go's type-system is far from perfect, and it's famous lack of type safe enums/unions makes me cry a little bit inside.</p>
type</span> Color</span> int</span></span>
</span>
const</span> (</span></span>
  Red</span> Color</span> =</span> 2</span></span>
  Blue</span> Color</span> =</span> 3</span></span>
  Green</span> Color</span> =</span> 4</span></span>
)</span></span>
</span>
func</span> applyColor</span>(</span>color</span> Color</span>) {</span></span>
  result</span> :=</span> doSomethingWithColor</span>(</span>color</span>)</span></span>
  //...</span></span>
}</span></span>
</span>
func</span> main</span>() {</span></span>
  var</span> num</span> int</span> =</span> 2340</span></span>
  applyColor</span>(</span>num</span>)</span></span>
  // nothing wrong with this!, no problems here!   💀</span></span>
}</span></span></code></pre>
Although this is far from my only issue with Go, (I'm looking at you, single letter variable naming convention -_- ) (w http.ResponseWriter, r *http.Request)</code>,
modules, and capital-letter for public access... and mayyybe some more issues I won't get into right now 😅</p>
But what we did gain, on top of performance, type safety, long-lived processes, simpler deployments and the ability to move some functionality into independent services, was a very clear intention in our back-end of what is really happening. Go's stdlib net/http</code> imho, is the perfect</em> amount of abstraction
for writing most servers. It would be impossible to write back-ends in Go, and not have a solid fundamental understanding of what is going on. You are not
simply figuring out X framework's particular method of solving Y end-user problem, you are implementing the actual solution to it, by understanding
what needs to happen.</p>
Random example:</p>
Lets go with basic auth middleware. Many projects rely on some outside service (we actually do now rely on Ory</code> because we needed comprehensive identity mgmt for the environment), but still employ the good old JWT.</p>
In Go, a popular way of implementing auth would be to install the JWT</code> module, and create a Claims</code> struct with some user + session information, encrypt it,
and set the cookie in the browser. Then you make some middleware (which is exactly what you would think it would be, a function that accepts a handler, and returns a handler), to check the cookie, validate the JWT, and create a context.Context</code> key value pair with the claims you can pass around to subsequent
handlers, to make the relevant user session information available where you will need it.</p>
This is a good example of how you are just forced to solve the problem itself, as opposed to trying to figure out exactly how to use X frameworks particular abstraction layer, having to read their docs and make sure you are doing all the
required things in the right order to make this work. In Go, you are thinking about how to solve the specific problem of needing to validate each
request sent by the user, and it forces you to understand what is really happening behind the scenes. I find often times that the problem itself is far more trivial than trying to figure out how to use the API of whatever 'solution' is being sold to you as a convenience. This leaves newer developers clueless as to how these things are actually implemented, and only teaches them how to use X tool.</p>
Yes I know that this argument could be applied all the way down the stack, eventually ending in the semi brainrot argument of "you have to use only C or assembly for everything or it's a skill issue" that's been popular in some programming circles lately with the seemingly unemployed. But this is the particular layer where I'll pick my personal hill to die on.</p>
This abstraction level is fine for the majority of the web applications, and unless you need extreme performance, Go will serve quite well
as a versatile, fast and simple back-end or service. Go services can easily run in scratch</code> docker containers with almost no dependencies.
Deployments and development environments are very simple as everything integrates well in the docker/container ecosystem.</p>
Production php/laravel deployments are significantly more work and preparation, and you can say that containerizing is cheating, but
even container free I am able to deploy our Go application on an ec2</code> in 1/4 of the time it would take me to deploy our old codebase.</p>
That is enough of my rant for today. These aren't two languages that are commonly/should be compared as they have very little in common,
so I am not trying to do too much here. Overall I am much happier getting to work with Go every day over php, and so far it has been
a great experience.</p>
Does PHP suck as much as people say? No, definitely not.</p>
Do I particularly want to use it for anything more complex than a blog or simple application? No, probably not.</p>
At least with php, nobody is trying to use it for anything outside of it's intended domain. 🥲</p>

Thanks for reading this, questions or comments feel free to email me at preston@pthorpe92.dev</code></p>


Gratitude
Unknown — Mon, 13 Nov 2023 00:00:00 +0000
How I got here</code> is already far too long of a post, so I must include this for all the credits and gratitude I need to extend to those who made this possible.</p>
To my parents: who supported me every single day of the decade+ I have spent behind bars, and who never gave up on me, even when I definitely deserved it and had well past given up on myself.</p>
To Mark Spahr and Rachel Mclean Who I work with in the Education Technology department, who gave me a chance to prove myself, and without whom I would never have had any of the access or resources I needed to gain the skills, knowledge and experience that I have today. I owe virtually all of my success to them for giving me the opportunity.</p>
To Haley, Jessica, Nokie and the entire team/family at UnlockedLabs. Definitely the only company that was going to do anything with my resume but chuckle and throw it away. I couldn't be more proud to be part of the team.</p>
To Laura Rodas (Education Director), and Randy Liberty (Commissioner of the MDOC) and all who approved the policies that make things like this possible.</p>
and finally I just want to throw a thank you to Andreas Kling and The Primeagen, who I have been following for a long time, but who will most likely never read this, but nevertheless who inspired me with their stories and their work and who I look up to very much. I have been clean and sober for well over 4 years now, and it's not easy in this environment. It has been a huge inspiration to see success stories of others that have gone through addictions.</p>


OCaml + Rust: some comparisons
Unknown — Sun, 12 Nov 2023 00:00:00 +0000
Following up on the first impressions post, let's solve a
problem in OCaml and compare the Rust solution.</p>
In Response</em> to how well received my last post was</h4>
I thought I would follow up with some comparisons between how I would solve a simple problem (Advent of code 2022, day 3 part 1) in both languages and compare the solutions, while trying to portray the kind of thinking required (I think anyway) to begin to solve problems in (OCa)ml style languages.</p>

The only thing I will say first.. is these kind of simple
questions do not lend themselves to great comparisons,
because someone well versed in their language of choice, will
be much more likely and able to just solve the problem in a
neat, concise, way probably in a single function</p>
</blockquote>
The Problem</h3>
I'm not going to bother with the story, if you are interested,
you can read it here</a>.</p>
I will also be leaving out the reading in of the file, as
it isn't specific to this problem, and everyone will usually
have an 'aoc' page of helper fn's for this anyway.</p>
The Basics</h5>
We need to take a string, split it in half, and then find the</p>
one single char that occurs in both substrings, and sum the val</p>
given to each char (a-z = 1-26) (A-Z = 27-52)</p>
Pretty simple right? If it wasn't for assigning the values to</p>
chars (or if they were just ascii values), then I'd say that's</p>
a good-ol' python one-liner</code> if I ever saw one.</p>
For the sake of simplicity. We need:</p>
"vJrwpWtwJgWrhcsFMMfFFhFp"</span></span>
</span>
to split into:</span></span>
</span>
("vJrwpWtwJgWr", "hcsFMMfFFhFp")</span></span>
      ^                      ^</span></span>
p is the char that occurs in both substrings</span></span>
</span>
     the value of p is 16</span></span>
</span>
we simply add this value for each original string.</span></span></code></pre>Because I am an Ocaml n00b</h4>
We will break this up into 3 simple functions (and a main), and try</em> to
solve the problem in relatively the same steps in both</em> languages,
so we can give them a fair</code> comparison of basic operations.</p>


Split the list of strings, into a list of tuples.</p>
</li>

With that input, find and return the suspect char.</p>
</li>

Perform some lookup or calculation to get and sum the values of the chars.</p>
</li>

Call these from a main function + print to stdout.</p>
</li>
</ol>
🦀 🦀</p>
// *** We iterate over the list of strings, split and collect into vec of tuples ***</span></span>
fn</span> split_string</span>(</span>s</span>:</span> Vec</span><</span>&</span>str</span>></span>)</span> -></span> Vec</span><</span>(</span>&</span>str</span>,</span> &</span>str</span>)</span>></span> {</span></span>
    s</span>.</span>iter</span>()</span></span>
        .</span>map</span>(</span>|</span>line</span>|</span> {</span></span>
            let</span> mid</span> =</span> line</span>.</span>len</span>()</span> /</span> 2</span>;</span></span>
            let</span> (</span>first</span>,</span> second</span>)</span> =</span> line</span>.</span>split_at</span>(</span>mid</span>);</span></span>
            (</span>first</span>,</span> second</span>)</span></span>
        })</span></span>
        .</span>collect</span>()</span></span>
}</span></span>
// *** Find the character that occurs in both substrings ***</span></span>
fn</span> find_dupe</span>(</span>s</span>:</span> (</span>&</span>str</span>,</span> &</span>str</span>))</span> -></span> Option</span><</span>char</span>></span> {</span></span>
    s</span>.</span>0</span>.</span>chars</span>()</span>.</span>find</span>(</span>|&</span>c</span>|</span> s</span>.</span>1</span>.</span>contains</span>(</span>c</span>))</span></span>
}</span></span>
// *** Map the char to its value ***</span></span>
fn</span> map_value</span>(</span>c</span>:</span> char</span>)</span> -></span> usize</span> {</span></span>
    if</span> c</span>.</span>is_lowercase</span>() {</span></span>
        ('</span>a</span>'</span>..=</span>'</span>z</span>')</span></span>
            .</span>enumerate</span>()</span> // enumerate gives us index, value so we return the index + 1 for the char</span></span>
            .</span>find_map</span>(</span>|</span>(</span>i</span>,</span> x</span>)</span>|</span> if</span> x</span> ==</span> c</span> {</span> Some</span>(</span>i</span> +</span> 1</span>) }</span> else</span> {</span> None</span> })</span></span>
            .</span>unwrap_or</span>(</span>0</span>)</span></span>
    }</span> else</span> {</span></span>
        ('</span>A</span>'</span>..=</span>'</span>Z</span>')</span></span>
            .</span>enumerate</span>()</span></span>
            .</span>find_map</span>(</span>|</span>(</span>i</span>,</span> x</span>)</span>|</span> if</span> x</span> ==</span> c</span> {</span> Some</span>(</span>i</span> +</span> 27</span>) }</span> else</span> {</span> None</span> })</span></span>
            .</span>unwrap_or</span>(</span>0</span>)</span></span>
    }</span></span>
}</span></span>
fn</span> main</span>() {</span></span>
    let</span> file</span> =</span> std</span>::</span>fs</span>::</span>read_to_string</span>(</span>"input"</span>)</span>.</span>unwrap</span>();</span></span>
    let</span> s</span> =</span> file</span>.</span>split</span>('</span>\n</span>')</span>.</span>collect</span>::</span><</span>Vec</span><</span>&</span>str</span>>></span>();</span></span>
    let</span> spltlines</span> =</span> split_string</span>(</span>s</span>);</span></span>
    let</span> total</span> =</span> spltlines</span></span>
        .</span>iter</span>()</span></span>
        .</span>map</span>(</span>|</span>t</span>|</span> {</span></span>
            if let</span> Some</span>(</span>val</span>)</span> =</span> find_dupe</span>(</span>*</span>t</span>) {</span></span>
                map_value</span>(</span>val</span>)</span></span>
            }</span> else</span> {</span></span>
                0</span></span>
            }</span></span>
        })</span></span>
        .</span>sum</span>::</span><</span>usize</span>></span>();</span></span>
  println!</span>(</span>"</span>{</span>total</span>}</span>"</span>);</span></span>
}</span></span>
// As I'm sure some will point out, we definitely could have made this a lot more concise</span></span>
// But for the sake of comparison, we will leave the naive implementation</span></span></code></pre>
So this was a touch more painful than I thought it would be...
Forcing myself to split those up into three different functions
when I would normally for such a simple problem just cram it all
into one, being able to operate more by chaining iterators.</p>
But I think you could definitely come up with worse examples.</p>
Time for the 🐫'Caml 🐪</p>
For those of you who aren't familiar with Ocaml, I'll try to briefly
explain my very limited understanding.</p>
(* In Ocaml, functions from other modules are called with 'Module.fn' syntax,</span></span>
so String.length is the length fn from the String module. I do like how Mod</span></span>
names are capitalized, instead of the public fn name like in Go. *)</span></span>
</span>
(* Here we have a recursive function(denoted by the rec keyword)</span></span>
called 'split_string' that takes our input lst, and an</span></span>
accumulator (an empty list in which to put the result: our tuples) *)</span></span>
</span>
let</span> rec</span> split_string</span> (</span>lst</span>:</span> string</span> list</span>)</span> acc</span>:</span> (</span>string</span> *</span> string</span>)</span> list</span> =</span></span>
  match</span> lst</span> with</span></span>
  |</span> []</span> -></span> acc</span></span>
  |</span> hd</span> ::</span>tl</span> -></span> let</span> len</span> =</span> (</span>String</span>.</span>length hd</span>)</span> /</span> 2</span> in</span></span>
              let</span> substr_a</span> =</span> String</span>.</span>sub hd</span> 0</span> len</span> in</span></span>
              let</span> substr_b</span> =</span> String</span>.</span>sub hd len len</span> in</span></span>
              split_string tl</span> ((</span>substr_a</span>,</span> substr_b</span>)</span>::</span>acc</span>)</span></span>
;;</span></span>
(* We pattern-match on the list, and while there is still some strings remaining,</span></span>
we get the length of the string(hd) and then call String.sub on(which takes the string,</span></span>
the position at which the substring starts, and the length of the substring) to split</span></span>
it into the tuple. we append the tuple to the accumulator with the 'cons' :: operator</span></span>
and then call the function recursively with the accumulator + the remainder of the list*)</span></span>
------------------------------------------------------------------------------------------</span></span>
</span>
(* Here we have a function with no parameters that simply computes</span></span>
the integer values of lower+uppercase chars and creates a (key val) style list</span></span>
of (char int) tuples e.g. ('p', 16) ('q', 17)... so pretty much seems to me like a</span></span>
super inefficient hashmap. I'm sure all you OCaml wizards are cringing rn *)</span></span>
</span>
let</span> char_to_int</span> ()</span>:</span> (</span>char</span> *</span> int</span>)</span> list</span> =</span></span>
  let</span> rec</span> mapping</span> acc start_char end_char value</span> =</span></span>
    if</span> start_char</span> ></span> end_char</span> then</span></span>
      acc</span></span>
    else</span></span>
      mapping</span> ((</span>start_char</span>,</span> value</span>)</span> ::</span> acc</span>) (</span>Char</span>.</span>chr</span> (</span>Char</span>.</span>code start_char</span> +</span> 1</span>))</span> end_char</span> (</span>value</span> +</span> 1</span>)</span></span>
  in</span></span>
  let</span> lower</span> =</span> mapping</span> []</span> 'a' 'z'</span> 1</span> in</span></span>
  let</span> upper</span> =</span> mapping</span> []</span> 'A' 'Z'</span> 27</span> in</span></span>
  List</span>.</span>rev_append lower upper</span></span>
</span>
(* We call that function, and then use List.assoc to get the value in our key, val tuple *)</span></span>
let</span> char_to_int_mapping</span> =</span> char_to_int</span> ()</span></span>
let</span> find_value</span> char_to_find</span> =</span></span>
  try</span></span>
    let</span> result</span> =</span> List</span>.</span>assoc char_to_find char_to_int_mapping</span> in</span></span>
    Some</span> result</span></span>
  with</span> Not_found</span> -></span> None</span></span>
;;</span></span>
-------------------------------------------------------------------------------</span></span>
(* Simple function to find the char between the two substrings, utilizing</span></span>
 String.fold_left which allows us to apply a fn on each char of a string,</span></span>
so we call Strings.contains on the other string on each char, and append it *)</span></span>
</span>
let</span> find_common_char</span> (</span>s1</span>,</span> s2</span>)</span> =</span></span>
  let</span> common_chars</span> =</span></span>
    String</span>.</span>fold_left</span></span>
      (</span>fun</span> acc c</span> -></span> if</span> String</span>.</span>contains s2 c</span> then</span> c</span> ::</span> acc</span> else</span> acc</span>)</span></span>
      []</span> s1</span></span>
  in</span></span>
  match</span> common_chars</span> with</span></span>
  |</span> []</span> -></span> None</span></span>
  |</span> common_char</span> ::</span> _</span> -></span> Some</span> common_char</span></span>
;;</span></span>
</span>
(* Another recursive function to sum up the values of the chars. we cheated in</span></span>
rust by having this in the main fn. but notice how, here we cannot call</span></span>
.unwrap_or() or similar fn's, we must pattern match *)</span></span>
</span>
let</span> rec</span> sum_vals</span> (</span>lst</span>:</span> (</span>string</span> *</span> string</span>)</span> list</span>) (</span>sum</span>:</span> int</span>)</span>:</span> int</span> =</span></span>
  match</span> lst</span> with</span></span>
  |</span> []</span> -></span> sum</span></span>
  |</span> hd</span>::</span>tl</span> -></span></span>
      sum_vals tl sum</span> +</span> (</span></span>
        match</span> find_common_char hd</span> with</span></span>
            |</span> None</span> -></span> 0</span></span>
            |</span> Some</span> ch</span> -></span> match</span> find_value ch</span> with</span></span>
                        |</span> Some</span> n</span> -></span> n</span></span>
                        |</span> None</span> -></span> 0</span>)</span></span>
;;</span></span>
</span>
let</span> lines</span> =</span> split_lines</span> (</span>read_lines</span> "input"</span>)</span> []</span> in</span></span>
let</span> ans</span> =</span> sum_vals lines</span> 0</span> in</span></span>
print_int ans</span></span>
</span>
(*^ Here would be our main fn, we are just calling the above functions, read_lines</span></span>
is the helper fn that reads the file in to a list of strings *)</span></span></code></pre>
Now, I'm sure that was a mess, but I have only been messing with the 'Caml for a
couple weeks now. Please feel free to hit me up with corrections/pointers
if you are an Ocaml wizard, I'd love to learn as much as I can. You'll have to
email me though, am I far too lazy to setup sqlite for comments.</p>
I imagine that as I get more comfortable with the standard library I will
end up using built in features where currently I might use recursion. However I am starting
to enjoy the style and find it quite satisfying to produce an effective recursive solution.</p>
I haven't yet looked into the other common data structures either, so I am sure
there are plenty of other improvements that can be made.</p>
If you were unfamiliar with either language.. god help you, the whole post probably looked
pretty scary depending on what background you have.</p>
As I have learned a tiny bit more, it is definitely growing on me, and tasks
I was tripping over even a few days ago are becoming a little more natural. The
farther I get, the more I realize the similarities and as I go on, whatever advantage
there might be from using a language that was heavily influenced by it,
will probably start to show more and more.</p>
I hope to provide (maybe more concise next time) more of my journey and thoughts on the 'Caml,
which I can say, I most definitely recommend to anyone who is interested in
challenging their idea and perception of programming, while getting to learn a language
with some very different syntax but also some great FP concepts/features that
is actually used outside of Academia (Haskell we're looking at you)</p>

Note: This is meant to be a comparison to show people who are interested in learning 🐫
just some of the initial differences, and structure, to see what their experience might be like,
as they begin to play around with the language.
I try to learn every day, and the more that I learn, the more I realize that I am an expert at nothing.</p>
</blockquote>
Thanks for reading this, feel free to reach out to me if you are interested in discussing these
topics, or if you have any corrections/pointers for me. I can be found on the eza</a> matrix/gitter</a> channel.
or you can email me at p@eza.rocks</a></p>


EDIT:</p>
</blockquote>
A solution from an actual</em> OCaml programmer</h1>
This excellent, concise solution was emailed to me by Yawar Amin https://github.com/yawaramin</a></p>
(after finding him on github, it became immediately obvious that he is indeed, one of the before-mentioned OCaml wizards)</p>
module</span> SC</span> =</span> Set</span>.</span>Make</span>(</span>Char</span>)</span> (* Module for set of char *)</span></span>
</span>
let</span> prio</span> ch</span> =</span></span>
  let</span> code</span> =</span> Char</span>.</span>code ch</span> in</span></span>
  if</span> code</span> >=</span> 97</span> then</span> code</span> -</span> 96</span> (* a-z *)</span></span>
  else</span> code</span> -</span> 38</span> (* A-Z *)</span></span>
</span>
let</span> common</span> string</span> =</span></span>
  let</span> len</span> =</span> String</span>.</span>length string</span> in</span></span>
  let</span> half_len</span> =</span> len</span> /</span> 2</span> in</span></span>
  let</span> first</span> =</span> SC</span>.</span>of_seq</span> (</span>String</span>.</span>to_seq</span> (</span>String</span>.</span>sub string</span> 0</span> half_len</span>))</span> in</span></span>
  let</span> second</span> =</span> SC</span>.</span>of_seq</span> (</span>String</span>.</span>to_seq</span> (</span>String</span>.</span>sub string half_len</span></span>
half_len</span>))</span> in</span></span>
  let</span> common_char</span> =</span> SC</span>.</span>inter first second</span> in</span> (* Set intersection *)</span></span>
  assert</span> (</span>SC</span>.</span>cardinal common_char</span> =</span> 1</span>)</span>;</span> (* Assert set contains only</span></span>
one member *)</span></span>
  SC</span>.</span>min_elt common_char</span> (* Min element must be the common character *)</span></span>
</span>
let</span> common_prio</span> string</span> =</span> string</span> |></span> common</span> |></span> prio</span></span>
let</span> result</span> list</span> =</span> List</span>.</span>fold_left</span> (</span>fun</span> sum string</span> -></span> sum</span> +</span> common_prio</span></span>
string</span>)</span> list</span></span></code></pre>
This clearly takes advantage of data structures in the standard library that I have not yet explored. (Obviously it's implementation of Set</code>, but also seq</a>): A seq</code> or sequence is type of list that is lazily evaluated, increasing performance but also allowing for things like conceptually infinite sequences. I guess sequences</code> are found in Haskell and F# (called a stream</code> in Scala), and can be best compared to Iterators in Rust. This sending me down a pretty deep rabbit hole that could take up a whole if not several posts, learning about the use of lazy</code>, seq</code>, and iter</code>, and the differences between them.</p>
I think this highlights one of the hurdles of a new paradigm, is it forces you to almost hyper focus on trivial tasks, causing you to forget the bigger picture, and not reach for the tools you would in other languages.</p>
For the sake of keeping with the nature of the post: I thought I would try to implement the exact</em> same solution in Rust, mimicking it as closely as possible. (fn names and all)</p>
fn</span> prio</span>(</span>ch</span>:</span> char</span>)</span> -></span> i32</span> {</span></span>
    let</span> code</span> =</span> ch</span> as u32</span>;</span></span>
    if</span> code</span> >=</span> 97</span> {</span></span>
        (</span>code</span> -</span> 96</span>)</span> as i32</span> // a-z</span></span>
    }</span> else</span> {</span></span>
        (</span>code</span> -</span> 38</span>)</span> as i32</span> // A-Z</span></span>
    }</span></span>
}</span></span>
</span>
fn</span> common</span>(</span>string</span>: &</span>str</span>)</span> -></span> char</span> {</span></span>
    let</span> len</span> =</span> string</span>.</span>len</span>();</span></span>
    let</span> half_len</span> =</span> len</span> /</span> 2</span>;</span></span>
</span>
    let</span> first</span>:</span> HashSet</span><</span>_</span>></span> =</span> string</span>.</span>chars</span>()</span>.</span>take</span>(</span>half_len</span>)</span>.</span>collect</span>();</span></span>
    let</span> second</span>:</span> HashSet</span><</span>_</span>></span> =</span> string</span>.</span>chars</span>()</span>.</span>skip</span>(</span>half_len</span>)</span>.</span>take</span>(</span>half_len</span>)</span>.</span>collect</span>();</span></span>
</span>
    let</span> common_char</span>:</span> HashSet</span><</span>_</span>></span> =</span> first</span>.</span>intersection</span>(</span>&</span>second</span>)</span>.</span>cloned</span>()</span>.</span>collect</span>();</span></span>
</span>
    *</span>common_char</span>.</span>iter</span>()</span>.</span>next</span>()</span>.</span>unwrap</span>()</span> // Min element must be the common character</span></span>
}</span></span>
// To demonstrate '|>'</span></span>
fn</span> common_prio</span>(</span>string</span>: &</span>str</span>)</span> -></span> i32</span> {</span></span>
    prio</span>(</span>common</span>(</span>string</span>))</span></span>
}</span></span>
</span>
fn</span> result</span>(</span>list</span>:</span> Vec</span><</span>&</span>str</span>></span>)</span> -></span> i32</span> {</span></span>
    list</span>.</span>iter</span>()</span>.</span>fold</span>(</span>0</span>,</span> |</span>sum</span>,</span> string</span>|</span> sum</span> +</span> common_prio</span>(</span>string</span>))</span></span>
}</span></span>
</span>
fn</span> main</span>() {</span></span>
    let</span> strings</span> =</span> fs</span>::</span>read_to_string</span>(</span>"input"</span>)</span>.</span>unwrap</span>();</span></span>
    let</span> splitstr</span> =</span> strings</span>.</span>lines</span>()</span>.</span>collect</span>::</span><</span>Vec</span><</span>_</span>>></span>();</span></span>
    let</span> result_value</span> =</span> result</span>(</span>splitstr</span>);</span></span>
    println!</span>(</span>"Result: </span>{}</span>"</span>,</span> result_value</span>);</span></span>
}</span></span></code></pre>
Notice how awesome the |></code> operator is in OCaml (the "pipe" operator, as it's so appropriately called), and what the same thing looks like in Rust. (Really any C based lang, but here in Rust, and unless you were to implement trait methods and then chain them).</p>
Seeing the comparison really makes me appreciate the functional style even more. Big thank you to Yawar for the solution 👍.</p>


Hackers guide to the hackernews front page
Unknown — Fri, 10 Nov 2023 00:00:00 +0000
Humorous article, completely unrelated to, and written before, the others ended up actually on the front page.</p>
The Goal:</strong></h1>


To get a post on the front page of the infamous Orange site...</li>
</ul>
The Plan:</strong></h1>


There are four guaranteed strategies. No one knows why they work, but you may pick one, give it a shot, and tell me I'm wrong.</li>
</ul>
Option 1:</h3>

Put RISC-V</strong> in the title.</li>
</ul>

That's it. I don't care what else the title says, it could be announcing that you have the rear bumper of a 1971 Chevette for sale,
that shit is on a rocket-ship straight to number 1.</p>
</blockquote>
Option 2:</h3>


Go to wikipedia.com</code>, to the 'search' bar, and proceed to allow your cat to walk across on your keyboard several times back and forth..</p>
Do this until some semblance of an english word or phrase appears in the text-box, then press the Search button. Now, you may need to do this
multiple times because the first initial results might not be odd or eccentric enough.</p>
Perhaps try your other cat, or if you do not have another cat
in your household, go find a neighborhood cat, and entice him/her/they to traverse your keyboard.</p>
</li>
</ul>
I have heard that dvorak</code> keyboards may be superior for this, although I must admit I heard it from an Uber driver so I'm not going to risk my
internet reputation and tell all of you like it's a fact.. It's not like I read it on wikipedia article*</a>. We will have to ask ThePrimeagen to confirm this.</p>
The best way to provoke this behavior from cats, is to appear to the cat, that you are in an important 'zoom' meeting, or are otherwise occupied on your computer
doing some task that would be greatly impeded by said feline should they decide to go for a stroll down qwerty</em>-lane.</p>
You will know when the search results are right, because the topic will be so obscure, that you will sit back in awe of who could possibly have
thought that this information was relevant or important to anyone else, while still sounding like something that, if someone told you Columbia university
pays 14 million dollars a year in grant money to research, you wouldn't second guess it.</p>
Option 3:</h3>

Link to an Open source project/technology that is old enough at this point that you might even consider leaving it off your resume, but not so old that it gets caught up with the hundreds of posts every day that simply link to the clearly still-maintained website of some ancient technology from 1962; that somehow looks like the website too, was designed in the same time period, in preparation for the eventual release of the internet.</li>
</ul>
Some suggestions:</strong></p>
Blosxom</a>  (I shit you not, as I was google'ing for this, </p>
Zope</a></p>
CoffeeScript</a></p>
XHTML</a></p>
Apache Tomcat</a></p>
Option 4:</h3>

ShowHN:</strong> Build something in the upper left corner of a chart where complexity and difficulty is the Y axis and real-world usefulness is its X.</li>
</ul>
Some recommendations are:</p>


A common-lisp to vimscript-8 transpiler.</p>
</li>

Rewrite any of the coreutils in Javascript.</p>
</li>

A web framework in APL (or your choice of array language).</p>
</li>

Implement (at least some part of) the JVM</em> in a scripting language</p>
</li>

Implement either Flappy Bird</em> or Conways game of life</em> in your choice of:</p>


GNU Sed, Bash, elisp, etc. Just ask yourself, if Richard Stallman was going to implement flappy bird...</p>
</li>

The Typescript Typesystem</p>
</li>

Inside a computer you build inside a video-game (minecraft, etc)</p>
</li>

Dreamberd</p>
</li>
</ul>
</li>
</ul>
*You know, that for just a moment... you believed that it was a real article..</p>
Thanks for reading this, I had a laugh writing it.</p>


Ocaml: first impressions
Unknown — Thu, 09 Nov 2023 00:00:00 +0000
The Why</h4>
I'm somewhat of a language nerd to begin with, and it was Rust that originally got me interested in the whole functional paradigm.
Not due to it's lineage, but the heavy use of chained iterator methods in favor over traditional loops. This is one of the more intimidating hurdles
for newcomers to the language, but after getting used to it, rarely will you see anyone write a for</code> loop again.</p>
Rant:</strong> The argument can be made that what I'm talking about is more similar to Javascript-land where a bunch of pseudo-functional methods and terminology are
used, but it in reality is as far away from functional programming as useState()</code> would imply it is.
At least Rust has a few things to back it up, as at least a language that will support you should you desire to write pure functional code. I would argue that for beginners, Rust will simply force you to write functional, immutable code, whether they really understand it or otherwise, as the amount of .clone()</code>'ing required for a newcomer to compile a non-trivial program will practically almost guarantee that nothing was actually &mut</code>ated*.</p>
So this got me interested in watching some GOTO</em>, strangeloop</em>, and other conference talks about functional programming. Upon discovering @artemslab</a>
(definitely go check it out, absolutely fantastic</em> high quality, very in-depth content) on YouTube and watching some videos, and reading about and it's runtime integer representations and memory allocation, that really got me fascinated with the language. One of the programs that he demonstrates is a huge static analysis tool written in Ocaml, called 'Infer', that I had never heard of, but seeing the scale of this project and the adoption by some huge companies I was super impressed and even more excited to maybe build my next side project in Ocaml, if for nothing but the learning experience. (and possibly a little street cred in the Rust community, if I'm being honest ;)</p>
In the past I typically wouldn't hesitate to just sudo pacman -Sy </code> whatever compiler or interpreter I needed, to get started playing around with whatever new language I was interested in. However since getting full-time employment, on top of the existing OSS stuff, as well as doing some contracted gigs for the facility I'm in, I haven't had so much as a moment to spend on anything fun quite some time.</p>
Anyways, as usual I'm off track here. My point is that I thought</em> I had a good introduction to the world of FP, and I thought</em> since I'd heard that many of the
features and some of the syntax we love from Rust came directly from tha 'Caml itself, I thought I would love it immediately and thought perhaps picking it up would be at least as simple as picking up any other language and I could count on the same method I rely on.</p>
Well, I think anyone who has tried to jump from C based languages to the 'ml' or FP world, can probably tell you how foolish of a thought that was.</p>
My plan was to do the entire Advent of Code in Ocaml, then I was going to pick an ambitious project that I would otherwise use Rust for, and simply force myself
to use Ocaml. I find this is the best way to learn a language, is just take on a project that is significantly more ambitious than your current skill level, and by the time you finish it, you'll have not only saved yourself a bunch of wasted time on tutorials that aren't actually teaching you anything but how to copy from someones screen in a youtube video, but you'll have made something cool :D (hopefully).</p>
I have two gripes with the language (so far)..</h3>
Where are the types? 👀</h3>
I underestimated how much I would be thrown off by the fact that the types are heavily</em> inferred, and are almost never explicitly declared, from what I can tell. I found that the combination of this and the strange syntax really screws me up, as now I am having problems inferring my own</em> types at this point...</p>
When dealing with lots of recursive functions, you find yourself looking up often at the function signature to view the parameters, but the concise syntax makes it difficult to mentally parse quickly.</p>
Example, random code snippet from ocaml.org:</p>
  let</span> group</span> list sizes</span> =</span></span>
    let</span> initial</span> =</span> List</span>.</span>map</span> (</span>fun</span> size</span> -></span> size</span>,</span> []</span>)</span> sizes</span> in</span></span>
</span>
  let</span> prepend</span> p list</span> =</span></span>
    let</span> emit</span> l acc</span> =</span> l</span> ::</span> acc</span> in</span></span>
    let</span> rec</span> aux</span> emit acc</span> =</span> function</span></span>
      |</span> []</span> -></span> emit</span> []</span> acc</span></span>
      |</span> (</span>n</span>,</span> l</span>)</span> as</span> h</span> ::</span> t</span> -></span></span>
         let</span> acc</span> =</span> if</span> n</span> ></span> 0</span> then</span> emit</span> ((</span>n</span> -</span> 1</span>,</span> p</span> ::</span> l</span>)</span> ::</span> t</span>)</span> acc</span></span>
                   else</span> acc</span> in</span></span>
         aux</span> (</span>fun</span> l acc</span> -></span> emit</span> (</span>h</span> ::</span> l</span>)</span> acc</span>)</span> acc t</span></span>
    in</span></span>
    aux emit</span> []</span> list</span></span>
  in</span></span>
  let</span> rec</span> aux</span> =</span> function</span></span>
    |</span> []</span> -></span> [</span>initial</span>]</span></span>
    |</span> h</span> ::</span> t</span> -></span> List</span>.</span>concat_map</span> (</span>prepend h</span>) (</span>aux t</span>)</span></span>
  in</span></span>
  let</span> all</span> =</span> aux list</span> in</span></span>
  let</span> complete</span> =</span> List</span>.</span>filter</span> (</span>List</span>.</span>for_all</span> (</span>fun</span> (</span>x</span>,</span> _</span>)</span> -></span> x</span> =</span> 0</span>))</span> all</span> in</span></span>
    List</span>.</span>map</span> (</span>List</span>.</span>map snd</span>)</span> complete</span>;;</span></span>
val</span> group</span> :</span> '</span>a</span> list</span> -></span> int</span> list</span> -></span> '</span>a</span> list</span> list</span> list</span> =</span> <</span>fun</span>></span></span>
;;</span></span></code></pre>
Even when you learn what it actually going on here, and it makes immediate sense to you.. It must take a while before it becomes easier to read.</p>
Remember recursion? How about linked lists?</h3>
Two things every Computer science student learns, and then are immediately told to never use again*...</p>
Anything that can be done recursively can be done iteratively (and most optimizing compilers will replace your recursion anyway), and linked lists are slow and inefficient with modern CPU caches, and you should almost never use them (and if you're a web dev, well, you just wouldn't anyway).</p>
You could describe Ocaml as a language that relies heavily</strong> on both</em> of these things. This feels even more strange that it might seem it would, and I have
found myself stumbling on absolutely remedial tasks.</p>
Yes, there are libraries with other data structures, and there are in fact for</code> loops (and even while</code> loops), However I feel like I will end up learning more of the language and find that iterator methods will be used more so than any of these, much as in Rust.</p>
The Good:</h1>


Thought provoking. If you don't believe there is anything to gain from being forced to think about something you do every day in a completely different way... well I'm surprised you made it this far into the post.</p>
</li>

Awesome mixture of !significant-whitespace and !brackets. And how cool is the comment syntax? (* doesn't it just feel like.. middle-eastern-ish *);;</p>
</li>

Tooling: is way better than it has any right to be, considering the size of it's user-base and community.</p>
</li>

Great conversation starter, really cool and helpful community.</p>
</li>
</ul>
Lots more to learn for sure but these were my initial thoughts anyway, I do really want to like it so we will see how much that helps. I just definitely did not get the head-start I was expecting, but I don't think that's a negative thing it just means there is more to learn.</p>
*obviously I know this isn't true and an extreme exaggeration, but I thought it was funny. There's always someone who will 'well ackshually' you when
you think you've been completely clear.</p>
Keep in mind, this is simply my first impression. I will be continuing to learn and will be periodically writing some updates on how my feelings change.</h2>
Also, once again have to plug @artemslab</a>. Doesn't have NEARLY enough youtube subscribers for how good the content is.</p>


How I got here
Unknown — Tue, 07 Nov 2023 00:00:00 +0000
My story, and how this is all possible</p>
Introduction</h3>
My name is Preston Thorpe, I'm 31 years old and I've spent just under 10 years of my life in Prison (all for non-violent drug crimes.)
I am currently incarcerated at Mountain View Correctional Facility in Charleston, Maine. More importantly, far more importantly, I am a software developer,
open source maintainer, contributor and all around enthusiast. I love rust, linux, Neovim, Alacritty, Tmux, tiling window managers (DWM), and doing everything possible from the command line.</p>
Idk who will even end up reading this, but thats the whole idea of blogs right? Ideally maybe the Primeagen will read it at some point,
I feel like it's right up his alley. I have lots of respect for him, I really appreciate him sharing his own story and particularly the message
of working hard for what you want, and also to stay away from bullshit/drugs. He has actually been a pretty big inspiration for me in the tech world
and I've been following him for over a year now. So on the off-chance that you're reading this: Thanks for everything you do, you have been an enormous inspiration to me, and the only source of entertainment I've allowed myself for the last year+.</p>
Anyway...</strong></p>
The (relatively) short version: I was a stupid 17yr old computer geek, spending my highschool years in the '05-'09 warez and torrent scenes(r.i.p. oink.cd) and was kicked out of my parents house for being a stupid 17yr old like many kids are, but instead of ending up broke and homeless sleeping in my car or on couches (I did that for a bit too though), and coming back to Mom + Dad's apologetic, as I'm sure was their plan; I ended up stumbling head-first into the world of the pre silk-road/Dark-web internet wholesale drugs. This was a place where chemicals could be bought for pennies from sketchy companies in India, China, and there were vendors in Canada, Holland, Germany, etc that sold bulk of whatever you wanted all by email referral.</p>
So instead of coming back home broke and apologetic, I ended up pretty deep into this and soon was making tens of thousands of dollars a week, very much unapologetically. At age 20, I was arrested the first time. I was caught with MDMA coming in the mail from Vancouver, and some marijuana coming from california (the latter of which is what I am currently serving my time for right now).</p>
Prison:</strong></p>
I have found that I adapt pretty well to just about whatever situation I happen to be put in, Prison was no different. Although, these people were very different
from the hippie/raver/music festival people I dealt with on the street. These people were hardcore addicts, who many of were into crimes and a lifestyle that I
had no part of. The thing about prison is that it has its own sub-culture, it's own lingo, it's own set of morality.. and this is enforced by an extremely negative
group-think, strict mindset that we are criminals: we hate cops, we hate rats, pedophiles, thieves(only if you steal from other inmates, you can steal whatever
you want from any grandma on the street) and anyone that has ever "PC"'d (asked for Protective Custody due to fear of violence from another inmate(s)).
This mindset is strictly enforced on everyone living in population, and spending a few years around such negativity will change anyone. In prison, there is
nothing to do: except get tattoos, work out, and wait for some kind of drama to unfold so you can either watch something go down, or participate in it.</p>
A few years later, I left prison with $0 in my pocket (lawyers and commissary are expensive, and nobody pays you what they owe you when you come in),
to a rooming house with hallways that smelled like crack-smoke and were filled with parole officers and junkies. I was left with the difficult choice
of either living there and walking to a temp agency with hopes of making $10.50/hour doing manual labor (without an ID or social security card at this point),
or getting on a bus to NYC to see some associates, and coming back in a week or so with $15-25k in my pocket and living in comfy luxury hotels until
I could rent an apartment... I chose the latter: and obviously, was back in prison after a short 14 months of addiction and misery.</p>
I have been incarcerated now on this sentence since May of 2017, I came in with a terrible attitude, a terrible outlook on life, and no hope for my future.
I have spent almost 3 years all-together in 22-23hr solitary confinement over the years (10% of my life?), and at one point I had truly become one of the people
that looked at me when I first came to prison at age 20, and asked "Is this your first bid?", to which everyone replies "first? I"m not coming back here!",
to which there is always laughter from the older heads. Everyone says that, but everyone comes back...</p>
Growing up:</strong></p>
My poor behavior and the perceived 'influence' I had in the relatively small prison system that I came from, ended up getting me put on an inter-state transfer list
after 13mo in solitary confinement (for no disciplinary action.) "Safety and security of the facility" is the official reason it says on my classification
paperwork. Little did I know, this would be the single best thing that would ever happen to me.</p>
Because I had no disciplinary action, and I guess very some fortunate timing, I got transferred to Maine, which is not a state that I thought was even
an option as it does not have a reputation for taking 'problem' inmates from other states.</p>
The prison system in Maine is different from all others, as the inmate population is insulated from most of the street/prison gangs that generally run
things. Not having to deal with the Politics and the drama was refreshing, and a bit of a shock at first. Quickly I realized that my time could just be
spent on myself, trying to improve my life or just do what I wanted with my time. I began to focus my time on things like teaching myself about organic chemistry, and finance, options trading, and I began for the first time to realize that I might be capable of more than what I had resided on, and that I could end doing whatever I put my mind to.</p>
I had previously accepted my fate, as one must when you get a 10+ year sentence, otherwise you'll go insane constantly thinking about what you could have done different or what you could have been doing with your life. But I had accepted my fate, as a criminal. I had accepted the identity, and the lifestyle, and the misery, and the mindset that comes along with it.</p>
One day, I woke up and looked around me, and had what can only be described as an epiphany. All of a sudden, all the conversations I would hear about drugs
and war stories, all just sounded like absolute bullshit to me, and I was suddenly ashamed that I ever participated in it. From that point, I no longer
felt like it was okay or normal that I was locked up in a cage, and the acceptance and self-identity as being a 'criminal' no longer felt like something
I was okay with.</p>
When it was available to me, I enrolled in College through the University of Maine Augusta, and before classes even started, I was completely enthralled
with the idea of learning how to program again (it had been 15 years since I had done some PHP/Perl + simple websites, so not a lot had left over).
But wait.. Who else has an opportunity to spend 12+ hours a day learning something for years? With no other obligations or responsibilities? It all of a
sudden clicked, that I would dedicate the remainder of my time to learning everything I could about computer science, and teaching myself software development.
The next ~three years would be spent exactly this way, to the very day I write this. There is zero substitution for hard work, and I intend
on continuing to use every moment of my remaining sentence to improve myself and my skills.</p>
During the pandemic, staff were sympathetic to the residents here, who were unable to see loved ones, and unable to spend their time on the internet
when everything was shut down and we were locked in our rooms like they had. This brought about the internet being brought into a prison system,
for probably the first time in the country. Being technically inclined, led me the opportunity to work for the education department and I ended up
helping considerably in getting things working properly. And I still do.</p>
After helping with the network for some time and implementing some software solutions for the DOC, when I asked for permission to seek remote
employment, they obliged. I was one of the first people in the country to to be allowed to be on "remote work-release" from a medium security prison.</p>
I believe this highlights the fact that you cannot take opportunities that you are not given, and I would never</em>
be in the position I am now: a completely changed man, working full-time from my cell, with a career doing what I
love not only programming, but also helping improve the system.(details</a>); if this opportunity wasn't given to me in
the form of a laptop and limited internet access. This is an incredibly rare situation, Maine is the only</em> state in
the country where these particular opportunities exist at all (and for a limited number of residents, currently just
those enrolled in College, and there are many restrictions).</p>
I am very grateful for the opportunity, but I recognize that this is very much the exception and not the rule, and the
success of the Maine model of corrections should highlight the absolutely embarrassing lack of opportunities in the rest
of the system, to do anything but become a bitter, broke criminal; deprived of not just your freedom, family, financial
security and reputation, but also of your self-identity as someone worth investing in changing. We need to do better as a
society, and understand that, yes, there are people in the system that deserve this kind of punishment, but a large majority
of our prison population are just regular people... non-violent drug offenders like myself. There are plenty more, like me,
that are capable of being responsible, productive, tax paying members of society if given the opportunity, but you cannot
expect anyone to change when you just lock them up in a cage with a bunch of other criminals where there is a subculture of
endless negativity.</p>
If you are interested in helping support the education of incarcerated individuals, please consider donating to a company like the one I currently work for, Unlocked Labs</a> or similar non-profits like The Last Mile</a> that are working to improve the system of mass-incarceration and provide opportunities for the incarcerated to learn and grow.</p>
Thanks for taking the time to read this...</p>
-P</p>

EDIT: Wow.. I definitely wasn't expecting this to get to the front page of HN. In light of this, and some harsh, but undoubtedly fair comments that were made(as well as some completely ignorant ones). I would like to add some clarification on a few things.</p>
</blockquote>
I have been an opioid addict for over 10 years, and to this day prescribed suboxone, being treated for OUD as well as recently having been cured of HEP C from years of drug abuse. As an addict and someone that has seen this lifestyle, and drugs in particular claim the lives of countless friends (2 of them extremely close to me. RIP Mark Bochner, Scott Young.) I am absolutely</strong>, and forever will be ashamed of any participation I had in that lifestyle and for perpetuating the scourge of addiction. I would like to apologize publicly to anyone that has been affected by this, and there is no excuse for my actions, other than the same (terrible) one I give am forced to give for all of them. That I was young, ignorant, addicted, and I held absolutely no value for own life, nor lives of others. I am genuinely sorry and ashamed on a level that could never be properly communicated, and I understand that although I will have to live with that for the rest of my life, many others will not get that chance.</p>
In the system, people are very open and accepting of their fate. Jokes are often made about how short someone was 'on the street', or bets will be placed how quickly someone will be back (always in months). Everyone speaks about release as if it is temporary, like a vacation from their accepted fate of prison or death. This is contagious, and when someone loses all value for their own life in acceptance of that fate, it creates a population of miserable, hopeless people with nothing to lose. As stated in the original post: I did not value my own life, nor did I believe I was capable of anything more than the life I had resided to.</p>

Update 10/03/24: it's been almost a year since I posted this.</p>
</blockquote>
In May I was promoted to Senior Developer + tech lead for the product team, so it's been very exciting leading development with a team of 6 going on 7 people now, and all of the other responsibilities that come along with it. I have had the chance to conduct technical interviews for two developer positions we filled, and I have spent the past few months teaching/mentoring another resident in a different facility in Maine who I was put in contact with. I had heard he matched my level of discipline by putting in 14 hour days learning every single day. That has been the most rewarding thing I've done, because he put in a tremendous amount of work and we are now hiring him this week and I am fortunate enough to see and have directly contributed to the same life altering changes that have happened to me over the past few years.</p>
I cannot recommend enough that devs volunteer some time teaching or mentoring, not only do you learn a ton yourself but it is really exciting seeing how efficiently you can teach someone else the things that you had to meticulously piece together yourself. Optimizing someone else's learning path scratches the same automate/optimize itch we devs get naturally, and the feeling that you are positively impacting someone's life is a great feeling.</p>

Update 06/16/2025:</p>
</blockquote>
Update available on Turso</a>'s blog</p>

pthorpe92.dev

The absolute beginners guide to databasemaxxing

You can do better

Rewriting SQLite: Adding a new join algorithm

Join Algorithms: A Brief Tour</h2> Most databases implement a small family of join algorithms:</p>

How Nested Loop Joins Work (and how SQLite makes them fast)</h3> The nested loop join really is as simple as it sounds. The outer loop iterates over one table, and for each row, the inner loop searches the other for matches.</p> SQLite does add several layers of optimization on top of this:</p>

Checkpointing performance in Write-Ahead-Logs

Learn systems, not langauges

whois pthorpe92.dev

Working on databases from prison, How I got here, part 2.

Calling Rust from cursed Go

The program in question:</h3> ==============================</p>

Calling Rust from cursed Go</h2> ==============================</p> Purego makes registering foreign symbols very simple. When the driver is registered, I dlopen()</code> the library and</p>

Magic isn't real

PHP/Laravel: Why we rewrote our app in Go

Gratitude

OCaml + Rust: some comparisons

Because I am an Ocaml n00b</h4> We will break this up into 3 simple functions (and a main), and try</em> to solve the problem in relatively the same steps in both</em> languages, so we can give them a fair</code> comparison of basic operations.</p>

Hackers guide to the hackernews front page

Ocaml: first impressions

How I got here