With this change, rqlite nodes running in "on-disk" mode build the database in memory first, and move it to the disk just before the Raft system starts. This means that on-disk nodes now initialize almost as quickly as in-memory nodes.
rqlite used to work like this, but suffered a regression due to a change in how Hashicorp Raft worked. The manner it changed in was not public, so relying on it was always fragile.
This PR changes Raft Log Entry encoding from JSON to Protobuf. Furthermore, larger Raft commands (which can result from batching SQL statements, or individually long SQL statements) are compressed before encoding.
This primary reason for this change is to reduce IO load since that is one of the largest performance bottlenecks. It will also reduce internode traffic.
Legacy JSON-encoded commands are still handled by this code, so this change is backwards-compatible with previous releases in the v5 series.
A non-voting node doesn't participate in Raft consensus, but does
subscribe to the committed log entries originating with the leader.
This means a non-voting node keeps up-to-date with the state machine,
without impacting write-latency. These non-voting nodes can provide
read scalability for the cluster.
This results in significant duplicated code, but is easier to follow.
The previous code was buggy when it came to redirection handling.
Longer term tool needs to be rebuilt to use a proper Go SQL-compliant
package (yet to be written).
With this change the cluster metadata (arbitrary key-value data associated with each node) is now broadcast across the cluster using the standard consensus mechanism. Specifically the use case for this metadata is to allow all nodes know the HTTP API address of all other nodes, for the purpose of redirecting requests to the leader.
This change removed the need for multiplexing two logical connections
over the single Raft TCP connection, which greatly simplifies the
networking code generally.
Original PR https://github.com/rqlite/rqlite/pull/434
Philip O'Toole