fluidB
/
fluidb-old
1
0
Fork 0
Code Pull Requests Activity

Update README.md

Browse Source
master
Timothy Cyrus 9 years ago
parent 12de94fe59
commit 5d8cfd9cc7
1 changed files with 144 additions and 106 deletions
Show Stats Download Patch File Download Diff File

250
README.md
Unescape Escape View File

@ -1,4 +1,4 @@
rqlite [![Circle CI](https://circleci.com/gh/otoolep/rqlite/tree/master.svg?style=svg)](https://circleci.com/gh/otoolep/rqlite/tree/master) [![GoDoc](https://godoc.org/github.com/otoolep/rqlite?status.png)](https://godoc.org/github.com/otoolep/rqlite)
rqlite [![Circle CI](https://circleci.com/gh/otoolep/rqlite/tree/master.svg?style=svg)](https://circleci.com/gh/otoolep/rqlite/tree/master) [![GoDoc](https://godoc.org/github.com/otoolep/rqlite?status.svg)](https://godoc.org/github.com/otoolep/rqlite)
======
*Detailed background on rqlite can be found on [these blog posts](http://www.philipotoole.com/tag/rqlite/). Note that master represents 2.0 development (which is still in progress), with a new API and Raft consensus module. If you want to work with 1.0 rqlite, you can find it [here](https://github.com/otoolep/rqlite/releases/tag/v1.0).*
@ -12,24 +12,30 @@ rqlite gives you the functionality of a fault-tolerant, replicated relational da
Download and run rqlite like so (tested on 64-bit Kubuntu 14.04 and OSX):
mkdir rqlite # Or any directory of your choice.
cd rqlite/
export GOPATH=$PWD
go get -t github.com/otoolep/rqlite/...
$GOPATH/bin/rqlited ~/node.1
```bash
mkdir rqlite # Or any directory of your choice.
cd rqlite/
export GOPATH=$PWD
go get -t github.com/otoolep/rqlite/...
$GOPATH/bin/rqlited ~/node.1
```
This starts a rqlite server listening on localhost, port 4001. This single node automatically becomes the leader. To see all available command-line options, execute:
$GOPATH/bin/rqlited -h
```bash
$GOPATH/bin/rqlited -h
```
### Forming a Cluster
While not strictly necessary to run rqlite, running multiple nodes means the SQLite database is replicated.
Start a second and third node (so a majority can still form in the event of a single node failure) like so:
$GOPATH/bin/rqlited -http localhost:4003 -raft :4004 -join :4001 ~/node.2
$GOPATH/bin/rqlited -http localhost:4005 -raft :4006 -join :4001 ~/node.3
```bash
$GOPATH/bin/rqlited -http localhost:4003 -raft :4004 -join :4001 ~/node.2
$GOPATH/bin/rqlited -http localhost:4005 -raft :4006 -join :4001 ~/node.3
```
*(This assumes you've set `GOPATH` as in the above section.)*
Under each node will be an SQLite file, which should remain in consensus. You can create clusters of any size, but clusters of 3, 5, and 7 nodes are most practical.
@ -37,17 +43,19 @@ Under each node will be an SQLite file, which should remain in consensus. You ca
### Restarting a node
If a node needs to be restarted, perhaps because of failure, don't pass the `-join` option. Using the example nodes above, if node 2 needed to be restarted, do so as follows:
$GOPATH/bin/rqlited -http localhost:4005 -raft :4006 ~/node.3
```bash
$GOPATH/bin/rqlited -http localhost:4005 -raft :4006 ~/node.3
```
On restart it will rejoin the cluster and apply any changes to the local sqlite database that took place while it was down. Depending on the number of changes in the Raft log, restarts may take a little while.
### Vagrant
Alternatively you can use a [Vagrant](https://www.vagrantup.com/) environment. To do so, simply [install Vagrant](https://docs.vagrantup.com/v2/installation/index.html) on your machine, a virtualization system such as VirtualBox, and execute the following commands:
~~~bash
```bash
$ cd $GOPATH/src/github.com/otoolep/rqlite
$ CLUSTER_SIZE=3 vagrant up rqlite
~~~
```
This will start a Vagrant box and install rqlite with all required dependencies. This will form a cluster with `CLUSTER_SIZE` nodes.
@ -55,9 +63,9 @@ To execute queries against the cluster you can either ssh directly to the Vagran
To terminate the Vagrant box simply execute:
~~~bash
```bash
$ vagrant destroy rqlite
~~~
```
## Data API
rqlite exposes an HTTP API allowing the database to be modified such that the changes are replicated. Queries are also executed using the HTTP API, though the SQLite database could be queried directly. Modifications go through the Raft log, ensuring only changes committed by a quorum of rqlite nodes are actually executed against the SQLite database. Queries do not go through the Raft log, however, since they do not change the state of the database, and therefore do not need to be captured in the log.
@ -67,69 +75,81 @@ All responses from rqlite are in the form of JSON.
### Writing Data
To write data successfully to the database, you must create at least 1 table. To do this, perform a HTTP POST, with a `CREATE TABLE` SQL command encapsulated in a JSON array, in the body of the request. For example:
curl -XPOST localhost:4001/db/execute?pretty -H "Content-Type: application/json" -d '[
"CREATE TABLE foo (id integer not null primary key, name text)"
]'
```bash
curl -XPOST localhost:4001/db/execute?pretty -H "Content-Type: application/json" -d '[
"CREATE TABLE foo (id integer not null primary key, name text)"
]'
```
where `curl` is the [well known command-line tool](http://curl.haxx.se/).
To insert an entry into the database, execute a second SQL command:
curl -XPOST 'localhost:4001/db/execute?pretty' -H "Content-Type: application/json" -d '[
"INSERT INTO foo(name) VALUES(\"fiona\")"
]'
```bash
curl -XPOST 'localhost:4001/db/execute?pretty' -H "Content-Type: application/json" -d '[
"INSERT INTO foo(name) VALUES(\"fiona\")"
]'
```
The response is of the form:
{
"results": [
{
"last_insert_id": 1,
"rows_affected": 1,
"time": 0.00886
}
],
"time": 0.0152
}
```json
{
"results": [
{
"last_insert_id": 1,
"rows_affected": 1,
"time": 0.00886
}
],
"time": 0.0152
}
```
The use of the URL param `pretty` is optional, and results in pretty-printed JSON responses. Time is measured in seconds.
You can confirm that the data has been writen to the database by accessing the SQLite database directly.
$ sqlite3 ~/node.3/db.sqlite
SQLite version 3.7.15.2 2013-01-09 11:53:05
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select * from foo;
1|fiona
```bash
$ sqlite3 ~/node.3/db.sqlite
SQLite version 3.7.15.2 2013-01-09 11:53:05
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select * from foo;
1|fiona
```
Note that this is the SQLite file that is under `node 3`, which is not the node that accepted the `INSERT` operation.
### Bulk Updates
Bulk updates are supported. To execute multipe statements in one HTTP call, simply include the statements in the JSON array:
curl -XPOST 'localhost:4001/db/execute?pretty' -H "Content-Type: application/json" -d "[
\"INSERT INTO foo(name) VALUES('fiona')\",
\"INSERT INTO foo(name) VALUES('sinead')\"
]"
```bash
curl -XPOST 'localhost:4001/db/execute?pretty' -H "Content-Type: application/json" -d "[
\"INSERT INTO foo(name) VALUES('fiona')\",
\"INSERT INTO foo(name) VALUES('sinead')\"
]"
```
The response is of the form:
{
"results": [
{
"last_insert_id": 1,
"rows_affected": 1,
"time": 0.00759015
},
{
"last_insert_id": 2,
"rows_affected": 1,
"time": 0.00669015
}
],
"time": 0.869015
}
```json
{
"results": [
{
"last_insert_id": 1,
"rows_affected": 1,
"time": 0.00759015
},
{
"last_insert_id": 2,
"rows_affected": 1,
"time": 0.00669015
}
],
"time": 0.869015
}
```
A bulk update is contained within a single Raft log entry, so the network round-trips between nodes in the cluster are amortized over the bulk update. This should result in better throughput, if it is possible to use this kind of update.
@ -138,39 +158,45 @@ Querying data is easy. The most important thing to know is that, by default, que
For a single query simply perform a HTTP GET, setting the query statement as the query parameter `q`:
curl -G localhost:4001/db/query?pretty --data-urlencode 'q=SELECT * FROM foo'
```bash
curl -G localhost:4001/db/query?pretty --data-urlencode 'q=SELECT * FROM foo'
```
The response is of the form:
{
"results": [
{
"columns": [
"id",
"name"
],
"values": [
[
1,
"fiona"
],
[
2,
"sinead"
]
```json
{
"results": [
{
"columns": [
"id",
"name"
],
"values": [
[
1,
"fiona"
],
"time": 0.0150043
}
],
"time": 0.0220043
}
[
2,
"sinead"
]
],
"time": 0.0150043
}
],
"time": 0.0220043
}
```
The behaviour of rqlite when more than 1 query is passed via `q` is undefined. If you want to execute more than one query per HTTP request, perform a POST, and place the queries in the body of the request as a JSON array. For example:
curl -XPOST 'localhost:4001/db/query?pretty' -H "Content-Type: application/json" -d '[
"SELECT * FROM foo",
"SELECT * FROM bar"
]'
```bash
curl -XPOST 'localhost:4001/db/query?pretty' -H "Content-Type: application/json" -d '[
"SELECT * FROM foo",
"SELECT * FROM bar"
]'
```
Another approach is to read the database file directly via `sqlite3`, the command-line tool that comes with SQLite. As long as you can be sure the file you access is under the leader, the records returned will be accurate and up-to-date.
@ -186,17 +212,21 @@ Since queries do not involve consensus, why must they served by the leader? This
This is why, even though queries do not involve consensus, they must be processed by the leader. If you wish to disable the leader check, and let queries be served regardless of leader state, add `noleader` to the URL. For example:
curl -G 'localhost:4001/db/query?pretty&noleader' --data-urlencode 'q=SELECT * FROM foo'
```bash
curl -G 'localhost:4001/db/query?pretty&noleader' --data-urlencode 'q=SELECT * FROM foo'
```
Due to the way rqlite works, there is a very small window (milliseconds) where a node has been disposed as leader, but has not yet changed its internal state. Therefore, even with the leader check in place, there is a very small window of time where out-of-date results could be returned.
### Transactions
Transactions are supported. To execute statements within a transaction, add `transaction` to the URL. An example of the above operation executed within a transaction is shown below.
curl -XPOST 'localhost:4001/db/execute?pretty&transaction' -H "Content-Type: application/json" -d "[
\"INSERT INTO foo(name) VALUES('fiona')\",
\"INSERT INTO foo(name) VALUES('sinead')\"
]"
```bash
curl -XPOST 'localhost:4001/db/execute?pretty&transaction' -H "Content-Type: application/json" -d "[
\"INSERT INTO foo(name) VALUES('fiona')\",
\"INSERT INTO foo(name) VALUES('sinead')\"
]"
```
When a transaction takes place either both statements will succeed, or neither. Performance is *much, much* better if multiple SQL INSERTs or UPDATEs are executed via a transaction. Note the execution ceases the moment any single query results in an error.
@ -205,23 +235,27 @@ The behaviour of rqlite when using `BEGIN`, `COMMIT`, or `ROLLBACK` to control t
### Handling Errors
If an error occurs while processing a statement, it will be marked as such in the response. For example.
curl -XPOST 'localhost:4001/db/execute?pretty' -H "Content-Type: application/json" -d "[
\"INSERT INTO foo(name) VALUES('fiona')\",
\"INSERT INTO nonsense\"
]"
{
"results": [
{
"last_insert_id": 3,
"rows_affected": 1,
"time": 182.033
},
{
"error": "near \"nonsense\": syntax error"
}
],
"time": 2.478862
}
```bash
curl -XPOST 'localhost:4001/db/execute?pretty' -H "Content-Type: application/json" -d "[
\"INSERT INTO foo(name) VALUES('fiona')\",
\"INSERT INTO nonsense\"
]"
```
```json
{
"results": [
{
"last_insert_id": 3,
"rows_affected": 1,
"time": 182.033
},
{
"error": "near \"nonsense\": syntax error"
}
],
"time": 2.478862
}
```
## Performance
rqlite replicates SQLite for fault-tolerance. It does not replicate it for performance. In fact performance is reduced somewhat due to the network round-trips.
@ -241,14 +275,18 @@ Pass `-mem` to `rqlited` at start-up to enable an in-memory database.
## Status API
A status API exists, which dumps some basic diagnostic and statistical information, as well as basic information about the underlying Raft node. Assuming rqlite is started with default settings, rqlite status is available like so:
curl localhost:4001/status?pretty
```bash
curl localhost:4001/status?pretty
```
The use of the URL param `pretty` is optional, and results in pretty-printed JSON responses.
## Backups
rqlite supports hot-backing up a node as follows. Retrieve and write a consistent snapshot of the underlying SQLite database to a file like so:
curl localhost:4001/db/backup -o bak.sqlite3
```bash
curl localhost:4001/db/backup -o bak.sqlite3
```
The node can then be restored by loading this database file via `sqlite3` and executing `.dump`. You can then use the output of the dump to replay the entire database back into brand new node (or cluster), *with the exception* of `BEGIN TRANSACTION` and `COMMIT` commands. You should ignore those commands in the `.dump` output.

Write Preview
Loading…
Cancel
Save