1
0
Fork 0

311 Commits (bb2473ca7be306f6dc4d63c8a8365909b9af0d02)

Author SHA1 Message Date
Philip O'Toole f03900b282 Log type of database on start-up 3 years ago
Philip O'Toole b83424484d Enhance SQLite database status 3 years ago
Philip O'Toole 6edff5ba24 Deal with SQLite connection "caching"?
https://github.com/mattn/go-sqlite3/issues/966
3 years ago
Philip O'Toole 6dc9583dab Implement concurrent reads and writes
Disk doesn't support it yet, as it has not moved to WAL mode. Perhaps
wait for next release to do that.
3 years ago
Philip O'Toole e0da96dd55 Remove support for specifying SQLite DSN
This is a breaking change. However this feature hasn't been tested and
allows end-users to break the system too easily. Low-level control over
the SQLite database is better done with PRAGMA commands where possible.
3 years ago
Philip O'Toole 8693694d14 Add current SQLite journal mode to status output 3 years ago
Philip O'Toole 503825dbaf Remove Abort functionality
No longer relevant since each Exec and Query gets its own connection.
3 years ago
Philip O'Toole 561bb47a14 Port DB layer to sql/database
Empty statements now cause panic.
3 years ago
Philip O'Toole 5f5f7dbbea Sync use of DB connection
Possible, albeit simple, fix for https://github.com/rqlite/rqlite/issues/830
3 years ago
Philip O'Toole 67b0368288 It's create in memory, not open 3 years ago
Philip O'Toole 0a55f0bed1 It's create, not open 3 years ago
Philip O'Toole 01e2403015 Count various Join scenarios 3 years ago
Philip O'Toole 4591e2df35 Don't ignore -join even if previous state exists 3 years ago
Philip O'Toole f0417f2241 Correct typo in comment 3 years ago
Philip O'Toole 58b567c27a Only check non-voting nodes by default 3 years ago
Philip O'Toole 4fd5555c41 ABL 3 years ago
Philip O'Toole f939d33fe5 Remove Metadata concept from system
System testing passes.
3 years ago
Philip O'Toole b2bcdb584d Tweak Store API 3 years ago
Philip O'Toole b579e5c59f Shut down Raft first 4 years ago
Philip O'Toole 1dc156b965 Add total Raft directory size to Store stats 4 years ago
Philip O'Toole fd4b76a419 Convert Raft stats to numbers where possible 4 years ago
Philip O'Toole 125074ab6e Only shutdown BoltDB when Raft is done with it 4 years ago
Philip O'Toole 3ff13cda74 More system testing 4 years ago
Philip O'Toole 51c745ea23 More in-mem and on-disk system test 4 years ago
Philip O'Toole ae16a65d8d Handle snapshots with database data
This shouldn't happen normally, but technically could.
4 years ago
Philip O'Toole f8e4795d44 Merge branch 'master' into test-snap-restore-more 4 years ago
Philip O'Toole c2f40500a2 Close Bolt on Store close 4 years ago
Philip O'Toole e4c60bed7a Merge branch 'master' into test-snap-restore-more 4 years ago
Philip O'Toole ca1f8e1752 Whitebox testing of Noops on Store 4 years ago
Philip O'Toole aa2fec4a1f More useful store test helper functions 4 years ago
Philip O'Toole 0b84667b45 Fix on-disk create in Apply 4 years ago
Philip O'Toole a8559b49e8
Merge pull request #754 from rqlite/noop-store-test
Support Noop commands in Raft Log
4 years ago
Philip O'Toole 9b8016e079 Simple Noop unit test 4 years ago
Philip O'Toole 824dab5536 Doesn't work because Store can't be re-opened
Due to network issues.
4 years ago
Philip O'Toole d31105e0f0
Add missing return 4 years ago
Philip O'Toole ed080fe9d0 Close database before Restore 4 years ago
Philip O'Toole 9c53b0f8b7 Consolidate database creation functions 4 years ago
Philip O'Toole c782b43f2d
Build on-disk databases in-memory first
With this change, rqlite nodes running in "on-disk" mode build the database in memory first, and move it to the disk just before the Raft system starts. This means that on-disk nodes now initialize almost as quickly as in-memory nodes.
4 years ago
Philip O'Toole 4551bd30da Add LastCommandIndex 4 years ago
Philip O'Toole 33aa4522cf Better timing from first to last log 4 years ago
Philip O'Toole a4185172da Time snapshot, restore, and startup times 4 years ago
Philip O'Toole 80e0db9f52 Make logic easier to read 4 years ago
Philip O'Toole 9ea54b608f Use DeserializeInMemoryWithDSN for in-mem restore 4 years ago
Philip O'Toole fa43fd4dc0 Enhance in-mem snapshot unit test 4 years ago
Philip O'Toole ef1f0f1d57 Sync queries and restores 4 years ago
Philip O'Toole 0f1352fb01 Remove duplicate pre-restoration Close() 4 years ago
Philip O'Toole ce7a3a6912 Merge branch 'master' into no-disk-during-restore 4 years ago
Philip O'Toole 0b3a41bd3f
Clearer comment 4 years ago
Philip O'Toole 535d690d7c Tigthen up Store-level locking
Previous locking didn't look quite correct, and with the advent of the
Serialize functionality, it needs to be more precise.
4 years ago
Philip O'Toole 0d0bb3f44a Close database before restoration
This change also adds better logging, so the restoration process can be
better observed.
4 years ago
Philip O'Toole eb01e8dc9d Skip temp file when restoring in-memory database
This saves a read and write to disk, the size of the SQLite database.
4 years ago
Philip O'Toole db64f4c923 Don't use temp file during database snapshot
Snapshotting the database does not now need to write to disk, and copy
the SQLite database straight to RAM, for handoff to the Raft subsystem.
4 years ago
Philip O'Toole 93a78f09a0 Add call to Serialize function 4 years ago
Philip O'Toole d560356d8d Fix spelling mistakes 4 years ago
Philip O'Toole 8576ed3cfd Remove unused byte slice address 4 years ago
Philip O'Toole bb31aef20a ABL 4 years ago
Philip O'Toole 5295a0e6b1 Trivial cleanups 4 years ago
Philip O'Toole 9b0852e711 Implement policy over trailing logs
Don't allow control over trailing logs directly, just implement a policy
that scales the number by 1.25 relative to Snapshot threshold. Without
doing this it may be that setting the snapshot threshold lower doesn't
actually affect SQLite start up times, due to log application. Perhaps
this really needs exposure at the command line, but the command line is
starting to get complicated.
4 years ago
Philip O'Toole 2e6a93faa3 Convenience function to get both indexes at once 4 years ago
Philip O'Toole 8e11534139 Support getting log first and last index 4 years ago
Philip O'Toole 8cb88d7744 Count number of legacy commands unmarshaled 4 years ago
Philip O'Toole 4f0c3092b3 Better command decoding panic message 4 years ago
Philip O'Toole 7780a0987b Simplify variable declaration 4 years ago
Philip O'Toole e52b4c58c3 Factor out binary value functions 4 years ago
Philip O'Toole 011bc00784
Add SQLite database page-centric size to status output 4 years ago
Philip O'Toole 6d77d904e7
Compress SQLite database in Raft snapshot
Before Raft truncates the log, it needs to retrieve a snapshot of the SQLite database to represent the log entries that are removed during that truncation. Raft then writes that snapshot to disk. To reduce both disk IO and disk space this change compresses that SQLite snapshot before it is written to disk, and decompresses it during a restore operation.

This change also deals with older SQLite snapshots, which were not compressed.
4 years ago
Philip O'Toole d5322a1909
Ignore join request if node already part of cluster (#711)
rqlite used to work like this, but suffered a regression due to a change in how Hashicorp Raft worked. The manner it changed in was not public, so relying on it was always fragile.
4 years ago
Philip O'Toole 8c2298941a ABL 4 years ago
Philip O'Toole 59360b4792 ABL 4 years ago
Philip O'Toole 6575de779d
Use Protobuf for encoding Raft Log commands
This PR changes Raft Log Entry encoding from JSON to Protobuf. Furthermore, larger Raft commands (which can result from batching SQL statements, or individually long SQL statements) are compressed before encoding.

This primary reason for this change is to reduce IO load since that is one of the largest performance bottlenecks. It will also reduce internode traffic.

Legacy JSON-encoded commands are still handled by this code, so this change is backwards-compatible with previous releases in the v5 series.
4 years ago
Philip O'Toole b4867713e4 Better variable name, Query -> SQL 4 years ago
Philip O'Toole b6ebdf3ab1 Support setting Leader Lease timeout 4 years ago
Philip O'Toole c4b4cf2a51 Perform more testing of queries 4 years ago
Philip O'Toole 8a4a5c4901
Improve comments re opening of Store 4 years ago
Philip O'Toole f961bb4e44 Fix typos and comments 4 years ago
Philip O'Toole 1480d8898f Handle Raft commands missing Parameterized values
This allows this code to run with Raft logs from previous versions.
4 years ago
Philip O'Toole ac862e4091 Add system test for parameterized statements
Fix some bugs along the way too.
4 years ago
Philip O'Toole 212d4e5c86 Store and Raft layer support parameterized queires 4 years ago
Philip O'Toole d840adc2a9 Convert Store to use new DB Statements 4 years ago
Philip O'Toole f81ef9024e Add Raft log size on disk to status 4 years ago
Philip O'Toole e6e8ef36d2 Fix Snapshot test
If the snapshot count didn't change, the test didn't actually fail.
4 years ago
Philip O'Toole c4a89ede8d Add helper function for Raft log size 5 years ago
Philip O'Toole 487c0e3f8c Set Snapshot Interval to non-zero value
Snapshotting was *never* functional due to the zero interval value.
5 years ago
Philip O'Toole 9c4758ade7 Add comment 5 years ago
Philip O'Toole a1d0607fcd Sleep 1s so freshness interval passes 5 years ago
wangfenjin 3de2e675ba add strong level test
Change-Id: Ib91ffb22807993ffb7743b7fe7d0b41a62d0e696
5 years ago
wangfenjin 9c4151c37b move comment down
Change-Id: I2f8c435adab5e1c46da2e6b91205ad4df850decc
5 years ago
wangfenjin 9c295ca704 check level before freshness
Change-Id: I6d736ad09eac9f003f087adaa9c1f51691fd5c69
5 years ago
Philip O'Toole 536cac4fdf Allow log level control for Raft module
Default log level is INFO.
5 years ago
Philip O'Toole abb7772105 Support leader freshness with None consistency
This change allows the read request to specify the maximum time the node
receiving the request may have last heard from the cluster leader. It
only applies to a read consistency selection of None.
5 years ago
Philip O'Toole 81f5f6faf1 Add support for non-voting nodes
A non-voting node doesn't participate in Raft consensus, but does
subscribe to the committed log entries originating with the leader.
This means a non-voting node keeps up-to-date with the state machine,
without impacting write-latency. These non-voting nodes can provide
read scalability for the cluster.
5 years ago
Philip O'Toole 9d595e2ead ABL 5 years ago
Philip O'Toole e0cf01cefc Protect against possible nil pointer 5 years ago
Philip O'Toole a23af88235 Write backup and dump files to a writer
This avoids making an extra copy in RAM. Port of original PR at
https://github.com/rqlite/rqlite/pull/461/.
5 years ago
Philip O'Toole f57ace7da2 Broadcast Store meta via standard consensus
With this change the cluster metadata (arbitrary key-value data associated with each node) is now broadcast across the cluster using the standard consensus mechanism. Specifically the use case for this metadata is to allow all nodes know the HTTP API address of all other nodes, for the purpose of redirecting requests to the leader.

This change removed the need for multiplexing two logical connections
over the single Raft TCP connection, which greatly simplifies the
networking code generally.

Original PR https://github.com/rqlite/rqlite/pull/434
5 years ago
Philip O'Toole e376812907 Add method to Store that returns leader ID
With the advent of Raft IDs, this distinction matters.
5 years ago
Philip O'Toole 8a0e1ce8f0 Move to Hashicorp Raft v1 5 years ago
Philip O'Toole 7352c376c7
Don't ignore Join error, actually return it
Clearly a typo.
5 years ago
Philip O'Toole 085d2d23d8 ABL 5 years ago
Philip O'Toole 0b8cd8536f Remove apparently duplicated code 5 years ago