1
0
Fork 0

175 Commits (7e05f42fe39159f20d684ba2aabc2934911eae1a)

Author SHA1 Message Date
Philip O'Toole 102869eb6d Unit test Provide() on the Store 1 year ago
Philip O'Toole 1794724ef0 Remove unused code and fix error handling 1 year ago
Philip O'Toole ff001bc5a8 Migrate to a Protos data model for Notify Requests 2 years ago
Philip O'Toole a782513a11 Migrate to a join-request proto 2 years ago
Philip O'Toole 5636febf9a Add IsVoter() and add to status/ output 2 years ago
Philip O'Toole 90b4dc0745 Silently ignore unchanged node self-joins 2 years ago
Philip O'Toole 4131cdb743 Unit test stepdown on shutdown 2 years ago
Philip O'Toole 98575d727b Start adding remote remove node
More testing, including unit testing, required.
2 years ago
Philip O'Toole e74bfc182a Use new Encoder everywhere 2 years ago
Philip O'Toole edec152ec9 Fix Store unit test 2 years ago
Philip O'Toole e158ae8a72 Start HTTP server ASAP 2 years ago
Eng Zer Jun da103c9b3e
test: fix failing Test_OpenStoreSingleNode on Windows
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2 years ago
Eng Zer Jun 6444926be7
test: fix failing Test_SingleNodeSnapshotInMem on Windows
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2 years ago
Eng Zer Jun 6969165330
test: fix failing Test_SingleNodeSnapshotOnDisk on Windows
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2 years ago
Eng Zer Jun 5141eff5fc
test: use `T.TempDir` to create temporary test directory
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2 years ago
Philip O'Toole 47d79c859a Log database backup time 2 years ago
Philip O'Toole 31bba4c577 Move to Backup proto model 2 years ago
Philip O'Toole 67a6042afb More style changes 2 years ago
Philip O'Toole 7bfc7afef0 Minor style changes to test logic 2 years ago
Abirdcfly 1b033f3b58 fix minor unreachable code caused by t.Fatal
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2 years ago
Philip O'Toole 9dbf590634 Better unit test 2 years ago
Philip O'Toole dc2b9f5b6f Ignore freshness when serving queries on Leader
It doesn't make sense to check the last leader-contact time when the
node itself is the leader. Any read, when served by the leader, is going
to be within the freshness bound.

Fixes https://github.com/rqlite/rqlite/issues/1048
2 years ago
Philip O'Toole da8d4fcbe9 Unit test that pre-existing data is gone 2 years ago
Philip O'Toole 89b132ac26 Fix Store-level unit tests 2 years ago
Philip O'Toole 9783ef24b1 Switch to error on self-join attempt 3 years ago
Philip O'Toole e4fed0cee3
Add bootstrap-expect (#974)
Add cluster-boostrap
3 years ago
Philip O'Toole 4aea326959
Consul and etcd autoclustering (#957)
This PR introduces new node-discovery integration with Consul and etcd. By using one of those systems with rqlite, automatic clustering of rqlite is much easier.
3 years ago
Philip O'Toole 63b7d24814 Init rand seed once at Store level 3 years ago
Philip O'Toole 24a5dc8c4b Much better random file names 3 years ago
Philip O'Toole 58adcb87c9 Fix Store unit tests 3 years ago
Philip O'Toole f3a9b79b34 Close obsolete Listeners during Store testing 3 years ago
Philip O'Toole 05534cc94c Don't use Windows file paths as IDs 3 years ago
Philip O'Toole 7093cb25d3 Fix race in test 3 years ago
Philip O'Toole 9fb114c07f Recover test after snapshot 3 years ago
Philip O'Toole 519a91e635 Actual recovery node test passes on Linux 3 years ago
Philip O'Toole f9bcde0722 Back to no-change test 3 years ago
Philip O'Toole e2e199633a
Fix JSON string in unit test 3 years ago
Philip O'Toole c111df7ed5 Simpler node-recovery test 3 years ago
Philip O'Toole 92d5bc8b64 Unit test fixes (?) 3 years ago
Philip O'Toole 1b158d09fb Simple unit test of node-recovery 3 years ago
Philip O'Toole dd71c07c73 ABL 3 years ago
Philip O'Toole 00f4c6bc35 Wait, or timeout, for LeaderID to be set
Address https://ci.appveyor.com/project/otoolep/rqlite/builds/40607771
failures.
3 years ago
Philip O'Toole 5c63ff9bff Check for WaitForLeader errors 3 years ago
Philip O'Toole c9fdd7d05d Ensure SQLite file was actually created 3 years ago
Philip O'Toole 105a894fe5 Support explicit SQLite path at Store level 3 years ago
Philip O'Toole 9aecafedbe Cluster client resolves advertised local address
Also, improve HTTP stats.
3 years ago
Philip O'Toole 52ef9836ed Minor store-level test cleanup 3 years ago
Philip O'Toole 3580e341a8 Store-level testing passes 3 years ago
Philip O'Toole 916342395e Add Store-level FK constraint test 3 years ago
Philip O'Toole fc76327b46 Use FSM index in tests 3 years ago
Philip O'Toole b8d72f3fc4 New DB and FSM indexes to track state 3 years ago
Philip O'Toole e0da96dd55 Remove support for specifying SQLite DSN
This is a breaking change. However this feature hasn't been tested and
allows end-users to break the system too easily. Low-level control over
the SQLite database is better done with PRAGMA commands where possible.
3 years ago
Philip O'Toole 503825dbaf Remove Abort functionality
No longer relevant since each Exec and Query gets its own connection.
3 years ago
Philip O'Toole f939d33fe5 Remove Metadata concept from system
System testing passes.
3 years ago
Philip O'Toole b2bcdb584d Tweak Store API 3 years ago
Philip O'Toole ca1f8e1752 Whitebox testing of Noops on Store 4 years ago
Philip O'Toole aa2fec4a1f More useful store test helper functions 4 years ago
Philip O'Toole 9b8016e079 Simple Noop unit test 4 years ago
Philip O'Toole 824dab5536 Doesn't work because Store can't be re-opened
Due to network issues.
4 years ago
Philip O'Toole fa43fd4dc0 Enhance in-mem snapshot unit test 4 years ago
Philip O'Toole 6d77d904e7
Compress SQLite database in Raft snapshot
Before Raft truncates the log, it needs to retrieve a snapshot of the SQLite database to represent the log entries that are removed during that truncation. Raft then writes that snapshot to disk. To reduce both disk IO and disk space this change compresses that SQLite snapshot before it is written to disk, and decompresses it during a restore operation.

This change also deals with older SQLite snapshots, which were not compressed.
4 years ago
Philip O'Toole 8c2298941a ABL 4 years ago
Philip O'Toole 59360b4792 ABL 4 years ago
Philip O'Toole 6575de779d
Use Protobuf for encoding Raft Log commands
This PR changes Raft Log Entry encoding from JSON to Protobuf. Furthermore, larger Raft commands (which can result from batching SQL statements, or individually long SQL statements) are compressed before encoding.

This primary reason for this change is to reduce IO load since that is one of the largest performance bottlenecks. It will also reduce internode traffic.

Legacy JSON-encoded commands are still handled by this code, so this change is backwards-compatible with previous releases in the v5 series.
4 years ago
Philip O'Toole c4b4cf2a51 Perform more testing of queries 4 years ago
Philip O'Toole 212d4e5c86 Store and Raft layer support parameterized queires 4 years ago
Philip O'Toole e6e8ef36d2 Fix Snapshot test
If the snapshot count didn't change, the test didn't actually fail.
4 years ago
Philip O'Toole 9c4758ade7 Add comment 5 years ago
Philip O'Toole a1d0607fcd Sleep 1s so freshness interval passes 5 years ago
wangfenjin 3de2e675ba add strong level test
Change-Id: Ib91ffb22807993ffb7743b7fe7d0b41a62d0e696
5 years ago
wangfenjin 9c295ca704 check level before freshness
Change-Id: I6d736ad09eac9f003f087adaa9c1f51691fd5c69
5 years ago
Philip O'Toole abb7772105 Support leader freshness with None consistency
This change allows the read request to specify the maximum time the node
receiving the request may have last heard from the cluster leader. It
only applies to a read consistency selection of None.
5 years ago
Philip O'Toole 81f5f6faf1 Add support for non-voting nodes
A non-voting node doesn't participate in Raft consensus, but does
subscribe to the committed log entries originating with the leader.
This means a non-voting node keeps up-to-date with the state machine,
without impacting write-latency. These non-voting nodes can provide
read scalability for the cluster.
5 years ago
Philip O'Toole a23af88235 Write backup and dump files to a writer
This avoids making an extra copy in RAM. Port of original PR at
https://github.com/rqlite/rqlite/pull/461/.
5 years ago
Philip O'Toole f57ace7da2 Broadcast Store meta via standard consensus
With this change the cluster metadata (arbitrary key-value data associated with each node) is now broadcast across the cluster using the standard consensus mechanism. Specifically the use case for this metadata is to allow all nodes know the HTTP API address of all other nodes, for the purpose of redirecting requests to the leader.

This change removed the need for multiplexing two logical connections
over the single Raft TCP connection, which greatly simplifies the
networking code generally.

Original PR https://github.com/rqlite/rqlite/pull/434
5 years ago
Philip O'Toole e376812907 Add method to Store that returns leader ID
With the advent of Raft IDs, this distinction matters.
5 years ago
Philip O'Toole 8a0e1ce8f0 Move to Hashicorp Raft v1 5 years ago
Philip O'Toole 0b8cd8536f Remove apparently duplicated code 5 years ago
Philip O'Toole 0471877f26 Abort transaction if load operation fails 5 years ago
Philip O'Toole d47808a26b Add expvar stats to Store 5 years ago
Philip O'Toole 2a1b6c65c3 New QueryRequest and ExecuteRequest types
The Query() and Execute() functions on the Store now take a complex type
that encapsulates the statements, and associated parameters. This will
make it easier to add more control parameters when making requests.
7 years ago
Philip O Toole 5ed6ef0ae8 Covert dumps to queries in HTTP service
There is no need for special logic at the Store level.
8 years ago
Philip O Toole e6586f41f2 Use single load for dump processing
Writing a sufficiently sophisicated parser is too much work, and unlikely to be successful soon. Instead this change simply loads the entire dump as one command, and allows the underlying SQLite support to parse it correctly. This will definitely work, but since the load goes over Raft, it may hit limits with regards to network transfer sizes. Right now any limitations in that area are unknown.

Therefore this functionality remains somewhat experimental.
8 years ago
Tycho Andersen 326ca757d7 add an API for using custom raft.PeerStore/log.Logger
This is useful in case the server needs to store other metadata (e.g. auth
data) along side the peer list.

The logging bit is handy in case something has its own logging framework
that it wants to use.

Signed-off-by: Tycho Andersen <tycho.andersen@canonical.com>
8 years ago
Philip O'Toole da21606644 Move chinook package to idiomatic testdata dir
https://github.com/golang/go/issues/14715
8 years ago
Philip O'Toole 634f55211c Unit test with Chinook database
https://chinookdatabase.codeplex.com/
8 years ago
Philip O'Toole eda120ffaa Correct unit tests 8 years ago
Philip O'Toole f87ee412e2 Correct unit tests
'PRAGMA foreign_keys=OFF' is no longer filtered.
8 years ago
Philip O'Toole 36368d6786 Start removing batch load 8 years ago
Philip O Toole b95af88367 Test SQL loading with blank lines 8 years ago
Philip O Toole 6ee9cdbbdf Start integration of SQL line parser 8 years ago
Philip O Toole 2daac8704b Support batching during load 8 years ago
Philip O Toole 38f9b6ff20 Test loading no commands 8 years ago
Philip O Toole ab743b1ead Test loading an empty file 8 years ago
Philip O Toole ccd48161f3 Unit test Store Load() 8 years ago
Philip O'Toole 7dd164915b Check database errors at store level 8 years ago
Benjamen Keroack b67e56196e add cluster state and leader functions to store 8 years ago
Philip O'Toole 47539cf8e1 Add RemoveNode() to Store 8 years ago
Philip O'Toole ef18cd5b17 Expose Raft Peers() on Store 8 years ago
Philip O Toole bf2047b45b 'go lint' fixes 8 years ago