Eng Zer Jun
da103c9b3e
test: fix failing Test_OpenStoreSingleNode on Windows
...
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2 years ago
Eng Zer Jun
6444926be7
test: fix failing Test_SingleNodeSnapshotInMem on Windows
...
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2 years ago
Eng Zer Jun
6969165330
test: fix failing Test_SingleNodeSnapshotOnDisk on Windows
...
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2 years ago
Eng Zer Jun
5141eff5fc
test: use `T.TempDir` to create temporary test directory
...
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2 years ago
Philip O'Toole
0e8e266f96
Merge pull request #1090 from rqlite/fix-remote-load-errors
...
Fix remote-load error handling
2 years ago
Philip O'Toole
893fbb0249
More store-level load operation logging
2 years ago
Philip O'Toole
65cd556549
Fix remote-load error handling
...
Error from remote node was being dropped.
2 years ago
Philip O'Toole
5f33bffa0f
More logging and resolution checks
2 years ago
Philip O'Toole
b0f17b86c7
Notified node checks address resolution
...
If a non-resolvable address is passed to the Store, and then Store then
adds that to the config, it can result in a cluster that seems to have
insufficient members (since the remote node can't be contacted). This
may trigger a new leader election.
2 years ago
Philip O'Toole
47d79c859a
Log database backup time
2 years ago
Philip O'Toole
9fc6ea16e9
Even better defer sequence
2 years ago
Philip O'Toole
555883d1a2
Correct order of deferred calls
2 years ago
Philip O'Toole
54aec1ca33
Tighten up backup code
2 years ago
Philip O'Toole
31bba4c577
Move to Backup proto model
2 years ago
Philip O'Toole
67a6042afb
More style changes
2 years ago
Philip O'Toole
7bfc7afef0
Minor style changes to test logic
2 years ago
Abirdcfly
1b033f3b58
fix minor unreachable code caused by t.Fatal
...
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2 years ago
Philip O'Toole
9dbf590634
Better unit test
2 years ago
Philip O'Toole
dc2b9f5b6f
Ignore freshness when serving queries on Leader
...
It doesn't make sense to check the last leader-contact time when the
node itself is the leader. Any read, when served by the leader, is going
to be within the freshness bound.
Fixes https://github.com/rqlite/rqlite/issues/1048
2 years ago
Philip O'Toole
da8d4fcbe9
Unit test that pre-existing data is gone
2 years ago
Philip O'Toole
89b132ac26
Fix Store-level unit tests
2 years ago
Philip O'Toole
b03ed99eda
Update comment
2 years ago
Philip O'Toole
304124952c
Add Store level changes
...
Needs unit testing, and HTTP testing won't compile.
2 years ago
Philip O'Toole
2729a17279
Move some functions to standalone form
2 years ago
Philip O'Toole
9783ef24b1
Switch to error on self-join attempt
3 years ago
Philip O'Toole
a524ed4979
Ignore join request if node ID is same
3 years ago
Philip O'Toole
e4fed0cee3
Add bootstrap-expect ( #974 )
...
Add cluster-boostrap
3 years ago
Philip O'Toole
ad7986630d
Better command-line flags
3 years ago
Philip O'Toole
ce099087ee
Control whether bbolt sync freelist to disk
...
When true, skips syncing freelist to disk. This improves the database write performance under normal operation, but requires a full database re-sync during recovery.# Please enter the commit message for your changes.
3 years ago
Philip O'Toole
490f654f36
Move to etcd's fork of BoltDB
...
bboltdb is the version the Consul now runs.
3 years ago
Philip O'Toole
4aea326959
Consul and etcd autoclustering ( #957 )
...
This PR introduces new node-discovery integration with Consul and etcd. By using one of those systems with rqlite, automatic clustering of rqlite is much easier.
3 years ago
Philip O'Toole
63b7d24814
Init rand seed once at Store level
3 years ago
Philip O'Toole
24a5dc8c4b
Much better random file names
3 years ago
Philip O'Toole
58adcb87c9
Fix Store unit tests
3 years ago
Philip O'Toole
ca556044e3
Merge branch 'master' of github.com:rqlite/rqlite into refactor-main-bootstrap
3 years ago
Philip O'Toole
2ffb29394f
Fix typo
3 years ago
Philip O'Toole
f5c03d4b60
Try simple refactor of bootstrap
3 years ago
Philip O'Toole
769557db7e
Better variable name
3 years ago
chermehdi
3bfcbffc37
Fix typos and Grammar issues
3 years ago
Philip O'Toole
6509070ce8
More STRONG queries to deal with races
3 years ago
Philip O'Toole
f3a9b79b34
Close obsolete Listeners during Store testing
3 years ago
Philip O'Toole
33bee34f89
More Store restart testing
3 years ago
Philip O'Toole
d74c05c2db
Support disabling in-memory init of on-disk
3 years ago
Philip O'Toole
003301aee6
Minor fixes to log messages ( #920 )
...
Minor fixes to log messages
3 years ago
Philip O'Toole
0b621cbec8
More logging tweaks
3 years ago
Philip O'Toole
5ce51bc2ba
Better comments
3 years ago
Philip O'Toole
05534cc94c
Don't use Windows file paths as IDs
3 years ago
Philip O'Toole
f9f35bf95c
Dump file
3 years ago
Philip O'Toole
7093cb25d3
Fix race in test
3 years ago
Philip O'Toole
9fb114c07f
Recover test after snapshot
3 years ago
Philip O'Toole
519a91e635
Actual recovery node test passes on Linux
3 years ago
Philip O'Toole
f9bcde0722
Back to no-change test
3 years ago
Philip O'Toole
e2e199633a
Fix JSON string in unit test
3 years ago
Philip O'Toole
c111df7ed5
Simpler node-recovery test
3 years ago
Philip O'Toole
92d5bc8b64
Unit test fixes (?)
3 years ago
Philip O'Toole
1b158d09fb
Simple unit test of node-recovery
3 years ago
Philip O'Toole
1abd3007ca
Rename peers.json after successful recovery
3 years ago
Philip O'Toole
cc1de72796
Ensure raft/ exists inside data directory
3 years ago
Philip O'Toole
abccc6ced3
Better function name
3 years ago
Philip O'Toole
d9ad3c080a
Add untested recover-node functionality
3 years ago
Philip O'Toole
32239b5053
Move new function to earlier per convention
3 years ago
Philip O'Toole
4ab72a8194
Refactor to make recover-cluster code easier
3 years ago
Philip O'Toole
5dd4aeed1e
More Snapshot metrics
3 years ago
Philip O'Toole
38eef8ee82
Merge pull request #890 from rqlite/more-abl
...
ABL
3 years ago
Philip O'Toole
dd71c07c73
ABL
3 years ago
Philip O'Toole
0e34dc49ca
Add BoltDB stats
3 years ago
Philip O'Toole
d6be09ba65
Update Snapshot comment
3 years ago
Philip O'Toole
5d48a18950
Check if slice is empty in addition to being nil
3 years ago
Philip O'Toole
bb002bb81b
Block query transactions during serialize
3 years ago
Philip O'Toole
a6da8d474f
Ensure (any) directory for on-disk-path exists
3 years ago
Philip O'Toole
1f33dbb2cb
Fix up Store-open logging
3 years ago
Philip O'Toole
00f4c6bc35
Wait, or timeout, for LeaderID to be set
...
Address https://ci.appveyor.com/project/otoolep/rqlite/builds/40607771
failures.
3 years ago
Philip O'Toole
5c63ff9bff
Check for WaitForLeader errors
3 years ago
Philip O'Toole
c9fdd7d05d
Ensure SQLite file was actually created
3 years ago
Philip O'Toole
105a894fe5
Support explicit SQLite path at Store level
3 years ago
Philip O'Toole
a6346ba9d4
Fix unit test -- start muxes
3 years ago
Philip O'Toole
9aecafedbe
Cluster client resolves advertised local address
...
Also, improve HTTP stats.
3 years ago
Philip O'Toole
d85bb55270
Merge branch 'master' of github.com:rqlite/rqlite into snapshot-gauges
3 years ago
Philip O'Toole
dd692ecf4f
Add gauge-like metric for Snapshot times
3 years ago
Philip O'Toole
e5a94f3592
Merge branch 'master' of github.com:rqlite/rqlite into cluster-execute-queries
3 years ago
Philip O'Toole
9d87a15194
Small simplication
3 years ago
Philip O'Toole
43e60a0469
Perform leader check first for STRONG query
...
Don't wait until Raft layer returns it. There is a small chance of a
false leadership detection, but the cluster would have to be in flux
anyway for that to be the case. Execute does this already.
3 years ago
Philip O'Toole
52ef9836ed
Minor store-level test cleanup
3 years ago
Philip O'Toole
fd98114839
HTTP-level testing passes
3 years ago
Philip O'Toole
3580e341a8
Store-level testing passes
3 years ago
Philip O'Toole
916342395e
Add Store-level FK constraint test
3 years ago
Philip O'Toole
a8d02bc807
Better JSON rendering of DB conf object
3 years ago
Philip O'Toole
deb4dabf2e
Thread through foreign key constraint controls
3 years ago
Philip O'Toole
9ed32972b1
Start adding FK option support
3 years ago
Philip O'Toole
0536bc3c1c
ABL
3 years ago
Philip O'Toole
74d0780a5f
Merge pull request #845 from rqlite/sqlite-options
...
Add SQLite compile-time options to status output
3 years ago
Philip O'Toole
39d0f2ad4a
Merge branch 'master' of github.com:rqlite/rqlite into sqlite-options
3 years ago
Philip O'Toole
fc76327b46
Use FSM index in tests
3 years ago
Philip O'Toole
b8d72f3fc4
New DB and FSM indexes to track state
3 years ago
Philip O'Toole
ea723ee830
Ping databases on open to ensure full availability
3 years ago
Philip O'Toole
696bac655f
Merge branch 'master' of github.com:rqlite/rqlite
3 years ago
Philip O'Toole
f03900b282
Log type of database on start-up
3 years ago
Philip O'Toole
b83424484d
Enhance SQLite database status
3 years ago
Philip O'Toole
6edff5ba24
Deal with SQLite connection "caching"?
...
https://github.com/mattn/go-sqlite3/issues/966
3 years ago
Philip O'Toole
6dc9583dab
Implement concurrent reads and writes
...
Disk doesn't support it yet, as it has not moved to WAL mode. Perhaps
wait for next release to do that.
3 years ago
Philip O'Toole
e0da96dd55
Remove support for specifying SQLite DSN
...
This is a breaking change. However this feature hasn't been tested and
allows end-users to break the system too easily. Low-level control over
the SQLite database is better done with PRAGMA commands where possible.
3 years ago
Philip O'Toole
8693694d14
Add current SQLite journal mode to status output
3 years ago
Philip O'Toole
503825dbaf
Remove Abort functionality
...
No longer relevant since each Exec and Query gets its own connection.
3 years ago
Philip O'Toole
561bb47a14
Port DB layer to sql/database
...
Empty statements now cause panic.
3 years ago
Philip O'Toole
5f5f7dbbea
Sync use of DB connection
...
Possible, albeit simple, fix for https://github.com/rqlite/rqlite/issues/830
3 years ago
Philip O'Toole
67b0368288
It's create in memory, not open
3 years ago
Philip O'Toole
0a55f0bed1
It's create, not open
3 years ago
Philip O'Toole
01e2403015
Count various Join scenarios
3 years ago
Philip O'Toole
4591e2df35
Don't ignore -join even if previous state exists
3 years ago
Philip O'Toole
f0417f2241
Correct typo in comment
3 years ago
Philip O'Toole
58b567c27a
Only check non-voting nodes by default
3 years ago
Philip O'Toole
4fd5555c41
ABL
3 years ago
Philip O'Toole
f939d33fe5
Remove Metadata concept from system
...
System testing passes.
3 years ago
Philip O'Toole
b2bcdb584d
Tweak Store API
3 years ago
Philip O'Toole
b579e5c59f
Shut down Raft first
4 years ago
Philip O'Toole
1dc156b965
Add total Raft directory size to Store stats
4 years ago
Philip O'Toole
fd4b76a419
Convert Raft stats to numbers where possible
4 years ago
Philip O'Toole
125074ab6e
Only shutdown BoltDB when Raft is done with it
4 years ago
Philip O'Toole
3ff13cda74
More system testing
4 years ago
Philip O'Toole
51c745ea23
More in-mem and on-disk system test
4 years ago
Philip O'Toole
ae16a65d8d
Handle snapshots with database data
...
This shouldn't happen normally, but technically could.
4 years ago
Philip O'Toole
f8e4795d44
Merge branch 'master' into test-snap-restore-more
4 years ago
Philip O'Toole
c2f40500a2
Close Bolt on Store close
4 years ago
Philip O'Toole
e4c60bed7a
Merge branch 'master' into test-snap-restore-more
4 years ago
Philip O'Toole
ca1f8e1752
Whitebox testing of Noops on Store
4 years ago
Philip O'Toole
aa2fec4a1f
More useful store test helper functions
4 years ago
Philip O'Toole
0b84667b45
Fix on-disk create in Apply
4 years ago
Philip O'Toole
a8559b49e8
Merge pull request #754 from rqlite/noop-store-test
...
Support Noop commands in Raft Log
4 years ago
Philip O'Toole
9b8016e079
Simple Noop unit test
4 years ago
Philip O'Toole
824dab5536
Doesn't work because Store can't be re-opened
...
Due to network issues.
4 years ago
Philip O'Toole
d31105e0f0
Add missing return
4 years ago
Philip O'Toole
ed080fe9d0
Close database before Restore
4 years ago
Philip O'Toole
9c53b0f8b7
Consolidate database creation functions
4 years ago
Philip O'Toole
c782b43f2d
Build on-disk databases in-memory first
...
With this change, rqlite nodes running in "on-disk" mode build the database in memory first, and move it to the disk just before the Raft system starts. This means that on-disk nodes now initialize almost as quickly as in-memory nodes.
4 years ago
Philip O'Toole
4551bd30da
Add LastCommandIndex
4 years ago
Philip O'Toole
33aa4522cf
Better timing from first to last log
4 years ago
Philip O'Toole
a4185172da
Time snapshot, restore, and startup times
4 years ago
Philip O'Toole
80e0db9f52
Make logic easier to read
4 years ago
Philip O'Toole
9ea54b608f
Use DeserializeInMemoryWithDSN for in-mem restore
4 years ago
Philip O'Toole
fa43fd4dc0
Enhance in-mem snapshot unit test
4 years ago
Philip O'Toole
ef1f0f1d57
Sync queries and restores
4 years ago
Philip O'Toole
0f1352fb01
Remove duplicate pre-restoration Close()
4 years ago
Philip O'Toole
ce7a3a6912
Merge branch 'master' into no-disk-during-restore
4 years ago
Philip O'Toole
0b3a41bd3f
Clearer comment
4 years ago
Philip O'Toole
535d690d7c
Tigthen up Store-level locking
...
Previous locking didn't look quite correct, and with the advent of the
Serialize functionality, it needs to be more precise.
4 years ago
Philip O'Toole
0d0bb3f44a
Close database before restoration
...
This change also adds better logging, so the restoration process can be
better observed.
4 years ago
Philip O'Toole
eb01e8dc9d
Skip temp file when restoring in-memory database
...
This saves a read and write to disk, the size of the SQLite database.
4 years ago
Philip O'Toole
db64f4c923
Don't use temp file during database snapshot
...
Snapshotting the database does not now need to write to disk, and copy
the SQLite database straight to RAM, for handoff to the Raft subsystem.
4 years ago
Philip O'Toole
93a78f09a0
Add call to Serialize function
4 years ago
Philip O'Toole
d560356d8d
Fix spelling mistakes
4 years ago
Philip O'Toole
8576ed3cfd
Remove unused byte slice address
4 years ago
Philip O'Toole
bb31aef20a
ABL
4 years ago
Philip O'Toole
5295a0e6b1
Trivial cleanups
4 years ago
Philip O'Toole
9b0852e711
Implement policy over trailing logs
...
Don't allow control over trailing logs directly, just implement a policy
that scales the number by 1.25 relative to Snapshot threshold. Without
doing this it may be that setting the snapshot threshold lower doesn't
actually affect SQLite start up times, due to log application. Perhaps
this really needs exposure at the command line, but the command line is
starting to get complicated.
4 years ago
Philip O'Toole
2e6a93faa3
Convenience function to get both indexes at once
4 years ago
Philip O'Toole
8e11534139
Support getting log first and last index
4 years ago
Philip O'Toole
8cb88d7744
Count number of legacy commands unmarshaled
4 years ago
Philip O'Toole
4f0c3092b3
Better command decoding panic message
4 years ago
Philip O'Toole
7780a0987b
Simplify variable declaration
4 years ago
Philip O'Toole
e52b4c58c3
Factor out binary value functions
4 years ago
Philip O'Toole
011bc00784
Add SQLite database page-centric size to status output
4 years ago
Philip O'Toole
6d77d904e7
Compress SQLite database in Raft snapshot
...
Before Raft truncates the log, it needs to retrieve a snapshot of the SQLite database to represent the log entries that are removed during that truncation. Raft then writes that snapshot to disk. To reduce both disk IO and disk space this change compresses that SQLite snapshot before it is written to disk, and decompresses it during a restore operation.
This change also deals with older SQLite snapshots, which were not compressed.
4 years ago
Philip O'Toole
d5322a1909
Ignore join request if node already part of cluster ( #711 )
...
rqlite used to work like this, but suffered a regression due to a change in how Hashicorp Raft worked. The manner it changed in was not public, so relying on it was always fragile.
4 years ago
Philip O'Toole
8c2298941a
ABL
4 years ago
Philip O'Toole
59360b4792
ABL
4 years ago
Philip O'Toole
6575de779d
Use Protobuf for encoding Raft Log commands
...
This PR changes Raft Log Entry encoding from JSON to Protobuf. Furthermore, larger Raft commands (which can result from batching SQL statements, or individually long SQL statements) are compressed before encoding.
This primary reason for this change is to reduce IO load since that is one of the largest performance bottlenecks. It will also reduce internode traffic.
Legacy JSON-encoded commands are still handled by this code, so this change is backwards-compatible with previous releases in the v5 series.
4 years ago
Philip O'Toole
b4867713e4
Better variable name, Query -> SQL
4 years ago
Philip O'Toole
b6ebdf3ab1
Support setting Leader Lease timeout
4 years ago
Philip O'Toole
c4b4cf2a51
Perform more testing of queries
4 years ago
Philip O'Toole
8a4a5c4901
Improve comments re opening of Store
4 years ago
Philip O'Toole
f961bb4e44
Fix typos and comments
4 years ago
Philip O'Toole
1480d8898f
Handle Raft commands missing Parameterized values
...
This allows this code to run with Raft logs from previous versions.
4 years ago
Philip O'Toole
ac862e4091
Add system test for parameterized statements
...
Fix some bugs along the way too.
4 years ago
Philip O'Toole
212d4e5c86
Store and Raft layer support parameterized queires
4 years ago
Philip O'Toole
d840adc2a9
Convert Store to use new DB Statements
4 years ago
Philip O'Toole
f81ef9024e
Add Raft log size on disk to status
4 years ago
Philip O'Toole
e6e8ef36d2
Fix Snapshot test
...
If the snapshot count didn't change, the test didn't actually fail.
4 years ago
Philip O'Toole
c4a89ede8d
Add helper function for Raft log size
5 years ago
Philip O'Toole
487c0e3f8c
Set Snapshot Interval to non-zero value
...
Snapshotting was *never* functional due to the zero interval value.
5 years ago
Philip O'Toole
9c4758ade7
Add comment
5 years ago
Philip O'Toole
a1d0607fcd
Sleep 1s so freshness interval passes
5 years ago
wangfenjin
3de2e675ba
add strong level test
...
Change-Id: Ib91ffb22807993ffb7743b7fe7d0b41a62d0e696
5 years ago
wangfenjin
9c4151c37b
move comment down
...
Change-Id: I2f8c435adab5e1c46da2e6b91205ad4df850decc
5 years ago
wangfenjin
9c295ca704
check level before freshness
...
Change-Id: I6d736ad09eac9f003f087adaa9c1f51691fd5c69
5 years ago
Philip O'Toole
536cac4fdf
Allow log level control for Raft module
...
Default log level is INFO.
5 years ago
Philip O'Toole
abb7772105
Support leader freshness with None consistency
...
This change allows the read request to specify the maximum time the node
receiving the request may have last heard from the cluster leader. It
only applies to a read consistency selection of None.
5 years ago
Philip O'Toole
81f5f6faf1
Add support for non-voting nodes
...
A non-voting node doesn't participate in Raft consensus, but does
subscribe to the committed log entries originating with the leader.
This means a non-voting node keeps up-to-date with the state machine,
without impacting write-latency. These non-voting nodes can provide
read scalability for the cluster.
5 years ago
Philip O'Toole
9d595e2ead
ABL
5 years ago
Philip O'Toole
e0cf01cefc
Protect against possible nil pointer
5 years ago
Philip O'Toole
a23af88235
Write backup and dump files to a writer
...
This avoids making an extra copy in RAM. Port of original PR at
https://github.com/rqlite/rqlite/pull/461/ .
5 years ago
Philip O'Toole
f57ace7da2
Broadcast Store meta via standard consensus
...
With this change the cluster metadata (arbitrary key-value data associated with each node) is now broadcast across the cluster using the standard consensus mechanism. Specifically the use case for this metadata is to allow all nodes know the HTTP API address of all other nodes, for the purpose of redirecting requests to the leader.
This change removed the need for multiplexing two logical connections
over the single Raft TCP connection, which greatly simplifies the
networking code generally.
Original PR https://github.com/rqlite/rqlite/pull/434
5 years ago
Philip O'Toole
e376812907
Add method to Store that returns leader ID
...
With the advent of Raft IDs, this distinction matters.
5 years ago
Philip O'Toole
8a0e1ce8f0
Move to Hashicorp Raft v1
5 years ago
Philip O'Toole
7352c376c7
Don't ignore Join error, actually return it
...
Clearly a typo.
5 years ago
Philip O'Toole
085d2d23d8
ABL
5 years ago
Philip O'Toole
0b8cd8536f
Remove apparently duplicated code
5 years ago
Philip O'Toole
0e69be3949
Store layer supports generating SQL format backups
...
Port PR453.
5 years ago
Philip O'Toole
0471877f26
Abort transaction if load operation fails
5 years ago
Philip O'Toole
ccca1ecfa9
Fix DSN key in Store status output
5 years ago
Philip O'Toole
d47808a26b
Add expvar stats to Store
5 years ago
Philip O'Toole
2e91858e1e
Allow Raft election timeout to be set
6 years ago
Philip O'Toole
6937a377ae
Allow control of shutdown-on-remove
...
Defaults to leaving Raft run if the node is removed.
6 years ago
Philip O'Toole
efec4d4e42
Return redirect if remove on follower
...
Fixes issue #391 .
7 years ago
Philip O'Toole
b5b2cb21db
Log Store directory path on startup
7 years ago
Philip O Toole
6bbdc88c1d
ABL
...
Always be linting.
7 years ago
Philip O'Toole
2a1b6c65c3
New QueryRequest and ExecuteRequest types
...
The Query() and Execute() functions on the Store now take a complex type
that encapsulates the statements, and associated parameters. This will
make it easier to add more control parameters when making requests.
7 years ago
Philip O'Toole
303be73dcb
Add missing Store params to diagnostics
7 years ago
Philip O Toole
a59e23cae0
Comment some unit tests
8 years ago
Philip O'Toole
8f2d8e8aa0
Copy JSON peers decode functionality from Raft
...
The main rqlite code needs this logic to check if the Discovery Service
should even be accessed. This code makes assumptions about how the Raft
module works, which is subject to change. The new unit tests should
capture that.
8 years ago
Philip O'Toole
2617962f5f
Typo
8 years ago
WanliTian
4cb5035172
remove store.readPeersJSON
8 years ago
Philip O Toole
aa5c946a40
Linting
8 years ago
Philip O'Toole
851f486ff8
Make waiting for logs optional
8 years ago
Philip O'Toole
059ab0757b
Allow Store to wait for initial logs to be applied
...
Better fix for issue #260 .
8 years ago
Philip O Toole
4b6c936d53
Fix comment typos
8 years ago
Philip O Toole
ae0d119d83
Allow Raft Apply timeout to be configurable
8 years ago
Philip O Toole
5ed6ef0ae8
Covert dumps to queries in HTTP service
...
There is no need for special logic at the Store level.
8 years ago
Philip O Toole
e6586f41f2
Use single load for dump processing
...
Writing a sufficiently sophisicated parser is too much work, and unlikely to be successful soon. Instead this change simply loads the entire dump as one command, and allows the underlying SQLite support to parse it correctly. This will definitely work, but since the load goes over Raft, it may hit limits with regards to network transfer sizes. Right now any limitations in that area are unknown.
Therefore this functionality remains somewhat experimental.
8 years ago
Philip O Toole
098578bda8
Clarify guarantees of some functions
8 years ago
Philip O Toole
c2ab24378c
Lint fixes
8 years ago
Philip O Toole
2fb99e5b20
Use Store logger everywhere in Store
8 years ago
Tycho Andersen
dc2128d02a
add a raw snapshotting API
...
This allows us to grab a raw snapshot of the databse.
v2: s/RawDBSnapshot/Database, remove reference to raft from the comment.
v3: add a leader arg to Database to do a leader check
v4: fix doc string.
Signed-off-by: Tycho Andersen <tycho.andersen@canonical.com>
8 years ago
Tycho Andersen
326ca757d7
add an API for using custom raft.PeerStore/log.Logger
...
This is useful in case the server needs to store other metadata (e.g. auth
data) along side the peer list.
The logging bit is handy in case something has its own logging framework
that it wants to use.
Signed-off-by: Tycho Andersen <tycho.andersen@canonical.com>
8 years ago
Philip O'Toole
7ac3450ab2
Suport observer deregistration
8 years ago
Tycho Andersen
aa47089a32
expose raft's observer API via store
...
Signed-off-by: Tycho Andersen <tycho.andersen@canonical.com>
8 years ago
Tycho Andersen
4cf094c6a4
store: use ErrNotLeader everywhere
...
Signed-off-by: Tycho Andersen <tycho.andersen@canonical.com>
8 years ago
Philip O'Toole
4242a641b1
Fix typo in comment
8 years ago
Philip O'Toole
da21606644
Move chinook package to idiomatic testdata dir
...
https://github.com/golang/go/issues/14715
8 years ago
Philip O'Toole
634f55211c
Unit test with Chinook database
...
https://chinookdatabase.codeplex.com/
8 years ago
Philip O'Toole
eda120ffaa
Correct unit tests
8 years ago
Philip O'Toole
c00f1b460b
Accept BEGIN and COMMIT on load()
8 years ago
Philip O'Toole
39ebf50cb7
Remove explicit control of FK constraints
...
It's not clear whether it's the right thing to do, and FK constraints
can still be controlled via PRAGMA commands.
8 years ago
Philip O'Toole
9ffa309f69
Stop database-level control of FK during load
8 years ago
Philip O'Toole
ed304b7840
Remove useless comment
8 years ago
Philip O'Toole
f87ee412e2
Correct unit tests
...
'PRAGMA foreign_keys=OFF' is no longer filtered.
8 years ago
Philip O'Toole
36368d6786
Start removing batch load
8 years ago
Philip O'Toole
68793cfcbb
Use a transaction for loaded batch
...
Bump the batch size to 1000 too.
8 years ago
Philip O Toole
b95af88367
Test SQL loading with blank lines
8 years ago
Philip O Toole
f44c08a530
Skip loading blank lines
8 years ago
Philip O Toole
9c180d8f43
SQL parser now strips newlines
8 years ago
Philip O Toole
1f5cfc65d8
Trim leading and trailing newlines
8 years ago
Philip O Toole
84f3be9c10
Add missing package
8 years ago
Philip O Toole
b87de293ce
Move \n ; trimming to SQL line parser
8 years ago
Philip O Toole
6ee9cdbbdf
Start integration of SQL line parser
8 years ago
Philip O Toole
3726d1bb20
Add DB conf to diagnostics
8 years ago
Philip O Toole
cb7686fa46
Diagnostics show actual FK constraint state
8 years ago
Philip O Toole
2daac8704b
Support batching during load
8 years ago
Philip O Toole
38f9b6ff20
Test loading no commands
8 years ago
Philip O Toole
ab743b1ead
Test loading an empty file
8 years ago
Philip O Toole
ccd48161f3
Unit test Store Load()
8 years ago