Philip O'Toole
c93447e29f
Delete chunked load entries in log after loading
10 months ago
Philip O'Toole
0dcc67f6f1
Perform full snapshot after chunked load
...
This must happen as chunked loading directly changes the SQLite file,
invalidating any existing Snapshots. Also, add missing unit tests for
chunked loading.
10 months ago
Philip O'Toole
31aef91d17
Add FullNeeded() to Snapshot store
...
It may be necessary to override the FullNeeded definition, so wrap it.
10 months ago
Philip O'Toole
8211af961d
Move some VACUUM checks to the Store
10 months ago
Philip O'Toole
5c7996fd98
Support optional VACUUM on backup
10 months ago
Philip O'Toole
a0bf10b021
Move VACUUM to DB layer
10 months ago
Philip O'Toole
dfb8a60c17
Move Store Provide to a distinct type
10 months ago
Philip O'Toole
3ae4255e62
Backup-related cleanups
10 months ago
Philip O'Toole
4c31c480f5
Better name
10 months ago
Philip O'Toole
5a8a2744ff
Support skipping verifying frame data
...
This halves the Compacting Scan time, and doesn't seem necessary since
rqlite is the only system writing the WAL files.
10 months ago
Philip O'Toole
ee13427c67
Tigher logging
10 months ago
Philip O'Toole
bd9e96ea02
Integrate Compacting WAL writer
10 months ago
Philip O'Toole
d53a34f765
Don't log too much
10 months ago
Philip O'Toole
ba98e88a7b
Update store.go
10 months ago
Philip O'Toole
70d5e6de34
Check join and notify addresses are resolvable
10 months ago
Philip O'Toole
66c1002c32
Remove obsolete metrics
11 months ago
Philip O'Toole
b2663de630
Rename snapshot2 -> snapshot
11 months ago
Philip O'Toole
c11d059551
Better function name
11 months ago
Philip O'Toole
10e6f1b115
Clean up snapshot FDs
11 months ago
Philip O'Toole
93566415e3
Just run bad test
11 months ago
Philip O'Toole
47d877cc25
Don't close dst file until done
11 months ago
Philip O'Toole
09712788c9
Store now builds with snapshot2
11 months ago
Philip O'Toole
6a60108f0c
Start work on Restore and Recover
...
Builds, but still needs more work. It's referencing snapshot(1).
11 months ago
Philip O'Toole
d34c35df31
No need to Store-level WAL selection
11 months ago
Philip O'Toole
5e3d481a6e
Fill out methods on SnapshotStore v2
12 months ago
Philip O'Toole
7c420407c2
WIP snapshot2
...
Close() on the Sink is done (first pass). What's needed next are all the
methods on the Store (Open, List, FullNeeded). I also need to decide
what getSnapshots should really do -- return names or a slice of meta?
12 months ago
Philip O'Toole
974f939ffe
Add an interface between Store and Snapshot Store
...
This will make it easier to swap new implementations in.
1 year ago
Philip O'Toole
be17f2a3cc
Correct variable assignment order
1 year ago
Philip O'Toole
53b432a391
More DB and Snapshot stats
1 year ago
Philip O'Toole
59e6a23b02
Clean-up logging
1 year ago
Philip O'Toole
2fff800273
Substantially shrink Test_MultiNodeClusterLargeQueuedWrites
1 year ago
Philip O'Toole
378dd4b2a1
Simpler logging of first application
1 year ago
Philip O'Toole
5ab7069aa0
Update store.go
1 year ago
Philip O'Toole
a4f7431108
Log first time a log is applied
1 year ago
Philip O'Toole
ad86a2989b
Use non-racy function for Leader ID and address
1 year ago
Philip O'Toole
225b5346f5
Add basic stats for Snapshot store
1 year ago
Philip O'Toole
fedbdb5478
Enable GZIP snapshot compression
...
Tests are failing.
1 year ago
Philip O'Toole
d2c5acc447
Wrap transport as passthru
1 year ago
Philip O'Toole
47e9bbde49
Fix single-node upgrade test
1 year ago
Philip O'Toole
dda86be0d3
Better Store open-and-close logging
1 year ago
Philip O'Toole
c44d020c11
Only attempt to checkpoint once
1 year ago
Philip O'Toole
d6b9c5b39e
Close streams
1 year ago
Philip O'Toole
ee8b23ba94
More logging
1 year ago
Philip O'Toole
e8e0381622
Log when Store closes
1 year ago
Philip O'Toole
a6134b02bc
Instrument the FSM snapshot process
1 year ago
Philip O'Toole
76867a682f
Time FSM snapshot creation
1 year ago
Philip O'Toole
ab6ca783e1
Checkpoint database during Recovery
1 year ago
Philip O'Toole
a9e895bee2
Some cleanups
1 year ago
Philip O'Toole
47476d5b58
Integrate new snapshoting with Store
...
- Snapshot
- Restore
- RecoverNode
1 year ago
Philip O'Toole
5eb4e38cbc
Add comments and errors
1 year ago
Philip O'Toole
e4d5ffb8a9
Flesh our Store check()
1 year ago
Philip O'Toole
07d49961d8
Remove unused functions
1 year ago
Philip O'Toole
9e31a054aa
RecoverNode now uses on-disk database
1 year ago
Philip O'Toole
88ce311b37
Remove all WAL-related files when loading new DB
1 year ago
Philip O'Toole
b931e387be
First pass at removing in-memory support
1 year ago
Philip O'Toole
f06e547ab5
Less disk IO during load
1 year ago
Philip O'Toole
f5517c0f66
Correct comment
1 year ago
Philip O'Toole
9bdf048975
Better naming of temp backup file
1 year ago
Philip O'Toole
b5cb889e1e
Auto-restore doesn't require ready() check
1 year ago
Philip O'Toole
b7633143cc
Restore from file uses chunking
1 year ago
Philip O'Toole
5ca5ce342e
Support internode LoadChunk()
1 year ago
Philip O'Toole
1c8e52c5f7
Test_SingleNodeLoadBinaryFromReader passes
...
That's good, but much more extensive testing needed, and this needs to
replace the existing load path -- and then we move to 8.0.
1 year ago
Philip O'Toole
93a752ae0e
Better variable names
1 year ago
Philip O'Toole
1c6f691ff8
Integrate chunking - WIP
...
I need more sophisticated dechunker management.
1 year ago
Philip O'Toole
13d47b24d8
Store can load from an io.Reader
...
This may be helpful for chunked-loads.
1 year ago
Philip O'Toole
a40c43725f
More logic reversal
1 year ago
Philip O'Toole
f81aad8fd3
Revert logic for clarity
1 year ago
Philip O'Toole
21dff4a68c
Remove on-disk-startup control
...
With the move to WAL and "synchronous mode" to OFF, on-disk startup
times are very close to in-memory. There is no need for this control
anymore and it complicates the start-up code.
1 year ago
Philip O'Toole
f7ced46377
Count number of DB stats errors
1 year ago
Philip O'Toole
36fa7f2afd
Detect DB stats error and log
1 year ago
Philip O'Toole
7ce2c6b99a
Fix comment
1 year ago
Philip O'Toole
7432bceca3
Since we're passed a ReadCloser, close it
1 year ago
Philip O'Toole
4342650c33
Fix RecoverNode
1 year ago
Philip O'Toole
ad73779285
Periodically record actually applied index
...
This ensures the system doesn't apply uncommitted log entries to the
FSM, if it runs the in-memory startup. There have been no reports of
this in the field -- it is an edge case -- but this removes any chance
of it.
1 year ago
Philip O'Toole
abfe056b4c
Remove any -wal and -shm files on shutdown
...
If in WAL mode, open-and-close again to remove the -wal file. This is not strictly necessary, since any on-disk database files will be removed when rqlite next starts, but it leaves the directory containing the database file in a cleaner state.
1 year ago
Philip O'Toole
b2f28d1181
Better WAL control
1 year ago
Philip O'Toole
2f7672d897
Enable WAL mode
1 year ago
Philip O'Toole
585f9c9c05
Use Snapshot decoder in Store
1 year ago
Philip O'Toole
c87931c45c
Factor Snapshot creation into own module
1 year ago
Philip O'Toole
3f462eecfa
Remove all on-disk SQLite files on startup
1 year ago
Philip O'Toole
7e1b8d12d7
Add full WAL support to DB layer
1 year ago
Philip O'Toole
eed215c664
Only 1 Snapshot needs to be retained
1 year ago
Philip O'Toole
b9253e6b41
Improve logging in Store
1 year ago
Philip O'Toole
4a7ac2850f
rqlite doesn't (yet) support WAL mode SQLite data
1 year ago
Philip O'Toole
92bc513242
Move FSMSnapshot to own source file
1 year ago
Philip O'Toole
7badba4a7a
Try fast paths
1 year ago
Philip O'Toole
76ba50ca86
Log last (if any) WaitForLeader error
1 year ago
Philip O'Toole
ef403cdcc6
Fix up WaitForLeader
...
Add dedicated error codes too.
1 year ago
Philip O'Toole
c05e091cd1
Add WaitForRemoval
1 year ago
Philip O'Toole
c23d3e60b1
Reuse existing consistency level
1 year ago
Philip O'Toole
18b76ebd4f
go fmt
1 year ago
Philip O'Toole
2f8d4e95e0
Update store.go
1 year ago
Philip O'Toole
eea101a1d5
Fix duplicate import
1 year ago
Philip O'Toole
f7cdbdea68
Handle Command_COMMAND_TYPE_EXECUTE_QUERY in Store
...
Works! Store.Request() needs unit testing however.
1 year ago
Philip O'Toole
b2b0b188e9
Factor out functions
1 year ago
Philip O'Toole
89f472743b
Start adding Store.Request()
1 year ago
Philip O'Toole
7680483793
Tighten up remove
1 year ago
Philip O'Toole
e64d78b809
Support removing self from cluster on shutdown
1 year ago
Philip O'Toole
02b6f7df61
selfLeaderChange is not just about restores
1 year ago
Philip O'Toole
159b127101
Clean up comments
1 year ago
Philip O'Toole
0c6be22478
Close auto-restore file
1 year ago
Philip O'Toole
1fa7cf73a7
Log any failure to delete auto-restore file
1 year ago
Philip O'Toole
1941c947c7
Fix comment
1 year ago
Philip O'Toole
466c757dcf
Add long-missing init call to Store stats
1 year ago
Philip O'Toole
ee55a35c03
SetRestorePath unit testing
1 year ago
Philip O'Toole
e7919a906f
Avoid the race during testing
1 year ago
Philip O'Toole
31467482a3
Store-level multi-node auto-restore testing
...
This test ensures that auto-restoring works as expected when
Bootstrapping is enabled.
1 year ago
Philip O'Toole
bf1e1cb1cd
Add first tested version of Store auto-restore
1 year ago
Philip O'Toole
5c2e8e7ad6
Add channel which gets signal on becoming Leader
1 year ago
Philip O'Toole
833d9d3a6f
Perform older checks first
1 year ago
Philip O'Toole
5e3f294dfa
Only check for a ready store if making changes
1 year ago
Philip O'Toole
2ce3c5f34c
Check Store is ready in more places
1 year ago
Philip O'Toole
57ca03014b
More integration and testing of Store readiness
1 year ago
Philip O'Toole
716f84ff28
Store supports registering "ready" channels
1 year ago
Philip O'Toole
372e4dd4a9
Store implements simpler Provide() interface
1 year ago
Philip O'Toole
5fcc222f73
Better Store comment
1 year ago
Philip O'Toole
ae97060f2c
Implement the DataProvider interface on the Store
1 year ago
Philip O'Toole
519df067cc
Check for more possible errors with peers
...
https://github.com/rqlite/rqlite/issues/1217
1 year ago
Philip O'Toole
35df643a0b
Remove unneeded type assertions
1 year ago
Philip O'Toole
1794724ef0
Remove unused code and fix error handling
1 year ago
Philip O'Toole
93b3308502
Skip any files that disappear during walking
1 year ago
Philip O'Toole
ff001bc5a8
Migrate to a Protos data model for Notify Requests
2 years ago
Philip O'Toole
a782513a11
Migrate to a join-request proto
2 years ago
Philip O'Toole
fdf5fce053
Update store.go
2 years ago
Philip O'Toole
2730fa06e5
Empty config means not a voter
2 years ago
Philip O'Toole
5636febf9a
Add IsVoter() and add to status/ output
2 years ago
Philip O'Toole
defde21281
Merge remote-tracking branch 'origin' into ignore-join-explicit-if-state
2 years ago
Philip O'Toole
90b4dc0745
Silently ignore unchanged node self-joins
2 years ago
Philip O'Toole
54d61c7866
Update store.go
2 years ago
Philip O'Toole
a2899a1cdd
Support stepping down before shutting down
2 years ago
Philip O'Toole
0702e39ace
Add ResetStats
2 years ago
Philip O'Toole
4da5b6322f
Log message could be confusing with remote remove
2 years ago
Philip O'Toole
98575d727b
Start adding remote remove node
...
More testing, including unit testing, required.
2 years ago
Philip O'Toole
bfc35eaf7c
Simplify reap command line flags ( #1118 )
...
Simplify reap command line flags
2 years ago
Philip O'Toole
220424c7d1
Simple refactor, better Go style
2 years ago
Philip O'Toole
07620cab41
Reap nodes ( #1114 )
...
Support automatic reaping of nodes
2 years ago
Philip O'Toole
e158ae8a72
Start HTTP server ASAP
2 years ago
Philip O'Toole
0e8e266f96
Merge pull request #1090 from rqlite/fix-remote-load-errors
...
Fix remote-load error handling
2 years ago
Philip O'Toole
893fbb0249
More store-level load operation logging
2 years ago
Philip O'Toole
65cd556549
Fix remote-load error handling
...
Error from remote node was being dropped.
2 years ago
Philip O'Toole
5f33bffa0f
More logging and resolution checks
2 years ago
Philip O'Toole
b0f17b86c7
Notified node checks address resolution
...
If a non-resolvable address is passed to the Store, and then Store then
adds that to the config, it can result in a cluster that seems to have
insufficient members (since the remote node can't be contacted). This
may trigger a new leader election.
2 years ago
Philip O'Toole
47d79c859a
Log database backup time
2 years ago
Philip O'Toole
9fc6ea16e9
Even better defer sequence
2 years ago
Philip O'Toole
555883d1a2
Correct order of deferred calls
2 years ago
Philip O'Toole
54aec1ca33
Tighten up backup code
2 years ago
Philip O'Toole
31bba4c577
Move to Backup proto model
2 years ago
Philip O'Toole
dc2b9f5b6f
Ignore freshness when serving queries on Leader
...
It doesn't make sense to check the last leader-contact time when the
node itself is the leader. Any read, when served by the leader, is going
to be within the freshness bound.
Fixes https://github.com/rqlite/rqlite/issues/1048
2 years ago
Philip O'Toole
89b132ac26
Fix Store-level unit tests
2 years ago
Philip O'Toole
b03ed99eda
Update comment
2 years ago
Philip O'Toole
304124952c
Add Store level changes
...
Needs unit testing, and HTTP testing won't compile.
2 years ago
Philip O'Toole
2729a17279
Move some functions to standalone form
2 years ago
Philip O'Toole
9783ef24b1
Switch to error on self-join attempt
3 years ago
Philip O'Toole
a524ed4979
Ignore join request if node ID is same
3 years ago
Philip O'Toole
e4fed0cee3
Add bootstrap-expect ( #974 )
...
Add cluster-boostrap
3 years ago
Philip O'Toole
ad7986630d
Better command-line flags
3 years ago
Philip O'Toole
ce099087ee
Control whether bbolt sync freelist to disk
...
When true, skips syncing freelist to disk. This improves the database write performance under normal operation, but requires a full database re-sync during recovery.# Please enter the commit message for your changes.
3 years ago
Philip O'Toole
490f654f36
Move to etcd's fork of BoltDB
...
bboltdb is the version the Consul now runs.
3 years ago
Philip O'Toole
4aea326959
Consul and etcd autoclustering ( #957 )
...
This PR introduces new node-discovery integration with Consul and etcd. By using one of those systems with rqlite, automatic clustering of rqlite is much easier.
3 years ago
Philip O'Toole
ca556044e3
Merge branch 'master' of github.com:rqlite/rqlite into refactor-main-bootstrap
3 years ago
Philip O'Toole
2ffb29394f
Fix typo
3 years ago
Philip O'Toole
f5c03d4b60
Try simple refactor of bootstrap
3 years ago
Philip O'Toole
769557db7e
Better variable name
3 years ago
chermehdi
3bfcbffc37
Fix typos and Grammar issues
3 years ago
Philip O'Toole
33bee34f89
More Store restart testing
3 years ago
Philip O'Toole
d74c05c2db
Support disabling in-memory init of on-disk
3 years ago
Philip O'Toole
003301aee6
Minor fixes to log messages ( #920 )
...
Minor fixes to log messages
3 years ago
Philip O'Toole
0b621cbec8
More logging tweaks
3 years ago
Philip O'Toole
5ce51bc2ba
Better comments
3 years ago
Philip O'Toole
05534cc94c
Don't use Windows file paths as IDs
3 years ago
Philip O'Toole
f9f35bf95c
Dump file
3 years ago
Philip O'Toole
7093cb25d3
Fix race in test
3 years ago
Philip O'Toole
9fb114c07f
Recover test after snapshot
3 years ago
Philip O'Toole
519a91e635
Actual recovery node test passes on Linux
3 years ago
Philip O'Toole
1b158d09fb
Simple unit test of node-recovery
3 years ago
Philip O'Toole
1abd3007ca
Rename peers.json after successful recovery
3 years ago
Philip O'Toole
cc1de72796
Ensure raft/ exists inside data directory
3 years ago
Philip O'Toole
abccc6ced3
Better function name
3 years ago
Philip O'Toole
d9ad3c080a
Add untested recover-node functionality
3 years ago
Philip O'Toole
32239b5053
Move new function to earlier per convention
3 years ago
Philip O'Toole
4ab72a8194
Refactor to make recover-cluster code easier
3 years ago
Philip O'Toole
5dd4aeed1e
More Snapshot metrics
3 years ago
Philip O'Toole
38eef8ee82
Merge pull request #890 from rqlite/more-abl
...
ABL
3 years ago
Philip O'Toole
dd71c07c73
ABL
3 years ago
Philip O'Toole
0e34dc49ca
Add BoltDB stats
3 years ago
Philip O'Toole
d6be09ba65
Update Snapshot comment
3 years ago
Philip O'Toole
5d48a18950
Check if slice is empty in addition to being nil
3 years ago
Philip O'Toole
bb002bb81b
Block query transactions during serialize
3 years ago
Philip O'Toole
a6da8d474f
Ensure (any) directory for on-disk-path exists
3 years ago
Philip O'Toole
1f33dbb2cb
Fix up Store-open logging
3 years ago
Philip O'Toole
105a894fe5
Support explicit SQLite path at Store level
3 years ago
Philip O'Toole
a6346ba9d4
Fix unit test -- start muxes
3 years ago
Philip O'Toole
dd692ecf4f
Add gauge-like metric for Snapshot times
3 years ago
Philip O'Toole
9d87a15194
Small simplication
3 years ago
Philip O'Toole
43e60a0469
Perform leader check first for STRONG query
...
Don't wait until Raft layer returns it. There is a small chance of a
false leadership detection, but the cluster would have to be in flux
anyway for that to be the case. Execute does this already.
3 years ago
Philip O'Toole
fd98114839
HTTP-level testing passes
3 years ago
Philip O'Toole
3580e341a8
Store-level testing passes
3 years ago
Philip O'Toole
deb4dabf2e
Thread through foreign key constraint controls
3 years ago
Philip O'Toole
0536bc3c1c
ABL
3 years ago
Philip O'Toole
74d0780a5f
Merge pull request #845 from rqlite/sqlite-options
...
Add SQLite compile-time options to status output
3 years ago