Philip O'Toole
bf18b3fab5
Remove some excessive logging
9 months ago
Philip O'Toole
07f41056b2
Friendly dir size
9 months ago
Philip O'Toole
7dd161c386
Don't ever use zero tickers
9 months ago
Philip O'Toole
a8cb7bf142
Always stop tickers
9 months ago
Philip O'Toole
8230b11088
Cleaner WAL goroutine handling
...
Add a unit test too, checking if WAL-triggered snapshotting actually
happens.
9 months ago
Philip O'Toole
d881d8a78c
Merge remote-tracking branch 'origin' into command-processor
9 months ago
Philip O'Toole
4fa94cc465
Fix logic
9 months ago
Philip O'Toole
4095561a3f
Only report Snapshot stats if available
9 months ago
Philip O'Toole
4d605ac9eb
Store WAL path in store, to avoid races
...
https://app.circleci.com/pipelines/github/rqlite/rqlite/3966/workflows/a460ab69-2827-4a72-9a79-1aceda68ee7d/jobs/31288
9 months ago
Philip O'Toole
40c5426b9f
Refactor into CommandProcessor
9 months ago
Philip O'Toole
c501d2d48f
Don't close nil channel
9 months ago
Philip O'Toole
db00fb3c99
Don't check for WAL size if 0 interval
9 months ago
Philip O'Toole
7c97fc4347
Support snaphotting on WAL size
9 months ago
Philip O'Toole
aac54ef1c0
User snapshot channel not needed
9 months ago
Philip O'Toole
1262de501a
Set trailing logs for user-requested snapshot
9 months ago
Philip O'Toole
20cf8cb38e
More variable corrections
9 months ago
Philip O'Toole
d9a96db815
More counting of Snapshotting
9 months ago
Philip O'Toole
04dd6a417d
Correct use of metric
9 months ago
Philip O'Toole
8f625d86f1
More comment updates
9 months ago
Philip O'Toole
19a2970dd1
Improve comments
9 months ago
Philip O'Toole
9b00510382
Make Notify requirements clearer
9 months ago
Philip O'Toole
954503dc66
Move Store to Layer and create NewLayer in tcp
9 months ago
Philip O'Toole
ad916270d2
Bring go mod import path into compliance
9 months ago
Philip O'Toole
9b193f2f4c
More coverage cleanup
9 months ago
Philip O'Toole
2e81a92e3a
Fix defer ordering
9 months ago
Philip O'Toole
5942e986a8
Simplify and correct DB path logic
9 months ago
Philip O'Toole
f6b0be13ed
Create temp SQLite files in same location as db
9 months ago
Philip O'Toole
54a2487807
Don't test WAL removal
...
Look, it could always happen if the node is crashed hard. A WAL file
could be left around at anytime, just accept it and don't bother with
the cleanup.
9 months ago
Philip O'Toole
5f9c9aad2d
Don't re-open and close on shutdown
9 months ago
Philip O'Toole
5f3e854c4a
Remove some blank lines
9 months ago
Philip O'Toole
364128c9b0
Tighter code
9 months ago
Philip O'Toole
10129ed312
Confirm restored data looks like a SQLite file
9 months ago
Philip O'Toole
c4e09c1f47
Noop earlier - make sure it's successful
9 months ago
Philip O'Toole
4988c29951
Log at DB level if open times are higher
9 months ago
Philip O'Toole
e7ab1389c9
Periodically log boot progres
9 months ago
Philip O'Toole
140f106acc
Add ReadFrom to Store
9 months ago
Philip O'Toole
e6b703a0c0
Remove ability to trigger chunked-loading
...
The low-level Raft system still recognizes the Chunk command, so that
this system can interop with older versions, which might still have a
Chunk command in logs.
9 months ago
Philip O'Toole
1e34b89cd9
Don't load invalid database
9 months ago
Philip O'Toole
3cc47bceee
Revert to simpler one-shot load for SQLite files
9 months ago
Philip O'Toole
cc4addf7e0
Noop() returns ApplyFuture
9 months ago
Philip O'Toole
7032ac7319
Don't truncate to 0 trailing entries
...
The Raft code seems to be have an unsigned int overflow problem when
this happens.
9 months ago
Philip O'Toole
caca666757
Remap "load in progress" to error type
9 months ago
Philip O'Toole
8708d51746
Fix GoDoc
9 months ago
Philip O'Toole
6fdd454632
Introduce explicit FSM type
...
This will make the code cleaner, and help clean up un-closed file
handles.
9 months ago
Philip O'Toole
abc0898c5f
Close all dechunkers at Store shutdown
9 months ago
Philip O'Toole
e3fa8584c0
Store can handle snapshotting empty WAL files
9 months ago
Philip O'Toole
36ad021fdd
Don't ignore errors
10 months ago
Philip O'Toole
df76560168
Sync and report number of loads in progress
10 months ago
Philip O'Toole
b0cd5c5a0c
Add more Load stats
10 months ago
Philip O'Toole
e0f361509a
Handle aborted chunks at lowest level
10 months ago
Philip O'Toole
dd6f386450
Refactor chunk loading
10 months ago
Philip O'Toole
94898ce43e
Track loads in progress
10 months ago
Philip O'Toole
44fa23b714
Use TrailingLogs = 0 to truncate log
10 months ago
Philip O'Toole
264e9822eb
Standardize on chunk size
10 months ago
Philip O'Toole
d726a4c780
Correct and simplify logic
10 months ago
Philip O'Toole
a96112b41d
Snapshot log after one-command load
10 months ago
Philip O'Toole
1f1ebf209f
Don't delete last log
10 months ago
Philip O'Toole
8adc11fb83
Snapshot in a goroutine
10 months ago
Philip O'Toole
c93447e29f
Delete chunked load entries in log after loading
10 months ago
Philip O'Toole
0dcc67f6f1
Perform full snapshot after chunked load
...
This must happen as chunked loading directly changes the SQLite file,
invalidating any existing Snapshots. Also, add missing unit tests for
chunked loading.
10 months ago
Philip O'Toole
31aef91d17
Add FullNeeded() to Snapshot store
...
It may be necessary to override the FullNeeded definition, so wrap it.
10 months ago
Philip O'Toole
8211af961d
Move some VACUUM checks to the Store
10 months ago
Philip O'Toole
5c7996fd98
Support optional VACUUM on backup
10 months ago
Philip O'Toole
a0bf10b021
Move VACUUM to DB layer
10 months ago
Philip O'Toole
dfb8a60c17
Move Store Provide to a distinct type
10 months ago
Philip O'Toole
3ae4255e62
Backup-related cleanups
10 months ago
Philip O'Toole
4c31c480f5
Better name
10 months ago
Philip O'Toole
5a8a2744ff
Support skipping verifying frame data
...
This halves the Compacting Scan time, and doesn't seem necessary since
rqlite is the only system writing the WAL files.
10 months ago
Philip O'Toole
ee13427c67
Tigher logging
10 months ago
Philip O'Toole
bd9e96ea02
Integrate Compacting WAL writer
10 months ago
Philip O'Toole
d53a34f765
Don't log too much
10 months ago
Philip O'Toole
ba98e88a7b
Update store.go
10 months ago
Philip O'Toole
70d5e6de34
Check join and notify addresses are resolvable
10 months ago
Philip O'Toole
66c1002c32
Remove obsolete metrics
11 months ago
Philip O'Toole
b2663de630
Rename snapshot2 -> snapshot
11 months ago
Philip O'Toole
c11d059551
Better function name
11 months ago
Philip O'Toole
10e6f1b115
Clean up snapshot FDs
11 months ago
Philip O'Toole
93566415e3
Just run bad test
11 months ago
Philip O'Toole
47d877cc25
Don't close dst file until done
11 months ago
Philip O'Toole
09712788c9
Store now builds with snapshot2
11 months ago
Philip O'Toole
6a60108f0c
Start work on Restore and Recover
...
Builds, but still needs more work. It's referencing snapshot(1).
11 months ago
Philip O'Toole
d34c35df31
No need to Store-level WAL selection
11 months ago
Philip O'Toole
5e3d481a6e
Fill out methods on SnapshotStore v2
12 months ago
Philip O'Toole
7c420407c2
WIP snapshot2
...
Close() on the Sink is done (first pass). What's needed next are all the
methods on the Store (Open, List, FullNeeded). I also need to decide
what getSnapshots should really do -- return names or a slice of meta?
12 months ago
Philip O'Toole
974f939ffe
Add an interface between Store and Snapshot Store
...
This will make it easier to swap new implementations in.
1 year ago
Philip O'Toole
be17f2a3cc
Correct variable assignment order
1 year ago
Philip O'Toole
53b432a391
More DB and Snapshot stats
1 year ago
Philip O'Toole
59e6a23b02
Clean-up logging
1 year ago
Philip O'Toole
2fff800273
Substantially shrink Test_MultiNodeClusterLargeQueuedWrites
1 year ago
Philip O'Toole
378dd4b2a1
Simpler logging of first application
1 year ago
Philip O'Toole
5ab7069aa0
Update store.go
1 year ago
Philip O'Toole
a4f7431108
Log first time a log is applied
1 year ago
Philip O'Toole
ad86a2989b
Use non-racy function for Leader ID and address
1 year ago
Philip O'Toole
225b5346f5
Add basic stats for Snapshot store
1 year ago
Philip O'Toole
fedbdb5478
Enable GZIP snapshot compression
...
Tests are failing.
1 year ago
Philip O'Toole
d2c5acc447
Wrap transport as passthru
1 year ago
Philip O'Toole
47e9bbde49
Fix single-node upgrade test
1 year ago
Philip O'Toole
dda86be0d3
Better Store open-and-close logging
1 year ago
Philip O'Toole
c44d020c11
Only attempt to checkpoint once
1 year ago
Philip O'Toole
d6b9c5b39e
Close streams
1 year ago
Philip O'Toole
ee8b23ba94
More logging
1 year ago
Philip O'Toole
e8e0381622
Log when Store closes
1 year ago
Philip O'Toole
a6134b02bc
Instrument the FSM snapshot process
1 year ago
Philip O'Toole
76867a682f
Time FSM snapshot creation
1 year ago
Philip O'Toole
ab6ca783e1
Checkpoint database during Recovery
1 year ago
Philip O'Toole
a9e895bee2
Some cleanups
1 year ago
Philip O'Toole
47476d5b58
Integrate new snapshoting with Store
...
- Snapshot
- Restore
- RecoverNode
1 year ago
Philip O'Toole
5eb4e38cbc
Add comments and errors
1 year ago
Philip O'Toole
e4d5ffb8a9
Flesh our Store check()
1 year ago
Philip O'Toole
07d49961d8
Remove unused functions
1 year ago
Philip O'Toole
9e31a054aa
RecoverNode now uses on-disk database
1 year ago
Philip O'Toole
88ce311b37
Remove all WAL-related files when loading new DB
1 year ago
Philip O'Toole
b931e387be
First pass at removing in-memory support
1 year ago
Philip O'Toole
f06e547ab5
Less disk IO during load
1 year ago
Philip O'Toole
f5517c0f66
Correct comment
1 year ago
Philip O'Toole
9bdf048975
Better naming of temp backup file
1 year ago
Philip O'Toole
b5cb889e1e
Auto-restore doesn't require ready() check
1 year ago
Philip O'Toole
b7633143cc
Restore from file uses chunking
1 year ago
Philip O'Toole
5ca5ce342e
Support internode LoadChunk()
1 year ago
Philip O'Toole
1c8e52c5f7
Test_SingleNodeLoadBinaryFromReader passes
...
That's good, but much more extensive testing needed, and this needs to
replace the existing load path -- and then we move to 8.0.
1 year ago
Philip O'Toole
93a752ae0e
Better variable names
1 year ago
Philip O'Toole
1c6f691ff8
Integrate chunking - WIP
...
I need more sophisticated dechunker management.
1 year ago
Philip O'Toole
13d47b24d8
Store can load from an io.Reader
...
This may be helpful for chunked-loads.
1 year ago
Philip O'Toole
a40c43725f
More logic reversal
1 year ago
Philip O'Toole
f81aad8fd3
Revert logic for clarity
1 year ago
Philip O'Toole
21dff4a68c
Remove on-disk-startup control
...
With the move to WAL and "synchronous mode" to OFF, on-disk startup
times are very close to in-memory. There is no need for this control
anymore and it complicates the start-up code.
1 year ago
Philip O'Toole
f7ced46377
Count number of DB stats errors
1 year ago
Philip O'Toole
36fa7f2afd
Detect DB stats error and log
1 year ago
Philip O'Toole
7ce2c6b99a
Fix comment
1 year ago
Philip O'Toole
7432bceca3
Since we're passed a ReadCloser, close it
1 year ago
Philip O'Toole
4342650c33
Fix RecoverNode
1 year ago
Philip O'Toole
ad73779285
Periodically record actually applied index
...
This ensures the system doesn't apply uncommitted log entries to the
FSM, if it runs the in-memory startup. There have been no reports of
this in the field -- it is an edge case -- but this removes any chance
of it.
1 year ago
Philip O'Toole
abfe056b4c
Remove any -wal and -shm files on shutdown
...
If in WAL mode, open-and-close again to remove the -wal file. This is not strictly necessary, since any on-disk database files will be removed when rqlite next starts, but it leaves the directory containing the database file in a cleaner state.
1 year ago
Philip O'Toole
b2f28d1181
Better WAL control
1 year ago
Philip O'Toole
2f7672d897
Enable WAL mode
1 year ago
Philip O'Toole
585f9c9c05
Use Snapshot decoder in Store
1 year ago
Philip O'Toole
c87931c45c
Factor Snapshot creation into own module
1 year ago
Philip O'Toole
3f462eecfa
Remove all on-disk SQLite files on startup
1 year ago
Philip O'Toole
7e1b8d12d7
Add full WAL support to DB layer
1 year ago
Philip O'Toole
eed215c664
Only 1 Snapshot needs to be retained
1 year ago
Philip O'Toole
b9253e6b41
Improve logging in Store
1 year ago
Philip O'Toole
4a7ac2850f
rqlite doesn't (yet) support WAL mode SQLite data
1 year ago
Philip O'Toole
92bc513242
Move FSMSnapshot to own source file
1 year ago
Philip O'Toole
7badba4a7a
Try fast paths
1 year ago
Philip O'Toole
76ba50ca86
Log last (if any) WaitForLeader error
1 year ago
Philip O'Toole
ef403cdcc6
Fix up WaitForLeader
...
Add dedicated error codes too.
1 year ago
Philip O'Toole
c05e091cd1
Add WaitForRemoval
1 year ago
Philip O'Toole
c23d3e60b1
Reuse existing consistency level
1 year ago
Philip O'Toole
18b76ebd4f
go fmt
1 year ago
Philip O'Toole
2f8d4e95e0
Update store.go
1 year ago