14858a37d4
Use last DB mod time in Provider
9 months ago
e14d065e3d
Comments and counters
9 months ago
2ff37f9362
Implement Check() on Store Provider
9 months ago
03c9441e94
Clearer comment
9 months ago
f3ba52c749
Merge remote-tracking branch 'origin' into backup-vacuum-fast-path
9 months ago
bccc83b7d5
Add fast path for vacuum, non-compressed backups
9 months ago
a087d509ac
Upload Provider uses Snapshot-locking backup
9 months ago
cf23612147
ABL
9 months ago
b62e7960c9
Switch to "BestSpeed" for all internode traffic
9 months ago
d7611c8b9f
Fix Store unit test
9 months ago
a5f2bd8d74
Allow WAL files for /boot
9 months ago
720926a61a
Simpler return
9 months ago
d0810106c3
Even simpler
9 months ago
0d7008ff9e
Better name
9 months ago
ec11fb4793
Simpler check
9 months ago
4fde9fe572
Refactor tests
9 months ago
4fb36f1049
More tests
9 months ago
9c2e730995
Support compressed backups at Store level
9 months ago
0b51616f50
Merge pull request #1567 from rqlite/use-progress
...
Refactor to make more use of `progress` module
9 months ago
63e3f8c80f
Refactor to make more use of `progress` module
9 months ago
36ee3faf08
Move some state-related functions to own file
9 months ago
d1cc802292
Fix return codes in HasData
...
This function should actually work fine, can wait for next release.
9 months ago
583a4820f5
Add HasData, which returns true if there is data
9 months ago
c315c0ecf6
Update cas.go
9 months ago
1002d2a29f
ABL
9 months ago
fe4747ba0e
Merge pull request #1557 from rqlite/remove-wal-req
...
Remove restriction on non-WAL restore
9 months ago
21dc186186
Remove restriction on non-WAL restore
...
Never really had a strong reason for it.
9 months ago
f06eba4bb8
Fix logic
9 months ago
6333b8cd82
Check for error with specific text
9 months ago
f18fb66665
Better logic
9 months ago
c1cd29e163
Ignore nothing to snapshot
9 months ago
3d88672c60
Add fast-path backup
9 months ago
8bfa54d700
Unit test CAS
9 months ago
5c5b226303
Working
9 months ago
bf18b3fab5
Remove some excessive logging
9 months ago
07f41056b2
Friendly dir size
9 months ago
7dd161c386
Don't ever use zero tickers
9 months ago
a8cb7bf142
Always stop tickers
9 months ago
8230b11088
Cleaner WAL goroutine handling
...
Add a unit test too, checking if WAL-triggered snapshotting actually
happens.
9 months ago
d881d8a78c
Merge remote-tracking branch 'origin' into command-processor
9 months ago
4fa94cc465
Fix logic
9 months ago
4095561a3f
Only report Snapshot stats if available
9 months ago
4d605ac9eb
Store WAL path in store, to avoid races
...
https://app.circleci.com/pipelines/github/rqlite/rqlite/3966/workflows/a460ab69-2827-4a72-9a79-1aceda68ee7d/jobs/31288
9 months ago
40c5426b9f
Refactor into CommandProcessor
9 months ago
c501d2d48f
Don't close nil channel
9 months ago
db00fb3c99
Don't check for WAL size if 0 interval
9 months ago
7c97fc4347
Support snaphotting on WAL size
9 months ago
aac54ef1c0
User snapshot channel not needed
9 months ago
1262de501a
Set trailing logs for user-requested snapshot
9 months ago
20cf8cb38e
More variable corrections
9 months ago
d9a96db815
More counting of Snapshotting
9 months ago
04dd6a417d
Correct use of metric
9 months ago
8f625d86f1
More comment updates
9 months ago
19a2970dd1
Improve comments
9 months ago
9b00510382
Make Notify requirements clearer
9 months ago
954503dc66
Move Store to Layer and create NewLayer in tcp
9 months ago
ad916270d2
Bring go mod import path into compliance
9 months ago
9b193f2f4c
More coverage cleanup
9 months ago
2e81a92e3a
Fix defer ordering
9 months ago
dd1f511f0a
Add unit test for cleanup
9 months ago
5942e986a8
Simplify and correct DB path logic
9 months ago
f6b0be13ed
Create temp SQLite files in same location as db
9 months ago
54a2487807
Don't test WAL removal
...
Look, it could always happen if the node is crashed hard. A WAL file
could be left around at anytime, just accept it and don't bother with
the cleanup.
9 months ago
5f9c9aad2d
Don't re-open and close on shutdown
9 months ago
5f3e854c4a
Remove some blank lines
9 months ago
364128c9b0
Tighter code
9 months ago
10129ed312
Confirm restored data looks like a SQLite file
9 months ago
c4e09c1f47
Noop earlier - make sure it's successful
9 months ago
4988c29951
Log at DB level if open times are higher
9 months ago
e7ab1389c9
Periodically log boot progres
9 months ago
140f106acc
Add ReadFrom to Store
9 months ago
e6b703a0c0
Remove ability to trigger chunked-loading
...
The low-level Raft system still recognizes the Chunk command, so that
this system can interop with older versions, which might still have a
Chunk command in logs.
9 months ago
1e34b89cd9
Don't load invalid database
9 months ago
3cc47bceee
Revert to simpler one-shot load for SQLite files
9 months ago
cc4addf7e0
Noop() returns ApplyFuture
9 months ago
7032ac7319
Don't truncate to 0 trailing entries
...
The Raft code seems to be have an unsigned int overflow problem when
this happens.
9 months ago
0ef35f1cbc
Fix restart issue after chunked-loading
9 months ago
caca666757
Remap "load in progress" to error type
9 months ago
5c86e2ae70
Better name
9 months ago
c4407bf616
More comment fixes
9 months ago
8708d51746
Fix GoDoc
9 months ago
6fdd454632
Introduce explicit FSM type
...
This will make the code cleaner, and help clean up un-closed file
handles.
9 months ago
abc0898c5f
Close all dechunkers at Store shutdown
9 months ago
e3fa8584c0
Store can handle snapshotting empty WAL files
9 months ago
36ad021fdd
Don't ignore errors
9 months ago
e0c6861ca8
Improve test
9 months ago
e6ddf27072
Unit test sync between chunked loading and Snapshotting
10 months ago
59e928d058
Remove superfluous blank line
10 months ago
df76560168
Sync and report number of loads in progress
10 months ago
b0cd5c5a0c
Add more Load stats
10 months ago
e0f361509a
Handle aborted chunks at lowest level
10 months ago
dd6f386450
Refactor chunk loading
10 months ago
94898ce43e
Track loads in progress
10 months ago
44fa23b714
Use TrailingLogs = 0 to truncate log
10 months ago
264e9822eb
Standardize on chunk size
10 months ago
d726a4c780
Correct and simplify logic
10 months ago
a96112b41d
Snapshot log after one-command load
10 months ago
1f1ebf209f
Don't delete last log
10 months ago
97e9b14bdf
Fix race during testing
10 months ago
8adc11fb83
Snapshot in a goroutine
10 months ago
4b272d04f1
Better use of counters in testing
10 months ago
c39d3472bc
Check snapshot occurs at unit level
10 months ago
c93447e29f
Delete chunked load entries in log after loading
10 months ago
0dcc67f6f1
Perform full snapshot after chunked load
...
This must happen as chunked loading directly changes the SQLite file,
invalidating any existing Snapshots. Also, add missing unit tests for
chunked loading.
10 months ago
31aef91d17
Add FullNeeded() to Snapshot store
...
It may be necessary to override the FullNeeded definition, so wrap it.
10 months ago
8211af961d
Move some VACUUM checks to the Store
10 months ago
5c7996fd98
Support optional VACUUM on backup
10 months ago
a0bf10b021
Move VACUUM to DB layer
10 months ago
f26f15476e
Actually add Provider type
10 months ago
dfb8a60c17
Move Store Provide to a distinct type
10 months ago
3ae4255e62
Backup-related cleanups
10 months ago
4c31c480f5
Better name
10 months ago
5a8a2744ff
Support skipping verifying frame data
...
This halves the Compacting Scan time, and doesn't seem necessary since
rqlite is the only system writing the WAL files.
10 months ago
ee13427c67
Tigher logging
10 months ago
bd9e96ea02
Integrate Compacting WAL writer
10 months ago
d53a34f765
Don't log too much
10 months ago
ba98e88a7b
Update store.go
10 months ago
70d5e6de34
Check join and notify addresses are resolvable
10 months ago
66c1002c32
Remove obsolete metrics
11 months ago
b2663de630
Rename snapshot2 -> snapshot
11 months ago
c11d059551
Better function name
11 months ago
10e6f1b115
Clean up snapshot FDs
11 months ago
93566415e3
Just run bad test
11 months ago
47d877cc25
Don't close dst file until done
11 months ago
09712788c9
Store now builds with snapshot2
11 months ago
6a60108f0c
Start work on Restore and Recover
...
Builds, but still needs more work. It's referencing snapshot(1).
11 months ago
d34c35df31
No need to Store-level WAL selection
11 months ago
5e3d481a6e
Fill out methods on SnapshotStore v2
12 months ago
7c420407c2
WIP snapshot2
...
Close() on the Sink is done (first pass). What's needed next are all the
methods on the Store (Open, List, FullNeeded). I also need to decide
what getSnapshots should really do -- return names or a slice of meta?
12 months ago
974f939ffe
Add an interface between Store and Snapshot Store
...
This will make it easier to swap new implementations in.
1 year ago
be17f2a3cc
Correct variable assignment order
1 year ago
5b4ff9b8ce
Fix test code
1 year ago
53b432a391
More DB and Snapshot stats
1 year ago
59e6a23b02
Clean-up logging
1 year ago
2fff800273
Substantially shrink Test_MultiNodeClusterLargeQueuedWrites
1 year ago
378dd4b2a1
Simpler logging of first application
1 year ago
5ab7069aa0
Update store.go
1 year ago
a4f7431108
Log first time a log is applied
1 year ago
ad86a2989b
Use non-racy function for Leader ID and address
1 year ago
225b5346f5
Add basic stats for Snapshot store
1 year ago
1ffb44cd38
ABL
1 year ago
854fba0e66
Better way to detect previous close
1 year ago
d78d2f4d9d
Short circuit EOF from decompressor
...
The gzip NewReader() seems to hang sometimes when reopened with the old
reader. Therefore just record that we have closed, and never reopen it.
1 year ago
3eb4607a74
GoDoc
1 year ago
a49385e1d3
Add end-to-end compress-decompress test
1 year ago
52aa68497d
Better comment on disabling multistream
1 year ago
06a94e50bf
Raft snapshotting runs to 100%
...
Still failing later.
1 year ago
cb384a9df2
More test tweaks
1 year ago
2395f30ddf
Be sure we don't write footer twice
1 year ago
57eabce899
Protect against double-close of NodeTransport
1 year ago
365abac57c
Simple unit test of NewNodeTransport
1 year ago
fedbdb5478
Enable GZIP snapshot compression
...
Tests are failing.
1 year ago
195981868f
Typo
1 year ago
6efdf9f2b9
Add Decompressor
1 year ago
1fe81713c9
More Compressor tests
1 year ago
55b3aaa805
Add initial version of Gzipping reader
1 year ago
d2c5acc447
Wrap transport as passthru
1 year ago
47e9bbde49
Fix single-node upgrade test
1 year ago
cb989d09a2
Some changes from PR v2
1 year ago
1a24557c19
Merge remote-tracking branch 'origin' into wal-snapshot-store-v3
1 year ago
208fc20659
Fix more uses of random
1 year ago
54713b63f4
Merge remote-tracking branch 'origin' into wal-snapshot-store-v3
1 year ago
c51b8364e8
Fix import path
1 year ago
2288cfe2a9
random only needed by store
1 year ago
c1a8b4a13a
Use global, sync'ed random source
...
Almost unbelievably it appears tests on AppVeyor get the same value when
seeding the random generator. See https://ci.appveyor.com/project/otoolep/rqlite/builds/47853399
1 year ago
c6e214de1a
Remember store attributes
1 year ago
dda86be0d3
Better Store open-and-close logging
1 year ago
c44d020c11
Only attempt to checkpoint once
1 year ago
d6b9c5b39e
Close streams
1 year ago
ee8b23ba94
More logging
1 year ago
e8e0381622
Log when Store closes
1 year ago
a6134b02bc
Instrument the FSM snapshot process
1 year ago
76867a682f
Time FSM snapshot creation
1 year ago
ab6ca783e1
Checkpoint database during Recovery
1 year ago
a9e895bee2
Some cleanups
1 year ago
bd0f78391e
Remove unneeded FSM snapshot code
1 year ago
47476d5b58
Integrate new snapshoting with Store
...
- Snapshot
- Restore
- RecoverNode
1 year ago
5eb4e38cbc
Add comments and errors
1 year ago
e4d5ffb8a9
Flesh our Store check()
1 year ago
48ea4bbfd8
Use a single header
1 year ago
ea8dc3ed18
More test fix-up post removal of in-memory
1 year ago
6895525482
Rename some tests
1 year ago
aa72408bc5
Remove racy use of rand
1 year ago
07d49961d8
Remove unused functions
1 year ago
9e31a054aa
RecoverNode now uses on-disk database
1 year ago
88ce311b37
Remove all WAL-related files when loading new DB
1 year ago
b931e387be
First pass at removing in-memory support
1 year ago
23b8b6e5c9
Actually rename
1 year ago
2ac27c2ffc
Move Store tests back to single directory
1 year ago
65c6a56468
Remove the file
1 year ago
88d448851d
Blank out on-disk store tests
1 year ago
02b7ea7955
Close Store so temp storage can be cleaned up
1 year ago
e47c7a1b7b
Stress-test snapshotting
1 year ago
e497ee75f9
Fix typo
1 year ago
f06e547ab5
Less disk IO during load
1 year ago
f5517c0f66
Correct comment
1 year ago
9bdf048975
Better naming of temp backup file
1 year ago
b5cb889e1e
Auto-restore doesn't require ready() check
1 year ago
b7633143cc
Restore from file uses chunking
1 year ago
5ca5ce342e
Support internode LoadChunk()
1 year ago
Philip O'Toole