wangfenjin
9c4151c37b
move comment down
...
Change-Id: I2f8c435adab5e1c46da2e6b91205ad4df850decc
5 years ago
wangfenjin
9c295ca704
check level before freshness
...
Change-Id: I6d736ad09eac9f003f087adaa9c1f51691fd5c69
5 years ago
Philip O'Toole
536cac4fdf
Allow log level control for Raft module
...
Default log level is INFO.
5 years ago
Philip O'Toole
abb7772105
Support leader freshness with None consistency
...
This change allows the read request to specify the maximum time the node
receiving the request may have last heard from the cluster leader. It
only applies to a read consistency selection of None.
5 years ago
Philip O'Toole
81f5f6faf1
Add support for non-voting nodes
...
A non-voting node doesn't participate in Raft consensus, but does
subscribe to the committed log entries originating with the leader.
This means a non-voting node keeps up-to-date with the state machine,
without impacting write-latency. These non-voting nodes can provide
read scalability for the cluster.
5 years ago
Philip O'Toole
9d595e2ead
ABL
5 years ago
Philip O'Toole
e0cf01cefc
Protect against possible nil pointer
5 years ago
Philip O'Toole
a23af88235
Write backup and dump files to a writer
...
This avoids making an extra copy in RAM. Port of original PR at
https://github.com/rqlite/rqlite/pull/461/ .
5 years ago
Philip O'Toole
f57ace7da2
Broadcast Store meta via standard consensus
...
With this change the cluster metadata (arbitrary key-value data associated with each node) is now broadcast across the cluster using the standard consensus mechanism. Specifically the use case for this metadata is to allow all nodes know the HTTP API address of all other nodes, for the purpose of redirecting requests to the leader.
This change removed the need for multiplexing two logical connections
over the single Raft TCP connection, which greatly simplifies the
networking code generally.
Original PR https://github.com/rqlite/rqlite/pull/434
5 years ago
Philip O'Toole
e376812907
Add method to Store that returns leader ID
...
With the advent of Raft IDs, this distinction matters.
5 years ago
Philip O'Toole
8a0e1ce8f0
Move to Hashicorp Raft v1
5 years ago
Philip O'Toole
7352c376c7
Don't ignore Join error, actually return it
...
Clearly a typo.
5 years ago
Philip O'Toole
085d2d23d8
ABL
5 years ago
Philip O'Toole
0b8cd8536f
Remove apparently duplicated code
5 years ago
Philip O'Toole
0e69be3949
Store layer supports generating SQL format backups
...
Port PR453.
5 years ago
Philip O'Toole
0471877f26
Abort transaction if load operation fails
5 years ago
Philip O'Toole
ccca1ecfa9
Fix DSN key in Store status output
5 years ago
Philip O'Toole
d47808a26b
Add expvar stats to Store
5 years ago
Philip O'Toole
2e91858e1e
Allow Raft election timeout to be set
6 years ago
Philip O'Toole
6937a377ae
Allow control of shutdown-on-remove
...
Defaults to leaving Raft run if the node is removed.
6 years ago
Philip O'Toole
efec4d4e42
Return redirect if remove on follower
...
Fixes issue #391 .
7 years ago
Philip O'Toole
b5b2cb21db
Log Store directory path on startup
7 years ago
Philip O Toole
6bbdc88c1d
ABL
...
Always be linting.
7 years ago
Philip O'Toole
2a1b6c65c3
New QueryRequest and ExecuteRequest types
...
The Query() and Execute() functions on the Store now take a complex type
that encapsulates the statements, and associated parameters. This will
make it easier to add more control parameters when making requests.
7 years ago
Philip O'Toole
303be73dcb
Add missing Store params to diagnostics
7 years ago
Philip O Toole
a59e23cae0
Comment some unit tests
8 years ago
Philip O'Toole
8f2d8e8aa0
Copy JSON peers decode functionality from Raft
...
The main rqlite code needs this logic to check if the Discovery Service
should even be accessed. This code makes assumptions about how the Raft
module works, which is subject to change. The new unit tests should
capture that.
8 years ago
Philip O'Toole
2617962f5f
Typo
8 years ago
WanliTian
4cb5035172
remove store.readPeersJSON
8 years ago
Philip O Toole
aa5c946a40
Linting
8 years ago
Philip O'Toole
851f486ff8
Make waiting for logs optional
8 years ago
Philip O'Toole
059ab0757b
Allow Store to wait for initial logs to be applied
...
Better fix for issue #260 .
8 years ago
Philip O Toole
4b6c936d53
Fix comment typos
8 years ago
Philip O Toole
ae0d119d83
Allow Raft Apply timeout to be configurable
8 years ago
Philip O Toole
5ed6ef0ae8
Covert dumps to queries in HTTP service
...
There is no need for special logic at the Store level.
8 years ago
Philip O Toole
e6586f41f2
Use single load for dump processing
...
Writing a sufficiently sophisicated parser is too much work, and unlikely to be successful soon. Instead this change simply loads the entire dump as one command, and allows the underlying SQLite support to parse it correctly. This will definitely work, but since the load goes over Raft, it may hit limits with regards to network transfer sizes. Right now any limitations in that area are unknown.
Therefore this functionality remains somewhat experimental.
8 years ago
Philip O Toole
098578bda8
Clarify guarantees of some functions
8 years ago
Philip O Toole
c2ab24378c
Lint fixes
8 years ago
Philip O Toole
2fb99e5b20
Use Store logger everywhere in Store
8 years ago
Tycho Andersen
dc2128d02a
add a raw snapshotting API
...
This allows us to grab a raw snapshot of the databse.
v2: s/RawDBSnapshot/Database, remove reference to raft from the comment.
v3: add a leader arg to Database to do a leader check
v4: fix doc string.
Signed-off-by: Tycho Andersen <tycho.andersen@canonical.com>
8 years ago
Tycho Andersen
326ca757d7
add an API for using custom raft.PeerStore/log.Logger
...
This is useful in case the server needs to store other metadata (e.g. auth
data) along side the peer list.
The logging bit is handy in case something has its own logging framework
that it wants to use.
Signed-off-by: Tycho Andersen <tycho.andersen@canonical.com>
8 years ago
Philip O'Toole
7ac3450ab2
Suport observer deregistration
8 years ago
Tycho Andersen
aa47089a32
expose raft's observer API via store
...
Signed-off-by: Tycho Andersen <tycho.andersen@canonical.com>
8 years ago
Tycho Andersen
4cf094c6a4
store: use ErrNotLeader everywhere
...
Signed-off-by: Tycho Andersen <tycho.andersen@canonical.com>
8 years ago
Philip O'Toole
4242a641b1
Fix typo in comment
8 years ago
Philip O'Toole
da21606644
Move chinook package to idiomatic testdata dir
...
https://github.com/golang/go/issues/14715
8 years ago
Philip O'Toole
634f55211c
Unit test with Chinook database
...
https://chinookdatabase.codeplex.com/
8 years ago
Philip O'Toole
eda120ffaa
Correct unit tests
8 years ago
Philip O'Toole
c00f1b460b
Accept BEGIN and COMMIT on load()
8 years ago
Philip O'Toole
39ebf50cb7
Remove explicit control of FK constraints
...
It's not clear whether it's the right thing to do, and FK constraints
can still be controlled via PRAGMA commands.
8 years ago
Philip O'Toole
9ffa309f69
Stop database-level control of FK during load
8 years ago
Philip O'Toole
ed304b7840
Remove useless comment
8 years ago
Philip O'Toole
f87ee412e2
Correct unit tests
...
'PRAGMA foreign_keys=OFF' is no longer filtered.
8 years ago
Philip O'Toole
36368d6786
Start removing batch load
8 years ago
Philip O'Toole
68793cfcbb
Use a transaction for loaded batch
...
Bump the batch size to 1000 too.
8 years ago
Philip O Toole
b95af88367
Test SQL loading with blank lines
8 years ago
Philip O Toole
f44c08a530
Skip loading blank lines
8 years ago
Philip O Toole
9c180d8f43
SQL parser now strips newlines
8 years ago
Philip O Toole
1f5cfc65d8
Trim leading and trailing newlines
8 years ago
Philip O Toole
84f3be9c10
Add missing package
8 years ago
Philip O Toole
b87de293ce
Move \n ; trimming to SQL line parser
8 years ago
Philip O Toole
6ee9cdbbdf
Start integration of SQL line parser
8 years ago
Philip O Toole
3726d1bb20
Add DB conf to diagnostics
8 years ago
Philip O Toole
cb7686fa46
Diagnostics show actual FK constraint state
8 years ago
Philip O Toole
2daac8704b
Support batching during load
8 years ago
Philip O Toole
38f9b6ff20
Test loading no commands
8 years ago
Philip O Toole
ab743b1ead
Test loading an empty file
8 years ago
Philip O Toole
ccd48161f3
Unit test Store Load()
8 years ago
Philip O Toole
31560160a1
Check FK state before load
8 years ago
Philip O Toole
4c6ee1ad39
Return error code during load if necessary
8 years ago
Philip O Toole
6007790ae4
Break loading when done
8 years ago
Philip O Toole
3286bf064d
Start adding load-from-dump
8 years ago
Philip O'Toole
729221bf57
join endpoint returns HTTP 301 if necessary
8 years ago
Philip O Toole
95cb4baf2a
More cyclo changes
8 years ago
Philip O Toole
7bdedf8cd9
Reduce cyclo complexity of opening store
8 years ago
Philip O'Toole
c40fff4d46
Support controlling FK constraints at startup
...
This was possible previously, but would need to be set everytime on
startup via the API. This change allows it to set at startup AND enables
foreign constraint checking by default.
8 years ago
Philip O'Toole
7dd164915b
Check database errors at store level
8 years ago
Philip O'Toole
0b13a1321d
Simplify error handling for Update Peers
8 years ago
Philip O Toole
aa467cffcd
Fix misspelling in comment
8 years ago
Philip O'Toole
2982058a81
Support setting Raft heartbeat timeout
8 years ago
Philip O'Toole
8a05236768
Support configurable snapshot thresholds
8 years ago
Benjamen Keroack
b67e56196e
add cluster state and leader functions to store
8 years ago
Philip O'Toole
83026e4612
Add Peers to status output
8 years ago
Philip O'Toole
47539cf8e1
Add RemoveNode() to Store
8 years ago
Philip O'Toole
ef18cd5b17
Expose Raft Peers() on Store
8 years ago
Philip O Toole
bf2047b45b
'go lint' fixes
9 years ago
Philip O'Toole
8c0fee4b06
Include Raft dir in status
9 years ago
Philip O'Toole
5380c4d7e4
Map Raft "not leader" error to Store error ( #112 )
9 years ago
Philip O'Toole
c04ff92d7a
Fill our API peers lookups ( #110 )
9 years ago
Philip O'Toole
fac48629bc
Start initial integration of mux ( #109 )
...
Integrate TCP mux with cluster and store
This change allows any node, including followers, to use the Raft log to make changes to a cluster-wide state.
9 years ago
Philip O Toole
5aefda9983
Move to new rqlite org
9 years ago
Philip O'Toole
e8c92e088c
Don't join unless actually necessary
9 years ago
Philip O'Toole
a02f44f089
Remove cluster-like code from Store
9 years ago
Philip O'Toole
9164550aab
Remove debug print
9 years ago
Philip O'Toole
bdeda47774
Initial implementation of Meta server
...
This should really be a distinct cluster package.
9 years ago
Philip O'Toole
0a3454a675
Methods on store to set and get API peers
9 years ago
Philip O'Toole
0bffa38319
Start implementing cluster meta comms
9 years ago
Philip O'Toole
16e3125149
Add support for cluster meta
...
This change required that the cluster meta become part of the Raft
snapshot.
9 years ago
Philip O'Toole
99ebaf9767
Add support for peersSub
9 years ago
Philip O'Toole
ed56f77714
Add hierarchy to Raft commands
9 years ago
Philip O'Toole
099c7c18ac
Enable TCP muxing layer for Raft commns
9 years ago
Philip O'Toole
ef554e33a0
Update sqlite3 status output
9 years ago
Philip O'Toole
a11b5fe2fe
Unit test restoring in-memory databases
9 years ago
Philip O'Toole
91a552b7ec
Further store-layer updates
9 years ago
Philip O'Toole
31cda32a38
'go fmt' fixes
9 years ago
Philip O'Toole
1ea4edd515
Store layer adapted to new DB layer
9 years ago
Philip O Toole
c07c11f134
Fix trivial typo
9 years ago
Philip O Toole
cafbc0722b
'go fmt' fixes
9 years ago
Philip O Toole
940c872dfd
Explicitly test file and in-mem
9 years ago
Philip O'Toole
b26bc3444c
Unit test snapshot and restore
9 years ago
Philip O'Toole
7fd7a9a385
Finish multi-node cluster testing
9 years ago
Philip O'Toole
5e688bab93
Clarify it's localhost
9 years ago
Philip O'Toole
cc3c75d9bf
Re-enable multi-node testing on CirceCI
9 years ago
Philip O'Toole
5bd3ab574d
Idiomatic network layer function names
9 years ago
Philip O'Toole
9693d7dc95
Obey 'timings' flag at database level
9 years ago
Philip O Toole
5f5a666a06
Move to "weak" and "strong" for consistency levels
9 years ago
Philip O Toole
1a16588450
Fix 'timings' JSON tag
9 years ago
Philip O Toole
ac92530a6d
Add sqlite file size to status
9 years ago
Philip O Toole
db1fb33fd3
Allow timing control
...
Fixes issue #75 .
9 years ago
Philip O Toole
8f674219eb
Re-enable timings
9 years ago
Philip O Toole
1649b83d14
Fix unexpected command 'typ'
9 years ago
Philip O Toole
11c18cd769
Service and store support consistency levels
...
Fix issue #5 .
9 years ago
Philip O Toole
20a05fd0a1
Refactor Raft command to support queries
9 years ago
Philip O'Toole
7b628686bf
Add SQLite version to diagnostics
9 years ago
Philip O'Toole
62c17291b4
More diagnostic information
9 years ago
Philip O Toole
e7e212411a
Ensure Raft dir exists
9 years ago
Philip O'Toole
4524d014eb
Leader redirection almost complete
9 years ago
Philip O'Toole
9d614b7c8d
Add explicit error for "no leader"
9 years ago
Philip O'Toole
3f1a3b3777
Fix Raft snapshots of in-memory databases
...
The snapshot code assumed all databases were disk-based.
9 years ago
Philip O'Toole
f28c0b650e
Precise checking of CircleCI
...
Issue #64 .
9 years ago
Philip O'Toole
534a1bd59b
Unit test transactions at the store level
9 years ago
Philip O'Toole
8cb852d1cf
Prettier store address in status
9 years ago
Philip O'Toole
59052ec0b4
More status output
9 years ago
Philip O'Toole
e88f97ed06
Support cluster-level leader checks for queries
9 years ago
Philip O Toole
b282aede1b
'go fmt' fixes
9 years ago
Philip O Toole
a653a5feb5
Skip multi-node test on CircleCI
...
Seems like there might be something special about their networking.
9 years ago
Philip O Toole
ce0a2ce269
Usage and comment fixes
9 years ago
Philip O Toole
f1963221ca
Small fixes and formatting
9 years ago
Philip O Toole
cc73f0abbc
Default to leader check before backup
9 years ago
Philip O Toole
2e7ae59245
Complete multi-node replication test
9 years ago
Philip O Toole
b058458a0f
Start adding multi-node tests
9 years ago
Philip O Toole
347109e4bb
More unit tests of store
9 years ago
Philip O Toole
d0f689c07c
Start adding store-level unit tests
9 years ago
Philip O Toole
81a3af3b7e
Allow leader check for queries
9 years ago
Philip O Toole
3ffed779f8
Move to POST for bulk queries
...
Backup endpoint also added, but not fully-functional.
9 years ago
Philip O Toole
170a916dd5
fmt and lint fixes
9 years ago
Philip O Toole
9c02e1edb6
Make it easier to understand DSN
9 years ago
Philip O Toole
4ff67c2c90
Support in-memory databases
9 years ago
Philip O Toole
2ba973d7b7
Allow control of time display from db layer
...
This is not a great solution, and somewhat of a hack to make
unit-testing easier. However, it will allow control over "time"
in the response in the future.
9 years ago
Philip O'Toole
a8a03ddb07
Fix comment typo
9 years ago
Philip O Toole
7c75bd9494
Finalize name for sqlite file
9 years ago
Philip O Toole
17aea81d75
Add Raft statistics
9 years ago
Philip O Toole
ee4ffe1d15
Comment cleanups
9 years ago
Philip O Toole
b638fa7b87
Finish snapshot restore
...
Compiles, but not tested.
9 years ago
Philip O Toole
a47da4065f
Correctly handle database removal
9 years ago
Philip O Toole
35803cdf75
Shutdown Store on exit
9 years ago
Philip O Toole
a6cb950ff4
Allow DSN config at command line
...
It might still need to be richer, so end-users could specify an
in-memory SQLite database. Specifying a DSN would require the user
to supply the full path to the SQLite database. This is OK. However,
the code then needs to be able to parse out the path to the database
so it can remove it before start up.
9 years ago
Philip O Toole
179ad650f3
Apply() and Snapshot NOT called concurrently
...
No reason to worry about this.
9 years ago
Philip O Toole
2ebec9b973
Start adding support for snapshots
...
This shows that passing the database into the Raft module is probably
not going to work, since the database could be opened in two ways -
directly at startup, and be completely restored from the log, or with
a combination of restoring from a snapshot, followed by the remaining
log entries. In both cases the database must be opened using the
requested DSN settings.
A detailed config object for controlling SQLite behaviour, is probably
best, and it should be passed to the Raft store on start up.
Finally, file-level copying of the SQLite file can only take place if
no transaction is in effect. This might be handled by the use of a
RWLock. The write-lock is taken during Execute() and Snapshot, but
the Read lock is taken during Query(). Unfortunately this may reduce
the concurrency of inserts and updates. Perhaps the Backup call on
the Go SQLite library might be better, but it might be slow.
9 years ago
Philip O Toole
d30d2fffa4
Query path now works
9 years ago
Philip O Toole
ab71c3bdd2
Execute working now
9 years ago
Philip O Toole
639264db4f
Hook up handleExecute
9 years ago
Philip O Toole
28f4a73773
Simplify use of ApplyFuture
9 years ago
Philip O Toole
3ab8e01d91
Start threading ApplyFuture for Execute response
9 years ago
Philip O Toole
d70a9cb0ed
Push database into Raft store
...
Allows the DSN to be fully set, if requested.
9 years ago
Philip O'Toole
99f5e63f49
Build fixes
9 years ago
Philip O'Toole
73d880c93c
Start support for SQLite DSN
...
This isn't complete, since it prepends the Raft directory to the
filename, which would break a DSN.
9 years ago
Philip O'Toole
77b9077d9d
Pass Execute results up
...
Doesn't compile.
9 years ago
Philip O'Toole
63c74cabea
Fully build with new db package
9 years ago
Philip O'Toole
f6984f4134
Small wrapper around sqlite3
9 years ago
Philip O'Toole
4f110e26ea
Update README with desired behavior
9 years ago
Philip O'Toole
b95c84d16e
Add database interface to Store
...
CREATE TABLE now works.
9 years ago
Philip O'Toole
e43b0bc421
Initialize SQLite store
9 years ago
Philip O'Toole
12aef34625
Start integrating actual SQLite support
...
Builds without error.
9 years ago
Philip O'Toole
d7d46f79c0
Start refactor for Hashicorp
...
Heavily influenced by hraftd. rqlited builds without error.
9 years ago