Philip O'Toole
316ee52b4c
Fill in empty types using actual queried data
1 year ago
Philip O'Toole
36fa7f2afd
Detect DB stats error and log
1 year ago
Philip O'Toole
c552dba575
Record how long checkpointing takes
1 year ago
Philip O'Toole
b64efbdee1
Better testing
1 year ago
Philip O'Toole
539e578227
Add WAL-validity checking functions
1 year ago
Philip O'Toole
d940779045
More unit testing of WAL-replay
1 year ago
Philip O'Toole
50fbdc73fd
Tigther code
1 year ago
Philip O'Toole
a9796eeb36
Better testing
1 year ago
Philip O'Toole
6cd8d30749
Fix Store-level database-backup test
1 year ago
Philip O'Toole
ca67ee4cff
Remove debug statement
1 year ago
Philip O'Toole
68fc185c76
More WAL DB open testing
1 year ago
Philip O'Toole
19d83ff1ca
More enforcement of DELETE mode in-and-out
1 year ago
Philip O'Toole
0463a92fde
Fix Serialize() to work with WAL mode
...
This function is even more resource (disk, RAM) intensive, but it's goal
is to be correct, not efficient.
1 year ago
Philip O'Toole
4e639d4f17
Add more detail to comments
...
Serialize() is problematic with WAL mode databases. It needs close study
and may have to be retired before enabling WAL mode. For example
Serialize() is called by Snapshotting. Snapshotting would need to move
to Backup, at for on-disk mode.
1 year ago
Philip O'Toole
33b56af639
Fix Backup for WAL mode
1 year ago
Philip O'Toole
3f462eecfa
Remove all on-disk SQLite files on startup
1 year ago
Philip O'Toole
26ad4913c9
Correct comment
1 year ago
Philip O'Toole
6b32b7f4d2
Add function which returns size of WAL
1 year ago
Philip O'Toole
7e1b8d12d7
Add full WAL support to DB layer
1 year ago
Philip O'Toole
df33a4aa42
DB layer supports enabling WAL mode
1 year ago
Philip O'Toole
da11903312
Correct test failure error
1 year ago
Philip O'Toole
67941699b6
Add some WAL-related control code
...
Not actually called yet.
1 year ago
Philip O'Toole
5e37ff9c55
Fix typo in variable name
1 year ago
Philip O'Toole
f58bd0cf7b
Add some important PRAGMA state to DB stats
1 year ago
Philip O'Toole
dd9b3f2695
PRAGMA synchronous=OFF for on-disk databases
1 year ago
Philip O'Toole
4a7ac2850f
rqlite doesn't (yet) support WAL mode SQLite data
1 year ago
Philip O'Toole
98a356504e
Fix import cycle
1 year ago
Philip O'Toole
ca12df38ce
Test cleanup
1 year ago
Philip O'Toole
2fd62f3f64
Isolation testing only applies to on-disk
1 year ago
Philip O'Toole
d8f3a72365
Common DB-layer tests
1 year ago
Philip O'Toole
ca6f071fb5
Split up DB-level unit testing
1 year ago
Philip O'Toole
5ff28d354f
Set Error on correct field
1 year ago
Philip O'Toole
9924fe75f5
Some small improvements
1 year ago
Philip O'Toole
3058d3e9fc
Fix transaction handling in db.Request()
1 year ago
Philip O'Toole
59786e818b
Initial unit tests for db.Requests()
...
Seems to work well.
1 year ago
Philip O'Toole
4cbb541e9b
DB Request code-complete except for transactions
1 year ago
Philip O'Toole
fe176a9752
Merge branch 'query-exec-stmt-with-conn' into unified-endpoint
1 year ago
Philip O'Toole
497c6b3340
Elevate queryer type
1 year ago
Philip O'Toole
3d652d929e
More factoring of DB query layer
1 year ago
Philip O'Toole
de70664bf1
More code in db.Request()
1 year ago
Philip O'Toole
6fcf85ef40
Flag major issue
1 year ago
Philip O'Toole
1987f9b07b
Flesh out db.Request
...
Needs full unit testing.
1 year ago
Philip O'Toole
b2b0b188e9
Factor out functions
1 year ago
Philip O'Toole
014559bb06
Another DB read-only test
1 year ago
Philip O'Toole
f84b258f99
Better variable name
1 year ago
Philip O'Toole
c3e7044945
Final read-only test
1 year ago
Philip O'Toole
2af5fc26aa
Move comment
1 year ago
Philip O'Toole
8f45c4c8c9
Factor out read-only check
1 year ago
Philip O'Toole
bb868595c1
Start prepping DB layer
1 year ago
Philip O'Toole
92a604c4ec
Move SQLite validity checks to db module
1 year ago
Philip O'Toole
fb1e5e3903
Use correct errors (thanks GPT!)
1 year ago
Philip O'Toole
95a689b3cf
Some improvements thanks to ChatGPT 4
2 years ago
Philip O'Toole
c1ba9157c2
More unit test of nil parameters
2 years ago
Philip O'Toole
41bf265b02
Fix more handling of nil named values
2 years ago
Philip O'Toole
053bde0ed6
Upgrade to latest SQLite driver
2 years ago
Philip O'Toole
3d555a73cb
Add unit test showing malformed database issue
...
https://github.com/rqlite/rqlite/issues/1103
2 years ago
Philip O'Toole
0702e39ace
Add ResetStats
2 years ago
Philip O'Toole
e3cc4eb5d0
go mod updates ( #1120 )
...
* go mod updates
* CHANGELOG
2 years ago
Philip O'Toole
e74bfc182a
Use new Encoder everywhere
2 years ago
Philip O'Toole
0071867e34
Don't call t.Fatal() from goroutine
...
It wouldn't abort the test anyway, so just log.
2 years ago
Eng Zer Jun
5141eff5fc
test: use `T.TempDir` to create temporary test directory
...
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2 years ago
Philip O'Toole
54aec1ca33
Tighten up backup code
2 years ago
Philip O'Toole
221907f8a9
Support nil as paramterized values
2 years ago
Philip O'Toole
c8b9fd72da
Sanity check supplied SQLite data
2 years ago
Philip O'Toole
c91ca45ba4
Expose more DB attributes
2 years ago
Philip O'Toole
a6a3567a4d
DB layer returns in-memory status
2 years ago
Philip O'Toole
d84fb85f01
Don't HTML escape JSON output
3 years ago
Philip O'Toole
869f3547b7
Update to SQLite 3.38
3 years ago
Philip O'Toole
ebb2ff32c2
Send rand() once in DB layer
3 years ago
Philip O'Toole
24a5dc8c4b
Much better random file names
3 years ago
chermehdi
3bfcbffc37
Fix typos and Grammar issues
3 years ago
Philip O'Toole
7038b86dcb
One final named params unit test
3 years ago
Philip O'Toole
01849e77d5
Add db-level support for named paremeters
3 years ago
Philip O'Toole
239cdc04b5
Don't run heavy memory tests in parallel
...
Let's see if this helps these tests run faster, especially with -race,
and prevent CircleCI timeouts.
3 years ago
Philip O'Toole
eab9183acd
Better comment in DB code
3 years ago
Philip O'Toole
51bb54488b
Explicitly handle column type nil (null)
3 years ago
Philip O'Toole
c639f21cde
Return time types in RFC3339 format
3 years ago
Philip O'Toole
28063bfa0d
Support timestamp field
3 years ago
Philip O'Toole
53e4b9ae3d
ABL
3 years ago
Philip O'Toole
437d76bc1c
Update db.go
3 years ago
Philip O'Toole
bed9dc2a49
Update db_test.go
3 years ago
Philip O'Toole
393a1c67a0
Fix unit tests
3 years ago
Philip O'Toole
3ed0909746
Read-only statements in query endpoint
3 years ago
Philip O'Toole
519a91e635
Actual recovery node test passes on Linux
3 years ago
Philip O'Toole
2bf7638a8d
Fix debug code in test
3 years ago
Philip O'Toole
01895e501f
Serialize on-disk databases by simply reading file
...
This is safe according to https://sqlite.org/howtocorrupt.html and
avoids any sqlite3_serialize() limitations.
3 years ago
Philip O'Toole
c0b22b0d03
Update CHANGELOG
3 years ago
Philip O'Toole
08eb74493d
Add unit test for >1GiB in-memory databases
...
>2GiB in-memory are still not supported.
3 years ago
Philip O'Toole
44e47686ee
Tests that demonstrate current isolation behavior
3 years ago
Philip O'Toole
a4c4374daf
Update CHANGELOG
3 years ago
Philip O'Toole
3907132761
Add detailed SQLite memstats to status/
3 years ago
Philip O'Toole
aa8df67592
DB-level unit tests pass
3 years ago
Philip O'Toole
161abb094d
Add db-level FK constraints unit test
3 years ago
Philip O'Toole
deb4dabf2e
Thread through foreign key constraint controls
3 years ago
Philip O'Toole
2cad491f40
Better creation of SQLite DSNs
3 years ago
Philip O'Toole
f7b390ab28
Adding JSON1 unit test
3 years ago
Philip O'Toole
96f9e85ee3
More concurrent query testing
3 years ago
Philip O'Toole
3cee11de6f
Count number of query errors
3 years ago
Philip O'Toole
6c69e8a9b5
Upgrade to new compile options
3 years ago
Philip O'Toole
ea723ee830
Ping databases on open to ensure full availability
3 years ago
Philip O'Toole
953e82f7c4
Add SQLite compile-time options to status output
3 years ago
Philip O'Toole
11da97f455
Fix typo
3 years ago
Philip O'Toole
497d3dd873
Remove obsolete foreign key constraint control
3 years ago
Philip O'Toole
c086eba427
Remove debug statement
3 years ago
Philip O'Toole
b83424484d
Enhance SQLite database status
3 years ago
Philip O'Toole
cc8b924e64
Fix WaitGroup usage in test
3 years ago
Philip O'Toole
4f2818d037
Add concurrent query test
3 years ago
Philip O'Toole
eec7494d62
DB size requires a read-write connection
3 years ago
Philip O'Toole
6edff5ba24
Deal with SQLite connection "caching"?
...
https://github.com/mattn/go-sqlite3/issues/966
3 years ago
Philip O'Toole
bf51947ca9
Better error messages
3 years ago
Philip O'Toole
d101e10469
Test writing via the query connection
...
It should fail.
3 years ago
Philip O'Toole
854cf362fd
Don't specify mode=rw on file open
...
Not sure why.
3 years ago
Philip O'Toole
6dc9583dab
Implement concurrent reads and writes
...
Disk doesn't support it yet, as it has not moved to WAL mode. Perhaps
wait for next release to do that.
3 years ago
Philip O'Toole
db1768c63f
Initial implementation of two database connections
3 years ago
Philip O'Toole
90a0d397c6
Switch from mode=memory to vfs=memdb for in-mem DB
3 years ago
Philip O'Toole
791a1b0b72
Remove obsolete references to DSN
3 years ago
Philip O'Toole
e0da96dd55
Remove support for specifying SQLite DSN
...
This is a breaking change. However this feature hasn't been tested and
allows end-users to break the system too easily. Low-level control over
the SQLite database is better done with PRAGMA commands where possible.
3 years ago
Philip O'Toole
2924ed407d
Upgrade to rqlite/go-sqlite3 v1.20.4
3 years ago
Philip O'Toole
028419af0f
Fix race in in-memory database testing
3 years ago
Philip O'Toole
1ee7a9eb17
Limit in-memory connection pool to 1 connection
...
https://github.com/mattn/go-sqlite3/issues/959#issuecomment-890283264
3 years ago
Philip O'Toole
911df4c03b
Don't leak a database if deserialization fails
...
Fixes https://github.com/rqlite/rqlite/issues/837 .
3 years ago
Philip O'Toole
8693694d14
Add current SQLite journal mode to status output
3 years ago
Philip O'Toole
503825dbaf
Remove Abort functionality
...
No longer relevant since each Exec and Query gets its own connection.
3 years ago
Philip O'Toole
561bb47a14
Port DB layer to sql/database
...
Empty statements now cause panic.
3 years ago
Philip O'Toole
eb8dcf74ff
Start moving to SQL layer
...
Builds.
3 years ago
Philip O'Toole
5f5f7dbbea
Sync use of DB connection
...
Possible, albeit simple, fix for https://github.com/rqlite/rqlite/issues/830
3 years ago
Philip O'Toole
c5e67a548c
Even more records during db test
4 years ago
Philip O'Toole
665d9913bb
Add top-level Deserialize() to db layer
4 years ago
Philip O'Toole
b83b2b241d
Merge branch 'master' into no-disk-during-restore
4 years ago
Philip O'Toole
18a4f38701
Add Copy() method to db layer
4 years ago
Philip O'Toole
ce7a3a6912
Merge branch 'master' into no-disk-during-restore
4 years ago
Philip O'Toole
535d690d7c
Tigthen up Store-level locking
...
Previous locking didn't look quite correct, and with the advent of the
Serialize functionality, it needs to be more precise.
4 years ago
Philip O'Toole
7900adbf53
Add Deserialize to DB layer
4 years ago
Philip O'Toole
9210297413
Unit test DB serialize
4 years ago
Philip O'Toole
93a78f09a0
Add call to Serialize function
4 years ago
Philip O'Toole
488a9554ea
Switch to rqlite fork of go-sqlite3
4 years ago
Philip O'Toole
011bc00784
Add SQLite database page-centric size to status output
4 years ago
Philip O'Toole
6575de779d
Use Protobuf for encoding Raft Log commands
...
This PR changes Raft Log Entry encoding from JSON to Protobuf. Furthermore, larger Raft commands (which can result from batching SQL statements, or individually long SQL statements) are compressed before encoding.
This primary reason for this change is to reduce IO load since that is one of the largest performance bottlenecks. It will also reduce internode traffic.
Legacy JSON-encoded commands are still handled by this code, so this change is backwards-compatible with previous releases in the v5 series.
4 years ago
Philip O'Toole
b4867713e4
Better variable name, Query -> SQL
4 years ago
Philip O'Toole
ee9f929735
ABL
4 years ago
Philip O'Toole
f961bb4e44
Fix typos and comments
4 years ago
Philip O'Toole
69ab554b58
Unit test parameterized statements
4 years ago
Philip O'Toole
1075315a11
DB Query now accepts complex Statements
4 years ago
Philip O'Toole
db992d1dce
DB Execute now accepts complex Statements
4 years ago
Philip O'Toole
3cfb0c3910
JSON types are also text
5 years ago
Philip O'Toole
b283ecc86d
Ensure backup is correctly closed
...
Also catch any error that occurs during close.
5 years ago
Philip O'Toole
0e69be3949
Store layer supports generating SQL format backups
...
Port PR453.
5 years ago
Philip O'Toole
43ea24fb82
DB layer supports generating SQL format backups
...
Port of PR451.
5 years ago
Philip O'Toole
0471877f26
Abort transaction if load operation fails
5 years ago
Philip O'Toole
6d5d4b3d81
Fix unit tests due to SQLite master table changes
7 years ago
Philip O Toole
6bbdc88c1d
ABL
...
Always be linting.
7 years ago
Philip O'Toole
bca0e43218
Unit test JOIN at the database level
7 years ago
Philip O'Toole
20f0eb505c
Add test showing OK PRAGMA table_info
8 years ago
Philip O'Toole
86a59adc57
Unit test query of SQLite master table
8 years ago
Philip O Toole
ffffd99f33
Test Common Table Expressions at database layer
8 years ago
Philip O Toole
b0e7432dc3
Check for nil row from Execer
8 years ago
Philip O'Toole
9a7ff4c979
Handle "" types
...
Add unit test for concatenated queries. This deals with issue #240 but
this may be a hack. I need to dig into this issue more.
8 years ago
Philip O'Toole
75417f6d1e
Start handling the various SQLite TEXT types
...
This is the start of a code change to handle this comprehensively.
8 years ago
Philip O'Toole
d140b1e8c7
More database-level stats
8 years ago
Philip O'Toole
48f7bb2c50
Add low-level DB expvar stats
8 years ago
Philip O Toole
efc49511a5
Test multiline statements at DB level
8 years ago
Philip O Toole
6815f09873
Database can return FK constraint state
8 years ago
Philip O'Toole
1de7848362
Clean up and simplify DB tests
8 years ago
Philip O'Toole
6b3704a9fa
Add db test helper functions
8 years ago
Philip O'Toole
60443bee0b
Reduce test cyclo complexity
8 years ago
Philip O'Toole
cb12dc17bf
Reduce cyclo complexity of query code
8 years ago
Philip O'Toole
9fce85b059
Add UNIQUE constraint test
8 years ago
Philip O'Toole
acb592aa0b
Tighten logic for FK constraint control
8 years ago
Philip O'Toole
c40fff4d46
Support controlling FK constraints at startup
...
This was possible previously, but would need to be set everytime on
startup via the API. This change allows it to set at startup AND enables
foreign constraint checking by default.
8 years ago
Philip O'Toole
56778f1764
'go lint' fixes
8 years ago
Philip O'Toole
b248b0f4e1
Trivial formatting changes
8 years ago
Benjamen Keroack
3f064afc31
db: only convert row value to string if the column type is text
8 years ago
Philip O Toole
bf2047b45b
'go lint' fixes
8 years ago
Philip O'Toole
6fde29ff62
Fix typos
9 years ago
Philip O'Toole
6405dc01a8
Add function for loading in-memory db from file
9 years ago
Philip O'Toole
31cda32a38
'go fmt' fixes
9 years ago
Philip O'Toole
f4c9ff0c3b
Start with new DB layer
9 years ago
Philip O'Toole
721445acd0
Update db.go
9 years ago
Philip O'Toole
9b4f0d409e
More package doc for db
9 years ago
Philip O'Toole
2b08c194ea
Update db.go
9 years ago
Philip O'Toole
7b628686bf
Add SQLite version to diagnostics
9 years ago
Philip O Toole
1554934e88
'go fmt' fixes
9 years ago
Zac Medico
285c8d4be6
Use sqlite3.SQLiteDriver directly, fix issue #73
9 years ago
Philip O Toole
300d58cfd3
'go fmt' fixes
9 years ago
Zac Medico
9f348a6171
Use SQLiteRows.DeclTypes() to fix issue #52
9 years ago
Zac Medico
f62787624c
Bypass sql.DB for queries (for direct SQLiteRows access)
...
It's necessary to bypass sql.DB in order to gain access to SQLiteRows
instances, which is the first step toward inclusion of result column
type declarations in query results for issue #52 .
9 years ago
Philip O'Toole
3f1a3b3777
Fix Raft snapshots of in-memory databases
...
The snapshot code assumed all databases were disk-based.
9 years ago
Philip O Toole
f1963221ca
Small fixes and formatting
9 years ago
Philip O Toole
3ffed779f8
Move to POST for bulk queries
...
Backup endpoint also added, but not fully-functional.
9 years ago
Philip O Toole
524e6170bf
Final tests of backup
9 years ago
Philip O Toole
8b3c4202cc
Work for database backup
...
Tests are failing.
9 years ago
Philip O Toole
fac68c3cdd
Return time in seconds
9 years ago
Philip O Toole
170a916dd5
fmt and lint fixes
9 years ago
Philip O Toole
4ff67c2c90
Support in-memory databases
9 years ago
Philip O Toole
2ba973d7b7
Allow control of time display from db layer
...
This is not a great solution, and somewhat of a hack to make
unit-testing easier. However, it will allow control over "time"
in the response in the future.
9 years ago
Philip O Toole
f33ffc2936
Measure time per statement
9 years ago
Philip O Toole
1b41f789b8
Remove debug panic()
9 years ago
Philip O Toole
297479bc4e
Make multi-SELECT work
9 years ago
Philip O Toole
ee4ffe1d15
Comment cleanups
9 years ago
Philip O Toole
a6cb950ff4
Allow DSN config at command line
...
It might still need to be richer, so end-users could specify an
in-memory SQLite database. Specifying a DSN would require the user
to supply the full path to the SQLite database. This is OK. However,
the code then needs to be able to parse out the path to the database
so it can remove it before start up.
9 years ago