f6b3f8dc5c
Simplify encoding checks across actions
...
This commit also removes the forceful `__private` module naming for the
`dbtest` proc macro. Supplied modules can now have any name.
3 years ago
c298f55b23
Use primitive casts to reduce jumps
3 years ago
f4fbdcae16
Fix encoding check correctness in actions
3 years ago
6fe3f53aa9
Use ptr offsets instead of index
3 years ago
947327f379
Simplify response writing/handling ( #222 )
...
* Use `BoolTable` to simplify resps
* Fix inversion of table
* Use BLUT and NLUT wherever possible
3 years ago
48ff6003af
Remove `keylen` and add `exists` and `del` tests
...
`keylen` checks the length of value for a given key. This cannot work
with listmaps
3 years ago
b930c5d31b
Add listmap compatibility to `keylen` and `lskeys`
3 years ago
68b9c9b81f
Add listmap compatibility with `exists`
3 years ago
8ba3cb8028
Add `del` compatibility with listmaps
3 years ago
ca77ca9f80
Add sanity tests for wrong-model error string
3 years ago
7246b14115
Add `lmod insert` tests
3 years ago
ed01c2fa82
Add `lmod remove` tests and fix `lget` tests
...
The error string length was incorrectly set causing the `lget` test to
fail
3 years ago
d0ee66ac4c
Add remaining `lget` tests and add `lmod` tests
3 years ago
ce99e974b9
Fix array extend impl ptr eq check
...
The end of allocation is reached when curptr > endptr
3 years ago
acef265b04
Fix list tests
3 years ago
8d43a70bd2
Add `lget` and `lset` tests
3 years ago
677c4a844a
Merge branch 'model/impl-lists' into next
3 years ago
9b4b140af7
Add listmap on-disk storage tests
3 years ago
41a49aaf11
Add `POP` subaction to `lmod`
3 years ago
9a2622c2ad
Add `insert` subaction to `lmod`
3 years ago
02628d6ff3
Add `REMOVE` subaction to `lmod`
3 years ago
c66206b9eb
Add `PUSH` subaction to `lmod`
3 years ago
0278f6063d
Add `lmod` and `CLEAR` subaction
3 years ago
5238298ac4
Add `VALUEAT` and `LIMIT` subactions to `lget`
3 years ago
0e8d60df0a
Add `lget` for getting all list elements and len
3 years ago
a04f676aa5
Add `lset` for list creation
3 years ago
26ed51a077
Simplify modelcode computation
3 years ago
3fc9967bf2
Add modelcode tests
3 years ago
7a8a22c04a
Prevent usage of lists as a key type
3 years ago
55457cf097
Add DDL support for listmaps
3 years ago
47ace776a4
Fix tests
3 years ago
4e68c4bd49
Add listmap se/de support in storage engine
3 years ago
0ee5f69d50
Add basic listmap support
3 years ago
cdbe1a81eb
Add more listmap corruption tests
3 years ago
ff59627e95
Add encoding tests for mutating KVE actions
3 years ago
8c95ef0049
Upgrade deps
3 years ago
3e4bc1e6fc
Add remove method
3 years ago
ef3e617147
Add basic listmap definition
3 years ago
2c2e5f0ea6
Add listmap bytemarks
3 years ago
234623a0cc
Add corruption tests
3 years ago
c560cc68d8
Add multi-element listmap tests
3 years ago
b8c8cda1d6
Add tests for list-based maps
3 years ago
21157b2797
Use `RawSliceIter` and `RawSliceIterBorrowed`
...
These iterators (if you'd call them) provide safe abstractions over raw
pointers, enabling us to easily use them in the storage engine, reducing
verbosity and redundancy.
3 years ago
ad403421d4
Fix storage engine bug for zero-sized map entries
...
gt was applicable here instead of ge, since for zero-sized elements, we
may have already reached the end of allocation.
3 years ago
7dc17a0757
Fix de of empty lists or zero-sized elements
3 years ago
6ad2793e76
Deconstruct se/de methods for lists
3 years ago
08b7b7b821
Add list deserialization
3 years ago
2154fba359
Add `raw_serialize_list` for list based maps
3 years ago
a1ea9c3111
Upgrade deps and document parser
3 years ago
1dfaa0e4a0
Switch to using `next_unchecked`
3 years ago
831d84d65e
Fix forwarding of buffer before query execution
...
This has the ability to introduce UB because advancing the cursor might
invalidate the source pointers obtained by `protocol::Parser::parse()`
That's why we only forward the cursor after the query has been
executed. This is absolutely fine because even if another query
packet has been buffered, we will only forward by the amount we read
and not anything more.
3 years ago
f1cf7e8796
Reduce `memcpy`s in Skyhash protocol impl
...
This commit does an extreme amount of `unsafe` work to reduce the amount
of data that is copied around. Previously, we:
1. Copied in from the buffer
2. Copied again when we did a ptr::read
This changes that to:
1. Zero-copies for operations that can operate on borrowed values
(i.e values by ref like GET, KEYLEN, ..)
2. One copy for operations that need owned values
Overall, this may not immediately seem beneficial for small queries
(afterall, computer memory is very fast), but for large keys -- this can
make a very significant difference. However, a big difference can be noticed
in memory usage under heavy load (with as much as a tenfold drop in some
cases).
But to make this possible, we have to break free from Rust's borrowing
rules as we can't normally pass around refs easily. So, we just turn
everything into raw pointers and operate on them directly. This is why
we introduce a significant amount of unsafe code.
3 years ago
111d50b75e
Release v0.7.0 ( #210 )
...
* Simplify query gen in sky-bench and bump versions
* Upgrade to the latest driver version
3 years ago
cdc33b349e
Fix wrong arg length being silently ignored
...
Where actions were supposed to report an action error, they silently ran
their significant part, ignoring the rest. This was fixed in:
- `DROP TABLE`
- `INSPECT KEYSPACES`
- `INSPECT KEYSPACE <ksid>`
- `INSPECT TABLE <entity>`
- `USE <entity>`
Tests for the same were added
3 years ago
4ef28ea7e8
Upgrade deps
3 years ago
76e0cf607c
Switch `MPOP` to returning a `Typed Array`
...
This makes its behavior similar to that of the other mutating actions.
3 years ago
d744eaa2f6
Fix `HEYA` impl and add `HEYA` actiondoc
...
Also added heya echo test
3 years ago
c45bfd03c8
Fix critical bug in flush routine
...
When a new instance is created, we need to:
1. Create the tree
This ONLY creates the directories
2. Create the PRELOAD
This is critical because this is our check for a new instance
3. Flush the tables
This is important because we have never flushed the tables/ks before.
If we don't do this -- the server would fail to start with a
`directory not found` error.
3 years ago
3bd34ae5c9
Merge branch 'model/tsymbol' into next
3 years ago
d7ac4ebe56
Fix inspect tests
3 years ago
a24e03da60
Use `TypedArray`s in inspect commands
3 years ago
596b593526
Fix detection of invalid container name ( #207 )
...
* Fix parser allowing `$` in container name
* Fix query engine tests
3 years ago
95504a8be6
Fix `FlatArrayWriter` impl and `mpop` tests
...
Update driver version
3 years ago
2ed3335a1e
Upgrade all interfaces to use the latest driver
...
Fix lskeys impl
3 years ago
e389ee7860
Simplify encoding checks across actions
...
Checking all encoding errors beforehand simplifies a lot of things for
us. At the same time, this saves a lot of bandwidth as we don't have to
write encoding errors for each element -- instead we just write one.
3 years ago
3a6c48727b
Upgrade all actions to use typed arrays
3 years ago
f7d64689a6
Use `TypedArrayWriter` for typed Skyhash arrays
3 years ago
0d2cfc661c
Use tsymbol to determine if binary or unicode
...
This enables clients to know whether they should expect binary data or
an unicode string.
3 years ago
dd12ea599e
Take values by ref in `KVEngine` impl methods
3 years ago
ec10d3f962
Fix `pop` and `mpop` tests
3 years ago
fa2a4c2611
Add the `mpop` action and update the `pop` action
3 years ago
1092bdfd65
Merge pull request #198 from skytable/storage/snapshotengine
3 years ago
27a5562651
Don't panic on unimplemented pipeline query type
...
The return for inpsect table queries for Keymap model tables was fixed
3 years ago
4ba70aa19b
Remove ser/de implementations
...
These impls are no longer needed
3 years ago
42f3251d2c
Upgrade deps
...
Also added docs for Skymap
3 years ago
da8462eda3
Switch hasher implementation
3 years ago
153f940ff6
Merge branch 'memory/improve-cmap' into next
3 years ago
c17e44ddf7
Fix Skymap borrowed iter impl
3 years ago
9da0cdafee
Upgrade interfaces to use Skymap
3 years ago
3759992cf8
Add iterators
3 years ago
7498add90c
Add get, get_mut and entry methods to Skymap
3 years ago
28133ddc5f
Remove trailing semicolons from macros ( #201 )
...
This fixes warnings on the latest nightly.
See https://github.com/rust-lang/rust/issues/79813
3 years ago
db60133dc3
Add basic Skymap methods
3 years ago
bb19d024ea
Ignore errors in run loop
...
This fixes CVE-2021-37625
3 years ago
8b7de7173e
Use flocks for pid file to enable auto release
...
This can help in situations where the process is forcefully terminated.
3 years ago
891f9a2e06
Remove unused deps
...
Also simplified new instance check
3 years ago
5a0d3017a5
Support non-interactive TLS passphrase input
3 years ago
d54652c21e
Fix config not parsing `false` in ssl.only key
3 years ago
bb14b62805
Reduce disk accesses with a trip switch
3 years ago
0d3bfe486e
Use specialized result type for strong actions
3 years ago
252dd9c08d
Add supdate concurrency tests
...
Also added changelog entry
3 years ago
39856cdfd5
Add sset concurrency tests
3 years ago
a8b716c892
Add sdel concurrency tests
3 years ago
62b58f1a9f
Decompose strong actions' snapshotting core
3 years ago
b47fcc2c88
Use snapshot isolation for strong actions
...
This makes strong actions far more reliable than the previous
implementation.
3 years ago
2d5d32216c
Add some general borrow optimizations
3 years ago
b6fcb4c035
Add `drop keyspace <name> force` ( #192 )
...
* Add forceful dropping of keyspaces
This commit also improves the reliability of `drop keyspace` in general
* Add changelog
* Add tests for `force_drop_keyspace`
* Upgrade deps
3 years ago
e32b3e8ea1
Destructure methods in `BorrowedEntityGroup`
3 years ago
69df98c69a
Only copy into `ObjectID` for `create table`
3 years ago
3f3f381c50
Avoid copies into `ObjectID`
3 years ago
133400b846
Only run mutating DDL queries if state is okay
3 years ago
ee14656354
Use volatile tables for tests
3 years ago
49fa843eb2
Parse the `swapks` header if it is provided
3 years ago
f06e9ccdb5
Add FQE tests for flushdb
3 years ago
abcb60463f
Add tests for `inspect`
3 years ago
6924524c4e
Add ddl tests
3 years ago
c5e4de9538
Fix lskeys and add tests for entity based queries
3 years ago
bc9abd7ac3
Enable `lskeys` to accept entities
3 years ago
32fcbc2075
Add changelog and bump up version
3 years ago
c1249ccbf9
Prevent abuse of `system` table
3 years ago
fed4597208
Parse volatile property from DDL queries
3 years ago
c9e55451f3
Add inspection queries
...
This lets the user explore a keyspace/table.
3 years ago
c36cbe69e4
Remove `HTable`
...
Our entire storage infrastructure has changed and this is no longer
needed.
3 years ago
0a670a6555
Remove compat
...
We'll be shipping a migration tool, so this isn't required anymore.
3 years ago
fb07d385fd
Support entity groups in `flushdb` and `dbsize`
3 years ago
1025933cfb
Run tests parallelly in the `testsuite` keyspace
...
This saves us time and simplifies things. A lot.
3 years ago
728c71f84f
Fix SE tests
3 years ago
28f825910e
Fix tests and strong actions
3 years ago
d43c3dc1cf
Fix storage engine tests
3 years ago
a6fc09f990
Enable entity group based table deletion
3 years ago
8e71ef2a01
Fix tests
3 years ago
2f39e6808b
Enable entity group based table creation
3 years ago
5402028b26
Add `use keyspace` and `use keyspace:table`
3 years ago
2f7a2d546b
Add branch hints to parser
3 years ago
6eaf580ac8
Add entity group parsing
3 years ago
3249fcb347
Add `drop table` and `drop keyspace` queries
3 years ago
d6e8db7d8f
Add `create table` and `create keyspace` queries
3 years ago
c1064a7cd1
Implement and upgrade to `Corestore`
3 years ago
a5f735e977
Add `drop table` and `drop keyspace`
3 years ago
608e008a65
Add `create table` and `create keyspace`
3 years ago
5bab0fb91b
Use global flush lock to coordinate disk access
3 years ago
6e17ef6d5e
Add system keyspace and remove `_system` table
3 years ago
16ac791ff5
Add flush methods for snapshots
3 years ago
03518c22c3
Add corestore impl
3 years ago
421ff19405
Use registry for handling global state
3 years ago
0e88d76444
Simplify hints
3 years ago
d1f6916251
Make mksnap use `action!` and simplify `IoResult`
3 years ago
c9545a30f1
Do not panic on unknown data type
3 years ago
327953dce0
Fix qe returning full resp packet instead of group
3 years ago
9535975d03
Create the dir tree if the instance is new
3 years ago
b2c0b9ecf2
Enable fixed len mutable params in `action!` macro
3 years ago
589ef85e2c
Simplify action impls with `action!` macro
3 years ago
66c9822f5d
Add routines to unflush an entire `Memstore`
3 years ago
a919b1c934
Add tests for flush and unflush routines
3 years ago
454704574b
Fix flush routines
3 years ago
2393efb590
Encode model bytemark into `PARTMAP`
3 years ago
610144f78e
Add unflush routines
3 years ago
6baa61176f
Add volatility tests
3 years ago
66b9ac27af
Write storage type into PARTMAP
...
This commit adds a storage_type segment to the PARTMAP disk file. This
contains information about the storage type of the table.
Is it volatile? Is it persistent? 8-bits were added for future
improvements.
3 years ago
1d403c0d1a
Add flush routines
3 years ago
e89417cbc6
Fix preload generation and add preload decoding
3 years ago
87d79650ce
Add preload generation
3 years ago
bca8df5863
Add methods to create directory tree
3 years ago
058b1ef1c6
Replace ns with ks
3 years ago
ad48e5478c
Add `Integer64BufferRaw` and use it in `resp`
3 years ago
71ab845d02
Fix zeroed impl for `Array<T, N>`
3 years ago
48e29b6ec6
Add `Integer32Buffer` for faster encoding/decoding
3 years ago
5790e99a98
Fix filename generator and use LUT for 32-bit ints
3 years ago
ca55694904
And `interface` for fs
3 years ago
b162756f80
Avoid unnecessary referencing
...
Even though the compiler will do immediate derefs, let us be explicit.
3 years ago
03a229104d
Fix alignment and auto static lifetime causing UB
3 years ago
c24e83c0a7
Encode all sizes to little endian
...
This is very convenient for us and we will provide advanced byte
ordering only if our users demand for it.
3 years ago
e1dfa12ba4
Add runtime panic check on 32-bit or lower
3 years ago
5adc269e11
Add endian info and pointer-width check
...
When a file created by a 64-bit system is read on a 16/32 bit system,
there may be a size overflow. If so, we should do a runtime panic.
3 years ago
74a0592fbc
Simplify transmutation
...
Also use smaller vectors for faster tests to avoid problems with
thread stack sizes on Windows.
3 years ago
c20302ef75
Add storage module for custom encoding/decoding
3 years ago
7c835d03b3
Add NS/KS swap headers in protocol
3 years ago
17d1c472b6
Use proc macro for uninit array magic
...
The proc macro does some magic to give us a const array with the full
size without having to manually write it. Magic!
3 years ago
fd3e06beda
Fix array length check assertion
3 years ago
e5b0588cca
Fix UB due to use of from_const_array
3 years ago
f4379d5688
Use Array[64] for NS/KS names
...
We limit the sizes of keyspaces/namespaces because very long names may
cause fs errors on some file systems.
3 years ago
8faf653d2e
Fix ser/de for `Coremap<Array, Array>`
3 years ago
bd679f9b79
Document `Array` and `IArray`
3 years ago
b997afe89a
Fix push function in `Array`
3 years ago
4a27f83e6b
Impl `Send` and `Sync` for `Array`
3 years ago
0067b7d1b7
Implement `Array` type
...
This allows us to have fixed size arrays right on the stack
3 years ago
5ff045bd93
Impl deserialize for `IArray`
3 years ago
e25d13afff
Add BP optimizations
3 years ago
036b507de8
Impl Serialize for IArray
3 years ago
4fe7aa7050
Add manip methods to `IArray`
3 years ago
6c00ffae3b
Add methods to add items to `IArray`
3 years ago
3739aa54fd
Add basic `IArray` impl
3 years ago
e3d749ac20
Add methods to drop keyspaces and tables
3 years ago
845ef82060
Use static slices for responses
3 years ago
26a22c3102
Add dtor tests for Lazy
3 years ago
a7e11cc281
Add methods to create ns, ks and tables
3 years ago
bbcbe0756b
Add basic methods to memstore
3 years ago
7ea890765d
Correct kvengine defs and add custom `Lazy` type
3 years ago
74126ec7cb
Add variable width characters and failure test
...
This is just for sanity
3 years ago
013be0058b
Add emoji tests
3 years ago
8749cfa134
Add memstore
3 years ago
c953b88695
Add test with bincode
3 years ago
b1383bf8c9
Add encoding evaluation in `KVEngine`
3 years ago
a276091726
Account for perl errors
3 years ago
5f75df7109
Add some optimizations for unicode checks
3 years ago
ead3f62ded
Add failure cases for unicode
3 years ago
bdfaf6ec4d
Ensure that perl doesn't error
3 years ago
5e4cd5be4b
Use deterministic finite automaton for validation
...
A dual stream approach provides even more speed improvements
3 years ago
8cfab3f7d3
Add convenience macros
3 years ago
efec980fa6
Add basic `BufferBlockReader` definition
3 years ago
Sayan Nandan
Aaron Hill