Release v0.6.1 (#164)

* Explicitly fsync and relax CPU on snap busy-loop

This commit also switches to using global `VERSION` and `URL` statics
than defining it per-crate.

* Add changelog entry and bump up version

* Optimize `dbtest` macro and rm redundant allocs

* Upgrade deps
next
Sayan 3 years ago committed by GitHub
parent 0c33395a09
commit e553c5172b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -2,25 +2,29 @@
All changes in this project will be noted in this file.
**Unreleased**
## Version 0.6.1 [2021-06-07]
* Snapshotting failure will now poison the database (customizable through CLI options or the configuration file)
[see [#160](https://github.com/skytable/skytable/pull/160)]
* Added file-locking on Solaris [see [#162](https://github.com/skytable/skytable/pull/162)]
> No breaking changes
- Snapshotting failure will now poison the database (customizable through CLI options or the configuration file)
[see [#160](https://github.com/skytable/skytable/pull/160)]
- Added file-locking on Solaris [see [#162](https://github.com/skytable/skytable/pull/162)]
- Fixed missing explicit `fsync` or `FlushFileBuffers` after writing data
- Optimized wait on snapshot busy-loop using `_mm_pause` (on x86/x86_64) or `__yield` (on aarch64/arm)
## Version 0.6.0 [2021-05-27]
> Breaking changes!
* ⚠ Dropped support for Terrapipe 1.0 (reached EOL)
* ⚠ New disk storage format with much faster reads and/or writes
* Added support for Skyhash 1.0 (see [#147](https://github.com/skytable/skytable/pull/147))
* Fixed persistence bugs (see [#151](https://github.com/skytable/skytable/pull/151))
* Fixed bugs in background services (see [#152](https://github.com/skytable/skytable/pull/152))
* Make BGSAVE recoverable (see [#153](https://github.com/skytable/skytable/pull/153))
* Added `lskeys` action (see [#155](https://github.com/skytable/skytable/pull/155))
* Added `compat` module (see [#158](https://github.com/skytable/skytable/pull/158))
* Added backward compatibility for storage formats all the way upto 0.3.0. See [this wiki article](https://github.com/skytable/skytable/wiki/Disk-storage-formats) for more information
- ⚠ Dropped support for Terrapipe 1.0 (reached EOL)
- ⚠ New disk storage format with much faster reads and/or writes
- Added support for Skyhash 1.0 (see [#147](https://github.com/skytable/skytable/pull/147))
- Fixed persistence bugs (see [#151](https://github.com/skytable/skytable/pull/151))
- Fixed bugs in background services (see [#152](https://github.com/skytable/skytable/pull/152))
- Make BGSAVE recoverable (see [#153](https://github.com/skytable/skytable/pull/153))
- Added `lskeys` action (see [#155](https://github.com/skytable/skytable/pull/155))
- Added `compat` module (see [#158](https://github.com/skytable/skytable/pull/158))
- Added backward compatibility for storage formats all the way upto 0.3.0. See [this wiki article](https://github.com/skytable/skytable/wiki/Disk-storage-formats) for more information
### Upgrading existing clients
@ -32,15 +36,15 @@ Please refer to [this wiki article](https://github.com/skytable/skytable/wiki/Di
### Improvements in the new protocol (Skyhash)
* Upto 40% lower bandwidth requirements
* Upto 30% faster queries
* Support for recursive arrays
* More robust and well tested than Terrapipe
- Upto 40% lower bandwidth requirements
- Upto 30% faster queries
- Support for recursive arrays
- More robust and well tested than Terrapipe
### Internal codebase improvements
* BGSAVE is now tested by the test suite
* `skysh` and `sky-bench` both use the [Rust client driver for Skytable](https://github.com/skytable/client-rust).
- BGSAVE is now tested by the test suite
- `skysh` and `sky-bench` both use the [Rust client driver for Skytable](https://github.com/skytable/client-rust).
## Version 0.5.3 [2021-05-13]
@ -52,55 +56,54 @@ Fix persistence (see [#150](https://github.com/skytable/skytable/issues/150))
> No breaking changes
* `sky-bench` is less agressive and runs sanity test before benchmarking
* `skyd` now locks the data file (the `data.bin` file)
* The data directory structure has been changed (see #144) (all files are now stored in ./data/*)
* Fixed 'Connection Forcibly Closed' errors on Windows (see #110)
* Add support for line-editing and keyboard shortcuts on `skysh` (see #142)
* Fixed problems while parsing snapshots in the snapshot directory (see #144)
* The old data directory structure has been deprecated (see #144)
* Official support for 32-bit platforms (see #139)
* Tier-2 Support for Linux musl (x86_64) (see #136, #135)
- `sky-bench` is less agressive and runs sanity test before benchmarking
- `skyd` now locks the data file (the `data.bin` file)
- The data directory structure has been changed (see #144) (all files are now stored in ./data/\*)
- Fixed 'Connection Forcibly Closed' errors on Windows (see #110)
- Add support for line-editing and keyboard shortcuts on `skysh` (see #142)
- Fixed problems while parsing snapshots in the snapshot directory (see #144)
- The old data directory structure has been deprecated (see #144)
- Official support for 32-bit platforms (see #139)
- Tier-2 Support for Linux musl (x86_64) (see #136, #135)
## Version 0.5.1 [2021-03-17]
> No breaking changes
* Built-in TLS/SSL support
* Custom host/port settings in `sky-bench`
* Mock keys can be created with `sky-bench`
* Security patch for VE/S/00001
* Escaping for spaces in `skysh`
* `tdb` is now called `skyd` (short for 'Skytable Daemon')
- Built-in TLS/SSL support
- Custom host/port settings in `sky-bench`
- Mock keys can be created with `sky-bench`
- Security patch for VE/S/00001
- Escaping for spaces in `skysh`
- `tdb` is now called `skyd` (short for 'Skytable Daemon')
## Version 0.5.0 [2020-11-19]
> This release introduces breaking changes!
* Command line configuration added to `tdb`
* ⚠ Positional arguments in `tsh` and `tdb-bench` have been removed
* `MKSNAP` now has an _enhanced version_ which enables snapshots to be created even if they're disabled on the server side
* If `BGSAVE` fails, no more writes will be accepted on the server side. All commands that try to modify data will return an internal server error
* `tdb-bench` now provides JSON output with the `--json` flag
* The `Dockerfile` was fixed to use command line arguments instead of the configuration file which caused problems
* The `enabled` key under the `snapshots` key in the configuration file has been removed
- Command line configuration added to `tdb`
- ⚠ Positional arguments in `tsh` and `tdb-bench` have been removed
- `MKSNAP` now has an _enhanced version_ which enables snapshots to be created even if they're disabled on the server side
- If `BGSAVE` fails, no more writes will be accepted on the server side. All commands that try to modify data will return an internal server error
- `tdb-bench` now provides JSON output with the `--json` flag
- The `Dockerfile` was fixed to use command line arguments instead of the configuration file which caused problems
- The `enabled` key under the `snapshots` key in the configuration file has been removed
### Upgrading
* Users who ran `tsh` like `tsh 172.17.0.1 2003` will now need to run:
- Users who ran `tsh` like `tsh 172.17.0.1 2003` will now need to run:
``` shell
```shell
tsh -h 172.17.0.1 -p 2003
```
* Users who ran `tdb-bench` like `tdb-bench 10 100000 4` will now need to run:
- Users who ran `tdb-bench` like `tdb-bench 10 100000 4` will now need to run:
``` shell
```shell
tdb-bench -c 10 -q 100000 -s 4
```
* To enable snapshots, you just have to add the key: there is no need for the `enabled` key. To disable snapshots, you just have to omit the `snapshot` key (block) from your configuration file
- To enable snapshots, you just have to add the key: there is no need for the `enabled` key. To disable snapshots, you just have to omit the `snapshot` key (block) from your configuration file
## Version 0.4.5 [2020-10-29]
@ -139,13 +142,13 @@ This release adds support for configuration files
Changes:
* Actions added: `MSET` , `MGET` , `MUPDATE`
* Terrapipe 1.0
* Improved terminal output
- Actions added: `MSET` , `MGET` , `MUPDATE`
- Terrapipe 1.0
- Improved terminal output
Fixes:
* Explicit handling for incomplete responses in `tsh`
- Explicit handling for incomplete responses in `tsh`
### Migrating existing clients

14
Cargo.lock generated

@ -401,7 +401,7 @@ checksum = "789da6d93f1b866ffe175afc5322a4d76c038605a1c3319bb57b06967ca98a36"
[[package]]
name = "libsky"
version = "0.6.0"
version = "0.6.1"
dependencies = [
"bytes",
"lazy_static",
@ -823,16 +823,16 @@ dependencies = [
[[package]]
name = "signal-hook-registry"
version = "1.3.0"
version = "1.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "16f1d0fef1604ba8f7a073c7e701f213e056707210e9020af4528e0101ce11a6"
checksum = "e51e73328dc4ac0c7ccbda3a494dfa03df1de2f46018127f60c693f2648455b0"
dependencies = [
"libc",
]
[[package]]
name = "sky-bench"
version = "0.6.0"
version = "0.6.1"
dependencies = [
"clap",
"devtimer",
@ -847,7 +847,7 @@ dependencies = [
[[package]]
name = "sky_macros"
version = "0.6.0"
version = "0.6.1"
dependencies = [
"proc-macro2",
"quote",
@ -856,7 +856,7 @@ dependencies = [
[[package]]
name = "skyd"
version = "0.6.0"
version = "0.6.1"
dependencies = [
"bincode",
"bytes",
@ -886,7 +886,7 @@ dependencies = [
[[package]]
name = "skysh"
version = "0.6.0"
version = "0.6.1"
dependencies = [
"bytes",
"clap",

@ -1,6 +1,6 @@
[package]
name = "skysh"
version = "0.6.0"
version = "0.6.1"
authors = ["Sayan Nandan <ohsayan@outlook.com>"]
edition = "2018"
@ -13,8 +13,8 @@ bytes = "1.0.1"
clap = { version = "2.33.3", features = ["yaml"] }
openssl = { version = "0.10.34", features = ["vendored"] }
tokio-openssl = "0.6.1"
rustyline = "8.0.0"
rustyline = "8.2.0"
crossterm = "0.19.0"
skytable = { git = "https://github.com/skytable/client-rust", branch = "next", features = ["async"], default-features = false }
lazy_static = "1.4.0"
regex = "1.5.4"
regex = "1.5.4"

@ -29,6 +29,8 @@ use clap::load_yaml;
use clap::App;
use crossterm::terminal::{Clear, ClearType};
use crossterm::{cursor, execute};
use libsky::URL;
use libsky::VERSION;
use readline::config::Configurer;
use readline::{error::ReadlineError, Editor};
use rustyline as readline;
@ -36,7 +38,6 @@ use skytable::AsyncConnection;
use std::io::stdout;
use std::process;
use std::process::exit;
const MSG_WELCOME: &'static str = "Skytable v0.6.0";
const ADDR: &str = "127.0.0.1";
/// This creates a REPL on the command line and also parses command-line arguments
@ -67,7 +68,7 @@ pub async fn start_repl() {
};
let mut runner = Runner::new(con);
if let Some(eval_expr) = matches.value_of("eval") {
if eval_expr.len() == 0 {
if eval_expr.is_empty() {
return;
}
runner.run_query(&eval_expr).await;
@ -78,7 +79,7 @@ pub async fn start_repl() {
editor.set_history_ignore_dups(true);
let _ = editor.load_history(".sky_history");
println!("Connected to skyhash://{}:{}", host, port);
println!("{}", MSG_WELCOME);
println!("Skytable v{} | {}", VERSION, URL);
loop {
match editor.readline("skysh> ") {
Ok(line) => match line.to_lowercase().as_str() {

@ -25,7 +25,6 @@
*/
mod argparse;
use tokio;
mod runner;
#[tokio::main]

@ -122,7 +122,7 @@ impl Runner {
Response::ParseError => {
println!("ERROR: The client failed to deserialize data sent by the server")
}
x @ _ => {
x => {
println!(
"The server possibly sent a newer data type that we can't parse: {:?}",
x

@ -1,6 +1,6 @@
[package]
name = "libsky"
version = "0.6.0"
version = "0.6.1"
authors = ["Sayan Nandan <ohsayan@outlook.com>"]
edition = "2018"
@ -11,4 +11,4 @@ lazy_static = "1.4.0"
bytes = "1.0.1"
termcolor = "1.1.2"
regex = "1.5.4"
skytable = { git = "https://github.com/skytable/client-rust", branch = "next", features = ["dbg"], default-features = false }
skytable = { git = "https://github.com/skytable/client-rust", branch = "next", features = ["dbg"], default-features = false }

@ -35,6 +35,10 @@ use std::error::Error;
pub type TResult<T> = Result<T, Box<dyn Error>>;
/// The size of the read buffer in bytes
pub const BUF_CAP: usize = 8 * 1024; // 8 KB per-connection
/// The current version
pub static VERSION: &str = env!("CARGO_PKG_VERSION");
/// The URL
pub static URL: &str = "https://github.com/skytable/skytable";
use std::str::FromStr;
@ -45,7 +49,7 @@ lazy_static::lazy_static! {
pub fn split_into_args(q: &str) -> Vec<String> {
let args: Vec<String> = RE
.find_iter(q)
.map(|val| val.as_str().replace("'", "").replace("\"", "").to_owned())
.map(|val| val.as_str().replace("'", "").replace("\"", ""))
.collect();
args
}

@ -32,10 +32,10 @@ pub mod terminal {
/// Write to stdout with
pub fn write_with_col<T: fmt::Display>(item: T, color: Option<Color>) -> fmt::Result {
let mut stdout = StandardStream::stdout(ColorChoice::Always);
if let Err(_) = stdout.set_color(ColorSpec::new().set_fg(color)) {
if stdout.set_color(ColorSpec::new().set_fg(color)).is_err() {
return Err(fmt::Error);
}
if let Err(_) = write!(&mut stdout, "{}", item) {
if write!(&mut stdout, "{}", item).is_err() {
return Err(fmt::Error);
}
if stdout.reset().is_err() {

@ -1,6 +1,6 @@
[package]
name = "skyd"
version = "0.6.0"
version = "0.6.1"
authors = ["Sayan Nandan <ohsayan@outlook.com>"]
edition = "2018"
build = "build.rs"
@ -33,12 +33,12 @@ jemallocator = "0.3.2"
winapi = { version = "0.3.9", features = ["fileapi"] }
[target.'cfg(unix)'.build-dependencies]
cc = "1.0.67"
cc = "1.0.68"
[dev-dependencies]
tokio = { version = "1.6.0", features = ["test-util"] }
tokio = { version = "1.6.1", features = ["test-util"] }
skytable = { git = "https://github.com/skytable/client-rust", features = ["async"], default-features = false, branch = "next" }
devtimer = "4.0.1"
[target.'cfg(unix)'.dependencies]
libc = "0.2.94"
libc = "0.2.95"

@ -102,6 +102,10 @@ impl FileLock {
// Now write to the file
self.file.write_all(bytes)
}
/// Sync all metadata and flush buffers before returning
pub fn fsync(&self) -> Result<()> {
self.file.sync_all()
}
#[cfg(test)]
pub fn try_clone(&self) -> Result<Self> {
Ok(FileLock {

@ -94,16 +94,21 @@ fn deserialize(file: Vec<u8>) -> TResult<HTable<Data, Data>> {
/// Flush the in-memory table onto disk
///
/// This functions takes the entire in-memory table and writes it to the disk,
/// more specifically, the `data/data.bin` file
/// to the provided file behind the [`FileLock`]. This method will **automatically fsync**. You
/// do not need to explicitly fsync unless you'd like to waste CPU time
pub fn flush_data(file: &mut flock::FileLock, data: &Coremap<Data, Data>) -> TResult<()> {
let encoded = data.serialize()?;
file.write(&encoded)?;
file.fsync()?;
Ok(())
}
/// This function will write serialized data to disk and will **automatically fsync**. You
/// do not need to explicitly fsync unless you'd like to waste CPU time
pub fn write_to_disk(file: &Path, data: &Coremap<Data, Data>) -> TResult<()> {
let mut file = fs::File::create(&file)?;
let encoded = data.serialize()?;
file.write_all(&encoded)?;
file.sync_all()?;
Ok(())
}

@ -31,6 +31,7 @@ use crate::coredb::CoreDB;
use crate::coredb::SnapshotStatus;
use crate::diskstore;
use chrono::prelude::*;
use core::hint::spin_loop as let_the_cpu_relax;
#[cfg(test)]
use io::Result as IoResult;
use regex::Regex;
@ -260,6 +261,9 @@ impl<'a> SnapshotEngine<'a> {
.is_busy()
{
// Endlessly wait for a lock to be free
// we'll not yield to the system scheduler but let the machine instructions
// optimize away things
let_the_cpu_relax();
}
// So we acquired a lock

@ -33,6 +33,8 @@
use crate::config::BGSave;
use crate::config::PortConfig;
use crate::config::SnapshotConfig;
use libsky::URL;
use libsky::VERSION;
use std::io::{self, prelude::*};
mod config;
use std::env;
@ -63,8 +65,6 @@ use jemallocator::Jemalloc;
/// Jemallocator - this is the default memory allocator for platforms other than msvc
static GLOBAL: Jemalloc = Jemalloc;
/// The version text
static MSG: &str = "Skytable v0.6.0 | https://github.com/skytable/skytable";
/// The terminal art for `!noart` configurations
static TEXT: &str = "\n█████████ ██ ██  ██ ████████  █████  ██████  ██  ███████ \n████ ██   ██  ██     ██    ██   ██ ██   ██ ██  ██      \n████████████  ████   ██  ███████ ██████  ██  █████  \n██████  ██   ██  ██   ██ ██   ██ ██  ██     \n█████████ ██  ██  ██  ██  ██ ██████  ███████ ███████ \n ";
@ -130,15 +130,15 @@ async fn check_args_and_get_cfg() -> (PortConfig, BGSave, SnapshotConfig, Option
let binding_and_cfg = match cfg {
Ok(config::ConfigType::Custom(cfg, file)) => {
if cfg.is_artful() {
println!("{}\n{}", MSG, TEXT);
println!("Skytable v{} | {}\n{}", VERSION, URL, TEXT);
} else {
println!("{}", MSG);
println!("Skytable v{} | {}", VERSION, URL);
}
log::info!("Using settings from supplied configuration");
(cfg.ports, cfg.bgsave, cfg.snapshot, file)
}
Ok(config::ConfigType::Def(cfg, file)) => {
println!("{}\n{}", MSG, TEXT);
println!("Skytable v{} | {}\n{}", VERSION, URL, TEXT);
log::warn!("No configuration file supplied. Using default settings");
(cfg.ports, cfg.bgsave, cfg.snapshot, file)
}

@ -1,6 +1,6 @@
[package]
name = "sky-bench"
version = "0.6.0"
version = "0.6.1"
authors = ["Sayan Nandan <ohsayan@outlook.com>"]
edition = "2018"
@ -11,8 +11,8 @@ rand = "0.8.3"
devtimer = "4.0.1"
skytable = { git = "https://github.com/skytable/client-rust", branch = "next" }
clap = { version = "2.33.3", features = ["yaml"] }
serde = { version = "1.0.125", features = ["derive"] }
serde = { version = "1.0.126", features = ["derive"] }
serde_json = "1.0.64"
regex = "1.5.4"
lazy_static = "1.4.0"
libsky = {path = "../libsky"}
libsky = { path = "../libsky" }

@ -64,7 +64,7 @@ mod benchtool {
}
impl Netpool {
/// Create a new `Netpool` instance with `size` number of connections (and threads)
pub fn new(size: usize, host: &String) -> Netpool {
pub fn new(size: usize, host: &str) -> Netpool {
assert!(size > 0);
let (sender, receiver) = mpsc::channel();
let receiver = Arc::new(Mutex::new(receiver));
@ -95,7 +95,7 @@ mod benchtool {
// We have to write something to the socket
connection.write_all(&someaction).unwrap();
// Ignore whatever we get, we don't need them
connection.read(&mut vec![0; 1024]).unwrap();
let _ = connection.read(&mut vec![0; 1024]).unwrap();
}
WhatToDo::Nothing => {
// A termination signal - just close the stream and
@ -179,7 +179,7 @@ mod benchtool {
}
println!("Sanity test succeeded");
// now push in the port to the host string
host.push_str(":");
host.push(':');
host.push_str(&port.to_string());
let mut rand = thread_rng();
if let Some(matches) = matches.subcommand_matches("testkey") {
@ -350,7 +350,7 @@ mod benchtool {
if !connection
.run_simple_query(&query)
.unwrap()
.eq(&Response::Item(Element::String(value.to_owned())))
.eq(&Response::Item(Element::String(value)))
{
return Err("GET test failed".into());
}
@ -369,7 +369,7 @@ mod benchtool {
/// Returns the number of queries/sec
fn calc(reqs: usize, time: u128) -> f64 {
reqs as f64 / (time as f64 / 1_000_000_000 as f64)
reqs as f64 / (time as f64 / 1_000_000_000_f64)
}
fn ran_string(len: usize, rand: impl rand::Rng) -> String {

@ -1,6 +1,6 @@
[package]
name = "sky_macros"
version = "0.6.0"
version = "0.6.1"
authors = ["Sayan Nandan <ohsayan@outlook.com>"]
edition = "2018"
@ -12,4 +12,4 @@ proc-macro = true
[dependencies]
syn = {version = "1.0.72", features = ["full"]}
quote = "1.0.9"
proc-macro2 = "1.0.26"
proc-macro2 = "1.0.27"

@ -126,47 +126,44 @@ fn parse_test_module(args: TokenStream, item: TokenStream) -> TokenStream {
let args = syn::parse_macro_input!(args as syn::AttributeArgs);
let mut skips = Vec::new();
for arg in args {
match arg {
syn::NestedMeta::Meta(syn::Meta::NameValue(namevalue)) => {
let ident = namevalue.path.get_ident();
if ident.is_none() {
let msg = "Must have specified ident";
return syn::Error::new_spanned(namevalue, msg)
.to_compile_error()
.into();
}
match ident.unwrap().to_string().to_lowercase().as_str() {
"skip" => {
let skip_lit = namevalue.lit.clone();
let span = skip_lit.span();
skips = match parse_string(skip_lit, span, "skip") {
Ok(s) => s,
Err(_) => {
return syn::Error::new_spanned(
namevalue,
"Expected a value for argument `skip`",
)
.to_compile_error()
.into();
}
}
.split_whitespace()
.map(|val| val.to_string())
.collect();
}
x => {
let msg = format!("Unknown attribute {} is specified; expected `skip`", x);
return syn::Error::new_spanned(namevalue, msg)
if let syn::NestedMeta::Meta(syn::Meta::NameValue(namevalue)) = arg {
let ident = namevalue.path.get_ident();
if ident.is_none() {
let msg = "Must have specified ident";
return syn::Error::new_spanned(namevalue, msg)
.to_compile_error()
.into();
}
match ident.unwrap().to_string().to_lowercase().as_str() {
"skip" => {
let skip_lit = namevalue.lit.clone();
let span = skip_lit.span();
skips = match parse_string(skip_lit, span, "skip") {
Ok(s) => s,
Err(_) => {
return syn::Error::new_spanned(
namevalue,
"Expected a value for argument `skip`",
)
.to_compile_error()
.into();
}
}
.split_whitespace()
.map(|val| val.to_string())
.collect();
}
x => {
let msg = format!("Unknown attribute {} is specified; expected `skip`", x);
return syn::Error::new_spanned(namevalue, msg)
.to_compile_error()
.into();
}
}
_ => (),
}
}
let modname = &input.ident;
if modname.to_string() != "__private" {
if *modname != "__private" {
return syn::Error::new_spanned(
modname,
"By convention, all the modules using the `dbtest` macro have to be called `__private`",

Loading…
Cancel
Save