Multimodel distributed in memory database management system

Table of Contents

1. About The Project
2. Getting Started
   • Building
   • Installation
3. Gratitudes
4. Features
5. Requirements
6. Cases
7. Cluster
8. Troubles
9. Contributing
10. License
11. FAQ
12. Roadmap
13. Contact
14. [Limitations] (#Limitations)

About The Project

fluidB is Multimodel in-memory database management system an easy-to-use. It's lightweight, distributed which uses SQLite as its storage engine for small datasests and futrum which uses Riak for large datasests.

fluidB is simple to deploy, operating it is very straightforward, and its clustering capabilities provide you with fault-tolerance and high-availability. [fluidB is available only for Unix-like systems only]. fluidB uses Raft to achieve consensus across all the instances of the SQLite databases, ensuring that every change made to the system is made to a quorum of SQLite databases, or none at all. fluidB gives you the functionality of a rock solid, fault-tolerant, replicated relational database, but with very easy installation, deployment, and operation. With it you've got a lightweight and reliable distributed relational data store. Think etcd or Consul, but with relational data modelling also available. You could use fluidB as part of a larger system, as a central store for some critical relational data, without having to run larger, more complex distributed databases.

Finally, if you're interested in understanding how distributed systems actually work, fluidB is a good example to study. Much thought has gone into its and implementation, with clear separation between the various components, including storage, distributed consensus, and API.

Key features

• Trivially easy to deploy, with no need to separately install SQLite.
• Super-simple to use, with a straightforward HTTP API (https://fluidB.io/docs/api/). A command-line interface is also available, as are various client libraries.
• Fully replicated production-grade SQL database, with full access to SQLite full-text search and JSON document support
• Multiple options for node-discovery and automatic clustering, including integration with Kubernetes, Consul, etcd and DNS, allowing clusters to be dynamically created.
• Extensive security and encryption support , including node-to-node encryption.
• Choice of read consistency levels, and support for choosing [write performance over durability
• Optional read-only (non-voting) nodes, which can add read scalability to the system.
• A form of transaction support.
• Hot backups, as well as [load directly from SQLite https://fluidB.io/docs/guides/restore/.

Gratitudes

• Philip O' Toole, I wish to express my appreciation for all your efforts!!!
  

Requirements

• Hardware: Intel or AMD
• Processor: 64-bit
• RAM: 1 GB or higher
• Nodes: 5 (strongly recomended)
• Operating System: UNIX-like only (Linux, BSD(except OpenBSD), MacOS X) Windows isn't supported

Cases

• Logs ageregation
• Caching system
• Queue server
• Pattern "Central dispatcher" (for sharding)
• Solution that sits in front of multiple data sources and allow them to be treated as a single json database
• Web-server and database for hosting static web-sites

Getting Started

Basic modules

fluidB The project consists of two main modules:

1. fluidB- relational storage engine for small datasests
2. futrum- nosql storage engine for large datasests

Building

1. Install Golang 19.4 or higher build-essential in your operating system

2. Clone the fluidb-repo

   git clone https://source.fluidb.icu/fluidB/fluidb
   
   
   

3. Change folder and running compilation

 cd fluidb && go install ./...

4. Change directory

cd .. go/bin/

5. Configuration fluidB

chmod +x fconf.sh && ./fconf.sh

6. Running server fluidb

  ./fluidbd ~/node.1

fluidB is Multimodel in-memory database management system an easy-to-use. It's lightweight, distributed which uses SQLite as its storage engine for small datasests and futrum which uses Riak for large datasests.

fluidB is simple to deploy, operating it is very straightforward, and its clustering capabilities provide you with fault-tolerance and high-availability. [fluidB is available only for Unix-like systems only]. fluidB uses Raft to achieve consensus across all the instances of the SQLite databases, ensuring that every change made to the system is made to a quorum of SQLite databases, or none at all. fluidB gives you the functionality of a rock solid, fault-tolerant, replicated relational database, but with very easy installation, deployment, and operation. With it you've got a lightweight and reliable distributed relational data store. Think etcd or Consul, but with relational data modelling also available. You could use fluidB as part of a larger system, as a central store for some critical relational data, without having to run larger, more complex distributed databases.

Finally, if you're interested in understanding how distributed systems actually work, fluidB is a good example to study. Much thought has gone into its and implementation, with clear separation between the various components, including storage, distributed consensus, and API. Key features

Trivially easy to deploy, with no need to separately install SQLite.
Super-simple to use, with a straightforward HTTP API (https://fluidB.io/docs/api/). A command-line interface is also available, as are various client libraries.
Fully replicated production-grade SQL database, with full access to SQLite full-text search and JSON document support
Multiple options for node-discovery and automatic clustering, including integration with Kubernetes, Consul, etcd and DNS, allowing clusters to be dynamically created.
Extensive security and encryption support , including node-to-node encryption.
Choice of read consistency levels, and support for choosing [write performance over durability
Optional read-only (non-voting) nodes, which can add read scalability to the system.
A form of transaction support.
Hot backups, as well as [load directly from SQLite https://fluidB.io/docs/guides/restore/.

Gratitudes

Philip O' Toole, I wish to express my appreciation for all your efforts!!!

Requirements

Hardware: Intel or AMD
Processor: 64-bit
RAM: 1 GB or higher
Nodes: 5 (strongly recomended)
Operating System: UNIX-like only (Linux, BSD(except OpenBSD), MacOS X) Windows isn't supported

Cases

Logs ageregation
Caching system
Queue server
Pattern "Central dispatcher" (for sharding)
Solution that sits in front of multiple data sources and allow them to be treated as a single json database
Web-server and database for hosting static web-sites

Getting Started Basic modules

fluidB The project consists of two main modules:

fluidB- relational storage engine for small datasests
futrum- nosql storage engine for large datasests

Building

Install Golang 19.4 or higher build-essential in your operating system

This single node automatically becomes the leader. You can pass -h to fluidBd to list all configuration options. Installation

For Linux-systems you can install fluidB automaticly using installation script from our site:

Installation Script for Debian-based systems Installation Script for Red-Hat-based systems Installation Script for Arch-based systems

Open the link above
Download the script
Copy the tar-archive in your home directory
Open you terminal and then print (−terminalprompt)‘‘‘−terminalprompt)‘‘‘ chmod +x install.sh && ./install.sh```
Wait for the installation process to complete, after installation the application will start automatically

Start working with fluidB

To begin with, let's check that fluidB is running at all:

$ clif 127.0.0.1:4001> CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT) 0 row affected (0.000668 sec) 127.0.0.1:4001> .schema +-----------------------------------------------------------------------------+ | sql | +-----------------------------------------------------------------------------+ | CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT) | +-----------------------------------------------------------------------------+ 127.0.0.1:4001> INSERT INTO foo(name) VALUES("fiona") 1 row affected (0.000080 sec) 127.0.0.1:4001> SELECT * FROM foo +----+-------+ | id | name | +----+-------+ | 1 | fiona | +----+-------+

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

Fork the Project
Create your Feature Branch (git checkout -b feature/AmazingFeature)
Commit your Changes (git commit -m 'Add some AmazingFeature')
Push to the Branch (git push origin feature/AmazingFeature)
Open a Pull Request

License

Distributed under the BSD 3 License. See LICENSE for more information. FAQ This page provides answers to frequently asked questions regarding fluidB

What is the main idea of your project? The key idea of our product is that it solves the problems of processing, storing and analyzing a large amount of continuously incoming data, mainly geolocation data. It is intended primarily for application software developers and DBAs (Database Administrators). Our main goal is to provide the user with a simple yet flexible and functional tool for working with data.


How is your project different from KeyDB?

When developing our product, we took KeyDB as a basis, as an improved version of the Redis sub-database, easy and understandable to learn and operate. The difference between fluidB and KeyDB lies in the implementation of a shell module that allows you to run external modules (programs) that extend the basic functionality of the application without explicitly specifying the path to the module in the configuration file.

What is the multi-model database management system? A multi-model subdivision is such a subdivision that allows you to store data using different data storage schemes. For example, along with relational data (stored in the form of tables), data can be stored in the form of associative array, etc. Thus, incredible flexibility in storing data is achieved, depending on the type of data, the most suitable tool for storing and processing it is used.


Is there a fluidB implementation for the Windows operating system? No, there is no implementation of FluidB for the Windows operating system. our application is client-server, with on-board caching subdivision, focused on use mainly on servers, and the vast majority of servers run under the Linux operating system. We do not plan to release a version for Windows in the future.


Is fluidB in Linux repositories?

No, FluidB is still distributed as source.


Is there a docker image of the project? No, there is no docker image, because docker, like any virtualization environment, forms an additional layer of abstraction, which complicates both the development process itself and the program operation process. Our goal: "To create a high-performance subassembly that is as easy to use as possible"


Why are you confident that you will be able to create a high-quality, competitive software product that people need?

For several reasons. First, because our product is based on the well-proven Redis DBMS. Secondly, our software product is already used as a backend in the ws-stickleback project "Biology and Soil" of the Faculty of the Department of Ichthyology and Hydrobiology, St. Petersburg State University

Is there a paid version of your software product with advanced functionality?

There is such a version, it is calledEnterprise-version, this version includes both the functionality required by the client (discussed individually with each client) and the default functionality.

How is authorization used?

In the traditional sense, there is no authorization. We have a funny login form that we wrote to show how you can quickly implement it using JavaScript, it is used by default in our fluwc web interface, where you can make "typical requests" in the browser. FluidB is not intended to be publicly displayed. This is an application that works on the Internet, so we do not have authorization at this stage.

How does your product name stand for?

The name of our project is deciphered as follows: FluidB, this is a reference to the English word "fluid", which means "fluid" - a physical term meaning "a state of matter with parameters above critical", which very accurately characterizes our product. A critical state of a substance means a state in which it disappears the difference between its liquid and vapor phases. This is illustrated by the colors of the logo: At first, there is no load on the application (blue Greek letter "Phi"), then it increases sharply (three red subsequent letters), then the load begins to decrease and evenly distribute- orange symbol, differential, showing the rate of change of the value of the variable during balancing, and finally it is evenly distributed and becomes normal (seventh blue symbol). In our case, it should be interpreted as follows: thanks to our product, the line between low loads and very high loads is erased, i.e. to. it helps balance the load. A couple of words should be said separately about the symbol of the "flower" standing in front of the inscription: It personifies the mathematical graph of objects, tk. our subd is a multi-model supporting graph data storage model.

I have decided to purchase the Enterprise version of your product, where should I contact?

You shoud write the letter in email: gvsafronov@gmail.com Roadmap

Raft implementation for core databases (fluidB-engine)
Improve installation scripts
Improve futrum installation script
Research riak and improve it as a database-engine (futrum) for big datasets
Develop application-engine for fuidB
Develop hash-sharding
Rewrite temporal model

Cluster

While not strictly necessary to run fluidB, running multiple nodes means you'll have a fault-tolerant cluster. Start two more nodes, allowing the cluster to tolerate failure of a single node, like so:

fluidBd -node-id 2 -http-addr localhost:4003 -raft-addr localhost:4004 -join http://localhost:4001 ~/node.2
fluidBd -node-id 3 -http-addr localhost:4005 -raft-addr localhost:4006 -join http://localhost:4001 ~/node.3

This demonstration shows all 3 nodes running on the same host. In reality you probably wouldn't do this, and then you wouldn't need to select different -http-addr and -raft-addr ports for each fluidB node.

With just these few steps you've now got a fault-tolerant, distributed relational database. For full details on creating and managing real clusters, including running read-only nodes, check out this documentation.
Contact

Grigoriy Safronov - gvsafronov@gmail.com

Project Link: https://fluidb.icu
Limitations

    In-memory databases are currently limited to 2GiB (2147483648 bytes) in size. You can learn more about possible ways to get around this limit in the [documentation].

    Because fluidB peforms statement-based replication certain non-deterministic functions, e.g. RANDOM(), are rewritten by fluidB before being passed to the Raft system and SQLite.

    This has not been extensively tested, but you can directly read the SQLite file under any node at anytime, assuming you run in "on-disk" mode. However there is no guarantee that the SQLite file reflects all the changes that have taken place on the cluster unless you are sure the host node itself has received and applied all changes.

    In case it isn't obvious, fluidB does not replicate any changes made directly to any underlying SQLite file, when run in "on disk" mode. If you change the SQLite file directly, you may cause fluidB to fail. Only modify the database via the HTTP API.

    SQLite dot-commands such as .schema or .tables are not directly supported by the API, but the supports some very similar functionality. This is because those commands are features of the sqlite3 command, not SQLite itself.

Работает на Gitea Версия: 1.19.3 Страница: 88ms Шаблон: 7ms
Лицензии API

       cd fluidb && go install ./...

4. Configuration fluidB

  chmod +x fconf.sh && ./fconf.sh
  

5. Running server fluidb

  ./fluidbd ~/node.1

This single node automatically becomes the leader. You can pass -h to fluidBd to list all configuration options.

Installation

For Linux-systems you can install fluidB automaticly using installation script from our site:

Installation Script for Debian-based systems
Installation Script for Red-Hat-based systems
Installation Script for Arch-based systems

1. Open the link above
2. Download the script
3. Copy the tar-archive in your home directory
4. Open you terminal and then print (-terminal prompt) ``` chmod +x install.sh && ./install.sh```
5. Wait for the installation process to complete, after installation the application will start automatically
   

Start working with fluidB

To begin with, let's check that fluidB is running at all:

$ clif
127.0.0.1:4001> CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT)
0 row affected (0.000668 sec)
127.0.0.1:4001> .schema
+-----------------------------------------------------------------------------+
| sql                                                                         |
+-----------------------------------------------------------------------------+
| CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT)               |
+-----------------------------------------------------------------------------+
127.0.0.1:4001> INSERT INTO foo(name) VALUES("fiona")
1 row affected (0.000080 sec)
127.0.0.1:4001> SELECT * FROM foo
+----+-------+
| id | name  |
+----+-------+
| 1  | fiona |
+----+-------+

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the BSD 3 License. See LICENSE for more information.

FAQ

This page provides answers to frequently asked questions regarding fluidB

• What is the main idea of your project? The key idea of our product is that it solves the problems of processing, storing and analyzing a large amount of continuously incoming data, mainly geolocation data. It is intended primarily for application software developers and DBAs (Database Administrators). Our main goal is to provide the user with a simple yet flexible and functional tool for working with data.

• How is your project different from KeyDB?

When developing our product, we took KeyDB as a basis, as an improved version of the Redis sub-database, easy and understandable to learn and operate. The difference between fluidB and KeyDB lies in the implementation of a shell module that allows you to run external modules (programs) that extend the basic functionality of the application without explicitly specifying the path to the module in the configuration file.

• What is the multi-model database management system? A multi-model subdivision is such a subdivision that allows you to store data using different data storage schemes. For example, along with relational data (stored in the form of tables), data can be stored in the form of associative array, etc. Thus, incredible flexibility in storing data is achieved, depending on the type of data, the most suitable tool for storing and processing it is used.

• Is there a fluidB implementation for the Windows operating system? No, there is no implementation of FluidB for the Windows operating system. our application is client-server, with on-board caching subdivision, focused on use mainly on servers, and the vast majority of servers run under the Linux operating system. We do not plan to release a version for Windows in the future.

• Is fluidB in Linux repositories?
  
  No, FluidB is still distributed as source.

• Is there a docker image of the project? No, there is no docker image, because docker, like any virtualization environment, forms an additional layer of abstraction, which complicates both the development process itself and the program operation process. Our goal: "To create a high-performance subassembly that is as easy to use as possible"

• Why are you confident that you will be able to create a high-quality, competitive software product that people need?
  

For several reasons. First, because our product is based on the well-proven Redis DBMS. Secondly, our software product is already used as a backend in the ws-stickleback project "Biology and Soil" of the Faculty of the Department of Ichthyology and Hydrobiology, St. Petersburg State University

• Is there a paid version of your software product with advanced functionality?

There is such a version, it is calledEnterprise-version, this version includes both the functionality required by the client (discussed individually with each client) and the default functionality.

• How is authorization used?

In the traditional sense, there is no authorization. We have a funny login form that we wrote to show how you can quickly implement it using JavaScript, it is used by default in our fluwc web interface, where you can make "typical requests" in the browser. FluidB is not intended to be publicly displayed. This is an application that works on the Internet, so we do not have authorization at this stage.

• How does your product name stand for?

The name of our project is deciphered as follows: FluidB, this is a reference to the English word "fluid", which means "fluid" - a physical term meaning "a state of matter with parameters above critical", which very accurately characterizes our product. A critical state of a substance means a state in which it disappears the difference between its liquid and vapor phases. This is illustrated by the colors of the logo: At first, there is no load on the application (blue Greek letter "Phi"), then it increases sharply (three red subsequent letters), then the load begins to decrease and evenly distribute- orange symbol, differential, showing the rate of change of the value of the variable during balancing, and finally it is evenly distributed and becomes normal (seventh blue symbol). In our case, it should be interpreted as follows: thanks to our product, the line between low loads and very high loads is erased, i.e. to. it helps balance the load. A couple of words should be said separately about the symbol of the "flower" standing in front of the inscription: It personifies the mathematical graph of objects, tk. our subd is a multi-model supporting graph data storage model.

• I have decided to purchase the Enterprise version of your product, where should I contact?

You shoud write the letter in email: gvsafronov@gmail.com

Roadmap

• Raft implementation for core databases (fluidB-engine)
• Improve installation scripts
• Improve futrum installation script
• Research riak and improve it as a database-engine (futrum) for big datasets
• Develop application-engine for fuidB
• Develop hash-sharding
• Rewrite temporal model

Cluster

While not strictly necessary to run fluidB, running multiple nodes means you'll have a fault-tolerant cluster. Start two more nodes, allowing the cluster to tolerate failure of a single node, like so:
```bash
fluidBd -node-id 2 -http-addr localhost:4003 -raft-addr localhost:4004 -join http://localhost:4001 ~/node.2
fluidBd -node-id 3 -http-addr localhost:4005 -raft-addr localhost:4006 -join http://localhost:4001 ~/node.3

This demonstration shows all 3 nodes running on the same host. In reality you probably wouldn't do this, and then you wouldn't need to select different -http-addr and -raft-addr ports for each fluidB node.

With just these few steps you've now got a fault-tolerant, distributed relational database. For full details on creating and managing real clusters, including running read-only nodes, check out this documentation.

Contact

Grigoriy Safronov - gvsafronov@gmail.com

Project Link: https://fluidb.icu

Limitations

• In-memory databases are currently limited to 2GiB (2147483648 bytes) in size. You can learn more about possible ways to get around this limit in the [documentation].

• Because fluidB peforms statement-based replication certain non-deterministic functions, e.g. RANDOM(), are rewritten by fluidB before being passed to the Raft system and SQLite.

• This has not been extensively tested, but you can directly read the SQLite file under any node at anytime, assuming you run in "on-disk" mode. However there is no guarantee that the SQLite file reflects all the changes that have taken place on the cluster unless you are sure the host node itself has received and applied all changes.

• In case it isn't obvious, fluidB does not replicate any changes made directly to any underlying SQLite file, when run in "on disk" mode. If you change the SQLite file directly, you may cause fluidB to fail. Only modify the database via the HTTP API.

• SQLite dot-commands such as .schema or .tables are not directly supported by the API, but the supports some very similar functionality. This is because those commands are features of the sqlite3 command, not SQLite itself.