|
|
|
@ -5,33 +5,37 @@
|
|
|
|
# `cozo`
|
|
|
|
# `cozo`
|
|
|
|
|
|
|
|
|
|
|
|
A general-purpose, transactional, relational database
|
|
|
|
A general-purpose, transactional, relational database
|
|
|
|
that uses Datalog for query and focuses on graph data and algorithms.
|
|
|
|
that uses Datalog for query, is embeddable, and focuses on graph data and algorithms.
|
|
|
|
|
|
|
|
|
|
|
|
## Features
|
|
|
|
## Features
|
|
|
|
|
|
|
|
|
|
|
|
* Relational database with [Datalog](https://en.wikipedia.org/wiki/Datalog) as the query language
|
|
|
|
* Relational database with [Datalog](https://en.wikipedia.org/wiki/Datalog) as the query language
|
|
|
|
* Recursive queries, especially recursion through (safe) aggregation, capable of expressing complex graph operations and algorithms
|
|
|
|
* Recursive queries, recursion through (safe) aggregations, capable of expressing complex graph operations and
|
|
|
|
* Fixed rules providing efficient whole-graph algorithms which integrate seamlessly with Datalog
|
|
|
|
algorithms
|
|
|
|
* Rich set of built-in functions and aggregations
|
|
|
|
* Fixed rules for efficient whole-graph algorithms which integrate seamlessly with Datalog
|
|
|
|
* Only a single executable, trivial to deploy and run
|
|
|
|
* Rich set of built-in functions and aggregations
|
|
|
|
* [Embeddable](https://cozodb.github.io/current/manual/setup.html#embedding-cozo), can run in the same process as the application
|
|
|
|
* Easy to use from any programming language, or as a standalone program
|
|
|
|
* Easy to use from any programming language
|
|
|
|
* [Embeddable](https://cozodb.github.io/current/manual/setup.html#embedding-cozo), with ready-to-use bindings for
|
|
|
|
* Special support for [Jupyter](https://jupyter.org/) notebooks for integration with the Python DataScience ecosystem
|
|
|
|
Python, NodeJS and Java
|
|
|
|
|
|
|
|
* Single executable, trivial to deploy and run
|
|
|
|
|
|
|
|
* [Jupyter](https://jupyter.org/) notebooks integration, plays well with the DataScience ecosystem
|
|
|
|
* Modern, clean, flexible syntax, informative error messages
|
|
|
|
* Modern, clean, flexible syntax, informative error messages
|
|
|
|
|
|
|
|
|
|
|
|
## Teasers
|
|
|
|
## Teasers
|
|
|
|
|
|
|
|
|
|
|
|
Here `*route` is a relation with two columns `src` and `dst`,
|
|
|
|
Here `*route` is a relation with two columns `src` and `dst`,
|
|
|
|
representing a route between those airports.
|
|
|
|
representing a route between those airports.
|
|
|
|
|
|
|
|
|
|
|
|
Find airports reachable by one stop from Frankfurt Airport (code `FRA`):
|
|
|
|
Find airports reachable by one stop from Frankfurt Airport (code `FRA`):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
TODO replace with images
|
|
|
|
|
|
|
|
|
|
|
|
```js
|
|
|
|
```js
|
|
|
|
?[dst] := *route{src: 'FRA', dst: stop},
|
|
|
|
?[dst] := *route{src: 'FRA', dst: stop},
|
|
|
|
*route{src: stop, dst}
|
|
|
|
*route{src: stop, dst}
|
|
|
|
```
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Find airports reachable from Frankfurt with any number of stops
|
|
|
|
Find airports reachable from Frankfurt with any number of stops
|
|
|
|
with code starting with the letter `A`:
|
|
|
|
with code starting with the letter `A`:
|
|
|
|
|
|
|
|
|
|
|
|
```js
|
|
|
|
```js
|
|
|
|
@ -58,15 +62,35 @@ starting[airport] := airport = 'FRA'
|
|
|
|
?[src, dst, cost, path] <~ ShortestPathDijkstra(*route[], starting[])
|
|
|
|
?[src, dst, cost, path] <~ ShortestPathDijkstra(*route[], starting[])
|
|
|
|
```
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## Learning Cozo
|
|
|
|
Nice error messages when things go wrong:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
xxx
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Getting started
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
First we need to get Cozo installed on the local machine. As Cozo is embeddable,
|
|
|
|
|
|
|
|
there are lots of options for how we run Cozo. Follow one of the following guides that suits you best:
|
|
|
|
|
|
|
|
|
|
|
|
* Start with the [Tutorial](https://nbviewer.org/github/cozodb/cozo/blob/main/docs/tutorial/tutorial.ipynb) to learn the basics;
|
|
|
|
* Embedded in Python with JupyterLab (best interactive experience, recommended if you are comfortable with the Python
|
|
|
|
* Continue with the [Manual](https://cozodb.github.io/current/manual/) to understand the fine points.
|
|
|
|
ecosystem)
|
|
|
|
|
|
|
|
* Embedded in Python
|
|
|
|
|
|
|
|
* Embedded in NodeJS (Javascript)
|
|
|
|
|
|
|
|
* Embedded in Java (or any JVM language)
|
|
|
|
|
|
|
|
* Embedded in Rust
|
|
|
|
|
|
|
|
* Client/server with HTTP API
|
|
|
|
|
|
|
|
* Embedded in C/C++, or any language that has a C FFI
|
|
|
|
|
|
|
|
(Golang, R, Haskell, CommonLisp, Julia, Fortran, C#, Swift, ...)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
After you have it installed, you can start learning CozoScript:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
* Start with the [Tutorial](https://nbviewer.org/github/cozodb/cozo/blob/main/docs/tutorial/tutorial.ipynb) to learn the
|
|
|
|
|
|
|
|
basics;
|
|
|
|
|
|
|
|
* Continue with the [Manual](https://cozodb.github.io/current/manual/) for the fine points.
|
|
|
|
|
|
|
|
|
|
|
|
## Bug reports, discussions
|
|
|
|
## Bug reports, discussions
|
|
|
|
|
|
|
|
|
|
|
|
If you encounter a bug, first search for [past issues](https://github.com/cozodb/cozo/issues) to see
|
|
|
|
If you encounter a bug, first search for [past issues](https://github.com/cozodb/cozo/issues) to see
|
|
|
|
if it has already been reported. If not, open a new issue.
|
|
|
|
if it has already been reported. If not, open a new issue.
|
|
|
|
Please provide sufficient information so that we can diagnose the problem faster.
|
|
|
|
Please provide sufficient information so that we can diagnose the problem faster.
|
|
|
|
|
|
|
|
|
|
|
|
Other discussions about Cozo should be in [GitHub discussions](https://github.com/cozodb/cozo/discussions).
|
|
|
|
Other discussions about Cozo should be in [GitHub discussions](https://github.com/cozodb/cozo/discussions).
|
|
|
|
@ -84,67 +108,68 @@ in specific situations:
|
|
|
|
* You have a lot of interconnected relations
|
|
|
|
* You have a lot of interconnected relations
|
|
|
|
and the usual queries need to relate many relations together.
|
|
|
|
and the usual queries need to relate many relations together.
|
|
|
|
In other words, you need to query a complex graph.
|
|
|
|
In other words, you need to query a complex graph.
|
|
|
|
* An example is a system granting permissions to users for specific tasks.
|
|
|
|
* An example is a system granting permissions to users for specific tasks.
|
|
|
|
In this case, users may have roles,
|
|
|
|
In this case, users may have roles,
|
|
|
|
belong to an organization hierarchy, and tasks similarly have organizations
|
|
|
|
belong to an organization hierarchy, and tasks similarly have organizations
|
|
|
|
and special provisions associated with them.
|
|
|
|
and special provisions associated with them.
|
|
|
|
The granting process itself may also be a complicated rule encoded as data
|
|
|
|
The granting process itself may also be a complicated rule encoded as data
|
|
|
|
within the database.
|
|
|
|
within the database.
|
|
|
|
* With a traditional database,
|
|
|
|
* With a traditional database,
|
|
|
|
the corresponding SQL tend to become
|
|
|
|
the corresponding SQL tend to become
|
|
|
|
an entangled web of nested queries, with many tables joined together,
|
|
|
|
an entangled web of nested queries, with many tables joined together,
|
|
|
|
and maybe even with some recursive CTE thrown in. This is hard to maintain,
|
|
|
|
and maybe even with some recursive CTE thrown in. This is hard to maintain,
|
|
|
|
and worse, the performance is unpredictable since query optimizers in general
|
|
|
|
and worse, the performance is unpredictable since query optimizers in general
|
|
|
|
fail when you have over twenty tables joined together.
|
|
|
|
fail when you have over twenty tables joined together.
|
|
|
|
* With Cozo, on the other hand, [Horn clauses](https://en.wikipedia.org/wiki/Horn_clause)
|
|
|
|
* With Cozo, on the other hand, [Horn clauses](https://en.wikipedia.org/wiki/Horn_clause)
|
|
|
|
make it easy to break
|
|
|
|
make it easy to break
|
|
|
|
the logic into smaller pieces and write clear, easily testable queries.
|
|
|
|
the logic into smaller pieces and write clear, easily testable queries.
|
|
|
|
Furthermore, the deterministic evaluation order makes identifying and solving
|
|
|
|
Furthermore, the deterministic evaluation order makes identifying and solving
|
|
|
|
performance problems easier.
|
|
|
|
performance problems easier.
|
|
|
|
* Your data may be simple, even a single table, but it is inherently a graph.
|
|
|
|
* Your data may be simple, even a single table, but it is inherently a graph.
|
|
|
|
* We have seen an example in the [Tutorial](https://nbviewer.org/github/cozodb/cozo/blob/main/docs/tutorial/tutorial.ipynb):
|
|
|
|
* We have seen an example in
|
|
|
|
the air route dataset, where the key relation contains the routes connecting airports.
|
|
|
|
the [Tutorial](https://nbviewer.org/github/cozodb/cozo/blob/main/docs/tutorial/tutorial.ipynb):
|
|
|
|
* In traditional databases, when you are given a new relation,
|
|
|
|
the air route dataset, where the key relation contains the routes connecting airports.
|
|
|
|
you try to understand it by running aggregations on it to collect statistics:
|
|
|
|
* In traditional databases, when you are given a new relation,
|
|
|
|
what is the distribution of values, how are the columns correlated, etc.
|
|
|
|
you try to understand it by running aggregations on it to collect statistics:
|
|
|
|
* In Cozo you can do the same exploratory analysis,
|
|
|
|
what is the distribution of values, how are the columns correlated, etc.
|
|
|
|
except now you also have graph algorithms that you can
|
|
|
|
* In Cozo you can do the same exploratory analysis,
|
|
|
|
easily apply to understand things such as: what is the most _connected_ entity,
|
|
|
|
except now you also have graph algorithms that you can
|
|
|
|
how are the nodes connected, and what are the _communities_ structure within the nodes.
|
|
|
|
easily apply to understand things such as: what is the most _connected_ entity,
|
|
|
|
|
|
|
|
how are the nodes connected, and what are the _communities_ structure within the nodes.
|
|
|
|
* Your data contains hidden structures that only become apparent when you
|
|
|
|
* Your data contains hidden structures that only become apparent when you
|
|
|
|
identify the _scales_ of the relevant structures.
|
|
|
|
identify the _scales_ of the relevant structures.
|
|
|
|
* Examples are most real networks, such as social networks,
|
|
|
|
* Examples are most real networks, such as social networks,
|
|
|
|
which have a very rich hierarchy of structures
|
|
|
|
which have a very rich hierarchy of structures
|
|
|
|
* In a traditional database, you are limited to doing nested aggregations and filtering,
|
|
|
|
* In a traditional database, you are limited to doing nested aggregations and filtering,
|
|
|
|
i.e. a form of multifaceted data analysis. For example, you can analyze by gender, geography,
|
|
|
|
i.e. a form of multifaceted data analysis. For example, you can analyze by gender, geography,
|
|
|
|
job or combinations of them. For structures hidden in other ways,
|
|
|
|
job or combinations of them. For structures hidden in other ways,
|
|
|
|
or if such categorizing tags are not already present in your data,
|
|
|
|
or if such categorizing tags are not already present in your data,
|
|
|
|
you are out of luck.
|
|
|
|
you are out of luck.
|
|
|
|
* With Cozo, you can now deal with emergent and fuzzy structures by using e.g.
|
|
|
|
* With Cozo, you can now deal with emergent and fuzzy structures by using e.g.
|
|
|
|
community detection algorithms, and collapse the original graph into a coarse-grained
|
|
|
|
community detection algorithms, and collapse the original graph into a coarse-grained
|
|
|
|
graph consisting of super-nodes and super-edges.
|
|
|
|
graph consisting of super-nodes and super-edges.
|
|
|
|
The process can be iterated to gain insights into even higher-order emergent structures.
|
|
|
|
The process can be iterated to gain insights into even higher-order emergent structures.
|
|
|
|
This is possible in a social network with only edges and _no_ categorizing tags
|
|
|
|
This is possible in a social network with only edges and _no_ categorizing tags
|
|
|
|
associated with nodes at all,
|
|
|
|
associated with nodes at all,
|
|
|
|
and the discovered structures almost always have meanings correlated to real-world events and
|
|
|
|
and the discovered structures almost always have meanings correlated to real-world events and
|
|
|
|
organizations, for example, forms of collusion and crime rings.
|
|
|
|
organizations, for example, forms of collusion and crime rings.
|
|
|
|
Also, from a performance perspective,
|
|
|
|
Also, from a performance perspective,
|
|
|
|
coarse-graining is a required step in analyzing the so-called big data,
|
|
|
|
coarse-graining is a required step in analyzing the so-called big data,
|
|
|
|
since many graph algorithms have high complexity and are only applicable to
|
|
|
|
since many graph algorithms have high complexity and are only applicable to
|
|
|
|
the coarse-grained small or medium networks.
|
|
|
|
the coarse-grained small or medium networks.
|
|
|
|
* You want to understand your live business data better by augmenting it into a _knowledge graph_.
|
|
|
|
* You want to understand your live business data better by augmenting it into a _knowledge graph_.
|
|
|
|
* For example, your sales database contains product, buyer, inventory, and invoice tables.
|
|
|
|
* For example, your sales database contains product, buyer, inventory, and invoice tables.
|
|
|
|
The augmentation is external data about the entities in your data in the form of _taxonomies_
|
|
|
|
The augmentation is external data about the entities in your data in the form of _taxonomies_
|
|
|
|
and _ontologies_ in layers.
|
|
|
|
and _ontologies_ in layers.
|
|
|
|
* This is inherently a graph-theoretic undertaking and traditional databases are not suitable.
|
|
|
|
* This is inherently a graph-theoretic undertaking and traditional databases are not suitable.
|
|
|
|
Usually, a dedicated graph processing engine is used, separate from the main database.
|
|
|
|
Usually, a dedicated graph processing engine is used, separate from the main database.
|
|
|
|
* With Cozo, it is possible to keep your live data and knowledge graph analysis together,
|
|
|
|
* With Cozo, it is possible to keep your live data and knowledge graph analysis together,
|
|
|
|
and importing new external data and doing analysis is just a few lines of code away.
|
|
|
|
and importing new external data and doing analysis is just a few lines of code away.
|
|
|
|
This ease of use means that you will do the analysis much more often, with a perhaps much wider scope.
|
|
|
|
This ease of use means that you will do the analysis much more often, with a perhaps much wider scope.
|
|
|
|
|
|
|
|
|
|
|
|
## Status of the project
|
|
|
|
## Status of the project
|
|
|
|
|
|
|
|
|
|
|
|
Cozo is very young and **not** production-ready yet,
|
|
|
|
Cozo is very young and **not** production-ready yet,
|
|
|
|
but we encourage you to try it out for your use case.
|
|
|
|
but we encourage you to try it out for your use case.
|
|
|
|
Any feedback is welcome.
|
|
|
|
Any feedback is welcome.
|
|
|
|
|
|
|
|
|
|
|
|
@ -165,26 +190,29 @@ Further down the road:
|
|
|
|
* More tuning options
|
|
|
|
* More tuning options
|
|
|
|
* Streaming/reactive data
|
|
|
|
* Streaming/reactive data
|
|
|
|
* Extension system
|
|
|
|
* Extension system
|
|
|
|
* The core of Cozo should be kept small at all times. Additional functionalities should be in extensions for the user to choose from.
|
|
|
|
* The core of Cozo should be kept small at all times. Additional functionalities should be in extensions for the
|
|
|
|
* What can be extended: datatypes, functions, aggregations, and fixed algorithms.
|
|
|
|
user to choose from.
|
|
|
|
* Extensions should be written in a compiled language such as Rust or C++ and compiled into a dynamic library, to be loaded by Cozo at runtime.
|
|
|
|
* What can be extended: datatypes, functions, aggregations, and fixed algorithms.
|
|
|
|
* There will probably be a few "official" extension bundles, such as
|
|
|
|
* Extensions should be written in a compiled language such as Rust or C++ and compiled into a dynamic library, to be
|
|
|
|
* arbitrary precision arithmetic
|
|
|
|
loaded by Cozo at runtime.
|
|
|
|
* full-text "indexing" and searching
|
|
|
|
* There will probably be a few "official" extension bundles, such as
|
|
|
|
* relations that can emulate spatial and other types of non-lexicographic indices
|
|
|
|
* arbitrary precision arithmetic
|
|
|
|
* reading from external databases directly
|
|
|
|
* full-text "indexing" and searching
|
|
|
|
* more exotic graph algorithms
|
|
|
|
* relations that can emulate spatial and other types of non-lexicographic indices
|
|
|
|
|
|
|
|
* reading from external databases directly
|
|
|
|
|
|
|
|
* more exotic graph algorithms
|
|
|
|
|
|
|
|
|
|
|
|
Ideas and discussions are welcome.
|
|
|
|
Ideas and discussions are welcome.
|
|
|
|
|
|
|
|
|
|
|
|
## Storage engine
|
|
|
|
## Storage engine
|
|
|
|
|
|
|
|
|
|
|
|
Cozo is written in Rust, with [RocksDB](http://rocksdb.org/) as the storage engine
|
|
|
|
Cozo is written in Rust, with [RocksDB](http://rocksdb.org/) as the storage engine
|
|
|
|
(this may change in the future).
|
|
|
|
(this may change in the future).
|
|
|
|
We manually wrote the C++/Rust bindings for RocksDB with [cxx](https://cxx.rs/).
|
|
|
|
We manually wrote the C++/Rust bindings for RocksDB with [cxx](https://cxx.rs/).
|
|
|
|
|
|
|
|
|
|
|
|
## Licensing
|
|
|
|
## Licensing
|
|
|
|
|
|
|
|
|
|
|
|
The contents of this project are licensed under AGPL-3.0 or later, except:
|
|
|
|
The contents of this project are licensed under AGPL-3.0 or later, except:
|
|
|
|
* Files under `cozorocks/` are licensed under MIT, or Apache-2.0, or BSD-3-Clause;
|
|
|
|
|
|
|
|
|
|
|
|
* Files under `cozorocks/`, `python/`, `nodejs/`, `java/` and `c/` are licensed under MIT, or Apache-2.0, or BSD-3-Clause;
|
|
|
|
* Files under `docs/` are licensed under CC BY-SA 4.0.
|
|
|
|
* Files under `docs/` are licensed under CC BY-SA 4.0.
|
Ziyang Hu
2 years ago
