updated documentation and added blocking doc

master
Dvir Volk 8 years ago
parent e7495e0234
commit 2a9b54f7b9

@ -0,0 +1,265 @@
Blocking commands in Redis modules
===
Redis has a few blocking commands among the built-in set of commands.
One of the most used is `BLPOP` (or the symmetric `BRPOP`) which blocks
waiting for elements arriving in a list.
The interesting fact about blocking commands is that they do not block
the whole server, but just the client calling them. Usually the reason to
block is that we expect some external event to happen: this can be
some change in the Redis data structures like in the `BLPOP` case, a
long computation happening in a thread, to receive some data from the
network, and so forth.
Redis modules have the ability to implement blocking commands as well,
this documentation shows how the API works and describes a few patterns
that can be used in order to model blocking commands.
How blocking and resuming works.
---
_Note: You may want to check the `helloblock.c` example in the Redis source tree
inside the `src/modules` directory, for a simple to understand example
on how the blocking API is applied._
In Redis modules, commands are implemented by callback functions that
are invoked by the Redis core when the specific command is called
by the user. Normally the callback terminates its execution sending
some reply to the client. Using the following function instead, the
function implementing the module command may request that the client
is put into the blocked state:
RedisModuleBlockedClient *RedisModule_BlockClient(RedisModuleCtx *ctx, RedisModuleCmdFunc reply_callback, RedisModuleCmdFunc timeout_callback, void (*free_privdata)(void*), long long timeout_ms);
The function returns a `RedisModuleBlockedClient` object, which is later
used in order to unblock the client. The arguments have the following
meaning:
* `ctx` is the command execution context as usually in the rest of the API.
* `reply_callback` is the callback, having the same prototype of a normal command function, that is called when the client is unblocked in order to return a reply to the client.
* `timeout_callback` is the callback, having the same prototype of a normal command function that is called when the client reached the `ms` timeout.
* `free_privdata` is the callback that is called in order to free the private data. Private data is a pointer to some data that is passed between the API used to unblock the client, to the callback that will send the reply to the client. We'll see how this mechanism works later in this document.
* `ms` is the timeout in milliseconds. When the timeout is reached, the timeout callback is called and the client is automatically aborted.
Once a client is blocked, it can be unblocked with the following API:
int RedisModule_UnblockClient(RedisModuleBlockedClient *bc, void *privdata);
The function takes as argument the blocked client object returned by
the previous call to `RedisModule_BlockClient()`, and unblock the client.
Immediately before the client gets unblocked, the `reply_callback` function
specified when the client was blocked is called: this function will
have access to the `privdata` pointer used here.
IMPORTANT: The above function is thread safe, and can be called from within
a thread doing some work in order to implement the command that blocked
the client.
The `privdata` data will be freed automatically using the `free_privdata`
callback when the client is unblocked. This is useful **since the reply
callback may never be called** in case the client timeouts or disconnects
from the server, so it's important that it's up to an external function
to have the responsibility to free the data passed if needed.
To better understand how the API works, we can imagine writing a command
that blocks a client for one second, and then send as reply "Hello!".
Note: arity checks and other non important things are not implemented
int his command, in order to take the example simple.
int Example_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv,
int argc)
{
RedisModuleBlockedClient *bc =
RedisModule_BlockClient(ctx,reply_func,timeout_func,NULL,0);
pthread_t tid;
pthread_create(&tid,NULL,threadmain,bc);
return REDISMODULE_OK;
}
void *threadmain(void *arg) {
RedisModuleBlockedClient *bc = arg;
sleep(1); /* Wait one second and unblock. */
RedisModule_UnblockClient(bc,NULL);
}
The above command blocks the client ASAP, spawining a thread that will
wait a second and will unblock the client. Let's check the reply and
timeout callbacks, which are in our case very similar, since they
just reply the client with a different reply type.
int reply_func(RedisModuleCtx *ctx, RedisModuleString **argv,
int argc)
{
return RedisModule_ReplyWithSimpleString(ctx,"Hello!");
}
int timeout_func(RedisModuleCtx *ctx, RedisModuleString **argv,
int argc)
{
return RedisModule_ReplyWithNull(ctx);
}
The reply callback just sends the "Hello!" string to the client.
The important bit here is that the reply callback is called when the
client is unblocked from the thread.
The timeout command returns `NULL`, as it often happens with actual
Redis blocking commands timing out.
Passing reply data when unblocking
---
The above example is simple to understand but lacks an important
real world aspect of an actual blocking command implementation: often
the reply function will need to know what to reply to the client,
and this information is often provided as the client is unblocked.
We could modify the above example so that the thread generates a
random number after waiting one second. You can think at it as an
actually expansive operation of some kind. Then this random number
can be passed to the reply function so that we return it to the command
caller. In order to make this working, we modify the functions as follow:
void *threadmain(void *arg) {
RedisModuleBlockedClient *bc = arg;
sleep(1); /* Wait one second and unblock. */
long *mynumber = RedisModule_Alloc(sizeof(long));
*mynumber = rand();
RedisModule_UnblockClient(bc,mynumber);
}
As you can see, now the unblocking call is passing some private data,
that is the `mynumber` pointer, to the reply callback. In order to
obtain this private data, the reply callback will use the following
fnuction:
void *RedisModule_GetBlockedClientPrivateData(RedisModuleCtx *ctx);
So our reply callback is modified like that:
int reply_func(RedisModuleCtx *ctx, RedisModuleString **argv,
int argc)
{
long *mynumber = RedisModule_GetBlockedClientPrivateData(ctx);
/* IMPORTANT: don't free mynumber here, but in the
* free privdata callback. */
return RedisModule_ReplyWithLongLong(ctx,mynumber);
}
Note that we also need to pass a `free_privdata` function when blocking
the client with `RedisModule_BlockClient()`, since the allocated
long value must be freed. Our callback will look like the following:
void free_privdata(void *privdata) {
RedisModule_Free(privdata);
}
NOTE: It is important to stress that the private data is best freed in the
`free_privdata` callback becaues the reply function may not be called
if the client disconnects or timeout.
Also note that the private data is also accessible from the timeout
callback, always using the `GetBlockedClientPrivateData()` API.
Aborting the blocking of a client
---
One problem that sometimes arises is that we need to allocate resources
in order to implement the non blocking command. So we block the client,
then, for example, try to create a thread, but the thread creation function
returns an error. What to do in such a condition in order to recover? We
don't want to take the client blocked, nor we want to call `UnblockClient()`
because this will trigger the reply callback to be called.
In this case the best thing to do is to use the following function:
int RedisModule_AbortBlock(RedisModuleBlockedClient *bc);
Practically this is how to use it:
int Example_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv,
int argc)
{
RedisModuleBlockedClient *bc =
RedisModule_BlockClient(ctx,reply_func,timeout_func,NULL,0);
pthread_t tid;
if (pthread_create(&tid,NULL,threadmain,bc) != 0) {
RedisModule_AbortBlock(bc);
RedisModule_ReplyWithError(ctx,"Sorry can't create a thread");
}
return REDISMODULE_OK;
}
The client will be unblocked but the reply callback will not be called.
Implementing the command, reply and timeout callback using a single function
---
The following functions can be used in order to implement the reply and
callback with the same function that implements the primary command
function:
int RedisModule_IsBlockedReplyRequest(RedisModuleCtx *ctx);
int RedisModule_IsBlockedTimeoutRequest(RedisModuleCtx *ctx);
So I could rewrite the example command without using a separated
reply and timeout callback:
int Example_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv,
int argc)
{
if (RedisModule_IsBlockedReplyRequest(ctx)) {
long *mynumber = RedisModule_GetBlockedClientPrivateData(ctx);
return RedisModule_ReplyWithLongLong(ctx,mynumber);
} else if (RedisModule_IsBlockedTimeoutRequest) {
return RedisModule_ReplyWithNull(ctx);
}
RedisModuleBlockedClient *bc =
RedisModule_BlockClient(ctx,reply_func,timeout_func,NULL,0);
pthread_t tid;
if (pthread_create(&tid,NULL,threadmain,bc) != 0) {
RedisModule_AbortBlock(bc);
RedisModule_ReplyWithError(ctx,"Sorry can't create a thread");
}
return REDISMODULE_OK;
}
Functionally is the same but there are people that will prefer the less
verbose implementation that concentrates most of the command logic in a
single function.
Working on copies of data inside a thread
---
An interesting pattern in order to work with threads implementing the
slow part of a command, is to work with a copy of the data, so that
while some operation is performed in a key, the user continues to see
the old version. However when the thread terminated its work, the
representations are swapped and the new, processed version, is used.
An example of this approach is the
[Neural Redis module](https://github.com/antirez/neural-redis)
where neural networks are trained in different threads while the
user can still execute and inspect their older versions.
Future work
---
An API is work in progress right now in order to allow Redis modules APIs
to be called in a safe way from threads, so that the threaded command
can access the data space and do incremental operations.
There is no ETA for this feature but it may appear in the course of the
Redis 4.0 release at some point.

@ -46,6 +46,8 @@ Read this before starting, as it's more than an API reference.
3. [TYPES.md](TYPES.md) - Describes the API for creating new data structures inside Redis modules, 3. [TYPES.md](TYPES.md) - Describes the API for creating new data structures inside Redis modules,
copied from the Redis repo. copied from the Redis repo.
4. [BLOCK.md](BLOCK.md) - Describes the API for blocking a client while performing asynchronous tasks on a separate thread.
# Quick Start Guide # Quick Start Guide

@ -1,11 +1,12 @@
# Native types in Redis modules Native types in Redis modules
===
Redis modules can access Redis built-in data structures both at a high level, Redis modules can access Redis built-in data structures both at high level,
by calling Redis commands, and at a low level, by manipulating the data structures by calling Redis commands, and at low level, by manipulating the data structures
directly. directly.
By using these capabilities in order to build new abstractions on top of existing By using these capabilities in order to build new abstractions on top of existing
Redis data structures, or by using the strings DMA (direct memory access) in order to encode modules Redis data structures, or by using strings DMA in order to encode modules
data structures into Redis strings, it is possible to create modules that data structures into Redis strings, it is possible to create modules that
*feel like* they are exporting new data types. However, for more complex *feel like* they are exporting new data types. However, for more complex
problems, this is not enough, and the implementation of new data structures problems, this is not enough, and the implementation of new data structures
@ -17,53 +18,56 @@ the API exported by the Redis modules system in order to create new data
structures and handle the serialization in RDB files, the rewriting process structures and handle the serialization in RDB files, the rewriting process
in AOF, the type reporting via the `TYPE` command, and so forth. in AOF, the type reporting via the `TYPE` command, and so forth.
## Overview of native types Overview of native types
---
A module exporting a native type is composed of the following parts: A module exporting a native type is composed of the following main parts:
* The implementation of some kind of new data structure and of commands operating on the new data structure. * The implementation of some kind of new data structure and of commands operating on the new data structure.
* A set of callbacks that handle: RDB save, RDB load, AOF rewrite, releasing of a value associated with a key (for example, through `DEL`), * A set of callbacks that handle: RDB saving, RDB loading, AOF rewriting, releasing of a value associated with a key, calculation of a value digest (hash) to be used with the `DEBUG DIGEST` command.
and calculating a value digest (hash) to be used with the `DEBUG DIGEST` command (not implemented).
* A 9 characters name that is unique to each module native data type. * A 9 characters name that is unique to each module native data type.
* An encoding version, used to persist into RDB files a module-specific data version, so that a module will be able to * An encoding version, used to persist into RDB files a module-specific data version, so that a module will be able to load older representations from RDB files.
load older representations from RDB files.
While handling RDB load, save and AOF rewrite may look complex as a first glance, the modules API provides a very While to handle RDB loading, saving and AOF rewriting may look complex as a first glance, the modules API provide very high level function for handling all this, without requiring the user to handle read/write errors, so in practical terms, writing a new data structure for Redis is a simple task.
high level of functionality for handling these tasks, without requiring the user to handle read/write errors, so in practical
terms, writing a new data structure for Redis is a simple task.
A **very easy** to understand but complete example of native type implementation A **very easy** to understand but complete example of native type implementation
is available inside the Redis distribution in the `/modules/hellotype.c` file. is available inside the Redis distribution in the `/modules/hellotype.c` file.
The reader is encouraged to read the documentation by looking at this example The reader is encouraged to read the documentation by looking at this example
implementation to see how things are applied in the practice. implementation to see how things are applied in the practice.
## Registering a new data type Registering a new data type
===
In order to register a new native type with the Redis core, the module declares a global variable that will hold a reference to the data type. In order to register a new native type into the Redis core, the module needs
The API to register the data type will return a data type reference stored in the global variable. to declare a global variable that will hold a reference to the data type.
The API to register the data type will return a data type reference that will
be stored in the global variable.
static RedisModuleType *MyType; static RedisModuleType *MyType;
#define MYTYPE_ENCODING_VERSION 0 #define MYTYPE_ENCODING_VERSION 0
int RedisModule_OnLoad(RedisModuleCtx *ctx) { int RedisModule_OnLoad(RedisModuleCtx *ctx) {
MyType = RedisModule_CreateDataType("MyType-AZ", MYTYPE_ENCODING_VERSION, RedisModuleTypeMethods tm = {
MyTypeRDBLoad, MyTypeRDBSave, MyTypeAOFRewrite, MyTypeDigest, .version = REDISMODULE_TYPE_METHOD_VERSION,
MyTypeFree); .rdb_load = MyTypeRDBLoad,
.rdb_save = MyTypeRDBSave,
.aof_rewrite = MyTypeAOFRewrite,
.free = MyTypeFree
};
MyType = RedisModule_CreateDataType("MyType-AZ",
MYTYPE_ENCODING_VERSION, &tm);
if (MyType == NULL) return REDISMODULE_ERR; if (MyType == NULL) return REDISMODULE_ERR;
} }
A single API call registers the new type, with a number of function pointers passed as arguments. The prototype of As you can see from the example above, a single API call is needed in order to
`RedisModule_CreateDataType` is the following: register the new type. However a number of function pointers are passed as
arguments. Certain are optionals while some are mandatory. The above set
moduleType *RedisModule_CreateDataType(RedisModuleCtx *ctx, of methods *must* be passed, while `.digest` and `.mem_usage` are optional
const char *name, int encver, and are currently not actually supported by the modules internals, so for
moduleTypeLoadFunc rdb_load, now you can just ignore them.
moduleTypeSaveFunc rdb_save,
moduleTypeRewriteFunc aof_rewrite,
moduleTypeDigestFunc digest,
moduleTypeFreeFunc free);
The `ctx` argument is the context received from the `OnLoad` function. The `ctx` argument is the context that we receive in the `OnLoad` function.
The type `name` is a 9 character name in the character set that includes The type `name` is a 9 character name in the character set that includes
from `A-Z`, `a-z`, `0-9`, plus the underscore `_` and minus `-` characters. from `A-Z`, `a-z`, `0-9`, plus the underscore `_` and minus `-` characters.
@ -72,12 +76,15 @@ ecosystem, so be creative, use both lower-case and upper case if it makes
sense, and try to use the convention of mixing the type name with the name sense, and try to use the convention of mixing the type name with the name
of the author of the module, to create a 9 character unique name. of the author of the module, to create a 9 character unique name.
**NOTE:** It is very important that the name is exactly 9 chars or the
registration of the type will fail. Read more to understand why.
For example if I'm building a *b-tree* data structure and my name is *antirez* For example if I'm building a *b-tree* data structure and my name is *antirez*
I'll call my type **btree1-az**. The name, converted to a 64 bit integer, I'll call my type **btree1-az**. The name, converted to a 64 bit integer,
is stored inside the RDB file when saving the type, and will be used when the is stored inside the RDB file when saving the type, and will be used when the
RDB data is loaded in order to resolve which module can load the data. If Redis RDB data is loaded in order to resolve what module can load the data. If Redis
finds no matching module, the integer is converted back to a name in order to finds no matching module, the integer is converted back to a name in order to
provide some clue to the user about the missing module in order to load provide some clue to the user about what module is missing in order to load
the data. the data.
The type name is also used as a reply for the `TYPE` command when called The type name is also used as a reply for the `TYPE` command when called
@ -85,7 +92,7 @@ with a key holding the registered type.
The `encver` argument is the encoding version used by the module to store data The `encver` argument is the encoding version used by the module to store data
inside the RDB file. For example I can start with an encoding version of 0, inside the RDB file. For example I can start with an encoding version of 0,
but later when I release version 2 of my module, I can switch encoding to but later when I release version 2.0 of my module, I can switch encoding to
something better. The new module will register with an encoding version of 1, something better. The new module will register with an encoding version of 1,
so when it saves new RDB files, the new version will be stored on disk. However so when it saves new RDB files, the new version will be stored on disk. However
when loading RDB files, the module `rdb_load` method will be called even if when loading RDB files, the module `rdb_load` method will be called even if
@ -93,28 +100,33 @@ there is data found for a different encoding version (and the encoding version
is passed as argument to `rdb_load`), so that the module can still load old is passed as argument to `rdb_load`), so that the module can still load old
RDB files. RDB files.
The remaining arguments `rdb_load`, `rdb_save`, `aof_rewrite`, `digest` and The last argument is a structure used in order to pass the type methods to the
`free` are all callbacks with the following prototypes and uses: registration function: `rdb_load`, `rdb_save`, `aof_rewrite`, `digest` and
`free` and `mem_usage` are all callbacks with the following prototypes and uses:
typedef void *(*RedisModuleTypeLoadFunc)(RedisModuleIO *rdb, int encver); typedef void *(*RedisModuleTypeLoadFunc)(RedisModuleIO *rdb, int encver);
typedef void (*RedisModuleTypeSaveFunc)(RedisModuleIO *rdb, void *value); typedef void (*RedisModuleTypeSaveFunc)(RedisModuleIO *rdb, void *value);
typedef void (*RedisModuleTypeRewriteFunc)(RedisModuleIO *aof, RedisModuleString *key, void *value); typedef void (*RedisModuleTypeRewriteFunc)(RedisModuleIO *aof, RedisModuleString *key, void *value);
typedef size_t (*RedisModuleTypeMemUsageFunc)(void *value);
typedef void (*RedisModuleTypeDigestFunc)(RedisModuleDigest *digest, void *value); typedef void (*RedisModuleTypeDigestFunc)(RedisModuleDigest *digest, void *value);
typedef void (*RedisModuleTypeFreeFunc)(void *value); typedef void (*RedisModuleTypeFreeFunc)(void *value);
* `rdb_load` is called when loading data from the RDB file. It loads data in the same format as `rdb_save` produces. * `rdb_load` is called when loading data from the RDB file. It loads data in the same format as `rdb_save` produces.
* `rdb_save` is called when saving data to the RDB file. * `rdb_save` is called when saving data to the RDB file.
* `aof_rewrite` is called when the AOF is being rewritten, and the module needs to tell Redis the sequence of commands * `aof_rewrite` is called when the AOF is being rewritten, and the module needs to tell Redis what is the sequence of commands to recreate the content of a given key.
required to recreate the content of a given key. * `digest` is called when `DEBUG DIGEST` is executed and a key holding this module type is found. Currently this is not yet implemented so the function ca be left empty.
* `digest` is called when `DEBUG DIGEST` is executed and a key holding this module type is found. Currently this is not * `mem_usage` is called when the `MEMORY` command ask for the total memory consumed by a specific key, and is used in order to get the amount of bytes used by the module value.
yet implemented so the function can be left empty. * `free` is called when a key with the module native type is deleted via `DEL` or in any other mean, in order to let the module reclaim the memory associated with such a value.
* `free` is called when a key with the module native type is deleted via `DEL` or by other meand, in order to let the
module reclaim the memory associated with such a value. Ok, but *why* modules types require a 9 characters name?
---
### *Why* modules types require a 9 characters name? Oh, I understand you need to understand this, so here is a very specific
explanation.
When Redis persists to RDB files, modules specific data types need to be be persisted as well. RDB files are When Redis persists to RDB files, modules specific data types require to
sequences of key-value pairs like the following: be persisted as well. Now RDB files are sequences of key-value pairs
like the following:
[1 byte type] [key] [a type specific value] [1 byte type] [key] [a type specific value]
@ -124,22 +136,22 @@ course this is not enough, we need the information needed to link a specific
value with a specific module type that is able to load and handle it. value with a specific module type that is able to load and handle it.
So when we save a `type specific value` about a module, we prefix it with So when we save a `type specific value` about a module, we prefix it with
a 64 bit integer. 64 bits is large enough to store the information needed a 64 bit integer. 64 bits is large enough to store the informations needed
in order to lookup the module that can handle that specific type, but is in order to lookup the module that can handle that specific type, but is
short enough that we can prefix each module value we store inside the RDB short enough that we can prefix each module value we store inside the RDB
without making the final RDB file too big. At the same time, this solution without making the final RDB file too big. At the same time, this solution
of prefixing the value with a 64 bit *signature* does not require doing of prefixing the value with a 64 bit *signature* does not require to do
strange things like defining in the RDB header a list of modules specific strange things like defining in the RDB header a list of modules specific
types. Everything is pretty simple. types. Everything is pretty simple.
What you can store in 64 bits in order to identify a given module in So, what you can store in 64 bits in order to identify a given module in
a reliable way? Well if you build a character set of 64 symbols, you can a reliable way? Well if you build a character set of 64 symbols, you can
easily store 9 characters of 6 bits, and you are left with 10 bits, that easily store 9 characters of 6 bits, and you are left with 10 bits, that
are used in order to store the *encoding version* of the type, so that are used in order to store the *encoding version* of the type, so that
the same type can evolve in the future and provide a different and more the same type can evolve in the future and provide a different and more
efficient or updated serialization format for RDB files. efficient or updated serialization format for RDB files.
The 64 bit prefix stored before each module value is as follows: So the 64 bit prefix stored before each module value is like the following:
6|6|6|6|6|6|6|6|6|10 6|6|6|6|6|6|6|6|6|10
@ -148,9 +160,9 @@ encoding version.
When the RDB file is loaded back, it reads the 64 bit value, masks the final When the RDB file is loaded back, it reads the 64 bit value, masks the final
10 bits, and searches for a matching module in the modules types cache. 10 bits, and searches for a matching module in the modules types cache.
When a match is found, the method that loads the RDB file value is called When a matching one is found, the method to load the RDB file value is called
with the 10 bits encoding version as an argument, so that the module knows with the 10 bits encoding version as argument, so that the module knows
what version of the data layout to load and whether it can support multiple versions. what version of the data layout to load, if it can support multiple versions.
Now the interesting thing about all this is that, if instead the module type Now the interesting thing about all this is that, if instead the module type
cannot be resolved, since there is no loaded module having this signature, cannot be resolved, since there is no loaded module having this signature,
@ -158,7 +170,8 @@ we can convert back the 64 bit value into a 9 characters name, and print
an error to the user that includes the module type name! So that she or he an error to the user that includes the module type name! So that she or he
immediately realizes what's wrong. immediately realizes what's wrong.
## Setting and getting keys Setting and getting keys
---
After registering our new data type in the `RedisModule_OnLoad()` function, After registering our new data type in the `RedisModule_OnLoad()` function,
we also need to be able to set Redis keys having as value our native type. we also need to be able to set Redis keys having as value our native type.
@ -169,7 +182,7 @@ and to test if a given key is already associated to a value of a specific data
type. type.
The API uses the normal modules `RedisModule_OpenKey()` low level key access The API uses the normal modules `RedisModule_OpenKey()` low level key access
interface in order to deal with this. Here is an example of setting a interface in order to deal with this. This is an eaxmple of setting a
native type private data structure to a Redis key: native type private data structure to a Redis key:
RedisModuleKey *key = RedisModule_OpenKey(ctx,keyname,REDISMODULE_WRITE); RedisModuleKey *key = RedisModule_OpenKey(ctx,keyname,REDISMODULE_WRITE);
@ -178,7 +191,7 @@ native type private data structure to a Redis key:
The function `RedisModule_ModuleTypeSetValue()` is used with a key handle open The function `RedisModule_ModuleTypeSetValue()` is used with a key handle open
for writing, and gets three arguments: the key handle, the reference to the for writing, and gets three arguments: the key handle, the reference to the
native type obtained during the type registration, and finally a `void*` native type, as obtained during the type registration, and finally a `void*`
pointer that contains the private data implementing the module native type. pointer that contains the private data implementing the module native type.
Note that Redis has no clues at all about what your data contains. It will Note that Redis has no clues at all about what your data contains. It will
@ -199,7 +212,7 @@ We can also test for a key to have our native type as value:
However for the calls to do the right thing, we need to check if the key However for the calls to do the right thing, we need to check if the key
is empty, if it contains a value of the right kind, and so forth. So is empty, if it contains a value of the right kind, and so forth. So
the idiomatic code to implement a command writing to our native type the idiomatic code to implement a command writing to our native type
would be: is along these lines:
RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1], RedisModuleKey *key = RedisModule_OpenKey(ctx,argv[1],
REDISMODULE_READ|REDISMODULE_WRITE); REDISMODULE_READ|REDISMODULE_WRITE);
@ -210,7 +223,7 @@ would be:
return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE); return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE);
} }
Then, if we successfully verified the key is not of the wrong type, and Then if we successfully verified the key is not of the wrong type, and
we are going to write to it, we usually want to create a new data structure if we are going to write to it, we usually want to create a new data structure if
the key is empty, or retrieve the reference to the value associated to the the key is empty, or retrieve the reference to the value associated to the
key if there is already one: key if there is already one:
@ -225,7 +238,8 @@ key if there is already one:
} }
/* Do something with 'data'... */ /* Do something with 'data'... */
## Free method Free method
---
As already mentioned, when Redis needs to free a key holding a native type As already mentioned, when Redis needs to free a key holding a native type
value, it needs help from the module in order to release the memory. This value, it needs help from the module in order to release the memory. This
@ -240,15 +254,16 @@ assuming our data structure is composed of a single allocation:
RedisModule_Free(value); RedisModule_Free(value);
} }
However an actual implementation would call some function that performs a more However a more real world one will call some function that performs a more
complex memory deallocation, by casting the void pointer to some structure complex memory reclaiming, by casting the void pointer to some structure
and freeing all the resources composing the value. and freeing all the resources composing the value.
## RDB load and save methods RDB load and save methods
---
The RDB save and load callbacks need to create and load a The RDB saving and loading callbacks need to create (and load back) a
representation of the data type on disk. Redis offers a high level API representation of the data type on disk. Redis offers an high level API
that can automatically store the following types inside the RDB file: that can automatically store inside the RDB file the following types:
* Unsigned 64 bit integers. * Unsigned 64 bit integers.
* Signed 64 bit integers. * Signed 64 bit integers.
@ -256,12 +271,12 @@ that can automatically store the following types inside the RDB file:
* Strings. * Strings.
It is up to the module to find a viable representation using the above base It is up to the module to find a viable representation using the above base
types. However, note that while the integer and double values are stored types. However note that while the integer and double values are stored
and loaded in an architecture and *endianess* agnostic way, if you use and loaded in an architecture and *endianess* agnostic way, if you use
the raw string save API to save a structure on disk, for example, you the raw string saving API to, for example, save a structure on disk, you
have to implement those details yourself. have to care those details yourself.
This is the list of functions performing RDB save and load: This is the list of functions performing RDB saving and loading:
void RedisModule_SaveUnsigned(RedisModuleIO *io, uint64_t value); void RedisModule_SaveUnsigned(RedisModuleIO *io, uint64_t value);
uint64_t RedisModule_LoadUnsigned(RedisModuleIO *io); uint64_t RedisModule_LoadUnsigned(RedisModuleIO *io);
@ -277,7 +292,7 @@ This is the list of functions performing RDB save and load:
The functions don't require any error checking from the module, that can The functions don't require any error checking from the module, that can
always assume calls succeed. always assume calls succeed.
As an example, imagine a native type that implements an array of As an example, imagine I've a native type that implements an array of
double values, with the following structure: double values, with the following structure:
struct double_array { struct double_array {
@ -285,7 +300,7 @@ double values, with the following structure:
double *values; double *values;
}; };
The `rdb_save` method may look like the following: My `rdb_save` method may look like the following:
void DoubleArrayRDBSave(RedisModuleIO *io, void *ptr) { void DoubleArrayRDBSave(RedisModuleIO *io, void *ptr) {
struct dobule_array *da = ptr; struct dobule_array *da = ptr;
@ -294,8 +309,8 @@ The `rdb_save` method may look like the following:
RedisModule_SaveDouble(io,da->values[j]); RedisModule_SaveDouble(io,da->values[j]);
} }
We stored the number of elements followed by each double What we did was to store the number of elements followed by each double
value. So when later when loading the structure in the `rdb_load` value. So when later we'll have to load the structure in the `rdb_load`
method we'll do something like this: method we'll do something like this:
void *DoubleArrayRDBLoad(RedisModuleIO *io, int encver) { void *DoubleArrayRDBLoad(RedisModuleIO *io, int encver) {
@ -314,14 +329,14 @@ method we'll do something like this:
return da; return da;
} }
The load callback just reconstructed the data structure from the data The load callback just reconstruct back the data structure from the data
store in the RDB file. we stored in the RDB file.
Note that while there is no error handling on the API that writes and reads Note that while there is no error handling on the API that writes and reads
from disk, the load callback may still return NULL on errors in case what from disk, still the load callback can return NULL on errors in case what
it reads does not look correct. Redis will just panic in that case. it reads does not look correct. Redis will just panic in that case.
AOF rewrite AOF rewriting
--- ---
void RedisModule_EmitAOF(RedisModuleIO *io, const char *cmdname, const char *fmt, ...); void RedisModule_EmitAOF(RedisModuleIO *io, const char *cmdname, const char *fmt, ...);
@ -335,28 +350,30 @@ Allocating memory
--- ---
Modules data types should try to use `RedisModule_Alloc()` functions family Modules data types should try to use `RedisModule_Alloc()` functions family
in order to allocate, reallocate and release heap memory used to implement the native data structures in order to allocate, reallocate and release heap memory used to implement the native data structures (see the other Redis Modules documentation for detailed information).
(see the other Redis Modules documentation for detailed information).
There are more advantages than simple accounting for the memory used by the module: This is not just useful in order for Redis to be able to account for the memory used by the module, but there are also more advantages:
* Redis uses the `jemalloc` allocator, that often prevents fragmentation problems that could be caused by using the libc allocator. * Redis uses the `jemalloc` allcator, that often prevents fragmentation problems that could be caused by using the libc allocator.
* When loading strings from the RDB file, the native types API can return strings allocated directly with `RedisModule_Alloc()`, * When loading strings from the RDB file, the native types API is able to return strings allocated directly with `RedisModule_Alloc()`, so that the module can directly link this memory into the data structure representation, avoiding an useless copy of the data.
enabling the module to directly link this memory into the data structure representation, avoiding a useless copy of the data.
Even when using external libraries that implement data structures, the Even if you are using external libraries implementing your data structures, the
allocation functions provided by the module API are perfectly compatible with allocation functions provided by the module API is exactly compatible with
`malloc()`, `realloc()`, `free()` and `strdup()`, so converting the libraries `malloc()`, `realloc()`, `free()` and `strdup()`, so converting the libraries
in order to use these functions should be trivial. in order to use these functions should be trivial.
In the case of a 3rd party external library that uses libc `malloc()`, a convenient In case you have an external library that uses libc `malloc()`, and you want
approach would use simple macros in order to replace the libc calls with the Redis API calls. Something like this could work: to avoid replacing manually all the calls with the Redis Modules API calls,
an approach could be to use simple macros in order to replace the libc calls
with the Redis API calls. Something like this could work:
#define malloc RedisModule_Alloc #define malloc RedisModule_Alloc
#define realloc RedisModule_Realloc #define realloc RedisModule_Realloc
#define free RedisModule_Free #define free RedisModule_Free
#define strdup RedisModule_Strdup #define strdup RedisModule_Strdup
Note that **mixing libc calls with Redis API calls will result in trouble and crashes**, so when replacing calls using However take in mind that mixing libc calls with Redis API calls will result
macros, make sure that all the calls are correctly replaced, and that the code with the substituted calls will never, into troubles and crashes, so if you replace calls using macros, you need to
for example, attempt to call `RedisModule_Free()` with a pointer allocated using libc `malloc()`. make sure that all the calls are correctly replaced, and that the code with
the substituted calls will never, for example, attempt to call
`RedisModule_Free()` with a pointer allocated using libc `malloc()`.

Loading…
Cancel
Save