redis-oxide

redis-oxide copied to clipboard

#+AUTHOR: David Briggs #+STARTUP: SHOWALL

• Redis Oxide

[[https://github.com/dpbriggs/redis-oxide/actions][https://github.com/dpbriggs/redis-oxide/workflows/Redis%20Oxide%20Pipeline/badge.svg]]

A multi-threaded implementation of redis written in rust 🦀.

This project is intended to be a drop-in replacement for redis. It's under construction at the moment.

[[https://i.imgur.com/8Zb0gu5.png][https://i.imgur.com/8Zb0gu5.png]]

** Design

=redis-oxide= is a black-box multi-threaded re-implementation of redis, backed by [[https://tokio.rs/][tokio]]. It features data-structure key-space/lock granularity, written entirely in safe rust. It's currently protocol compatible with redis, so you should be able to test it out with your favourite tools.

The multi-threaded nature has advantages and disadvantages. On one hand, =KEYS *= isn't particularly crippling for the server as it'll just keep a thread busy. On the other hand, there's some lock-juggling overhead, especially for writes, which messes with tokio.

** Building / Running

There's currently no official release for the project. You can compile and install it yourself with the following command:

: cargo install --git https://github.com/dpbriggs/redis-oxide

Note: This project requires the rust nightly. You can use [[https://rustup.rs/][rustup]] to install it.

Once it compiles you should be able to run it with =~ redis-oxide=.

If you wish to download and run it yourself, you can do the following

#+begin_example ~ git clone https://github.com/dpbriggs/redis-oxide ~ cd redis-oxide ~ cargo run #+end_example

Then use your favorite redis client. Eg. =redis-cli=:

#+begin_example ~ redis-cli 127.0.0.1:6379> set foo bar OK 127.0.0.1:6379> get foo "bar" #+end_example

Or using the redis library for python:

#+begin_src python import redis from pprint import pprint

r = redis.Redis() r.set('foobar', 'foobar') pprint(r.get('foobar'))

for i in range(100): r.rpush('list', i)

list_res = r.lrange('list', 0, -1)

pprint(list_res[0:3]) pprint(sum(map(int, list_res)))

total = 0 for i in range(100): total += int(r.lpop('list')) pprint(total) #+end_src

Which will print:

#+begin_src python b'foobar' [b'0', b'1', b'2'] 4950 4950 #+end_src

** Things left to do

*** Basic Datastructures

• [X] Keys
• [X] Sets
• [X] Lists
• [X] Hashes
• [ ] HyperLogLog
• [ ] Geo
• [-] Sorted Sets
  • [X] Basic Functionality
  • [ ] Still need some operations
• [ ] Strings

We should solidify the above before working on the more complex bits, but contributions are welcome :)

*** Redis Compatibility

• [X] Resp / server
• [ ] Database compatibility
  • [ ] Unsure if this is a good thing -- may be better to port existing dumps.
• [ ] Blocking / Concurrent Ops (ttl/save-on-x-ops)
• [ ] CLI / config compatibility
• [ ] Authentication

** Contribution Guide

Conduct: =Have fun, please don't be a jerk.=

Contact: Make an issue or PR against this repo, or send an email to [email protected]=. If you know of a better forum, please suggest it!

NOTE: DO NOT USE THE REDIS SOURCE CODE IN ANY WAY!

This project is under active development, so things are a little messy.

The general design of =redis-oxide= is:

• A Command (=set foo bar=) is read off the socket and passed to the translate function in =src/ops.rs=.
  • The parser generates a =RedisValue=, which is the lingua franca of =redis-oxide=.
• This gets converted to an =Ops::XYZ(XYZOps::Foobar(..))= enum object, which is consumed by the =op_interact= function.
  • A macro is used to provide automate this.
• This operation is executed against the global =State= object (using the =op_interact= function)
  • This will return an =ReturnValue= type, which is a more convenient form of =RedisValue=.
  • This =ReturnValue= is converted and sent back to the client.

Therefore, if you want to do something like implement =hashes=, you will need to:

1. Add a new struct member in =State=.
   1. You first define the type: =type KeyHash = DashMap<Key, HashMap<Key, Value>>=
   2. Then add it to State: =pub hashes: KeyHash=
2. Define a new file for your data type, =src/hashes.rs=.
   1. Keep your type definitions in =src/types.rs=!
3. Create an enum to track your commands, =op_variants! { HashOps, HGet(Key, Key), HSet(Key, Key, Value) }=
4. Implement parsing for your enum in =src/ops.rs=.
   1. You should be able to follow the existing parsing infrastructure. Should just be extra entries in =translate_array= in =src/ops.rs=.
   2. You will need to add your return type to the =ok!= macro. Just copy/paste an existing line.
   3. You should return something like =ok!(HashOps::HSet(x, y, z))=.
   4. A stretch goal is to automate parsing.
5. Implement a =async *_interact= for your type; I would follow existing implementations (eg. =src/keys.rs=).
   1. I would keep the redis docs open, and play around with the commands in the web console (or wherever) to determine behavior.
   2. Add a new match entry in the =async op_interact= function in =src/ops.rs=.
6. Test it! (follow existing testing bits; eg. =src/keys.rs=).
7. Please add the commands to the list below.
   1. If you're using emacs, just fire up the server and evaluate the babel block below (see =README.org= source)
   2. Alternatively, copy the script into a terminal and copy/paste the output below. (see raw =README.org=)

** Implemented Commands

#+BEGIN_SRC python :results output raw :format org :exports results import redis

r = redis.StrictRedis(decode_responses=True)

all_commands = r.execute_command('printcmds')

for command in all_commands: command_name, ops = command[0], command[1:] print(f'*** {command_name}\n') for op in ops: print(f'- ={op}=') print('\n') #+END_SRC

#+RESULTS: *** KeyOps

• =Set (Key, Value)=
• =MSet (RVec<(Key, Value)>)=
• =Get (Key)=
• =MGet (RVec<Key>)=
• =Del (RVec<Key>)=
• =Rename (Key, Key)=
• =RenameNx (Key, Key)=

*** ListOps

• =LIndex (Key, Index)=
• =LLen (Key)=
• =LPop (Key)=
• =LPush (Key, RVec<Value>)=
• =LPushX (Key, Value)=
• =LRange (Key, Index, Index)=
• =LSet (Key, Index, Value)=
• =LTrim (Key, Index, Index)=
• =RPop (Key)=
• =RPush (Key, RVec<Value>)=
• =RPushX (Key, Value)=
• =RPopLPush (Key, Key)=
• =BLPop (Key, UTimeout)=
• =BRPop (Key, UTimeout)=

*** HashOps

• =HGet (Key, Key)=
• =HSet (Key, Key, Value)=
• =HExists (Key, Key)=
• =HGetAll (Key)=
• =HMGet (Key, RVec<Key>)=
• =HKeys (Key)=
• =HMSet (Key, RVec<(Key, Value)>)=
• =HIncrBy (Key, Key, Count)=
• =HLen (Key)=
• =HDel (Key, RVec<Key>)=
• =HVals (Key)=
• =HStrLen (Key, Key)=
• =HSetNX (Key, Key, Value)=

*** SetOps

• =SAdd (Key, RVec<Value>)=
• =SCard (Key)=
• =SDiff (RVec<Value>)=
• =SDiffStore (Key, RVec<Value>)=
• =SInter (RVec<Value>)=
• =SInterStore (Key, RVec<Value>)=
• =SIsMember (Key, Value)=
• =SMembers (Key)=
• =SMove (Key, Key, Value)=
• =SPop (Key, Option<Count>)=
• =SRandMembers (Key, Option<Count>)=
• =SRem (Key, RVec<Value>)=
• =SUnion (RVec<Value>)=
• =SUnionStore (Key, RVec<Value>)=

*** ZSetOps

• =ZAdd (Key, RVec<(Score, Key)>)=
• =ZRem (Key, RVec<Key>)=
• =ZRange (Key, Score, Score)=
• =ZCard (Key)=
• =ZScore (Key, Key)=
• =ZPopMax (Key, Count)=
• =ZPopMin (Key, Count)=
• =ZRank (Key, Key)=

*** BloomOps

• =BInsert (Key, Value)=
• =BContains (Key, Value)=

*** StackOps

• =STPush (Key, Value)=
• =STPop (Key)=
• =STPeek (Key)=
• =STSize (Key)=

*** HyperLogLogOps

• =PfAdd (Key, RVec<Value>)=
• =PfCount (RVec<Key>)=
• =PfMerge (Key, RVec<Key>)=

*** MiscOps

• =Keys ()=
• =Exists (Vec<Key>)=
• =Pong ()=
• =FlushAll ()=
• =FlushDB ()=
• =Echo (Value)=
• =PrintCmds ()=
• =Select (Index)=
• =Script (Value)=
• =EmbeddedScript (Value, Vec<RedisValueRef>)=
• =Info ()=