A Web Admin Console for Redis, Part One
Redis seems to grow more popular — and more important — every year. I first heard about it five or so years ago, but it's
been around quite a bit longer than that. As a cache it works admirably well: its memory usage scales pretty linearly with its contents,
and response times are hard to match. It is just a cache, though — it doesn't come with much in the way of an admin interface.
If you install it locally, you get the redis-cli command-line tool as part of the bundle which covers the basics of routine maintenance tasks.
Still, searching for keys is a big hassle on the command-line. I often find myself needing
to scan through a Redis instance for keys that match a specific pattern and modify or remove them on a case-by-case basis. Although I can,
and have, scripted this in Python, there are occasions where I'd like to browse through them. It's come up often enough that I finally
broke down and put together a simple web-based Redis browser.
About 20 or so years ago, I would have made this a pure desktop app, but web UI's are too useful to ignore anymore. I can't do it completely in Javascript, though, because even running locally Javascript can't connect to a Redis instance. Instead, I put together a basic web server in Python (because Dear God, anything but Node) that handles the connection to Redis and a web UI that interacts with it.
Writing a basic server-side web app in Python is not too difficult.
Modern Python installations include a simple web server that you can easily customize by extending SimpleHTTPRequestHandler
and overriding do_verb (where verb is one of GET, POST, DELETE, etc.)
to customize behavior. What I want here is a server that opens a persistent connection to a Redis instance and accepts a scan
request as a GET query. Since it's supposed to be a single-user "server" I don't need to worry about memory management or
re-entrancy much and can just keep the current query in global state rather than managing it in a session as I technically should.
I'll start with a simple server that accepts a Redis host as a command-line parameter and just looks up the value of a key in the redis cache when a properly-formed request is supplied.
import sys
import json
import redis
from http.server import HTTPServer
from http.server import SimpleHTTPRequestHandler
redisConn = None
class RedisHTTPRequestHandler(SimpleHTTPRequestHandler):
def parseQuery(self, path):
if path.find('?') > -1:
query = path.split('?')[1]
return {key: value for [key, value] in map(lambda x: x.split('='), query.split('&'))}
else:
return {}
def ensure_required_params(self, params, requiredKeys):
missingKeys = set(requiredKeys).difference(set(params.keys()))
if len(missingKeys) > 0:
self.send_error(400, 'Missing required parameters: %s' % (','.join(missingKeys)))
return False
else:
return True
def send_text(self, obj):
jsonResponse = json.dumps(obj)
self.send_response(200, 'OK')
self.send_header('Content-Type', 'application/json')
self.send_header('Content-Length', len(jsonResponse))
self.end_headers()
self.wfile.write(bytes(jsonResponse, 'UTF-8'))
def do_GET(self):
if self.path.startswith('/get'):
params = self.parseQuery(self.path)
if self.ensure_required_params(params, ['key']):
val = redisConn.get(params['key'])
self.send_text(val)
else:
super().do_GET()
if __name__ == '__main__':
if len(sys.argv) < 2:
print('Usage: %s [redis host name]' % sys.argv[0])
sys.exit()
redisConn = redis.Redis(sys.argv[1], decode_responses=True)
server_address = ('', 8000)
httpd = HTTPServer(server_address, RedisHTTPRequestHandler)
httpd.serve_forever()
Listing 1: Simple Server to query Redis instance
It's a bit long but should be easy enough to follow: if a GET request is received with the path get, it's expected to
contain at least a key parameter. If so, it looks up the key and returns the value. Otherwise, it lets the parent class
default respond to the GET, whose behavior is to return a response from the local file system (with minimal security checks,
so don't run this on a remote server!)
Ok, so far so good, now we need an interface. The default page will be pretty spartan and just prompt the user to input
a key. When the get button is pressed, the value of the key will be returned.
<html>
<head>
<title>Redis Maintenance Tool</title>
<style>
#error {
color: red;
}
</style>
<script>
function issueGetRequest(path, onSuccess) {
document.getElementById("error").innerHTML = "";
var req = new XMLHttpRequest();
req.open("GET", path);
req.onreadystatechange = function(evt) {
var req = evt.target;
if (req.readyState == 4) {
if (req.status == 200) {
onSuccess(req.responseText);
} else {
document.getElementById("error").innerHTML = req.statusText;
}
}
}
req.send();
}
function get() {
var key = document.getElementById("key").value;
issueGetRequest('/get?key=' + key, function(txt) {
document.getElementById("target").innerHTML = txt;
});
}
</script>
</head>
<body>
Key: <input id="key" /><br/>
<button onclick="get()">Scan</button>
<div id="target"></table>
<div id="error"></div>
</body>
</html>
Listing 2: Landing page
Interesting, but not very useful... I can do the exact same thing from the command line without the UI after all. What I want to be able
to do here is put in a key pattern and get back all of the matches. The redis scan command is designed for exactly
this, but it's a little bit trickier to use than you might think if you're not familiar with it. If you issue the command scan 0 match cust_* count 100, for instance,
you would probably expect to get back the first 100 keys in the cache that match the pattern "cust_*". That's not how scan works, though:
instead, it queries the "first" 100 keys and returns the ones that match the pattern, if any. So you might mistakenly believe, if you
get back an empty response from scan, that nothing in the cache matches the pattern, but if you keep querying, you may
eventually find something.
So what I actually want is a query that keeps running scans until it finds a set number of matches and returns them
— and to be useful, I want to be able to save my state and allow paging back and forth through the results. This is a bit tricky
because it requires some level of state maintenance. I could do this by scanning the entire keyspace and storing the results, but that's
a bit wasteful if I'm only paging through a few hundred at a time. Instead what I'll do is to keep track of the cursor value of the result
of the previous page and the current page, along with where in the Redis result the page demarcation landed.
Every time you issue a scan to Redis, you get back a (possibly empty) list of matches along with a "next page" cursor which
you're supposed to pass on the next scan to query the next n possible keys. It's designed as forward-only, though
— if you want to support paging backwards, you have to store all the previous cursors in order to potentially page back over the
responses. Note that I'm not designing for multiple active scans although Redis itself does actually allow for this.
So, with each scan, I can encounter one of three cases:
- the
scanresults include exactly as many elements as I want on a page - the
scanresults include fewer elements than I want on a page - the
scanresults include more elements than I want on a page
I can accomplish this with a new route in my Python server named scan that accepts three parameters: pattern,
size and page. I'll maintain (global) state about the currently active query and if its pattern or size change,
I'll reset the whole thing.
redisConn = None
activeQuery = {
'pattern': None,
'size': 0,
'pages': [
{'cursor': '0', 'position': 0}
]
}
DEFAULT_SCAN_SIZE=100
class RedisHTTPRequestHandler(SimpleHTTPRequestHandler):
...
def do_GET(self):
...
elif self.path.startswith('/scan'):
params = self.parseQuery(self.path)
if self.ensure_required_params(params, ['pattern', 'size', 'page']):
pattern = params['pattern']
size = int(params['size'])
page = int(params['page'])
if activeQuery['pattern'] != pattern or activeQuery['size'] != size:
# New query, reset the whole thing
activeQuery['pattern'] = pattern
activeQuery['size'] = size
activeQuery['pages'] = [{'cursor': '0', 'position': 0}]
if page < len(activeQuery['pages']):
cursor = activeQuery['pages'][page]['cursor']
position = activeQuery['pages'][page]['position']
# The keys returned to the user are collected here
pageKeys = []
# True unless Redis has no more results
more = True
while size > 0:
(nextCursor, keys) = redisConn.scan(cursor, pattern, DEFAULT_SCAN_SIZE)
remainingKeys = keys[position:]
if nextCursor == 0:
# last scan result from Redis
pageKeys += remainingKeys
more = False
break
else:
if len(remainingKeys) > size:
pageKeys += remainingKeys[:size]
position += size
else:
pageKeys += remainingKeys
cursor = nextCursor
position = 0
size -= len(remainingKeys)
if page == len(activeQuery['pages']) - 1:
activeQuery['pages'].append({'cursor': cursor, 'position': position})
Listing 3: Retrieve one page worth of results in (possibly multiple) redis scans
All that's left to do now is to retrieve the key's values themselves. To speed things up, I'll use a pipe to batch up requests and responses:
pipe = redisConn.pipeline()
for key in pageKeys:
pipe.get(key)
values = pipe.execute()
self.send_text({'page': list(zip(pageKeys, values)), 'more': more})
Listing 4: Retrieve keys, too
Back to the UI side, I have to update the JavaScript to present the results and a navigation control.
function scan(page) {
var query = document.getElementById("query").value;
var size = document.getElementById("size").value;
issueGetRequest('/scan?pattern=' + query + '&size=' + size + '&page=' + page, function(txt) {
var result = JSON.parse(txt);
var tbl = document.getElementById("tbl");
tblHtml = '<tr><th>Key</th><th>value</th></tr>';
for (var i = 0; i < result.page.length; i++) {
tblHtml += '<tr><td>' + result.page[i][0] + '</td></tr>'
}
tblHtml += '<tr><td>' +
(page > 0 ? '<a href="#" onclick="scan(' + (page - 1) + ')">prev</a> ' : '') +
'</td><td>' +
(result.more ? '<a href="#" onclick="scan(' + (page + 1) + ')">next</a>' : '') +
'</td></tr>';
tbl.innerHTML = '<table>' + tblHtml + '</table>';
if (page == 0 && !result.more) {
document.getElementById("error").innerHTML = "No matches found";
}
});
}
...
Query: <input id="query" /><br/>
Size: <input id="size" /><br/>
<button onclick="scan(0)">Scan</button>
<div id="tbl"></div>
Listing 5: Display query results in a table
This puts a bit more load on the Redis server than necessary — if the page size is much less than the scan size, I'll end up re-requesting the same results from Redis over and over. I do want the scan size to be pretty big, too: if I query a pattern that doesn't match too many keys, I'll end up scanning the whole keyspace to satisfy it: if my scan size is small relative to my key count, the query will end up taking a long time. A good compromise is to cache the most recent scan result from Redis, but otherwise go back to the server for the next (or previous) page.
I don't worry too much about what happens if the keys change while I'm running the scan — there's really not all that much I can do about that, anyway. Most times when I use this, I'm looking for a relatively small handful of keys that match a specific pattern, anyway. The results are spartan, but useful. Still, once I find what I'm looking for, I usually want to delete it or edit it. I'll tackle modifications in my next post, stay tuned!
Add a comment:
Completely off-topic or spam comments will be removed at the discretion of the moderator.
You may preserve formatting (e.g. a code sample) by indenting with four spaces preceding the formatted line(s)
