RedisLookupPlugin Plugin for Graylog

Plugin to add Redis Data Adapter in read/write to graylog so you can store and retrieve key/values and lists of string values from pipelines
Support Redis authentication (with password and username/password)

Required Graylog version: 5.0 and later

Installation

Download the plugin and place the .jar file in your Graylog plugin directory. The plugin directory is the plugins/ folder relative from your graylog-server directory by default and can be configured in your graylog.conf file.

Restart graylog-server and you are done.

Development

You can improve your development experience for the web interface part of your plugin dramatically by making use of hot reloading. To do this, do the following:

• git clone https://github.com/Graylog2/graylog2-server.git
• cd graylog2-server/graylog2-web-interface
• ln -s $YOURPLUGIN plugin/
• npm install && npm start

Usage

About data types

• "Strings" lookup table will use GET/SET redis commands. Designed to set, get, expire, delete keys.
• "Streams" is designed for a "fire-and-forget" usage. Only "lookup_set_value" will work. Please note there is currently no TTL on stream, so you need to have an external process to purge your stream if you do not want it to grow indefinitely.

Usage in pipelines

• Create data adapter, cache (or not), lookup table
• Use 'lookup_set_value(lookup_table, key, value, [ttl])' to create or update key in redis
• Use 'lookup_value(lookup_table, key)' to get key value
• Use 'lookup_clear_key(lookup_table, key)' to remove key
• Use 'lookup_has_value(lookup_table, key)' to test key existence
• Use 'lookup_assign_ttl(lookup_table, key, ttl)' to change TTL of existing key
• Use 'lookup_set_string_list(lookup_table, key, value, [ttl])' to create a list named "key"
• Use 'lookup_add_string_list(lookup_table, key, value, [keep_duplicates])' to add value list to existing list
• Use 'lookup_remove_string_list(lookup_table, key, value) to remove a string from list "key"

By default single value keys will be created in Redis with the default TTL defined at data adapter creation time

Be aware that only setting TTL with 'lookup_assign_ttl' of 'lookup_set_string_list' alter TTL value in Redis ; so a list created with a TTL of 3600 will expire in 3600 seconds, even if it was updated with 'lookup_add_string_list' some seconds before expiration.

Known bugs

• Deletion via lookup_remove_string_list, lookup_clear_key or keep_duplicates=false sometimes not done.
• lookup_add_string_list : Each character of a string list is considered an item

Getting started

This project is using Maven 3 and requires Java 8 or higher.

• Clone this repository.
• Run mvn package to build a JAR file.
• Optional: Run mvn jdeb:jdeb and mvn rpm:rpm to create a DEB and RPM package respectively.
• Copy generated JAR file in target directory to your Graylog plugin directory.
• Restart the Graylog.

Plugin Release

We are using the maven release plugin:

$ mvn release:prepare
[...]
$ mvn release:perform

This sets the version numbers, creates a tag and pushes to GitHub. Travis CI will build the release artifacts and upload to GitHub automatically.