One Hat Cyber Team

Edit File: mod_redis.html

<!DOCTYPE html>
<html>
<head>
<title>ProFTPD module mod_redis</title>
</head>

<hr>
<center>
<h2>ProFTPD module <code>mod_redis</code></h2>
</center>
<hr>

The <code>mod_redis</code> module enables ProFTPD support for caching data in
<a href="https://redis.io">Redis</a> servers, using the
<a href="https://github.com/redis/hiredis">hiredis</a> client library.

<h2>Directives</h2>
<ul>
 <li><a href="#RedisEngine">RedisEngine</a>
 <li><a href="#RedisLog">RedisLog</a>
 <li><a href="#RedisLogFormatExtra">RedisLogFormatExtra</a>
 <li><a href="#RedisLogOnCommand">RedisLogOnCommand</a>
 <li><a href="#RedisLogOnEvent">RedisLogOnEvent</a>
 <li><a href="#RedisOptions">RedisOptions</a>
 <li><a href="#RedisSentinel">RedisSentinel</a>
 <li><a href="#RedisServer">RedisServer</a>
 <li><a href="#RedisTimeouts">RedisTimeouts</a>
</ul>

<hr>
<h3><a name="RedisEngine">RedisEngine</a></h3>
Syntax: RedisEngine on|off 
Default: RedisEngine off 
Context: server config, <code>&lt;VirtualHost&gt;</code>, <code>&lt;Global&gt;</code> 
Module: mod_redis 
Compatibility: 1.3.6rc5 and later

The <code>RedisEngine</code> directive enables or disables the
<code>mod_redis</code> module, and thus the configuration of Redis support for
the <code>proftpd</code> daemon.

<hr>
<h3><a name="RedisLog">RedisLog</a></h3>
Syntax: RedisLog path|"none" 
Default: None 
Context: server config, <code>&lt;VirtualHost&gt;</code>, <code>&lt;Global&gt;</code> 
Module: mod_redis 
Compatibility: 1.3.6rc5 and later

The <code>RedisLog</code> directive is used to specify a log file for
<code>mod_redis</code>'s reporting on a per-server basis. The
file parameter given must be the full path to the file to use for
logging.

Note that this path must not be to a world-writable directory and,
unless <code>AllowLogSymlinks</code> is explicitly set to on
(generally a bad idea), the path must not be a symbolic link.

<hr>
<h3><a name="RedisLogFormatExtra">RedisLogFormatExtra</a></h3>
Syntax: RedisLogFormatExtra format-name json-object 
Default: None 
Context: server config, <code>&lt;VirtualHost&gt;</code>, <code>&lt;Global&gt;</code>, <code>&lt;Anonymous&gt;</code>, <code>&lt;Directory&gt;</code> 
Module: mod_redis 
Compatibility: 1.3.8rc1 and later

The <code>RedisLogFormatExtra</code> directive configures "extra" data to be
added to the JSON logging, performed by <code>RedisLogOnCommand</code> and
<code>RedisLogOnEvent</code>.

The format-name parameter indicates that the json-object extra
data should be added to that payload. This allows adding of custom JSON object
key/value pairs.

Example:
<pre>
 LogFormat file-transfers "%h %l %u %t \"%r\" %s %b"

RedisLogOnCommand RETR,STOR file-transfers %m:ftpxfer:%u
 RedisLogFormatExtra file-transfers {"custom_property1":"%m","custom_property2":"custom_value"}
</pre>

<hr>
<h3><a name="RedisLogOnCommand">RedisLogOnCommand</a></h3>
Syntax: RedisLogOnCommand "none"|commands format-name [key] 
Default: None 
Context: server config, <code>&lt;VirtualHost&gt;</code>, <code>&lt;Global&gt;</code>, <code>&lt;Anonymous&gt;</code>, <code>&lt;Directory&gt;</code> 
Module: mod_redis 
Compatibility: 1.3.6rc5 and later

The <code>RedisLogOnCommand</code> directive configures the use of Redis for
logging. Whenever one of the comma-separated list of commands
occurs, <code>mod_redis</code> will compose a JSON object, using the
<a href="mod_log.html#LogFormat"><code>LogFormat</code></a> named by
format-name as a template for the fields to include in the
JSON object. The JSON object of that event will then be appended to a list
stored in Redis, using format-name as the default key name. Multiple
<code>RedisLogOnCommand</code> directives can be used, for different log formats
for different events.

The optional key parameter, if present, specifies the value to use
as the Redis key. This key parameter supports all of the
<code>LogFormat</code> variables, thus you can use e.g.:
<pre>
 RedisLogOnCommand RETR,STOR xferlog %m:ftpxfer:%u
</pre>

More on the use of Redis logging, including a table showing how
<code>LogFormat</code> variables are mapped to JSON object keys can be found
<a href="#Logging">here</a>.

Example:
<pre>
 LogFormat file-transfers "%h %l %u %t \"%r\" %s %b"

# Only log to Redis for uploads in this directory
  &lt;Directory /path/to/inbox&gt;
    RedisLogOnCommand APPE,STOR,STOU file-transfers
  &lt;/Directory&gt;

# Only log to Redis for downloads from this directory
  &lt;Directory /path/to/pub&gt;
    RedisLogOnCommand RETR file-transfers
  &lt;/Directory&gt;

# ...but prevent Redis logging in this subdirectory
 &lt;Directory /path/to/pub/subdir&gt;
 RedisLogOnCommand none
 &lt;/Directory&gt;
</pre>

Note that <code>RedisLogOnCommand</code> does not currently
support the logging classes that the <code>ExtendedLog</code> directive
supports.

<hr>
<h3><a name="RedisLogOnEvent">RedisLogOnEvent</a></h3>
Syntax: RedisLogOnEvent "none"|events format-name [key] 
Default: None 
Context: server config, <code>&lt;VirtualHost&gt;</code>, <code>&lt;Global&gt;</code>, <code>&lt;Anonymous&gt;</code>, <code>&lt;Directory&gt;</code> 
Module: mod_redis 
Compatibility: 1.3.7rc1 and later

The <code>RedisLogOnEvent</code> directive configures the use of Redis for
logging. Whenever one of the comma-separated list of events
occurs, <code>mod_redis</code> will compose a JSON object, using the
<a href="mod_log.html#LogFormat"><code>LogFormat</code></a> named by
format-name as a template for the fields to include in the
JSON object. The JSON object of that event will then be appended to a list
stored in Redis, using format-name as the default key name. Multiple
<code>RedisLogOnEvent</code> directives can be used, for different log formats
for different events.

More on the use of Redis logging, including a table showing how
<code>LogFormat</code> variables are mapped to JSON object keys can be found
<a href="#Logging">here</a>.

Example:
<pre>
 LogFormat sessions "%{iso8601} %a"
 RedisLogOnEvent CONNECT,DISCONNECT sessions
</pre>

In addition to specific FTP commands, the events list can specify
"ALL", for logging on all commands. Or it can include the
"CONNECT" and "DISCONNECT" events, which can be useful for logging the
start and end times of a session. Note that
<code>RedisLogOnEvent</code> does support the logging classes
that the <code>ExtendedLog</code> directive supports.

<hr>
<h3><a name="RedisOptions">RedisOptions</a></h3>
Syntax: RedisOptions opt1 ... 
Default: None 
Context: server config, <code>&lt;VirtualHost&gt;</code>, <code>&lt;Global&gt;</code> 
Module: mod_redis 
Compatibility: 1.3.7rc1 and later

The <code>RedisOptions</code> directive is used to configure various optional
behavior of <code>mod_redis</code>.

Example:
<pre>
 RedisOptions NoReconnect
</pre>

The currently implemented options are:
<ul>
 <li><code>NoReconnect</code> 
 
 If the connection to Redis breaks unexpectedly, <code>mod_redis</code>
 will attempt to reconnect automatically. Use this option to
 disable the automatic reconnection.
 </li>
</ul>

<hr>
<h3><a name="RedisSentinel">RedisSentinel</a></h3>
Syntax: RedisSentinel host1 ... ["master" name] [ssl:true] [ssl-ca:&lt;path&gt;] [ssl-cert:&lt;path&gt;] [ssl-key:&lt;path&gt;] 
Default: None 
Context: server config, <code>&lt;VirtualHost&gt;</code>, <code>&lt;Global&gt;</code> 
Module: mod_redis 
Compatibility: 1.3.7rc1 and later

The <code>RedisSentinel</code> directive is used to configure a list of IP
addresses/ports of Redis Sentinels that the <code>mod_redis</code> module is
to use, for discovering the location of named master database. For
example:
<pre>
 # Configure two Sentinels; the first master discovered will be used
 RedisSentinel 1.2.3.4:26379 5.6.7.8:36379

# Configure two Sentinels on the default Sentinel port, and
  # look for the location of the master named "proftpd".
  RedisSentinel 1.2.3.4 5.6.7.8 master proftpd

# Configure three Sentinels including an IPv6 address, with explicit ports,
 # and look for the location of the "proftpd" master.
 RedisSentinel redis1.example.com:26379 1.2.3.4:26379 [::1]:26379 master proftpd
</pre>

The <code>RedisSentinel</code> directive can be used instead of the
<code>RedisServer</code> directive, for discovering the Redis server to use.
However, if your Redis server requires authentication, or
supports multiple databases, then you will need to use
<code>RedisServer</code> as well:
<pre>
 # Use this to configure our password and database
 RedisServer 127.0.0.1:6379 redis redisr0cks 2

# And use Sentinels to discover the true address/port
 RedisSentinel 1.2.3.4 5.6.7.8 9.10.11.12
</pre>

In ProFTPD 1.3.8rc1 and later, it is possible to configure SSL/TLS parameters
when connecting to Redis Sentinel. Most of the time, all that is needed for
the SSL/TLS session is the CA (Certificate Authority) to use, for verifying the
certificate presented by the Redis sentine. Thus:
<pre>
 RedisSentinel ... ssl:true ssl-ca:/path/to/cacert.pem
</pre>
If your Redis sentinel is configured to require SSL/TLS mutual authentication
(also called "client auth"), you may need the <code>ssl-cert:</code> and
<code>ssl-key:</code> parameters as well:
<pre>
 RedisSentinel ... ssl:true \
 ssl-ca:/path/to/cacert.pem \
 ssl-cert:/path/to/client-cert.pem \
 ssl-key:/path/to/client-key.pem
</pre>

<hr>
<h3><a name="RedisServer">RedisServer</a></h3>
Syntax: RedisServer server [username] [password] [db-index][ssl:true] [ssl-ca:&lt;path&gt;] [ssl-cert:&lt;path&gt;] [ssl-key:&lt;path&gt;] 
Default: None 
Context: server config, <code>&lt;VirtualHost&gt;</code>, <code>&lt;Global&gt;</code> 
Module: mod_redis 
Compatibility: 1.3.6rc5 and later

The <code>RedisServer</code> directive is used to configure the IP address/port
of the Redis server that the <code>mod_redis</code> module is to use. For
example:
<pre>
 RedisServer 1.2.3.4:6379
</pre>
or, for an IPv6 address, make sure the IPv6 address is enclosed in square
brackets:
<pre>
 RedisServer [::ffff:1.2.3.4]:6379
</pre>

Alternatively, you can configure a Unix domain socket path using e.g.:
<pre>
 RedisServer /var/run/redis.sock
</pre>

Optional username and password parameters can be provided,
for Redis servers which are password protected.

As of ProFTPD 1.3.7rc1, the optional db-index parameter can be
provided, for selecting the server-side Redis database by index:
<pre>
 RedisServer 1.2.3.4:6379 user passwd 2
</pre>
Connecting to a Redis server without password authentication but still
selecting the database would be done using the empty string for the username
and password:
<pre>
 RedisServer 1.2.3.4:6379 "" "" 2
</pre>

In ProFTPD 1.3.8rc1 and later, it is possible to configure SSL/TLS parameters
when connecting to Redis. Most of the time, all that is needed for the SSL/TLS
session is the CA (Certificate Authority) to use, for verifying the certificate
presented by the Redis server. Thus:
<pre>
 RedisServer ... ssl:true ssl-ca:/path/to/cacert.pem
</pre>
If your Redis server is configured to require SSL/TLS mutual authentication
(also called "client auth"), you may need the <code>ssl-cert:</code> and
<code>ssl-key:</code> parameters as well:
<pre>
 RedisServer ... ssl:true \
 ssl-ca:/path/to/cacert.pem \
 ssl-cert:/path/to/client-cert.pem \
 ssl-key:/path/to/client-key.pem
</pre>

<hr>
<h3><a name="RedisTimeouts">RedisTimeouts</a></h3>
Syntax: RedisTimeouts connect-millis io-millis 
Default: RedisTimeouts 500 500 
Context: server config, <code>&lt;VirtualHost&gt;</code>, <code>&lt;Global&gt;</code> 
Module: mod_redis 
Compatibility: 1.3.6rc5 and later

The <code>RedisTimeouts</code> directive configures timeouts to be used
when communicating with the Redis server. The connect-millis
parameter specifies a timeout, in milliseconds, to use when first
connecting to the Redis server. The io-millis parameter specifies
a timeout, in milliseconds, to use both when sending commands to Redis, and
when reading responses.

The default is 500 milliseconds for both timeouts:
<pre>
 RedisTimeouts 500 500
</pre>

<hr>
<h2><a name="Installation">Installation</a></h2>
The <code>mod_redis</code> module is distributed with ProFTPD. To enable
support and use of the Redis protocol in your <code>proftpd</code> daemon,
use the <code>--enable-redis</code> configure option:
<pre>
 $ ./configure --enable-redis ...
 $ make
 $ make install
</pre>
This option causes the <code>mod_redis</code> module to be compiled into
<code>proftpd</code>.

You may also need to tell <code>configure</code> how to find the
<code>hiredis</code> header and library files:
<pre>
 $ ./configure --enable-redis \
 --with-includes=/path/to/hiredis/include \
 --with-libraries=/path/to/hiredis/lib
</pre>

<hr>
<h2><a name="Usage">Usage</a></h2>

Configuring Redis for use by other modules, e.g. <code>mod_ban</code>
or <code>mod_tls_redis</code>:
<pre>
 &lt;IfModule mod_redis.c&gt;
 RedisEngine on
 RedisLog /var/log/ftpd/redis.log
 RedisServer 127.0.0.1:6379
 &lt;/IfModule&gt;
</pre>

This example shows the use of Redis logging for all commands:
<pre>
 &lt;IfModule mod_redis.c&gt;
 RedisEngine on
 RedisLog /var/log/ftpd/redis.log
 RedisServer 127.0.0.1:6379

LogFormat redis "%h %l %u %t \"%r\" %s %b"
 RedisLogOnCommand ALL redis
 &lt;/IfModule&gt;
</pre>

<a name="Logging"></a>
Redis Logging 
When using Redis logging, the following table shows how <code>mod_redis</code>
converts a <code>LogFormat</code> variable into the key names in the JSON
logging objects:
<table border=1 summary="Redis LogFormat Variables">
 <tr>
 <td><code>LogFormat</code> Variable</td>
 <td>Key</td>
 </tr>

<tr>
 <td>&nbsp;<code>%A</code>&nbsp;</td>
 <td>anon_password</td>
 </tr>

<tr>
 <td>&nbsp;<code>%a</code>&nbsp;</td>
 <td>remote_ip</td>
 </tr>

<tr>
 <td>&nbsp;<code>%b</code>&nbsp;</td>
 <td>bytes_sent</td>
 </tr>

<tr>
 <td>&nbsp;<code>%c</code>&nbsp;</td>
 <td>connection_class</td>
 </tr>

<tr>
 <td>&nbsp;<code>%E</code>&nbsp;</td>
 <td>session_end_reason</td>
 </tr>

<tr>
 <td>&nbsp;<code>%{epoch}</code>&nbsp;</td>
 <td>Unix timestamp, in seconds since Jan 1 1970.</td>
 </tr>

<tr>
 <td>&nbsp;<code>%F</code>&nbsp;</td>
 <td>transfer_path</td>
 </tr>

<tr>
 <td>&nbsp;<code>%{file-modified}</code>&nbsp;</td>
 <td>file_modified</td>
 </tr>

<tr>
 <td>&nbsp;<code>%g</code>&nbsp;</td>
 <td>group</td>
 </tr>

<tr>
 <td>&nbsp;<code>%H</code>&nbsp;</td>
 <td>server_ip</td>
 </tr>

<tr>
 <td>&nbsp;<code>%h</code>&nbsp;</td>
 <td>remote_dns</td>
 </tr>

<tr>
 <td>&nbsp;<code>%I</code>&nbsp;</td>
 <td>session_bytes_rcvd</td>
 </tr>

<tr>
 <td>&nbsp;<code>%{iso8601}</code>&nbsp;</td>
 <td>timestamp</td>
 </tr>

<tr>
 <td>&nbsp;<code>%J</code>&nbsp;</td>
 <td>command_params</td>
 </tr>

<tr>
 <td>&nbsp;<code>%L</code>&nbsp;</td>
 <td>local_ip</td>
 </tr>

<tr>
 <td>&nbsp;<code>%l</code>&nbsp;</td>
 <td>identd_user</td>
 </tr>

<tr>
 <td>&nbsp;<code>%m</code>&nbsp;</td>
 <td>command</td>
 </tr>

<tr>
 <td>&nbsp;<code>%{microsecs}</code>&nbsp;</td>
 <td>microsecs</td>
 </tr>

<tr>
 <td>&nbsp;<code>%{millisecs}</code>&nbsp;</td>
 <td>millisecs</td>
 </tr>

<tr>
 <td>&nbsp;<code>%O</code>&nbsp;</td>
 <td>session_bytes_sent</td>
 </tr>

<tr>
 <td>&nbsp;<code>%p</code>&nbsp;</td>
 <td>local_port</td>
 </tr>

<tr>
 <td>&nbsp;<code>%{protocol}</code>&nbsp;</td>
 <td>protocol</td>
 </tr>

<tr>
 <td>&nbsp;<code>%r</code>&nbsp;</td>
 <td>raw_command</td>
 </tr>

<tr>
 <td>&nbsp;<code>%S</code>&nbsp;</td>
 <td>response_msg</td>
 </tr>

<tr>
 <td>&nbsp;<code>%s</code>&nbsp;</td>
 <td>response_code</td>
 </tr>

<tr>
 <td>&nbsp;<code>%T</code>&nbsp;</td>
 <td>transfer_secs</td>
 </tr>

<tr>
 <td>&nbsp;<code>%t</code>&nbsp;</td>
 <td>local_time</td>
 </tr>

<tr>
 <td>&nbsp;<code>%{transfer-failure}</code>&nbsp;</td>
 <td>transfer_failure</td>
 </tr>

<tr>
 <td>&nbsp;<code>%{transfer-status}</code>&nbsp;</td>
 <td>transfer_status</td>
 </tr>

<tr>
 <td>&nbsp;<code>%U</code>&nbsp;</td>
 <td>original_user</td>
 </tr>

<tr>
 <td>&nbsp;<code>%V</code>&nbsp;</td>
 <td>server_dns</td>
 </tr>

<tr>
 <td>&nbsp;<code>%v</code>&nbsp;</td>
 <td>server_name</td>
 </tr>

<tr>
 <td>&nbsp;<code>%{version}</code>&nbsp;</td>
 <td>server_version</td>
 </tr>

<tr>
 <td>&nbsp;<code>%w</code>&nbsp;</td>
 <td>rename_from</td>
 </tr>
</table>

In addition to the standard <code>LogFormat</code> variables, the
<code>mod_redis</code> module also adds a "connecting" key for events
generated when a client first connects, and a "disconnecting" key for events
generated when a client disconnects. These keys can be used for determining
the start/finish events for a given session.

Here is an example of the JSON-formatted records generated, using the above
example configuration:
<pre>
 {"connecting":true,"timestamp":"2013-08-21 23:08:22,171"}
 {"command":"USER","timestamp":"2013-08-21 23:08:22,278"}
 {"user":"proftpd","command":"PASS","timestamp":"2013-08-21 23:08:22,305"}
 {"user":"proftpd","command":"PASV","timestamp":"2013-08-21 23:08:22,317"}
 {"user":"proftpd","command":"LIST","bytes_sent":432,"transfer_secs":4.211,"timestamp":"2013-08-21 23:08:22,329"}
 {"user":"proftpd","command":"QUIT","timestamp":"2013-08-21 23:08:22,336"}
 {"disconnecting":true,"user":"proftpd","timestamp":"2013-08-21 23:08:22,348"}
</pre>
Notice that for a given event, not all of the <code>LogFormat</code>
variables are filled in. If <code>mod_redis</code> determines that a given
<code>LogFormat</code> variable has no value for the logged event, it will
simply omit that variable from the JSON object.

Another thing to notice is that the generated JSON object ignores the textual
delimiters configured by the <code>LogFormat</code> directive; all that
matters are the <code>LogFormat</code> variables which appear in the directive.

<a name="FAQ"></a>
Frequently Asked Questions

<a name="SQLLog"></a>
Question: How can I convert this SQL logging into the
equivalent Redis logging?
<pre>
 SQLNamedQuery upload FREEFORM "INSERT INTO ftplogs ('userid', 'server_ip', 'transfer_date', 'operation', 'protocol', 'client_ip', 'transfer_time', 'bytes_transfer', 'file_hash_type', 'file_hash', 'file_path', 'transfer_status') VALUES ('%u', '%H', NOW(), '%r', '%{protocol}', '%a', '%T', '%b', '%{note:mod_digest.algo}', '%{note:mod_digest.digest}', '%f', '%{transfer-status}')"
 SQLLog STOR upload
</pre>
Answer: Since the JSON object key names are hardcoded
in <code>mod_redis</code>, converting the above <code>SQLNamedQuery</code>
into a suitable/matching <code>LogFormat</code> is the necessary step. Thus
for example it might become:
<pre>
 LogFormat upload "%u %H %{YYYY-MM-DD HH:MM:SS}t %r %{protocol} %a %T %b %{note:mod_digest.algo} %{note:mod_digest.digest} %f %{transfer-status}"
 RedisLogOnCommand STOR upload
</pre>
Note that <code>LogFormat</code> does not provide a <code>NOW()</code>
function, unlike many SQL databases, thus the <code>%t</code> variable is
needed to provide/fill in that timestamp.

</body>
</html>