Security

• Overview
• Secure InfluxDB and Kapacitor
• Kapacitor Security
• Secure Kapacitor and Chronograf

Overview

This document covers the basics of securing the open-source distribution of Kapacitor.

When seeking to secure Kapacitor it is assumed that the Kapacitor server will be communicating with an already secured InfluxDB server. It will also make its tasks and alerts available to a Chronograf installation.

The following discussion will cover configuring Kapacitor to communicate with a secure InfluxDB server, enabling TLS in Kapacitor and connecting a TLS enabled Kapacitor server to Chronograf.

Authentication and Authorization are not fully implemented in the open-source Kapacitor distribution, but are available as a feature of Enterprise Kapacitor.

Secure InfluxDB and Kapacitor

InfluxDB can secure its communications with TLS on the transport layer and with authentication into the database. How to enable TLS and authentication and authorization in InfluxDB is covered in the InfluxDB documentation, in the sections HTTPS Setup and Authentication and Authorization respectively.

Kapacitor configuration supports both HTTPS communications and Authentication with InfluxDB. Parameters can be set directly in the configuration file, as environment variables or over Kapacitor’s HTTP API.

An overview of Kapacitor configuration is provided in the Configuration document.

Kapacitor and InfluxDB HTTPS

To activate a TLS connection the urls strings in the influxdb servers configuration will need to contain the https protocol. Furthermore either a PEM encoded public key and certificate pair or a PEM encoded CA file will need to be specified.

When testing with a self-signed certificate it is also important to switch off certificate verification with the property insecure-skip-verify. Failure to do so will result in x509 certificate errors as follows:

ts=2018-02-19T13:26:11.437+01:00 lvl=error msg="failed to connect to InfluxDB, retrying..." service=influxdb cluster=localhost err="Get https://localhost:8086/ping: x509: certificate is valid for lenovo-TP02, not localhost"

In the configuration file these values are set according to the following example.

Example 1 – TLS Configuration Properties for InfluxDB – kapacitor.conf

[[influxdb]]
  # Connect to an InfluxDB cluster
  # Kapacitor can subscribe, query and write to this cluster.
  # Using InfluxDB is not required and can be disabled.
  enabled = true
  default = true
  name = "localhost"
  urls = ["https://localhost:8086"]
  timeout = 0
  # Absolute path to pem encoded CA file.
  # A CA can be provided without a key/cert pair
  #   ssl-ca = "/etc/ssl/influxdata-selfsigned-incl-pub-key.pem"
  # Absolutes paths to pem encoded key and cert files.
     ssl-cert = "/etc/ssl/influxdb-selfsigned.crt"
     ssl-key = "/etc/ssl/influxdb-selfsigned.key"
...
   insecure-skip-verify = false
...
   subscription-protocol = "https"     
...   

The relevant properties in Example 1 are:

• urls – note the protocol is https and not http.
• ssl-cert and ssl-key – to indicate the location of the certificate and key files.
• insecure-skip-verify – for testing with a self-signed certificate set this to true otherwise it should be false, especially in production environments.
• subscription-protocol – to declare the correct protocol for subscription communications. For example if Kapacitor is to run on HTTP then this should be set to "http", however if Kapacitor is to run on “HTTPS” then this should be set to "https".

Note that when a CA file contains the certificate and key together the property ssl-ca can be used in place of ssl-cert and ssl-key.

As environment variables these properties can be set as follows:

Example 2 – TLS Configuration Properties for InfluxDB – ENVARS

KAPACITOR_INFLUXDB_0_URLS_0="https://localhost:8086"
KAPACITOR_INFLUXDB_0_SSL_CERT="/etc/ssl/influxdb-selfsigned.crt"
KAPACITOR_INFLUXDB_0_SSL_KEY="/etc/ssl/influxdb-selfsigned.key"
KAPACITOR_INFLUXDB_0_INSECURE_SKIP_VERIFY=true
KAPACITOR_INFLUXDB_0_SUBSCRIPTION_PROTOCOL="https"

When using Systemd to manage the Kapacitor daemon the above parameters can be stored in the file /etc/default/kapacitor.

Kapacitor to InfluxDB TLS configuration over HTTP API

These properties can also be set using the HTTP API. To get the current InfluxDB part of the Kapacitor configuration, use the following curl command:

curl -ks http://localhost:9092/kapacitor/v1/config/influxdb | python -m json.tool > kapacitor-influxdb.conf

This results in the following file:

Example 3 – The InfluxDB part of the Kapacitor configuration

{
    "elements": [
        {
            "link": {
                "href": "/kapacitor/v1/config/influxdb/localhost",
                "rel": "self"
            },
            "options": {
                "default": true,
                "disable-subscriptions": false,
                "enabled": true,
                "excluded-subscriptions": {
                    "_kapacitor": [
                        "autogen"
                    ]
                },
                "http-port": 0,
                "insecure-skip-verify": false,
                "kapacitor-hostname": "",
                "name": "localhost",
                "password": true,
                "ssl-ca": "",
                "ssl-cert": "/etc/ssl/influxdb-selfsigned.crt",
                "ssl-key": "/etc/ssl/influxdb-selfsigned.key",
                "startup-timeout": "5m0s",
                "subscription-mode": "cluster",
                "subscription-protocol": "https",
                "subscription-path": "",
                "subscriptions": {},
                "subscriptions-sync-interval": "1m0s",
                "timeout": "0s",
                "udp-bind": "",
                "udp-buffer": 1000,
                "udp-read-buffer": 0,
                "urls": [
                    "https://localhost:8086"
                ],
                "username": "admin"
            },
            "redacted": [
                "password"
            ]
        }
    ],
    "link": {
        "href": "/kapacitor/v1/config/influxdb",
        "rel": "self"
    }
}

Properties can be updated by POSTing a JSON document containing the field `“set”`` followed by the properties to be modified.

For example, the following command switches off the insecure-skip-verify property.

curl -kv -d '{ "set": { "insecure-skip-verify": false } }' http://localhost:9092/kapacitor/v1/config/influxdb/
...
upload completely sent off: 43 out of 43 bytes
< HTTP/1.1 204 No Content
< Content-Type: application/json; charset=utf-8
< Request-Id: 189e9abb-157b-11e8-866a-000000000000
< X-Kapacitor-Version: 1.7.6~n201802140813
< Date: Mon, 19 Feb 2018 13:45:07 GMT
<
* Connection #0 to host localhost left intact

Similar commands:

• To change the URLS:

  curl -kv -d '{ "set": { "urls": [ "https://lenovo-TP02:8086" ]} }' https://localhost:9092/kapacitor/v1/config/influxdb/
  

• To set the subscription-protocol:

  curl -kv -d '{ "set": { "subscription-protocol": "https" } }' https://localhost:9092/kapacitor/v1/config/influxdb/
  

• If Kapacitor is behind a reverse proxy, set the subscription-path to append to the InfluxDB subscription URL:

  curl -kv -d '{ "set": { "subscription-path": "/path/behind/reverse-proxy" } }' https://localhost:9092/kapacitor/v1/config/influxdb/
  

• To set the path to the CA Certificate:

  curl -kv -d '{ "set": { "ssl-ca": "/etc/ssl/influxdata-selfsigned-incl-pub-key.pem" } }' https://localhost:9092/kapacitor/v1/config/influxdb/
  

Other properties can be set in a similar fashion.

Kapacitor and InfluxDB Authentication

An additional security mechanism available in InfluxDB is Authentication and Authorization. Kapacitor can be configured to communicate with InfluxDB using a username:password pair. These properties can be set in the configuration file, as environment variables or over the HTTP API.

Example 4 – InfluxDB Authentication Parameters – kapacitor.conf

[[influxdb]]
  # Connect to an InfluxDB cluster
  # Kapacitor can subscribe, query and write to this cluster.
  # Using InfluxDB is not required and can be disabled.
  enabled = true
  default = true
  name = "localhost"
  urls = ["https://localhost:8086"]
  username = "admin"
  password = "changeit"
  timeout = 0
...

The relevant parameters in Example 4 are username and password.

These can also be set as environment variables.

Example 5 – InfluxDB Authentication Parameters – ENVARS

KAPACITOR_INFLUXDB_0_USERNAME="admin"
KAPACITOR_INFLUXDB_0_PASSWORD="changeit"

When using Systemd to manage the Kapacitor daemon the above parameters can be stored in the file /etc/defaults/kapacitor.

Alternately they can be set or updated over the HTTP API.

$ curl -kv -d '{ "set": { "username": "foo", "password": "bar" } }' https://localhost:9092/kapacitor/v1/config/influxdb/

Kapacitor Security

Open-source Kapacitor offers TLS for encrypting communications to the HTTP API.

Kapacitor over TLS

This feature can be enabled in the configuration http group of the configuration. Activation requires simply setting the property https-enabled to true and then providing a path to a certificate with the property, https-certificate. If your certificate’s private key is separate, specify the path to the private key using the https-private-key property.

The following example shows how this is done in the kapacitor.conf file.

Example 6 – Enabling TLS in kapacitor.conf

[http]
  # HTTP API Server for Kapacitor
  # This server is always on,
  # it serves both as a write endpoint
  # and as the API endpoint for all other
  # Kapacitor calls.
  bind-address = ":9092"
  log-enabled = true
  write-tracing = false
  pprof-enabled = false
  https-enabled = true
  https-certificate = "/etc/ssl/influxdata-selfsigned.crt"
  https-private-key = "/etc/ssl/influxdata-selfsigned.key"

These values can also be set as environment variables as shown in the next example.

Example 7 – Enabling TLS as ENVARS

KAPACITOR_HTTP_HTTPS_ENABLED=true
KAPACITOR_HTTP_HTTPS_CERTIFICATE="/etc/ssl/influxdata-selfsigned.crt"
KAPACITOR_HTTP_HTTPS_PRIVATE_KEY="/etc/ssl/influxdata-selfsigned.key"

However, they cannot be set over the HTTP API.

Please remember, that when Kapacitor is running on HTTPS, this needs to be reflected in the subscription-protocol property for the [[influxdb]] group of the Kapacitor configuration. See Example 1 above. The value of this property needs to be set to https. Failure to do so will result in a TLS handshake error with the message oversized record received with length 21536 in the Kapacitor log as shown here:

ts=2018-02-19T13:23:49.684+01:00 lvl=error msg="2018/02/19 13:23:49 http: TLS handshake error from 127.0.0.1:49946: tls: oversized record received with length 21536\n" service=http service=httpd_server_errors

If for any reason TLS is switched off, this property needs to be reset to http. Failure to do so will result in the inability of InfluxDB to push subscribed data to Kapacitor with a message in the InfluxDB log like the following:

mar 05 17:02:40 algonquin influxd[32520]: [I] 2018-03-05T16:02:40Z Post https://localhost:9092/write?consistency=&db=telegraf&precision=ns&rp=autogen: http: server gave HTTP response to HTTPS client service=subscriber

Kapacitor command-line client with HTTPS

Once HTTPS has been enabled the Kapacitor command line client will need to be supplied the -url argument in order to connect. If a self-signed or other certificate is used, which has not been added to the system certificate store, an addition argument -skipVerify will also need to be provided.

$ kapacitor -url https://localhost:9092 -skipVerify list tasks
ID                                                 Type      Status    Executing Databases and Retention Policies
chronograf-v1-3586109e-8b7d-437a-80eb-a9c50d00ad53 stream    enabled   true      ["telegraf"."autogen"]

Kapacitor Authentication and Authorization

To ensure Kapacitor requires a username and password to connect, enable basic authentication. To do this, set up the username, password, and auth-enabled parameters in the [http] section of kapacitor.conf.

Kapacitor also supports using InfluxDB Enterprise to manage authentication and authorization for interactions with the Kapacitor API. For instructions, see “Set up InfluxDB Enterprise authorizations”.

Note on HTTP API Configuration and Restarting Kapacitor

Please be aware that when configuration values are set using the HTTP API, that these values will persist in the Kapacitor database even after restart. To switch off these overrides on restart set the property skip-config-overrides to true either in the configuration file (kapacitor.conf) or as an environment variable (KAPACITOR_SKIP_CONFIG_OVERRIDES).

When troubleshooting connection issues after restart, check the HTTP API, for example at http://localhost:9092/kapacitor/v1/config. This can be especially useful if Kapacitor to InfluxDB communications do not seem to be respecting values seen in the file kapacitor.conf or in environment variables.

Deny specific CIDR ranges

To deny access to Kapacitor from certain CIDR ranges (classes of IP addresses), use the -blacklist-cidrs flag. Pass a comma-separated list of CIDRs to deny for most HTTP GET/POST operations:

kapacitord -blacklist-cidrs 10.0.0.0/8,0.0.0.0/32

Disable specific alert handlers

Use the -disable-handlers flag to disable a set of alert handlers. Pass a comma-separated list of handlers:

kapacitord -disable-handlers exec,httppost

Secure Kapacitor and Chronograf

With Kapacitor configured with HTTPS/TLS enabled many users will want to add Kapacitor to their connection configuration in Chronograf. The primary requirement for this to work is to have the base signing certificate installed on the host where the Chronograf service is running. With most operating systems this should already be the case.

When working with a self-signed certificate, this means installing the self-signed certificate into the system.

Install a Self-Signed Certificate on Debian

As an example of installing a self-signed certificate to the system, in Debian/Ubuntu any certificate can be copied to the directory /usr/local/share/ca-certificates/ and then the certificate store can be rebuilt.

$ sudo cp /etc/ssl/influxdb-selfsigned.crt /usr/local/share/ca-certificates/
$ sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Replacing debian:influxdb-selfsigned.pem
done.
done.

If a self-signed or other certificate has been added to the system the Chronograf service needs to be restarted to gather the new certificate information.

$ sudo systemctl restart chronograf.service

Adding a Kapacitor Connection in Chronograf

The following instructions apply to the Chronograf UI. If Chronograf has been installed it can be found by default at port 8888 (e.g. http://localhost:8888).

1. In the left side navigation bar open the Configuration page. This will show all available InfluxDB connections. In the row containing the InfluxDB connection for which a Kapacitor connection is to be added, click the link Add Kapacitor Connection. This will load the Add a New Kapacitor Connection page.

Image 1 – Adding a Kapacitor Connection

2. In the Connection Details group fill in such details as a name for the connection and click the Connect button.

Image 2 – Kapacitor Connection Details

3. If the certificate is installed on the system a success notification will appear.

Image 3 – Kapacitor Connection Success

If an error notification is returned check the Chronograf log for proxy errors. For example:

mar 06 13:53:07 lenovo-tp02 chronograf[12079]: 2018/03/06 13:53:07 http: proxy error: x509: certificate is valid for locahlost, not localhost

4. Also tabbed forms for editing and adding Kapacitor Handler Endpoints will appear. In wider screens they will be to the right of the Connection Details group. In narrower screens they will be below the Connection Details group.

Image 4 – Configure Kapacitor Handler Endpoints

At this point Kapacitor can be used to generate alerts and TICKscripts through Chronograf. These features are available through the Alerting item in the left navigation bar.