Setup

Requirements

The only requirement for Kibanalytics is a server with Gitopen in new window and Docker-Composeopen in new window installed. All required services are build and abtracted Docker containers.

Source Code

Gitopen in new window is a free and open source distributed version control system.

The easiest way to get Kibanalytics source code is by installing Git and running the following command:

git clone https://github.com/kibanalytics/kibanalytics.git

Before starting Kibanalytics, it's mandatory to provide some basic configuration files and environment variales.

Environment Variables

Kibanalytics will look for a .env file at the root of the project folder to apply settings to the server.

TIP

Use .env.example as base for your environment variables setup. If you just want start straightaway, just rename .env.example to .env and use all default values.

cp .env.example .env

WARNING

It's recomended to change the EXPRESS_SESSION_SECRET and ELASTICSEARCH_PASSWORD environment variables default values before running Kibanalytics in production.

VariableTypeAllowed Values
NODE_ENVstring'development', 'production'
NODE_CLUSTERint0, 1
NODE_LISTENstringValid host IP and port number
EXPRESS_HELMETint0, 1
EXPRESS_GZIPint0, 1
EXPRESS_CORSint0, 1
EXPRESS_ALLOWED_ORIGINS[string]Any valid RegExp
EXPRESS_SESSION_NAMEstringAny string
EXPRESS_SESSION_SECRETstringAny string
EXPRESS_SESSION_COOKIE_MAX_AGEintNumber >= 0
EXPRESS_SESSION_COOKIE_SAME_SITEstring'0', '1', lax', 'none' or 'strict'
EXPRESS_SESSION_COOKIE_SECUREint0, 1
EXPRESS_SESSION_DURATIONintNumber >= 0
EXPRESS_VALIDATE_JSON_SCHEMAint0, 1
EXPRESS_ANONYMIZE_USER_IPint0, 1
EXPRESS_PUBLIC_FOLDERint0, 1
SENTRY_DSNstringValid Sentry DSN URL
LOGSTASH_REDIS_HOSTstringValid Redis host
LOGSTASH_REDIS_PORTintValid Redis host port number
REDIS_LISTENstringValid host IP and port number
REDIS_QUEUE_SERVER_URIstringValid Redis URI
REDIS_SESSION_SERVER_URIstringValid Redis URI
TWEMPROXY_LISTENstringValid host IP and port number
ELASTICSEARCH_LISTENstringValid host IP and port number
ELASTICSEARCH_URIstringValid Elasticsearch URI
ELASTICSEARCH_SECURITYbooltrue, false
ELASTICSEARCH_USERNAMEstringAny string
ELASTICSEARCH_PASSWORDstringAny string
KIBANA_LISTENstringValid host IP and port number
DOCKER_LOG_MAX_SIZEstringNumber >= 1 suffixed with magnitude
DOCKER_LOG_MAX_FILEintNumber >= 1

NODE_ENV

Flag which indicates whether the server is running on development or production mode.

If you're running Node.js outside of docker enviroment, set the value of NODE_ENV to "development".

NODE_CLUSTER

Flag to enable or disable Node.js clusteringopen in new window.

This can drastically improve the performance on multicore deployments by starting one process for each core and distributing the load of requests to the web server.

NODE_LISTEN

Expose Node Docker service host ip and port number for external connections.

EXPRESS_HELMET

Define several security HTTP response headers to secure the web server.

Uses Helmetopen in new window library for Express web servers.

If the Kibanalytics web server will be used in conjunction with a reverse proxy server, is recomended to disable Helmet and set all security HTTP headers directly on the proxy.

EXPRESS_GZIP

Flag to enable or disable HTTP response compress.

If the Kibanalytics web server will be used in conjunction with a reverse proxy server, is recomended to disable Kibanalytics compression and let the reverse proxy take care of it.

EXPRESS_CORS

Flag to enable or disable CORSopen in new window. Used with EXPRESS_ALLOWED_ORIGINS environment variable.

EXPRESS_ALLOWED_ORIGINS

Comma separated RegExp values of allowed origins (for example .*virail.[a-z..]+) for CORS. Used with EXPRESS_CORS environment variable.

EXPRESS_SESSION_NAME

The name of the session cookie to set in the response (and read from in the request).

TIP

If you have multiple apps running on the same hostname then you need to use distinct session cookies from each one. The simplest way is to set different name session names per app.

EXPRESS_SESSION_SECRET

This is the secret used to sign the session cookie. Using a secret that cannot be guessed will reduce the ability to hijack a session by guessing the session name.

Changing the secret value will invalidate all existing sessions.

Specifies the number (in milliseconds) to use when calculating the Expires Set-Cookie attribute. This is done by taking the current server time and adding maxAge milliseconds to the value to calculate an Expires datetime. By default, 7776000000 (90d) is set.

Specifies the boolean or string to be the value for the SameSite Set-Cookie attribute. By default, this is '1'.

'1' will set the SameSite attribute to Strict for strict same site enforcement.

'0' will not set the SameSite attribute.

'lax' will set the SameSite attribute to Lax for lax same site enforcement.

'none' will set the SameSite attribute to None for an explicit cross-site cookie.

'strict' will set the SameSite attribute to Strict for strict same site enforcement.

Specifies the boolean value for the Secure Set-Cookie attribute. When '1', the Secure attribute is set, otherwise it is not. By default, the Secure attribute is '0'.

EXPRESS_SESSION_DURATION

Session duration is defined as the time frame during which there are regular active interactions occurring between a user. The session is timed out when there is no activity from the user for a predefined time duration of 1800000ms (30 minutes by default).

EXPRESS_VALIDATE_JSON_SCHEMA

Enable custom event payload validation.

EXPRESS_ANONYMIZE_USER_IP

Hash user IP address for privacy.

EXPRESS_PUBLIC_FOLDER

Serve the public folder on the web server. Used to for development & testing.

SENTRY_DSN

Enable Sentryopen in new window performance and error tracking.

Disable with empty value.

LOGSTASH_REDIS_HOST

Redis queue server hostname.

LOGSTASH_REDIS_PORT

Redis queue server port.

REDIS_LISTEN

Expose Redis Docker service host ip and port number for external connections.

REDIS_QUEUE_SERVER_URI

Redis queue server connection URI. Needs to use the same values from LOGSTASH_REDIS_HOST and LOGSTASH_REDIS_PORT variables. This is necessary because some services needs to use hostname / port and others only accept URI connection string.

REDIS_SESSION_SERVER_URI

Redis session server connection URI.

TWEMPROXY_LISTEN

Expose Twemproxy Docker service host ip and port number for external connections.

ELASTICSEARCH_LISTEN

Expose Elasticsearch Docker service host ip and port number for external connections.

ELASTICSEARCH_URI

Elasticsearchopen in new window server connection URI.

To point to the local Elasticsearch server started with Docker Compose, use the value "http://elasticsearch:9200" as it is an alias to the internal Docker network Elasticsearch host.

If NODE_ENV is set to "development", Node.js server will override ELASTICSEARCH_URI value to "http://localhost:9200".

ELASTICSEARCH_SECURITY

Enable or disable Elasticsearch security. By enabling this flag, Kibana access will also require an username & password.

ELASTICSEARCH_USERNAME

Elasticsearch username. Used with ELASTICSEARCH_SECURITY=true.

ELASTICSEARCH_PASSWORD

Elasticsearch password. Used with ELASTICSEARCH_SECURITY=true.

KIBANA_LISTEN

Expose Kibana Docker service host ip and port number for external connections.

DOCKER_LOG_MAX_SIZE

Docker log file max size.

DOCKER_LOG_MAX_FILE

Docker log files max count.

Install

Docker-Composeopen in new window is a tool that helps to define and share multi-container applications. As Kibanalytics is composed by a stack of open source projects, it's easier for a quick start to use containers.

It's necessary to install Docker and Docker-Compose to the host/server where Kibanalytics will run.

Docker instalation setupopen in new window

Docker-Compose instalation setupopen in new window

After completing all the setup steps from Docker and Docker-Compose, to start Kibanalytics using the default configuration, run the following commands:

cd kibanalytics
cp .env.example .env
cp -r .config.example .config
docker-compose --profile local --profile production up -d --build

If you're starting Kibanalytics for the first time, it's necessary to create a Kibana index patternopen in new window to be able to visualize the collected data.

To make this process easier, we provide a script to automate this process and also load the default dashboards. To run the script, execute the following command:

docker-compose exec node npm run load-dashboards

After all containers started, you can open Kibana to check events and metrics by accessing http://localhost:5601/app/dashboardsopen in new window.

TIP

Kibana server take some minutes to boot, so don't panic if you try to access the Kibana URL and nothing shows up.

If you started Kibanalytics with EXPRESS_PUBLIC_FOLDER=1 and EXPRESS_PORT=3000, you can also access http://localhost:3000open in new window to interact with some example pages to generate events.

Advanced Configurations

It's possible to provide some aditional advanced configurations to Kibanalytics engine gears by editing the corresponding files in "/.config" folder.

Redis

To provide aditional settings to the local Redis instance, edit ".config/redis/redis.conf" file. For more information about the available setting options, please check Redis configurationopen in new window docs.

Twemproxy

To provide aditional settings to the local Twemproxy instance, edit ".config/twemproxy/twemproxy.yml" file. For more information about the available setting options, please check Twemproxy Configurationopen in new window docs.

Logstash

To provide aditional settings to the local Logstash instance, edit ".config/logstash/logstash.yml" file. For more information about the available setting options, please check Logstash Configurationopen in new window docs.

Also, it's possible to edit que main Logstash pipeline by editing ".config/logstash/pipeline/tracker.conf" file. For more information about pipeline settings, please check Configuring Logstashopen in new window docs.

Kibana

To provide aditional settings to the local Kibana instance, edit ".config/kibana/kibana.yml" file. For more information about the available setting options, please check Configure Kibanaopen in new window docs.