Datalizer [Eco Hacker Farm]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Datalizer ======

The Datalizer allows to monitor, analyze and alert based on data that is collected. The data is stored in a optimized structure that allows discarding old values, aggregate them and push new data in a efficient way.

====== Setup ======

The datalizer is build on a docker-capable machine for separation of concerns. We setup two containers and are able to add multiple collectors based on our requirements. "store" and "visualize" are the names of the containers. "store" is a influxdb that exposes a REST interface on port 8086 and a collectd compatible UDP listener on port 25826. "visualize" is a webapp "grafana" that can build beautiful graphs from the data stored in the influxdb. ''collectd'' is the container that fetches data. To collect hardware information it may be required to install collectd in the host directly, the pattern is the same then.

===== Store =====

Store is a influxdb ([[https://en.wikipedia.org/wiki/InfluxDB|Read More]]). InfluxDB is a time-series database written in GO and is OpenSource. It supports retention policies that allow us to keep the amount of data at bay.
To store the cpu, network and load values of a single computer with a 30s accuracy, influxdb takes 100mb per week, 400mb per month or 5gb per year.

Influxdb can be installed without any dependency on a bare debian:

<code>
wget -q https://dl.influxdata.com/influxdb/releases/influxdb_${INFLUXDB_VERSION}_amd64.deb
dpkg -i influxdb_${INFLUXDB_VERSION}_amd64.deb
</code>

To start the influxdb server ''influxd'' is run.

To review the data stored in the database the ''influx'' command is used:

<code>
influx
create database collectd_db
show databases
use collectd_db
show series
select * from thermal_value
</code>

===== Visualize =====

Visualization is done using Grafana ([[https://grafana.com/|Read More]]). Grafana is a webapp written in Javascript and GO, an open source, feature rich metrics dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus and InfluxDB.
It allows to define overviews (Dashboards) for certain areas and allows to define different users (Viewers, Admins)

Grafana can be installed with minimal dependencies on a bare debian:

<code>
apt-get -y install curl apt-transport-https ca-certificates
echo 'deb https://packagecloud.io/grafana/stable/debian/ jessie main' > /etc/apt/sources.list.d/grafana.list
curl https://packagecloud.io/gpg.key | apt-key add -
apt-get update && apt-get -y install grafana
</code>

After the installation grafana needs to be started with ''/usr/sbin/grafana-server'' in the ''/usr/share/grafana'' directory.
It creates a sqlite database in ''/usr/share/grafana/data/grafana.db'' that holds the dashboard configuration and the configured users.

When the application is running, point a browser to port 3000 on the host and login using ''admin:admin''. Change the organisation and default admin password before defining any dashboards.

===== Collectd =====

Collectd ([[https://en.wikipedia.org/wiki/Collectd|Read More]]) is a Unix daemon that collects, transfers and stores performance data of computers and network equipment. The acquired data is meant to help system administrators maintain an overview over available resources to detect existing or looming bottlenecks.

Collectd is plugin based and available for many platforms including openwrt. It allows us to monitor the performance of the infrastructure with its core computers to detect issues before the users.

To install collectd on a bare debian you can choose to install collectd or collectd-core. the collectd package pulls various X-related dependencies that should be avoided on servers. The collectd-core does come with all the useful plugins but without configuration.

<code>
apt-get install -y collectd-core
</code>

To push the values that collectd creates to the ''store'' the network plugin is used. It supports encryption of the data (but you can still see the kind (cpu/network/..) of data in the packets).
Collectd pushes the fqdn of the machine (hostname -f) when the configuration does not override the hostname. It is advised to configure the host correctly before starting the daemon so the influxdb does contain the correct hostname.
A minimal configuration that collects the network traffic along with the load and thermal values of the machine will need the following plugins. The config is derived from the gentoo-default config with the defaults commented out and you should read the ''collectd.conf(5)'' manpage to find what the plugins are doing. The network plugin is responsible to send the collected data to the store:

<code>
#Hostname    "localhost"
#FQDNLookup   true
#BaseDir     "/var/lib/collectd"
PIDFile     "/run/collectd/collectd.pid"
#PluginDir   "/usr/lib64/collectd"
#TypesDB     "/usr/share/collectd/types.db"

##LoadPlugin aggregation
##LoadPlugin barometer
LoadPlugin battery
##LoadPlugin conntrack
LoadPlugin cpu
LoadPlugin cpufreq
##LoadPlugin cpusleep
LoadPlugin df
LoadPlugin disk
##LoadPlugin hddtemp
LoadPlugin interface
## LoadPlugin ipmi
LoadPlugin load
LoadPlugin memory
##LoadPlugin multimeter
LoadPlugin network
##LoadPlugin ping
LoadPlugin sensors
LoadPlugin swap
#LoadPlugin tcpconns
LoadPlugin thermal
LoadPlugin uptime
LoadPlugin users
##LoadPlugin vmem
LoadPlugin wireless

#<Plugin "barometer">
#   Device            "/dev/i2c-0";
#   Oversampling      512
#   PressureOffset    0.0
#   TemperatureOffset 0.0
#   Normalization     2
#   Altitude          238.0
#   TemperatureSensor "myserver/onewire-F10FCA000800/temperature"
#</Plugin>

#<Plugin "battery">
#  ValuesPercentage false
#  ReportDegraded false
#</Plugin>

#<Plugin cpu>
#  ReportByCpu true
#  ReportByState true
#  ValuesPercentage false
#</Plugin>

#<Plugin df>
#	Device "/dev/hda1"
#	Device "192.168.0.2:/mnt/nfs"
#	MountPoint "/home"
#	FSType "ext3"
#	IgnoreSelected false
#	ReportByDevice false
#	ReportInodes false
#	ValuesAbsolute true
#	ValuesPercentage false
#</Plugin>

#<Plugin disk>
#	Disk "/^[hs]d[a-f][0-9]?$/"
#	IgnoreSelected false
#	UseBSDName false
#	UdevNameAttr "DEVNAME"
#</Plugin>

#<Plugin hddtemp>
#  Host "127.0.0.1"
#  Port "7634"
#</Plugin>

#<Plugin interface>
#	Interface "eth0"
#	IgnoreSelected false
#	ReportInactive true
#	UniqueName false
#</Plugin>

#<Plugin ipmi>
#	Sensor "some_sensor"
#	Sensor "another_one"
#	IgnoreSelected false
#	NotifySensorAdd false
#	NotifySensorRemove true
#	NotifySensorNotPresent false
#</Plugin>

#<Plugin load>
#        ReportRelative true
#</Plugin>

#<Plugin memory>
#	ValuesAbsolute true
#	ValuesPercentage false
#</Plugin>

<Plugin network>
#	# client setup:
	Server "store" "25826"
#	TimeToLive 128
#
#	# server setup:
#	Listen "ff18::efc0:4a42" "25826"
#	<Listen "239.192.74.66" "25826">
#		SecurityLevel Sign
#		AuthFile "/etc/collectd/passwd"
#		Interface "eth0"
#	</Listen>
#	MaxPacketSize 1452
#
#	# proxy setup (client and server as above):
#	Forward true
#
#	# statistics about the network plugin itself
#	ReportStats false
#
#	# "garbage collection"
#	CacheFlush 1800
</Plugin>

#<Plugin ping>
#	Host "host.foo.bar"
#	Interval 1.0
#	Timeout 0.9
#	TTL 255
#	SourceAddress "1.2.3.4"
#	Device "eth0"
#	MaxMissed -1
#</Plugin>

#<Plugin protocols>
#	Value "/^Tcp:/"
#	IgnoreSelected false
#</Plugin>

#<Plugin sensors>
#	SensorConfigFile "/etc/sensors.conf"
#	Sensor "it8712-isa-0290/temperature-temp1"
#	Sensor "it8712-isa-0290/fanspeed-fan3"
#	Sensor "it8712-isa-0290/voltage-in8"
#	IgnoreSelected false
#</Plugin>

#<Plugin swap>
#	ReportByDevice false
#	ReportBytes true
#	ValuesAbsolute true
#	ValuesPercentage false
#</Plugin>

#<Plugin table>
#	<Table "/proc/slabinfo">
#		Instance "slabinfo"
#		Separator " "
#		<Result>
#			Type gauge
#			InstancePrefix "active_objs"
#			InstancesFrom 0
#			ValuesFrom 1
#		</Result>
#		<Result>
#			Type gauge
#			InstancePrefix "objperslab"
#			InstancesFrom 0
#			ValuesFrom 4
#		</Result>
#	</Table>
#</Plugin>

#<Plugin tcpconns>
#	ListeningPorts false
#	AllPortsSummary false
#	LocalPort "25"
#	RemotePort "25"
#</Plugin>

#<Plugin thermal>
#	ForceUseProcfs false
#	Device "THRM"
#	IgnoreSelected false
#</Plugin>

#<Plugin vmem>
#	Verbose false
#</Plugin>
</code>

====== Container ======

Putting it all together, a multi container setup is defined using the docker-compose.yml:

<code>
collectd:
  build: collectd
  links:
    - store

store:
  build: store
  volumes:
    - ./volumes/store:/var/lib/influxdb:rw
    - ./store/config/influxdb.conf:/etc/influxdb/influxdb.conf:ro
    - ./store/config/collectd.types.db:/opt/collectd/share/collectd/types.db:ro
  ports:
    #  collectd port
    - "25826:25826/udp"

visualize:
  build: visualize
  volumes:
    - ./visualize/config/grafana-server:/etc/default/grafana-server:ro
    - ./visualize/config/grafana.ini:/etc/grafana/grafana.ini:ro
    - ./volumes/visualize/grafana.log:/var/log/grafana/grafana.log:rw
    - ./volumes/visualize/grafana.sqlite3.db:/usr/share/grafana/data/grafana.db:rw
  ports:
    - "80:3000"
  links:
    - store

</code>

As you can see multiple volumes are defined that need to be prepared prior to the first build, otherwise docker-compose will create directories where files are required:

<code>
touch ./volumes/visualize/grafana.sqlite3.db
touch ./volumes/visualize/grafana.log
mkdir -p ./volumes/store
</code>

Now ''Dockerfiles'' need to be prepared:

<code>
# store/Dockerfile
FROM debian:latest

RUN apt-get update && apt-get install -y wget curl

RUN gpg \
    --keyserver hkp://ha.pool.sks-keyservers.net \
    --recv-keys 05CE15085FC09D18E99EFB22684A14CF2582E0C5

ENV INFLUXDB_VERSION 1.2.1
RUN wget -q https://dl.influxdata.com/influxdb/releases/influxdb_${INFLUXDB_VERSION}_amd64.deb.asc && \
    wget -q https://dl.influxdata.com/influxdb/releases/influxdb_${INFLUXDB_VERSION}_amd64.deb && \
    gpg --batch --verify influxdb_${INFLUXDB_VERSION}_amd64.deb.asc influxdb_${INFLUXDB_VERSION}_amd64.deb && \
    dpkg -i influxdb_${INFLUXDB_VERSION}_amd64.deb && \
    rm -f influxdb_${INFLUXDB_VERSION}_amd64.deb*

EXPOSE 8086
EXPOSE 25826

CMD ["influxd"]
</code>

<code>
# visualize/Dockerfile
FROM debian:latest

RUN apt-get update && apt-get -y install curl apt-transport-https ca-certificates

RUN echo 'deb https://packagecloud.io/grafana/stable/debian/ jessie main' > /etc/apt/sources.list.d/grafana.list \
 && curl https://packagecloud.io/gpg.key | apt-key add - \
 && apt-get update && apt-get -y install grafana

WORKDIR /usr/share/grafana

CMD /usr/sbin/grafana-server
</code>

Run ''docker-compose build'' to create the images for the containers. This takes some time based on the hardware and internetuplink. We use the time to create the configuration.

store/config/collectd.types.db
<code>
absolute		count:ABSOLUTE:0:U
apache_bytes		count:COUNTER:0:134217728
apache_connections	count:GAUGE:0:65535
apache_idle_workers	count:GAUGE:0:65535
apache_requests		count:COUNTER:0:134217728
apache_scoreboard	count:GAUGE:0:65535
arc_counts		demand_data:COUNTER:0:U, demand_metadata:COUNTER:0:U, prefetch_data:COUNTER:0:U, prefetch_metadata:COUNTER:0:U
arc_l2_bytes		read:COUNTER:0:U, write:COUNTER:0:U
arc_l2_size		value:GAUGE:0:U
arc_ratio		value:GAUGE:0:U
arc_size		current:GAUGE:0:U, target:GAUGE:0:U, minlimit:GAUGE:0:U, maxlimit:GAUGE:0:U
ath_nodes		value:GAUGE:0:65535
ath_stat		value:COUNTER:0:4294967295
bitrate			value:GAUGE:0:4294967295
bytes			value:GAUGE:0:U
cache_ratio		value:GAUGE:0:100
cache_result		value:COUNTER:0:4294967295
cache_size		value:GAUGE:0:4294967295
charge			value:GAUGE:0:U
compression		uncompressed:COUNTER:0:U, compressed:COUNTER:0:U
compression_ratio	value:GAUGE:0:2
connections		value:COUNTER:0:U
conntrack		entropy:GAUGE:0:4294967295
contextswitch		contextswitches:DERIVE:0:U
counter			value:COUNTER:U:U
cpufreq			value:GAUGE:0:U
cpu			value:COUNTER:0:4294967295
current			value:GAUGE:U:U
delay			seconds:GAUGE:-1000000:1000000
derive			value:DERIVE:0:U
df			used:GAUGE:0:1125899906842623, free:GAUGE:0:1125899906842623
df_complex		value:GAUGE:0:U
df_inodes		value:GAUGE:0:U
disk_latency		read:GAUGE:0:U, write:GAUGE:0:U
disk_merged		read:COUNTER:0:4294967295, write:COUNTER:0:4294967295
disk_octets		read:COUNTER:0:17179869183, write:COUNTER:0:17179869183
disk_ops		read:COUNTER:0:4294967295, write:COUNTER:0:4294967295
disk_ops_complex	value:COUNTER:0:4294967296
disk_time		read:COUNTER:0:1000000, write:COUNTER:0:1000000
dns_answer		value:COUNTER:0:65535
dns_notify		value:COUNTER:0:65535
dns_octets		queries:COUNTER:0:125000000, responses:COUNTER:0:125000000
dns_opcode		value:COUNTER:0:65535
dns_qtype_cached	value:GAUGE:0:4294967295
dns_qtype		value:COUNTER:0:65535
dns_query		value:COUNTER:0:65535
dns_question		value:COUNTER:0:65535
dns_rcode		value:COUNTER:0:65535
dns_reject		value:COUNTER:0:65535
dns_request		value:COUNTER:0:65535
dns_resolver		value:COUNTER:0:65535
dns_response		value:COUNTER:0:65535
dns_transfer		value:COUNTER:0:65535
dns_update		value:COUNTER:0:65535
dns_zops		value:COUNTER:0:65535
email_check		value:GAUGE:0:U
email_count		value:GAUGE:0:U
email_size		value:GAUGE:0:U
entropy			entropy:GAUGE:0:4294967295
fanspeed		value:GAUGE:0:U
file_size		bytes:GAUGE:0:U
files			value:GAUGE:0:U
frequency		frequency:GAUGE:0:U
frequency_offset	ppm:GAUGE:-1000000:1000000
fscache_stat		value:COUNTER:0:4294967295
fork_rate		value:DERIVE:0:U
gauge			value:GAUGE:U:U
http_request_methods	count:COUNTER:0:134217728
http_requests		count:COUNTER:0:134217728
http_response_codes	count:COUNTER:0:134217728
humidity		value:GAUGE:0:100
if_collisions		value:COUNTER:0:4294967295
if_dropped		rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295
if_errors		rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295
if_multicast		value:COUNTER:0:4294967295
if_octets		rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295
if_packets		rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295
if_rx_errors		value:COUNTER:0:4294967295
if_tx_errors		value:COUNTER:0:4294967295
invocations		value:DERIVE:0:U
io_octets		rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295
io_packets		rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295
ipt_bytes		value:COUNTER:0:134217728
ipt_packets		value:COUNTER:0:134217728
irq			value:COUNTER:U:65535
latency			value:GAUGE:0:65535
links			value:GAUGE:0:U
load			shortterm:GAUGE:0:100, midterm:GAUGE:0:100, longterm:GAUGE:0:100
memcached_command	value:COUNTER:0:U
memcached_connections	value:GAUGE:0:U
memcached_items		value:GAUGE:0:U
memcached_octets	rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295
memcached_ops		value:COUNTER:0:134217728
memory			value:GAUGE:0:281474976710656
multimeter		value:GAUGE:U:U
mysql_commands		value:COUNTER:0:U
mysql_handler		value:COUNTER:0:U
mysql_locks		value:COUNTER:0:U
mysql_log_position	value:COUNTER:0:4294967295
mysql_octets		rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295
mysql_qcache		hits:COUNTER:0:U, inserts:COUNTER:0:U, not_cached:COUNTER:0:U, lowmem_prunes:COUNTER:0:U, queries_in_cache:GAUGE:0:U
mysql_threads		running:GAUGE:0:U, connected:GAUGE:0:U, cached:GAUGE:0:U, created:COUNTER:0:U
nfs_procedure		value:COUNTER:0:4294967295
nginx_connections	value:GAUGE:0:U
nginx_requests		value:COUNTER:0:134217728
node_octets		rx:COUNTER:0:U, tx:COUNTER:0:U
node_rssi		value:GAUGE:0:255
node_stat		value:COUNTER:0:4294967295
node_tx_rate		value:GAUGE:0:127
operations		value:COUNTER:0:4294967295
percent			percent:GAUGE:0:100.1
pg_blks			value:COUNTER:0:U
pg_db_size		value:GAUGE:0:U
pg_n_tup_c		value:COUNTER:0:U
pg_n_tup_g		value:GAUGE:0:U
pg_numbackends		value:GAUGE:0:U
pg_scan			value:COUNTER:0:U
pg_xact			value:COUNTER:0:U
ping_droprate		value:GAUGE:0:100
ping			ping:GAUGE:0:65535
ping_stddev		value:GAUGE:0:65535
players			value:GAUGE:0:1000000
power			value:GAUGE:0:U
protocol_counter	value:COUNTER:0:U
ps_count		processes:GAUGE:0:1000000, threads:GAUGE:0:1000000
ps_cputime		user:COUNTER:0:16000000, syst:COUNTER:0:16000000
ps_pagefaults		minflt:COUNTER:0:9223372036854775807, majflt:COUNTER:0:9223372036854775807
ps_disk_octets		read:DERIVE:0:U, write:DERIVE:0:U
ps_disk_ops		read:DERIVE:0:U, write:DERIVE:0:U
ps_rss			value:GAUGE:0:9223372036854775807
ps_code			value:GAUGE:0:9223372036854775807
ps_data			value:GAUGE:0:9223372036854775807
ps_stacksize		value:GAUGE:0:9223372036854775807
ps_state		value:GAUGE:0:65535
ps_vm			value:GAUGE:0:9223372036854775807
queue_length		value:GAUGE:0:U
response_time		value:GAUGE:0:U
records			count:GAUGE:0:U
route_etx		value:GAUGE:0:U
route_metric		value:GAUGE:0:U
routes			value:GAUGE:0:U
serial_octets		rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295
signal_noise		value:GAUGE:U:0
signal_power		value:GAUGE:U:0
signal_quality		value:GAUGE:0:U
snr			value:GAUGE:0:U
spam_check		value:GAUGE:0:U
spam_score		value:GAUGE:U:U
swap_io			value:DERIVE:0:1099511627776
swap			value:GAUGE:0:1099511627776
tcp_connections		value:GAUGE:0:4294967295
temperature		value:GAUGE:-273.15:U
threads			value:GAUGE:0:U
time_dispersion		seconds:GAUGE:-1000000:1000000
timeleft		timeleft:GAUGE:0:U
time_offset		seconds:GAUGE:-1000000:1000000
total_bytes		value:DERIVE:0:U
total_requests		value:DERIVE:0:U
total_time_in_ms	value:DERIVE:0:U
total_values		value:DERIVE:0:U
uptime			value:GAUGE:0:4294967295
users			users:GAUGE:0:65535
virt_cpu_total		ns:COUNTER:0:256000000000
virt_vcpu		ns:COUNTER:0:1000000000
vmpage_action		value:COUNTER:0:4294967295
vmpage_faults		minflt:COUNTER:0:9223372036854775807, majflt:COUNTER:0:9223372036854775807
vmpage_io		in:COUNTER:0:4294967295, out:COUNTER:0:4294967295
vmpage_number		value:GAUGE:0:4294967295
voltage_threshold	value:GAUGE:U:U, threshold:GAUGE:U:U
voltage			value:GAUGE:U:U
vs_memory		value:GAUGE:0:9223372036854775807
vs_processes		value:GAUGE:0:65535
vs_threads		value:GAUGE:0:65535
stations		value:GAUGE:0:256
</code>

store/config/influxdb.conf

<code>
[meta]
  dir = "/var/lib/influxdb/meta"

[data]
  dir = "/var/lib/influxdb/data"
  engine = "tsm1"

wal-dir = "/var/lib/influxdb/wal"


[[collectd]]
  enabled = true
  port = 25826
  database = "collectd_db"
  typesdb = "/opt/collectd/share/collectd/types.db"
</code>

visualize/config/grafana-server

<code>
GRAFANA_USER=grafana

GRAFANA_GROUP=grafana

GRAFANA_HOME=/usr/share/grafana

LOG_DIR=/var/log/grafana

DATA_DIR=/var/lib/grafana

MAX_OPEN_FILES=10000

CONF_DIR=/etc/grafana

CONF_FILE=/etc/grafana/grafana.ini

RESTART_ON_UPGRADE=false

PLUGINS_DIR=/var/lib/grafana/plugins
</code>

visualize/config/grafana.ini

<code>
##################### Grafana Configuration Example #####################
#
# Everything has defaults so you only need to uncomment things you want to
# change

# possible values : production, development
; app_mode = production

# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
; instance_name = ${HOSTNAME}

#################################### Paths ####################################
[paths]
# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
#
;data = /var/lib/grafana
#
# Directory where grafana can store logs
#
;logs = /var/log/grafana
#
# Directory where grafana will automatically scan and look for plugins
#
;plugins = /var/lib/grafana/plugins

#
#################################### Server ####################################
[server]
# Protocol (http or https)
;protocol = http

# The ip address to bind to, empty will bind to all interfaces
;http_addr =

# The http port  to use
;http_port = 3000

# The public facing domain name used to access grafana from a browser
;domain = localhost

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false

# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
;root_url = http://localhost:3000

# Log web requests
;router_logging = false

# the path relative working path
;static_root_path = public

# enable gzip
;enable_gzip = false

# https certs & key file
;cert_file =
;cert_key =

#################################### Database ####################################
[database]
# You can configure the database connection by specifying type, host, name, user and password
# as seperate properties or as on string using the url propertie.

# Either "mysql", "postgres" or "sqlite3", it's your choice
;type = sqlite3
;host = 127.0.0.1:3306
;name = grafana
;user = root
# If the password contains # or ; you have to wrap it with trippel quotes. Ex """#password;"""
;password =

# Use either URL or the previous fields to configure the database
# Example: mysql://user:secret@host:port/database
;url =

# For "postgres" only, either "disable", "require" or "verify-full"
;ssl_mode = disable

# For "sqlite3" only, path relative to data_path setting
;path = grafana.db

#################################### Session ####################################
[session]
# Either "memory", "file", "redis", "mysql", "postgres", default is "file"
;provider = file

# Provider config options
# memory: not have any config yet
# file: session dir path, is relative to grafana data_path
# redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=grafana`
# mysql: go-sql-driver/mysql dsn config string, e.g. `user:password@tcp(127.0.0.1:3306)/database_name`
# postgres: user=a password=b host=localhost port=5432 dbname=c sslmode=disable
;provider_config = sessions

# Session cookie name
;cookie_name = grafana_sess

# If you use session in https only, default is false
;cookie_secure = false

# Session life time, default is 86400
;session_life_time = 86400

#################################### Analytics ####################################
[analytics]
# Server reporting, sends usage counters to stats.grafana.org every 24 hours.
# No ip addresses are being tracked, only simple counters to track
# running instances, dashboard and error counts. It is very helpful to us.
# Change this option to false to disable reporting.
;reporting_enabled = true

# Set to false to disable all checks to https://grafana.net
# for new vesions (grafana itself and plugins), check is used
# in some UI views to notify that grafana or plugin update exists
# This option does not cause any auto updates, nor send any information
# only a GET request to http://grafana.net to get latest versions
;check_for_updates = true

# Google Analytics universal tracking code, only enabled if you specify an id here
;google_analytics_ua_id =

#################################### Security ####################################
[security]
# default admin user, created on startup
;admin_user = admin

# default admin password, can be changed before first start of grafana,  or in profile settings
;admin_password = admin

# used for signing
;secret_key = SW2YcwTIb9zpOOhoPsMm

# Auto-login remember days
;login_remember_days = 7
;cookie_username = grafana_user
;cookie_remember_name = grafana_remember

# disable gravatar profile images
;disable_gravatar = false

# data source proxy whitelist (ip_or_domain:port separated by spaces)
;data_source_proxy_whitelist =

[snapshots]
# snapshot sharing options
;external_enabled = true
;external_snapshot_url = https://snapshots-origin.raintank.io
;external_snapshot_name = Publish to snapshot.raintank.io

# remove expired snapshot
;snapshot_remove_expired = true

# remove snapshots after 90 days
;snapshot_TTL_days = 90

#################################### Users ####################################
[users]
# disable user signup / registration
;allow_sign_up = true

# Allow non admin users to create organizations
;allow_org_create = true

# Set to true to automatically assign new users to the default organization (id 1)
;auto_assign_org = true

# Default role new users will be automatically assigned (if disabled above is set to true)
;auto_assign_org_role = Viewer

# Background text for the user field on the login page
;login_hint = email or username

# Default UI theme ("dark" or "light")
;default_theme = dark

[auth]
# Set to true to disable (hide) the login form, useful if you use OAuth, defaults to false
;disable_login_form = false

#################################### Anonymous Auth ##########################
[auth.anonymous]
# enable anonymous access
;enabled = false

# specify organization name that should be used for unauthenticated users
;org_name = Main Org.

# specify role for unauthenticated users
;org_role = Viewer

#################################### Github Auth ##########################
[auth.github]
;enabled = false
;allow_sign_up = true
;client_id = some_id
;client_secret = some_secret
;scopes = user:email,read:org
;auth_url = https://github.com/login/oauth/authorize
;token_url = https://github.com/login/oauth/access_token
;api_url = https://api.github.com/user
;team_ids =
;allowed_organizations =

#################################### Google Auth ##########################
[auth.google]
;enabled = false
;allow_sign_up = true
;client_id = some_client_id
;client_secret = some_client_secret
;scopes = https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email
;auth_url = https://accounts.google.com/o/oauth2/auth
;token_url = https://accounts.google.com/o/oauth2/token
;api_url = https://www.googleapis.com/oauth2/v1/userinfo
;allowed_domains =

#################################### Generic OAuth ##########################
[auth.generic_oauth]
;enabled = false
;name = OAuth
;allow_sign_up = true
;client_id = some_id
;client_secret = some_secret
;scopes = user:email,read:org
;auth_url = https://foo.bar/login/oauth/authorize
;token_url = https://foo.bar/login/oauth/access_token
;api_url = https://foo.bar/user
;team_ids =
;allowed_organizations =

#################################### Grafana.net Auth ####################
[auth.grafananet]
;enabled = false
;allow_sign_up = true
;client_id = some_id
;client_secret = some_secret
;scopes = user:email
;allowed_organizations =

#################################### Auth Proxy ##########################
[auth.proxy]
;enabled = false
;header_name = X-WEBAUTH-USER
;header_property = username
;auto_sign_up = true
;ldap_sync_ttl = 60
;whitelist = 192.168.1.1, 192.168.2.1

#################################### Basic Auth ##########################
[auth.basic]
;enabled = true

#################################### Auth LDAP ##########################
[auth.ldap]
;enabled = false
;config_file = /etc/grafana/ldap.toml
;allow_sign_up = true

#################################### SMTP / Emailing ##########################
[smtp]
;enabled = false
;host = localhost:25
;user =
;password =
;cert_file =
;key_file =
;skip_verify = false
;from_address = admin@grafana.localhost

[emails]
;welcome_email_on_sign_up = false

#################################### Logging ##########################
[log]
# Either "console", "file", "syslog". Default is console and  file
# Use space to separate multiple modes, e.g. "console file"
;mode = console file

# Either "trace", "debug", "info", "warn", "error", "critical", default is "info"
;level = info

# optional settings to set different levels for specific loggers. Ex filters = sqlstore:debug
;filters =


# For "console" mode only
[log.console]
;level =

# log line format, valid options are text, console and json
;format = console

# For "file" mode only
[log.file]
;level =

# log line format, valid options are text, console and json
;format = text

# This enables automated log rotate(switch of following options), default is true
;log_rotate = true

# Max line number of single file, default is 1000000
;max_lines = 1000000

# Max size shift of single file, default is 28 means 1 << 28, 256MB
;max_size_shift = 28

# Segment log daily, default is true
;daily_rotate = true

# Expired days of log file(delete after max days), default is 7
;max_days = 7

[log.syslog]
;level =

# log line format, valid options are text, console and json
;format = text

# Syslog network type and address. This can be udp, tcp, or unix. If left blank, the default unix endpoints will be used.
;network =
;address =

# Syslog facility. user, daemon and local0 through local7 are valid.
;facility =

# Syslog tag. By default, the process' argv[0] is used.
;tag =


#################################### AMQP Event Publisher ##########################
[event_publisher]
;enabled = false
;rabbitmq_url = amqp://localhost/
;exchange = grafana_events

;#################################### Dashboard JSON files ##########################
[dashboards.json]
;enabled = false
;path = /var/lib/grafana/dashboards

#################################### Alerting ######################################
[alerting]
# Makes it possible to turn off alert rule execution.
;execute_alerts = true

#################################### Internal Grafana Metrics ##########################
# Metrics available at HTTP API Url /api/metrics
[metrics]
# Disable / Enable internal metrics
;enabled           = true

# Publish interval
;interval_seconds  = 10

# Send internal metrics to Graphite
[metrics.graphite]
# Enable by setting the address setting (ex localhost:2003)
;address =
;prefix = prod.grafana.%(instance_name)s.

#################################### Internal Grafana Metrics ##########################
# Url used to to import dashboards directly from Grafana.net
[grafana_net]
;url = https://grafana.net

#################################### External image storage ##########################
[external_image_storage]
# Used for uploading images to public servers so they can be included in slack/email messages.
# you can choose between (s3, webdav)
;provider =

[external_image_storage.s3]
;bucket_url =
;access_key =
;secret_key =

[external_image_storage.webdav]
;url =
;username =
;password =
</code>

Now all images should be built and with the configuration in place ''docker-compose up -d'' will launch the store and the visualize servers.
The store needs to create the database before any values are stored:

<code>
docker exec -it datalizer_store_1 /bin/bash
#> influx
i> create database collectd_db
</code>

When a collectd host is configured to push to the datalizer, values should show up almost immediately (''show series'' in influx). To debug, use tcpdump port 25826 to see if values are posted and investigate the logfiles.

====== Push Data ======

When the first values arrive in the store, all you can do is review the cpu, load or other technical values. Our goal is to collect environment values from various sensors as well. There are two kinds of data: live-values and historical values. Live values can be pushed every time they are measured while historical values come from some storage (eg a sd-card).

===== Live Data =====

To push live data from a network device, a simple curl-request with some parameters can be used to create the series:

<code>
curl -i -XPOST \
  'http://datalizer:8086/write?db=${INFLUXDB_NAME}&rp=${RETENTION_POLICY}' \
  --data-binary "${SERIES_NAME},${TAG_NAME}=${TAG_VALUE} value=${VALUE}"
</code>

The INFLUXDB_NAME is a database created for a specific purpose, the RETENTION_POLICY allows to push to that database and ensure the values are dropped after some time. The SERIES_NAME and TAG_NAME is used to differentiate the values when building graphs.

A practical example would post the a temperature of 10.0C of the soil in the north side of the garden to the database 'garden' that has a 'five_years' retention policy:

<code>
curl -i -XPOST \
  'http://datalizer:8086/write?db=garden&rp=five_years' \
  --data-binary "temperature,type=soil,type_location=north value=10.0"
</code>

===== Historic Data =====

To push historic data, the values need to be transformed from their source-format (eg a csv file) to the influx line format. Depending on the type of source, the conversion-process should be automated so future pushes are a simple task. Lets use a imaginary CSV source from our power-meter and push the values to the influxdb.

<code>
1.2.2010 22:30:00 5000
1.2.2010 22:45:00 2000
1.2.2010 23:00:00 0
</code>

As you can see, the format can not 'just' be used, it needs conversion from a weird date (its the first of february right) to the influx nanoseconds since 1970-01-01T00:00:00.0000. (''date -d "INPUT" +%s%N'') Often a simple ''cat|sed|awk'' helps but sometimes a more sophisticated language is required to create the following output for the csv:

<code>
power_consumption,kitchen_kettle 5000 1265063400000000000
power_consumption,kitchen_kettle 2000 1265060700000000000
power_consumption,kitchen_kettle 0000 1265061600000000000
</code>

Up to 5000 entries at once can be written using the multiple points interface:

<code>
curl -i -XPOST 'http://datalizer:8086/write?db=electro' @filename
</code>

[[https://docs.influxdata.com/influxdb/v1.2/guides/writing_data/|Read More]]

====== Considerations ======

Be aware that some of the values can have privacy implications. For example network traffic may allow you to analyze bottlenecks or rouge devices but may also be used to prove that someone was in the hackerspace downloading exactly the 1.1GB "operating system" that was illegal to download. Therefore each measurement should be considered for its implications and a retention policy should avoid having that data stored for a extended period. [[https://docs.influxdata.com/influxdb/v1.2/guides/downsampling_and_retention/|Read More on how to define Retention policies.]]

Discuss and publish the measurements with their policy in the community to prevent surprises.

To make the data useful for everyone, the dashboards from the ''visualize'' container need to be designed with the right units and useful related graphs at one display. The users should allow to setup public displays that show the current network throughput or the forecast weather compared to actual sensor data.