Datalizer
The Datalizer allows to monitor, analyze and alert based on data that is collected. The data is stored in a optimized structure that allows discarding old values, aggregate them and push new data in a efficient way.
Setup
The datalizer is build on a docker-capable machine for separation of concerns. We setup two containers and are able to add multiple collectors based on our requirements. “store” and “visualize” are the names of the containers. “store” is a influxdb that exposes a REST interface on port 8086 and a collectd compatible UDP listener on port 25826. “visualize” is a webapp “grafana” that can build beautiful graphs from the data stored in the influxdb. collectd is the container that fetches data. To collect hardware information it may be required to install collectd in the host directly, the pattern is the same then.
Store
Store is a influxdb (Read More). InfluxDB is a time-series database written in GO and is OpenSource. It supports retention policies that allow us to keep the amount of data at bay. To store the cpu, network and load values of a single computer with a 30s accuracy, influxdb takes 100mb per week, 400mb per month or 5gb per year.
Influxdb can be installed without any dependency on a bare debian:
wget -q https://dl.influxdata.com/influxdb/releases/influxdb_${INFLUXDB_VERSION}_amd64.deb
dpkg -i influxdb_${INFLUXDB_VERSION}_amd64.deb
To start the influxdb server influxd is run.
To review the data stored in the database the influx command is used:
influx create database collectd_db show databases use collectd_db show series select * from thermal_value
Visualize
Visualization is done using Grafana (Read More). Grafana is a webapp written in Javascript and GO, an open source, feature rich metrics dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus and InfluxDB. It allows to define overviews (Dashboards) for certain areas and allows to define different users (Viewers, Admins)
Grafana can be installed with minimal dependencies on a bare debian:
apt-get -y install curl apt-transport-https ca-certificates echo 'deb https://packagecloud.io/grafana/stable/debian/ jessie main' > /etc/apt/sources.list.d/grafana.list curl https://packagecloud.io/gpg.key | apt-key add - apt-get update && apt-get -y install grafana
After the installation grafana needs to be started with /usr/sbin/grafana-server in the /usr/share/grafana directory.
It creates a sqlite database in /usr/share/grafana/data/grafana.db that holds the dashboard configuration and the configured users.
When the application is running, point a browser to port 3000 on the host and login using admin:admin. Change the organisation and default admin password before defining any dashboards.
Collectd
Collectd (Read More) is a Unix daemon that collects, transfers and stores performance data of computers and network equipment. The acquired data is meant to help system administrators maintain an overview over available resources to detect existing or looming bottlenecks.
Collectd is plugin based and available for many platforms including openwrt. It allows us to monitor the performance of the infrastructure with its core computers to detect issues before the users.
To install collectd on a bare debian you can choose to install collectd or collectd-core. the collectd package pulls various X-related dependencies that should be avoided on servers. The collectd-core does come with all the useful plugins but without configuration.
apt-get install -y collectd-core
To push the values that collectd creates to the store the network plugin is used. It supports encryption of the data (but you can still see the kind (cpu/network/..) of data in the packets).
Collectd pushes the fqdn of the machine (hostname -f) when the configuration does not override the hostname. It is advised to configure the host correctly before starting the daemon so the influxdb does contain the correct hostname.
A minimal configuration that collects the network traffic along with the load and thermal values of the machine will need the following plugins. The config is derived from the gentoo-default config with the defaults commented out and you should read the collectd.conf(5) manpage to find what the plugins are doing. The network plugin is responsible to send the collected data to the store:
#Hostname "localhost" #FQDNLookup true #BaseDir "/var/lib/collectd" PIDFile "/run/collectd/collectd.pid" #PluginDir "/usr/lib64/collectd" #TypesDB "/usr/share/collectd/types.db" ##LoadPlugin aggregation ##LoadPlugin barometer LoadPlugin battery ##LoadPlugin conntrack LoadPlugin cpu LoadPlugin cpufreq ##LoadPlugin cpusleep LoadPlugin df LoadPlugin disk ##LoadPlugin hddtemp LoadPlugin interface ## LoadPlugin ipmi LoadPlugin load LoadPlugin memory ##LoadPlugin multimeter LoadPlugin network ##LoadPlugin ping LoadPlugin sensors LoadPlugin swap #LoadPlugin tcpconns LoadPlugin thermal LoadPlugin uptime LoadPlugin users ##LoadPlugin vmem LoadPlugin wireless #<Plugin "barometer"> # Device "/dev/i2c-0"; # Oversampling 512 # PressureOffset 0.0 # TemperatureOffset 0.0 # Normalization 2 # Altitude 238.0 # TemperatureSensor "myserver/onewire-F10FCA000800/temperature" #</Plugin> #<Plugin "battery"> # ValuesPercentage false # ReportDegraded false #</Plugin> #<Plugin cpu> # ReportByCpu true # ReportByState true # ValuesPercentage false #</Plugin> #<Plugin df> # Device "/dev/hda1" # Device "192.168.0.2:/mnt/nfs" # MountPoint "/home" # FSType "ext3" # IgnoreSelected false # ReportByDevice false # ReportInodes false # ValuesAbsolute true # ValuesPercentage false #</Plugin> #<Plugin disk> # Disk "/^[hs]d[a-f][0-9]?$/" # IgnoreSelected false # UseBSDName false # UdevNameAttr "DEVNAME" #</Plugin> #<Plugin hddtemp> # Host "127.0.0.1" # Port "7634" #</Plugin> #<Plugin interface> # Interface "eth0" # IgnoreSelected false # ReportInactive true # UniqueName false #</Plugin> #<Plugin ipmi> # Sensor "some_sensor" # Sensor "another_one" # IgnoreSelected false # NotifySensorAdd false # NotifySensorRemove true # NotifySensorNotPresent false #</Plugin> #<Plugin load> # ReportRelative true #</Plugin> #<Plugin memory> # ValuesAbsolute true # ValuesPercentage false #</Plugin> <Plugin network> # # client setup: Server "store" "25826" # TimeToLive 128 # # # server setup: # Listen "ff18::efc0:4a42" "25826" # <Listen "239.192.74.66" "25826"> # SecurityLevel Sign # AuthFile "/etc/collectd/passwd" # Interface "eth0" # </Listen> # MaxPacketSize 1452 # # # proxy setup (client and server as above): # Forward true # # # statistics about the network plugin itself # ReportStats false # # # "garbage collection" # CacheFlush 1800 </Plugin> #<Plugin ping> # Host "host.foo.bar" # Interval 1.0 # Timeout 0.9 # TTL 255 # SourceAddress "1.2.3.4" # Device "eth0" # MaxMissed -1 #</Plugin> #<Plugin protocols> # Value "/^Tcp:/" # IgnoreSelected false #</Plugin> #<Plugin sensors> # SensorConfigFile "/etc/sensors.conf" # Sensor "it8712-isa-0290/temperature-temp1" # Sensor "it8712-isa-0290/fanspeed-fan3" # Sensor "it8712-isa-0290/voltage-in8" # IgnoreSelected false #</Plugin> #<Plugin swap> # ReportByDevice false # ReportBytes true # ValuesAbsolute true # ValuesPercentage false #</Plugin> #<Plugin table> # <Table "/proc/slabinfo"> # Instance "slabinfo" # Separator " " # <Result> # Type gauge # InstancePrefix "active_objs" # InstancesFrom 0 # ValuesFrom 1 # </Result> # <Result> # Type gauge # InstancePrefix "objperslab" # InstancesFrom 0 # ValuesFrom 4 # </Result> # </Table> #</Plugin> #<Plugin tcpconns> # ListeningPorts false # AllPortsSummary false # LocalPort "25" # RemotePort "25" #</Plugin> #<Plugin thermal> # ForceUseProcfs false # Device "THRM" # IgnoreSelected false #</Plugin> #<Plugin vmem> # Verbose false #</Plugin>
Container
Putting it all together, a multi container setup is defined using the docker-compose.yml:
collectd:
build: collectd
links:
- store
store:
build: store
volumes:
- ./volumes/store:/var/lib/influxdb:rw
- ./store/config/influxdb.conf:/etc/influxdb/influxdb.conf:ro
- ./store/config/collectd.types.db:/opt/collectd/share/collectd/types.db:ro
ports:
# collectd port
- "25826:25826/udp"
visualize:
build: visualize
volumes:
- ./visualize/config/grafana-server:/etc/default/grafana-server:ro
- ./visualize/config/grafana.ini:/etc/grafana/grafana.ini:ro
- ./volumes/visualize/grafana.log:/var/log/grafana/grafana.log:rw
- ./volumes/visualize/grafana.sqlite3.db:/usr/share/grafana/data/grafana.db:rw
ports:
- "80:3000"
links:
- store
As you can see multiple volumes are defined that need to be prepared prior to the first build, otherwise docker-compose will create directories where files are required:
touch ./volumes/visualize/grafana.sqlite3.db touch ./volumes/visualize/grafana.log mkdir -p ./volumes/store
Now Dockerfiles need to be prepared:
# store/Dockerfile
FROM debian:latest
RUN apt-get update && apt-get install -y wget curl
RUN gpg \
--keyserver hkp://ha.pool.sks-keyservers.net \
--recv-keys 05CE15085FC09D18E99EFB22684A14CF2582E0C5
ENV INFLUXDB_VERSION 1.2.1
RUN wget -q https://dl.influxdata.com/influxdb/releases/influxdb_${INFLUXDB_VERSION}_amd64.deb.asc && \
wget -q https://dl.influxdata.com/influxdb/releases/influxdb_${INFLUXDB_VERSION}_amd64.deb && \
gpg --batch --verify influxdb_${INFLUXDB_VERSION}_amd64.deb.asc influxdb_${INFLUXDB_VERSION}_amd64.deb && \
dpkg -i influxdb_${INFLUXDB_VERSION}_amd64.deb && \
rm -f influxdb_${INFLUXDB_VERSION}_amd64.deb*
EXPOSE 8086
EXPOSE 25826
CMD ["influxd"]
# visualize/Dockerfile FROM debian:latest RUN apt-get update && apt-get -y install curl apt-transport-https ca-certificates RUN echo 'deb https://packagecloud.io/grafana/stable/debian/ jessie main' > /etc/apt/sources.list.d/grafana.list \ && curl https://packagecloud.io/gpg.key | apt-key add - \ && apt-get update && apt-get -y install grafana WORKDIR /usr/share/grafana CMD /usr/sbin/grafana-server
Run docker-compose build to create the images for the containers. This takes some time based on the hardware and internetuplink. We use the time to create the configuration.
store/config/collectd.types.db
absolute count:ABSOLUTE:0:U apache_bytes count:COUNTER:0:134217728 apache_connections count:GAUGE:0:65535 apache_idle_workers count:GAUGE:0:65535 apache_requests count:COUNTER:0:134217728 apache_scoreboard count:GAUGE:0:65535 arc_counts demand_data:COUNTER:0:U, demand_metadata:COUNTER:0:U, prefetch_data:COUNTER:0:U, prefetch_metadata:COUNTER:0:U arc_l2_bytes read:COUNTER:0:U, write:COUNTER:0:U arc_l2_size value:GAUGE:0:U arc_ratio value:GAUGE:0:U arc_size current:GAUGE:0:U, target:GAUGE:0:U, minlimit:GAUGE:0:U, maxlimit:GAUGE:0:U ath_nodes value:GAUGE:0:65535 ath_stat value:COUNTER:0:4294967295 bitrate value:GAUGE:0:4294967295 bytes value:GAUGE:0:U cache_ratio value:GAUGE:0:100 cache_result value:COUNTER:0:4294967295 cache_size value:GAUGE:0:4294967295 charge value:GAUGE:0:U compression uncompressed:COUNTER:0:U, compressed:COUNTER:0:U compression_ratio value:GAUGE:0:2 connections value:COUNTER:0:U conntrack entropy:GAUGE:0:4294967295 contextswitch contextswitches:DERIVE:0:U counter value:COUNTER:U:U cpufreq value:GAUGE:0:U cpu value:COUNTER:0:4294967295 current value:GAUGE:U:U delay seconds:GAUGE:-1000000:1000000 derive value:DERIVE:0:U df used:GAUGE:0:1125899906842623, free:GAUGE:0:1125899906842623 df_complex value:GAUGE:0:U df_inodes value:GAUGE:0:U disk_latency read:GAUGE:0:U, write:GAUGE:0:U disk_merged read:COUNTER:0:4294967295, write:COUNTER:0:4294967295 disk_octets read:COUNTER:0:17179869183, write:COUNTER:0:17179869183 disk_ops read:COUNTER:0:4294967295, write:COUNTER:0:4294967295 disk_ops_complex value:COUNTER:0:4294967296 disk_time read:COUNTER:0:1000000, write:COUNTER:0:1000000 dns_answer value:COUNTER:0:65535 dns_notify value:COUNTER:0:65535 dns_octets queries:COUNTER:0:125000000, responses:COUNTER:0:125000000 dns_opcode value:COUNTER:0:65535 dns_qtype_cached value:GAUGE:0:4294967295 dns_qtype value:COUNTER:0:65535 dns_query value:COUNTER:0:65535 dns_question value:COUNTER:0:65535 dns_rcode value:COUNTER:0:65535 dns_reject value:COUNTER:0:65535 dns_request value:COUNTER:0:65535 dns_resolver value:COUNTER:0:65535 dns_response value:COUNTER:0:65535 dns_transfer value:COUNTER:0:65535 dns_update value:COUNTER:0:65535 dns_zops value:COUNTER:0:65535 email_check value:GAUGE:0:U email_count value:GAUGE:0:U email_size value:GAUGE:0:U entropy entropy:GAUGE:0:4294967295 fanspeed value:GAUGE:0:U file_size bytes:GAUGE:0:U files value:GAUGE:0:U frequency frequency:GAUGE:0:U frequency_offset ppm:GAUGE:-1000000:1000000 fscache_stat value:COUNTER:0:4294967295 fork_rate value:DERIVE:0:U gauge value:GAUGE:U:U http_request_methods count:COUNTER:0:134217728 http_requests count:COUNTER:0:134217728 http_response_codes count:COUNTER:0:134217728 humidity value:GAUGE:0:100 if_collisions value:COUNTER:0:4294967295 if_dropped rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295 if_errors rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295 if_multicast value:COUNTER:0:4294967295 if_octets rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295 if_packets rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295 if_rx_errors value:COUNTER:0:4294967295 if_tx_errors value:COUNTER:0:4294967295 invocations value:DERIVE:0:U io_octets rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295 io_packets rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295 ipt_bytes value:COUNTER:0:134217728 ipt_packets value:COUNTER:0:134217728 irq value:COUNTER:U:65535 latency value:GAUGE:0:65535 links value:GAUGE:0:U load shortterm:GAUGE:0:100, midterm:GAUGE:0:100, longterm:GAUGE:0:100 memcached_command value:COUNTER:0:U memcached_connections value:GAUGE:0:U memcached_items value:GAUGE:0:U memcached_octets rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295 memcached_ops value:COUNTER:0:134217728 memory value:GAUGE:0:281474976710656 multimeter value:GAUGE:U:U mysql_commands value:COUNTER:0:U mysql_handler value:COUNTER:0:U mysql_locks value:COUNTER:0:U mysql_log_position value:COUNTER:0:4294967295 mysql_octets rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295 mysql_qcache hits:COUNTER:0:U, inserts:COUNTER:0:U, not_cached:COUNTER:0:U, lowmem_prunes:COUNTER:0:U, queries_in_cache:GAUGE:0:U mysql_threads running:GAUGE:0:U, connected:GAUGE:0:U, cached:GAUGE:0:U, created:COUNTER:0:U nfs_procedure value:COUNTER:0:4294967295 nginx_connections value:GAUGE:0:U nginx_requests value:COUNTER:0:134217728 node_octets rx:COUNTER:0:U, tx:COUNTER:0:U node_rssi value:GAUGE:0:255 node_stat value:COUNTER:0:4294967295 node_tx_rate value:GAUGE:0:127 operations value:COUNTER:0:4294967295 percent percent:GAUGE:0:100.1 pg_blks value:COUNTER:0:U pg_db_size value:GAUGE:0:U pg_n_tup_c value:COUNTER:0:U pg_n_tup_g value:GAUGE:0:U pg_numbackends value:GAUGE:0:U pg_scan value:COUNTER:0:U pg_xact value:COUNTER:0:U ping_droprate value:GAUGE:0:100 ping ping:GAUGE:0:65535 ping_stddev value:GAUGE:0:65535 players value:GAUGE:0:1000000 power value:GAUGE:0:U protocol_counter value:COUNTER:0:U ps_count processes:GAUGE:0:1000000, threads:GAUGE:0:1000000 ps_cputime user:COUNTER:0:16000000, syst:COUNTER:0:16000000 ps_pagefaults minflt:COUNTER:0:9223372036854775807, majflt:COUNTER:0:9223372036854775807 ps_disk_octets read:DERIVE:0:U, write:DERIVE:0:U ps_disk_ops read:DERIVE:0:U, write:DERIVE:0:U ps_rss value:GAUGE:0:9223372036854775807 ps_code value:GAUGE:0:9223372036854775807 ps_data value:GAUGE:0:9223372036854775807 ps_stacksize value:GAUGE:0:9223372036854775807 ps_state value:GAUGE:0:65535 ps_vm value:GAUGE:0:9223372036854775807 queue_length value:GAUGE:0:U response_time value:GAUGE:0:U records count:GAUGE:0:U route_etx value:GAUGE:0:U route_metric value:GAUGE:0:U routes value:GAUGE:0:U serial_octets rx:COUNTER:0:4294967295, tx:COUNTER:0:4294967295 signal_noise value:GAUGE:U:0 signal_power value:GAUGE:U:0 signal_quality value:GAUGE:0:U snr value:GAUGE:0:U spam_check value:GAUGE:0:U spam_score value:GAUGE:U:U swap_io value:DERIVE:0:1099511627776 swap value:GAUGE:0:1099511627776 tcp_connections value:GAUGE:0:4294967295 temperature value:GAUGE:-273.15:U threads value:GAUGE:0:U time_dispersion seconds:GAUGE:-1000000:1000000 timeleft timeleft:GAUGE:0:U time_offset seconds:GAUGE:-1000000:1000000 total_bytes value:DERIVE:0:U total_requests value:DERIVE:0:U total_time_in_ms value:DERIVE:0:U total_values value:DERIVE:0:U uptime value:GAUGE:0:4294967295 users users:GAUGE:0:65535 virt_cpu_total ns:COUNTER:0:256000000000 virt_vcpu ns:COUNTER:0:1000000000 vmpage_action value:COUNTER:0:4294967295 vmpage_faults minflt:COUNTER:0:9223372036854775807, majflt:COUNTER:0:9223372036854775807 vmpage_io in:COUNTER:0:4294967295, out:COUNTER:0:4294967295 vmpage_number value:GAUGE:0:4294967295 voltage_threshold value:GAUGE:U:U, threshold:GAUGE:U:U voltage value:GAUGE:U:U vs_memory value:GAUGE:0:9223372036854775807 vs_processes value:GAUGE:0:65535 vs_threads value:GAUGE:0:65535 stations value:GAUGE:0:256
store/config/influxdb.conf
[meta] dir = "/var/lib/influxdb/meta" [data] dir = "/var/lib/influxdb/data" engine = "tsm1" wal-dir = "/var/lib/influxdb/wal" [[collectd]] enabled = true port = 25826 database = "collectd_db" typesdb = "/opt/collectd/share/collectd/types.db"
visualize/config/grafana-server
GRAFANA_USER=grafana GRAFANA_GROUP=grafana GRAFANA_HOME=/usr/share/grafana LOG_DIR=/var/log/grafana DATA_DIR=/var/lib/grafana MAX_OPEN_FILES=10000 CONF_DIR=/etc/grafana CONF_FILE=/etc/grafana/grafana.ini RESTART_ON_UPGRADE=false PLUGINS_DIR=/var/lib/grafana/plugins
visualize/config/grafana.ini
##################### Grafana Configuration Example #####################
#
# Everything has defaults so you only need to uncomment things you want to
# change
# possible values : production, development
; app_mode = production
# instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty
; instance_name = ${HOSTNAME}
#################################### Paths ####################################
[paths]
# Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used)
#
;data = /var/lib/grafana
#
# Directory where grafana can store logs
#
;logs = /var/log/grafana
#
# Directory where grafana will automatically scan and look for plugins
#
;plugins = /var/lib/grafana/plugins
#
#################################### Server ####################################
[server]
# Protocol (http or https)
;protocol = http
# The ip address to bind to, empty will bind to all interfaces
;http_addr =
# The http port to use
;http_port = 3000
# The public facing domain name used to access grafana from a browser
;domain = localhost
# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false
# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
;root_url = http://localhost:3000
# Log web requests
;router_logging = false
# the path relative working path
;static_root_path = public
# enable gzip
;enable_gzip = false
# https certs & key file
;cert_file =
;cert_key =
#################################### Database ####################################
[database]
# You can configure the database connection by specifying type, host, name, user and password
# as seperate properties or as on string using the url propertie.
# Either "mysql", "postgres" or "sqlite3", it's your choice
;type = sqlite3
;host = 127.0.0.1:3306
;name = grafana
;user = root
# If the password contains # or ; you have to wrap it with trippel quotes. Ex """#password;"""
;password =
# Use either URL or the previous fields to configure the database
# Example: mysql://user:secret@host:port/database
;url =
# For "postgres" only, either "disable", "require" or "verify-full"
;ssl_mode = disable
# For "sqlite3" only, path relative to data_path setting
;path = grafana.db
#################################### Session ####################################
[session]
# Either "memory", "file", "redis", "mysql", "postgres", default is "file"
;provider = file
# Provider config options
# memory: not have any config yet
# file: session dir path, is relative to grafana data_path
# redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=grafana`
# mysql: go-sql-driver/mysql dsn config string, e.g. `user:password@tcp(127.0.0.1:3306)/database_name`
# postgres: user=a password=b host=localhost port=5432 dbname=c sslmode=disable
;provider_config = sessions
# Session cookie name
;cookie_name = grafana_sess
# If you use session in https only, default is false
;cookie_secure = false
# Session life time, default is 86400
;session_life_time = 86400
#################################### Analytics ####################################
[analytics]
# Server reporting, sends usage counters to stats.grafana.org every 24 hours.
# No ip addresses are being tracked, only simple counters to track
# running instances, dashboard and error counts. It is very helpful to us.
# Change this option to false to disable reporting.
;reporting_enabled = true
# Set to false to disable all checks to https://grafana.net
# for new vesions (grafana itself and plugins), check is used
# in some UI views to notify that grafana or plugin update exists
# This option does not cause any auto updates, nor send any information
# only a GET request to http://grafana.net to get latest versions
;check_for_updates = true
# Google Analytics universal tracking code, only enabled if you specify an id here
;google_analytics_ua_id =
#################################### Security ####################################
[security]
# default admin user, created on startup
;admin_user = admin
# default admin password, can be changed before first start of grafana, or in profile settings
;admin_password = admin
# used for signing
;secret_key = SW2YcwTIb9zpOOhoPsMm
# Auto-login remember days
;login_remember_days = 7
;cookie_username = grafana_user
;cookie_remember_name = grafana_remember
# disable gravatar profile images
;disable_gravatar = false
# data source proxy whitelist (ip_or_domain:port separated by spaces)
;data_source_proxy_whitelist =
[snapshots]
# snapshot sharing options
;external_enabled = true
;external_snapshot_url = https://snapshots-origin.raintank.io
;external_snapshot_name = Publish to snapshot.raintank.io
# remove expired snapshot
;snapshot_remove_expired = true
# remove snapshots after 90 days
;snapshot_TTL_days = 90
#################################### Users ####################################
[users]
# disable user signup / registration
;allow_sign_up = true
# Allow non admin users to create organizations
;allow_org_create = true
# Set to true to automatically assign new users to the default organization (id 1)
;auto_assign_org = true
# Default role new users will be automatically assigned (if disabled above is set to true)
;auto_assign_org_role = Viewer
# Background text for the user field on the login page
;login_hint = email or username
# Default UI theme ("dark" or "light")
;default_theme = dark
[auth]
# Set to true to disable (hide) the login form, useful if you use OAuth, defaults to false
;disable_login_form = false
#################################### Anonymous Auth ##########################
[auth.anonymous]
# enable anonymous access
;enabled = false
# specify organization name that should be used for unauthenticated users
;org_name = Main Org.
# specify role for unauthenticated users
;org_role = Viewer
#################################### Github Auth ##########################
[auth.github]
;enabled = false
;allow_sign_up = true
;client_id = some_id
;client_secret = some_secret
;scopes = user:email,read:org
;auth_url = https://github.com/login/oauth/authorize
;token_url = https://github.com/login/oauth/access_token
;api_url = https://api.github.com/user
;team_ids =
;allowed_organizations =
#################################### Google Auth ##########################
[auth.google]
;enabled = false
;allow_sign_up = true
;client_id = some_client_id
;client_secret = some_client_secret
;scopes = https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email
;auth_url = https://accounts.google.com/o/oauth2/auth
;token_url = https://accounts.google.com/o/oauth2/token
;api_url = https://www.googleapis.com/oauth2/v1/userinfo
;allowed_domains =
#################################### Generic OAuth ##########################
[auth.generic_oauth]
;enabled = false
;name = OAuth
;allow_sign_up = true
;client_id = some_id
;client_secret = some_secret
;scopes = user:email,read:org
;auth_url = https://foo.bar/login/oauth/authorize
;token_url = https://foo.bar/login/oauth/access_token
;api_url = https://foo.bar/user
;team_ids =
;allowed_organizations =
#################################### Grafana.net Auth ####################
[auth.grafananet]
;enabled = false
;allow_sign_up = true
;client_id = some_id
;client_secret = some_secret
;scopes = user:email
;allowed_organizations =
#################################### Auth Proxy ##########################
[auth.proxy]
;enabled = false
;header_name = X-WEBAUTH-USER
;header_property = username
;auto_sign_up = true
;ldap_sync_ttl = 60
;whitelist = 192.168.1.1, 192.168.2.1
#################################### Basic Auth ##########################
[auth.basic]
;enabled = true
#################################### Auth LDAP ##########################
[auth.ldap]
;enabled = false
;config_file = /etc/grafana/ldap.toml
;allow_sign_up = true
#################################### SMTP / Emailing ##########################
[smtp]
;enabled = false
;host = localhost:25
;user =
;password =
;cert_file =
;key_file =
;skip_verify = false
;from_address = admin@grafana.localhost
[emails]
;welcome_email_on_sign_up = false
#################################### Logging ##########################
[log]
# Either "console", "file", "syslog". Default is console and file
# Use space to separate multiple modes, e.g. "console file"
;mode = console file
# Either "trace", "debug", "info", "warn", "error", "critical", default is "info"
;level = info
# optional settings to set different levels for specific loggers. Ex filters = sqlstore:debug
;filters =
# For "console" mode only
[log.console]
;level =
# log line format, valid options are text, console and json
;format = console
# For "file" mode only
[log.file]
;level =
# log line format, valid options are text, console and json
;format = text
# This enables automated log rotate(switch of following options), default is true
;log_rotate = true
# Max line number of single file, default is 1000000
;max_lines = 1000000
# Max size shift of single file, default is 28 means 1 << 28, 256MB
;max_size_shift = 28
# Segment log daily, default is true
;daily_rotate = true
# Expired days of log file(delete after max days), default is 7
;max_days = 7
[log.syslog]
;level =
# log line format, valid options are text, console and json
;format = text
# Syslog network type and address. This can be udp, tcp, or unix. If left blank, the default unix endpoints will be used.
;network =
;address =
# Syslog facility. user, daemon and local0 through local7 are valid.
;facility =
# Syslog tag. By default, the process' argv[0] is used.
;tag =
#################################### AMQP Event Publisher ##########################
[event_publisher]
;enabled = false
;rabbitmq_url = amqp://localhost/
;exchange = grafana_events
;#################################### Dashboard JSON files ##########################
[dashboards.json]
;enabled = false
;path = /var/lib/grafana/dashboards
#################################### Alerting ######################################
[alerting]
# Makes it possible to turn off alert rule execution.
;execute_alerts = true
#################################### Internal Grafana Metrics ##########################
# Metrics available at HTTP API Url /api/metrics
[metrics]
# Disable / Enable internal metrics
;enabled = true
# Publish interval
;interval_seconds = 10
# Send internal metrics to Graphite
[metrics.graphite]
# Enable by setting the address setting (ex localhost:2003)
;address =
;prefix = prod.grafana.%(instance_name)s.
#################################### Internal Grafana Metrics ##########################
# Url used to to import dashboards directly from Grafana.net
[grafana_net]
;url = https://grafana.net
#################################### External image storage ##########################
[external_image_storage]
# Used for uploading images to public servers so they can be included in slack/email messages.
# you can choose between (s3, webdav)
;provider =
[external_image_storage.s3]
;bucket_url =
;access_key =
;secret_key =
[external_image_storage.webdav]
;url =
;username =
;password =
Now all images should be built and with the configuration in place docker-compose up -d will launch the store and the visualize servers.
The store needs to create the database before any values are stored:
docker exec -it datalizer_store_1 /bin/bash #> influx i> create database collectd_db
When a collectd host is configured to push to the datalizer, values should show up almost immediately (show series in influx). To debug, use tcpdump port 25826 to see if values are posted and investigate the logfiles.
Push Data
When the first values arrive in the store, all you can do is review the cpu, load or other technical values. Our goal is to collect environment values from various sensors as well. There are two kinds of data: live-values and historical values. Live values can be pushed every time they are measured while historical values come from some storage (eg a sd-card).
Live Data
To push live data from a network device, a simple curl-request with some parameters can be used to create the series:
curl -i -XPOST \
'http://datalizer:8086/write?db=${INFLUXDB_NAME}&rp=${RETENTION_POLICY}' \
--data-binary "${SERIES_NAME},${TAG_NAME}=${TAG_VALUE} value=${VALUE}"
The INFLUXDB_NAME is a database created for a specific purpose, the RETENTION_POLICY allows to push to that database and ensure the values are dropped after some time. The SERIES_NAME and TAG_NAME is used to differentiate the values when building graphs.
A practical example would post the a temperature of 10.0C of the soil in the north side of the garden to the database 'garden' that has a 'five_years' retention policy:
curl -i -XPOST \ 'http://datalizer:8086/write?db=garden&rp=five_years' \ --data-binary "temperature,type=soil,type_location=north value=10.0"
Historic Data
To push historic data, the values need to be transformed from their source-format (eg a csv file) to the influx line format. Depending on the type of source, the conversion-process should be automated so future pushes are a simple task. Lets use a imaginary CSV source from our power-meter and push the values to the influxdb.
1.2.2010 22:30:00 5000 1.2.2010 22:45:00 2000 1.2.2010 23:00:00 0
As you can see, the format can not 'just' be used, it needs conversion from a weird date (its the first of february right) to the influx nanoseconds since 1970-01-01T00:00:00.0000. (date -d “INPUT” +%s%N) Often a simple cat|sed|awk helps but sometimes a more sophisticated language is required to create the following output for the csv:
power_consumption,kitchen_kettle 5000 1265063400000000000 power_consumption,kitchen_kettle 2000 1265060700000000000 power_consumption,kitchen_kettle 0000 1265061600000000000
Up to 5000 entries at once can be written using the multiple points interface:
curl -i -XPOST 'http://datalizer:8086/write?db=electro' @filename
Considerations
Be aware that some of the values can have privacy implications. For example network traffic may allow you to analyze bottlenecks or rouge devices but may also be used to prove that someone was in the hackerspace downloading exactly the 1.1GB “operating system” that was illegal to download. Therefore each measurement should be considered for its implications and a retention policy should avoid having that data stored for a extended period. Read More on how to define Retention policies.
Discuss and publish the measurements with their policy in the community to prevent surprises.
To make the data useful for everyone, the dashboards from the visualize container need to be designed with the right units and useful related graphs at one display. The users should allow to setup public displays that show the current network throughput or the forecast weather compared to actual sensor data.