cookbook 'elk_forwarder', '= 3.0.2'
elk_forwarder (5) Versions 3.0.2 Follow0
Installs logstash-forwarder for use in an ELK Cluster
cookbook 'elk_forwarder', '= 3.0.2', :supermarket
knife supermarket install elk_forwarder
knife supermarket download elk_forwarder
elk_forwarder
Installs and configures the logstash-forwarder to forward specified logs to specified servers
Requirements
-
golang
cookbook
Attributes
You can also find comments in attributes/default.rb
General Attributes
Key | Type | Description | Default |
---|---|---|---|
['elk_forwarder']['install_type'] |
String | source or package | package |
['elk_forwarder']['config_dir'] |
String | Where to put config | /etc/logstash-forwarder |
['elk_forwarder']['service_name'] |
String | The Service Name | logstash-forwarder |
['elk_forwarder']['log_dir'] |
String | Directory to log to | /var/log/logstash-forwarder |
['elk_forwarder']['install_dir'] |
String | Directory to install to | /opt/logstash-forwarder |
['elk_forwarder']['daemon_args'] |
String | Extra args for the forwarder | -spool-size 5 |
['elk_forwarder']['syslog']['facility'] |
String | The Syslog facility. | local0 |
Config File Attributes
The node['elk_forwarder']['config']
hash closely mimics the logstash forwarder config file format, with the only difference of the files
key contains a hash instead of an array
The following table is namespaced under node['elk_forwarder']['config']
so prepend node['elk_forwarder']['config']
to the key column
Key | Type | Description | Default |
---|---|---|---|
['network']['servers'] | Array | An array of logstash agent address:port values | [] |
['network']['ssl certificate'] | String | The path to find the SSL Certificate | /etc/pki/tls/certs/logstash-forwarder/cert.pem |
['network']['ssl key'] | String | The path to find the SSL Private Key | /etc/pki/tls/certs/logstash-forwarder/key.pem |
['network']['ssl ca'] | String | The path to find the SSL CA Certificate | /etc/pki/tls/certs/logstash-forwarder/ca.pem |
['network']['timeout'] | String | Seconds to wait before connecting to next server | 15 |
['files'] | Hash | The List of files to track and associated fields to add | {} |
The node['elk_forwarder']['config']['files']
hash is probably the most useful,
check out the Usage section for instructions on how to configure files to forward
Usage
Thats great but how do I use it.
Point to your servers
Put this in a recipe, probably in your base cookbook
servers = search( :node, "role:elk_server AND chef_environment:#{node.chef_environment}" ).map do |node| "#{node['fqdn']}:5043" end node.set['elk_forwarder']['config']['network']['servers'] = servers
Grab your Lumberjack Certificate
This one is pretty much up to you, but there is a built in recipe for this
that pulls a certificate from a data bag item. I will use that as an example
assuming that you stored your CA certificate in the ca certificate
key in the
logstash
data bag item in the certificates
data bag
In your Attributes file
set['elk_forwarder']['certs']['ca_data_bag'] = 'certificates' set['elk_forwarder']['certs']['ca_data_bag_item'] = 'logstash' set['elk_forwarder']['certs']['ca_data_bag_item_key'] = 'ca certificate'
In your Recipe
include_recipe 'elk_forwarder::certs'
Configure files to forward
From your Attributes
set['elk_forwarder']['config']['files']['myapp']['paths'] = ['/var/log/myapp.log'] set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = 'myapp' set['elk_forwarder']['config']['files']['myapp']['fields']['foo'] = 'bar'
Or from your recipe. This one loops through a list of apps
apps.each do |app| node.set['elk_forwarder']['config']['files']['myapp']['paths'] = ["/var/log/#{app}.log"] node.set['elk_forwarder']['config']['files']['myapp']['fields']['type'] = 'rails_app' node.set['elk_forwarder']['config']['files']['myapp']['fields']['app'] = app end
As a side note all attribute modifications need to happen at compile time
Recipes
default
Installs and configures the logstash-forwarder
- Install logstash-forwarder using the
_source
or_package
recipes - Include various recipes for this cookbook:
elk_forwarder::configure
configure
Configures the forwarder with the ['elk_forwarder']['config']
hash
- Creates the config file:
#{node['elk_forwarder']['config_dir']}/logstash-forwarder.conf
certs
Installs SSL Certs and Keys from data bags to the paths specified in these atts:
node['elk_forwarder']['network']['ssl ca']
node['elk_forwarder']['network']['ssl certificate']
node['elk_forwarder']['network']['ssl key']
The following attributes are used to determine the location of the certs/keys
default['elk_forwarder']['cert_data_bag'] = 'certificates' default['elk_forwarder']['cert_data_bag_item'] = 'logstash'
The data bag you specify is expected to be in the following format:
{ "id": "elk_forwarder", "data": { "ca": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----", "certificate": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----\n" } }
Usage
Include this recipe in a wrapper cookbook:
metadata.rb
depends 'elk_forwarder', '~> 1.0'
recipes/your_recipe.rb
include_recipe 'elk_forwarder::default'
Certificates
Generating and distributing SSL Certificates is out of scope for this cookbook
unless you want to try out the certs recipe which just distributes certs from a
data bag. If you want to take care of certs yourself then your wrapper cookbook
will need to configure the SSL Certificates and Keys. See the
Certificate Notes
on the logstash forwarder repo for help.
Remember to set these attributes to the filenames where you store the certs:
node['elk_forwarder']['network']['ssl ca']
-
node['elk_forwarder']['network']['ssl certificate']
(optional) -
node['elk_forwarder']['network']['ssl key']
(optional)
Contributing
- Fork the repository on Github
- Create a named feature branch (like
add_component_x
) - Write your change
- Write tests for your change (if applicable)
- Run the tests with
kitchen test
, ensuring they all pass - Submit a Pull Request using Github
License and Authors
Author:: EverTrue, inc. (devops@evertrue.com)
Change Log for elk_forwarder
All notable changes to this project will be documented in this file.
This project adheres to Semantic Versioning.
Unreleased
Added
Changed
Removed
[3.0.2] - 2015-11-05
Fixed
- Build path to be in file_cache_path
[3.0.1] - 2015-08-07
Fixed
- Fix certificate miscommunication
[3.0.0] - 2015-08-07
Changed
- Massive refactoring (much of attribute names changed)
- Change test environment name from
_default
todev
- Stop testing on Ubuntu 12.04
[2.0.0] - 2015-06-16
Changed
- Breaking: All Configuration to just be through attributes, no LWRP unfortunately
- Breaking: The
['config']['files']
attribute has been changed from an array to a hash - Added a mocking mode for running tests
Removed
- The
logstash_forwarder_log
LWRP as it was causing problematic race conditions
1.0.0 - 2015-04-27
Added
- Initial Release
Foodcritic Metric
3.0.2 passed this metric
3.0.2 passed this metric