Logpeck - A Simple, RESTful Log Collector

Objectives

Logpeck aims to be an easy-to-use module that parsing and collecting contents from log file and posting into specific storage system, such as ElasticSearch, Influxdb, Kafka. We want to control collection tasks remotely with HTTP API (NONE configuration file).

Getting Started

Installation

From Binary (linux only)

• Download installation package logpeck_0.5.0.deb
• Run sudo dpkg -i logpeck_0.5.0.deb
• Run sudo service logpeck start (or sudo supervisorctl update if supervisor is avalible)

From Source Code

• Download source code: Release page v0.5.0
• Build: go build cmd/logpeckd/logpeckd.go
• Launch: ./logpeckd -config logpeckd.conf
• We can also use supervisor or other service management software to manage logpeck process.

Web UI

We highly recommend to install logpeck-kibana-plugin into Kibana. With this plugin, we can control all machines and collection tasks conveniently. At the same time, we can take advantage of powerful searching and visualization features of Kibana.

RESTful API

We can also control collection tasks with RESTful API. See more

Documentation

• Task Configuration
• Frequently Asked Questions

Dependencies

• BurntSushi/toml: configuration management
• Sirupsen/logrus: logging
• bitly/go-simplejson: json parser
• yuin/gopher-lua: lua virtual machine
• boltdb/bolt: local storage
• go-zoo/bone: http multiplexer
• hpcloud/tail: watching log file
• Shopify/sarama: kafka client

Saulte to all these excellent projects.

Discussion

Any suggestions or questions, please create an issue to feedback.