Logging

Due to a limitation on the docker engines logging implementation, lcp-compose is required to jump through few hoops to make sure that log files are captured to file-system as well as shown on docker logs command output.

Details

If the default docker logging driver is switched from json-file to anything else other than journald, the docker logs command stops working. Since lcp-compose is meant to be a development tool, it is essential to have quick and reliable access to container's standard output. The straight forward solution to this problem would be to simply switch to the journald driver and use systemd to redirect logs to specific destinations. However, systemd is yet to see widespread adoption and making it a hard requirement would make lcp-compose inoperable in many systems (including TeamCity build agents).

Solution

Introduce logspout, a containerized program that is designed to solve this specific problem. Logspout maps the docker socket and uses the docker engine APIs to tail logs from each running container. These logs streams can then be directly forwarded to a syslog destination.

Architecture

+------------+
|   orders   |
+-----+------+                                       +-------------------+
      |                         +--------------------+  splunkforwarder  |
      |                         |                    +----------+--------+
+-----v------+                  v                               |
|  logspout  |           /path/to/log/dir/                      |
+-----+------+                  ^                               |
      |                         |                    +----------v--------+
      |                         |                    |       splunk      |
+-----v------+                  |                    +-------------------+
|   syslog   +------------------+                     http://splunk:8000/
+------------+

Logspout appends syslog Structured-Data tags to identify the source container ID and service name.
Syslog (rsyslogd) inspects the Structured-Data tags in incoming log lines to create corresponding log files with the format <service>/<log_type>-<container_id>.log in the configured logging directory.
Splunkforwarder monitors the logging directory for files and forwards them a containerized Splunk instance to be indexed.

Unattended/Non-Splunk Usage

For unattended builds or lcp-compose usage without splunk services, the user can preserve the log directory for manual inspection/tailing.

Limitations

There is no distinction between stderr and stdout.
Services which mount the logs folder (that do not log to stdout) must volume map /path/to/lcpcompose/logging_dir/<service> to the containers /syslog-logs path.

Configuration

Syslog configuration and Splunk configurations are stored in lcp-compose and can be inspected at .lcp-compose/configuration/ under the directories "splunk", "splunkforwarder", and "syslog".