I'm glad to see docker stabilizing but it's very disappointing to not see any changes addressing the logging support that was promised in 1.0 (https://news.ycombinator.com/item?id=7712138 reply by shykes). The infinitely increasing-in-size and unconfigurable logfiles generated by containers in docker are basically a dealbreaker unless you bake your own logging solution into each container and avoid stdin/stdout.
Yeah, we spent some time with logging providers to get feedback on the design, and had to go back to the drawing board a little bit.
We're going to show the result of that tomorrow :)
We decided to ship 1.0 anyway just for the sheer volume of bugfixes. But don't worry we are still on a monthly release schedule. We are far from finished shipping stuff!
This sounds like a good fit for LoggerFS[1] - if at all possible I'd try to avoid logs being local to a given host, and instead pipe them off to a central log processing host.
I've found starting docker containers using something like systemd works really well. Basically you do something like this:
# systemd unit file
Description=My Service
After=docker.service
Requires=docker.service
[Service]
ExecStart=/usr/bin/docker run username/app
Notice I'm not using the '-d' flag in the docker run command. That forces all output to stdout/stderr, which will be routed to the systemd journal. Sadly this only works for apps that can log to the console instead of files.
I wonder if a very simple solution would be enough: with a one-line change, the logs could be created as named pipes instead of files. Then you could just write some upstart scripts/systemd units that do
cat /run/docker/mypipe >/dev/log
And you'd be good.
---
I'm not sure this is the right approach, though. In my opinion, docker shouldn't log at all—containers are unix processes, and Unix processes log to ephemeral stream-sink endpoints (think AWS SNS topics) called stdout and stderr. To collect and persist those logs, the Unix Way is to use shell redirection to "attach" those endpoints to subscribers.
Now, in Docker's daemonized run mode, these pipes are just hanging loose. But docker provides the "docker attach" command precisely so you can reattach them to something (or as many somethings) as you like. All you have to do if to get sensible logging if you're using e.g. Upstart is to create a service that runs "docker attach <mycontainer>": the logs will flow from the container's stdout/stderr to the service's stdout/stderr, and Upstart will catch them and handle them sensibly from there.
So, having done something like that, the important bit would actually be making sure docker doesn't do its own broken logging at all, except in the case of test containers that will never be attached to. Actually, this also cuts across the use (or lack thereof) of the --rm flag: it'd make sense for containers started with --rm to not log. (The generalization of the --rm semantics would seem to put such containers in the same category as e.g. EC2 instance-store instances.)
In general you ought to just be able to "docker run" in the foreground from your systemd script, and then stdin/stdout will go into the journal where you can handle things like any other file. This works now, except that the logs are also duplicated into /var/lib/docker.
FWIW, I've found running supervisord for all containerized apps with a log redirect to a host-mounted path to be a perfectly reasonable solution.
I would hate to see docker go down a path where it folds in its own logging "framework", which, I feel, would be going too far (poor separation of concerns, etc).
Well from a 12-factor app perspective, having each container dump its output to stderr/stdout is basically perfect as long as docker has a pluggable mechanism for dealing with that output. The "docker attach" API is all that you need, since you can attach prior to starting the container. Then you just need a way to tell docker not to save the logs via some "--disable-logging" option to "docker run" and "docker start".
Using a log shipper is probably the best workaround in this scenario
1.) logrotate at the end of the day
2.) have your logshipper watch docker log folder for each container
3) Log shipper ( or collector) ships files as they are updated to a central server and you are free to archive or delete as you wish.
Many of the central logging systems can detect rolled logs, so this set up is not much of a stretch.
I do agree more configurable logging is a bit of an oversight, especially for something like Docker.
Is a linux tool like logrotate acceptable here? Wouldn't this be able to rotate the std-out infinitely increasing log files? Yes this isn't built in to docker, but does seem like a reasonable solution.
I've never seen a recipe documenting whether docker can handle a logrotate. Most daemons require you to HUP them after rotation. In general logrotate is a complete hack in the way it interacts with daemons.
There are docker commands (docker logs) that are dependent on this format. The primary issue is that if you deal with logs yourself via attaching to the container and piping them somewhere, the logs are still duplicated to these json files. It's unexpected, a waste of space and insecure.
That being said, logrotate would sort of work with caveats since the json files are one entry per line last I checked.
