Logs are great. They are easy when you’re running on one or two boxes. However, they get harder when you’re running in a modest distributed environment and even more so if you’ve embraced micro-services and containers. I’ve compiled some notes and thoughts on getting Logentries running in a Kubernetes environment.
At Fairwinds, we useKubernetesfor our clients. Centralized logs are critical for them to understand their environments and for us to assist in troubleshooting. Sometimes, a client will have existing logging tools or services they would like us to use. This is the case with my current client, who utilizesLogentries.
Getting it running in Kubernetes was not very difficult, but some things were learned. Since Kubernetes is all about containers, it makes sense that a log collector for Logentries should also run as a container. The folks at Logentries clearly feel the same way and have made thelogentries/docker-logentriesimage available on docker hub. The instructions discuss how to run in a container, but it is limited to straightdockerenvironment. Kubernetes is based on docker, but you do run things differently.
Getting Logentries Up and Running
I run the Logentries collector as aDaemonSetin its ownServiceAccountunder thekube-systemnamespace. The reason for running it asDaemonSetis to ensure we run one of these containers on every node. A dedicatedServiceAccountisn’t necessary, but helps to isolate things. You can also use a different namespace if you’d rather not add tokube-system, but it fits pretty well since this is for logging everything.
Logentries allows you to send logs to different locations named Log Sets. In order to get logs to show up in Log Sets, you will need to create the Log Set and then an access token. Since this is a secret, we’ll store it as such. Here is an examplesecret.ymlto create thelogentries_tokenin Kubernetes:
Which you apply withkubectl apply -f daemonset.yml.
The above should work if you have a standard type setup, similar to the one you get from kops. If you’ve done custom installation work, you may need to adjust some things, like the location of the socket etc.
The above is the default configuration which will send container logs, container statistics, and docker events to Logentries. Should you wish to alter the logging that happens via the flags, I’ve added them in commented fashion. Details on those flags can be found ondocker hub: logentries/docker-logentries/.
In general, it works as advertised. Once theDaemonSetis running, the logs will flow. Memory utilization of each container or Pod is under 100MB and CPU consumption has been modest thus far.
The logs produced by the containers are in JSON form and the-jswitch makes sure that things pass through properly. The JSON that arrives appears clean and fully functional.
The first thing I noticed was that it can be tricky to find the logs for a singleDeployment. For example, depending on how your images or containers are named, you will likely need to use regular expressions. The simple search will not find substrings. If we have a name ofpod-staging_nginx-workersearching fornginx, it will not return results, butname=/.*nginx.*/will.
Now that we have logs for only the pod or container we’re interested in, there is the issue of all the stats that are also part of the found results. To reduce the results to only logs from the container itself, you can add another clause:AND line. Every line logged will be stored in a JSON attribute namedline,thus forcing the search to also find only those items gets us what we need:name=/.*nginx.*/ AND line.
Another issue still in search of an answer is the use case where you’d like a Log Set forapplication Xandapplication Y. With a classic Logentries collector, which accesses the log files directly, you can configure which Log Set the logs from a given file will go. In a Kubernetes environment, where the logs are harvested via a docker API, that is not possible. Unfortunately, Logentries does not offer a way to slice and tag once the logs are received from docker. One possible option would be to run a Logentries container alongside each pod and utilize--matchByNameor--matchByImageand--no-statsand--no-dockerEvents. Clearly, that will lead to a large number logging containers. I will be investigating this a bit more, but the default is likely the only reasonable approach for now.
In conclusion, Logentries is perfectly usable and easy to set up with Kubernetes.