Managing Java logs with logstash and Kibana

Logstash is a tool to analyze and manage logs. You can collect, parse, store and forward log entries. The data is stored in elasticsearch. You can then analyse, visualise and search through the logs using Kibana. To make it easy for you to try, the basic logstash distribution combines all three tools to make it easy for you to install.

The main strength of logstash compared to many other log management solution is the ability to dissect your log lines and split in in the field, allowing you to search and correlate logs better.

For example, a log line like

~myApp 2013-12-06T17:43:04.234+0100 [0.0.0.0-http-10.32.92.147-8080-3] INFO  b.v.a.d.l.MyClass - logging something

can be split in the application, timestamp, server, thread, severity, class and message.

Also combinations of lines can be recognized allowing stack traces or other multi-line log messages to not be split. For example the following log entry for a REST call.

~dc-devt 2013-12-06T17:43:04.234+0100 [0.0.0.0-http-10.32.92.147-8080-3] INFO  b.v.a.d.l.PreProcessLoggingInterceptor - 
Service: GET http://10.32.92.147:8080/appContext/rest/service
UserId: itsmeagain
Response types application/json
Query Parameters: 
	limit -> [10]
	sortColumn -> [number]
	start -> [0]
Path parameters: 
Reply type: class myapp.PagedList
Output document:
{...contents snipped...}
Duration: 0.078s

To make this possible, you have to match your logging format with your logstash configuration.
I use the following logback format configuration:

<pattern>~%property{log.appkey} %date{yyyy-MM-dd'T'HH:mm:ss.SSSZ} [%property{log.server}-%thread] %-5level %logger{36} - %msg%n%xEx</pattern>

The resulting log lines are quite long as they contain additional information to allow aggregating logs from many applications and servers on one logstash server, they are not really meant for direct consumption.

In logstash, you can configure how to process the input. Let’s look at the configuration matching the format above.
Let’s start by defining the log sources.

input {
  file {
    type => "syslog"
    path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
  }
 
  file {
    type => "logfile"
    path => [ "/home/joachim/temp/*.log", "/home/joachim/temp/logs/*.log" ]
  }
}

In this case just define the files which need to be scanned. It is very practical that this supports wildcards.
I did a test using UDP to send an receive messages. This is crazy fast (as expected) but unfortunately the most of the messages never reached logstash (I expect they were dropped while logstash was still handling a previous message).

The output is sent to elasticsearch for storage. This will allow Kibana to access the data so you can visualize the result.

output {
  elasticsearch 
  {
    embedded => true
  }
}

Now we need to add the filtering of the log messages. First I want to combine the lines which are part of the same log message. To allow us to figure out which lines that belong together, I included a tilde as marker to indicate that a new log message starts. The filter basically says that all lines which do not start with a tilde need to be appended to the previous line.

filter {
 
  multiline {
    pattern => "^[\[~]"
    negate => true
    what => "previous"
  }
 
  # add other filters
}

We already defined the filter block, so additional filters need to be added inside that block.

We need to disect the log fields according to the pattern we configured.

grok {
  match => [ 
    "message", "~%{NOTSPACE:application} %{TIMESTAMP_ISO8601:timestamp} \[%{DATA:server}\-%{DATA:thread}\] %{LOGLEVEL:severity}\s+%{JAVAFILE:category} \- %{GREEDYDATA:shortmessage}"
  ]
}

I also add some rules to extract additional fields when present. This extract data from specific log messages like the logging of the REST calls above.

grok {
    match => [ "message", "Duration: %{NUMBER:duur:float}s" ]
    tag_on_failure => [] 
  }
 
  grok {
    match => [ "message", "UserId: %{WORD:ldapid}" ]
    tag_on_failure => [] 
  }
 
  grok {
    match => [ "message", "Service: %{WORD:http_command} %{URI:endpoint}" ]
    tag_on_failure => [] 
  }

Finally, make sure the timestamp from the message is used as timestamp of the log entry (otherwise the time of processing the message is used).

date {
  # 2013-09-23T11:27:14.177+0200
  match => [
      "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
  ]
}

With these settings and some logs in which are processed, the logs can be investigated and interesting things can be done.

The dashboard which is included in the logstash distribution, Kibana allows you to customize the view on the log data in your elasticsearch database.
You can customize Kibana by adding several dashboards. In each dashboard you can add rows and for each row you can add additional panels which display the information in a specific way. To access the data, you can define filters and query and these can be combined. For example the following two rows in the dashboard show (based on the configuration above) an overview of events, an overview of logged errors and an idea of the number of total, mean and maximum duration of the REST calls.

logstash-graphics

When viewing the details of the events you can choose the columns which are visible and by clicking on the line see the full details of a log entry.

logstash-events

There is also a view of the fields (used to select the event column) which also allow quick summaries to be displayed.

logstash-fields

17 Comments

  1. What you wrote was actually very logical. However,
    think on this, suppose you wrote a catchier post title?
    I am not suggesting your information is not solid, but what if you added a post title that makes people desire more?
    I mean Managing Java logs with logstash and Kibana trying to solve IT problems is kinda vanilla.

    You might glance at Yahoo’s front page and watch how they create news
    titles to get viewers interested. You might add a video or a pic or two to get readers
    interested about what you’ve got to say. In my opinion, it might make your posts a little bit more interesting.

  2. Thanks for ones marvelous posting! I actually enjoyed reading it, you may be a great author.I
    will ensure that I bookmark your blog and may come back later in life.
    I want to encourage you to definitely continue your great job, have a nice holiday weekend!

  3. strongly says:

    Excellent blog here! Also your site loads up fast! What
    host are you using? Can I get your affiliate link to your host?
    I wish my website loaded up as quickly as yours lol

    my website … strongly

  4. Ӏ all tɦee time used to read piece off writing in newqs papoers but now as I
    amm a user of net thus rom now I am using net for articles, tɦanks to web.

  5. I came across your site a week ago as well as started to follow your site content regularly.
    I haven’t left a comment on any sort of blog site just yet but I was
    considering to start soon. It’s actually exciting to
    really contribute to an article even if it’s only a blog.
    I really liked going through a couple of your articles.
    Fantastic articles no doubt. I’ll keep visiting your blog often. I learned
    lots from you. Many thanks!

    Here is my website – Best Acne Treatment (bestnaturalacnetreatment.yolasite.Com)

  6. gros nichons says:

    Encorе un très bon poste : j’en parlerai dans la
    semaine аvec certains dde mes voisins

  7. Hi there I am so grateful I found your webpage, I really found you by accident, while I was looking on Bing for something else, Anyhow
    I am here now and would just like to say kudos for a fantastic post and a all round exciting blog
    (I also love the theme/design), I don’t have time to read through it all at the moment
    but I have bookmarked it and also included your RSS feeds,
    so when I have time I will be back to read more, Please do keep up the awesome b.

    my page: new diamond earrings

  8. An impressive share! I’ve just forwarded
    this onto a friend who was doing a little research on this.
    And hhe actually bought me lunch because I discovered
    it for him… lol. So allow me to reword this….

    Thanks for the meal!! But yeah, thanks for spending the time to talk about this subject here onn your internbet
    site.

  9. Enhorabuena! y mil gracias!

    He de pronunciar que tu entrada en el blog me ha sido efectivamente productivo!

    Termometro de cocina

  10. donzie says:

    Is there any way to strip the timestamp off the from of the message when it is displayed since it is already contained in the @timestamp column? I find it difficult visually scan logs for important message when all I see is timestamp information in the message column.

  11. donzie says:

    Also, thank you for you post!!

  12. Excellent items from you, man. I have have in mind your
    stuff previous to and you’re just extremely fantastic.
    I actually like what you’ve bought here, really like what you’re saying and the best way through which you say it.
    You’re making it enjoyable and you still care for to stay it wise.
    I can not wait to read far more from you. That is really a wonderful web site.

  13. I was curious if you ever thought of changing the
    layout of your site? Its very well written; I love what youve got to say.
    But maybe you could a little more in the way of content so people could connect with it better.
    Youve got an awful lot of text for only having one or 2 images.
    Maybe you could space it out better?

  14. bassura city says:

    What’s up, jսst աanted to tell yoս, Ι enjoyed thiѕ blog post.
    It wwas helpful. Ҡeep on posting!

    Visit my website bassura city

  15. Take 3,000 to 8,000 milligrams daily in divided doses. They have
    to know the side effects what they are getting from smoking and
    also they have to know the way of quitting smoking.
    Be sure you anticipate exposure during outdoor activities (mowing the lawn, raking leaves).

  16. I all tҺе time emailed this website post page to all
    my contacts, ѕince іf like to read іt then my contacts will
    toо.

    My website :: judi online (http://taruhan88.com)

  17. Thank you for some other informative web site. Where else may I am getting that type of info written in such
    a perfect means? I have a challenge that I am simply now operating on, and I have
    been at the glance out for such info.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

question razz sad evil exclaim smile redface biggrin surprised eek confused cool lol mad twisted rolleyes wink idea arrow neutral cry mrgreen

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>