Author Archives: justme

Building an executable jar with embedded web server

To make using and deploying our application easier, I want it to be easy to run the application. Instead of requiring the installation of a servlet server like Tomcat, Jetty or a full blown application server, I aim to use an embedded server.

The end result is the ability to run our application using the following command:

java -jar application.jar

Possibly using some additional parameters like

java -DserverPort=8080 -DshutdownPort=8089 -jar application.jar

As web server I chose Undertow, a relatively new server which is also used in WildFly.

I want the result jar to still show the included libraries, so I do not want to make an uber-jar using something like the maven shade plugin. I rather want to build a jar which includes jars with all the dependencies.

Building this jar can be done using the maven assembly plugin, using a configuration like the following:

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.3 http://maven.apache.org/xsd/assembly-1.1.3.xsd">
    <id>with-deps</id>
    <formats>
        <format>jar</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <fileSets>
        <fileSet>
            <outputDirectory>/</outputDirectory>
            <directory>${basedir}/target/classes</directory>
            <excludes>
                <exclude>.PLACEHOLDER.txt</exclude>
            </excludes>
        </fileSet>
    </fileSets>
    <dependencySets>
        <dependencySet>
            <excludes>
                <exclude>ggg:aaa-exe</exclude>
            </excludes>
            <outputDirectory>/lib</outputDirectory>
            <useProjectArtifact>true</useProjectArtifact>
            <unpack>false</unpack>
            <scope>runtime</scope>
        </dependencySet>
    </dependencySets>
</assembly>

I use a separate module to build the jar. In this module the main pom mainly contains all the application dependencies and the following configuration:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
 
    <parent>
        <groupId>ggg</groupId>
        <artifactId>aaa</artifactId>
        <version>vvv-SNAPSHOT</version>
    </parent>
 
    <artifactId>aaa-exe</artifactId>
    <packaging>jar</packaging>
 
    <name>Build executable jar</name>
 
    <dependencies>
        <!-- module dependencies which build the application -->
    </dependencies>
 
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.5.4</version>
                <configuration>
                    <descriptors>
                        <descriptor>${basedir}/src/assembly/include-deps.xml</descriptor>
                    </descriptors>
                    <archive>
                        <manifest>
                            <mainClass>myapp.WarRunner</mainClass>
                            <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
                        </manifest>
                        <manifestEntries>
                            <Implementation-Version>${project.version}</Implementation-Version>
                        </manifestEntries>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id> <!-- this is used for inheritance merges -->
                        <phase>package</phase> <!-- bind to the packaging phase -->
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

This module will now build two jars, a normal jar (very small) and one with the dependencies which has the “with-deps” suffix. The latter can be used to start the application.

In the module itself we need only two classes.

We need a main class to start the application (the “mainClass” which is referenced in the manifest). This should start the application. To assure that the embedded jars are used, we need a special classloader. Apart from that, the main method refers to a different class which runs the application.

public final class WarRunner {
 
    private WarRunner() {
        // hide constructor
    }
 
    /**
     * Run de "real" runner from the application.
     *
     * @param args command line parameters
     */
    public static void main(String[] args) {
        try {
            JarClassLoader.invokeMain("myapp.AppRunner", args);
        } catch (Throwable e) {
            System.err.println("Cannot run myapp.AppRunner: " + e.getMessage());
            e.printStackTrace();
        }
    }
 
}

The second class is the JarClassLoader. This is based on the code from embedded jar classloader, but with some fixes.

public class JarClassLoader extends URLClassLoader {
 
    private static boolean isJar(String fileName) {
        return fileName != null && fileName.toLowerCase().endsWith(".jar");
    }
 
    private static File jarEntryAsFile(JarFile jarFile, JarEntry jarEntry) throws IOException {
        String name = jarEntry.getName().replace('/', '_');
        int i = name.lastIndexOf(".");
        String extension = i > -1 ? name.substring(i) : "";
        File file = File.createTempFile(name.substring(0, name.length() - extension.length()) + ".", extension);
        file.deleteOnExit();
        try (InputStream input = jarFile.getInputStream(jarEntry)) {
            try (OutputStream output = new FileOutputStream(file)) {
                int readCount;
                byte[] buffer = new byte[4096];
                while ((readCount = input.read(buffer)) != -1) {
                    output.write(buffer, 0, readCount);
                }
                return file;
            }
        } 
    }
 
    /**
     * Build classpath with extra jars in it.
     * @param urls urls
     * @param parent parent class loader
     */
    public JarClassLoader(URL[] urls, ClassLoader parent) {
        super(urls, parent);
        try {
            ProtectionDomain protectionDomain = getClass().getProtectionDomain();
            CodeSource codeSource = protectionDomain.getCodeSource();
            URL rootJarUrl = codeSource.getLocation();
            String rootJarName = rootJarUrl.getFile();
            if (isJar(rootJarName)) {
                addJarResource(new File(rootJarUrl.getPath()));
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
 
    private void addJarResource(File file) throws IOException {
        JarFile jarFile = new JarFile(file);
        addURL(file.toURL());
        Enumeration<JarEntry> jarEntries = jarFile.entries();
        while (jarEntries.hasMoreElements()) {
            JarEntry jarEntry = jarEntries.nextElement();
            if (!jarEntry.isDirectory() && isJar(jarEntry.getName())) {
                addJarResource(jarEntryAsFile(jarFile, jarEntry));
            }
        }
    }
 
    @Override
    public synchronized Class<?> loadClass(String name) throws ClassNotFoundException {
        try {
            Class<?> clazz = findLoadedClass(name);
            if (clazz == null) {
                clazz = findClass(name);
            }
            return clazz;
        } catch (ClassNotFoundException e) {
            return super.loadClass(name);
        }
    }
 
    /**
     * Invoke main method in given class using this classloader.
     * @param name class to invoke
     * @param args command line arguments
     * @throws ClassNotFoundException oops
     * @throws NoSuchMethodException oops
     * @throws InvocationTargetException oops
     */
    public static void invokeMain(String name, String[] args) throws ClassNotFoundException, NoSuchMethodException, InvocationTargetException {
        ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();
        try {
            final String mainMethodName = "main";
            JarClassLoader loader = new JarClassLoader(((URLClassLoader) ClassLoader.getSystemClassLoader()).getURLs(), contextClassLoader);
            Thread.currentThread().setContextClassLoader(loader); // replace contextClassloader
            Class<?> clazz = loader.loadClass(name);
            Method method = clazz.getMethod(mainMethodName, String[].class);
            method.setAccessible(true);
            int mods = method.getModifiers();
            if (method.getReturnType() != void.class || !Modifier.isStatic(mods) || !Modifier.isPublic(mods)) {
                throw new NoSuchMethodException(mainMethodName);
            }
            try {
                method.invoke(null, (Object) args); // Crazy cast "(Object)args" because param is: "Object... args"
            } catch (IllegalAccessException e) {
                // This should not happen, as we have disabled access checks
                System.err.println("Probleem during JarClassLoader.invokeMain: " + e.getMessage());
                e.printStackTrace();
            }
        } finally {
            Thread.currentThread().setContextClassLoader(contextClassLoader);
        }
    }
 
}

What remains to be done is write the AppRunner class (which should be in the module with the application code):

public final class AppRunner {
 
    private static final Logger LOG = LoggerFactory.getLogger(AppRunner.class);
 
    private static final String CONTEXT_PATH = "/app";
    private static final int SERVER_PORT = 8080;
    private static final int SHUTDOWN_PORT = 8089;
    private static final int SHUTDOWN_WAIT = 5;
 
    private static GracefulShutdownHandler shutdownHandler;
    private static Undertow server;
 
    private AppRunner() {
        // hide constructor
    }
 
    /**
     * Main method to start the application
     *
     * @param args command line parameters - not used at the moment - configuration through properties
     */
    public static void main(final String[] args) {
        int serverPort = getProperty("km.server.port").orElse(SERVER_PORT);
        int shutdownPort = getProperty("km.shutdown.port").orElse(SHUTDOWN_PORT);
        int shutdownWaitSeconds = getProperty("km.shutdown.wait").orElse(SHUTDOWN_WAIT);
 
        try {
            ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
            DeploymentInfo servletBuilder = Servlets.deployment()
                    .setClassLoader(AppRunner.class.getClassLoader())
                    .setContextPath(CONTEXT_PATH)
                    .setDeploymentName("app.war")
                    .addInitParameter("contextConfigLocation", "classpath:app-web-applicationContext.xml")
                    .addInitParameter("resteasy.logger.type", "SLF4J")
                    .addInitParameter("resteasy.wider.request.matching", "true")
                    .addWelcomePage("index.html")
                    .setDefaultSessionTimeout(60)
                    .addListener(new ListenerInfo(RequestContextListener.class))
                    .addListener(new ListenerInfo(ResteasyBootstrap.class))
                    .addListener(new ListenerInfo(SpringContextLoaderListener.class))
                    .addServlets(Servlets.servlet("RESTEasy", HttpServletDispatcher.class)
                            .addMapping("/rest/*"))
                    .setResourceManager(new ClassPathResourceManager(classLoader, "web"));
 
            DeploymentManager manager = Servlets.defaultContainer().addDeployment(servletBuilder);
            manager.deploy();
 
            HttpHandler servletHandler = manager.start();
            PathHandler path = Handlers.path(Handlers.redirect(CONTEXT_PATH))
                    .addPrefixPath(CONTEXT_PATH, servletHandler);
            shutdownHandler = Handlers.gracefulShutdown(path);
            server = Undertow.builder()
                    .addHttpListener(serverPort, "localhost")
                    .addHttpListener(shutdownPort, "localhost", exchange -> {
                        exchange.getResponseHeaders().put(Headers.CONTENT_TYPE, "text/plain");
                        exchange.getResponseSender().send(
                                String.format("Going to shutdown when all requests have ended or %s seconds, whichever occurs first.", shutdownWaitSeconds));
                        new Thread(() -> {
                            try {
                                shutdownHandler.shutdown();
                                shutdownHandler.awaitShutdown(shutdownWaitSeconds * 1000);
                            } catch (InterruptedException ie) {
                                LOG.warn("Wait for undertow requests to end was interrupted.", ie);
                            }
                            server.stop();
                            LOG.info("Gracefully shut down.");
                            System.exit(0);
                        }).run();
                    })
                    .setHandler(shutdownHandler)
                    .build();
            server.start();
 
        }  catch (ServletException e) {
            LOG.error("Kan servlet niet starten.", e);
        }
    }
 
    private static Optional<Integer> getProperty(String propertyName) {
        String propertyValue = System.getProperty("km.server.port");
        if (StringUtils.isNotBlank(propertyValue)) {
            try {
                return Optional.of(Integer.parseInt(propertyValue));
            } catch (NumberFormatException nfe) {
                LOG.error(String.format("Cannot parse property %s with value %s to a number, ignoring value.", propertyName, propertyValue));
            }
        }
        return Optional.empty();
    }
 
}

This code starts creates a servlet. It says that the resources to be served can be found in the “web” package (as there is no webapp (folder) you have to give the explicit location.
It also registers a shutdown handler. Any request on the shutdown port will block incoming requests and wait for max 5s until pending requests are all handled. Then the server is shut down.

Ubuntu Phone, synchronizing your contacts and calendar with ownCloud

I love Ubuntu Touch and have been using it on a LG Nexus 4 for a couple of months.
Now that my real Ubuntu phone has arrived in the shape of a BQ Aquaris 4.5 Ubuntu Edition, it is time to properly connect it to my address book and calendars which I have on ownCloud (thanks to Eliwan).

Ubuntu phone

Ubuntu is a great operating system for the smartphone. One of the extra attractions is the ability to be avoid disclosing private information to one of big IT companies like Apple, Google or Microsoft.

Unfortunately though, there is no user interface (should I say ‘app’) yet to enter the necessary configuration. You have to do this using the shell at this moment.

  • From the configuration app select ‘About this phone’ and then (at the bottom) ‘Developer mode’. Switch it on.
  • From a Ubuntu PC, run “adb shell”. If adb is not available it will tell you how to install it. If you don’t have a Ubuntu PC you can run the commands below from the shell app.
  • Create your ownCloud connection:
    syncevolution --configure --template WebDAV username=yourUserName password=yourPassWord syncURL=https://me.eliwan.net/cloud/remote.php/ keyring=no target-config@owncloud

    Replace ‘yourUserName’ with your username and ‘yourPassword’ with your password. The ‘https://me.eliwan.net/cloud’ should be replaced by your ownCloud root.

  • To sync your contacts:
    syncevolution --configure database=https://me.eliwan.net/cloud/remote.php/carddav/addressbooks/yourUserName/contacts backend=carddav target-config@owncloud myaddressbook
     
    syncevolution --configure --template SyncEvolution_Client sync=none syncURL=local://@owncloud username= password= owncloud
     
    syncevolution --configure sync=two-way backend=addressbook database= owncloud myaddressbook
     
    syncevolution --sync slow owncloud myaddressbook

    Replace ‘yourUserName’ with your username. The ‘https://me.eliwan.net/cloud’ should be replaced by your ownCloud root.

  • To sync your personal calendar:
    syncevolution --configure \
    	database=https://me.eliwan.net/cloud/remote.php/caldav/calendars/yourUserName/calendarName/ \
    	backend=caldav \
    	target-config@owncloud calendar-calendarName
     
    syncevolution --configure sync=two-way backend=calendar database= owncloud calendar-calendarName
     
    syncevolution --sync slow owncloud calendar-calendarName

    You can verify the link using the chain icon (CalDAV link) in the user interface.
    Replace ‘calendarName’ with the name of your calendar and ‘yourUserName’ with your username. The ‘https://me.eliwan.net/cloud’ should be replaced by your ownCloud root.

  • To create and sync additional calendars, you have to make sure an extra calendar is created and linked:
    syncevolution --configure \
    	database=https://me.eliwan.net/cloud/remote.php/caldav/calendars/yourUserName/calendarName/ \
    	backend=caldav \
    	target-config@owncloud calendar-calendarName
     
    syncevolution --create-database backend=calendar database=LocalName
    syncevolution --configure backend=calendar database=LocalName @default LocalName
    syncevolution --configure sync=two-way backend=calendar database=LocalName owncloud calendar-calendarName
     
    syncevolution --sync slow owncloud calendar-calendarName

    ‘LocalName’ needs to be replaced with the name you want for the calendar in the calendar app.
    You can verify the link using the chain icon (CalDAV link) in the user interface.
    Replace ‘calendarName’ with the name of your calendar and ‘yourUserName’ with your username. The ‘https://me.eliwan.net/cloud’ should be replaced by your ownCloud root.

There is one minor glitch I see at this moment. The ‘all-day events’ are currently visible in the calendar in both the day itself and the following day.

If you want to add the calendar app to the launcher which is displayed when swiping from the left you should open the app, then swipe from the left (all across the screen) to make sure the launcher stays on screen. With the calendar open the calendar icon should be there. Now indicate and hold the calendar item. This should display a little menu with the item ‘pin shortcut’. Select that.

Presto.

Filtering logs using logstash

Our application produces a lot of logs. Sometimes we want to extract some specific information from it, for example the timestamps and duration of the REST calls in it or have a specific list of all the errors (possible excluding some common ones).

You can use logstash to do this kind of filtering. In this case, extracting information from one or more log files into specific files.

Let’s start by defining the input files.

input {
 
  file {
    type => "logfile"
    path => [ "/home/joachim/temp/logs/*.log" ]
    sincedb_path => "/dev/null"
  }
 
}

We just specify where the logs we want to process can be found. Note that only changes which occur after logstash is started will be processed.

Our logged items can contain multiple lines so lines need to be combined. The data is also structured, so the fields can be extracted. This uses the format I already explained earlier.

filter {
 
  multiline {
    pattern => "^[\[~]"
    negate => true
    what => "previous"
    # enable_flush is not recommended but assures that the last log statement (multi-line) of the file is also processed
    enable_flush => true 
  }
 
  grok {
    match => [ 
      "message", "~%{NOTSPACE:application} %{TIMESTAMP_ISO8601:timestamp} \[%{DATA:server}\-%{DATA:thread}\] %{LOGLEVEL:severity}\s+%{JAVAFILE:category} \- %{GREEDYDATA:shortmessage}"
    ]
  }

Now extract some additional fields which are not available in each message.

  grok {
    match => [ "message", "Duration: %{NUMBER:duration:float}s" ]
    tag_on_failure => [] 
  }
 
  grok {
    match => [ "message", "ldapid: %{WORD:ldapid}" ]
    tag_on_failure => [] 
  }
 
  grok {
    match => [ "message", "role: %{WORD:role}" ]
    tag_on_failure => [] # deze regel matcht niet altijd
  }
 
  grok {
    match => [ "message", "organisation: %{WORD:organisation}" ]
    tag_on_failure => [] # deze regel matcht niet altijd
  }
 
  grok {
    #match => [ "message", "Service: %{WORD:http_command} %{URI:endpoint}" ]
    match => [ "message", "Service: %{WORD:http_command} %{NOTSPACE:endpoint}" ]
    tag_on_failure => [] 
  }

Specify the date format.

  date {
    # 2013-09-23T11:27:14.177+0200
    match => [
        "timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
    ]
  }

Now add some tags for the type of output. These are very practical for determining the output files. This is the last part of the filter section in the configuration file.

  # add tag to type of output
  if [severity] == "ERROR" {
    mutate {
      add_tag => "error"
    }
  }
  if "b.v.a.d.s.util.ProfilingSchedule" in [message] {
    mutate {
      add_tag => "profile"
    }
  }
  if [endpoint] =~ /http/ {
    mutate {
      add_tag => "endpoint"
    }
  }
 
}

Now put the details we want in different files. We have one file which only contains the errors, one file which contains the profiling info and one file which contains a CSV file (space separated) of REST requests.

output {
 
  if "error" in [tags] and ( "huidige_gebruiker" not in [endpoint] ) {
    file 
    {
      path => "dc-errors.txt"
      max_size => 20000000
      message_format => "%{message}"
    }
  }
 
  if "profile" in [tags] {
    file 
    {
      path => "dc-profile.txt"
      max_size => 20000000
      message_format => "%{message}"
    }
  }
 
  if "endpoint" in [tags] {
    file 
    {
      path => "dc-requests.txt"
      max_size => 20000000
      message_format => "%{timestamp} %{http_command} %{endpoint} %{duration}"
    }
  }
 
  #remove this line and uncomment lines below for debugging
  #stdout {
  #  message_format => "%{timestamp}"
  #}
 
}