Making JasperReports less memory hungry

We had an issue in production. We are producing reports using JasperReports. Some reports were not being generated under load. The application has a REST interface to produce a report. We had a specific 75 page documents which contains a two SVG logos on each page and 47 photos throughout the report.

We quickly started thinking that the amount of heap space used to generate the report was probably causing problems. So we tried some measurements. When simply starting the application and then generating the report, the application needed a heap of 460MB to succeed. Time to investigate and look for optimizations.

We improved the following:

  • Upgraded JasperReports to the latest version.
  • The images which needed to be included in the report were being sent as part of the request using base 64 encoding for the image. This request was logged. We improved this by
    • Removing the logging of the base 64 data.
    • Next we changed this by letting the application load the images as late as possible by including download details instead of base64 data. The image is now lazy-loaded to allow the image data to be garbage collected quickly.
  • The (svg) images which are displayed in the header and footer of each page now use the “isCached” attribute to allow JasperReports to cache the images instead of re-rendering for each page.
  • Our reports use styled text for all fields. However most fields do not contain any styling. To avoid excessive parsing of styled text (full XML parsing) we patched the JRStyledTextParser class JasperReports to only parse the text when it contains at least both a < and a > See the excerpt below. This is one change we measured separately and it saves us 10MB of heap space.
  • We enabled page virtualization. This reduces the number of pages which are kept in memory while rendering.

With these changes we were able to improve the situation substantially. It is now possible to start the application and generate a report with a heap of only 120MB.

The following patch was applied to JRStyledTextParser (adding one “if” to getStyledText).

public JRStyledText getStyledText(Map<Attribute,Object> parentAttributes, String text, boolean isStyledText, Locale locale)
{
    JRStyledText styledText = null;
    if (isStyledText && text.contains("<") && text.contains(">"))  // changed JVDA to prevent excessive parsing
    {
        try
        {
            styledText = parse(parentAttributes, text, locale);
        }
        catch (SAXException e)
        {
            //ignore if invalid styled text and treat like normal text
        }
    }
 
    if (styledText == null)
    {
        // using the original String object instead without creating a buffer and a String copy
        styledText = new JRStyledText(locale, text, parentAttributes);
    }
 
    return styledText;
    }

The lazy loading of the images is handles by wrapping the parameters which are passed to JasperReports. This basically delegates to the underlying map

/**
 * Special map for inside rowMaps which loads external images lazily.
 */
public class LazyLoadExternalImagesMap implements Map<String, Object> {
 
    private static final String EXTERNAL_IMAGE = "externalImage";
    private static final String EXTERNAL_IMAGE_REF = "externalImageRef";
 
    private static final String EXTERNAL_IMAGE_MIME = "mimeType";
    private static final String EXTERNAL_IMAGE_URL = "url";
    private static final String EXTERNAL_IMAGE_COOKIE = "cookie";
 
    private Map<String, Object> delegate;
    private ExternalDataHelper externalDataHelper;
 
    /**
     * Constructor.
     *
     * @param delegate delegate
     * @param externalDataHelper helper to load the external image
     */
    public LazyLoadExternalImagesMap(Map<String, Object> delegate, ExternalDataHelper externalDataHelper) {
        this.delegate = delegate;
        this.externalDataHelper = externalDataHelper;
    }
 
    @Override
    public Object get(Object key) {
        if (EXTERNAL_IMAGE.equals(key)) {
            ExternalImageTo eb = getExternalImageRef();
            if (null != eb) {
                InputStream is = externalDataHelper.get(eb);
                try {
                    return ImageIO.read(is);
                } catch (IOException ioe) {
                    throw new ServiceException("Cannot load external image " + eb + ".", ioe);
                } finally {
                    StreamUtil.close(is);
                }
            }
        }
        return delegate.get(key);
    }
 
    private ExternalImageTo getExternalImageRef() {
        Object ebRef = delegate.get(EXTERNAL_IMAGE_REF);
        if (ebRef instanceof Map) {
            Map map = (Map) ebRef;
            if (map.containsKey(EXTERNAL_IMAGE_MIME) && map.containsKey(EXTERNAL_IMAGE_URL)) {
                ExternalImageTo eb = new ExternalImageTo();
                eb.setMimeType(string(map.get(EXTERNAL_IMAGE_MIME)));
                eb.setUrl(string(map.get(EXTERNAL_IMAGE_URL)));
                eb.setCookie(string(map.get(EXTERNAL_IMAGE_COOKIE)));
                return eb;
            }
        }
        return null;
    }
 
    private String string(Object s) {
        if (null == s) {
            return null;
        }
        return s.toString();
    }
 
 
    // ----- delegated methods below
 
 
    @Override
    public int size() {
        return delegate.size();
    }
 
    @Override
    public boolean isEmpty() {
        return delegate.isEmpty();
    }
 
    @Override
    public boolean containsKey(Object key) {
        return delegate.containsKey(key);
    }
 
    @Override
    public boolean containsValue(Object value) {
        return delegate.containsValue(value);
    }
 
    @Override
    public Object put(String key, Object value) {
        return delegate.put(key, value);
    }
 
    @Override
    public Object remove(Object key) {
        return delegate.remove(key);
    }
 
    @Override
    public void putAll(Map<? extends String, ?> m) {
        delegate.putAll(m);
    }
 
    @Override
    public void clear() {
        delegate.clear();
    }
 
    @Override
    public Set<String> keySet() {
        return delegate.keySet();
    }
 
    @Override
    public Collection<Object> values() {
        return delegate.values();
    }
 
    @Override
    public Set<Entry<String, Object>> entrySet() {
        return delegate.entrySet();
    }
 
    @Override
    public boolean equals(Object o) {
        return delegate.equals(o);
    }
 
    @Override
    public int hashCode() {
        return delegate.hashCode();
    }
 
    @Override
    public Object getOrDefault(Object key, Object defaultValue) {
        return delegate.getOrDefault(key, defaultValue);
    }
 
    @Override
    public void forEach(BiConsumer<? super String, ? super Object> action) {
        delegate.forEach(action);
    }
 
    @Override
    public void replaceAll(BiFunction<? super String, ? super Object, ?> function) {
        delegate.replaceAll(function);
    }
 
    @Override
    public Object putIfAbsent(String key, Object value) {
        return delegate.putIfAbsent(key, value);
    }
 
    @Override
    public boolean remove(Object key, Object value) {
        return delegate.remove(key, value);
    }
 
    @Override
    public boolean replace(String key, Object oldValue, Object newValue) {
        return delegate.replace(key, oldValue, newValue);
    }
 
    @Override
    public Object replace(String key, Object value) {
        return delegate.replace(key, value);
    }
 
    @Override
    public Object computeIfAbsent(String key, Function<? super String, ?> mappingFunction) {
        return delegate.computeIfAbsent(key, mappingFunction);
    }
 
    @Override
    public Object computeIfPresent(String key, BiFunction<? super String, ? super Object, ?> remappingFunction) {
        return delegate.computeIfPresent(key, remappingFunction);
    }
 
    @Override
    public Object compute(String key, BiFunction<? super String, ? super Object, ?> remappingFunction) {
        return delegate.compute(key, remappingFunction);
    }
 
    @Override
    public Object merge(String key, Object value, BiFunction<? super Object, ? super Object, ?> remappingFunction) {
        return delegate.merge(key, value, remappingFunction);
    }
}

Installing Ubuntu on a Dell M3800

For a while I was having problems with my laptop having too little memory. It was a Dell XPS13 developer edition. A system I was very happy with, but the 8GB RAM was proving to be too little in some cases. So I wanted something more powerful. I chose the Dell M3800 because it has a recent i7, has 16GB RAM and a HiDpi screen with a 3200×1800 resolution. No “developer edition” though but (being a full-time Ubuntu Linux user) I went bought it anyway.

I installed Ubuntu 14.10 on it. and encountered a few issues.

  • The system was a bit slower to compile than my previous one even though it has more memory and a faster processor. Looking into this, it seems this was caused by overly aggressive settings to keep the temperature down. I tweaked the termald settings to kick in at 75 instead of 45 degrees and this halved the compile time! For details look at “Thermal Issues” and “Powerclamp”.
  • Parcellite (a tool which keeps clipboard memory) does not seem to work. I replaced this with diodon which seems to be a better tool anyway.
  • The system needs to be rebooted a lot more than my previous one. There are two problems which sometimes occur which need a reboot to fix. One is that the screen turns black – as if I am not doing anything except it kicks in after a second or two instead of five minutes. I turns back on when I move the mouse but gets annoying quickly. The other is that the wifi sometimes gets very disfunctinal, hardly being able to connect to the local network and when it does only has very limited bandwidth.
  • Sometimes the keyboard becomes quite unresponsive. This may be application related. I have encountered this most in pgAdmin and Inkscape. Inkscape also seems to crash very easily.

The HiDpi screen is a challenge in itself.

  • Ubuntu itself (or Unity) looks great. The trick is to set the “Scale for menu and title bars” setting in the display menu. I used “2” as setting so the screen looks as if I have a resolution of 1600×900 but with more detailed rendering.

  • Firefox also works fine. You just have to set “layout.css.devPixelsPerPx” to 2 in about:config.
  • The same trick works for thunderbird. To access about:config, go to Edit → Preferences → Advanced → Config editor.
  • Chromium only half works. Many windows and pop-up boxes are positioned and scaled wrong.
  • Unfortunately many application don’t (or only half) support the scale setting. For example pgAdminIII is mostly ok, but all rows in query results are only half the size of the text displayed in them.
  • Java applications also don’t respect the scale. In some cases this can be fixed by configuring the text size (for example in IntelliJ IDEA), but the menus and icons stay too small. For the many, installing jayatana (Unity global menu integration) really helps.

In short, the linux support for HiDpi displays is clearly a work in progress.

Database migrations using Flyway, multiple modules for one schema

In our project we selected Flyway to handle out database migrations.

This is a great library which can easily be configured to do the database migrations for you. For example using Spring all you need is:

<bean class="org.flywaydb.core.Flyway" init-method="migrate">
    <property name="dataSource" ref="dataSource" />
    <property name="locations" value="scripts" />
</bean>

Basically you point it to the data source and the package where the migration scripts can be found (or migration classes).

In our setup, the migrations script have names like the following (conforming to the default conventions):

  • V001__initial_schema.sql
  • V002__initial_data.sql
  • V003__additional_script.sql

Flyway will check if the selected database schema (default is “public”) is empty. If it is, the table which stores information about the database migrations is created and the found migrations are run. If the schema exists, it will use the table to figure out which migrations need to be performed. As a sanity check it verifies whether there are any conflicts between previous migrations and currently available migrations.

Unfortunately this does not work in our setup.

  • The project has multiple modules which all have their own migration scripts but share the public schema.
  • We also use Activiti which manages its database structure itself, creating the tables before Flyway kicks in. This causes Flyway to stop as the migrations table does not exit and the schema is non-empty.

Fortunately Flyway allows for this case using the following configuration (shown here for two modules).

<bean class="org.flywaydb.core.Flyway" init-method="migrate">
    <property name="dataSource" ref="dataSource" />
    <property name="table" value="module1_schema_version" />
    <property name="locations" value="scripts/module1" />
    <property name="initOnMigrate" value="true" />
    <property name="initVersion">
        <bean class="org.flywaydb.core.api.MigrationVersion">
            <constructor-arg index="0" value="000" />
        </bean>
    </property>
</bean>
<bean class="org.flywaydb.core.Flyway" init-method="migrate">
    <property name="dataSource" ref="dataSource" />
    <property name="table" value="module2_schema_version" />
    <property name="locations" value="scripts/module2" />
    <property name="initOnMigrate" value="true" />
    <property name="initVersion">
        <bean class="org.flywaydb.core.api.MigrationVersion">
            <constructor-arg index="0" value="000" />
        </bean>
    </property>
</bean>

There are a couple of tricks.

  • We use different tables for the migration info for each of the modules (using “table”).
  • We use a different classpath location for the migration script for each of the modules (using “locations”).
  • We assure Flyway always initialises itself (using “initOnMigrate”). This assures the migrations table is always created if it does not exist.
  • Assure all out scripts are run using “initVersion” setting it to “000”. When initOnMigrate is true, a record is written in the migratinos table to indicate that Flyway init was run. By default this is marked as version “1” which would then means that our first migration script (“V001__initial_schema.sql” in the example above) is not run as it is also version “1”.