JBossWorld Orlando, Tuning JBoss on linux

Faster than your shadow, notes from this session, to a large extent also mentioned on the slides (which will be available later and which contain more details about some of the linux settings and how to calculate some of the numbers). Most of the linux specific details are detailed for Red Hat linux variant, there may be variations for other distributions.

Know your peaks, don’t look at the averages.
Applications should be instrumented for performance analysis. This can be done using AOP or statistics in the containers and Hibernate statistics.

Understand where time is being spent !
– transactions is not enough, you need to know where time is being spent in your transactions
– don’t guess where time is lost, measure !

Modeling results
– If your load testing environment is not the same as your production environment, you need a model to relate the two.
— be conservative ! if your production env is twice the size of load testing env, don’t double your numbers !
— in all cases you will not get linear results

Tuning
– about 75% of performance problems is application related, not middleare or operating system
– do check
— connection pooling
— thread pools
— logging
— object and component pools
— caching
— clustering and replication

Connection pooling
– assure database pool is big enough, check JMX console to check pool utilization
– you probably wantg a min-pool-size for expected load, and a max-pool-size which allows some growth

Thead pooling
– see conf/jboss-service.xml for system thread pool, can be monitored in JMX console
– jbossweb server.xml has a threadpool
– check BlockingMode which is the behaviour when the threadpool queue is full, default (“run”) is probably safest, but “wait” could be useful when tuning (makes problems more visible)

Other pools
– conf/standardjboss.xml has JMSPool
– conf/standardjboss.xml has J2EE 1.4 container pools
– deploy/ejb3-interceptors-aop.xml has EJB3 pool

Logging
– diable console logging, this is expensive in a production environment
– turn down logging verbosity level if it is not necessary
– use asynchronous logging
– wrap debug statements with “if (debugEnabled())” to avoid the work of the parameter building and building the log message (in the logger)

Caching
– JBoss Cache is part of JBoss, it can be used to cache anything to you want
– EJB3 entities can be cached, see persistence.xml and deploy/ejb3-entity-cache-service.xml

Linux
– on 64 bit use Linux large memory page support (HugeTLB)
— default page size is 4kB, this is a lot of pages for 1GB
— large memory pages usually starts at 2MB pages
— large memory pages cannot be swapped to disk (useful)
— carefull all memoty assigned in large pages is no longer available to other applications
HOW
– Sun JVM : -XX:+UseLargePages,
– in etc/sysctl.conf
— kernel.sjmmax=n
— vm.nr_hugepages=n
– etc/security/limits.conf
– after starting the applications you should see a non-zero value for “HugePages_Rsvd” in /proc/meminfo

Tuning the JVM
– /etc/sysctl.conf set vm.swappiness to 1

Database tuning
– database cache is king
– use DIRECT_IO if your database supports it (but assure your db cache is big enough)
– utilize your database capability to divide tables, logs etc over different physical disks
– use large block or page sizes

Storage
– 4ms should be your average access times as upper bounds for your db
– you may need a storage system which can stripe over many drives
– if your use NAS or iSCSI SAN, isolate storage network from the rest and use jumbo frames (9000 byte MTU)

Storage and write caching
– when using disk write caching, favor solutions with solid state caches

Result for applying the above using a EJB3 application with two UI servlets : approx 3x improvement on tx throughput (mostly due to large page support on heap sizes above 2GB)
— set soft and hard memlock for users (possibly also root)
— vm.huge_tlb_shm_group=gid : shared group of users with large pages access

135 Comments

  1. mario says:

    hello,

    thaks for the tipps, gave me some ideas. is it enough to state the ejb3 pool sizes in the ejb3-interceptors-aop.xml or do i have to make an annotation in my application?

    thanks,

    mario

  2. joachim says:

    Just putting the pool sizes in the xml file should be sufficient.

Leave a Reply

Your email address will not be published. Required fields are marked *

question razz sad evil exclaim smile redface biggrin surprised eek confused cool lol mad twisted rolleyes wink idea arrow neutral cry mrgreen

*