Faster than your shadow, notes from this session, to a large extent also mentioned on the slides (which will be available later and which contain more details about some of the linux settings and how to calculate some of the numbers). Most of the linux specific details are detailed for Red Hat linux variant, there may be variations for other distributions.
Know your peaks, don’t look at the averages.
Applications should be instrumented for performance analysis. This can be done using AOP or statistics in the containers and Hibernate statistics.
Understand where time is being spent !
– transactions is not enough, you need to know where time is being spent in your transactions
– don’t guess where time is lost, measure !
– If your load testing environment is not the same as your production environment, you need a model to relate the two.
— be conservative ! if your production env is twice the size of load testing env, don’t double your numbers !
— in all cases you will not get linear results
– about 75% of performance problems is application related, not middleare or operating system
– do check
— connection pooling
— thread pools
— object and component pools
— clustering and replication
– assure database pool is big enough, check JMX console to check pool utilization
– you probably wantg a min-pool-size for expected load, and a max-pool-size which allows some growth
– see conf/jboss-service.xml for system thread pool, can be monitored in JMX console
– jbossweb server.xml has a threadpool
– check BlockingMode which is the behaviour when the threadpool queue is full, default (“run”) is probably safest, but “wait” could be useful when tuning (makes problems more visible)
– conf/standardjboss.xml has JMSPool
– conf/standardjboss.xml has J2EE 1.4 container pools
– deploy/ejb3-interceptors-aop.xml has EJB3 pool
– diable console logging, this is expensive in a production environment
– turn down logging verbosity level if it is not necessary
– use asynchronous logging
– wrap debug statements with “if (debugEnabled())” to avoid the work of the parameter building and building the log message (in the logger)
– JBoss Cache is part of JBoss, it can be used to cache anything to you want
– EJB3 entities can be cached, see persistence.xml and deploy/ejb3-entity-cache-service.xml
– on 64 bit use Linux large memory page support (HugeTLB)
— default page size is 4kB, this is a lot of pages for 1GB
— large memory pages usually starts at 2MB pages
— large memory pages cannot be swapped to disk (useful)
— carefull all memoty assigned in large pages is no longer available to other applications
– Sun JVM : -XX:+UseLargePages,
– in etc/sysctl.conf
– after starting the applications you should see a non-zero value for “HugePages_Rsvd” in /proc/meminfo
Tuning the JVM
– /etc/sysctl.conf set vm.swappiness to 1
– database cache is king
– use DIRECT_IO if your database supports it (but assure your db cache is big enough)
– utilize your database capability to divide tables, logs etc over different physical disks
– use large block or page sizes
– 4ms should be your average access times as upper bounds for your db
– you may need a storage system which can stripe over many drives
– if your use NAS or iSCSI SAN, isolate storage network from the rest and use jumbo frames (9000 byte MTU)
Storage and write caching
– when using disk write caching, favor solutions with solid state caches
Result for applying the above using a EJB3 application with two UI servlets : approx 3x improvement on tx throughput (mostly due to large page support on heap sizes above 2GB)
— set soft and hard memlock for users (possibly also root)
— vm.huge_tlb_shm_group=gid : shared group of users with large pages access