JBossWorld Orlando, Jboss clustering tuning clustering

Faster than your shadow, notes from this session, to a large extent also mentioned on the slides (which will be available later).

Which knobs can you turn… Focus on end-to-end performance.

Topology : session replication, all coming through apache which connects to multiple backends (using AJP) with sticky sessions (so apache will redirect the session to the same backend server).

Client setup
– Apache : mod-jk, workers.properties (list backend nodes), urimapping, modjk.conf
– JBossWeb : jvmRoute=”node1″ (from workers.properties), UseJK=”true”

Configuration
– web.xml :
– jboss-web.xml : replication granularity (what is replaicated) : SESSION (default) (all sesssion), ATTRIBUTE (only modified attributes), FIELD (check parts of attributes)
– jboss-web.xml : replication trigger : SET, SET_AND_GET, SET_AND_NON_PRIMITIVE_GET (default), in all cases only one replication per request (at end)

Replication mode
– synchronous : http blocks till replication is done
– asynchronous : only blocks until message is sent, faster but no guarantee that the changes have been received, bit less
– configuration : REPL_SYNC or REPL_ASYNC

Buddy vs total replication
– total (default) : replicate to all nodes of cluster, doesn’t scale well
– buddy : session is replicated to configurable number of backup “buddies”, (default is 1)
– configuration : set “buddyReplacationEnabled” in jboss-web-cluster.sar/META-INF/jboss-service.xml

TIP 1 : Use ATTRIBUTE granularity
– reduce amount of replicated data

TIP 2 : Use SET
– avoid unnecessary replication
– only works if getters have no side-effects and data is not modified after get (for non-primitive data)

TIP 3 : Use buddy replication
– replicate to N backups instead of all nodes
– maybe use of TCP for intra-cluster comm (makes the communication reliable, udp is better for multicast)
effect : big improvements when more than 2 nodes
caveat
– when node fails other nodes need to pick new buddy : large amount of data which needs to be replicated

TIP 4 : TCP vs UDP
– default is UDP and multicast : logical for total replication, for buddy UDP unicast is used
– TCP is a valid choice if number of peers is low
– some tests indicate that for small number of nodes and total replication, TCP is faster (up to 4), for buddy replication performance is similar

cost of REPL_SYNC
– total+udp : sync kills performance (esp for large number of nodes)
– buddy+tcp+sync : cost decreases with larger number of nodes

addition tips and tricks
– enable KeepAlive in httpd.conf
– connection pool sizes should be big enough to handle max # of concurrent connections
— remember nodes can fail adding load to survivors
— default AJP connector pool size is only 40 !
— both httpd.conf and server.xml
– use JBossWeb (which uses a native APR lib which scales better)
– JMX
— listThreadCpuUtilization(), listThreadDump()
— TomcatCluster MBean : look at contents of tree
– Unused HTTP sessions take up space
— set http session timeout
— call invalidate() when done with a session
– assure logging is tuned, use WARN level (can be changed at runtime if needed)
– apache logs : access_log, modjk.log can be big
– lose some fat, remove unneeded stuff (and check number of clusters, default has 4)
– seperate traffic for client, AJP and replication traffic (using different network ports if possible)
– use multiple mod-jk domains, eg instead of using 9 node full repl cluster, split using load balancer in three domains each with three servers

Outlook
– cooperate with apache to allow it to sense configuration from the JBoss configuration (and let JBoss tell Apache what the load on a node is)

Leave a Reply

Your email address will not be published. Required fields are marked *

question razz sad evil exclaim smile redface biggrin surprised eek confused cool lol mad twisted rolleyes wink idea arrow neutral cry mrgreen

*