Thursday, March 31, 2011

WebSphere Job Interview Sample Questions 002

Intermediate Questions

1. Please talk about your recent WebSphere work experience?

Worked on WebSphere 8
Worked on WebSphere eXtreme Scale 7.1
worked on the streamlining and stabilizing of a very large financial WebSphere system
worked on the troubleshooting of a WebSphere Virtual Enterprise system

2. Why do you think that you can be a good WebSphere engineer?

UNIX experience
Good scripting expertise
C, C++, Java language programming
Long years of working on a variety of releases of WebSphere
Long years of working on large and complex WebSphere projects

3. Please explain the major differences between WebSphere 6 and WebSphere 7?

new features in WAS V7


* support Java SE 6
* Java annotation and injection support to reduce complexity
* Web services specification updates
* Flexible management model with Administrative Agent and Job Manager
* Manage DataPower through administrative console
* Properties file based configuration
* Fine-grained administrative security extended to administrative console
* Multiple security domain support
* Security auditing support
* Integration with MQ WebSphere V7 using JCA 1.5 resoure adapter
* Centralized Installation Manager (CIM)
* Rational Application Developer for WAS V7.5

4. Can you tell the major differences between Type 2 JDBC driver and Type 4 JDBC driver?
Type 2 Driver - Native-API Driver specification schematic of the Native API driver. The JDBC type 2 driver, also known as the Native-API driver, is a database driver implementation that uses the client-side libraries of the database. The driver converts JDBC method calls into native calls of the database API. The type 2 driver is not written entirely in Java as it interfaces with non-Java code that makes the final database calls. The driver is compiled for use with the particular operating system. For platform interoperability, the Type 4 driver, being a full-Java implementation, is preferred over this driver. However the type 2 driver provides more functionality and better performance than the type 1 driver as it does not have the overhead of the additional ODBC function calls.

Advantages

* Better performance than Type 1 Driver (JDBC-ODBC bridge).
* Provides Fastest performance than all 3 drivers as it calls native APIs(MySQL,Oracle...etc).

Disadvantages

* The vendor client library needs to be installed on the client machine.
* Cannot be used in web-based application due to the client side software needed.
* Not all databases have a client side library
* This driver is platform dependent
* This driver supports all java applications except Applets

Type 4 Driver - Native-Protocol Driver, Schematic of the Native-Protocol driver
The JDBC type 4 driver, also known as the Direct to Database Pure Java Driver, is a database driver implementation that converts JDBC calls directly into the vendor-specific database protocol. The type 4 driver is written completely in Java and is hence platform independent. It is installed inside the Java Virtual Machine of the client. It provides better performance over the type 1 and 2 drivers as it does not have the overhead of conversion of calls into ODBC or database API calls. Unlike the type 3 drivers, it does not need associated software to work. As the database protocol is vendor-specific, separate drivers, usually vendor-supplied, need to be used to connect to different databases.

Advantages

* These drivers don't translate the requests into an intermediary format (such as ODBC), nor do they need a middleware layer to service requests. Thus the performance may be considerably improved.
* All aspects of the application to database connection can be managed within the JVM; this can facilitate easier debugging.

Disadvantage

* At client side, a separate driver is needed for each database.

5. What is the major function of Job Manager?

* New profile type to support flexible management
* Use administrative agent to register base server profiles with a job manager
* To manage multiple cells, register deployment managers with job manager directly
* Use job manager to queue jobs for registered profiles
* Registered profiles retain autonomy and can be managed without the job manager
* Scales to support large number of registered profiles

6. What are the major new features of JDK 1.5 GC?

*Annotation not deployment descriptor
*GC - more options available such as nursery and tenured sections
*JPA support - no more EJB entity bean (Thank God)

7. What are the major benefits of database connection pool?
A connection pool is a service provided by an application server that can be used to improve data access performance with a group of open connections to the database being maintained persistently. Each time an application needs access to the database, it takes a connection from the pool, access the database then returns the connection to the pool. The connection pool technique removes the overhead incurred when the connection is open and closed. Such overhead can be extremely high for an internet application, which could have a massive negative impact on web applications performance.

8. How do you configure JMS?
Java Messaging Service (JMS) is the standard API for accessing enterprise messaging systems from Java-based applications. It provides methods and functions that are directly implemented by the underlying messaging provider. WebSphere MQ is the premier messaging middleware provided by IBM.

Configure JMS resources

* Create JMS provider
* Create JMS connection factory
* Create JMS queue or topic
* Create listener port
* Create JMS activation specification

9. How to produce a heap dump? How to direct heap dump to a specific location?

kill -3
The dumps go to the working directory.

10.What UNIX commands can be used to find CPU usage or memory usage or network state?

top
topas
ps
vmstat
netstat

11.How do you tell disk usage?
This sounds simple, but I have seen WebSphere consultants did not know what to do when WAS did not start due to 100% disk usage.

df -k
du -k

12.How to configure Introscope 8.1?

*install Introscope agent on the app server
*define Java general argument that includes a profile file that tells where is the Introscope EM along with other relevant information such as port number - the profile file is on the app server - the path has to be included
*create custom service on the WebSphere server via admin console - this specifies the Introscope class
*modify the PBD files to specify metrics to monitor  IBM document on setting up the PBD.

1. A URL is not working. How do you find out why?
First, I would exclude 3DNS and BIG IP by directing trying to get to the URL to the web server transport port. Then, I would try to see if the WebSphere Application Server's port for this URL is working. The key is to determine if the request has ever made to the web layer and application server layer. That is the important first step.

It helps to test all the way from the browser if the network connectivity is working.

For static pages, focus on the web server by examining the access log and error log of the web server. Determine if the web server in general is functional. Web Server resources are adequate.

For application server, examine web container thread, JVM GC, and look at log files for tell-tale message.

Produce dumps (thread, heap, core) and examine these dumps while sending them to IBM with the opening of a PMR.

2. Some users are experiencing latency. How do you find out why?
In the event that some users are experiencing latency or slow response time we should examine the complete request processing chain involving network, load balancers, HTTP web servers, application servers, back end databases, and hardware.

First we examine the network layer such as firewalls or switches and so on to make sure they work properly. Network devices have processing capacity and use memory much like any other hardware resource. Adding additional hardware resources to a network device will increase the throughput.

Then we can investigate load balancers and HTTP web servers to see whether they are over loaded or not. If yes, the web server plug-in tuning might be needed.

The next step is to examine the processing time of application servers at JVM, Web Container and EJB Container layer. For example, check the JVM heap utilization. If GC is too frequent or if the pause is too, then, tune GC.

And finally, the connection pool usage between application servers and backend data sources such as database should be checked. If the database backend is slow, the whole system will be slow. It helps to examine the network connection from the WebSphere Application Server to the database backend and if the database is working as designed.


3. What are the major benefits of vertical clustering and horizontal clustering?
Vertical clustering (process redundancy) involves creating multiple application server processes on a single physical machine, providing application server failover as well as load balancing across multiple application servers. This topology does not provide an efficient fault tolerance because a failure of the operation system or the hardware on the physical machine itself might cause problems to all servers in the cluster.

Horizontal clustering (process and hardware redundancy) involves creating application servers on multiple machines to take advantage of the additional processing capacity available on each machine. Using horizontal scaling techniques also provides hardware failover support.

4. If you add substantial more vertical clustering trying to overcome a bottleneck at the web container layer, what you have to watch out for as a potential system capacity issue?
The first concern is to watch the system memory capacity since the vertical clustering involves multiple JVMs running on a single physical machine and each JVM requires enough memory to achieve desired performance. The memory requirement is heightened even more if the clustering is
configured using servlet and JSP caching in at the web container layer.

Next we should investigate if enough messaging resources are in place to support the heavy communication traffic and loading to the database servers and other enterprise information systems. Throughput and response time under heavy web requests originated from the web servers must be measured to ensure that the vertical clustering meets the expectations.

5. What are some of the application code logic and construct that prevents the use of vertical and horizontal clustering?
Unique counters, traffic routers, and any Java artifacts that depends on a unique serialized valued to perform its business function. This type of the problems can be avoided if the WebSphere engineer engages early in the JEE project and provide WebSphere application development best practices to the application developers and architects.

6. Explain the main improvements of Java 6
1. improved caching capability - shared cache among JVM - caching more Java objectis than static classes - persist Java objects
2. 64 bit reference pointer compression to prevent performance penalty using 64 bit poniters
3. JIT more complied code for better performance

WebSphere Virtual Enterprise questions

1. Can you explain the main features of WebSphere Virtual Enterprise (WVE)?

*dynamic resource allocation through ODR, dynamic cluster, and autonomic computing
*edition management to run multiple version of the JEE application and interrupt free upgrades
*policy based SLA ensurance to focus on key applications
*maximize middleware infrastructure investment
*rule based routing

2. Explain what is ODR?
"The WebSphere Virtual Enterprise On Demand Router (ODR) component provides capabilities such as workload balancing, prioritization, health monitoring, and dynamic operations for dynamic clusters. An ODR can be configured to provide multi-cluster routing, including dynamic clusters located in remote cells, and routing to other servers that are not running WebSphere Virtual Enterprise. The ODR can serve as a replacement for the HTTP server plug-in, but in many configurations both components are used. The HTTP server could be located in the demilitarized zone to serve static content and to provide an entry point to the private network where the ODR resides." - IBM documentation

Think of ODR as a highly intelligent Web Server or Web Server plugin. It does not only route traffic to application server clusters according to a fixed weighted method such as Round Robin. It has the intelligence to understand the state of the application server and adjusts itself in routing behavior. The correct configuration and timely and wise application or ODR fixes are critical in keeping your WebSphere Virtual Enterprise system stable. ODR is the most critical component of WebSphere Virtual Enterprise.

3. What is a dynamic cluster?
"A dynamic cluster is a server cluster that uses weights and workload
management to balance the workloads of its cluster members dynamically,
based on performance information that is collected from the cluster
members. Dynamic clusters enable application server virtualization. "  - IBM documentation

The key here is load - the dynamic cluster can adjust load based on load between its members. 

"A dynamic cluster is an application deployment target that 
can expand and contract depending on the workload in your environment.
  Dynamic clusters work with autonomic managers, including the
application   placement controller and the dynamic workload manager to
maximize   the use of your computing resources. Dynamic clusters are
required   for many of the WebSphere Virtual Enterprise  autonomic 
functions, including high availability and service policies." -  IBM documentation

The critical message here is dynamically increase or reduce the number of work JVM instances.

"Dynamic clusters are similar to the server clusters that you can 
create with WebSphere Application Server Network Deployment , but  key
differences exist that make dynamic clusters much more robust. For
complete life-cycle management servers, the product controls the
creation and deletion of server instances and can start and stop servers.
For assisted life-cycle management servers,the product can control the
state of servers by stopping and starting servers from a pool of
predefined server instances. " - IBM documentation

Well - WAS ND cluster is static and WebSphere Virtual Enterprise has dynamically changing clusters if you switch to automatic mode of operation among "Manual, Supervised, and Automatic".

4. Explain what is the current IBM recommended upper limit for core group?
The members of the core group talk to each other by inter-process communication (IPC). "A CoreGroup is a set of WebSphere servers having a Group Communications System (GCS) fabric, which can be used for Virtually Synchronous messaging. WebSphere environment allows mulitcasts among the members of a core group." Therefore, if the size of the core group is too large, hence the communication burden as well as possible performance degradation. The last IBM recommendation that I heard was  Keith Smith (the chief architect of WAS XD): 50 or less.

5. Explain what is the inter-process communication mechanism replacing core group (service bulletin board) for WVE 6.1.1?
WVE's Bulletin Board (WVEBB) uses scalable peer-to-peer technology and WVEBB removes all WVE dependencies on HA manager and core groups. This is an important architectural change that tends to make your WVE environment more stable. The Bulletin Board (WVEBB) is disabled by default.  For details, please see this WebSphere Virtual Enterprise article.

IBM WVE article

-----------------------------------------------------------------------------------------------------------------
Come across some JEE job interview questions - may be useful to WAS professionals as well since JEE application server is a subset of JEE specification: JEE Interview Questions

A very good WAS interview question sample with answers: WAS Interview Questions

Another with detailed answers: Questions with detailed answers

Keep the Admin Console Open

The WAS Admin Console times out in 30 minutes - very inconvenient.
You can use this Jython program to change the time out value or modify the file directly. (this code will not work unless you fix the indention that is meaningful to Jython. I cannot get this blog editor to do indention easily)

# This progam extends the time out value for WAS admin console
__author__="IBM Red Book Authors"
__date__ ="$Jan 28, 2011 8:55:48 AM$"
dep = AdminConfig.getid("/Deployment:isclite/" )
appDep = AdminConfig.list("ApplicationDeployment", dep )
sesMgmt = AdminConfig.list("SessionManager", appDep )
# check if existing sesMgmt there or not, if not then create a new one, if
# exist then modify it
if (sesMgmt == ""):
    # get applicationConfig to create new SessionManager
    appConfig = AdminConfig.list("ApplicationConfig", appDep )
    if (appConfig == ""):
        # create a new one
        appConfig = AdminConfig.create("ApplicationConfig", appDep, [])
        # then create a new SessionManager using new Application Config
        sesMgmt = AdminConfig.create("SessionManager", appConfig, [] )
    else:
        # create new SessionManager using the existing ApplicationConfig
        sesMgmt = AdminConfig.create("SessionManager", appConfig, [] )
        # endElse
# endIf
# get tuningParams config id
tuningParams = AdminConfig.showAttribute(sesMgmt, "tuningParams" )
if (tuningParams == ""):
    # create a new tuningParams
    AdminConfig.create("TuningParams", sesMgmt, [["invalidationTimeout", 1420]])
else:
    # modify the existing one
    AdminConfig.modify(tuningParams, [["invalidationTimeout", 1420]] )
    # endElse
# saving the configuration changes
AdminConfig.save()

Wednesday, March 30, 2011

WebSphere Job Interview Sample Questions 001

What is a virtual host?
Virtual hosting is a method for hosting multiple domain names on a computer using a single IP address. You can use admin console environment > virtual host to modify alias and assign port numbers (alias plus port numbers = virtual host)
 
How do you configure JDBC and JMS resources?
  • First create a JDBC provider (a JDBC provide encapsulates the JDBC driver for a specific database). From the admin console Resources > JDBC Providers. 
  • Then, create and configure the datasource that is associated with the proper JDBC provider. The datasource is the connection that the Java application uses to connect to the database. Resources > Data Sources are the panels to use in system configuration. 
  • Be particular consistent with the JNDI name of the data source. The JNDI is used in the Java application to access the database
  • Type 2 driver - you need client - Type 4 driver - pure Java 
  • For Type 4 Driver, you will need
    • server name
    • database name
    • port number
    • authentication mechanism along with security credentials (for example, username and password, authentication - JAAS) 
  • Connection pool - performance consideration - max and min connection size
  •  Configure MQ JMS provider
  • Configure Connection Factory and but very careful about the JNDI name because that it what the developers will use (to create sessions and then, consumers or providers) (binding mode with the local MQ queue manager or client mode where a client connection is needed to connect to a remote queue manager)
  • Then, create queue or topic
  • Finally, you have to configure the listener port to connect the dots
    Click on the picture to see a very clear picture
    • WebSphere Application Server > Communication > Messaging > Listener Port > New
    • The most important items are Connection factory JNDI name and these form the connection between the MDB, its connection to listen to and the destination for its messages  
What are the steps involved in deploying an application?
  • Make sure that the resources needed by the application are configured correctly such as JDBC, JMS, virtual host, among others
  • The ear file is readily available on a local or remote server
  • Use Application Panel of the admin console and choose new and follow the prompts
  • Recycle the server and perform post installation certification
  • Use web.xml to determine the context root and home page for testing - the following is the directory to look for the file -
    • C:\Program Files (x86)\IBM\WebSphere\AppServer\profiles\AppSrv04\installedApps\DOD02Node03Cell\ivtApp.ear\ivt_app.war\WEB-INF

What is session persistence? What mechanisms does Websphere have for session persistence?
Session persistency is option to recover the session data in case the server crashes or the session data is too valuable to lose. There are two ways to configure session persistency - database or memory to memory replication. The memory to memory replication has better performance and the database approach is more reliable. JSESSION ID is the server side artifact supporting session and prefer local session affinity are both good topics.

Have you used wsadmin? What are the different objects available in wsadmin?
  • AdminControl - deal with MEAN that represent live object
  • AdminConfi - deal with configuration stored in the configuration repository
  • AdminApp - update application metadata
  • AdminTask - used to access a set of task-oriented administrative commands that provide an alternative way to access the configuration commands
  • Help - print Help.AdminControl() - this gives you a list of the objects. If you want to read text about each area, you can use AdminControl.help()
Troubleshooting – give couple of scenarios where you were involved in troubleshooting
  • JVM crashes with memory exhaustion - caused by application logging logic failure and unbounded data structure to hold business records to log 
  • Ear files disappeared without obvious cause - WAS Virtual Enterprise anomaly - system automatically reinstalled application under certain circumstance - suspicion autonomic computing error
Monitoring tools – TPV, Introscope – have you used them?
  • TPV for light weight monitoring and performance tuning
  • Introscope dives deep into WAS and the application in the JVM
Describe the most complex topology you have worked with
  • KTC
  • CHS
Some basic UNIX commands, flavors of UNIX you have worked on.
  • Ultrix - coding - socket, pipe, broadcasting, multicast, distributed large scale cashing
  • Sun Solaris 9
  • AIX 6
  • Red Hat LINUX 9
  • Ubuntu 10

WebSphere eXtreme Scale Installation

For a while, this blog will switch direction to WebSphere eXtreme Scale 7.1 because I will help with an IBM Red Book effort on this.

Installation of WebSphere eXtreme Scale 7.1 can be standalone or work with WebSphere Application Server 7.1.

After download the software, you have to set HOME_JAVA environment variable. On Windows platform, it is a bit tricky because the WebSphere eXtreme Scale batch files do not like spaces in directory and file names. For example, the following will not work.

set JAVA_HOME=C:\Program Files (x86)\Java\jdk1.6.0_23

When you launch runcat.bat, you will have error messages complaining about 1) you need to define JAVA_HOME, 2) c:\Program is a not an internal or external command

The following will be acceptable to WebSphere eXtreme Scale batch files.

set JAVA_HOME=C:\Java\dks1.6.0_23

Then, after you launch runcat.bat at install_root\ObjectGrid\gettingStarted\, you will see below. Your WebSphere eXtreme Scale system is ready.

WebSphere eXtreme Scale ready to process requests

WebSphere Installation Factory to produce CIP and IIP

WebSphere Installation Factory is a tool to produce installation packages - customized installation package (CIP) or integrated installation package (IIP) to install WebSphere software products.

The objective of this tool is to reduce complexity and improve productivity of WebSphere product installation.

Tuesday, March 29, 2011

Centralized Installation Manager (CIM)

CIM is a tool to allow remote installation of WebSphere Application Server software (release, refresh pack, fix pack, and interim fixes) without logging into each node.

"The CIM does not take the place of Update Installer or Installation Factory for
WebSphere Software. CIM pushes the product binary files or maintenance to the
remote targets and invokes the standard installer or update installer tool to
perform the installation or apply updates on the target systems."

Unlike update installer, CIM is part of WAS 7 Admin Console. It uses update installer by keeping is as part of the installation package and getting it installed on the node to use for product installation.

CIM is new feature with WebSphere Application Server 7.0.

For more details, refer to IBM Centralized Installation Manager (CIM) red paper.

WebSphere Product Updates

What product updates
  • Release - a major version for WebSphere, for example, 7.1 (the first two digits)
  • Refresh pack - an update with both fixes and new functions, for example, 6.0.2 (the third digit)
  • Fix pack - fixes only, for example, 6.0.1.2 (the fourth digit)
  • Interim fix - an update to fix a specific problem or a number of specific problems
Product update strategy

Product updates are expensive engineering tasks because the extensive testing required. Therefore, a balanced must be achieved between the need to run a relative updated WebSphere product and resource and production environment stability consideration.

To get on the most recent release may not be the best idea. Any software needs time to mature. The current release minus one might be a sensible approach. 

Usually, two scheduled WebSphere product updates a year is adequate. These two may be a major releases, or refresh packs or even fix packs. It depends on many factors such as the WebSphere technology road map of your company as well as need of the JEE applications and the specifics of your enterprise engineering practice.

The technical process
  • Defining versions and updates as well as technical objectives
  • Use versionInfo.sh or versionInfo.bat determine the current product version
Version Information Report

  • Go to IBM WebSphere support site to locate the desired product update
  • Get the product update desired
  • Stop the servers and backup the configuration by using backupConfig.sh or backupConfig.bat
  • Extract the files into the WAS product root 
  • Install update installer from IBM Support Site
    • Download the proper update installer
    • Install the update installer - go to product root, for example c:\Program Files (x86)\IBM\WebSphere\UpdateInstaller
    • Launch install - is:javahome C:\Program Files (x86)\IBM\WebSphere\JDK\jre.pak\repository\package.java.jre\java
    • Then, install the installer by invoking the install.bat program again
  • Run the installer 
  • Restart all WAS processes - the servers on this node and Dmgr
  • Check the systemOut.log to see if the server starts cleanly
  • Repeat the same for other servers on the cell

Monday, March 28, 2011

Tivoli Performance Viewer

TPV is a useful tool, especially the charts for web container and JVM, among other. Here are some teaser questions:
  1. What are the PMI monitor levels?
    1. None
    2. Basic
    3. Extended
    4. All
    5. Custom
  2. What is the default setting?
    1. Basic
  3. Where to go for setting the PMI monitor levels?
    1. Use Admin Console - Performance Monitoring Infratructure (PMI) > server_name
    2. Use WSAdmin
  4. Where is the PMI data log file and what is the name of this log file?
    1.  Monitoring and Tuning > Peformance Viewer > Current Activity > server_name > Settings > Log
    2. then, you can define_log_file_name or use default that is "tpv.xml"
  5. What you can view by using TPV?
    1. DynaCache
    2. EJBModule
    3. EntityBean
    4. JDBCProvider
    5. J2CResourceAdapter
    6. JVM
    7. MessageDrivenBean
    8. ORB
    9. Server
    10. SessionManager
    11. StatefulSessionBean
    12. StatelessSessionBean
    13. SystemMetrics
    14. ThreadPool
    15. TransactionService
    16. WebModule
    17. Servlet
    18. WLMAppServer
    19. WebServicesService
    20. WSGW
  6. Can you use TPV to view sysem.log file?
    1. No, you cannot. TPV can view TPV performance data in tpv.xml file
  7. When you use TPV and when you use ITCAM or Introscope?
    1. When you need to know which ojbect in JVM is the problom
    2. When you want to track transaction

Sunday, March 27, 2011

WebSphere troubleshooting

Recently I had the good fortunate to meet with a very senior WebSphere system engineer who works for a large financial services company. The discussion focused on WebSphere troubleshooting. Here are the notes that  I took from the conversation.
  1. Have an end-to-end view in WebSphere troubleshooting, from browser all the way to the backend system.
  2. First, test JVM to see if it is working. Make sure that the JVM is up and running and there is no hang thread. Turn on verbose GC and look into system log and native_std.log for JVM related error message.
  3. From the browser, to be if the URL is working. If the return code is 500 internal error, this may be a JVM or plugin issue. If the return code is 404 page not found error, it may well be a web server problem. 
  4. Try to browse into the transport port of the web server and application server directly. If there URL works, then, you can exclude the web server and application server from the troubleshooting scope.
  5. Use "telnet server_name port_name"   to test network connectivity and server status or test other components of the system, for example MQ server with a port number of 1470.
  6. Look into the access log of the web server to see if any request has actually made to the web sever and not got stuck with the 3DNS or BIG IP. Also look into error logs to see if there are any plugin problems and SITEMINDER issues.
  7. If there is high CPU, usually it is bad application code.
  8. If there is high memory consumption, create heap dump with kill -3 helps. You can ship the dump to IBM for analysis if your work station does not have enough memory to run the Support Assistance suite of tools.
  9.  Check connection pool - a frequently seen problem is a bug in the JEE code that does not close the connection after using. This causes a connection leak. Use "telnet server_name 446" to examine the network connectivity between the WebSphere Application Server and the backend systems. This will also tell you if the server is actually up and running. Sometimes, the piling up of connections is due to a connectivity issue. Use TPV, Introscope, or ITCAM to inspect the connection pool as well as examine system log for connection timeout. 
  10. It helps tremendously if you have transaction monitoring capability.  Then, you know exactly where the transaction got stuck or slows down. Introscope provides this capability, though you need in-depth expertise in Introscope that takes time to build.
  11. The capability to monitor user experience and transaction is critical in troubleshooting.

Saturday, March 26, 2011

Performance Tuning

What is performance tuning?
Performance tuning is turn WebSphere an extensive number of "knobs" and parameters that you can use to enhance an application’s performance according to the specific needs of each application.

When to tune?
Do not try to fix anything that is not broken. Only do performance tuning at performance testing or when there is a performance problem identified.

Performance tuning strategy?
Have a holistic view of the WebSphere system from the geographical load balancer all the way to the back

What to tune 
What to tune depends on the specific JEE application. The most frequently tunable areas are the following.
  • JVM, especially for memory bound applications. Ensure that the heap size is large enough but not too large to cause a long pause. There is not fixed "gold rule" for heap size. For example, I have seen heap as large as 2GB that works well. It depends on the application. For application that has many small and short lived objects, you may want to experiment on having a large "nursery". Even though for a 64 bit system, there is theoretically no upper limit but the constraint of available physical memory, too large a heap certainly can cause problems due to the "stop the world" compact cycle of garbage collection. 
  • EJB container - EJB thread is a custom property. You want to use ITCAM or Introscope to detect thread usage and tune accordingly.
  • Web container for web applications - web container thread is the most common bottleneck, especially for high load systems. You have to watch for creeping increase of load. For a very stable WebSphere system, the stead but slow increase of load over the years may eventually become unstable. When thread saturation occurs, it may manifest itself as a JVM heap problem because of within the JVM, large number of requests are piling up. Increase of vertical clustering has been an effective means that I used over the years to fix this problem. Most of the application code supports vertical clustering, with the exception of  the rare situation where the application code uses a unique counter or software router kind of Java artifact. Quite a number of applications have logic that do not support horizontal cluster. For this type of JEE application, vertical clustering is the only means to achieve a level of inter-JVM fail-over and increase of thread processing capability. It is very helpful to work with the application development team not to design and code JEE applications that do not support vertical and horizontal clustering.
  • JDBC connection pool - a usually problem is the too low a setting for the load, if you see a number of waitForConnection exception and timeouts in the log files. Now it is time to test increased connection pool size. 
  • JMS - the number of connection, the max retry value, and the number of messages in a session are among the tunable parameters.
  • OS - The ulimit value, the network (MTU), and amount of memory, and the CPU allocation, among others, contribute to the performance of WebSphere Application Server.
  • Web Server - are you running in worker mode where you can take advantage of web server threads or refork mode where the web server process forks to serve each request? Thread settings, timeout values, and the location of the web server impact the performance of the WebSphere system. Look for warnings about reaching MaxClients. Here is a useful web server tuning guide on WAS 6 .
What are the "knobs" and parameters?
  1. JVM heap size
  2. Thread pool size
  3. Connection pool size
  4. Data source statement cache size
  5. ORB pass by reference 
  6. Servlet caching
JVM heap size 
  1. The increase of the heap size should be balanced with the time and pause needed for garbage collection. 
  2. The high (maximum) and low (minimum) heap settings should be equal to prevent the dynamic heap size adjustments.
  3. Default of 50 and 256 MG is usually inadequate (it is wrong to take these default settings as optimal or IBM recommended settings. For some applications, a relatively small heap may work the best while for others, a very large heap has the best performance. It is purely a matter of performance testing and tuning).
  4. The right heap size tuning can only be achieved via testing (turn verbose garbage collection on when testing).
  5. Using free heap after collection to isolate memory leak.
  6. garbage collection policy is a main GC tuning parameter. 
  • optthruput: (default) mark and sweep during garbage collection when the application is paused to maximize throughput.
  • optavgpause: mark and sweep while the application is running to minimize pause times to get the best response time.
  • gencon: manage short-lived and long-lived objects differently to provide a combination of lower pause times and high throughput.
    Thread pool size
    Minimum size The minimum number of threads that the contain will keep in the pool. After the number is reached with thread adding into the pool, this minimum pool is kept regarding they are busy are idle.
    Maximum size The maximum number of threads to maintain in the thread pool. To set this too high can cause JVM resource issues and halt the application
    Thread inactivity timeout The amount of inactivity (in milliseconds) that should elapse before a thread is reclaimed. A value of 0 indicates not to wait, and a negative value (less than 0) means to wait forever.

    Connection pool size 
    Making connection and tearing up the connection is resource intensive. A connection pool allows the reuse of a connection to improve performance.


    Minimum connections The minimum number of physical connections. If the size of the connection pool is at or below the minimum connection pool size, an unused timeout thread will not discard physical connections. The pool does not create connections only to maintain the minimum connection pool size.
    Maximum connections The maximum number of physical connections possible for this pool. If this number is reached, no new physical connections are created; requestors must wait until a physical connection returned to the pool, or until a ConnectionWaitTimeoutException is thrown, based on connection timeout. Too high a maximum connections value can stress or even overwhelm the back-end.
    Thread inactivity timeout The amount of inactivity (in milliseconds) before a thread is reclaimed. A value of 0 = no wait, and a negative value = wait forever.


    Data source statement cache size
    Data source statement cache size is for improve the performance of prepared statement and callable statement. Try to get the number of these statements and make the size equal to that statement. Then, test and increase the size till you see no discarded statement. This is how to adjust this knob. 

    Data sources > Derby JDBC Driver XA DataSource > WebSphere Application Server data source properties.

    ORB pass by reference
    This is the same like in C++. Pass by value method creates a new copy of the object. This method is more costly than pass by reference.Using the following panel to change this.

    Servers > Server Types > Application servers > server name > Container services > ORB Service 

    Servlet caching
    Use DynaCache to cache fragments for Servlet can improve performance. Servlet caching can be enabled in the administrative console by navigating to Servers => Application servers => server_name => Web container settings => Web container.

    Tools
    TPV, Introscope, ITCAM all provide help in performance tuning. However, without real time capability to monitor and measure all the way from browser to the backend 1) user experience, 2) transaction, your ability to fully understand what is going in the system is limited.

    Thursday, March 24, 2011

    WebSphere backup and restore

    1) Backup and restore configuration file or profile: please see this link by Sunil Patel.
    2) Backup a profile: manageprofiles.sh -backupProfile -argument

    Session management tuning

    Session tuning has three objectives to me.
    • performance (Write frequency, Scheduled invalidation, Tablespace and row size)
    • Special characterics desired (Session affinity, Write contents, Multirow schema, Session object size, Session cache size, persistent store IO,  )
    • problem avoidance (session replication, session overflow, session security, session serialization)

    Tuesday, March 22, 2011

    Application Binding

    What is application binding?

    Application binding could be viewed as a system configuration task. The purpose of the task is to inform WAS where is the physical artifact for a specific name. The mechanism for making this connection is JNDI and some configuration files of an application (ibm-xxx-bnd.xml or ibm-xxx-bnd.xmi).The xml suffix is for WAS 7 and xmi suffix for earlier WAS releases.

    What to bind?
    • EJB
    • Data Source for EJB
    • Backend ID for EJB model
    • MDB
    • Resources
    • JDBC
    • URL
    • JMS connection factory
    • Java Mail
    • Virtual host for web model
    • Message destination references
    When to bind?
    • Development time
    • Assembling time
    • Deployment time
    • Configuration time