Apache, Plugin, and WebSphere Application Server 7 on LINUX

Install apache 2.2 on Linux.

sudo apt-get install apache2

Download and install the plugin.

/etc/apache2/apache2.conf includes the /etc/apache2/httpd.conf file.

# Include all the user configurations:
Include httpd.conf

/etc/apache2/httpd.conf file includes the following codes. 

• LoadModule was_ap22_module /opt/IBM/WebSphere/Plugins/bin/mod_was_ap22_http.so
• WebSpherePluginConfig /opt/IBM/WebSphere/Plugins/config/apache01/plugin-cfg.xml

Note that it is was_ap22_mode, not was_ap20_mode. Otherwise, you may have an error complaining about a socket address format.

The .so file is the actually native code module performing the plugin function. The LoadModule directive is to load the native code at the initiation of the plugin proram. The second directive is telling the web server and plugin program where to find the configuration of the plugin program.

For example, when the web server gets a request:

http://DFAPP01.com/hello

The request is sent to the web server. However, as soon as the plugin is installed, according to IBM documentation, the plugin examines all the incoming requests for the web server and forward requests for static contents to the web server and dynamic pages to the app server. The plugin sends right page back the requesting browser.

Two more notes -

1. Installing IBM HTTP server 7 can encounter Java incompatibility, you may have to run the following, sometimes, without success. 

   java -cp `pwd`/setup.jar -Xms48m -Xmx384m run

2. Also, in Windows, running genPluginCfg.bat may run into a run time exception PLGC0033E that requires you to open an IBM PMR to manage.

"PLGC0033E: A run-time exception occurred while generating the plug-in configuration:
Explanation: A problem resulted from generating the configuration file.
User Response: Examine the exception for nested exceptions and error codes. Rerun the GenPluginCfg script with the -debug option set to yes (GenPluginCfg -debug yes), to see trace details. If the problem persists, see the problem determination on the WebSphere Application Server Support Web page on http://www.ibm.com/software/webservers/appserv/was/support/."

You can test the web server and web server plugin installation and cnfiguration, by using browser to bring up WAS default applications via the web server default transport 80. http://DFAPP01.com:80/hello

Star, stop, and restart and configuration change

sudo /etc/init.d/apache2 stop
sudo /etc/init.d/apache2 start
sudo /etc/init.d/apache2 restart

Recycle the application server after web server plugin changes.

Trace facilities (enabling and selecting components and logging configuration)

Question 1 - how many log files that one WebSphere Application Server uses?
Question 2 - how many set of trace or error message panels there are on the admin console?
Question 3 - the differences between JVM logs and Process logs?
Question 4 Which logs are the most important or the most frequently used in troubleshooting?

Answer 1 - 7 log files

• Diagnostic Trace (trace.log)
• JVM Logs (SystemOut.log, SystemErr.log)
• Process Logs (native_stdout.log, native_stderr.log)
• IBM Service Logs (activity.log - runtime events and can only be viewed by Log Analyzer)
• NCSA access and HTTP error logging (http_access.log)

Answer 2 - three set of panels and each set has three panels

Troubleshooting

• Logs and trace
• Configuration problems
• Class loader viewer

Configuration Validation

• Configuration error
• Configuration warning
• Configuration information

Diagnostic Provider

• Tests
• State Data
• Configuration Data

Runtime Messages

• Runtime error
• Runtime warning
• Runtime information

 Answer 3 - JVM logs have trace or error messages from the JVM and Process logs contains trace information and error messages from native code (native code is non-Java code, typically found in files with .dll, .exe, and .so extensions).

Answer 4- the JVM log files are the most frequently used in monitoring the server and troubleshooting if not the most important as well.

Here is an IBM Red Paper on tracing and logging - a very good one (some contents, for example, the collector tools are bit old. The current method is us AutoPD, not the collector. However, I am sure many are still using the collector to do MustGather).

Web server has the following logs files as well as the web server plugin.
– Access log: \logs\access.log
– Error log: \logs\error.log

dumpNameSpace

What kind of typical problems in troubleshooting involves using "dumpNameSpace.sh"?

sudo ./dumpNameSpace.sh -root server -port 2801

Enabling Verbose GC

Use these panels - Application servers > ClstAppSrv01 > Process definition > Java Virtual Machine, then, check the following

Verbose garbage collection 

Of course, you can always use wsadmin 

AdminTask.setJVMProperties('[-nodeName D-F-02Node01 -serverName ClstAppSrv01 -verboseModeClass false -verboseModeGarbageCollection true -verboseModeJNI false -runHProf false -hprofArguments -debugMode false -debugArgs" -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777" -executableJarFileName -genericJvmArguments -disableJIT false]')

How can I learn WebSphere Application Server 7 Network Edition?

You can download an evaluation copy of WAS 7 Base. However, IBM does not offer an evaluation download for WAS 7 ND. This poses a problem for those who want to learn WAS 7 ND. How to solve this problem?

My recommendation is to use the free download of  IBM WebSphere Application Server Network Deployment 8.0 Beta.  WebSphere technologies are mature and stable. Even WAS 8 ND (Beta) release has reached a high level of stability. It has all the WAS 7 ND functions and more. I have found that WAS 8 ND (Beta) very helpful in not only learning how to use WAS 7 ND, but also practicing on the flexibility management features of WAS 8 ND with Job Manager and Administrative Agent.

Here is the download site for WebSphere Application Server 8 ND (Beta) for Windows.

Generate Dumps

1) Use wsadmin jython code for thread dumps (be careful that indention is meaningful in jython)

------------------------------------------------------------------------------------------------------
# define a function that will find the JVM name and generate the dump
# the name of the jython program is wasgeneratethreaddump.py 

     def generateThreadDump(serverName):

     serverJVM = AdminControl.queryNames("type=JVM,process="+serverName+",*")

AdminControl.invoke(serverJVM,"dumpThreads")
# call the function
generateThreadDump("ClstAppSrv02")
-------------------------------------------------------------------------------------------------------

Use the following in wsadmin to call the jython program (windows):

wsadmin -lang jython -f  wasgeneratethreaddump.py

Then, you should see the dump file. You can use IBM Support Assistant to view and analyze the dump.

2) heap dump

A. Use admin console to set up automatic memory dump - this is only for testing and development OR when you have a severe production problem, because the generation of dump will cause the production system an actual pause. Use the following sequence of panels, then, check "Enable Performance and Diagnostic Advisor Framework (Runtime Performance Advisor)" 

Application servers > ClstAppSrv01 > Performance and Diagnostic Advisor Configuration

B. Use wsadmin and jython to generate a heap dump

------------------------------------------------------------------------------------------------------
# define a function that will find the JVM name and generate the dump
# the name of the jython program is wasgeneratethreaddump.py
def generateThreadDump(serverName):
     serverJVM = AdminControl.queryNames("type=JVM,process="+serverName+",*"
     AdminControl.invoke(serverJVM,"generateHeapDump")
# call the function
generateThreadDump("ClstAppSrv02")
-------------------------------------------------------------------------------------------------------

3) Of course, both can be generated from using kill -3 taking JVM  PID as argument in UNIX/LINUX environment (be careful - sometimes there are anomalies - once I got into trouble issuing kill -3 because of a bug in a monitoring tool. Generally speaking, wsadmin and jython is a safer route)

java dmp
javacore txt
Snap trc

SOAP/HTTP or SOAP/JMS

SOAP/HTTP

• For external endpoints
• Firewall friendly (web services exposed over internet)
• Supported on all platforms
• Clients can be simple and lightweight
• Dynamic caching (to improve performance)

 SOAP/JMS

• Intranet endpoints
• Assured delivery and only once delivery
• Publish and subscribe and asynchronous support
• Better scalability and reliability
• Marginal better performance
• Better support in middleware software
• Transaction support

WAS 7 and Web Services Standards Support

Keep the following for quick reference.

WS-I Basic Profile 1.1 - this is about interoperability and WSDL. It provides interoperability guidance for core Web Services specifications such as SOAP, WSDL, and UDDI. The profile uses Web Services Description Language (WSDL) to enable the description of services as sets of endpoints operating on messages.

JAX-WS - The Java API for XML Web Services (JAX-WS) is a Java programming language API for creating web services.

Java API for XML-based RPC (JAX-RPC) allows a Java application to invoke a Java-based Web Service with a known description while still being consistent with its WSDL description. It can be seen as Java RMIs over Web services. JAX-RPC 2.0 was renamed JAX-WS 2.0 (Java API for XML Web Services). JAX-RPC 1 is deprecated with Java EE 6.^[1]

UDDI V3 - Universal Description, Discovery and Integration is a platform-independent, Extensible Markup Language (XML)-based registry for businesses worldwide to list themselves on the Internet and a mechanism to register and locate web service applications.

WS-I Security - WS-Security (Web Services Security, short WSS) is a flexible and feature-rich extension to SOAP to apply security to web services. It is a member of the WS-* family of web service specifications and was published by OASIS.

WS-Transaction - WS-Transaction - a Web Services specification developed by BEA Systems, IBM, and Microsoft. The WS-Transaction specification describes coordination types that are used with the extensible coordination framework described in the WS-Coordination specification. It defines two coordination types: Atomic Transaction (AT) for individual operations, and Business Activity (BA) for long running transactions.

WebSphere Concepts - Groups and Clusters

It is sometimes confusing about all the WebSphere groups - node group, default group, core group, cluster, and dymanic cluster. Here is a quick summary.

node group -  a group of nodes of similarity within one cell. For example, a cell can have a two node groups of zOS and one node group of nodes on distributed platform. To me, the node group provides logical grouping of application servers and made system administration easier.

default group - usually, on WAS ND, if you do not specify node groups, all in one cell belongs to a defaultnodegroup.

cluster - A cluster is a collection of servers managed together. Clusters behave as a single application server entity to accomplish a job by parallel processing. The intent of clustering is high availability and failover. A frequently ignored important aspect of clustering is that it can be effectively used to increase system resources to a set of applications in order to improve throughput and overall JVM level capacity.

• all cluster member must be within the same cell
• all cluster member must be in the same node group
• a cluster member can only belong to one cluster
• a cell can have one ore more clusters
• all members must run the same set of applications

core group - a group of clusters in a high availablity envrinment. All of the application servers of a cluster included in a core group are automatically members of that core group. Core groups enables Websphere Application Server to provide high availability for applications  You can also configure core groups to communicate with each other using the core group bridge. The core groups can communicate within the same cell or across cells. Core group works with high availability manager and communication bridges. The intent of core group is high availability.

dynamic cluster - A dynamic cluster is a concept and mechanism used by WebSphere XD to manage a dynamic WebSphere environment. It monitors performance and load information and is able to dynamically create and remove cluster members based on the workload. The intent of dynamic cluster is dynamic provisioning of JVM as well as application edition management.

IBM Cloudburst, IBM WebSphere Cloudburst Appliances, and IBM WebSphere Application Server 7 Hypervisor Edition

The IBM Cloudburst is a fairly large server - looks like a highend mid-range server. If you do not want to run hypervisor on "general purpose" hardware, the IBM Cluadburst is where you want to go.

The IBM WebSphere Cloudburst Appliance is a thin box that looks like a DataPower appliance. It has WebSphere Applications server images and WAS topology patterns as well as system management tools. You can deploy these to IBM Cloudburst or virtualization ready servers (servers running hypervisor).

IBM WebSphere Application Server 7 Hypervisor Edition is the Websphere server that can be deployed and execute in a cloud.

A metaphor - what is a "bare metal" hypervisor?

To call "bare metal" hypervisor a hypervisor is not as good as calling it "hardware simulator", because to the guest operating systems, the "bare metal" hypervisor is simulated hardware.

WebSphere CloudBurst Appliance

Cost in IT acquisition and operation costs are cited in WebSphere CloudBurst Appliance advantages. Lowering costs are extremely important.

However, the most compelling motivation to use WebSphere CloudBurst Appliance may very well be reduced time to market. To design, build, configure, and test WebSphere infrastructure is a time consuming process, not to say that you have to take into consideration that acquisition of servers take time, especially for large corporations.

Anything that can help a project team to reduce the time to get on a working middleware infrastructure help way beyond reducing costs only. This is perhaps the greatest value of WebSphere CloudBurst Appliance.

In the world of large new WebSphere technology deployment, we need both "Agile in a Bird Case" and "Anarchy Led Engineering Discipline".

In the world of large scale new WebSphere deployment (development and engineerning), I think that we need to both loosing down and tightening up with new approaches.

In the area of development, the goal of working software has to take priority with Agile methodology leading the way. However, a compromise is needed in a very large and heavily regulated environment where documentation and approval process must be provided due to legal and privacy concerns. All the following Agile objectives remain true, while new adaptive approach can help.

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

However, the complete lack of long-planning is not possible for any modern IT projects, even we fully appreciate and adopt Agile Methodology. Maybe a "bird case" approach is in the Agile based projects, especially those taking on totally new technology and unchartered territories. An overall long-term project is in place that cut into manageable pieces - "bird cases" within which the bird - Agile based development fly freely. Hopefully, this will achieve the need for control large projects and taking the full advantage of Agile and Scrum methodology.

On the other hand, the infrastructure engineering must follow tighter design procedures, change process, operation manuals, and engineering standards. This is because the malfunction of infrastructure causes potentially heavy loss and disruption as well as the nature of infrastructure - it is time consuming and costly to fix some of the infrastructure issues, for example, change the design of your middleware in the middle of its production operations. At the same, a gradual, POC based approach works better for new technology adoption that will help with the following. The introduction of new technology demands a level of anarchy and a degree of chaos for the team to figure out what to do. However, this intended anarchy and purposeful chaos must lead to the following.

• Building the team's technical skills over the new technology
• Creating and documenting engineering processes - the foundation of engineering process optimization and automation
• Nurturing internal and external working relationships that are critical in designing, building, and operating the new infrastructure
• And the most importantly, delivering a gradually maturing IT infrastructure in support of the business drivers

Therefore, we have to seriously consider Agile and Scrum in  large new WebSphere technology development and deployment while making substantial adjustment to achieve success for providing a new IT paradigm of innovation for new business imperatives and objectives.

Job Manager is a bless

What is Job Manager? In a simplistic way, it is your centralized WebSphere "crontab". For example, you can schedule a weekly recycling of many clusters and servers. Then, the job manager will get the job done and send you an email.  What a bless for system engineer!

Add nodes to an administrative agent

To add nodes to an administrative agent, you have to use \AppServer\bin\registerNode command. The following is an exmaple. Note: this is NOT a wsadmin command!

set WAS_HOME=C:\program files (x86)\IBM\WebSphere\AppServer

set adminAgentProfileName=AdminAgent01
set adminAgentHostName=localhost
set adminAgentSoapPort=8877
set baseProfileName=AppSrv05

cd %WAS_HOME%\profiles\%adminAgentProfileName%\bin
registerNode.bat -profileName %adminAgentProfileName% -profilePath "%WAS_HOME%\profiles\%baseProfileName%" -host %adminAgentHostName% -conntype SOAP -port %adminAgentSoapPort%

Look for port numbers?

Look for port numbers for a JVM? Go to the config directory and look for serverindex.xml.
For example, if you look for the admin console port for an administrative agent, go to the below sub-directory.

\profiles\AdminAgent01\config\cells\DF-02AACell01\nodes\DF-02AANode01\serverindex.xml

Then, you find the port number, for example, the port to access admin console. Now it is possbile to access the admin console. Let me if you have a better way such as using wsadmin.

Messaging - What is Message Driven Bean?

What exactly is Message Driven Bean (MDB)

1. From the perspective of Java artefact, it is an EJB specified by EJB 2.1. It is a stateless session bean.
2. It is a consumer in terms of the messaging programming model.
3. It implements javax.ejb.MessageDrivenBean (container life-cycle management).
4. It implements javax.jms.MessageListener interface.
5. It implements ejbCreate() method.
6. The EJB container maintains a pool of method-ready MDB.
7. When a message comes, the container selects a MDB from the pool to process the message.
8. The container removes an MDB by invoking ejbRemove() method.
9. The transaction can be container managed or bean managed.
10. To associate an MDB with destination on SIB, use JMS activation specification.
11. To associate an MDB with a destination on MQ, use listener port.