Configure Email Notification in WebLogic Server

Introduction

If you are administering applications deployed on WebLogic Server or if you are operations personal responsible to manage a WebLogic Server environment you might have at least asked the following question once:

“Can I setup WebLogic Server to send automatic notifications?”

People always ask me whether WebLogic Server can send automatic notifications. My answer is - "Yes, WebLogic Server had always supported SNMP which can be configured to do standard SNMP Trap notifications to your enterprise monitoring system which can then be sent as an email or pager out to administrators". But the next question is - "I don't want to set up SNMP and I want WebLogic Server to directly send notifications. Can we do that?" Again the answer is - "Yes, starting from version 9.x WebLogic Server supports a new framework called WebLogic Diagnostic Framework - WLDF, which can be used for this purpose". Enterprise monitoring solutions like Oracle Enterprise Manager, HP Open View can monitor WebLogic Server from outside and can trigger notification among various other management and monitoring features. But you can also set up WebLogic Server to directly send many types of notifications. Let us see how to simply set up an email notification with WLS in this article.

Oracle WebLogic Server is Java EE certified application server which is the foundation of many Service Oriented Architecture (SOA) implementations. So many enterprises have their mission-critical applications deployed on WebLogic Server. WebLogic provides a scalable and highly available environment. If you are deploying your mission critical applications on Oracle WebLogic Server you can make use of WebLogic Diagnostic Framework (WLDF) to get diagnostic data from WebLogic Server instances and from applications deployed to them. This will help administrator and system operators to see more information from under the hoods.

This powerful feature will play a significant role in production environment where the application environment should alert operational personal when there is a performance or service level violation. This also eliminates the need to setup custom or third-party monitoring solutions for alerting and notification.

WebLogic Diagnostics Framework – Watch & Notification

One of the features of WLDF is Watch and Notification. You can configure server(s) with watch condition and respective notification(s) to be sent if the watch conditions are met. For example, if the number of request waiting on a JDBC Data Source to grab a connection exceeds 10 and if the free heap in the server is less than 5% then you can trigger a notification. The notification can be one or more of the following types:

• JMX Notification - Application(s) can register notification listener with WLDF to receive notifications.
• JMS Notification - WLDF can post a message to JMS destination to alert the situation which can be consumed by a message listener or MDBs to further processing.
• SNMP Notification - SNMP traps can be sent to alert SNMP managers.
• SMTP Notification - Email notifications through a Java Mail session.
• Image Notification - Generates a server image which contains information from different subsystem during the watch.

Figure: 1 WebLogic Domain

You can configure Watch conditions on the MBean data, Log messages and WLDF Event data. The notification can be one or more of the above notifications. Since email notification is a common request, I shall illustrate how to setup WLS to send email notification using an open source SMTP server - The Apache Java Enterprise Mail Server (Apache James). To download Apache James, go to - http://james.apache.org/server/ [or] http://people.apache.org/dist/james/server/binaries/. Download and extract the zip file to a convenient location (Eg. /usr/james-2.3.1rcl/). Start the server using run.sh from the bin folder under the JAMES_HOME. Setup a test user in James as per these instructions - http://james.apache.org/server/2.3.1/adding_users.html. The default admin username and password for James are root/root.

Ensure that the James mail server and the administration server for the domain are started. If not, use DOMAIN_HOME/startWebLogic.cmd/sh for starting the administration server of your domain and use JAMES_HOME/bin/run.bat/sh to start James Server. You can use console or WLST to configure the following setting. Let us use the web-based administration console for configuration. Log in to WebLogic Administration Console at - http://localhost:7001/console using an administrative account.

Creating JavaMail Session

First step is to make sure that the underlying resources for the notification are setup. In this case we are going to use SMTP notification so let us first create the Java Mail Session to connect to James Mail Server.

1. Click "Lock & Edit" to acquire a configuration lock
2. Expand Services > Mail Sessions and click "New" to create a new Java Mail Session.
3. Provide Name, JNDI Name and JavaMail Properties as follow:
4. Name - James-MailSession
5. JNDI Name - com.test.myjamesMailSession
6. JavaMail Properties:

mail.transport.protocol=smtp

mail.smtp.host=localhost

mail.smtp.user=admin

mail.smpt.password=password

7. Click next to target the mail session to a server (Eg. AdminServer or mgd_s1) and click finish.
8. Activate the changes by clicking "Activate Changes".

Figure: 2 Configuring JavaMail Session

If you are using a different mail server use appropriate information.

Creating WLDF Module

Next step in configuring watch and notification is to create a diagnostic module and target it to the server. You can use console or WLST to create a diagnostic module. The following steps illustrate the steps to create a WLDF system module:

1. Click "Lock & Edit" to acquire a configuration lock
2. Click Diagnostics > Diagnostics Modules and click "New" to create a new diagnostics module. Provide a name and description.
3. Name - Test_WLDF_Module
4. Description - This is a WLDF module for testing email notifications

5. Select the newly created module and select targets tab. Select the appropriate server for target as before
6. Click "Save" and "Activate"

Figure: 3 Configuring Diagnostic Module

Creating Watch & Notification

Each WebLogic Server instance can be configured with only one diagnostic module but you can target the same diagnostic module to multiple servers or clusters. Once the WLDF system module is created and targeted to the appropriate server then watches and notifications can be configured. The following steps illustrate how to create a Watch and a Notification.

To create a watch:

1. Click "Lock & Edit" to acquire a configuration lock
2. Navigate to the diagnostic module created above (Test_WLDF_Module) and select the "Watches and Notifications" tab and "Watches" sub-tab (if not already selected)
3. Click "New" to create a new Watch. Provide the name and select the type as "Collected Metrics" [Select "Collected Metrics" for inspecting Runtime MBean values, "Server Log" to watch log data and "Event Data" to watch instrumented data]
4. Name - Heap_Watch
5. Watch Type - Collected Metrics
6. Enable Watch - Selected
7. Click "Next"
8. Click "Add Expressions"
9. Ensure that "ServerRuntime" is selected and click "Next"
10. If you are running WebLogic Server on Java HotSpot VM, make sure "Select an MBean Type from the following list" is selected and select "weblogic.management.runtime.JVMRuntimeMBean" and click "Next". If you are using JRockit VM, make sure "Select an MBean Type from the following list" is selected and select "weblogic.management.runtime. JRockitRuntimeMBean" and click "Next".
11. Select the instance for the appropriate server from the list for "Instance" and click "Next"
12. For "Message Attribute" select "HeapFreePercent", "<" for "Operator" and type "85" for "Value". Click "Finish" twice to create a Watch
13. Activate the changes

Figure: 4-a Configuring Watch

Figure: 4-b Configuring Watch

The above watch will trigger the configured notification if the percentage of free heap is less than 90. You could make complex rule expression by combining multiple attributes from different MBean. For now we haven't configured any notification for this watch. We will create the notification and will associate it to the watch in the following steps.

To create a notification:

1. Click "Lock & Edit" to acquire a configuration lock
2. Navigate to the diagnostic module created above and select the "Watches and Notifications" tab and "Notification" sub-tab
3. Click "New" to create a new Notification
4. Select "SMTP (E-Mail) for Type and click "Next"
5. Provide a name - James_Email_Notification. Make sure the notification is enabled and click "Next".
6. Configure the following properties for the "Config Notification - SMTP Properties" page
7. Mail Session Name - James-MailSession
8. E-Mail Recipients - admin@localhost

9. Click "Finish".
10. Activate the changes

Figure: 5-a Configuring SMTP Notification

Figure: 5-b Configuring SMTP Notification

To associate the watch and the notification:

1. Click "Lock & Edit" to acquire a configuration lock
2. Navigate to the diagnostic module created above (Test_WLDF_Module) and select the "Watches and Notifications" tab and "Watches" sub-tab (if not already selected)
3. Select the watch you created earlier - Heap_Watch
4. Select the "Notifications" tab and move the "James_Email_Notification" from Available to Chosen
5. Click "Save"
6. Select the "Alarms" tab and select "Use an automatic reset alarm". Set the "Automatic reset period" to 6 and click "Save". This will ensure that the notifications are at least 90 seconds apart.
7. 5. Click "Activate Changes"

Figure: 6-a Associating Watch and Notification

Figure: 6-b Configuring Alarm

As we configured the watch type to be Collected Metrics, the attributes involved in the watch rule will be harvested and tested for watch rule expression(s). The default sampling period for the harvester is 300,000 milliseconds (or 5 minutes). So the time between samples will be 5 minutes. To change the sampling period for metrics collection:

1. Click "Lock & Edit" to acquire a configuration lock
2. Navigate to the diagnostic module created above (Test_WLDF_Module) and select the "Collected Metrics" tab
3. Ensure that "Enabled' check-box is selected
4. Change the "Sampling Period" as "120000" so that it will collect samples once every two minute

Figure: 7 Configuring Sampling Period

To receive the notifications sent by WebLogic Server as emails, install and configure an email client like Mozilla Thunderbird. The following is an example of the email received from WebLogic Server:

Using WLST

The configurations that are illustrated using Administration Console here can also be done using WLST. The following script is an example WLST script to configure a Javamail session, WLDF Module, Watch and Notification:

#Connect to the admin sever
connect('weblogic','Welcome1','t3://localhost:7001')
 
#Start a change session
edit()
startEdit()
cd('/')
 
#Create and configure a Javamail session
cmo.createMailSession('James-MailSession')
cd('/MailSessions/James-MailSession')
cmo.setJNDIName('com.test.myjamesMailSession')
cmo.setProperties({mail.smtp.user=user;, mail.smpt.password=password;, mail.smtp.host=localhost;, mail.transport.protocol=smtp;})
set('Targets',jarray.array([ObjectName('com.bea:Name=AdminServer,Type=Server')], ObjectName))
 
#Create and configure a WLDF Module
cd('/')
cmo.createWLDFSystemResource('Test_WLDF_Module')
cd('/SystemResources/Test_WLDF_Module')
cmo.setDescription('This is a WLDF module for testing email notifications.')
set('Targets',jarray.array([ObjectName('com.bea:Name=AdminServer,Type=Server')], ObjectName))
 
#Create and configure a WLDF Watch
cd('/WLDFSystemResources/Test_WLDF_Module/WLDFResource/Test_WLDF_Module/WatchNotification/Test_WLDF_Module')
cmo.createWatch('Heap_Watch')
cd('/WLDFSystemResources/Test_WLDF_Module/WLDFResource/Test_WLDF_Module/WatchNotification/Test_WLDF_Module/Watches/Heap_Watch')
cmo.setRuleType('Harvester')
cmo.setEnabled(true)
#For HotSpot VM
#cmo.setRuleExpression('(${ServerRuntime//[weblogic.management.runtime.JVMRuntimeMBean]com.bea:Name=AdminServer,ServerRuntime=AdminServer,Type=JVMRuntime//HeapFreePercent} < 85)')
#For JRockit VM
cmo.setRuleExpression('(${ServerRuntime//[weblogic.management.runtime.JRockitRuntimeMBean]com.bea:Name=AdminServer,ServerRuntime=AdminServer,Type=JRockitRuntimeMBean//HeapFreePercent} < 85)')
cmo.setAlarmType(None)
 
#Create and configure a WLDF SMTP Notification
cd('/WLDFSystemResources/Test_WLDF_Module/WLDFResource/Test_WLDF_Module/WatchNotification/Test_WLDF_Module')
cmo.createSMTPNotification('James_Email_Notification')
cd('/WLDFSystemResources/Test_WLDF_Module/WLDFResource/Test_WLDF_Module/WatchNotification/Test_WLDF_Module/SMTPNotifications/James_Email_Notification')
cmo.setEnabled(true)
cmo.setMailSessionJNDIName('com.test.myjamesMailSession')
set('Recipients',jarray.array([String('admin@localhost')], String))
cmo.setSubject(None)
cmo.setBody(None)
 
#Associate the watch and notification
cd('/WLDFSystemResources/Test_WLDF_Module/WLDFResource/Test_WLDF_Module/WatchNotification/Test_WLDF_Module/Watches/Heap_Watch')
set('Notifications',jarray.array([ObjectName('com.bea:Name=James_Email_Notification,Type=weblogic.diagnostics.descriptor.WLDFSMTPNotificationBean,Parent=[testdomain]/WLDFSystemResources[Test_WLDF_Module],Path=WLDFResource[Test_WLDF_Module]/WatchNotification[Test_WLDF_Module]/SMTPNotifications[James_Email_Notification]')], ObjectName))
cmo.setAlarmType('AutomaticReset')
cmo.setAlarmResetPeriod(60000)
 
#Set the sampling period
cd('/WLDFSystemResources/Test_WLDF_Module/WLDFResource/Test_WLDF_Module/Harvester/Test_WLDF_Module')
cmo.setSamplePeriod(120000)
cmo.setEnabled(true)
 
#Activate the changes
save()
activate()
 
#Exit
exit()

Would you like to have a view of WebLogic Server instances from a control tower ?!

If you have enterprise-wide management systems in place and are looking a way to integrate WebLogic Server instances then WebLogic Server SNMP is your option. You can use Simple Network Management Protocol (SNMP) to provide monitoring data from your WebLogic Server instances to your enterprise-wide management systems.

SNMP set of standards provides a framework for the definition of management information along with a protocol for the exchange of that information. The SNMP model assumes the existence of managers and agents. A manager is a software module responsible for managing part or all of the configuration on behalf of network management applications and users. An agent is a software module in a managed device responsible for maintaining local management information and delivering that information to a manager via SNMP. A management information exchange can be initiated by the manager (via polling) or by the agent (via a trap). Agents function as collection devices that gather and send data about the managed resource in response to a request from a manager.

WebLogic supports SNMP for a long time starting from version 5.1. WebLogic SNMP enables you to move towards a single management console and thus provide integrated systems management of WebLogic Server based applications. Also WebLogic SNMP enables you to connect WebLogic Server to popular management systems such as HP OpenView, IBM Tivoli, MRTG, and Sun SunNet Manager. This makes managing WebLogic Server more effective by providing a whole-system perspective instead of piecemeal solutions.

In WebLogic Server Version 10.x SNMP implementation went through a lot of changes. Some of the important changes are:

• WebLogic Server SNMP implementation now supports SNMPv3 with SNMPv1 and SNMPv2. SNMPv3 provides additional security compared to prior versions.

• Previously only one SNMP agent per domain which runs on the Administration Server now you can create multiple agents on different servers. This removes the dependency with Administration Servers.

• Now custome MBeans can be monitored using SNMP in addition to standard WLS MBeans.

• The old command-line utilities are not deprecated and a new command-line utility is introduced - weblogic.diagnostics.snmp.cmdline.Manager

• SNMP agents in WLS can now use TCP port in addition to UDP Port which was the only option in the previous versions. And many of the configuration changes in SNMP are now dynamic which doesn't require start.

For more information read the edocs - http://edocs.bea.com/wls/docs100/snmpman/snmpagent.html

With more and more system and server administrator coming to the paradigm of administering application servers and enterprise applications, this will provide them a way to integrate WebLogic with enterprise level monitoring solutions.

Is my Administration Server single point of failure?

The answer is No and Yes. Read on for more specific info....

WebLogic domain consists of Servers, Machines, Clusters, Applications and Configurations related to them. Administration sever is the controlling instance for the domain. It is the keeper of all the configuration. Ideally you don't want to host (target) any application or service that clients would access on the admin instance. Managed server(s) or cluster(s) are meant to host such applications and services that clients would access (servers other than admin server in a domain are called as managed servers).

Having said that, if an admin instance fails during the middle of the day what happens to my domain? It is the controlling instance so will it halt my domain? The answer is NO. Admin server will provide all the configurations to each managed server and they will be cached locally. There are limited dependency between the servers during runtime. So an admin server failure will not affect the running managed server(s) or cluster of managed server(s).

Alright, what about the "Yes" part of the above answer?! There are few services you will lose when an admin instance is not available. You cannot introduce any new configuration or change any existing configuration in the domain. Examples are: you cannot deploy new applications, you cannot change the JNDI name of a JMS destination hosted on one or more servers etc. You will also lose the domain log entries in the Domain Log file maintained by the administration server. Still you can use the Server Log maintained on each managed server for auditing or troubleshooting purpose. Also lastly the managed server(s) in the absence of admin server can only be started or restarted in Independence Mode (using the previously cached local configuration information).

• You are not expected to introduce new configuration or change any configuration on the fly in your production setup so losing the ability to do this will have literally no or little impact.
  
• You can use server log file entries for auditing and troubleshooting purposes.
  
• The managed server can be restarted or started if required during the absence of admin server using Independence Mode (which is a default setup now in WLS 10.x - no extra configuration required).

Admin server (for that matter any server in the domain) can be configured to automatically get restarted in case of failures using a feature called - whole server migration. I will talk about this in a later discussion.

So now you know why I said "No" and "Yes"! huh!

Managed Server Independence Mode

Starting from version 7 WebLogic managed servers can be started in the absence of an administration server. The feature that will enable a managed server to come up in the absence of administration server (MSI mode) is enabled by default. But some of the files required for MSI mode are only sent to the managed server only when another option (MSI file replication enabled) is ON.

This feature is same in version 8 except that the config.xml being copied and cached in the managed server is stored as msi-config.xml.

In version 9 the feature is still the same except that the complete config directory that contains the configuration file is copied by default to the managed servers. But some other files like the security (*.dat) file is not copied until the MSI file replication enabled option is ON.

Version 10 alleviate all the pains by making MSI mode and MSI file replication as a default option. So out of the box a managed server is enabled to run in independence mode. And all the files required to start a managed server by itself (/config dir and *.dat file) will be automatically copied to the managed server during startup and will be synced (automatically) with the master copy (in Admin Server).

Whats the deal with SNMPv1, v2 and v3

Here are some of the questions I got from someone few days ago. See my answers inline.

Q: The WLS SNMP agent supports 2 trap versions: V1, V2 and V3. What's the difference between these 3 versions?

[A] SNMPv1 and SNMPv2 are the two version of SNMP implementation. Until WLS 9.x only SNMPv1 and SNMPv2 are supported. Now we have SNMPv3 implemented in WLS 10. Both SNMPv1 and SNMPv2 use community strings to authenticate the packets as SNMP uses UDP (a broadcast protocol). The community string is sent in plain-text in every single SNMPv1 or SNMPv2 packet and the agent uses it to decide to process the packet or discard it. The main difference between SNMPv1 and SNMPv2 is that SNMPv2 added a few more packet types like the GETBULK PDU which enable you to request a large number of GET or GETNEXT in one packet.

SNMPv3 was designed to address the weak V1/V2 security. SNMPv3 is based on SNMPv2 (same packet types), but the main difference is that it is a lot more secure. It does not use community strings but users with passwords and SNMPv3 packets can be authenticated. For more info about SNMP Security from our edocs Security for SNMP.


Q: The SNMP architecture diagram demonstrates that you can poll MBean attribute values from Managed Servers. However, the SNMP commands such as SNMPWALK and SNMPGET don't allow you to specify the addresses and/or names of the Managed Servers that you'd like to poll. How can I tell which Managed Server(s) to poll?


[A] SNMP architecture until WLS 9.x doesn’t allow you to create multiple agents in a WLS domain. The agent implementation was only available in the administration server of the domain. The managed servers will be communicated by the agent when a SNMP manager talks to the SNMP agent running on the administration server. So if you want to get MBean attribute value from a managed server in a domain you must differentiate it using the OID. So you will still point your SNMPWALK and SNMPGET to the admin host and agent port. If you are not sure about the complete OIDs of the managed objects on the WebLogic managed servers you can use snmpwalk to get the root OID of the managed object (or) the attribute by suffixing the managed server's name to the community prefix.