Does your WLS File Store grow in size?

JMS Server in WLS only removes the messages that are expired once in every expiration scan interval (A JMS Server level config). The destination may have expired message sitting in them if the expiration scan interval property is set to a high value. For ex. 1800 will cause the JMS Server to scan for expired messages once in 30 mins. Still applications cannot receive the expired messages - they will be sitting in the destination. But it doesn’t automatically do any compaction per se. So the file size of the WLS File Store may keep increasing. This might cause slow startup of the server.

You can perform manual offline compaction in the following ways:

• There is a Java command-line utility that can do this trick "weblogic.store.Admin". See Edocs for more info.
  

• A WLST command called "compactstore" can also do the same trick. See Edocs for more info.

Note that this compaction is supported only for WLS File Stores and not for JDBC Stores as they don't suffer with the same issue of compaction.

The Java command-line utility "weblogic.store.Admin can also be used to dump the contents of a store into an XML file for troubleshooting purpose.

Play Hide & Seek with Administration Console

Everyone who work with WLS knows that the Administration Console is the light weight web based UI provided by BEA for configuring, monitoring and managing a WLS domain. Automation and Scripting can be done in many ways. See this post for more info.

Exposing the console in your production environment might be security issue. In those cases you can either disable the console or hide it. Needless to say if you disable the console then your only option is to depend on command-line and scripting tools for any other administration activity or to even enable the console back. But hiding will provide a level or security.

To disable the console:

• In the admin console navigate to --> General --> Advanced Options --> Console Enabled. De-select that option to disable console (restart required).
  

To hide the cosole:

• Enable the administration port on the domain so that console can only be accessed on the 2-way secured admin port. This will force all the clients to have valid certificate installed in their client (browser) to access the console.
• In the admin console navigate to --> General --> Advanced Options --> Console Context Path and change the context-path of the console to something unique and only known to the administrators.

All is fine. But one of the common question is how can I enable the console once I disable them. Like I said earlier you have to rely on the scripting tool to enable the admin console once it is disabled. You either perform this action interactively or by running a WLST script. See the following sample WLST script for more info:

connect("username","password")
edit()
startEdit()
cmo.setConsoleEnabled(true)
save()
activate()
disconnect()
exit()

You have to restart the administration server to get the console back. Have fun playing hide and seek with the console!

In WLS 10.3 with the Light Weight Server Installer, you can choose not to install WLS Admin Console when you install WLS. Along with the other installation options, this provides a way to reduce the server installation footprint and runtime memory footprint. Can't wait to play with WLS 10.3, you can now download the Tech Preview at http://commerce.bea.com/showproduct.jsp?family=WLS&major=10.3Tech&minor=-1
(as of Nov 9, 2007 WLS 10.3 is only a Tech Preview Release). I will write more on the WLS 10.3 release which is build on BEA microService Architecture (mSA) soon. Read more about BEA mSA here http://www.bea.com/framework.jsp?CNT=msa.jsp&FP=/content/.

Is my Administration Server single point of failure?

The answer is No and Yes. Read on for more specific info....

WebLogic domain consists of Servers, Machines, Clusters, Applications and Configurations related to them. Administration sever is the controlling instance for the domain. It is the keeper of all the configuration. Ideally you don't want to host (target) any application or service that clients would access on the admin instance. Managed server(s) or cluster(s) are meant to host such applications and services that clients would access (servers other than admin server in a domain are called as managed servers).

Having said that, if an admin instance fails during the middle of the day what happens to my domain? It is the controlling instance so will it halt my domain? The answer is NO. Admin server will provide all the configurations to each managed server and they will be cached locally. There are limited dependency between the servers during runtime. So an admin server failure will not affect the running managed server(s) or cluster of managed server(s).

Alright, what about the "Yes" part of the above answer?! There are few services you will lose when an admin instance is not available. You cannot introduce any new configuration or change any existing configuration in the domain. Examples are: you cannot deploy new applications, you cannot change the JNDI name of a JMS destination hosted on one or more servers etc. You will also lose the domain log entries in the Domain Log file maintained by the administration server. Still you can use the Server Log maintained on each managed server for auditing or troubleshooting purpose. Also lastly the managed server(s) in the absence of admin server can only be started or restarted in Independence Mode (using the previously cached local configuration information).

• You are not expected to introduce new configuration or change any configuration on the fly in your production setup so losing the ability to do this will have literally no or little impact.
  
• You can use server log file entries for auditing and troubleshooting purposes.
  
• The managed server can be restarted or started if required during the absence of admin server using Independence Mode (which is a default setup now in WLS 10.x - no extra configuration required).

Admin server (for that matter any server in the domain) can be configured to automatically get restarted in case of failures using a feature called - whole server migration. I will talk about this in a later discussion.

So now you know why I said "No" and "Yes"! huh!

Managed Server Independence Mode

Starting from version 7 WebLogic managed servers can be started in the absence of an administration server. The feature that will enable a managed server to come up in the absence of administration server (MSI mode) is enabled by default. But some of the files required for MSI mode are only sent to the managed server only when another option (MSI file replication enabled) is ON.

This feature is same in version 8 except that the config.xml being copied and cached in the managed server is stored as msi-config.xml.

In version 9 the feature is still the same except that the complete config directory that contains the configuration file is copied by default to the managed servers. But some other files like the security (*.dat) file is not copied until the MSI file replication enabled option is ON.

Version 10 alleviate all the pains by making MSI mode and MSI file replication as a default option. So out of the box a managed server is enabled to run in independence mode. And all the files required to start a managed server by itself (/config dir and *.dat file) will be automatically copied to the managed server during startup and will be synced (automatically) with the master copy (in Admin Server).

Scripting & Automation

Administrators always love scripting solutions. Because it is reusable and it can be executed by anyone. However it is not so easy to learn any scripting solution per se. A good example is UNIX shell scripting, where you must know the commands to build a scripting solution. But once you get ramped up it very simple to build a script.

WebLogic Server(WLS) provides many ways for an administrator to script different configuration and management tasks. Needless to say if the administrators are also Java developers they can use the management API used by WLS (JMX) for building such scripting solutions. If you are not a developer you don't have to panic you have other alternates that you can use.

If you are in WLS 8.1 or lower you can use anyone of the following solutions for scripting

• Weblogic.Admin utility
• WebLogic Ant tasks
• WebLogic Scripting Tool (WLST)
• Thirt-party solutions (WLShell etc.)

If you are in WLS 9.x or higher you still have all the above options but the Weblogic.Admin utility is deprecated and WLST is the recommended solution. But Weblogic.Admin is still for backward compatibility reasons.

But one of the challenge is for the administrator to ramp up on such scripting solutions provided by different vendors. WLS 10 solve that problem by providing a management plug-in to the light weight web-based UI (Administration Console) - WLST Script Generator. Using this you can record all your console interactions (Configurations) into a reusable WLST Script. It is like a little macro recorder that will solve your problem or ramping up on WLST if you are new to it.

The Administration Console does not record WLST commands for the following:

• Changes to the security data that is maintained by a security provider. For example, it doesn't record the commands if you add or remove users.
• Changes that are made to the application using console which will end up in deployment plans.
• Runtime operations that you perform on Control or Monitoring pages, such as starting servers, suspending a JDBC Connection Pool etc.

To read more about WLST go here.
To read more about recording WLST Scripts go here.

I shall share more tips on WLST in my later posts.

Whats the deal with SNMPv1, v2 and v3

Here are some of the questions I got from someone few days ago. See my answers inline.

Q: The WLS SNMP agent supports 2 trap versions: V1, V2 and V3. What's the difference between these 3 versions?

[A] SNMPv1 and SNMPv2 are the two version of SNMP implementation. Until WLS 9.x only SNMPv1 and SNMPv2 are supported. Now we have SNMPv3 implemented in WLS 10. Both SNMPv1 and SNMPv2 use community strings to authenticate the packets as SNMP uses UDP (a broadcast protocol). The community string is sent in plain-text in every single SNMPv1 or SNMPv2 packet and the agent uses it to decide to process the packet or discard it. The main difference between SNMPv1 and SNMPv2 is that SNMPv2 added a few more packet types like the GETBULK PDU which enable you to request a large number of GET or GETNEXT in one packet.

SNMPv3 was designed to address the weak V1/V2 security. SNMPv3 is based on SNMPv2 (same packet types), but the main difference is that it is a lot more secure. It does not use community strings but users with passwords and SNMPv3 packets can be authenticated. For more info about SNMP Security from our edocs Security for SNMP.


Q: The SNMP architecture diagram demonstrates that you can poll MBean attribute values from Managed Servers. However, the SNMP commands such as SNMPWALK and SNMPGET don't allow you to specify the addresses and/or names of the Managed Servers that you'd like to poll. How can I tell which Managed Server(s) to poll?


[A] SNMP architecture until WLS 9.x doesn’t allow you to create multiple agents in a WLS domain. The agent implementation was only available in the administration server of the domain. The managed servers will be communicated by the agent when a SNMP manager talks to the SNMP agent running on the administration server. So if you want to get MBean attribute value from a managed server in a domain you must differentiate it using the OID. So you will still point your SNMPWALK and SNMPGET to the admin host and agent port. If you are not sure about the complete OIDs of the managed objects on the WebLogic managed servers you can use snmpwalk to get the root OID of the managed object (or) the attribute by suffixing the managed server's name to the community prefix.

A common (and tricky) security question

One common question people always ask me is "If I forget my administrator password how can I reset it". It was simple during WebLogic Server 6.1 days where the password was stored in a plain text file - filerealm.properties. When they released version 7.0 of WebLogic Server(WLS), the security architecture went through a complete overhaul. They introduced a file based Embedded LDAP server for storing security related information in the domain.

First there are few recommendations that I want to share with you:

1. Create a backup admin user and keep it safe so that you can use that admin user in such disaster scenario. Make sure you keep that info in a safe place.
2. Take periodical back ups of your domain's embedded LDAP directory on the admin server machine(/Servers/AdminServer/data/ldap/).
3. Take periodical back ups of the individual providers in you real using WebLogic Server security import/export feature. (For more info see Export data from a security provider from edocs)
   

Following are some of the ways in which you can answer the above question:

• If you are in development, one option is you can simply recreate the domain from the existing domain by creating a Domain Template using Domain Template Builder. That way you don't loose any configuration that you did in the original domain but any security changes you made will be lost.

• If you have already created User Config file set for the admin user, run a JMX Script to change the admin user password using that User Config file set. Obviously you might want to recreate the User Config file set after you change the password. If you have no User Config file set created for the admin user then follow one of the following steps.

• Delete the ldap sub-directory under the adminserver folder and restart the adminserver(Obiviously make a backup before you delete). It will automatically rebuild the ldap from the *.ldift files in the domain directory (created when the domain was created Ex. DefaultAuthenticatorInit.ldift). This will work if you changed the admin password after creating the domain and you forgot/loose the new password but still remember the old password which you created when you created the domain (Strange use case ! huh). When using this method you will loose all the existing users and groups in the ldap so you might want to export the users and groups before you delete the ldap and import them back later. But this can be performed only if you still have access to the console or some pre written backup scripts which are using encrypted username and password in the form of user config file.

• If you don't remember the Adminstrator password at all (Original or Changed) and if you are ready to loose all the user and groups (if not backed up using export feature) in the existing ldap then delete the ldap sub-directory under the adminserver folder (Don't forget to create a backup). Edit the file 'DefaultAuthenticatorInit.ldift' in a text editor and locate the line 'dn: uid=adminusername,ou=people,ou=@realm@, dc=@domain@' (Obiviously make a backup before you edit). Edit the value for the attribute 'userpassword' to a known plain text value. Restart the admin server using the new credentials. The domain will recreate the ldap with the new admin username and password. If you have already exported the old users and groups import them back.

• What if you haven't backed up any users or groups and you also don't want to loose them. But still you want to recover the lost administration user password. In such scenario to recover the administrator password in a WebLogic domain, provided the default Admin role is not modified/removed perform the following:

• At the command line, change directory to the domain and run the setEnv script to set the PATH and CLASSPATH.

• Create a new DefaultAuthenticatorInit.ldift: run java weblogic.security.utils.AdminAccount tempadminusername temppassword ./
• Remove the initialized status file, DefaultAuthenticatormyrealmInit.initialized from the Domain/AdminServer/ldap subdirectory.
• Restart the server, using the new user identity.
• To change the old admin user identity, log into the admin console or JMX Scripts. (Optional)

• Delete and recreate the domain (Yuck!). I dont think you would ever end up doing this unless you change the default Admin role or his permissions. This is highly unlikely situation.

To be or not to be technical

I was thinking to start a technical blog for a long time and I kept on deferring it. By the way who am I and why am I starting this blog?!

My name is Balamurali Kothandarman and I go by the name of "Bala". I am a Delivery Technologist Sr. working with BEA Education Services in USA. My primary job function is to deliver training to our customers in various Technologies and Products. I see a lot of customers every week in my trainings. Many times I receive a follow up question from somone who took a training with me. The question may either be from the course material we use for the training or from a general product and technology stand point. Either way I see a lot of repeating questions. So I decided to do this blogging where not only customer who attend our trainings but also other who are interested can participate and collaborate in this blog regarding WebLogic Server and realated technologies.

I also have an oppurtunity to do blogging @ http://dev2dev.bea.com which is pretty good technical website from BEA. But I want to keep my blog more unofficial so here I am.

Feel free to collaborate or ask your questions regarding BEA products or technologies. I shall address them to the best of my knowledge. If I can't address a question directly, I shall get help from my co-workers who are an excellent team.