Login Taking Long? Blame Java’s DNS Look-up
The other day I answered a different article’s comment with a very brief mention of what I’ve been told by SAP-BO was a common tweak. The potential benefit of this simple tweak is large enough to merit its own article. Thank you MarcV for making this apparent. I would like to answer his questions here.
First of all, what tweak are we talking about?
Place entries in the hosts file on all web application servers/hosts/machines/boxes for each of the machines/hosts/servers/boxes running CMS. This tweak is a workaround to resolve Java’s poor, inefficient implementation of DNS look-up.
The computer’s hosts file lists hosts (meaning computers identifiable on the network) that are to be resolved locally and not through the computer’s DNS (domain name server). New hosts are added by listing the IP address of the new host first, followed by at least one space, then the new host’s fully qualified server name, then optionally one could add at least one space and the new host’s short name. Here are some links to a good tutorial of hosts files and a good discussion of hosts files.
Here is what MarcV said:
Julian,Can you expand on the last comment? I would love to know more about it, as common tweaks should be… common? I have never seen that tweak mentioned anywhere.
Currently, we host everything on one server (yeah, I know). Should we enter the server’s name and IP in the hosts file as if it were on a separate machine? Using Tomcat for the app server, that would affect pretty much everything that gets served up (logins, screen renders, WebI, Web Services…)
I’m all for speeding things up. What aspects would this affect? WebI/C Reports/LOV/???
Do you know of any consolidated source of “common tweaks”? I usually find one there here, one thing there… which means, you only know about the things that you might be researching if you’re noticing problems. Not knowing about Java having a poor implementation for DNS look-up, we might never have touched on that for our BOXI issues.
I agree, it seems that the use of the word common here was unwarranted. I did not use this tweak until it was mentioned to me by a BO support engineer after working through some deployment issues. He and others after him told me this was a common tweak for Java Application Servers and I took them at their word.
To the best of my knowledge the biggest performance improvement that one can expect from this tweak is to speed up initial login; however, it seems to me that some over operations run a little faster after the tweak is implemented.
In the case you have a one-stop-shop and you are running everything on one server, I believe that you may still see a performance improvement with this tweak. However, you may find that there is already an entry for your server in your server’s hosts file.
Repository of Common Tweaks
I wish there were a single location listing tweaks that one implement on their Business Objects XI system. Perhaps there is none, because those with the knowledge either don’t make the time to document and publish them or they don’t wish to give away for free very valuable information. Nevertheless, we can try to build one here if there is interest. I could start a “tweaks” page and through comments we could communicate our known tweaks which I would then incorporate in the main article.
Common BO-WebLogic Issues
Oracle-BEA WebLogic is not a technology I want to dive deeply into with this web site; however, it is often the Java Application Server of choice for many Business Objects XI environments because of its scalability and it tends to be preferred by other business technologists within companies. I personally work/struggle with it a lot and so I thought it a good idea to share a few issues and their resolutions in this article. I also would welcome anyone else to share their war stories as well. Be warned, I am writing this article to those who are already familiar with configuring/deploying BOXI on WebLogic.
Node Manager Service Won’t Start
This issue may be specific to Windows operating systems or it might apply to others as well. The symptoms are that you can run Node Manager from the command prompt but you cannot run it as a service.
CAUSE: If you have encountered this issue, look immediately at your “Path” environment variable. Is the final character a backslash ( \ )? If so this could be causing the script run by the service to fail due to some faulty programming. Most likely someone or some automated script/patch altered the “Path” variable and the problem did not occur until WebLogic was restarted. This issue actaully has nothing to do with Business Objects.
FIX: Just remove the backslash ( \ ) as the final character. If you really want you could also append a final semi-colon ( ; ) at the end.
Installing/Deploying Applications Fails
This may be a Unix/Linux only issue. When deploying Business Objects XI applications such as InfoViewApp, CmcApp, PlaformServices, or AnalyticalReporting you notice that you can never get them all successfully installed or started. Individually they may work just fine, but with all of them installed you suddenly start seeing infinite down apps and “Start running” status messages when you start them up. If WebLogic starts to feel like the twighlight zone you should start to question if it is hitting limits imposed by the OS.
CAUSE: There is an OS level setting that limits how many open files a single session/instance can have. In Linux this is stored in the “ulimit” under the setting “open files”. If this setting is too small (less than about 2048) you can expect issues. This is due to the fact that Business Objects applications/deployments have a very large number of files and WebLogic “opens” them all during deployment and application serving.
FIX: Increase the ulimit for “open files”, a.k.a. “nofiles” to 4000 or 4096 if you like. This should not impose any security threat to your system; however, after running the command “ulimit -a” you may discover that the limit imposed by your OS is too low to allow this increase. In such case, get it increased (see “etc/security/limits” or SAS has a good web page on this, too bad BO has no official comment on the topic). Keep in mind that you will need to also tell WebLogic to use more of these open files. The best way is to do so by appending a line in the “commEnv” file located in the “/weblogic/common/bin” folder.
The Admin Server Won’t Start
This is one of the most perplexing issues because there are not errors to point you in any direction. In short, the Admin Server cannot be started successfully via a service or via the command prompt. If watched closely you see that it progresses fine for a couple seconds and then it freezes. If you look closely at running it through the command prompt you will see that the last entries (“”) are related to security; this is your only helpful clue.
CAUSE: The “ldap” directory has become locked or corrupted by some errant process.
FIX: Stop all WL processes. Rename the “ldap” directory or compress it and delete it. Start the Admin Server. Viola! You just fixed it. WL will recreate the “ldap” directory. If you don’t even use WebLogic’s LDAP functionality then just move on, if you do use it, you may have some work still left to do. Alternatively you could look for the “lock” file (*.lck) and delete it.
Please feel free to share your own BO-WebLogic issues and their solutions. I hope this article becomes a place to share some successful resolutions and not a place to ask for help with WebLogic. There are better qualified web sites for that.
BO XI Distributed Environment: The Standalone Java Application Server
When you are building a small sandbox it is great to put everything all in one place and create a Business Objects solution that is a one-stop shop (a.k.a. putting all of your eggs in one basket). This works for lots of small-to-medium sized BO systems, but sometimes you want to do a little more. Sometimes you want some fault-tolerance, high-availability, and/or fault-tolerance. In any of these cases you might decide to build something really cool: a distributed Business Objects Enterprise XI 3.X Environment.
In this article, I would like to talk about putting your Java Web Application Server on a separate dedicated machine. You could chose Apache Tomcat, or your could get a little more crazy and go with Oracle’s WebLogic. Either way, the principles are the same.
Does BO Support Mixed Mode?
Perhaps your BO CMS, job, and reporting servers are all running on Windows machines. Perhaps you are thinking that you would like to run your Java applications on a Linux server. No problem. Business Objects has documented that it is acceptable to mix different Operating Systems in your BO environment as long as all servers running a particular component are of the same Operating System. What does this mean? It means that you would be running in a unsupported mode if you ran one CMS in your cluster on Windows and another CMS in the cluster on Linux. But if you put all of your web applications on Linux you should have no problems getting support and you should not expect any unusual issues. In fact, in my opinion, if your web application servers are not clustered, you could get away with running Apache on one Windows server and WebLogic on a Linux server, since they do not communicate with each other.
How Do I Create a Dedicated BO Web Application Server?
The hardest part is setting up your Java Web Application Server. I won’t help you here, but there is plenty of help out there in the great Internet to guide you in configuring Tomcat or even setting up WebLogic. Once this is setup all that you have to do is copy the WAR files over to the server (or you can even upload them from your client machine using the WebLogic Administration Console). Yes, this is all you have to do for Business Objects XI 3.0 and later.
Don’t I Need to Install BO on the Server?
No, do not believe the documentation. You do not need to install BO on the server as long as you can generate the WAR files from some other server. It is true! The documentation would have you believe that regardless you need to copy over at least the “deployment” folder and a “java” folder, but this is not true. These are ONLY needed if you are needing to run “wdeploy” on the Java Web Application Server. In my case, I can always use a CMS machine to generate my WebLogic and Tomcat WAR files so why would I want to put BO software on another server? If I put some of the BO binaries on my Java Web Application Server then I have to worry about maintaining/patching the software on that server as well. I don’t know about you, but I don’t need more work and more things to forget to do.
Where Do I Get the WAR Files From?
As I said, go look at the CMS machine. Follow the documentation* to help you find the “wdeploy” application (it is a command-line driven application, probably found here > wdeploy weblogic10 predeployall“. You do not need to configure any “config.*” files for your Java Application Server as these are ignored when running with the option “predeployall”. After you run the command successfully, take about 4 to 7 minutes, then you will be told where your WAR files will be located, for example:
What About XI R2, Do I Need to Put BO on the Web App Server?
All java deployments/application except for the “webcompadapter” are standalone applications that require nothing more than a supported JVM to be installed on the Java Web Application Server. The only application that uses the “webcompadapter” is CMC, so if you are intending to provide CMC access through some other method (such as a small unadvertised Tomcat instance installed on a CMS server) then you do not need to install any Business Objects binaries or libraries on the Java Web Application Server. The issue here is that the XIR2 still has some JNI calls in CMC, it was not fully converted from COM to Java and so the “Web Component Adapter” is required by the CMC and it requires that certain binaries of Business Objects Enterprise XI R2 be installed on the server.
————-
* Recommended published BO documentation is the “BusinessObjects Enterprise XI 3.1 SP2 Web Application Deployment Guide” or if not running Service Pack 2 then use the “Web Application Deployment Guide” for your OS (hint: Unix applies to both Unix and Linux). Of course, the file names will look more like the following: xi3-1_deployconfig_unix_en.pdf or plugin-xi31_sp2_webappd_unix_en.pdf (for example)
Note: I have not played enough with IIS and .NET to know much about deploying the different application there. I am sure the process is quite different, so I would not try to use this article as a guideline for any .NET BO web applications.
To Split Mode or Not to Split Mode, That is the Question?
Wishlist Item: A Distributed Business Objects Enterprise Environment
We all want our BOE environment to perform to the best of its ability. We want to be certain that no particular component of our environment’s architecture is a weak point or bottleneck. As a result, given the time and the hardware, most of us would love to deploy a distributed, well-balanced Business Objects Enterprise environment. In many cases this would include putting each tier or group of services on their own machine/node: the Intelligence Tier (CMS, FRS, etc.), the Processing Tier (Report/Processing, Job, etc.), Web Application Tier (where InfoView and CMC are deployed), and Web Tier (HTTP/HTTPS services). Aside for using multiple servers for each tier, if you read the documentation closely, you could be tempted to take your distribution to the next level.
What is “Split Mode”, “Split-Web”, “Split Deployment”?
The SAP Business Objects “Web Application Deployment Guide” explains that “you can deploy all web application resources on a single web application server (standalone mode), or to separate dynamic and static content for deployment to de-paired web and web application servers (split mode).” The dynamic content, the part requiring actual processing would be deployed to the dedicated Web Application Server and the static content (images and client side scripts, I believe) would be deployed to a dedicated Web Server. The reason behind creating a split mode web deployment would be to better distribute the load of Web Application Server, giving the otherwise under utilized Web Server a little more responsibility.
The following is a list of WDEPLOY supported split mode server combinations:
• Tomcat 5.5 (web app) with Apache 2.x (web)
• WebSphere 6.1 (web app) with IBM IHS (web)
• WebLogic (web app) with Apache 2.x
• Sun Java Application Server 8.1 and secondary web server
Ideals and Reality, Same or Different
My mother used to play a game with us (when we were children) that we called “same or different”. It quickly became and inside joke, a way of indirectly making a comment on something, for example, I would ask my mother something like “Business Objects, a simple/stable/scalable out-of-the-box experience; same or different”. Now I didn’t say that all of them were witty gems.
The truth of the matter is that even if you are successful at deploying Split Mode you won’t find that it makes your life, or that of your servers any easier. It is extremely difficult to get Split Mode working. The guides would have you believe that it is a simple matter of using different parameters with the WDEPLOY application. This may be true for creating the packages to deploy, but actually deploying them and getting them to function is quite a different task.
Benefits Outweighed by Costs
If you are able to get split web working then you have joined the ranks of a very select few and you have also probably burned through at least 3 or 4 of your SAP Business Objects Support Customer Message allocations. So what did all of that work get you? Not much, in fact, even if you are monitoring the servers closely you may not see any change in the actual load on the Web Application Server, even with a large user base. Not to mention, each time you install a FixPack or Service Pack you will have to recreate the magic all over again and ask yourself, was it really worth it?
These are not just my personal experiences and observations. Seasoned Business Objects consultants have expressed similar discontent with this feature. My limited Internet research (forums, mostly) also confirms my findings. I would love for anyone to share their own experience with split deployments in the comments.
