BOXI Grooming: Prune Your Input and Output FRS
Origin of a Cluttered FRS
Business Objects XI has a way of making more sub-folders than files. Seriously, take a close look at your Input and Output File Repository Servers, or rather your FRS on the disk. You will notice a crazy number of folders. As objects get deleted in Business Objects XI their corresponding files are removed from the FRS; however, their sub-folder is not. Since there is nearly a 1 to 1 ratio of files to folders this lack of folder clean-up leads to a lot of empty folders and an inefficient FRS (and any resulting file system back-up).
Keeping Your FRS House in Order
Since you see the most efficient system, you want to clean up this folder mess. Well, BO has a tool for that and you already own it. You can invoke it merely by putting the command-line parameter “-prune” on the Input FRS and the Output FRS.
Making -prune Work for You
I recommend that before you start you shut down your BOXI system’s web front-end. This will keep users from receiving errors or possibly impacting the prune activity. Yes, this requires downtime to do it properly and only experience will tell you how long it takes (depends on how much mess needs to be cleaned and how fast your servers are). Now in CCM shutdown your FRS servers.
Do the next steps one at a time unless you are in a rush. Modify the command-line parameters by appending the parameter “-prune”, be sure to include a space between it and the final parameter that was present before you started. Also you can add “-trace” as a switch to capture a log of the actions.
Completing your FRS Prune
Now start the FRS that you just modified and you will observe that it maintains a “starting” status until it finishes. You can watch it through your servers task manager, top, or other corresponding process manager. You could also watch it through the log file it creates with the included “-trace” parameter. When it finishes, remove the parameters you added and proceed to the next one. If both have completed the prune then you are done and as long as you removed the parameters that you added you are ready to roll.
Additional Information
I suggest counting sub-folders and files before and after. This will help you understand the impact of what you just did. Please keep in mind that the “PRUNE” command does not clean-up any dead pointers (objects in the CMS that lost the files the point to). Prune only cuts the dead branches and it doesn’t touch a single leaf or leafy branch. Enjoy the analogy 
More FRS Prune Resources
Over at “let’s learn business intelligence” they have a good video walk-through of a real, live FRS prune on a BOXI system. Check it out: http://www.letslearnbi.com/2009/05/business-objects-prune-and-trace.html.
To Split Mode or Not to Split Mode, That is the Question?
Wishlist Item: A Distributed Business Objects Enterprise Environment
We all want our BOE environment to perform to the best of its ability. We want to be certain that no particular component of our environment’s architecture is a weak point or bottleneck. As a result, given the time and the hardware, most of us would love to deploy a distributed, well-balanced Business Objects Enterprise environment. In many cases this would include putting each tier or group of services on their own machine/node: the Intelligence Tier (CMS, FRS, etc.), the Processing Tier (Report/Processing, Job, etc.), Web Application Tier (where InfoView and CMC are deployed), and Web Tier (HTTP/HTTPS services). Aside for using multiple servers for each tier, if you read the documentation closely, you could be tempted to take your distribution to the next level.
What is “Split Mode”, “Split-Web”, “Split Deployment”?
The SAP Business Objects “Web Application Deployment Guide” explains that “you can deploy all web application resources on a single web application server (standalone mode), or to separate dynamic and static content for deployment to de-paired web and web application servers (split mode).” The dynamic content, the part requiring actual processing would be deployed to the dedicated Web Application Server and the static content (images and client side scripts, I believe) would be deployed to a dedicated Web Server. The reason behind creating a split mode web deployment would be to better distribute the load of Web Application Server, giving the otherwise under utilized Web Server a little more responsibility.
The following is a list of WDEPLOY supported split mode server combinations:
• Tomcat 5.5 (web app) with Apache 2.x (web)
• WebSphere 6.1 (web app) with IBM IHS (web)
• WebLogic (web app) with Apache 2.x
• Sun Java Application Server 8.1 and secondary web server
Ideals and Reality, Same or Different
My mother used to play a game with us (when we were children) that we called “same or different”. It quickly became and inside joke, a way of indirectly making a comment on something, for example, I would ask my mother something like “Business Objects, a simple/stable/scalable out-of-the-box experience; same or different”. Now I didn’t say that all of them were witty gems.
The truth of the matter is that even if you are successful at deploying Split Mode you won’t find that it makes your life, or that of your servers any easier. It is extremely difficult to get Split Mode working. The guides would have you believe that it is a simple matter of using different parameters with the WDEPLOY application. This may be true for creating the packages to deploy, but actually deploying them and getting them to function is quite a different task.
Benefits Outweighed by Costs
If you are able to get split web working then you have joined the ranks of a very select few and you have also probably burned through at least 3 or 4 of your SAP Business Objects Support Customer Message allocations. So what did all of that work get you? Not much, in fact, even if you are monitoring the servers closely you may not see any change in the actual load on the Web Application Server, even with a large user base. Not to mention, each time you install a FixPack or Service Pack you will have to recreate the magic all over again and ask yourself, was it really worth it?
These are not just my personal experiences and observations. Seasoned Business Objects consultants have expressed similar discontent with this feature. My limited Internet research (forums, mostly) also confirms my findings. I would love for anyone to share their own experience with split deployments in the comments.
Tips for Installing FixPacks in Business Objects XI 3.1
Lately I have been faced with the need to install a FixPack on one of my Business Objects Enterprise XI 3.1 systems. Doing so has reminded me of some key points and tips that I wanted to share with the Business Objects community. FixPacking BO 3.1 is truly a different experience:
Not Your Father’s BO FixPack
Like everything else in life, FixPacks have changed and evolved with the Business Objects product. Some of the notable changes for BO XI 3.1 FixPacks are:
- FixPacks are no longer suite wide. You need the FixPack for the specific Business Objects product you wish to patch (such as Enterprise, Live Office, Crystal Reports 2008, etc.)
- You must first install the patch on a node where a CMS server is located. According to some SAP-BO support engineers the safest choice in a distributed environment is to install the FixPack one node/machine at a time. So if you have multiple CMS nodes, install on those first one at a time, then install on the other servers one at a time. This makes for a VERY lengthy FixPack deployment, but it is the safest method. I have also had several senior support engineers confirm that the only requirement is to patch 1 node hosting a CMS first, then after that patch is completely started you can patch all of the rest, staggering them by 5-10 minutes each.
- Stop all BO servers except for a single CMS, Input/Output File Repository Servers, CMS database, and Server Intelligence Agent (SIA). If a CMS in the cluster is not up and running the install will not work properly or at all (this is true for all servers, regardless of their role). The SI Agent must also remain up on all nodes to be patched.
- Czech and Finnish Language Packs are now available (as of FP 1.3/1.4)
FixPack Installer Beware
Here are a few cautions/warnings that stand out to me:
- EVERY Business Objects component must be on the same version. If you patch a dedicated CMS server, you must also patch any other nodes that are a part of your cluster, such as processing servers or Live Office servers, etc. Also any clients, Designer and DeskI need to be patched as well. Another reason to use WebI exclusively, no client patches needed (unless we are talking about WebI Rich Client).
- Don’t forget to redeploy the patched BusinessObjects web components. The deployment method will vary based on your original web component deployment.
- Temp Directory: When working with the Windows Installer Package, be sure to either manually extract the installation files to a desired location, or to change your user’s “TEMP” environment variable to the true location to which you want these very large and many files extracted. Actually, on second thought, ALWAYS change your “TEMP” path environment variable to a location with enough space (a few GB); I have seen the installer, in the midst of installing fill up my C: drive with temporary files that it works with after all of the unpacking and hours into the patch installation. Be careful here, plan ahead.
These are just a few of my notes and pointers for now on the top of my head. I will update this article later with my more complete notes or with any of your suggestions offered through the comments below.
Cleanly Stopping and Starting Business Objects Servers
I really cannot explain why it is that I have NEVER seen any documentation that advises Business Objects XI administrators on how to stop and start Business Objects properly. For those of us running part of our business on BO XI we are very concerned about minimizing errors for users and scheduled jobs while we are restarting Business Objects.
The Cleanest Method to Stop a Business Objects Environment
I have discussed this topic with my senior engineers and the following is based on the input I received from them and from my own experience and knowledge. I will label optional steps that will make your stop and start as graceful as possible; these are optional, but they are the best method to follow if you have the time to do so. Also when following the steps make certain that the step is complete and the server is completely stopped/started before proceeding to the next step. Additionally if you have a clustered environment, and you should if at all possible, then you can stop all servers of the same kind in any order or even simultaneously.:
- Please first make note of any pre-existing disabled servers to be sure that you do not enable them mistakenly later on. Screenshots are useful and fast, just be sure to save them.
- Shutdown your web/application layer. This will stop your users from launching new jobs and from getting strange errors as you are in the midst of your shutdown. You could just disable a proxy server (if you use one to cut off the access, but (Graceful Option) you may want to completely flush the system by completely stopping the web server.
- (Graceful Option) Through the Central Management Console (CMC) or through the Central Configuration Manager (CCM) Disable all BO servers except for the CMS, Input FRS, Output FRS, and Destination servers.
- (Graceful Option) Wait as long as reasonable/acceptable or until all user sessions/requests have cleared the disabled servers before proceeding to the next step. Usually 30 to 60 minutes is sufficient for any valid threads being processed.
- Shutdown the Event Servers (use CCM or CMC). This would stop any related scheduled jobs from launching.
- Shutdown all Job Servers (WebI, DeskI, Program…).
- Shutdown the Destination Servers.
- Shutdown all Report Servers (WebI, DeskI, Crystal…).
- Shutdown any other non-CMS Servers that are still up and running.
- Shutdown CMS. CMS should always be last. This is essential and it will make your CMS shutdown go much faster and more smoothly. It may also help reveal any problems that your CMS may be having (for example, if it won’t shutdown you’ll know it is not because of any other lingering servers in the cluster).
- With the CMS completely shutdown you are official and completely down.
The Cleanest Method to Start a Business Objects Environment
It is assumed that EVERYTHING is down prior to beginning these steps. If your environment is only partially down we strongly recommend that you first shut it down completely before attempting a start/restart. You want to have a clean environment so do yourself and your users this favor. When starting groups of the same kind of server you can start them one by one or simultaneously, jsut be sure that all are started before proceeding to the next step. (Graceful Option) If you disabled any servers prior to the shutdown (as part of a graceful shutdown) then you should enable them immediately after they have been started:
- Start the CMS servers. This will take a little while depending mostly on the number of objects in your environment.
- Start Destination Servers, Input FRS, and Output FRS.
- Start Event Servers. Also as a side note, please be aware that Event Servers must always be restarted following the restart of any CMS in the cluster.
- Start all Report Servers.
- Start all Job Servers.
- Start any other servers that have not yet been started.
- Start your web/application layer.
The Value of the (Graceful Stop/Start Method)
Disabling Business Objects XI servers allows the servers to retain and complete their current threads/work, but it stops it from accepting any new work. Nevertheless, this requires caution. If users retain access to the environment (web and application layer are up) while you are disabling servers and you disable ALL servers, or most of them to a point below capacity demands, then you will cause errors for users! Therefore, if the users do retain access to the environment disabling servers would only be done partially to the collection of similar servers (with reason). Also do not forget to enable the servers after the servers are restarted!!!
Please also see the article “The Best Way to Stop a Business Objects Server“.
CMS Cluster Summary and the CMSClusterMembers Registry Key
A “CMS Cluster” is a group of Business Objects XI Central Management Servers (abbreviated CMS, which is a BO service\daemon) that are interrelated and connected so that they may work in conjunction to manage the various other BO servers and provide most of the basic functionality of Business Objects. Clustering them allows for greater capacity management (each Central Management Server can generally support about 500 to 600 concurrent users, based on usage) and fail-over capability. In other words, having a CMS cluster consisting of at least 2 CMS is a VERY intelligent design decision for any Business Objects XI environment requiring any fault tolerance.
Creating a CMS cluster is not too complicated, but for now outside the scope of this article. You should understand that pretty much every other BO service will benefit by being aware of this CMS Cluster. The cluster name is defined in the specifications of each server’s CCM configuration (BusinessObjects XI R2) or that of its Server Intelligence Agent (Business Objects XI 3.X). The way that the BO server is aware of the members of the CMS cluster is through server operating system registry key. The name and path of this key on the Windows OS is:
HKEY_LOCAL_MACHINE\SOFTWARE\
One thing important to note about this key is that it is self-maintained. If a CMS is joined or removed from the CMS Cluster than the other BO servers running on a node will communicate with the CMS Cluster and update the registry key to reflect the valid members of the CMS Cluster. Now this is not exactly real-time updating, but rather it occurs at start-up of the BO server (I mean a service, such as Web Intelligence Report Server) or even at enabling of a disabled server.
Reflecting on what I’ve just written I a reminded that it is a good thing to create a CMS Cluster that only contains one CMS. There are no negative impacts that I know of and this allows for scalability as you can easily add in a CMS to the cluster with this already configured.
The Little Known Business Objects Program Object
Most people are only familiar with the various reporting tools in Business Objects. Of course, these are the reasons organizations set up a Business Objects instance in the first place. However, Business Objects XI has managed to include some additional functionality that goes beyond reporting. For example, BOXI can be used to publish “agnostic” documents, such as PDF, Word, Excel, or PowerPoint. In particular, nevertheless, I would like to discuss the unique functionality of the Business Objects Program Object.
A Business Objects Program Object can be used to execute a batch file, a shell script, or a binary executable. It can also run Java, JavaScript, and VBScript. Therefore, they is virtually no limit to what you can do with a BO Program Object, except for your imagination and standards. Once the association is made to the program object standard objects rights can be applied and most importantly you can schedule the object (which is where the power is).
There are a few caveats that you need to be aware of. The program will require a “Logon As” account that must have interactive login rights to the server on which the Program Job Server runs. Also, I should mention that you need a Program Job Server configured for your environment. When you create a program you are not creating a pointer to your script, but rather you are uploading your script/program to the CMS InfoStore, just like you would an Excel file if you were to load it to your Business Objects repository. This means that if you later alter the script you used to source the program object then you must create a replacement object and re-source the script.
Another couple points, when I create program objects I like to log in to CMC while logged in to the server since often I have created a driving batch file that calls other batch files on the server. Oh, and that brings up the very last point. If you are running multiple Program Job Servers and your script calls other scripts on the server you will need to place identical copies of the called scripts in the exact same path on all servers running a Program Job Server (anything less would be easier, but it would cause failures).
The following are some simple steps to create a functioning program object. They are basic, but they should get you on your way to lots of programmatic trouble. The grouping of the step correspond to the screenshots above. Click on each thumbnail to view the full-size image; I recommend opening them in another browser window (hint, right click the thumbnail image and select this option).
Step and Screenshot #1
a) In CMC, go to Objects and click the “New Object” button
b) Click on “Program” in the left-hand menu
c) Enter in the location of the program object source, the path needs to be local to the client you are using. You will need to create a test script to run if you don’t already have something in mind.
d) Select the “Program Type”
e) Select the target folder and/or categories
f) Click “Submit”
Step and Screenshot #2
a) You may edit the program object name which gets generated from the source file you loaded. The name can be anything.
b) Add other properties if desired. A good description is always helpful to others and your future self.
c) If you made changes, click “Update”.
Step and Screenshot #3
a) Click on the “Process” tab and the “Logon” sub-tab.
b) Put the account information for an account with log on rights, any rights less than interactive logon will result in a failed execution.
c) Click “Update” when done
You are welcome to experiment with the other tabs and sub-tabs, but please share you findings if you do.
Business Objects Classic Logging and Standard Tracing
If you have ever had a serious issue with your Business Objects environment that wasn’t easily fixed by a service pack, fix pack, or limited availability fix then you probably have had to enable logging on some or even all of your BusinessObjects servers. This is also frequently referred to as “tracing”. I’d like to discuss this briefly, but I want to say up-front that this article is not intended to replace any instructions you receive from SAP BO. Hopefully, it will just confirm information you receive from them, give you something to experiment with, or even prompt some deeper discussion between you and your BO Support engineer.
-Trace, or Dash Trace
I believe that -Trace is new to BO with the roll-out of BOXI, or Business Objects XI. My sources say that it comes from the “Crystal” technology and their for it has its own features and limitations. For this reason it may be especially useful on a BO WebI Report Server server with a particular issue, but on a different environment with a different Web Intelligence Report Server problem it may be useless. In my experience “-trace” can drastically impact performance depending on usage of the BO server and the limiters placed on the “-trace“.
It is called “-trace” because that is the name of the parameter that is used in the BO server’s command line to enable the tracing or logging. By the way, the command line parameters of BO servers/services are edited through CCM on the server. The parameter itself accepts no settings, but I believe that additional parameters can place configurations and limits on the trace logging (for example, “-maxlogfilesize“).
All of the tracing log files will be placed in the “\Program Files\Business Objects\BusinessObjects Enterprise 11.5\Logging” directory. So this can be a problem if the hard drive to which BO is installed does not have much room on it. It seems that by default BO servers are always doing some light tracing/logging to this folder. Once you begin “-trace” logging, you should see a huge spike in the file sizes and the file names should correspond to their servers more or less.
Business Objects Classic Logging, BO Classic Logging
Although BO was migrated to the Crystal platform we know that some fundamentals and tools remained virtually unchanged (for example Designer). Apparently under the hood there are still some things that can only be examined with what is now called “BO Classic Logging”. Apparently this is usually only for servers such as WebI Report Servers.
BO Classic Trace Logging is enabled through a collection of manual configurations on the server. The settings below are representative of possible settings, an example. If you are testing this out you could use these settings, but I would not place them in your “production” environment without consulting BO or some additional authority.
Create the following environment variables on the server:
- BO_TRACE_CONFIGFILE
Set this variable equal to the complete path to a file on your server, such asC:\Logs\BO_Trace.ini - BO_TRACE_LOGDIR
Set this variable equal to the complete path to a folder on your server, such asC:\Logs
Create the following environment variables on the server:
- Create a file named “BO_Trace.ini”. I think it can have any name that corresponds to the environment variable value and it must be placed in the location specified by the environment vairable value as well.
- In the file place the following code:
if (name == "busobj") {
active = true;
size = 100 * 1000;
keep = true;
importance = xs;
}
With either flavor of tracing or logging you will need to stop and start the affected BO servers in order to have the tracing take effect. In fact, to “-trace” logging you will need to stop the service first to make the command line parameter addition. You will need to monitor disk space constantly so that this doesn’t bring down your Business Objects environment. Since these are text files, the files compress rather well. Expect many similar sized files that have the start time/date stamp in the file name. Keep in mind that these will be on the server’s timezone and a file dated in the file name as Jan 19th can easily have data for Jan 19th, 20th, and 21st depending on your environment’s activity.
Note: Watch the sub-directory that is created with the name “wicdztrace”. This directory fills up with lots and lots of small files. They are a mess and will need to me deleted often.
Conclusions
Remember, now you know enough to be dangerous. I can’t support you if you crash your system with this. Test it out and engage BO and demand that they help you configure your logging to meet their needs and yours.
The Best Way to Stop a Business Objects Server
Minimal Impact of Administrative Actions on BOXI
I would like to think that I am unique in many ways, but I think that I am very similar to others when it comes to my desire to minimize the impact of my administrative actions on my Business Objects XI instances. One way that I accomplish this is to carefully shutdown Business Objects servers in a very particular and simple manner.
Changes to Web Intelligence Report Server Require Restart
From time to time we all make changes to our Business Objects environment. Frequently I find myself making changes to the the Web Intelligence Report Servers (WIRS). Whether it is tweaking command-line parameters, CMC properties, logging status, making some obscure registry setting, or even patching DLLs; I am changing my WIRS and I need to restart it in order for the change to take effect. In some cases, my client runs 24/7 and has no real downtime and in other cases I need to make the change immediately and I need them to take effect yesterday. 
The Cleanest Restart You Can Ever Perform
In most cases the changes I make will not take effect without a restart. However, if I go directly to the Central Configuration Manager (CCM) and just stop the server(s) and then restart it, any reports (edits, paging, refreshes, etc.) will fail on the user’s side (or for the scheduled job). So what can I do to guarantee this does not happen??? Disable the server. Wait. Stop server. Restart Server. Enable the server.
Detailed Cleanest BO Server Stopping Procedure
- Disable the server: do this either in the Central Management Console (CMC) or in CCM, which ever you prefer, but I think that internally BO engineers prefer to use the CCM (its the icon with the cylinder and check mark).
- Wait: there is no golden rule on how long to wait, any amount of waiting is still better than none. During this period the server is finishing whatever work it was assigned and the CMS is not assigning it any new requests. You can observe this on the metrics tab of the CMC page for the server (if your server has metrics). I generally wait 30 minutes and I watch the CPU utilization (CPU time) of the WIRS’s process (yes, in Windows) to see that it is dormant.
- Stop the server: this one is easy. Just go to the CCM and stop the server you are targeting. You can do this from any server in the cluster. You should watch the executable to be certain that it goes away, disappears, or dies.
- Restart the server: Again stay in CCM, allow for at least 15 seconds from the stop, but this will depend a lot on how long you waited since the disabling of the server, I mean if it still has any active threads it will take longer to stop (see previous step for proper shutdown before restart). Allow the server to get fully started and registered with the CMS.
- Enable the server: again use CCM or CMC to do this. I suggest you make sure the server is started for a period of 30 seconds before you enable it and let the CMS have it back for abuse.
Note: I believe this same technique should work well for all Business Objects server, not just Web Intelligence Report Servers.
Please let us all know if you have any comments, suggestions, or anything else on your mind loosely related to this topic by leaving a comment.
