Tips for Installing FixPacks in Business Objects XI 3.1

Lately I have been faced with the need to install a FixPack on one of my Business Objects Enterprise XI 3.1 systems. Doing so has reminded me of some key points and tips that I wanted to share with the Business Objects community. FixPacking BO 3.1 is truly a different experience:

Not Your Father’s BO FixPack

Like everything else in life, FixPacks have changed and evolved with the Business Objects product. Some of the notable changes for BO XI 3.1 FixPacks are:

• FixPacks are no longer suite wide. You need the FixPack for the specific Business Objects product you wish to patch (such as Enterprise, Live Office, Crystal Reports 2008, etc.)
• You must first install the patch on a node where a CMS server is located. According to some SAP-BO support engineers the safest choice in a distributed environment is to install the FixPack one node/machine at a time. So if you have multiple CMS nodes, install on those first one at a time, then install on the other servers one at a time. This makes for a VERY lengthy FixPack deployment, but it is the safest method. I have also had several senior support engineers confirm that the only requirement is to patch 1 node hosting a CMS first, then after that patch is completely started you can patch all of the rest, staggering them by 5-10 minutes each.
• Stop all BO servers except for a single CMS, Input/Output File Repository Servers, CMS database, and Server Intelligence Agent (SIA). If a CMS in the cluster is not up and running the install will not work properly or at all (this is true for all servers, regardless of their role). The SI Agent must also remain up on all nodes to be patched.
• Czech and Finnish Language Packs are now available (as of FP 1.3/1.4)

FixPack Installer Beware

Here are a few cautions/warnings that stand out to me:

• EVERY Business Objects component must be on the same version. If you patch a dedicated CMS server, you must also patch any other nodes that are a part of your cluster, such as processing servers or Live Office servers, etc. Also any clients, Designer and DeskI need to be patched as well. Another reason to use WebI exclusively, no client patches needed (unless we are talking about WebI Rich Client).
• Don’t forget to redeploy the patched BusinessObjects web components. The deployment method will vary based on your original web component deployment.
• Temp Directory: When working with the Windows Installer Package, be sure to either manually extract the installation files to a desired location, or to change your user’s “TEMP” environment variable to the true location to which you want these very large and many files extracted. Actually, on second thought, ALWAYS change your “TEMP” path environment variable to a location with enough space (a few GB); I have seen the installer, in the midst of installing fill up my C: drive with temporary files that it works with after all of the unpacking and hours into the patch installation. Be careful here, plan ahead.

These are just a few of my notes and pointers for now on the top of my head. I will update this article later with my more complete notes or with any of your suggestions offered through the comments below.

Cleanly Stopping and Starting Business Objects Servers

I really cannot explain why it is that I have NEVER seen any documentation that advises Business Objects XI administrators on how to stop and start Business Objects properly. For those of us running part of our business on BO XI we are very concerned about minimizing errors for users and scheduled jobs while we are restarting Business Objects.

The Cleanest Method to Stop a Business Objects Environment

I have discussed this topic with my senior engineers and the following is based on the input I received from them and from my own experience and knowledge. I will label optional steps that will make your stop and start as graceful as possible; these are optional, but they are the best method to follow if you have the time to do so. Also when following the steps make certain that the step is complete and the server is completely stopped/started before proceeding to the next step. Additionally if you have a clustered environment, and you should if at all possible, then you can stop all servers of the same kind in any order or even simultaneously.:

1. Please first make note of any pre-existing disabled servers to be sure that you do not enable them mistakenly later on. Screenshots are useful and fast, just be sure to save them.
2. Shutdown your web/application layer. This will stop your users from launching new jobs and from getting strange errors as you are in the midst of your shutdown. You could just disable a proxy server (if you use one to cut off the access, but (Graceful Option) you may want to completely flush the system by completely stopping the web server.
3. (Graceful Option) Through the Central Management Console (CMC) or through the Central Configuration Manager (CCM) Disable all BO servers except for the CMS, Input FRS, Output FRS, and Destination servers.
4. (Graceful Option) Wait as long as reasonable/acceptable or until all user sessions/requests have cleared the disabled servers before proceeding to the next step. Usually 30 to 60 minutes is sufficient for any valid threads being processed.
5. Shutdown the Event Servers (use CCM or CMC). This would stop any related scheduled jobs from launching.
6. Shutdown all Job Servers (WebI, DeskI, Program…).
7. Shutdown the Destination Servers.
8. Shutdown all Report Servers (WebI, DeskI, Crystal…).
9. Shutdown any other non-CMS Servers that are still up and running.
10. Shutdown CMS. CMS should always be last. This is essential and it will make your CMS shutdown go much faster and more smoothly. It may also help reveal any problems that your CMS may be having (for example, if it won’t shutdown you’ll know it is not because of any other lingering servers in the cluster).
11. With the CMS completely shutdown you are official and completely down.

The Cleanest Method to Start a Business Objects Environment

It is assumed that EVERYTHING is down prior to beginning these steps. If your environment is only partially down we strongly recommend that you first shut it down completely before attempting a start/restart. You want to have a clean environment so do yourself and your users this favor. When starting groups of the same kind of server you can start them one by one or simultaneously, jsut be sure that all are started before proceeding to the next step. (Graceful Option) If you disabled any servers prior to the shutdown (as part of a graceful shutdown) then you should enable them immediately after they have been started:

1. Start the CMS servers. This will take a little while depending mostly on the number of objects in your environment.
2. Start Destination Servers, Input FRS, and Output FRS.
3. Start Event Servers. Also as a side note, please be aware that Event Servers must always be restarted following the restart of any CMS in the cluster.
4. Start all Report Servers.
5. Start all Job Servers.
6. Start any other servers that have not yet been started.
7. Start your web/application layer.

The Value of the (Graceful Stop/Start Method)

Disabling Business Objects XI servers allows the servers to retain and complete their current threads/work, but it stops it from accepting any new work. Nevertheless, this requires caution. If users retain access to the environment (web and application layer are up) while you are disabling servers and you disable ALL servers, or most of them to a point below capacity demands, then you will cause errors for users! Therefore, if the users do retain access to the environment disabling servers would only be done partially to the collection of similar servers (with reason). Also do not forget to enable the servers after the servers are restarted!!!

Please also see the article “The Best Way to Stop a Business Objects Server“.

CMS Cluster Summary and the CMSClusterMembers Registry Key

A “CMS Cluster” is a group of Business Objects XI Central Management Servers (abbreviated CMS, which is a BO service\daemon) that are interrelated and connected so that they may work in conjunction to manage the various other BO servers and provide most of the basic functionality of Business Objects. Clustering them allows for greater capacity management (each Central Management Server can generally support about 500 to 600 concurrent users, based on usage) and fail-over capability. In other words, having a CMS cluster consisting of at least 2 CMS is a VERY intelligent design decision for any Business Objects XI environment requiring any fault tolerance.

Creating a CMS cluster is not too complicated, but for now outside the scope of this article. You should understand that pretty much every other BO service will benefit by being aware of this CMS Cluster. The cluster name is defined in the specifications of each server’s CCM configuration (BusinessObjects XI R2) or that of its Server Intelligence Agent (Business Objects XI 3.X). The way that the BO server is aware of the members of the CMS cluster is through server operating system registry key. The name and path of this key on the Windows OS is:

HKEY_LOCAL_MACHINE\SOFTWARE\\Business Objects\Suite 11.5\Enterprise\CMSClusterMembers

One thing important to note about this key is that it is self-maintained. If a CMS is joined or removed from the CMS Cluster than the other BO servers running on a node will communicate with the CMS Cluster and update the registry key to reflect the valid members of the CMS Cluster. Now this is not exactly real-time updating, but rather it occurs at start-up of the BO server (I mean a service, such as Web Intelligence Report Server) or even at enabling of a disabled server.

Reflecting on what I’ve just written I a reminded that it is a good thing to create a CMS Cluster that only contains one CMS. There are no negative impacts that I know of and this allows for scalability as you can easily add in a CMS to the cluster with this already configured.

What do the values of AUDIT_EVENT.ERROR_CODE mean?

If you have spent some time with your Business Objects XI auditing data then you have probably asked yourself what significance is of the the error code values for the AUDIT_EVENT table’s ERROR_CODE property. If you haven’t yet pondered this then I suspect you haven’t looked closely at your data, OR you don’t have enough data to have produced a suspicious error code.

What are those weird values for AUDIT_EVENT.ERROR_CODE?

Here are some of the values that I have seen over the years in my BO XI R2 auditing database. The list below is a distinct list of all error codes with their corresponding EVENT_TYPE_DESCRIPTION:

ERROR_CODE | EVENT_TYPE_DESCRIPTION
1 | List of values
5 | Get page
86 | Get page
87 | Get page
303985 | Generate SQL
2147500037 | Apply format
2147500037 | Document refresh
2147500037 | Get page
2147500037 | List of values
2147500037 | Select prompt
2147760471 | List of values
2147760472 | Document refresh
2147760472 | List of values
2147760642 | Document refresh
2147760642 | List of values
2147760644 | Document refresh
2147760644 | List of values
2147760646 | Document refresh
2147760646 | List of values
2147760675 | List of values
2147760677 | List of values
2147775010 | List of values
2147776034 | List of values
2147776072 | List of values
2147776088 | List of values
2147776301 | Get page
2147776326 | Get page
2147776341 | Apply format
2147777058 | Get page

I must state that this list is in no way comprehensive, but I think it is a good sample. It might have helped the search engine help you to reach this page. Anyway, the point here is that it doesn’t matter if we build a comprehensive list because it is near impossible/pointless to build an error code translation for each ERROR_CODE value.

What do all of those ERROR_CODE values mean?

Some of you may have that handy-dandy Business Objects error code cross-reference document (I should see about sharing it next time I find it) and you are thinking these error codes can be found there. Sorry, nice try, wrong, thanks for playing. Failing that look-up I asked around and this is what I found out:
Very Short Answer: nothing
Short Answer: they can only be used in deep debugging analysis
Long Answer: Those error codes are internal Web Intelligence error codes which depend on the workflow followed and on the specific API which failed. The error codes should be able to be referenced in the trace logs produced through active “-trace” logging on the corresponding WebI Report Server (more on this topic in our article “Business Objects Classic Logging and Standard Tracing“).

So can I find any meaning in the ERROR_CODE values?

In my experience the number of errors perceived by the users of the system exceed the number of non-zero/non-null ERROR_CODE values stored in the Business Objects XI auditing data. Truthfully, some of this makes sense. If a BOXI user gets an error immediately when they click on a report in InfoView then the CMS may never get a chance to capture the error to be able to write an auditing record. More truthfully, I don’t really know.

All I can say is that the best use of AUDIT_EVENT.ERROR_CODE values is to count them as Boolean values. Did an error occur, or did no error occur. Moreover, if you find that the ratio of errors to non-error events seems to disagree with other sources of error tracking information (or user perception) then the data still has value. It has relative value. You can track it to measure trends. If you see the ratio of errors to non-error events increase/decrease over time then you know something, and knowing is half the battle.

Mystery Solved?

My sources tell me that this is fixed in BO XI R3 SP1 (XI 3.1). In fact, if you want to dive deeper take a look at ADAPT01092225. The “fix” is to limit all AUDIT_EVENT.ERROR_CODE values to 0 or 1. So, if it were previously possible tie auditing data to Web Intelligence Report Server trace logging through the error code value, that ability is revoked in R3 SP1. Therefore, the only purpose remaining for this property is the relative trend value I detailed above. Good luck.

Querying the Busines Objects XI CMS InfoStore Database Tables

The question has come to my mind many times over the years and this month it has come to my inbox.
Thanks to Pluto for inspiring me to write this article.

The question: Is it possible to peer into the Business Objects XI CMS InfoStore (a.k.a. database repository) without Query Builder or the BOXI SDK????

The answer: No

All of my research and experience tells me the following:
Much of the meta data, the useful part especially, is stored in the CMS InfoStore in an encrypted binary format. No one has been able to decrypt this format yet, but I can’t really say that anyone has tried very hard. I certainly haven’t.

Some Things You Can Do with the CMS InfoStore’s Database Tables

The CMS InfoStore is mostly held in the database table named “CMS_INFOOBJECTS5″. Some other less significant data is held in the following tables: CMS_RELATIONS5, CMS_IDNUMBERS5, CMS_ALIASES5, and CMS_VERSION_INFO.

The structure of the CMS_INFOOBJECTS5 table in Oracle is as follows:

OBJECTID INTEGER NOT NULL,
PARENTID INTEGER NOT NULL,
TYPEID INTEGER NOT NULL,
OWNERID INTEGER NOT NULL,
LASTMODIFYTIME VARCHAR2(32 BYTE) NOT NULL,
OBJFLAGS INTEGER NOT NULL,
USERFLAGS INTEGER,
SCHEDULESTATUS INTEGER,
NEXTRUNTIME VARCHAR2(32 BYTE),
ALIASES RAW(255),
CRC VARCHAR2(32 BYTE) NOT NULL,
PROPERTIES BLOB NOT NULL,
SI_GUID VARCHAR2(56 BYTE),
SI_CUID VARCHAR2(56 BYTE),
SI_RUID VARCHAR2(56 BYTE),
SI_INSTANCE_OBJECT INTEGER,
SI_PLUGIN_OBJECT INTEGER,
SI_TABLE INTEGER,
SI_HIDDEN_OBJECT INTEGER,
SI_NAMEDUSER INTEGER,
SI_RECURRING INTEGER,
SI_RUNNABLE_OBJECT INTEGER,
SI_TYPEID_MACHINE INTEGER,
SI_KEYWORD VARCHAR2(255 BYTE),
SI_KEYWORDISTRUNCATED INTEGER,
LOV_KEY VARCHAR2(18 BYTE),
OBJNAME VARCHAR2(255 BYTE),
OBJNAMEISTRUNCATED INTEGER

The “properties”, “aliases”, and “objname” fields are where most of the goodies are to be found and these are the fields that are encrypted or in some way pretty much unusable to us without the SDK or Query Builder.

A New Hope for CMS_INFOOBJECTS5

Don’t completely give up on querying this table directly. There are still quite a few valuable data that can be retrieved from the table. Firstly, you should know that the following fields are indexed and there for with a large CMS InfoStore database you will want to use these for filtering your query:

PARENTID
OWNERID
TYPEID
LASTMODIFYTIME
NEXTRUNTIME
SCHEDULESTATUS
OBJECTID
SI_GUID
SI_CUID
SI_RUID
SI_INSTANCE_OBJECT
SI_PLUGIN_OBJECT
SI_TABLE
SI_HIDDEN_OBJECT
SI_NAMEDUSER
SI_RECURRING
SI_RUNNABLE_OBJECT
SI_TYPEID_MACHINE
SI_KEYWORD
SI_KEYWORDISTRUNCATED
LOV_KEY
OBJNAME (this is encrypted though)
OBJNAMEISTRUNCATED

Not all of these fields will have data, but some will and those that sporadically have data might meet your needs perfectly. I don’t really have a strong background in querying this table directly. So we are going to have to crack this nut together. Hopefully we can share our progress in the comments of this post and through updates from me to this article.

A Sample Use of a Direct Query on the CMS InfoStore

For example, I may want to extract a list of all si_cuid (the most unique identifier of an object) and objectid for objects that were created or modified on January 24th. To do this I would use the following query:

SELECT
objectid,
si_cuid
FROM
CMS_INFOOBJECTS5
WHERE
lastmodifytime LIKE ('2009 01 24%')

If you look to the table structure you will see that the field lastmodifytime is of a character type and therefore it can be queried as presented above.

Why would you want to do this query directly on the database and not in Business Objects’ Query Builder?

Short answer: performance and possibility. What I mean to say is that you can control more precisely the performance of the query you write against the table. In fact, a similar query done through Query Builder might perform so slow that you it would not be possible except if you ran it directly against the database table. The truth is that many valid reasons to directly query the database will still end up with you taking the output of that query and then running it through the SDK or Query Builder to obtain all of the properties corresponding to that output.

What other database tables can I query?

CMS_RELATIONS5 and CMS_VERISON_INFO seem to have the only other bit of useful data. CMS_VERSION_INFO has a single row with the repositories version info; I am not certain that you can detect service pack or fix pack level here, in fact, I think you cannot. The CMS_RELATIONS5 table seems to be interesting, but I have not come up with a useful reason to query it beyond tinkering.

For your reference the CMS_RELATIONS5 table has the following structure in Oracle:

PARENTID INTEGER NOT NULL,
CHILDID INTEGER NOT NULL,
ISMEMBER INTEGER NOT NULL,
ORDINAL INTEGER,
RELATIONSHIPID INTEGER NOT NULL

None of the data appears to be encrypted. It also appears to have almost a 1-to-1 ratio between its record count and the count of the objects in the repository as reported in Business Objects Central Management Console. The value of this table might increase if we were to discover the meaning of different relationshipid values.

Conclusion: Where do we go from here?

I feel that I have only given you enough information to advance your tinkering on the true back-end. Honestly, this is a new frontier for me. I hope to post more information here as I find the time to experiment with directly with the database or if I hear from anyone else that has experience they would like to share. I invite you to share your findings as you play with querying the database. Use BO Query Builder concurrently to help you understand more about what you are seeing. If you find you need some help with Query Builder then I do recommend that you take a look at our Business Objects Query Builder Guide

Business Objects Classic Logging and Standard Tracing

If you have ever had a serious issue with your Business Objects environment that wasn’t easily fixed by a service pack, fix pack, or limited availability fix then you probably have had to enable logging on some or even all of your BusinessObjects servers. This is also frequently referred to as “tracing”. I’d like to discuss this briefly, but I want to say up-front that this article is not intended to replace any instructions you receive from SAP BO. Hopefully, it will just confirm information you receive from them, give you something to experiment with, or even prompt some deeper discussion between you and your BO Support engineer.

-Trace, or Dash Trace

I believe that -Trace is new to BO with the roll-out of BOXI, or Business Objects XI. My sources say that it comes from the “Crystal” technology and their for it has its own features and limitations. For this reason it may be especially useful on a BO WebI Report Server server with a particular issue, but on a different environment with a different Web Intelligence Report Server problem it may be useless. In my experience “-trace” can drastically impact performance depending on usage of the BO server and the limiters placed on the “-trace“.

It is called “-trace” because that is the name of the parameter that is used in the BO server’s command line to enable the tracing or logging. By the way, the command line parameters of BO servers/services are edited through CCM on the server. The parameter itself accepts no settings, but I believe that additional parameters can place configurations and limits on the trace logging (for example, “-maxlogfilesize“).

All of the tracing log files will be placed in the “\Program Files\Business Objects\BusinessObjects Enterprise 11.5\Logging” directory. So this can be a problem if the hard drive to which BO is installed does not have much room on it. It seems that by default BO servers are always doing some light tracing/logging to this folder. Once you begin “-trace” logging, you should see a huge spike in the file sizes and the file names should correspond to their servers more or less.

Business Objects Classic Logging, BO Classic Logging

Although BO was migrated to the Crystal platform we know that some fundamentals and tools remained virtually unchanged (for example Designer). Apparently under the hood there are still some things that can only be examined with what is now called “BO Classic Logging”. Apparently this is usually only for servers such as WebI Report Servers.

BO Classic Trace Logging is enabled through a collection of manual configurations on the server. The settings below are representative of possible settings, an example. If you are testing this out you could use these settings, but I would not place them in your “production” environment without consulting BO or some additional authority.


Create the following environment variables on the server:

1. BO_TRACE_CONFIGFILE
   Set this variable equal to the complete path to a file on your server, such as C:\Logs\BO_Trace.ini
2. BO_TRACE_LOGDIR
   Set this variable equal to the complete path to a folder on your server, such as C:\Logs

Create the following environment variables on the server:

1. Create a file named “BO_Trace.ini”. I think it can have any name that corresponds to the environment variable value and it must be placed in the location specified by the environment vairable value as well.
2. In the file place the following code:
   if (name == "busobj") {
active = true;
size = 100 * 1000;
keep = true;
importance = xs;
}

With either flavor of tracing or logging you will need to stop and start the affected BO servers in order to have the tracing take effect. In fact, to “-trace” logging you will need to stop the service first to make the command line parameter addition. You will need to monitor disk space constantly so that this doesn’t bring down your Business Objects environment. Since these are text files, the files compress rather well. Expect many similar sized files that have the start time/date stamp in the file name. Keep in mind that these will be on the server’s timezone and a file dated in the file name as Jan 19th can easily have data for Jan 19th, 20th, and 21st depending on your environment’s activity.

Note: Watch the sub-directory that is created with the name “wicdztrace”. This directory fills up with lots and lots of small files. They are a mess and will need to me deleted often.

Conclusions

Remember, now you know enough to be dangerous. I can’t support you if you crash your system with this. Test it out and engage BO and demand that they help you configure your logging to meet their needs and yours.