CMS Cluster Summary and the CMSClusterMembers Registry Key
A “CMS Cluster” is a group of Business Objects XI Central Management Servers (abbreviated CMS, which is a BO service\daemon) that are interrelated and connected so that they may work in conjunction to manage the various other BO servers and provide most of the basic functionality of Business Objects. Clustering them allows for greater capacity management (each Central Management Server can generally support about 500 to 600 concurrent users, based on usage) and fail-over capability. In other words, having a CMS cluster consisting of at least 2 CMS is a VERY intelligent design decision for any Business Objects XI environment requiring any fault tolerance.
Creating a CMS cluster is not too complicated, but for now outside the scope of this article. You should understand that pretty much every other BO service will benefit by being aware of this CMS Cluster. The cluster name is defined in the specifications of each server’s CCM configuration (BusinessObjects XI R2) or that of its Server Intelligence Agent (Business Objects XI 3.X). The way that the BO server is aware of the members of the CMS cluster is through server operating system registry key. The name and path of this key on the Windows OS is:
HKEY_LOCAL_MACHINE\SOFTWARE\
One thing important to note about this key is that it is self-maintained. If a CMS is joined or removed from the CMS Cluster than the other BO servers running on a node will communicate with the CMS Cluster and update the registry key to reflect the valid members of the CMS Cluster. Now this is not exactly real-time updating, but rather it occurs at start-up of the BO server (I mean a service, such as Web Intelligence Report Server) or even at enabling of a disabled server.
Reflecting on what I’ve just written I a reminded that it is a good thing to create a CMS Cluster that only contains one CMS. There are no negative impacts that I know of and this allows for scalability as you can easily add in a CMS to the cluster with this already configured.
Hi,
Quite an interesting article. I am new to this site and already impressed. Would like to add some more points the article. One part is about Fail-over. Say we have 2 CMSs in a Cluster and suddenly one of them crashed while taking the requests. These requests will not be taken by other CMS for the following scenarios.
1.The CMS crashes before the request is actually received by the CMS (Between Client and CMS)
2.The CMS crashes before registering the request with the CMS Database
We had one very interesting learning. If you configure Auditor in a clustered environment, we need to make sure that ODBC System DSN name for Auditor connection should be same in all the machines. In case we have different System DSN Names, all the CMS in the cluster will not get started.
Regards
Aravind
Hi Aravind,
I am glad you are finding the article interesting; thanks for the feedback. Thanks for sharing your knowledge and experience. All of them are very important details.
I was not aware of the trouble with setting the auditing database information differently. Did you have to figure it out on your own or was there some kind of helpful error message that led you to the problem?
Thanks, Julian
Hi Julian,
Nice to hear from you.
In fact there was no obvious error message when we start the CMS. When we start CMS from one server, we noticed that the other CMS was getting stopped automatically. No error message was thrown. In order to identify the issue, we enabled -trace for CMS and captured the logs. With the help of SAP technical team, we identified that there was some ODBC connection error. As soon as we disabled the auditor from the cluster, both the CMS got started. Auditor System DSN name was made identical, then we were able to successfully start the CMS.
Thanks and Regards,
Aravind
Hi Julian,
This is an awesome website for getting deeper in to BusinessObjects. It is really helpful for us.
Cheers!!!
Arun Sasi
Wow Arun! Thanks for the compliment. I’d like to have more here, but all of the other demands on my time are making that difficult. Thanks for the encouragement to keep building on the site.
Thanks, Julian
Hi Julian,
how can i improve the report and universe performance?
Please explain me step by step procedure?
Hi ashok,
Your question is quite important, but it is quite wide open. If you could provide more information about which aspect of the performance you want to improve it would be helpful. My initial response is that the biggest overall improvement in performance of reports and universes will come from improving the performance of the query (which is a database issue and a query design issue), this will impact the performance of report refresh.
Please let me know what specifically you are concerned about improving and I will try to put together a new article (as it certainly deserves one) on the topic.
Thanks, Julian
Your article is very helpful. It is hard to find people discussing this topic. now my problem with cluster is: by accident, I added a new CMS server to current cluster. So now I have two CMS servers in one cluster. If I change the cluster name, it will change the other server’s cluster name too. I tried to give different cluster name to each CMS in registery, however, after the machine is rebooted, both of the CMS are still in the same cluster. In Central Management Console, we see both two servers there. You will see two CMS, two pager servers, two cacheservers, two jobservers under two different machine name. This is not what we want. We want to have one CMS server as production, the other one as test server. We do not want these two CMS interrelated and connected. In other words, in each machine’s CMC, we only want to see its own servers(CMS,page, cache, job), not both machines’.
I have read the administrator guide, but I don’t see any info regarding this issue. Can you shed some lights on? I appreciate it. Thank you very much and have a great holiday
Hi Sunny,
My mind is on vacation right now, so you are not going to get my deepest, validated ideas. Sorry.
My initial thought is, from the server you want to become your “test” environment, what if you delete the CMS using the SI Agent utility (BO XI 3.x) or CCM (BO XI R2)? Then you could remove them from the production CMC as well.
Then I would wipe out the cluster registry key on “test”. Then I would create a new CMS and hopefully designate a new “test” cluster name, such as “testenvironment”.
Please try this and report back if you can. Merry Christmas to you, by the way.
Hi Julian/Sunny:
I have the exact same issue as Sunny, running BO XI R2. We are migrating Oracle 10 -> 11 and I was doing some testing.
Specifically, in BO XI R2′s Central Configuration Manager (CCM) I am unable to find a way to “delete the CMS”. When I open the CCM, a menu at the top right above the services list labeled “Computer Name” lists both Crystal Management Servers (CMSs) in the cluster, but nowhere do I see a way to “delete/remove” one. When I stopped Crystal Management Server service on the good CMS and removed the registry extraneous key within HKEY_LOCAL_MACHINE\SOFTWARE\\Business Objects\Suite 11.5\Enterprise\CMSClusterMembers, upon restart of the CCM the servers were both still listed.
Any chance either of you have found such an action to “remove/delete” the CMS in BO XI R2?
Thanks,
Cris
Here are the steps that worked for me to “remove” a undesired CMS from a cluster in BO XI R2:
1) Search the registries of the relevant CMSs (confirmed clustered or not) for the names of all other suspected CMSs clustered. For example, search CMSa in the registry of CMSb, and via versa. You may find entries in the CMSClusterMembers, CurrentComputerList, and in some “instances” and such*. Remove them (backing up the parent tree like “HKEY_LOCAL_MACHINE\SOFTWARE\Business Objects” first would be wise).
2) On the “dead/moving” CMS machines, like “CMSb” in my example, use the CCM to stop the CMS service, then use the properties -> configuration to Specify a CMS Data Source of “test / test” (invalid names). I suspect you could connect to another valid CMS Database, and that might help cleanup a bit. Reboot the dying CMS.
3) Back on the good CMS, like “CMSa” in my example, connect to the Central Management Console (CMC), like http://CMSa/businessobjects/enterprise115/admin, and login as Administrator. Use home -> Servers and select the stopped servers that have “Machine Name” not in the cluster, like “CMSb” in my case. Delete these selected services using the button in the upper right, which I believe cleans up proprietary table entries that are not only annoying but can trigger “re-clustering” given the opportunity. Restart the good CMS.
* If you were unwise enough to clone a CMS and just rename it for a new instance (perhaps virtual machines), then be careful because some registry items may just refer to directories for logs and such. These should not be cleaned up. In our XP Pro testing case of “CMSa” cloned and renamed to “CMSb” for testing, there was a “Documents and Settings\CMSa” folder on CMSb used for logging and such. While we wanted to remove CSMb’s registry items “CMSa” that had resulted from cloning and clustering from connecting to the common DB1, directory entries were not applicable for removal. In general, I do not recommend cloning a CMS, though I have not found anything that is problematic – just this one confusion.
The boring details…
——————
With regards to BO XI R2: how clustering works internally is a mystery, but it is equally illusive what external symptoms indicate clustering is active. After a few more days on this, I found there are a number of configurations through the registry and in the Central Management Server (CMS) proprietary DB that trigger, and help implement, clustering with willing CMS servers. CMSs seem adept at setting these up, and in many cases will quickly reestablish them if partially damaged, but I still find no action that formally cleans them up and separates. But, sometimes, “you gotta’ keep ‘em separated”! That is: CMSs love to cluster, at the drop of a hat, and breaking them apart… well, remember those dogs in the neighborhood?
How do they cluster? If you connect a CMS to a CMS database that was created from a “clusterable” CMS install (I believe this is an option on install, though I can’t recall), the new CMS will be added to the cluster, which changes registry entries on that one and others clustered, and I claim modifies some entries in the proprietary CMS database tables.
I do agree there is a very high probability that clustering is only active if the CMSClusterMembers Registry Key is configured with multiple entries for the current cluster and the servers are running and communicating. Conversely, I find symptoms like the Central Configuration Manager (CCM) application menu at the top right labeled “Computer Name” do not always imply active clustering. In that case it is just a “recent” list, and while you would want to have the list show any real CMSs in your cluster, inclusion in the list does not confirm it is clustered. That one misled me, until I found and cleaned the recent list in HKEY_CURRENT_USER\Software\Business Objects\Configuration Manager “CurrentComputerList”. Inclusion within did not mean the CMS was definitely still “clustering”.
If CMSa and CMSb are both pointed at DB1 they cluster themselves, but subsequently connecting CMSb to DB2 leaves CMSa still showing confusing signs. At minimum, CMSa’s “Services” list (elaborated below in step #3) still lists CMSb “services” as “stopped”, which I find more disconcerting than the “Computer Name” menu – especially since our production system has a pinger that checks the list and throws alarms. So, things may have been OK, but I was not. I like to sleep at night.
Cleaning the CMSClusterMembers Registry Key on relevant CMS servers, then stopping others you wish to dis-associate from your CMS database, and then restarting (for good measure) the “good” ones might put you right enough. However, to avoid confusion you may have to do more cleanup, which are the 3 steps detailed in my steps at the top of this message.
A step further (CMSa/DB1 cloned to CMSb/DB2):
——————————————–
Finally, you may be foolish enough to also be cloning your CMS Database for testing, and this seems even worse. When you connect a CMS to the DB copy it will think it is part of the cluster, talking to your original CMS. I do not think this is a good idea; I doubt CMS authors had this kind of DB mess in mind when coding/testing. However, just having them cluster/connect did not mess up our source DB (though we did very little on the new CMS before shutting it down quickly).
While CMS DB copy as well as CMS copy is not advised, we confirmed that stopping the new CMSb from talking to CMSa before it could cluster avoided mess. We copied the CMS DB1 -> DB2 using an Oracle copy**. Then we duplicated virtual machine CMSa and renamed the OS to CMSb, then put in “\WINDOWS\system32\drivers\etc\hosts” entries on CMSb before we put it on the net, to stop them from talking (confirming they could not ping each other by name). For example, I added “172.16.0.250 CMSa” and “172.16.0.250 CMSa.lan.domain.com” to CMSb’s hosts file, where “172.16.0.250″ is a dead IP. With that, we could connect CMSb to DB2, and it worked without clustering CMSa/DB1 to death, though I would be cautious.
** (we also via the Crystal Copy in the CMS service, user the properties -> configuration to Specify a CMS)
Impressive comment/article Cris. Thanks!
CMSa/DB1 cloned to CMSb/DB2 – correction (DNS is not enough)
In simple cases, just pointing a CMS you want to remove from a cluster to “test/test”, or another valid cluster or new DB, may be sufficient. If not, fixing Julian’s registry entry CMSClusterMembers with reboot may be enough, or you can try my three steps above to dig deeper.
In CMSa/DB1 cloned to CMSb/DB2 it gets worse, and this follow-up is provided only for those who might have deeper problems (and to correct my claim that “DNS” fixes are enough). I do not recommend you try this stuff unless you have to, instead hope you might just find some tidbit of value in some way or another.
For CMSa/DB1 cloned to CMSb/DB2 just hacking the DNS via a hosts file was not sufficient. At first it looked good, then I discovered I had the double list of “services” listed in the “CMC” (http://CMSb/businessobjects/enterprise115/admin) and they looked messed up (CMSa’s looked “running/disable” and CMSb’s looked dead). Moreover, while the original server looked OK that night, in the morning it was clustered again, and CMSa listed double services too (though not as messed up).
Subsequent queries of the DBs and network tracing using Wireshark revealed that the services entries from the Database were enough to somehow allow the systems to talk despite having no DNS, and cluster. My only solution in this case was to clean up the clone before it could cluster to the original system, and clean up any mess still left after that.
With Wireshark I immediately saw the cloned CMS reached out to the original CMS, despite the fact I could not find the IP listed in the registry or DBs of the either server (and DNS/hosts was hacked). I admit I only searched for IP as a decimal value, and it is possible it is stored somewhere as hex or just as a long integer. But, in the end, Wireshark showed that once I clean up the services as well as the registry, on reboot the systems did not talk (cluster).
Here are my more dramtic cleanup steps (which I offer only as “insight” and do not recommend):
CMS config: Central Configuration Manager (CCM) application
CMC example URL: http://CMSb/businessobjects/enterprise115/admin, login as “Administrator”.
1. Point the “cloned/new” CMSb at an invalid “test/test” database (CMS config).
CMS config – stop the CMS service, then use the properties -> configuration -> specify
2. Copy CMSa database DB1 to DB2.
3. Point the “cloned/new” CMSb at the DB2 copy and then start it.
CMS config – stop the CMS service, then use the properties -> configuration -> specify
– systems will try and cluster here, you need to move fast -
4. Immediately use the “cloned/new” CMSb’s “Central Management Console” (CMC):
5. Go into “Servers”, select all services from the original CMSb, delete them using button in upper right.
6. Stop the “cloned/new” CMS service in the CMSb config.
7. Edit “cloned/new” registry and clean out any references to the original CMSb.
8. Reboot the “cloned/new” CMSb OS.
9. Repeat steps 7 & 8, getting concerned if it requires three loops.
10. Rename the cluster on the “cloned/new” CMSb.
CMS config – stop the CMS service, then use the properties -> configuration.
11. In the CMCb “Services” select any red services and use “enable” in upper right.
12. Confirm services on both machines using their CMCs.
13. If there are problems on either machine, do steps 5-9, then 11-13 again.
About the CMS DBs:
One can’t examine a CMS DB directly, since most columns are enciphered, and the relations and abstractions are extensive. Instead you need to go through code that gathers data into represented objects. Unfortunately, the interface probably does not reveal all the data so it is not conclusive.
However, using the BusinessObjects Enterprise SDK I was able to extract loads of data from the DB and search it as I pulled it out to confirm to my satisfaction that clustering is represented in the DB as well – this is might need help cleaning up. Sadly, since details are not conclusive, my conclusion is that the best cleanup is via registry, and restart, combined with CMC “services” operations.
I used the following docs in combination with some sample code:
http://www.sdn.sap.com/irj/boc/sdklibrary “XI Release 2″ ->
http://devlibrary.businessobjects.com/BusinessObjectsXIR2/en/devsuite.htm
BusinessObjects Enterprise SDK
.NET developer guide and API reference
SDK Fundimentals
How do I use the query language to retrieve classes from the CMS repository
Docs revealed three categories of objects that can be pulled from the CMS DB: CI_INFOOBJECTS, CI_SYSTEMOBJECTS, CI_APPOBJECTS. You pull these with SQL queries, which return rows of objects which vary within each category. All objects supported “toString()”, which gave substantial data insight (and could be used for substantial backwards engineering).
My psuedo code from ColdFusion testing to search an object category for data is below, which illustrates the idea. For performance reasons it is strongly recommended you only do this (select “*” from table) on a test system. Note that some of the items in the “toString()” were “xml” packets that do not show if you dump the output as HTML in a brouwser (view source to see them).
InfoStore = createObject(“java”, “com.businessobjects.InfoStore”);
EnterpriseSession = CrystalSession.create(“Administrator”, “CMSb_password”, “CMSb”, “secEnterprise”);
InfoStore.create(EnterpriseSession);
InfoObjects = InfoStore.query(“select * from CI_INFOOBJECTS”);
for (i=1; i lte arrayLen(InfoObjects); i=i+1) {
InfoObj = InfoObjects[i];
if (FindNoCase(“CMSa”,InfoOb.toString())) { // looking for CSMa entries in DB CSMb.
try {
writeoutput(“CI_INFOOBJECTS.” & InfoOb.getKind() & “ – “);
} catch (any excpt) {
writeoutput(“CI_INFOOBJECTS.[ERROR] – “); // getKind() not always supported.
}
writeoutput(Report.toString());
}
}
I hope this is the last you hear from me, I am sick of BO XI R2.
Cris Mooney
Hi,
We Have planned to Create cluster Of CMS servers. But we have planned to installed CMS servers on different physical machines to create a fail over cluster. Is it possible to install only CMS servers on different physical machines.If it is possible to cluster the CMS servers on different Physical machines then, whether we need to install BO on all the physical servers? or Is there any way to install Only CMS on other machines
Thanks,
Guru
You can cluster CMS however you want. You can create multiple CMS on the same machine, on the same SIA (BO XI 3.X only), and on different machines. The only limits are that if the CMS on are the same machine or same SIA (also on the same machine) then you must give each CMS a distinct port number. If you are cluster CMS on different machines then you can have them all use the same port (6400 is the default CMS port).
You do have to install BO on each server; however, you can choose during the installation to do a “Custom or Expand” option where you “Select which components you wish to install”. Then you can chose just to install CMS and deselect everything else.
Thanks A lot Julian,
Your Tips worked out for me.
Guru
Hi, I am new and working on XI 3.1 now.
I have two CMS in CMC and would like delete one.
Because I created one by mistake.
But the delete button under Manage greyed out just for CMS.
I can still delete other Objects.
How could I delete a CMS?
Hi J.J., the easiest way to remove a CMS is to reinitialize the Server Intelligence Agent that is hosting the CMS. This can be done through the “serverconfig” admin tool (add new SIA and just put the same name as the existing one); however, it will wipe out any of your settings on any other servers.
I am not looking at a 3.1 system now, but can you try to stop the CMS first and then try to delete it? Probably not, CMS are a bit “die-hard”.
Hi Julian,
thanks for the reply. The CMS was stopped from the beginning and I was not able to start or disable it. It was basically uncontrollable.
I will try your instruction.
thanks
J.J.
Hi,
I am working with this BOE Reports.My server is pointed to another machine,now I am moving it to another one.But well my question is,I din’t take the backup of the reports I created which are stored in CMS repository.Now since my server is not running,so is there is any other way to get the backup of these files,b;cause i will be reinstalling the system again.plz let me know ASAP
Hi Sunny,
Is your server not able to boot the OS? In such a case try out with another server using import wizard if input and output services of your server(Not running server).
Regards,
RK Raman.
Hi,
I have a 2 server cluster that has been setup with Network Load Balancing. I can find zero information on how to set this up so that the web tier can point to the NLB address rather than the individual server addresses. The server names seem to be hardcoded into the repository so that if I point to the NLB address, even though the traffic goes to the right server/port, BOXI looks up the NLB name, doesnt find it in the repository and thus fails. This is very frustrating. It is my understanding that if I point to a particular CMS server then I lose all the NLB benefits as all the load would go to that one server in the first instance (although would then be distributed by BOXI), and more critically if that particular server goes down then the system will break even though the other server is still up.
Any suggestions you might have are appreciated.
Never mind. As usual, as soon as you ask the question it works. Simply changing the name of the cluster to the NLB address in the SIA properties seems to be working. I had actually tried that configuration earlier and it wasnt working but I think maybe the CMS’ hadnt been restarted in the right order or something like that. There seems to be very little tolerance when installing with this particular configuration.
Hi Ian, I am glad you were able to get past this. I have certainly been in your shoes.
If I were you I would not user an NLB at all. All that you need is to use a BO cluster name. This creates a simple software mechanism that will check all CMS servers defined in the cluster before giving up. Usually, you would rename the default cluster name (often a server’s name) to a generic easy to remember name. Then you would just as this in your web tier as your default CMS. For example, in BO XI 3.X you would put this in InfoViewApp, CmcApp, and OpenDocument. You would also define your cluster CMS members in a clusters.config file placed in your PlatformServices web application. I should dedicate an article to this.