Sizing Limits to Web Intelligence Report Server Maximum Simultaneous Connections
In Business Objects XI R2 newly created Web Intelligence Report Servers default to 50 “Maximum Simultaneous Connections” (editable in the Central Management Console, or CMC). In Business Objects XI 3.X the “Maximum Simultaneous Connections” setting is defaulted to 100! Can we read anything in to these changes in default settings? Do the default settings mean anything at all with regards to the settings you should have on your system?
Sizing Limits to Maximum Simultaneous Connections
General consensus of our sources tell us that XI 3.X is better coded and can handle more connections than XIR2, with regards to WebI Report Servers. Nevertheless, it would be quite a leap to say that all other things as equal as possible that XI 3.X WebI Report Servers can handle twice as much traffic. Seriously, don’t count on it.
I feel I should add personal experience and best practice here. I see the default setting for “Maximum Simultaneous Connections” on the WebI Report Server as a maximum setting. I have personally never exceeded it nor witness it exceeded by anyone. It is generally held that if you need more Simultaneous Connections then you ought to add another WebI Report Server to your environment. Of course, keep in mind that SAP Business Objects’ general guideline is that you have no more than 1 Web Intelligence Report Server per available CPU core (for example, a server with 4 quad-core CPUs has 16 CPU cores); so there is a limit there too.
Real-World Web Intelligence Report Server Sizing
A production system will generally run better having more Web Intelligence Report Servers with lower Maximum Simultaneous Connections. For example, a server with 4 dual-core CPUs would run better having 8 WebI Rpt Servers each set at 30 Maximum Simultaneous Connections than it would having 4 WebI Rpt Servers each set at 60 Maximum Simultaneous Connections in CMC. In the real-world tuning and balancing usually are based on observed performance within configuration guidelines.
To get your sizing in the right place you should know what you maximum concurrent users are (logged-in users plus concurrent schedule jobs). This number is essentially how many Maximum Simultaneous Connections you will need. So let’s say you never have more than 100 concurrent users, but you can at those peak times also have 25 scheduled jobs. Keep in mind that most users will only do one process at a time, but some like me will be refreshing one report while editing another simultaneously. A scheduled job will always only be one connection. So you can safely say that you require only 150 simultaneous connections (with some wiggle room).
Now your server had only 4 CPU cores in it. So this one is easy 150 divided by 4 will give you 37.5. Round that up, because I suggest you have at least 150 and that you keep the same number on each WebI Report Server. So you can set each one’s “Maximum Simultaneous Connections” to 38 using the CMC. In XIR2 this might be pushing the limits, but in Business Objects XI 3.X this should be a comfortable setting.
Hi Julian,
Quite interesting article. This remind me about one of the error we faced regularly. The Error was
“Could not initialize the Report Engine Server”. When we check CCM, we can see Report Server status as Green, but in fact service would have hanged or not responding. This happens due to overloading of requests in Webi Report Server. Easy solution is just restart the service from CCM,though not a permanent solution, its quite handy. Permanent solution as explained above to add one more Webi Report Server. But as suggested by you we never edited “Maximum Simultaneous Connection”. This is indeed quite making lots of sense. Ideally how do we do the sizing, i don’t have much idea. Why don’t you start a new thread on Sizing? This would be really worth a try
Thanks and Regards
Aravind
Hi Aravind,
Yes, I was thinking a sizing topic is long overdue. I have a few guidelines from BO consulting engagements, BO documentation, and experience. I think if I put this all down as a new topic others might add their own experience and we could end up with a decent collection of guidelines.
This also makes me think that if I can find the time I should go back to some of the more commented topics and add some of the content in the main article to be sure that the search engines index it for the benefit of others (based on the possibly erroneous assumption that some search engines ignore comments).
Thanks, Julian
Our connection limit is set at 50 across 4 WebI servers. How can you monitor how many are being used? I see in the metrics that 300+ requests are being served, but not how many connections are used.
The only precise method of measuring connections is to look at the target databases. I have created reports in the past that returned the number of sessions based on connecting application from a database. If you have a lot of data sources or if you cannot get an account with privileges to query sessions than this won’t work for you.
On a side note, most BO consultants recommend one WebI server per CPU and that the connection limit be between 25 and 40 (depending on the RAM available, for example 8GB RAM would be good enough for 40). I need to write some sizing articles.
Thanks Julian, but we have 900+ Universes and over 600 Databases, so what you suggest is not possible. I was hoping there was somthing on the BO Report Server side to monitor at least a current connection count and connection queue. The Metrics in the CMC only show #Requests on each WebI server. I’m guessing there is one or more connections used per request…is that true? If I have 300 requests, and the server only handles 40 simultaneous connections, what’s happening to the other 250 requests? I’m assuming there’s a erquest queue somewhere.
It seems that the issue at hand is the definition of “requests” and of “connections”. It seems to me that “connections” refers the number of WebI client users. A “user” then is anyone or thing that needs the WebI Processing Server to do something. So if I open a WebI report for viewing, that is a connection. If I also have a WebI report open for editing, another connection. If a job server is refreshing a WebI report, another connection.
Now “requests”, that is a strange beast. I wonder if you always observe that requests is higher than connections. It can’t be queued requests for connections, because then users would be sitting there looking at hourglasses or errors. I need to experiment some more with this. I tried some tests in a new BOE XI 3.1 SP2 FP 2.4 environment I just built with no users and I found no surprises or revelations yet.
I’m glad I found this thread. Very useful. We are in a XI R2 (SP4) two-node clustered environment. Each node has one WIRS with a Maximum Simultaneous Connection setting of 50. We have 2 Quad Core CPU’s – but half of the cores are disabled in the bios for licensing purposes. We have approximately 700 users and a typical maximum user concurrency of less than 10%. Despite this, we’ve intermittently seen the Web Intelligence Report Server reaching its threshold of 50 Maximum Simultaneous Connections and then crashing, all while the other node’s WIRS is up and running and ready to accept connections. we do have a Cisco CSS load balancer on top of that architecture that routes the HTTP requests. It’s set to “Sticky Source IP”, which the vendor says overrides the default load balancing and forces the processing to one node only, which they say, is why the other node’s WIRS could not process requests once the first node’s WIRS crashed. Do you have any experience with CSS load balancers? Does this make sense to you? regardless, if that is indeed the case, would adding more WIRSs to each node resolve the issue?
Thanks,
Sleuth
The Sleuth is back! I don’t see how a load balancer comes in to play here. The CMS should be doing its own load balancing and routing requested WebI resources in an even distribution on average. The load balancer does share the load between the web/http servers, but nothing else. Beyond determining which web server a user receives it is out of their hands which CMS you are routed to (unless you did not setup your CMS cluster in correctly int he web app). So still, I can’t see how a load balancer would impact any WebI server assignments. CMS is the traffic cop for the cluster, nothing else.
But obviously, something is off. Do you have a balanced number of WebI Processing Servers on each machine? Do you have any server groups that are not evenly balanced across the cluster? What happens if you shutdown the WIRSs on the node that crashed, does the load shift to the other one?
Thanks for the feedback! I’ve never worked with load balancers, so when the SAP rep told me this “Sticky Source IP” setting is forcing traffic and processing to one server, I counted the load balancer among the possible causes for the problem. Regarding setting up the CMS cluster incorrectly in the web app, an outside consulting firm created a document instructing how to set this up. I will review that document with an SAP rep.
We only have one WIRS per machine, and we only have two machines in the cluster. We have no server groups set up. We haven’t tried shutting doen the WIRS on the node that crashed because it’s a pretty-much 24/7 environment with a small weekend maintenance window. I can try that this weekend.
Thanks,
Sleuth
Hi Sleuth, A guideline I have used on dedicated WIRS boxes is to run one WIRS per CPU (or core). So if your machines’ cores were all enabled you could run 8 WIRS on each box. Now, since your box is not dedicated, I would at least subtract from this the number of CMS running on the machine. So if you were running 1 CMS and all cores were enabled, then you could do 7 WIRS.
I know that you do not have all cores running (too bad) and that you are probably running EVERYTHING on the same box, (don’t worry I have done this). So if you are running just 4 cores on a machine then I would try running 2 or 3 WIRS on each machine. I would also try disabling all WIRS on one machine and seeing how the CMS handles sending all future requests to the other WIRS.
Hello,
I have a sort of side question. What do the ‘system connections’ represent in the CMC settings -> Metrics Screen. Logged on represent the concurrent CMC sessions, what about system connections ?
Thanks in advance.
Hi all,
We experiencing a problem with a xcelsius dashboard that loads data from 25 webi reports using LiveOffice connection and from what I’ve read here it seems that it might have to do with load balancing issues. The problem arises randomly when users are using the dashboard and suddenly they get a RWI error saying that the document couldn’t be obtained because of obsolete token.
The server has 8 CPUs and 16GB of RAM. At the beginning we set up 2 webi servers with 65000 max connections. Now I think it is not the best configuration.
Do you recommend me to set up 4 webi servers with a number of connections of 40? Would this be enough?
All comments are welcome. Thanks in advance.
cheers,
Christian
Hi Christian, 65000 max connections would not in itself harm the system (I think) until there is a load on the system. Once a load is received, such a high number would likely result in faulty and uneven load balancing by WebI process. In fact, you might be able to see this by monitoring your processes and notice that one is getting abused while the others are kicking back having an nice nap.
Depending on what else you are running and how heavily you are using it I would strongly consider running 1 WebI processing server per CPU and set each to 40-50 connections. You will know that your connections are too low when you start getting RWI errors (telling you know WebI processing connections are available for the requested action). 8 x 50 = 400. Do you think that your system has 400 concurrent WebI processes (interactive refreshes/views + schedule refreshes + report edits)? If so, you might push this number higher, but keep all servers at the same level and I would not go much higher than 100 connections each.
We are Could not initialize the Report Engine Server”. with Communication Exception..
Can someone plz advise on this.
What is the exact thing we need to look in..?
Any help would be appreciated
Since want to understand the terms, what is the difference between webi session and CMS Connection.?
Hi John,
For “Could not initialize the Report Engine Server” I would look very closely at CMS database connectivity, the database where your CMS repository / InfoStore is located. Make sure that you can reach this database from the server using the same account that you use to run the CMS.
WebI sessions are sessions assigned by the CMS internal load balancer to the WebI Processing Servers. They can natively open connections to data sources (sources of data in the reports) without the need for a Connection Server.
CMS Database Connections are the number of connections that the CMS attempts to open on its InfoStore / metadata repository. The number of connections are defined by the parameter “ndbqthreads” in XI R2 (10 maximum) and in XI 3.X these are a text box on the CMS editable in the CMC (default 14, max 99?).