Anatomy of BI EE Page Rendering - Use of HTTP/Web Server & Compression

There are many aspects/variables thatcan affect the performance of a BI Deployment. Generally whenever a BI Dashboard is slow the general tendency is to attribute the slowness in performance to eitherthe BI Reports or underlying databases. Of course, in 90% of the cases this will be truebut there are cases where some other factors affect the page load times. In thisblog post, we will try to investigate the various other factors like browser cache,HTTP Server, HTTP Server compression etc. Also, we will try to understand whathappens from a communication standpoint between the browser and the Server. How much of network traffic gets passed, what is the optimal bandwidth etc.

To basically find out what happens at a protocol/network level we will be using the Firebug & YSlowNetwork Monitoring extension. There are other network monitoring and sniffing tools available todo this, but i prefer the Firebug/YSlow (& LiveHTTPHeaders) extension as it gives a graphical representation of each GET/POST request madealong with the times and size. Lets load up the extension and point it to the SampleAppLite Overview dashboard (directly using a URL without the HTTP Server enabled) from a browser that has its browser cache cleared completely.

Instead of a single HTTP call we are having a lot of HTTP Calls. So, lets trace the output(rendered HTML) of the first HTTP call (http://108.166.89.170:7001/analytics/saw.dll?Dashboard&PortalPath=%2Fshared%2FSample%20Lite%2F_portal%2FQuickStart&page=Overview)

So what we have is, the first HTTP call brings all the necessary data from all the reports (with table, Pivot view etc). So,the first HTTP call does not bring in all the images, css & js files. Those are all done in incremental GET calls in parallel.To confirm, lets look at the HTTP response(just the response from server - not the HTML) of the first GET call (the same above url)

If you notice, the HTTP response back from the server has server references (like relative urls to the location of js, image &stylesheet files etc). To retrieve them, the browser has to make further calls back to the server and display them. Lets look at the overall size breakup of the Overview dashboard. The idea is to find out the overall size of an average dashboard and the breakup of sizes by individual image files, js files etc

Interesting. Java Scripts alone take up bulk of the overhead. The actual data is pretty less. A total of 122 js files and a totalof 49 image files. Thats a lot for a single page. Ok, now lets try to understand where browser cache comes into picture. To understandthat lets take a look at the breakup shown below on the overall network data size with and without browser cache.

If you notice, with browser caching alone (on image files, js files etc), there is a significant reduction of packet size transferred from the server to the client i.e. from 4MB to 140KB. Thats a lot of caching. Also, if you notice, there is no cachingfor the main/first HTTP request as expected. So, as a standard we need to have browser caching enabled for production environments as that can significantly reduce the number of HTTP calls as well as the size of packets. So what happens is, after receiving the first HTTP request the client browser decides whether to send a HTTPrequest to the fetch the relative URL images, js or whether to get it from the local browser cache (if it is populated). The LiveHTTPHeaders outputshows the requests below (after the browser cache is warmed up)

Very interesting. As you see, as soon as the browser cache is warmed up there are only limited number of GET requests backto the server.

To summarize, there are 3 things that we can notice

BI EE just does not make a single HTTP call(obvious but a definite thing to keep in mind) to the Server from the browser
With browser cache significant amount of data can be reduced from being passed on over the network
BI EE - to render a single web page makes close to 300-400 HTTP requests to get almost 4MB of data

So the question is what role does a HTTP Server play in all of this? Can HTTP Compression reduce the amount of traffic?

At a high level this is what happens

Step 1: User enters a BI EE URL and hits on enter in the browser. A HTTP request is passed on to the server.
Step 2: The server on receiving the request, does some internal processing(basically fires SQL to get the dataand then creates the HTML) and then formulates the response.
Step 3: The browser will receive the response. The sample response is nothing but the source of the HTML output page.
Step 4: Based on the first HTTP response it decides whether to use browser cache(if populated) or to make further http requests to the server
Step 5: If the browser cache is enabled, only those images,css,js files that are not cached will generate the HTTP request. The HTTPrequests will be generated in parallel. The number of parallel requests depend on the browser.
Step 6: HTTP request is generated for all other remaining files required for browser rendering. Modern browsers can start renderingthe page as and when the first HTTP response is received. Then after that once the individual images are retrieved, browsers can render them incrementally (hence we get an impression as though the pages are being painted especially on slow networkinstalls)

Pretty straightforward. But this does offer insights into what can be done to reduce the traffic and thereby reducing contention.

Browser Cache once enabled can reduce the traffic significantly atleast for static images. This can reduce the numberof HTTP requests.
BI EE transfers around 4 to 10 MB(or more) of traffic per page (not a single transfer - this happens in parallel in chunks).There is scope for compression.
Browser Cache though very good still cannot cache every static file/resource. This is best done by a separate HTTP server. Also, the first time access of new users to a website will be slow if we depend on browser cache (until that warms up). So,HTTP Server ideally is the best way to cache static resources so that all end user static file response will

Remember HTTP Server caching does not reduce the number of GET requests. But it just reduces un-necessary GET requests to the serverapplication server. So, if you want to reduce the CPU/Memory usage on the main application server by HTTP Server cachingthen HTTP Server has to reside on a completely different box than the main Application Server.

Having the Web Server and App Serveron the same box might not give that much of performance gains (though it still makes sense to isolate the main app server processfrom serving static resources - http server has to do that as it can serve in-memory as well).

So, lets try to take a look at the BI EE SampleAppLite Overview dashboard page through the Web Server URL (rather than the appserver port).

As you see not much of actual gain. But the gains will become apparent as more users start accessing the URL - HTTP Server memory resources will get cached in-memory in its process space and hence can serve out requests faster than the app server.

Now lets try to understand how the page load times relate to the network bandwidth. As we had seen earlier almost 4MB of data gets movedover the network for loading one page. Majority of it can be cached either at the http server layer or at the browser level. But still 4MBis a lot of data.

So, lets try adding a compression at the HTTP Server layer. The idea behind this is HTTP Server before sending the response will compress using gzip(that will be specified in the HTTP Header so that the client browser can uncompress). The client browser will then uncompress the packets using gzip again. To enable compression the following needs to be done on the httpd.conf file of the http server

Ensure that the following lines are not commented out in the httpd.conf file.

LoadModule expires_module "${ORACLE_HOME}/ohs/modules/mod_expires.so
LoadModule deflate_module "${ORACLE_HOME}/ohs/modules/mod_deflate.so

Add the following after the LoadModule section

Now lets take a look at the size of the data that gets transferred over the network.

As you see there is a reduction of almost 3MB in the data size that gets transferred. Also, if you take a look at the headerof the response, there is a content-encoding of gzip that gets enabled.

The compression is recommended only if you are on a WAN network. Remember, compression will require CPU both on the HTTP Server side as well as the client side. Also LAN networkswhere there is no issue of bandwidth, compression ideally should not be enabled.

HTTP Server Support:

As a footnote, if you are in the process of enabling HTTP Servers for your existing BI EE 11g installs, ensure that you are on theright patch sets of Oracle Web Tier. For example, BI EE 11.1.1.5 supports only 11.1.1.4 version of Oracle Web Tier. And 11.1.1.4version of Web Tier is not a direct install - instead one has to install 11.1.1.2 version and then upgrade that version to 11.1.1.4 using the patchset. But for BI EE 11.1.1.6, Oracle Web Tier 11.1.1.6 is supported (no Patchset for this).