Recently, we came across a problem with a new OBIEE 12c environment configured for a client. For this client, there was a requirement to regularly export reports with huge amounts of data to Excel. The reports were basically tables or pivot tables with over 100K rows and more than 20 columns. This functionality was working fine in their old OBIEE 11g environment, but when trying to similarly export the reports to Excel in 12c, they were getting the error below:
After some initial investigation, we found some clues in the sawlog.log file (located in <ORACLE_HOME>/user_projects/domains/bi/servers/obips1/logs in OBIEE 12c):
[2017-10-25T10:34:27+03:00] [OBIPS] [ERROR:16] [] [saw.rpc.server.handleConnection] [ecid: 005MzeMrkY5CWrK5qVT4iY000GhE00002h,0:135826] [tid: 493369088] [SI-Name: ] [IDD-Name: ] [IDD-GUID: ] [userId: ] Exception in handler thread. An error occurred during execution of "send". Broken pipe [Socket:31]
Error Codes: ETI2U8FA
Location: saw.rpc.variablemos.finalize, saw.rpc.server.responder, saw.rpc.server, saw.rpc.server.handleConnection, saw.rpc.server.dispatch, saw.threadpool.socketrpcserver, saw.threads
[[
File:socketrpcserver.cpp
Line:578
Location:
saw.rpc.server.handleConnection
saw.rpc.server.dispatch
saw.threadpool.socketrpcserver
saw.threads
]]
And in the JavaHost log jh.log (located in <ORACLE_HOME>/user_projects/domains/bi/servers/obips1/logs):
[2017-10-25T10:34:27.419+03:00] [messageprocessor] [WARNING] [] [saw.messageprocessor] [tid: 41] [ecid: d33c0ecb-a5f5-4bcc-8bad-a3fc6bc44fda-00000002,0] Unexpected exception. Connection will be closed[[
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at com.siebel.analytics.javahost.io.ChannelWithTimeout.write(ChannelWithTimeout.java:174)
at com.siebel.analytics.javahost.io.BlockingIOImpl.write(BlockingIOImpl.java:32)
at com.siebel.analytics.javahost.io.BlockingIOOutputStream.writeBuffer(BlockingIOOutputStream.java:64)
at com.siebel.analytics.javahost.io.BlockingIOOutputStream.flush(BlockingIOOutputStream.java:27)
at com.siebel.analytics.javahost.io.BlockingIOOutputStream.write(BlockingIOOutputStream.java:48)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:167)
at com.siebel.analytics.web.sawconnect.sawprotocol.SAWProtocolOutputStreamImpl.finishChunkInternal(SAWProtocolOutputStreamImpl.java:158)
at com.siebel.analytics.web.sawconnect.sawprotocol.SAWProtocolOutputStreamImpl.finishMessageOnServerSide(SAWProtocolOutputStreamImpl.java:184)
at com.siebel.analytics.web.sawconnect.sawprotocol.SAWProtocolOutputStreamImpl.finishMessageOnServerSide(SAWProtocolOutputStreamImpl.java:169)
at com.siebel.analytics.javahost.AbstractRpcCall.processMessage(AbstractRpcCall.java:299)
at com.siebel.analytics.javahost.MessageProcessorImpl.processMessage(MessageProcessorImpl.java:200)
at com.siebel.analytics.javahost.Listener$Job.run(Listener.java:223)
at com.siebel.analytics.javahost.standalone.SAJobManagerImpl.threadMain(SAJobManagerImpl.java:207)
at com.siebel.analytics.javahost.standalone.SAJobManagerImpl$1.run(SAJobManagerImpl.java:155)
at java.lang.Thread.run(Thread.java:748)
]]
Since we weren’t sure at the beginning if this was an issue with the Presentation Services or with other components, we edited the instanceconfig.xml file as a first measure to increase the limits on the number of rows returned by the reports. The instanceconfig.xml is located in /config/fmwconfig/biconfig/OBIPS in OBIEE 12c and is the main file to look at when facing issues related to limits on the number of cells, rows, or cells returned by reports.
Specifically, we increased all the values for the parameters controlling the maximum cells, pages, columns, rows, and sections for both Table and Pivot views. In the case of the Table view, we also added the DefaultRowsDisplayedInDownloadCSV tag, which was not present by default. This is how our Pivot and Table views looked like in our instanceconfig.xml file afterwards:
<Pivot>
<MaxCells>4000000000</MaxCells>
<MaxPagesToRollOutInDelivery>1000</MaxPagesToRollOutInDelivery>
<MaxVisibleColumns>500</MaxVisibleColumns>
<MaxVisiblePages>1000</MaxVisiblePages>
<MaxVisibleRows>100000</MaxVisibleRows>
<MaxVisibleSections>30</MaxVisibleSections>
<DefaultRowsDisplayed>100</DefaultRowsDisplayed>
<DefaultRowsDisplayedInDelivery>75</DefaultRowsDisplayedInDelivery>
<DefaultRowsDisplayedInDownload>2500000</DefaultRowsDisplayedInDownload>
<DisableAutoPreview>false</DisableAutoPreview>
</Pivot>
<Table>
<MaxCells>4000000000</MaxCells>
<MaxPagesToRollOutInDelivery>1000</MaxPagesToRollOutInDelivery>
<MaxVisiblePages>5000</MaxVisiblePages>
<MaxVisibleRows>9000000</MaxVisibleRows>
<MaxVisibleSections>5000</MaxVisibleSections>
<DefaultRowsDisplayed>100</DefaultRowsDisplayed>
<DefaultRowsDisplayedInDelivery>7500</DefaultRowsDisplayedInDelivery>
<DefaultRowsDisplayedInDownload>2500000</DefaultRowsDisplayedInDownload>
<DefaultRowsDisplayedInDownloadCSV>2500000</DefaultRowsDisplayedInDownloadCSV>
</Table>
The new tag for CSVs should fix the export issue when exporting the data in CSV format. However, we found that the problem persisted for exports to Excel after these changes. After checking the logs, we saw clearly that the problem was now with the JavaHost service. The JavaHost service gives Presentation Services the ability to use different functionalities contained in Java libraries that are needed for graphs and charts generation, advanced reporting, report exports, etc. In our case, the service was working correctly for reports and charts, but was throwing an error when exporting the data to Excel. As a secondary measure, we decided to increase the heap size of the JavaHost component. In the obijh.properties file located in /bi/modules/oracle.bi.cam.obijh/env/ we increased the memory reserved for JavaHost requests to 32GB (the default was 8GB):
MEM_ARGS=-Xmx32768M
We had a machine with almost 1TB of memory available, but we thought this was a reasonable limit that should be enough for any request to the JavaHost service.
In the configuration file for the JavaHost, the config.xml file located in /config/fmwconfig/biconfig/OBIJH/ we also increased the Socket timeout, which is the time that a JavaHost thread will wait for a message before considering it as “idle” and throwing an error:
<MessageProcessor> <!-- How much time worker thread should wait for a message before returning socket to the "idle" pool. Initial messages in the idle pool are handled using Java NIO Channels. --> <SocketTimeout>1200000</SocketTimeout> </MessageProcessor>
By default, this parameter is commented in the xml file, so the value for the timeout is 5 minutes. This wasn’t enough for the JavaHost service to process the large data request that we were executing, so we decided to increase it to 20 minutes in order to provide enough time for the service to process large quantities of data within reasonable limits.
For exporting the content of analyses into formats such as PDF, Microsoft Excel, or PowerPoint, the JavaHost service uses the core libraries of BI Publisher. These libraries are embedded within the JavaHost service, so you don’t need BI Publisher running or deployed in your environment for them to work. In the JavaHost configuration, elements related to the BI Publisher libraries are located within the XMLP element in the config.xml file. To export large quantities of rows to MS Excel, we also increased the InputStreamLimitInKB parameter and changed the ReadRequestBeforeProcessing value to false in this file, as recommended by Oracle in Doc ID 1595671.1:
<XMLP> <InputStreamLimitInKB>2097152</InputStreamLimitInKB> <ReadRequestBeforeProcessing>false</ReadRequestBeforeProcessing> </XMLP>
The InputStreamLimitInKB parameter specifies, in KB (1024 * X), the maximum input size for requests that are sent to JavaHost. The default value is 8192KB. A value of 0 deactivates this limit; however, this should be used for testing purposes only. A value too high may cause the JavaHost to become unstable or crash because of excessive resource allocation. The value should be configured to a reasonable value according to the demand of requests to the JavaHost (charts, graphs, and exports) and the size of the datasets. We set it at 2GB (2097152KB = 1024*2048).
The ReadRequestBeforeProcessing parameter specifies whether or not to wait to process the request until a file is completely read. For exports from OBIEE, it is recommended to set this parameter to false. This way, data is streamed to JavaHost gradually rather than saved to a file first and then processed, thereby improving performance.
After these changes and a full restart, we were still facing the same issue, only that now we had to wait 20 minutes until the error came out. We knew that we were heading in the right direction, but there was still something that was causing the JavaHost service to throw an error for these extremely large reports. So as per recommendation from Oracle support, and as a last measure, we edited the bridgeconfig.properties file, which is the configuration file for the Presentation Services Plug-In located in /config/fmwconfig/biconfig/, adding the following parameters to increase the socket timeout and connection pool maximum connections:
# Number of seconds to wait for a response from sawserver # Default value is 360 seconds oracle.bi.presentation.sawconnect.ConnectionPool.SocketReadTimeoutSec = 3600 # Maximum number of connections in pool. Default is a function of number of processors. # Default value is 128 oracle.bi.presentation.sawconnect.ConnectionPool.MaxConnections = 4096
After another restart of the whole OBIEE environment (including AdminServer), we tested the export again, and we were able to export more than 999K rows to Excel! Moreover, we noted that the export took less time after these modifications.
So, in summary, to be able to export large quantities of rows to Excel in OBIEE 12c, you will need to do the following modifications:
<DOMAIN_HOME>/config/fmwconfig/biconfig/OBIPS/instanceconfig.xml: increase the values for the default limits on rows, columns, and cells.
<ORACLE_HOME>/bi/modules/oracle.bi.cam.obijh/env/obijh.properties: increase the memory reserved for JavaHost requests (-Xmx parameter).
<DOMAIN_HOME>/config/fmwconfig/biconfig/OBIJH/config.xml: increase the socket time out and input stream limit, and set the read request before processing flag to false.
<DOMAIN_HOME>/config/fmwconfig/biconfig/bridgeconfig.properties: add the properties to increase the socket time out and the maximum number of connections in pool.
After this and a full restart, you should be able to export any report to Excel.
Conclusion
Even though OBIEE offers a complete solution for data consumption and analysis, Microsoft Excel is still a widely used tool across companies to view and share data. That is exactly why many clients still require the ability to export tabular data from OBIEE reports to Excel. Unfortunately, the default configuration of OBIEE 12c comes with some limitations for exporting large quantities of rows to Excel. In this article we give you some tips to overcome these limitations.
Contact us if you would like to hear more about this solution or any other related topics.
The post OBIEE 12c export to Excel issue appeared first on ClearPeaks.