11i/R12 Apache Common Issues with Solutions   3 comments

I dont know where I got this document. But I am thankful to the person who collected all these issues in one single Document.

Credits to that Person

Apache Common Issues

General Information to Troubleshoot Apache Issues.

1)If you mistype any directive in Apache (httpd.conf or any conf file), Apache will not start & it will not write even in error_log. In these case it will report any error message to file
$COMMON_TOP/admin/log/$CONTEXT_NAME/adapcctl.txt

So first check in the above file

2)Found *.pid file in Apache_Top/Apache/logs/ even after bounce of an Apache web server (When Apache starts it creates .pid file in logs directory and when you shutdown apache it cleans this .pid file). So if this file is already there then Apache will not start.
Fix: Simply move this file

3)Could not bind port (Port already in Use) or Sometimes apache and concurrent Manager will interchange their port no.’s while starting results error and both of them will not up
Fix: To overcome this up the services in this manner
1)Listener
2)Apache web server
3)Concurrent manager
4)adstrtal.sh

4)Error 500 – Internal server error
Fix: 1) Clear Cache (Application Server Cache)and start the services

5)If there are no error messages in error_log or error_log_pls
then there might be issues in starting Jserv component of web server (which caters Java requests by mod_jserv)

6)If you don’t know port number of your web server you can check it in file $IAS_ORACLE_HOME/Apache/Apache/conf/httpd.conf with directive Listen & Port

7)Once you are able to access http://hostname.domainname:port and if you are getting any error message that means your web server (Core Apache) is working but there are various components called as modules and there may be issues with those modules which we will check these modules . few modules of our interest are
-mod_jserv
-mod_pls
-mod_oprocmgr
-mod_ossl (If you have ssl enabled on web server)
-mod_osso (If single sign on server is enabled on web server)

8)Once in a while adapcctl.sh start will not work then ultimate you have to go and invoke apachectl script file in Apache top/apache/bin to start Apache webserver.

09)Error : Your session is no longer valid while trying to loging to applications.
The issue is resolved by changing the profile option session_cookie_domain to null from domain.com

10)HTTP-403 forbidden message after entering the username and password in the “Apps Logon Links” page. [Not authorised to view this page]
Check for Apps password in $APACHE_TOP/modplsql/cfg/wdbsvr.app ,Comment out custom_auth entry in $APACHE_TOP/modplsql/cfg/wdbsvr.app and bounce apache.

11)Error message: You have insufficient privileges for the current operation.
Verify and update SESSION_COOKIE_DOMAIN in ICX_parameters if it is wrong.

Example :

update icx_parameters set SESSION_COOKIE_DOMAIN=’NEW_VALUE’ where SESSION_COOKIE_DOMAIN=’OLD_VALUE’

12)AppsLocalLogin?.jsp page error outs….

  • Edit the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties
  • Set wrapper.bin.parameters=-DLONG_RUNNING_JVM=false
  • add the line wrapper.bin.parameters=-DCACHEMODE=LOCAL
  • Restarted Apache for these changes to take effect and retested the issue

13) Login to the applications URL fails with
500 Internal server Error
java.lang.NoClassDefFoundError at
oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect.java:126) at
oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect.java:170) at com.evermind[Oracle
Containers for J2EE 10g (10.1.3.0.0)
].server.http.HttpApplication.loadServlet(HttpApplication.java:2231) at com.evermind[Oracle
Containers for J2EE 10g (10.1.3.0.0)
].server.http.HttpApplication.findServlet(HttpApplication.java:4617) at com.evermind[Oracle
Containers for J2EE 10g (10.1.3.0.0)
].server.http.HttpApplication.findServlet(HttpApplication.java:4541) at com.evermind[Oracle
Containers for J2EE 10g (10.1.3.0.0)

Performed the below steps, but it did not resolved the issue
1) Bounced Apache server, cleared cache and validated ports
2) Bounced all Middle tier services and killed defunct processes
3) Bounced Unix Server

Solution
=======
Java object Caching Configuration parameter was Enabled in oc4j.properties
$INST_TOP/ora/10.1.3/j2ee/oacore/config
Disabled LONG_RUNNING_JVM=false from True
Bounced Apache server and could login to the applications URL without issues

                           ISSUES AND SOLUTIONS

ISSUE 1:

Description

When Apache script adapcctl.sh is issued.It shows that Apache is started.On checking the status Apache is stopped

Cause

The error log file in the folder
cd $APACHE_TOP/Apache/logs

Showed

fopen: No such file or directory httpd: could not log pid to file /var/opt/_/Apache/logs/httpd_pls.pid

SOLUTION 1

Created the missing directories /_/Apache/logs under /var/opt folder
Bounced Apache
**********************************************************************
ISSUE 2:

Description
oacore was not comming up in the configurator node.
opmnctl status

ias-component process-type pid status

OC4J oacore N/A Down
OC4J oacore N/A Down
OC4J oacore N/A Down
OC4J oacore N/A Down

Cause

Check the oacore error log.

Example:

Below Log file gives root cause. //mtlog/_/logs/ora/10.1.3/opmn/oacore_default_group_2/oacorestd.err

14:11:56 Error initializing server: //inst/apps/_/ora/10.1.3/j2ee/oacore/config/server.xml,
Fatal error at line 70 offset 1 in file://inst/apps/_/ora/10.1.3/j2ee/oacore/config/server.xml: .:
XML-20100: (Fatal Error) Expected ‘EOF’.
SOLUTION 2

Edit the /sid/inst/apps/SID_hostname/ora/10.1.3/j2ee/oacore/configserver.xml file and remove the extra line in the file. Restart and check oacore will be starting. This sudden change in server.xml seems to be because of a bug.
Bug – 6702510
************************************************************************
ISSUE 3

AppsLocalLogin?.jsp page was displaying errors

Cause

Extract from the Jserv log file =========================== [27/02/2008 16:57:14:447 CST] weboamLocal/oracle.apps.fnd.oam. servlet.ui.OAMServlet:
Sucessfully initialize oaosu.OAMServlet [27/02/2008 16:57:14:447 CST] Servlet Zone rootauohstrzt11 initia lization complete [27/02/2008 16:57:15:101 CST]
Creating noun for: oracle.jsp.JspServlet [27/02/2008 16:57:15:102 CST] Creating noun for: AppsLocalLogin?.jsp
[27/02/2008 16:57:15:120 CST] oracle.jsp.JspServlet: init [27/02/2008 16:57:28:066 CST] Creating noun for: oracle.jsp.JspServlet
[27/02/2008 16:57:28:067 CST] Creating noun for: AppsLocalLogin?.jsp [27/02/2008 16:57:28:078 CST] oracle.jsp.JspServlet: init
[27/02/2008 16:57:29:753 CST] JspServlet?: unable to dispatch to requested page: Exception:java.lang.NoClassDefFoundError

SOLUTION 3:

Edited the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.properties
a) Set wrapper.bin.parameters=-DLONG_RUNNING_JVM=false
b) Add the line wrapper.bin.parameters=-DCACHEMODE=LOCAL
c) Restarted Apache for these changes to take effect and retested the issue
********************************************************************
ISSUE 4

New Message when starting Apache using adapcctl.sh start

Cause

It takes 7/10 minutes to clear this cache directory if present
apache Web Server Listener is not running.

Starting Apache Web Server Listener (dedicated HTTP) …

Removing gantt cache directory
/ppomwi/applmgr/r12/apps/apps_st/comn/webapps/oacore/html/cabo/images/cache
Successfully removed the gantt cache directory:/tcge1i/applmgr/common/html/cabo/images/cache/gantt

Apache Web Server Listener (PLSQL) is not running.

Starting Apache Web Server Listener (dedicated PLSQL) …

SOLUTION 4

$COMMON_TOP/webapps/oacore/html/cabo/images/cache

Wait for atleast 7 to 10 minutes and this cache directory will be cleared and Apache will be started successfully
***********************************************************************
ISSUE 5

Apache server is up and running.
But adapcctl.sh status shows
adapcctl.sh version 115.54
Apache Web Server Listener is not running. Apache Web Server Listener (PLSQL) is not running.

adapcctl.sh: exiting with status 1

Cause

Incorrect Lock file path given in httpd.conf
LockFile? $APACHE_TOP/Apache/log/PPOMWI_httpd.lock
and Incorrect path given in adapcctl.sh script.

SOLUTION 5:
Modified the path of lock file in $APACHE_TOP/Apache/conf/httpd.conf file.
LockFile? /var/opt/SID_/httpd.lock
and Modified the path of adapcctl.sh file by commenting the wrong path
************************************************************************
ISSUE 6

After selecting the E-Business home page got internal server error Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.

Cause
Found that all the properties files as below
jserv.properties jserv_restrict.properties zone.properties zone_restrict.properties
were pointing to a dbc file name _.dbc
which was not physically present at the path
/sid/applmgr/11510/fnd/11.5.0/secure/_
$ cd //applmgr/11510/fnd/11.5.0/secure/_
$ ls .dbc

SOLUTION 6:

copied the file PROD.dbc
which was physically present at path
//applmgr/11510/fnd/11.5.0/secure/_
to the dbc file name which was being pointed by all the properties files
This resolved the issue.
**********************************************************************
ISSUE 7
Instance is very slow / inaccessible with a warning on login page:
Warning

Low-level logging is currently enabled. Your application will not perform as well while Low-level logging is on.

Cause
Logging was enabled through profile values:
FND: Debug Log Enabled — Yes
FND: Debug Log Level — Statement
FND: Debug Log Module — %
SLA: Enable Diagnostics — Yes
FND Validation Level — Error

SOLUTION 7
1. Change these values to:
FND: Debug Log Enabled — No
FND: Debug Log Level — ( Blank )
FND: Debug Log Module — ( Blank )
SLA: Enable Diagnostics — No
FND Validation Level — None

2. Bounce Apache on all the MTs and clear Apache cache.
*********************************************************************
ISSUE 8

Login to the applications URL fails with
500 Internal Server Error java.lang.NoClassDefFoundError at oracle.apps.fnd.sso.AppsLoginRedirect.AppsSetting(AppsLoginRedirect?.java:126) at oracle.apps.fnd.sso.AppsLoginRedirect.init(AppsLoginRedirect?.java:170) at com.evermind[Oracle Containers for J2EE? 10g (10.1.3.0.0) ].server.http.HttpApplication.loadServlet(HttpApplication?.java:2231) at com.evermind

Performed the below steps, but it did not resolved the issue
1) Bounced Apache server, cleared cache and validated ports
2) Bounced all Middle tier services and killed defunct processes
3) Bounced Unix Server

Cause
No Cause Entered

SOLUTION 8

Java object Caching Configuration parameter was Enabled in oc4j.properties $INST_TOP/ora/10.1.3/j2ee/oacore/config
Disabled LONG_RUNNING_JVM=false from True

Bounced Apache server and could login to the applications URL without issues

Java Caching is a mechanism used to improve performance by storing data in the Middle Tier JVM memory, instead of retrieving the data from the database every time it is requested.
***********************************************************************
ISSUE 9
When trying to access the URL (Either Big IP or the node level), it hangs for a long time and getting redirected to a different URL and gives a blank page. For example, When try to access the ipayments URL which is hosted on node2, it hangs for a long time and its getting redirected towards the Sales URL which is hosted on node3, and displays a blank page.

Cause
All the host reference and the URL directives are pointing towards node3 and sales URL respectively, in the below files. Hence every attempt tries to launch the sales URL and get fails as the apache configuration files and parameters are pointing towards the correct one.
jserv.conf forms.properties zone.properties jserv.properties

SOLUTION 9

Stop Apache. Edit the files jserv.conf, forms.properties, zone.properties, jserv.properties, jserv.conf to replace the host reference and the URL directives to the correct one (In our case, its node3 . Clear the cache and restart the Apache.

ISSUE 10
Apache login issues occurs in multi MT environment. Unable to login to oracle applications.

Cause
Issue due to distributed cache.

SOLUTION 10
Set is_distributed to false in javacache.xml Bounce Apache.

ISSUE 11
Apache server is down. Apache logfile did not provide useful information.
Started the Apache server, but the Apache Web listener did not started.
1) Verified no log files (Apache,Jserv) crossed 2GB limit. error_log, access.log, mod_jserv
2) Veified that the port is listening (netstat -an |grep ) Apache port(httpd.conf), pls listener(httpd_pls.conf)
3) Verified that the Apache and Jserv configuration files are not changed recently httpd.conf & Jserv.conf
4) Shutdown all the Middle tier services and check for ipcs (semaphores) did not resolve the outage.
$ ipcs on application middle tier as Application user ap

Cause
No Cause Entered

SOLUTION 11

Rebooted the server and started the services. All services came back normal.
************************************************************************
ISSUE 12
Unable to login to the home page URL. Login fails with the error:
Error Page You have encountered an unexpected error. Please contact the system administrator for assistance.

Click here for exception details

Checks:

Upon clicking the link, can observe the error:
oracle.apps.fnd.framework.OAException: Application: FND, Message Name: SQL_PL SQL_ERROR. Tokens: ROUTINE = createSession(int)(userId=6,sessionMode=’null’,’2D8 FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412?’); REASON = java.s ql.SQLException: ORA-01422: exact fetch returns more than requested number of ro ws ORA-06512: at “APPS.FND_SESSION_MANAGEMENT”, line 295 ORA-06512: at “APPS.FND_SESSION_MANAGEMENT”, line 455 ORA-06512: at line 1 ; ERRNO = 1422;

Cause
FND_NODES table was updated with the same server_id for all Nodes.

SOLUTION 12

Update the table FND_NODES with correct server_ids for all nodes as in $FND_TOP/secure/sid_host/sid.dbf
Bounced apache on all apache nodes.

and Run select node_name, server_id from fnd_nodes;
to confirm the values.

Below is the example for the solution:

Found three hosts using same server_id:

SQL> select node_name, server_id from fnd_nodes; NODE_NAME SERVER_ID
AUOHSACPS06 2D8FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412
AUOHSACPS03 2D8FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412
AUOHSACPS02 2D8FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412

reset server_id with original server_id from each hosts .dbc file @ $FND_TOP/secure:

SQL> select node_name, server_id from fnd_nodes; NODE_NAME SERVER_ID
AUOHSACPS06 2D8FF21843830F82E040558CDBC6195C27690802237516122831445560236127
AUOHSACPS03 2D90005D5A13C774E040558CDBC61D9436546148971863330582284123371818
AUOHSACPS02 2D8FCCC57459B66EE040558CDBC6118E44964630721347149122084914143412
************************************************************************
ISSUE 13

Users getting ‘404 page not found errors’ when tyring to access the URL

(OR)

Getting Exception in thread “main” java.lang.ClassFormatError: Truncated class file

(OR)

adstrtal.sh or adstpall.sh is throwing below error message

You are running adstrtal.sh version 115.16

Exception in thread “main” java.lang.ClassFormatError: oracle/apps/ad/context/AppsContext (Truncated class file) at java.lang.ClassLoader.defineClass0(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader?.java:539) at java.security.SecureClassLoader.defineClass(SecureClassLoader?.java:123) at java.net.URLClassLoader.defineClass(URLClassLoader?.java:251) at java.net.URLClassLoader.access$100(URLClassLoader?.java:55) at java.net.URLClassLoader$1.run(URLClassLoader?.java:194) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader?.java:187) at java.lang.ClassLoader.loadClass(ClassLoader?.java:289) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:274) at java.lang.ClassLoader.loadClass(ClassLoader?.java:235) at java.lang.ClassLoader.loadClassInternal(ClassLoader?.java:302) at oracle.apps.ad.autoconfig.ServiceControl.(ServiceControl?.java:319) at oracle.apps.ad.autoconfig.ServiceControl.main(ServiceControl?.java:662) Check logfile //applmgr/common/admin/log/_/08061104.log for details

Exiting with status 1

(OR)

[06/08/2007 11:03:12:181] (ERROR) an error returned handling request via protocol “ajpv12” [06/08/2007 11:03:12:181] (EMERGENCY) ajp12: can not connect to host 140.85.188.36:20021 [06/08/2007 11:03:13:184] (EMERGENCY) ajp12: can not connect to host 140.85.188.36:20021 [06/08/2007 11:03:15:038] (EMERGENCY) ajp12: auth did not receive challenge size [06/08/2007 11:03:15:038] (EMERGENCY) ajp12: auth fail

Cause

Extract from the JVM log file: //product/iAS/Apache/Jserv/logs/jvm/OACoreGroup.0.stderr
Exception in thread “main” java.lang.ClassFormatError: Truncated class file at org.apache.jserv.JServServletManager.load_init(JServServletManager?.java:765) at org.apache.jserv.JServServletManager.loadServlet(JServServletManager?.java:659) at org.apache.jserv.JServServletManager.loadStartupServlets(JServServletManager?.java:789) at org.apache.jserv.JServServletManager.init(JServServletManager?.java:447) at org.apache.jserv.JServ.start(JServ.java:625)

Verified that there are no log files that exceed 2GB limit (OS limit).
Clearing the Apache and Jserv cache did not resolve the issue. Unmount / mount product specific file system did not resolve

SOLUTION 13:

Try each of these solutions until the problem is resolved, in order to have the least impact on the system.
1. If you can determine the specific class file from the log files, use the ‘touch’ command.

e.g. touch classfile

If necessary, restart Apache and clear server cache.

2. Shut down all MT services, exit the Apps owner and ask the System Admin to unmount and remount the /sid/applmgr file system (mount point above $JAVA_TOP). Ask the System Admin to kill any processes that may be holding the file system open. Then log in to the Apps owner and restart MT services.

3. Shutdown all the MT services and ask System Admin to reboot the server if the actions above don’t resolve the outage.
************************************************************************
ISSUE 14:

Got login error – Internal Server Error after bounced all services on . find following message from Jserv log file:

$ tail -8 mod_jserv.log
[07/05/2007 18:41:06:915] (ERROR) ajp12: Servlet Error: java.lang.NoClassDefFoundError: null
[07/05/2007 18:41:06:916] (ERROR) an error returned handling request via protocol “ajpv12”
[07/05/2007 18:41:06:916] (ERROR) balance: 11906 internal servlet error in server auohsnops03.oracleoutsourcing.com:20720
[07/05/2007 18:41:06:916] (ERROR) an error returned handling request via protocol “balance”
[07/05/2007 20:07:52:732] (ERROR) ajp12: Servlet Error: java.lang.NoClassDefFoundError: null
No runaway, no lock on port 20720.

SOLUTION 14:
Shutdown all services on MT. Removed all log file (standard error…) from Jserv/jvm. Started All services with successful login.
************************************************************************
ISSUE 15
Customer reported Not able to login with “Internal Server Error” on .
We could not start some of services during bounced All Services with following error message: adapcctl.sh version 115.47
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.

Apache Web Server Listener is not running (dedicated HTTP) …
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.

Apache Web Server Listener is already stopped.
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
adapcctl.sh: exiting with status 2

Unable to login the application URL, facing the error while login:

Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, ap@domain.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

Cause:

SQL> select END_DATE from fnd_user where user_name=’GUEST’;

END_DATE
———
05-OCT-10

SOLUTION 15

We found out the apps and applsys were not in sync.
Just manual change the apps password from sqlplus.
Restarted all services and login successfully.

or

Update the FND_USER table with END_DATE=null and bounced apache
SQL> update fnd_user set END_DATE=null where user_name=’GUEST’;

1 row updated.

SQL> commit;

Commit complete.

SQL> select END_DATE from fnd_user where user_name=’GUEST’;

END_DATE
———–
************************************************************************

 

 

 
ISSUE 16

Unable to login to oracle applications:
error message: You have insufficient privileges for the current operation.

Checks

Unable to login to oracle applications:
error message: You have insufficient privileges for the current operation.

SOLUTION 16

Verify and update SESSION_COOKIE_DOMAIN in ICX_parameters if it is wrong.

Example :

update icx_parameters set SESSION_COOKIE_DOMAIN=’NEW_VALUE’ where SESSION_COOKIE_DOMAIN=’OLD_VALUE’;
************************************************************************
ISSUE 17

Apache home page is not coming up even after bounced
Found in jvm log asking for java version

SOLUTION 17

//product/iAS/Apache/Apache/bin/java.sh: /usr/java/jdk1.3.1_15/bin/java: not found

Check the version in /usr/java and update that version in java.sh file and Bounced Apache.
************************************************************************
ISSUE 18

Unable to access applications

cause
All applications are up and running.
“Solaris Error: 12: Not enough space” errors are encounterd in database listener logfile.

SOLUTION 18:

Check if the OS is SUN Solaris.
Check if /tmp filesystem is 100% full.
Check Swap space utilization.
Check the application process that is hagging resources. (Check if any sql sessions are hagging resources)
Clear the process in issue after getting required approval.

NOTE:
The above is applicable only for environments on Solaris OS.
************************************************************************
ISSUE 19

Not able to login to APPS. JSP Error:
Unable to create an user session or ORA-01403 : no data found has been detected in FND.SIGNON.AUDIT_USER

Checks

Check number of rows in dual table.
Select * from dual;
DUAL table should always contain only one row.
The table named DUAL is a small table in the data dictionary that Oracle and user-written programs can reference to guarantee a known result.
This table has one column called DUMMY and one row containing the value X.

SOLUTION 19

DUAL table should always contain only one row and one column.
Insert a row with value X into dual table if the dual table contains 0 rows. If the dual table contains more than one row then delete the rows by keeping one row.

The rows in dual table not equal to zero.
************************************************************************
ISSUE 20

Signature for “https URL redirecting to http URL”
Issue:
https URL redirecting to http URL and the page cannot be displayed error occurs.

SOLUTION 20

Check the httpd.conf and url_fw.conf, in url_fw.conf the Rewrite rule was
RewriteRule ^/$/OA_HTML/AppsLocalLogin.jsp.
Changed into
RewriteRule https://: /OA_HTML/AppsLocalLogin.jsp.

 

Advertisements

Posted October 30, 2013 by balaoracledba.com in Issues, OracleAppsR12

3 responses to “11i/R12 Apache Common Issues with Solutions

Subscribe to comments with RSS.

  1. great work sir
    thanking you

  2. Thanks for finally talking about >11i/R12 Apache Common Issues with Solutions | OracleAPPSDBA & Oracle DBA Blog
    <Loved it!

  3. Good to gather all the things in single page…Thanks a lot 🙂

    —Srinivas Adapaka

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: