Follow Us on Twitter

Enterprise Manager: Manage your Fusion Middleware logfiles

by Tony van Esch on June 1, 2012 · 0 comments

The reason for this blogpost is, that we had two Enterprise manager outages last few months due to logfile misconfiguration in Enterprise manager. The cases are not specific to Enterprise Manager and can be applied to any other Weblogic based Fusion Middleware setup.

So what were these outages:

  1. Mod_wl_ohs.log was not rotated and grew to a very large size affecting performance and availablility of EM. We fixed that following note 1271676.1. (Webtier log – mod_wl_ohs.log in the OMS Home is very Large in Size and not Rotated
  2. The weblogic server crashed with a SIGSEGV without a clue to what was wrong. Filesystem filespace was exhausted (BTW: Filesystems are monitored with Zenoss and not EM, but this machine was not monitored yet)

So what was filling up the filesystem? Searching the filesystem for large directories showed me two locations containing almost 7 GB of logfiles.

/u01/app/oracle/product/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/
em_upload_https_access_log.*
mod_wl_ohs.log.*
access_log.*
/u01/app/oracle/product/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs
EMGC_OMS1.out*
access.log*

These logfiles are rotated, which is good, but the resulting archives are not cleaned up. There are retention parameters to automatically cleanup old logfiles, but they haven’t been configured.

There are two components that need to be reconfigured

  • Webtier aka Oracle HTTP Server (OHS)
  • Weblogic server

Reconfigure Webtier aka Oracle HTTP Server (OHS)

Changing the logging configuration is done on the commandline. There is a special Oracle version of rotatelogs called odl_rotatelogs, which has extra retention parameters.

$ORACLE_HOME/../Oracle_WT/ohs/bin/odl_rotatelogs
Usage: /u01/app/oracle/product/Middleware/oms11g/../Oracle_WT/ohs/bin/odl_rotatelogs [-l] [-f] [-u:<utc offset in seconds>] <logfile> {<rotation time in seconds> [retention time in seconds] [base rotation start time] | <rotation size in megabytes> [max logsize in megabytes]}

Change the configuration to: rotate every 12 hours (43200 seconds) and keep archives for 7 days (604800 seconds).

Navigate to : $ORACLE_HOME/../../gc_inst/WebTierIH1/config/OHS/ohs1

httpd.conf: Rotate & retention for the acces_log

CustomLog "|${ORACLE_HOME}/../Oracle_WT/ohs/bin/odl_rotatelogs ${ORACLE_INSTANCE}/diagnostics/logs/${COMPONENT_TYPE}/${COMPONENT_NAME}/access_log 43200 604800" common

mod_wl_ohs.conf: Rotate & retention for the mod_wl_ohs.log

WLLogFile "|${ORACLE_HOME}/../Oracle_WT/ohs/bin/odl_rotatelogs ${ORACLE_INSTANCE}/diagnostics/logs/${COMPONENT_TYPE}/${COMPONENT_NAME}/mod_wl_ohs.log 43200 604800"

http_em.conf: Rotate & retention for multiple logfiles

CustomLog "|${ORACLE_HOME}/../Oracle_WT/ohs/bin/odl_rotatelogs /u01/app/oracle/product/Middleware/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_access_log 43200 604800" common env=!no-log
ErrorLog "|${ORACLE_HOME}/../Oracle_WT/ohs/bin/odl_rotatelogs /u01/app/oracle/product/Middleware/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_error_log 43200 604800"
CustomLog "|${ORACLE_HOME}/../Oracle_WT/ohs/bin/odl_rotatelogs /u01/app/oracle/product/Middleware/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_http_access_log 43200 604800" common env=!no-log
CustomLog "|${ORACLE_HOME}/../Oracle_WT/ohs/bin/odl_rotatelogs /u01/app/oracle/product/Middleware/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/access_log 43200 604800" common env=!no-log

Now restart the Webtier.

export ORACLE_INSTANCE=$ORACLE_HOME/../../gc_inst/WebTierIH1/
$ORACLE_HOME/opmn/bin/opmnctl status
$ORACLE_HOME/opmn/bin/opmnctl restartproc process-type=OHS
$ORACLE_HOME/opmn/bin/opmnctl status

Sidenote: In Enterprise Manager 12c which runs on fusion middleware 11.1.1.5 (10.3.5) the logging configuration in httpd.conf has changed to:

#Directives to setup logging via ODL
OraLogDir "${ORACLE_INSTANCE}/diagnostics/logs/${COMPONENT_TYPE}/${COMPONENT_NAME}"
OraLogMode odl-text
OraLogSeverity WARNING:32
OraLogRotationParams S 10:70

Reconfigure Weblogic server

Configuration of Weblogic servers can be done through the admin console.

Navigate to Servers > EMGC_OMS1 > Logging > General

Weblogic General Logging

Weblogic General Logging

The main server log is indeed already being rotated and has a retention.

In the advanced part of this configure the severity levels which need to be logged. Up the levels of severity to reduce logging activity.

General logging - advanced

General logging - advanced

Now lets configure the Weblogic access_log. Navigate to the HTTP tab and enable ‘limit number of retained files’. You can even disable logging to the HTTP access log of Weblogic.

Configure Weblogic access log

Configure Weblogic access log

There is one other log that needs to be handled and that is the standard out (stdout). This logfile is actually produced by the JVM itself, not Weblogic server (which runs on the JVM). But we can redirect stdout to the default loggers of Weblogic.

Navigate to Logging > general > advanced.

Logging: redirecting stdout

Logging: redirecting stdout

Enable Redirect stdout logging enabled. As an extra change the severity loglevel to warning. This affects ALL logfiles!

Restart your Weblogic server. When using Enterpise manager, restart through the commandline.

emctl stop oms
emctl start oms

Now your Fusion Middleware (Enterprise Manager) environment should be managing its own logfiles. All these changes have been checked for Enterprise manager 11g, Enterprise manager 12c and should work for Fusion Middleware in general too.

That’s all!

References

Good summary of OHS 11.1 & Weblogic server plugin
odl_rotatelogs does not rotate in the defined location (fixed as of 11.1.1.4)

11G Grid Control Performance: Webtier log – mod_wl_ohs.log in the OMS Home is very Large in Sizeand not Rotated [ID 1271676.1]

Fusion Middleware 11.1.1.2 (10.3.2): Managing logfiles and diagnostic data

Fusion Middleware 11.1.1.5 (10.3.5): Administration Console : View and configure logs

Enterprise Manager: Manage your Fusion Middleware logfiles , 5.0 out of 5 based on 4 ratings
Ratings:
VN:F [1.9.22_1171]
Rating: 5.0/5 (4 votes cast)

{ 0 comments… add one now }

Leave a Comment

 

Previous post:

Next post:

About Whitehorses
Company profile
Services
Technology

Whitehorses website

Home page
Whitebooks
Jobs

Follow us
Blog post RSS
Comment RSS
Twitter