The reason for this blogpost is, that we had two Enterprise manager outages last few months due to logfile misconfiguration in Enterprise manager. The cases are not specific to Enterprise Manager and can be applied to any other Weblogic based Fusion Middleware setup.
So what were these outages:
- Mod_wl_ohs.log was not rotated and grew to a very large size affecting performance and availablility of EM. We fixed that following note 1271676.1. (Webtier log – mod_wl_ohs.log in the OMS Home is very Large in Size and not Rotated
- The weblogic server crashed with a SIGSEGV without a clue to what was wrong. Filesystem filespace was exhausted (BTW: Filesystems are monitored with Zenoss and not EM, but this machine was not monitored yet)
So what was filling up the filesystem? Searching the filesystem for large directories showed me two locations containing almost 7 GB of logfiles.
/u01/app/oracle/product/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/ em_upload_https_access_log.* mod_wl_ohs.log.* access_log.*
/u01/app/oracle/product/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs EMGC_OMS1.out* access.log*
These logfiles are rotated, which is good, but the resulting archives are not cleaned up. There are retention parameters to automatically cleanup old logfiles, but they haven’t been configured.
There are two components that need to be reconfigured
- Webtier aka Oracle HTTP Server (OHS)
- Weblogic server
Reconfigure Webtier aka Oracle HTTP Server (OHS)
Changing the logging configuration is done on the commandline. There is a special Oracle version of rotatelogs called odl_rotatelogs, which has extra retention parameters.
$ORACLE_HOME/../Oracle_WT/ohs/bin/odl_rotatelogs
Usage: /u01/app/oracle/product/Middleware/oms11g/../Oracle_WT/ohs/bin/odl_rotatelogs [-l] [-f] [-u:<utc offset in seconds>] <logfile> {<rotation time in seconds> [retention time in seconds] [base rotation start time] | <rotation size in megabytes> [max logsize in megabytes]}
Change the configuration to: rotate every 12 hours (43200 seconds) and keep archives for 7 days (604800 seconds).
Navigate to : $ORACLE_HOME/../../gc_inst/WebTierIH1/config/OHS/ohs1
httpd.conf: Rotate & retention for the acces_log
CustomLog "|${ORACLE_HOME}/../Oracle_WT/ohs/bin/odl_rotatelogs ${ORACLE_INSTANCE}/diagnostics/logs/${COMPONENT_TYPE}/${COMPONENT_NAME}/access_log 43200 604800" common
mod_wl_ohs.conf: Rotate & retention for the mod_wl_ohs.log
WLLogFile "|${ORACLE_HOME}/../Oracle_WT/ohs/bin/odl_rotatelogs ${ORACLE_INSTANCE}/diagnostics/logs/${COMPONENT_TYPE}/${COMPONENT_NAME}/mod_wl_ohs.log 43200 604800"
http_em.conf: Rotate & retention for multiple logfiles
CustomLog "|${ORACLE_HOME}/../Oracle_WT/ohs/bin/odl_rotatelogs /u01/app/oracle/product/Middleware/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_access_log 43200 604800" common env=!no-log
ErrorLog "|${ORACLE_HOME}/../Oracle_WT/ohs/bin/odl_rotatelogs /u01/app/oracle/product/Middleware/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_https_error_log 43200 604800"
CustomLog "|${ORACLE_HOME}/../Oracle_WT/ohs/bin/odl_rotatelogs /u01/app/oracle/product/Middleware/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/em_upload_http_access_log 43200 604800" common env=!no-log
CustomLog "|${ORACLE_HOME}/../Oracle_WT/ohs/bin/odl_rotatelogs /u01/app/oracle/product/Middleware/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1/access_log 43200 604800" common env=!no-log
Now restart the Webtier.
export ORACLE_INSTANCE=$ORACLE_HOME/../../gc_inst/WebTierIH1/ $ORACLE_HOME/opmn/bin/opmnctl status $ORACLE_HOME/opmn/bin/opmnctl restartproc process-type=OHS $ORACLE_HOME/opmn/bin/opmnctl status
Sidenote: In Enterprise Manager 12c which runs on fusion middleware 11.1.1.5 (10.3.5) the logging configuration in httpd.conf has changed to:
#Directives to setup logging via ODL
OraLogDir "${ORACLE_INSTANCE}/diagnostics/logs/${COMPONENT_TYPE}/${COMPONENT_NAME}"
OraLogMode odl-text
OraLogSeverity WARNING:32
OraLogRotationParams S 10:70
Reconfigure Weblogic server
Configuration of Weblogic servers can be done through the admin console.
Navigate to Servers > EMGC_OMS1 > Logging > General
Weblogic General Logging
The main server log is indeed already being rotated and has a retention.
In the advanced part of this configure the severity levels which need to be logged. Up the levels of severity to reduce logging activity.
General logging - advanced
Now lets configure the Weblogic access_log. Navigate to the HTTP tab and enable ‘limit number of retained files’. You can even disable logging to the HTTP access log of Weblogic.
Configure Weblogic access log
There is one other log that needs to be handled and that is the standard out (stdout). This logfile is actually produced by the JVM itself, not Weblogic server (which runs on the JVM). But we can redirect stdout to the default loggers of Weblogic.
Navigate to Logging > general > advanced.
Logging: redirecting stdout
Enable Redirect stdout logging enabled. As an extra change the severity loglevel to warning. This affects ALL logfiles!
Restart your Weblogic server. When using Enterpise manager, restart through the commandline.
emctl stop oms emctl start oms
Now your Fusion Middleware (Enterprise Manager) environment should be managing its own logfiles. All these changes have been checked for Enterprise manager 11g, Enterprise manager 12c and should work for Fusion Middleware in general too.
That’s all!
References
Good summary of OHS 11.1 & Weblogic server plugin
odl_rotatelogs does not rotate in the defined location (fixed as of 11.1.1.4)
11G Grid Control Performance: Webtier log – mod_wl_ohs.log in the OMS Home is very Large in Sizeand not Rotated [ID 1271676.1]
Fusion Middleware 11.1.1.2 (10.3.2): Managing logfiles and diagnostic data
Fusion Middleware 11.1.1.5 (10.3.5): Administration Console : View and configure logs
Whitehorses is specialized in succesfully implementing Oracle SOA solutions: BPEL, OSB, WebLogic & BPM
