In Cloud Control 12c The Job Refresh From My Oracle Support fails with Error ORA-01882: Timezone Region Not Found When Setting My Oracle Support Preferred Credentials

Modified 20-FEB-2013

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later
Enterprise Manager for My Oracle Support - Version: 10.2.0.5 to 12.1.0.1.0 [Release: 10.2 to 12.1]
Generic Linux
Generic Unix

Symptoms

When setting My Oracle Support Preferred Credentials in the EM 12c Cloud Control console, by applying the following steps:

• Setup → Security → Preferred Credentials
• Then click on the "My Oracle Support Preferrred Credentials"
• Under "My Oracle Support Preferred Credentials" click on "Set MOS Credentials" throws following error:
  
  Refresh From My Oracle Support - Error: ORA-01882: timezone region not found ORA-06512: at "SYSMAN.EM_JOB_SCHEDULER", line 822 ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE",
line 3409 ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 3602 ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 4615 ORA-01403: no data found ORA-06512: at "SYSMAN.EM_JOB_OPS", line 235 ORA-06512: at "SYSMAN.GC_JOB", line 29 ORA-06512: at line 1

  This error may also be seen in the status of the 'Refresh From My Oracle Support' job.
  
  This error may also be seen in the status of the 'Opatch Update' job.

Cause

The OS Time zone is set to a non-standard setting.

The problem is that the 'MYT' timezone shown with the 'date' command does not match any of those in the repository database.

$ date
Sat Dec 17 03:29:22 MYT 2011

The following query run against the Repository database returns no rows for the 'MYT' time zone:

SQL> SELECT TZname, TzAbbrev FROM V$TIMEZONE_NAMES where TZname like '%MYT%' ORDER BY TzAbbrev;
no rows selected

The OS Timezone needs to match one of those returned by the following query against the repository database:

SQL> SELECT TO_CHAR(SYSTIMESTAMP,'TZR') FROM DUAL UNION SELECT DISTINCT tzname FROM v$timezone_names;

Solution

1. Please run this query against the Repository Database:
   
   SQL> SELECT TO_CHAR(SYSTIMESTAMP,'TZR') FROM DUAL UNION SELECT DISTINCT tzname FROM v$timezone_names;



2. Coordinate with your sysadmins to adjust the operating system timezone to match one of the above accordingly, the one that your database is set to.
   The operating system timezone and database timezone need to be the same. Indeed the database and operating system Timezone entries were not in sync.
   
   To achieve it, the files touched/referenced are:
   
   /usr/share/zoneinfo
   /etc/sysconfig/clock
   /etc/localtime
   
   Create a symbolic link for the Date/time:
   
   $ ln -sf /usr/share/zoneinfo/America/Denver /etc/localtime

Grid Agent Configuration: storage_report_metrics.pl Reports Error "snmhsutl.c:executable nmhs should have root suid enabled"

Modified 23-FEB-2012

Applies to

Enterprise Manager Base Platform - Version: 10.2.0.1 to 11.1.0.1 [Release: 10.2 to 11.1]
This problem can occur on any platform.

Symptoms

The <AGENT_HOME>/sysman/log/emagent_perl.trc in the Agent home has errors such as:

storage_report_metrics.pl: <timestamp>: WARN: STORAGE_REPORTS:error::snmhsutl.c:executable nmhs should have root suid enabled

The Host target in the Grid Console could have a metric collection error as below:

Target agentmachine.domain
Type Host
Metric Storage Data
Collection Timestamp <timestamp>
Error Type Collection Problem
Message snmhsutl.c:executable nmhs should have root suid enabled

Cause

The root.sh was not executed in the <AGENT_HOME> after installation.
Check the permissions of the nmhs binary:

cd <AGENT_HOME>/bin
ls -l nmhs*

• For a 10.2 Agent, the output should be:
  
  -rwsr-x--- 1 root oracle 46003 Jun 10 10:16 nmhs
  


• For a 11g Agent, the output should be:
  
  -rws--x--- 1 root oinstall 47287 Apr 28 23:13 nmhs

Solution

1. Stop the agent:
   
   cd <AGENT_HOME>/bin
emctl stop agent

   
2. Run root.sh from <AGENT_HOME> as the root user
   
   Or
   
   Change the permissions of the nmhs executable manually:
   • Login as the root user:
   • Execute:
   
   For a 10.2 Agent:
   
   # cd <AGENT_HOME>/bin
# chown root $ORACLE_HOME/bin/nmhs
# chmod 4750 $ORACLE_HOME/bin/nmhs

   For a 11G Agent:
   
   # cd <AGENT_HOME>/bin
# chown root $ORACLE_HOME/bin/nmhs
# chmod 4710 $ORACLE_HOME/bin/nmhs

   

   NOTE: It is recommended to fix this issue by running the <AGENT_HOME>/root.sh script, rather than manually modifying the permissions.
   The root.sh script makes lot more changes which are needed for ensuring that the Agent iscorrectly configured on the Unix machine.
   



3. Start the agent:
   
   cd <AGENT_HOME>/bin
emctl start agent

   
4. The metrics collection error "snmhsutl.c:executable nmhs should have root suid enabled" can be reset (after running root.sh) by executing the following commands on the agent host:
   
   $ emctl control agent runcollection <host_target_name>:host host_storage
   $ emctl upload
   
   Example:

$ emctl control agent runcollection agentmachine.domain:host host_storage
$ emctl upload

12c Cloud Control: Agent Target is not Seen in the Console After a Successful Installation

Modified 14-MAY-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
This problem can occur on any platform.

Symptoms

12.1.0.1 Agent has been newly installed on a machine using one of the installation methods like Push-Agent or silent Agent installation. Agent installation is successfully completed and is able
to start but the Agent target is not visible in the 12c Cloud Console.

• The Agent is started and running:
  
  emctl status agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.1.0
OMS Version : 12.1.0.1.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oracle/12.1.0
Agent Binaries : /u01/app/oracle/12.1.0/core/12.1.0.1.0
Agent Process ID : 7643
Parent Process ID : 7603
Agent URL : https://<agentmachine.domain>:3872/emd/main/
Repository URL : https://<omsmachine.domain>:4900/empbs/upload
Started at : 2011-11-26 11:47:26
Started by user : oracle
Last Reload : (none)
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload : 1
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 21.66%
Collection Status : Collections enabled
Last attempted heartbeat to OMS : 2011-11-26 11:47:33
Last successful heartbeat to OMS : 2011-11-26 11:47:33
---------------------------------------------------------------
Agent is Running and Ready

  
• The communication between the Agent and OMS machine is fine as seen in:
  
  emctl pingOMS
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD pingOMS completed successfully

  The communication between Agent and OMS is working both the ways.
  Also the OMS upload port and the Agent port is open and accessible i.e no firewall blocking the communication.
  


• Attempting a manual upload from the Agent fails with:
  
  cd <AGENT_HOME>/bin
emctl upload agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: Upload timed out before completion.
Number of files to upload before the uploadNow call: 1, total size (MB): 5.1784515E-4
Remaining number of files to upload: 1, total size (MB): 5.1784515E-4 (TIMEOUT)



• The <AGENT_INST>/sysman/log/gcagent.log file shows:
  
  <timestamp> [54:A60A3E87] INFO - >>> Dispatching request: UploadRequest (waitTime:-1)
<timestamp> [54:A60A3E87] INFO - >>> Reporting exception: oracle.sysman.emSDK.agent.client.exception.UploadException: Upload timed out before completion.
Number of files to upload before the uploadNow call: 1, total size (MB): 5.1784515E-4
Remaining number of files to upload: 1, total size (MB): 5.1784515E-4 (TIMEOUT) (request id 1)

  
• The <EM_CONFIG_HOME>/em/EMGC_OMS1/sysman/log/emoms_pbs.trc on the OMS machine shows:
  
  <timestamp> [[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)']
ERROR gcloader.Receiver logp.251 -
Couldn't get agent guid with given emdUrl:https://agentmachine.domain:3872/emd/main/
java.sql.SQLException: ORA-01403: no data found
ORA-06512: at "SYSMAN.EMD_LOADER", line 713
ORA-06512: at line 1



• In DEBUG level, the emoms_pbs.trc shows:
  
  <timestamp> [[ACTIVE] ExecuteThread: '22' for queue: @ 'weblogic.kernel.Default (self-tuning)']
DEBUG gcloader. ResourceManager logp.251 -
Received priority value as 2 of data type:metadata for emd:https://<agentmachine.domain:<port>/emd/main/
<timestamp> [[ACTIVE] ExecuteThread: '22' for queue: @ 'weblogic.kernel.Default (self-tuning)']
DEBUG gcloader.ResourceManager logp.251 -
Failed to get resource for agent:https://<agentmachine.domain:<port>/emd/main/priority2 dataTypemetadata


Changes

All existing Agents in the Cloud Console have one or more targets whose Lifecycle Status is set to Production or Mission Critical.

To verify, login to the Repository Database as the sysman user and execute:

SQL> SELECT COUNT(*), priority
FROM ( SELECT t.emd_url as emd_url, nvl(min(pe.priority),3) as priority
FROM em_global_Target_properties tp, mgmt_Targets t, gc_lifecycle_states pe
WHERE t.target_guid=tp.target_guid and t.emd_url is not null
and tp.lifecycle_status = pe.display_name(+) group by t.emd_url)
group by priority;

count(1) priority
-------- --------
1        1
1        2

where priority 1= Mission Critical and 2= Production.

To get the list of targets which have the above lifecycle status:

SQL> select target_name, target_type, emd_url from mgmt_targets where target_guid in
(select target_guid from em_global_target_properties where lifecycle_status in ('Production','MissionCritical'));

TARGET_NAME TARGET_TYPE EMD_URL
-------------------- ----------- ---------------------------------------
agentmachine1.domain host https://agentmachine1.domain:1830/emd/main/
agentmachine2.domain host https://agentmachine2.domain:3873/emd/main/

In this particular setup, there are only 2 Agents discovered in the Cloud Console and each Agent has one target each
where the Lifecycle Status is set to Production or Mission Critical.

Cause

As all the existing Agents have one or more targets whose lifecycle status is Production or Mission Critical,
the Loader is unable to allocate resources to add/save the target details sent by the newly installed Agent.
Due to this, automatic promotion of the Agent target does not occur and this target is listed as a
non-yet managed entities in the console, which can be verified using:

• Login to 12c Cloud Console as sysman user.
• Navigate to Setup → Add Target → Auto Discovery Results
• Click on the "Non-Host Targets" tab, the Agent target is listed here.

Solution

Download and apply the mandatory 12.1.0.1.0 Bundled Patch 1 to the OMS.

Workaround

There are two workarounds as listed below:

1. Manually promote the Agent target from this machine using these steps:
   • Login to the 12c Cloud Console as sysman user.
   • Navigate to Setup → Add Target → Auto Discovery Results
   • Click on the "Non-Host Targets" tab and select the Agent target.
   • Click on the "Promote" button.
   Once the target promotion is successful, the Agent target will be displayed in the Console.
   
   
2. For the Loader to be able to allocate the resources for the new Agent,it is necessary that there is atleast one Agent
   in the Console having all targets with Lifecycle Status set to a value of Stage/Test/Development/None.
   • From the output of the below query, pick one of the Agents:
     
     SQL> select target_name, target_type, emd_url from mgmt_targets where target_guid in
(select target_guid from em_global_target_properties where lifecycle_status in ('Production','MissionCritical'));

TARGET_NAME TARGET_TYPE EMD_URL
-------------------- ----------- ---------------------------------------
agentmachine1.domain host https://agentmachine1.domain:1830/emd/main/
agentmachine2.domain host https://agentmachine2.domain:3873/emd/main/

     
   • In the Cloud Console, navigate to the homepage of the target listed against this Agent
     and go to Target Setup → Properties. Click on the edit button and change the Lifecycle
     Status property value to Stage or Test or Development or None.
     
   
   
   • Similarly modify the lifecycle status of all the targets that are listed against this Agent by the above query.
   • Wait for 15 minutes and re-try a manual upload from the Agent machine:
     
     cd <AGENT_HOME>/bin
emctl upload

   
   The Agent target should now be visible in the Console.

How to Perform Agent Resynchronization in 12C Cloud Control?

Modified 09-JAN-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later [Release: 12.1 and later].
Information in this document applies to any platform.

Goal

How to perform Agent Resynchronization in 12C Cloud Control?

Solution

Follow the below mentioned steps to perform agent resynchronization:

1. Login to 12C console
2. Navigate to Setup → Agents
3. Click on the problematic agent
4. Click on drop down menu "Agent" and Choose "Resynchronization" in the menu list.
5. Choose the option to "Unblock agent on successful completion of agent resynchronization." and Click on Continue.
   
   

   



6. The resynchronization operation is submitted as a job as shown below:
   
   

   

   
   
7. Check the job's status of the resynchronization operation by clicking on the job name's link.
   
   

   

   
   
8. Check the job's log report of the resynchronization operation by clicking on the log report link.
   
   
   

   

How To Suppress Down Alerts From Cluster Resource ora.gsd (Oracle Global Services Daemon) In Grid Control

Modified 15-OCT-2012

Applies to

Enterprise Manager Base Platform - Version: 11.1.0.1 and later [Release: 11.1 and later]
This problem can occur on any platform.

Goal

How To Suppress Alerts From Cluster Resource ora.gsd (Oracle Global Services Daemon) in Grid Control?

Background Information

The function of GSD (10g and above) is to service requests for 9i RAC management clients and therefore when there are no 9i databases present, there is nothing for GSD to do. Consequently, there will be no impact on a RAC cluster if GSD is offline and 9i is not used. If GSD service is not required, it can be temporarily disabled.

In 11.2 GSD is disabled by default and the ora.gsd service will be show as target:offline, status:offline as follows:

[grid@node1 ~]$ crs_stat -t|grep gsd
ora.node1.gsd application OFFLINE OFFLINE
ora.node2.gsd application OFFLINE OFFLINE
ora.gsd ora.gsd.type OFFLINE OFFLINE

Fix

Ensure Grid Control is patched to PSU3 or Higher (Latest PSU would be preferable)

Apply patch 10297881 to the agent.

Then go to the metrics and thresholds page and add the key_value for resource_ora.gsd.type_ora.gsd. Set threshold to "not defined".

If required add resource_ora.oc4j.type_ora.oc4j and set threshold to "not defined".

Apply the changes

Then EM will not alert only for these resources.

EM 12c: Installing the Enterprise Manager Cloud Control 12.1.0.1 Agent through the Add Target Host Feature Results in a Directory Creation Error

Modified 06-SEP-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1 and later
Information in this document applies to any platform.

Symptoms

Installing the Cloud Control 12c agent through the Add Target Host Feature fails at agent start with the error:

Transferring Agent Software to Destination Host. The error is "Creation of directory /oracle/agent12c on host xxx Failed. Error Message:Not Available Exit Code : 1. Recommendation Not Available.

FileName
----------------
ui.log

2012-03-22_14-11-25:INFO: HOST : abc.com Intialization : FAILED Remote Prereq status : NULL INSTALL status : NULL
2012-03-22_14-11-25:INFO:Inside setDetailsBean :groupid group1 hostname dbtst3.uark.edu
2012-03-22_14-11-25:INFO:Dep vo map for host dbtst3.uark.eduis {Initialization=
[AdaDeploymentVo:
status: FAILED
[AdaActionVo: actionName: SSHValidations recommendation: null errorStream: null status: SUCCESS outStream: null
AdaActionVo]
[AdaActionVo: actionName: CopyAgentImage recommendation: Not Available errorStream: Error Message:Not Available
Exit Code :1 status: FAILED outStream: null
AdaActionVo]
AdaDeploymentVo]}
2012-03-22_14-11-25:INFO:Inside Build Progress Bar : Session Ended true Refresh Frequency 30000 Progress Bar Message Initialization Failed
2012-03-22_14-11-25:INFO:local prereqs failed/images/title_error.png Failed
2012-03-22_14-11-25:INFO:Since all nodes failed. Disable Continue ignoring failed host
2012-03-22_14-11-25:INFO:==================END populateCurrentSessionData ==================:2012-03-22_14-04-25-PM

Deploy.log:

2012-07-13_15-18-44:INFO:/bin/sh -c 'if [[ -d /oracle/agent_inst ]] ; then exit 0; else exit 1; fi' execution failed on node155.213.223.133
2012-07-13_15-18-44:INFO: ACTION Execution of command /bin/sh -c 'if [[ -d /oracle/agent_inst ]] ; then exit 0; else exit 1; fi' on host 155.213.223.133
2012-07-13_15-18-44:INFO: OUT
2012-07-13_15-18-44:INFO: ERR
2012-07-13_15-18-44:INFO: EXIT CODE1

2012-07-13_15-18-49:INFO: EXIT CODE2
2012-07-13_15-18-49:INFO: ERR /bin/ls: /oracle/agent_inst: No such file or directory
2012-07-13_15-18-49:INFO: EXIT CODE2
2012-07-13_15-18-49:INFO:Printing Exception :CommandException: err: /bin/ls: /oracle/agent_inst: No such file or directory
out: exitcode: 2
stacktrace:
null

2012-07-13_15-19-09:INFO:LOGGING XML LOC:/oracle/EM/middleware/oms/oui/prov/resources
2012-07-13_15-19-09:INFO:RemoteInterfaces startup called with REMOTENODES_PLATFORM=226 and REMOTE_TEMP_LOCATION=/tmp
2012-07-13_15-19-10:INFO:checkSharedPath:Init RI with REMOTE_PATH_PROPERTIES_LOC= /oracle/EM/middleware/oms/oui/prov/resources
2012-07-13_15-19-10:INFO:RUN AS not set during startup
2012-07-13_15-19-10:INFO:CLUSTER EXCEPTION found in startup
2012-07-13_15-19-10:INFO:Printing Exception :oracle.sysman.prov.remoteinterfaces.exception.ClusterException: Check for remote command execution setup failed on nodes 155.213.223.133.
at oracle.sysman.prov.remoteinterfaces.nativesystem.NativeSystem.startup(NativeSystem.java:538)
at oracle.sysman.prov.remoteinterfaces.clusterops.ClusterBaseOps.startup(ClusterBaseOps.java:425)
at oracle.sysman.prov.remoteinterfaces.clusterops.ClusterBaseOps.startup(ClusterBaseOps.java:338)

2012-07-13_15-19-12:INFO:=========Command Exception :Error Message:Not Available

Exit Code :1
2012-07-13_15-19-12:INFO:Updating Action CopyAgentImagewith Status FAILED and error Message :Error Message:Not Available

Exit Code :1 and problem Creation of directory /oracle on host 155.213.223.133 Failed and recommendation Not Available

Cause

Checking the ssh connectivity between the OMS and the target server using this command

<OMS_HOME>/oui/prov/resources/scripts/sshUserSetup.sh -setup -user oracle -hosts <hostname>

Shows this result:

sshUserSetup logs clearly says that "Checking if the remote hosts are reachable"

From OMS host

+ ping <Agent Machine Name>
>>oracle@oem ~ $ ping abc.domain.com
PING abc.domain.com (10.10.10.10) 56(84) bytes of data.

--- abc.domain.com ping statistics ---
1361 packets transmitted, 0 received, 100% packet loss, time 1361060ms


Solution

Agent deployment from the Cloud Control 12c console requires effective interaction through ssh as described in this reference:

Oracle(R) Enterprise Manager Cloud Control Basic Installation Guide
12c Release 1 (12.1.0.1)
Part Number E22624-09
Chapter 8 Installing Oracle Management Agent
Section titled: Before You Begin
and
Table 8-1 Prerequisites for Installing Oracle Management Agent
Download here

Ensure ssh is configured for effective OMS host interaction with the agent hosts as described there.

EM 12c Emctl Start OMS Returns Error Occurred: Cannot Run Program "/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not Enough Space

Modified 25-APR-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

When trying to start or stop OMS , It fails with the following errors:

./emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Starting WebTier...
Error Occurred: Cannot run program "/u01/app/oracle/product/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space
Please check /u01/app/oracle/product/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details

The following errors are found in <Middleware_Home>/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log:

2011-07-23 08:58:32,447 [main] ERROR wls.OMSController main.259 - OMSController failed for start oms
2011-07-23 08:58:32,449 [main] ERROR wls.OMSController main.260 - OMSController Error: Cannot run program "/u01/app/oracle/product/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space
java.io.IOException: Cannot run program "/u01/app/oracle/product/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:431)
at java.lang.Runtime.exec(Runtime.java:369)
at oracle.sysman.emctl.util.WebTierUtil.execCmd(WebTierUtil.java:136)
at oracle.sysman.emctl.util.WebTierUtil.stopWebTier(WebTierUtil.java:117)
at oracle.sysman.emctl.util.WebTierUtil.startWebTier(WebTierUtil.java:122)
at oracle.sysman.emctl.wls.OMSController.startOMS(OMSController.java:521)
at oracle.sysman.emctl.wls.OMSController.main(OMSController.java:214)
Caused by: java.io.IOException: error=12, Not enough space

Cause

The Swap space was configured less.

%swap -s total: 6708280k bytes allocated + 0k reserved = 6708280k used, 1680328k available

Less than 2Gb was not enough for stopping/starting the oms stack.

Solution

Make sure following things are taken care:

Allocate more Swap space. At least minimum of 4 to 5Gb should be available.

So that:

%swap -s
total: 6565136k bytes allocated + 0k reserved = 6565136k used, 6017776k available

And:

Make sure the location where OMS Instance Home resides (<Middleware_Home>/gc_inst), has enough free space.

12c Cloud Control: Which WLS Log Files Can be Removed / Purged Manually at Regular Intervals in 12c OMS Installation for Space Considerations?

Modified 19-APR-2012

Applies to

Enterprise Manager for Oracle Database - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

In a 12c OMS installation, automatic log rotation has been enabled for all the OMS and WLS logs after they reach a certain size.
Hence, creation of very large log/trace files should not be a concern. But, there are certain logs which are not purged automatically
after rotation and these files have to be manually deleted at regular intervals.

This document lists the log files which need to be purged manually on a periodic basis in a 12c OMS installation.

Fix

On the OMS machine, the following log files need to be manually purged:

• Webtier Logs in <EM_INSTANCE_HOME>/WebTierIH1/diagnostics/logs/OHS/ohs1:


• em_upload_http_access_log.*
• access_log.*
• em_upload_https_access_log.*
• ohs1-*.log
• console~OHS~1.log*
• mod_wl_ohs.log*


• Domain Logs:


• <EM_INSTANCE_HOME>/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/GCDomain.log*


• EMGC_ ADMINSERVER Logs in <EM_INSTANCE_HOME>/users_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs:


• EMGC_ADMINSERVER.out*
• EMGC_ADMINSERVER.log*
• access.log*


• Managed Server Logs:


• <EM_INSTANCE_HOME>/users_projects/domains/GCDomain/servers/EMGC_OMSn/logs/EMGC_OMS*.out*

NOTE:

1. In an additional OMS installation, there are no Domain logs or EMGC_ADMINSERVER logs as the Adminserver runs on the primary OMS machine.
So only the Webtier and Managed server out files need to be purged.

2. Automatic purging of older Domain logs and EMGC_ADMINSERVER logs can be enabled using the steps in our next technical tip:

"How to Enable Log Rotation Policy to Automatically Delete Older GCDomain.log, EMGC_ADMINSERVER.log and access.log Files?" on page 15

Timeout Setting Mos Credentials in Oracle em12c

Modified 16-SEP-2012

Applies to

Enterprise Manager for My Oracle Support - Version 12.1.0.1.0 and later
This problem can occur on any platform.

Symptoms

Cloud Control is unable to connect against My Oracle Support.
The test against https://updates.oracle.com succeeds.
But setting the preferred credentials for My Oracle Support will fail with: Timeout occurred while reading data from My Oracle Support

Cause

Proxy settings.are needed for connecting to My Oracle Support
The settings in Setup → Proxy Settings are only done for the HTTP proxy, but no settings are done for HTTP settings

Unlike the tip showing on this page the HTTPS proxy must be configured, too. Even if it is the same proxy server and port

Solution

Edit the proxy settings in Setup → Proxy Settings for Manual proxy configuration.

• Let the settings for the HTTP proxy untouched
• Add settings for HTTPS proxy

Automatic Diagnostic Cleanup - Auto purge

Modified 10-JAN-2012

Applies to

Oracle Server - Enterprise Edition - Version: 11.1.0.7 and later [Release: 11.1 and later ]
Information in this document applies to any platform.

Goal

How to increase the frequency of log auto purges in adrci?

Solution

Diagnostic data Purging in ADR is controlled by two attributes:

• SHORTP_POLICY which is used for automatically purging short-lived files, i.e. dump files, expressed in hours and defaults to 30 days.
• LONGP_POLICY which is used for automatically purging long-lived files, i.e. incidents, expressed in hours and defaults to 1 year.

ADR Home = /u01/app/oracle/diag/rdbms/orc11107/orc11107:
*************************************************************************
ADRID SHORTP_POLICY LONGP_POLICY LAST_MOD_TIME LAST_AUTOPRG_TIME LAST_MANUPRG_TIME ADRDIR_VERSION ADRSCHM_VERSION ADRSCHMV_SUMMARY ADRALERT_VERSION CREATE_TIME
-------------------- -------------------- -------------------- ---------------------------------------- ---------------------------------------- ---------------
3963614641 720 8760 2010-08-06 13:18:00 +03:00 2010-08-27 03:25:21 +03:00 1 2 65 1 2010-08-06 13:18:00 +03:00
1 rows fetched

Syntax: purge [[-i {id | start_id end_id}] | [-age mins [-type {ALERT|INCIDENT|TRACE|CDUMP|HM}]]]

11g Understanding Automatic Diagnostic Repository

Modified 27-MAY-2012

Applies to

Oracle Server - Enterprise Edition - Version 11.1.0.6 and later
This problem can occur on any platform.

Purpose

Beginning with Release 11g, Oracle Database includes an advanced fault diagnosability infrastructure for preventing, detecting, diagnosing, and resolving problems.
This note helps to understand the new 11g infrastructure known as Automatic Diagnostic Repository.

Scope

Audience: Oracle 11g Users and DBAs who want to know about ADR.

Details

What is New in 11g?

In 11g, RDBMS diagnostic data has been reorganized and is stored inside a common directory structure, the Automatic Diagnostic Repository (ADR). An ADR is a centralized directory structure where one can find trace files, alert messages, incident dumps, core files, etc.

Automatic Diagnostic Repository (ADR):

All trace files, core files, and the alert files are now organized into a directory structure comprising the Automatic Diagnostic Repository (ADR).

The ADR is a file-based repository for database diagnostic data. It has a unified directory structure across multiple instances and multiple products.
Beginning with Release 11g, the database, Automatic Storage Management (ASM), Cluster Ready Services (CRS), and other Oracle products or components store all diagnostic data in the ADR.
Each instance of each product stores diagnostic data underneath its own ADR home directory.
For example, in an Oracle Real Application Clusters environment with shared storage and ASM, each database instance and each ASM instance has a home directory within the ADR. ADR's unified directory structure, consistent diagnostic data formats across products and instances, and a unified set of tools enable customers and Oracle Support to correlate and analyze diagnostic data across multiple instances.

Problems and Incidents:

Problem: is a critical error in the database
Eg : ORA-600 , ORA-7445 , ORA-4031 etc.

Problem key : Every problem has a problem key, which is a text string that includes an error code (such as ORA-600) and in some cases, one or more error parameters.
Eg : ORA-600, ORA-4030

Incident : is a single occurance of a problem . Each incident has a numeric incident id.

Where is ADR located:

The location of the ADR is controlled by the Oracle "diagnostic_dest" parameter. Path specified in the "diagnostic_dest"
parameter defines the ADR root directory,ADR BASE. The first subdirectory inside an ADR (under the <ADR_BASE> directory) is always named "diag"

For example, if the "diagnostic_dest" and thus the <ADR_BASE> is specified as "$ORACLE_HOME/log", then expect to find the subdirectory "$ORACLE_HOME/log/diag".
Below this will be <ADR_HOME>. Any number of instances/components can share same ADR BASE. Under ADR BASE there will be individual ADR HOMES.
Under ADR BASE ,the address of an <ADR_HOME> will be similar to: "diag/<product_type>/<prod_id>/<instance_id>".
Inside each ADR home, you can find several subdirectories, each for storing a specific type of diagnostic data. Among the subdirectories,
you should be able to find TRACE, ALERT, INCIDENT, CDUMP etc.

ADR HOME contents:

You will find the following directories under ADR HOME -

ALERT: The alert directory contains the XML alert log

CDUMP: core dumps are stored in this directory

TRACE: Process trace files and Alert.log are stored in the trace directory. "Background_dump_dest" and "user_dump_dest" are now ignored in 11g.
Now all the trace files will be generated in "trace" directory.

INCIDENT: The incident directory stores dump files created when critical errors are encountered.
Each occurrence of a critical error (incident) is given its own incident directory, with the incident ID used to form the directory name.

METADATA: The metadata directory stores a series of files that contain diagnostic metadata.

HM: The HM directory stores reports for health checks

INCPKG,IR ,LCK, SWEEP : These directories contain internal diagnosability framework state.

DIAGNOSTIC_DEST - Default value:

If environment variable ORACLE_BASE is set, DIAGNOSTIC_DEST is set to the directory designated by ORACLE_BASE.
If environment variable ORACLE_BASE is not set, DIAGNOSTIC_DEST is set to ORACLE_HOME/log.

V$DIAG_INFO:

For each database , you can query V$DIAG_INFO to check its ADR locations.
This shows us the ADR BASE , ADR home , trace file locations , XML Alert location , incident dump locations ,
core dump and health monitor reports. Also gives us the default session trace (for the current session ) and number of problems ,
incidents reported in the database.

The potential issues are categorized into problem groups and a complete overview of particular errors can quickly be obtained.
Each problem is associated with a single error, such as ORA-00600 (with specific arguments), ORA-04031, etc., and can consist
of one or more so-called incidents (occurrences of the problem).

Details can be retrieved for both a problem and individual occurrences of a problem.

ADR Command Interpreter (ADRCI):

ADR is accessible via either a User Interface called Support Workbench from the Enterprise Manager UI,
or via a command line utility called ADRCI.

The following example focuses on the usage of the command line utility ADRCI to collect data about a specific problem.
When ADRCI is started, the utility needs to be informed which Oracle diagnostic information should be used to analyze an issue.
As indicated, several Oracle products are using the ADR and for each product one or more subdirectories exist in the ADR,
such as individual directories for the various instances running on the system, a subdirectory for the listener, etc.
The SHOW HOMES and SET HOME commands can be used to select which data subset should be accessed:

$ adrci
ADR base = "/u01/app/oracle"
adrci> show homes
ADR Homes:
diag/rdbms/v1120/v1120
diag/tnslsnr/oel5/listener
adrci> set home diag/rdbms/v1120/v1120

To get an overview of the various issues reported in the ADR for that environment, the SHOW PROBLEM command can be used:

adrci> show problem
ADR Home = /u01/app/oracle/diag/rdbms/v1120/v1120:
*************************************************************************************
PROBLEM_ID       PROBLEM_KEY       LAST_INCIDENT      LASTINC_TIME
---------------- ----------------- -------------- ----------------------------------
1               ORA 600 [16613]     3818            2012-02-01 09:23:09 +01:00
1 rows fetched

The PROBLEM_KEY column (or PROBLEM_ID, but the key is more explanatory) is used to obtain details about various occurrences
of the issue using the SHOW INCIDENT command:

adrci> show incident -p "problem_key='ORA 600 [16613]'"
ADR Home = /u01/app/oracle/diag/rdbms/v1120/v1120:
**************************************************************************
INCIDENT_ID       PROBLEM_KEY         CREATE_TIME
---------------- -------------------- ------------------------------------
3817               ORA 600 [16613]      2012-02-01 09:20:34 +01:00
3818               ORA 600 [16613]      2012-02-01 09:23:09 +01:00
2 rows fetched

In this case the ORA-600 [16613] was raised twice. Each incident has trace files and associated incident trace files generated.
To gather information about particular incidents for use in analysis of a Service Request, the following command can be used to
pack this information in a single archive using the command IPS PACK:

adrci> ips pack incident 3817
Generated package 1 in file /home/oracle/database/ORA600166_20120201093056_COM_1.zip, mode complete

The generated package ORA600166_20120201093056_COM_1.zip contains all relevant information required to start analyzing the issue:

• the trace file(s)
• the associated incident trace file(s)
• the instance alert file.

NOTE: the ORA-600 [16613] issue in this example has PROBLEM_ID 1 (as listed in the previous SHOW PROBLEM output).

These generated packages can be uploaded as an attachment to a Service Request via My Oracle Support for
detailed analysis of the issue. Whenever opening a Service Request with Oracle Global Software Support,
providing the package which contains all relevant trace information is highly recommended to avoid delays in resolution.

12c Cloud Control: How to Change the Default Login Timeout Value for the Enterprise Manager Cloud Console Connections

Modified 01-JAN-2013

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.1.0 [Release 12.1]
Information in this document applies to any platform.

Goal

This document explains how to change the Default Login Timeout Value which defines the maximum idle time after
which you will have to log in again to the Cloud Control console.

Solution

This configuration task is described in the following documentation:
Oracle Enterprise Manager Cloud Control Advanced Installation and Configuration Guide
12c Release 1 (12.1.0.1). Download here
Chapter 11 - Performing Additional Configuration Tasks
Topic "Modifying the Default Login Timeout Value"

For your convenience, the steps are provided below:

To prevent unauthorized access to the Cloud Control console, Enterprise Manager will automatically log you out of the
Cloud Control console when there is no activity for a predefined period of time.
For example, if you leave your browser open and leave your office, this default behavior prevents unauthorized users from using your
Enterprise Manager administrator account.

By default, if the system is inactive for 45 minutes or more, and then you attempt to perform an Enterprise Manager action,
you will be asked to log in to the Cloud Control console again.

CAUTION: As stated in the previous paragraphs, the timeout value for logging in to the Cloud Control console is defined in order to protect your system from unauthorized logins.
If you make changes to the login timeout value, be sure to consider the security implications of leaving your session open for other than the default timeout period.

1. Set the value for the OMS ORACLE_HOME environment variable
   Example:
   
   $ export ORACLE_HOME=/u01/app/oracle/product/Middleware/oms



2. Navigate to the directory OMS $ORACLE_HOME/bin
3. Issue the following command to increase the timeout to 60 minutes for example:
   
   $ ./emctl set property -name oracle.sysman.eml.maxInactiveTime -value 60 -sysman_pwd oracle12



4. Stop and restart the OMS to reflect the new property value
   
   $ ./emctl stop oms

$ ./emctl start oms

12c Cloud Control EM CLI: How to Install and Setup 12c EMCLI (Enterprise Manager Command Line Interface)?

Modified 08-OCT-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.1.0 [Release 12.1]
Information in this document applies to any platform.

Goal

This document lists the steps for installing and setting up Enterprise Manager Command Line Interface (emcli) in 12c Cloud Control.
The EM CLI enables you to access Enterprise Manager Grid Control functionality from text-based consoles (shells and command windows) for a variety of operating systems. You can call Enterprise Manager functionality using custom scripts, such as SQL*Plus, OS shell, Perl, or Tcl, thus easily integrating Enterprise Manager functionality with a company's business process.

Using EM CLI, you can perform Enterprise Manager Grid Control console-based operations, such as monitoring and managing targets, jobs, groups, blackouts, notifications, and alerts. EM CLI is intended for use by enterprise or system administrators writing scripts, such as shell/batch files, Perl, Tcl, or PHP, that provide workflow in the customer's business process. You can also use EM CLI commands interactively from an operating system console.

EM CLI is fully integrated with Enterprise Manager's security and user administration functions, enabling you to carry out operations using EM CLI with the same security and confidentiality as the Enterprise Manager Grid Control console.

NOTE:

- You should always use the latest version of EM CLI Client corresponding to the OMS version and not a lower version of EM CLI.

- EM CLI does not allow OS Script jobs to be run against database targets. The Enterprise Manager Cloud Control console, however, does allow this.

- EM CLI has only been certified for submitting OS Script and SQL Script jobs.

- To avoid an uncommon occurrence in which multiple EM CLI sessions are created on the OMS, Oracle recommends that you enter the login command before running a script containing EM CLI commands.

Fix

The EM CLI Client can be installed on any machine within your managed network and provides a subset of the functionality available in the Enterprise Manager Cloud Control console.

Prerequisites

Before installing EMCLI, we need to have the following in place:

• An existing 12c Cloud Control setup.
• Workstation where EM CLI will be installed should have JDK version1.6.0_25 or higher.
• Workstation where EM CLI will be installed should be running on the following platforms - Linux, Solaris, HPUX, Tru64, AIX, Windows with NTFS (client installation).

• Download the 12c EM CLI Kit to your workstation using the following URL:
  
  https://omsmachine.domain:port/em/console/emcli/download
  
  For example:
https://omsmachine.domain:7801/em/console/emcli/download



• Set the JAVA_HOME environment variable and make sure that it is part of your PATH variable: Make sure that this variable is set to the home of a JDK 1.6.0_25 or greater.
  
  For example:
export JAVA_HOME=/u03/jdk16/jdk1.6.0_27
export PATH=$JAVA_HOME/bin:$PATH



• Install the EM CLI client in any directory:
  
  java -jar emclikit.jar client -install_dir=<emcli client directory>
  
  For example:
java -jar emclikit.jar client -install_dir=/home/oracle/emcli
  

NOTE:

By default, emclikit.jar file available under

OMS_HOME/sysman/jlib

We can user the same jar file as another option.

• Setup the EM CLI client using the following command:
  
  emcli setup -dir=<directory of emcli> -url=<OMS URL> -user=<EM username for OMS on client machine>
  
  For example:
emcli setup -dir=/u03 -url=https://omsmachine.domain:7799/em -user=SYSMAN

  

  

• EMCLI Client can be configured to work with multiple OMS by doing multiple setups as shown below:
  
  
  • Setup EMCLI client for OMS1 at location dir1
    
    emcli setup -dir= -url=<URL of OMS1> -user=<EM Username for OMS1>
    
  
  
  • Setup EMCLI client for OMS2 at location dir2
    
    emcli setup -dir=<dir2> -url=<URL of OMS2> -user=<EM Username for OMS2>
    
  
  
  • Set the environment variable EMCLI_STATE_DIR to point to the setup dir for the OMS1
    
    setenv EMCLI_STATE_DIR <dir1>
    
    This will allow the EM CLI Client to work with OMS1
    
  
  
  • Set the environment variable EMCLI_STATE_DIR to point to the setup dir for the OMS2
    
    setenv EMCLI_STATE_DIR <dir2>
    
    This will allow the EM CLI Client to work with OMS2.
    
  
  

1. After setting up the EM CLI client, trying to verify the emcli version shows the EMCLI version as 11.1.0.1.0 as shown below:
   
   

   

   
   This is because, though the 12.1.0.1.0 EM CLI version was setup, as the 11.1 emcli client is in the PATH and is getting picked up.
   To prevent this, remove the 11.1 emcli from the PATH environment variable and verify the EM CLI version, it should provide the expected output:
   
   

   



2. The 'emcli sync' command fails with:
   
   Error: Session expired. Run emcli login to establish a session.

   This can occur if the emcli was not setup correctly.
   
   Refer to our other technical tip:
   
   

   "Cloud Control 12c emcli sync fails with SEVERE: oracle.sysman.emCLI.omsbrowser.OMSBrowserException" on page 7

Cloud Control 12c emcli sync fails with SEVERE: oracle.sysman.emCLI.omsbrowser.OMSBrowserException

Modified 05-DEC-2011

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later [Release 12.1 and later]
Information in this document applies to any platform.

Symptoms

OMS Version: 12.1.0.1

The command emcli login completes successfully.
The command emcli sync fails with:

Error: Session expired. Run emcli login to establish a session.

The file MIDDLEWARE_HOME/gc_inst/em/EMGC_OMS1/sysman/emcli/setup/.emcli/.emcli.log shows:

Nov 4, 2011 12:44:21 PM oracle.sysman.emCLI.verb.SetupVerb execute
INFO: SETUP: localdir=/u01/app/oracle/product/Middleware/gc_inst/em/EMGC_OMS1/sysman/emcli/log,
omsurl=https://<hostname>:7801/em, username=SYSMAN, trustall=true, validate=true, autologin=false, certvalidate=false
[...]

Nov 14, 2011 2:13:22 PM oracle.sysman.emCLI.verb.SynchronizeVerb execute
SEVERE:
oracle.sysman.emCLI.omsbrowser.OMSBrowserException
at oracle.sysman.emCLI.omsbrowser.LoginSystem.establishSession(LoginSystem.java:218)
at oracle.sysman.emCLI.omsbrowser.OMSBrowser.getPageInternal(OMSBrowser.java:653)
at oracle.sysman.emCLI.omsbrowser.OMSBrowser.getPage(OMSBrowser.java:595)
at oracle.sysman.emCLI.verb.SynchronizeVerb.synchronize(SynchronizeVerb.java:280)
at oracle.sysman.emCLI.verb.SynchronizeVerb.execute(SynchronizeVerb.java:162)
at oracle.sysman.emSDK.emCLI.CLIController.execute(CLIController.java:540)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at oracle.sysman.emSDK.emCLI.CLIController.go(CLIController.java:343)
at oracle.sysman.emSDK.emCLI.CLIController.main(CLIController.java:157)


Cause

emcli was not setup correctly at the Cloud Control installation time.

The first lines of the file emcli.log show that emcli was configured with the short name of the host where the OMS has been installed.

INFO: SETUP: localdir=/u01/app/oracle/product/Middleware/gc_inst/em/EMGC_OMS1/sysman/emcli/log, omsurl=https://<hostname>:7801/em, username=SYSMAN, trustall=true, validate=true, autologin=false, certvalidate=false

emcli should have instead been setup with the fully qualified host name: hostname.domainName.
The file emcli.log should have:

INFO: SETUP: localdir=/u01/app/oracle/product/Middleware/gc_inst/em/EMGC_OMS1/sysman/emcli/log, omsurl=https://<hostname.domainname>:7801/em,
username=SYSMAN, trustall=true, validate=true, autologin=false, certvalidate=false

Example:
hostname.domainname = mycloudhost.mydomain.com.

INFO: SETUP: localdir=/u01/app/oracle/product/Middleware/gc_inst/em/EMGC_OMS1/sysman/emcli/log, omsurl=https://mycloudhost.mydomain.com:7801/em, username=SYSMAN, trustall=true, validate=true, autologin=false, certvalidate=false


Solution

To fix the problem, please execute:

From the host where the OMS is running:

Setup emcli with the fully qualified name of the host where the OMS is running

$ cd $OMS_ORACLE_HOME/bin
$ ./emcli setup -url=https://<OMS Fully Qualified Host Name>:<Console Port>/em -username="sysman" -password="<your password>" -trustall

Example:

$cd /u01/app/oracle/product/Middleware/oms/bin
$ ./emcli setup -url="https://myhostname.mydomainname:7801/em" -username=sysman -password=XXXX -trustall


12c Cloud Control EM CLI: 12c: How to delete Agent/targets using EMCLI Delete Verb

Modified 01-JAN-2013

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

How to delete an Agent (and all its monitoried targets) or a specific target in EM 12c using EMCLI verb?

Fix

1. To delete an Agent and all its targets, we can use the following command:
   
   $<EMCLI_HOME>/emcli delete_target -name="hostname:3872" -type="oracle_emd" -delete_monitored_targets -async

   NOTE: The above deletes the Agent and ALL the monitored targets from the repository/console.
   

   
   "-delete_monitored_targets" : Delete the targets monitored by the specified Management Agent. Applicable only with the oracle_emd target type.
   


2. To delete any specific target:
   
   $<EMCLI_HOME>/emcli delete_target -name="<target_name>" -type="<target_type>" -delete_monitored_targets -async
   
   Example:

$./emcli get_targets -targets="DBname:oracle_database"

$./emcli delete_target -name="DB1" -type="oracle_database" -delete_monitored_targets -async

How to De-install the Enterprise Manager Cloud Control 12c Agent

Modified 22-FEB-2013

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.2.0 [Release 12.1]
Information in this document applies to any platform.

Purpose

This document provides Steps to Deintall Oracle Management Agent (Intelligent Agent) using various methods. It includes the following:

• Prerequisites
• Deinstalation Steps
• Post Deinstallation Steps

Scope

This document describes in detail the Steps for Deinstalling Oracle Enterprise Manager Cloud Control 12c and Oracle Management Agent (Management Agent) 12c using a Graphical and Silent methods.


Details

1. Before you deinstall a Management Agent, do the following:
   
   
   1. Stop the Agent using command from Management Agent home:
      
      $ emctl stop agent
      
      Example:
../agent12c/agent_inst/bin/emctl stop agent

   
   
   2. Wait for the Management Agent to go to the unreachable state in the Cloud Control console.
   3. It is mandatory to delete the Management Agent and their monitored targets using EMCLI as explained in our previous technical tip on this page.
      
      Example:
$ emcli login -username=SYSMAN
$ emcli sync
$ emcli delete_target -name="example.com:1836" -type="oracle_emd" -delete_monitored_targets
Or
      
      Manually remove an agent from 12C Cloud Control as explained in our next technical tip:
      
      

      "EM 12C: How to Manually Remove an Agent From 12C Cloud Control" on page 9

   
   
2. Deinstalling using Graphical Method:
   
   
   1. Invoke the installer from Management Agent home by running the following command:
      
      $<AGENT_HOME>/oui/bin/runInstaller -deinstall ORACLE_HOME=<absolute_path_to_agent_home> [-removeallfiles]
      
      Example:
$ /u01/app/oracle/agent/core/12.1.0.1.0/oui/bin/runInstaller -deinstall ORACLE_HOME=/u01/app/oracle/agent/core/12.1.0.1.0 -removeallfiles

   
   
   2. In the Installation wizard click on "Installed Products" button.
   3. On the Inventory screen, select the plug-in homes, and click "Remove" button.
   4. On the Inventory screen, select the sbin home, and click "Remove" button.
   5. On the Inventory screen, select the Management Agent, and click "Remove" button.
   
   Or
   
   
3. Deinstalling using Silent Method:
   
   
   1. Deinstall the plug-in homes:
      
      $/oui/bin/runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={absolute_path_to_plug-in_home}" -invPtrLoc
      
      Example:
$ ../agent/core/12.1.0.1.0/oui/bin/runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={../agent/plugins/oracle.sysman.emas.oms.plugin_12.1.0.1.0, ../agent/plugins/oracle.sysman.emct.oms.plugin_12.1.0.1.0}" -invPtrLoc /home/oracle/oraInst.loc

   
   
   2. Deinstall the sbin home:
      
      $/oui/bin/runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={absolute_path_to_sbin_directory}" -invPtrLoc
      
      Example:
$ ../agent/agent_inst/oui/bin/runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={../agent/sbin}" -invPtrLoc /home/oracle/oraInst.loc

   
   
   3. Deinstall the Management Agent:
      
      $/oui/bin/runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={absolute_path_to_agent_oracle_home}" -invPtrLoc
      
      Example:
$ ../agent/core/12.1.0.1.0/oui/bin/runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={../agent/core/12.1.0.1.0}" -invPtrLoc /home/oracle/oraInst.loc


   NOTE: Parameter -invPtrLoc is optional
   For Windows: Instead of runInstaller use setup.exe
   

   
   
4. (Only for Graphical Mode) Verify whether the Oracle Homes and other directories were successfully deinstalled. To do so, follow these steps:
   
   
   1. Invoke the installation wizard by running the following command from the Management Agent Home:
      
      $/oui/bin/runInstaller
      
      Example:
$ ../agent/core/12.1.0.1.0/oui/bin/runInstaller

   
   
   2. In the installation wizard, on the My Oracle Support Details screen, click on "Installed Products" button.
   3. On the Inventory screen, check whether or not the Oracle Homes and other directories you deinstalled appears. If the deinstallation was successful then those Oracle Homes and directories should not appear.
   
   
5. Open inventory.xml and check Agent plugin home and Agent sbin home entries are removed or not. These entries should be removed during deinstallation.
   
   

   NOTE: inventory.xml would be located under ContentsXML folder under the inventory location in oraInst.loc file.
   

   
   
6. Remove the Cloud Control Management Agent base directory:
   
   For UNIX platforms:
   
   $ rm -rf <absolute_path_to_agent_base_dir>
   
   Example:
$ rm -rf /u01/app/oracle/product/agent12c

   For Microsoft Windows platforms:
   
   C:\app\oracle> del <absolute_path_to_agent_base_dir>
   
   Example:
C:\app\oracle> del c:\app\oracle\product\agent12c


EM 12C: How to Manually Remove an Agent From 12C Cloud Control

Modified 29-JAN-2013

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

How to manually remove a 12.1.0.1 Agent in 12C Cloud Control.

Fix

Perform the following steps to delete an Agent from 12C Cloud Control:

NOTE: For Cloud Control 12c Release 12.1.0.1.0 - Before Deleing the Agent from Console, ensure that you delete the targets monitored by this agent.

1. Ensure that the agent on the target machine is not running:
   
   <AGENT_HOME>/bin> emctl status agent
<AGENT_HOME>/bin> emctl stop agent



2. Remove the Agent target manually from the console as follows:
   
   
   1. Login to 12C Cloud Control
   2. Navigate to Setup → Manage Cloud Control → Agents (Setup → Agents In cloud Control 12c Release 12.1.0.1)
   3. Go to the Home page of the Agent that is to be deleted
   4. Expand the drop-down menu near the "Agent"
   5. Expand the "Target Setup" option
   6. Select "Remove Target"
   7. Following confirmation dialogue box will be displayed (may be slightly different for different Cloud Control 12c Releases):

   "You have chosen to remove : (Agent). Do you want to proceed?"
   

   
   Select "yes" to delete the agent.
   
   In Cloud Control Release 12.1.0.2, the following dialog box will be displayed if the Agent is still monitoring targets:
   

   

   
   Click Continue. You can then remove all targets (using the "Select All" link) in the next page.
   You will then be able to remove the host/agent itself.

"12C Cloud Control: How to De-install 12C Agent" for deinstallation of Agent" on page 8.

NOTE: Ensure that all targets monitored by the Agent are removed including the host target; before proceeding with the Agent Deletion.
If the Agent is still monitoring any other targets, then the following error will be displayed in the console:

Unknown error.java.sql.SQLException: ORA-20242: Target myhost.oracle.com:3875:oracle_emd is monitoring other targets. It cannot be deleted
ORA-06512: at "SYSMAN.MGMT_ADMIN", line 484
ORA-06512: at "SYSMAN.MGMT_ADMIN", line 876
ORA-06512: at line 1

12c Cloud Control: Installation Fails with oracle.sysman.assistants.common.dbutil.SQLFatalErrorException:
java.sql.SQLException: ORA-04030: out of process memory

Modified 14-JUN-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

When installing 12c Cloud Control,but it failed at repository configuration stage.

Getting following error in rcu.log

Caused by: oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: java.sql.SQLException: ORA-04030: out of process memory when trying to allocate 103504 bytes (QERHJ hash-joi,kllcqc:kllcqslt)
ORA-06512: at "SYS.DBMS_AQADM", line 81
ORA-06512: at line 15
ORA-01403: no data found

Also Repository database alert.log shows

Errors in diag/rdbms/omsrepo/omsrepo/trace/omsrepo_j001_2375780.trc (incident=7086):
ORA-04030: out of process memory when trying to allocate 123416 bytes (QERHJ hash-joi,kllcqas:kllsltba)

Use ADRCI or Support Workbench to package the incident.

Cause

This error

java.sql.SQLException: ORA-04030: out of process memory when trying to allocate 103504 bytes (QERHJ hash-joi,kllcqc:kllcqslt)

caused by OS limit has been reached preventing oracle processes from allocating needed memory.

This can be checked by the following

ulimit -a
ulimit -ha

Sample out puts

oracle@abc:/home/oracle$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 31072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 4096

oracle@abc:/home/oracle$ ulimit -Ha
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 4194304
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) 4096


Solution

1. Increase the ulimits for stack to unlimited ulimit -s unlimited.
2. Make sure the ulimit -Hs is also set to unlimited.
3. Be sure to shutdown and restart the oracle server and listener so that the new ulimits take effect.
4. Re Try Instalation.

OCM fails to discover the Oracle Management System (OMS) 11.1.0.1.0

Modified 04-JAN-2012

Applies to

Oracle Configuration Manager - Version: 10.3.2.0 and later [Release: and later]
Oracle Configuration Manager - Version: 10.3.2.0 to 10.3.6.0.0
Information in this document applies to any platform.

Symptoms

You have installed successfully Oracle Configuration Manager 10.3.3.1.0 in the Grid Control Oracle Management Service (OMS) 11.1.0.1.0 ORACLE_HOME.
You check the xml files collected in the directory under the OMS $ORACLE_HOME/ccr/host/<host_name>/state/review.
You notice that there are only the following files :

• BB4D4FBF880D5CF02365405AA59D32D3-oracle_home_config.xml
• 949EAACD431225800C0BA2CBFA7D0BB5-livelink_config.xml
• DADDD6B3873D0957DD406B28E51DB598-ll_host_config.xml
• targetMap.xml

Cause

Beginning with the Grid Control release 11.1.0.1.0, the configuration properties of the OMS are stored in the file emgc.properties.

OCM needs to browse this file in order to discover the OMS and collects data.

OCM will look for this file in the directory specified by the environment variable ORACLE_CONFIG_HOME.

If this environment variable is not set when OCM is configured executing the command setupCCR, OCM will fail to discover the OMS.

Solution

In order to fix the issue, you need to follow the steps below:

1. Uninstall OCM. You need to uninstall OCM as the previous configuration removed the file setupCCR.
   
   
   1. From the OMS $ORACLE_HOME/ccr/bin directory execute:
      
      $ deployPackages -d $ORACLE_HOME/ccr/inventory/core.jar

   
   
   2. Remove the ccr directory and all directories and files under this directory
      
      $ rm -rf $ORACLE_HOME/ccr (on Unix)

c> rmdir /s/q %ORACLE_HOME%\ccr (On Windows)

2. Re-install OCM. Unzip the OCM zip file to the OMS $ORACLE_HOME
3. Look for the location of the file emgc.properties
   The location of the file emgc.properties is stored in the file OMS $ORACLE_HOME/sysman/config/emInstanceMapping.properties.
   
   Example: EMGC_OMS1=/u01/app/oracle/product/gc_inst/em/EMGC_OMS1/emgc.properties



4. Set the environment variable ORACLE_CONFIG_HOME.
   
   Example:
$ export ORACLE_CONFIG_HOME=/u01/app/oracle/product/gc_inst/em/EMGC_OMS1



5. Configure OCM from the OMS $ORACLE_HOME/ccr/bin directory:
   
   $ ./setupCCR

NOTE: In a multiple OMS Home environment, steps 1-5 above needs to be completed on each location also. This keeps all OMS locations in sync.

• <config_id>-oracle_home_config.xml
• <config_id>-livelink_config.xml
• <config_id>-oracle_oms_config.xml
• <config_id>-ll_host_config.xml
• targetMap.xml

Script for Checking the Grid Control Agent CPU, Memory & Threads Usage

Modified 10-SEP-2012

Applies to

Enterprise Manager Base Platform - Version: 10.2.0.1 and later [Release: 10.2 and later ]
Linux x86
IBM AIX on POWER Systems (64-bit)
Oracle Solaris on SPARC (64-bit)
Linux x86-64

Purpose

The script is intended to provide an overview of the Agent CPU, Memory & Threads as well as of the running, ready and scheduled collections.
The script sample interval is 2 minutes. This can be customized.

Software Requirements/Prerequisites

This is a shell script. The only prerequisite is that the agent is running while running the script.

Configuring the Script

1. Download the Script or save its content in a file called "agent_snap.sh" under Agent's <ORACLE_HOME/bin> directory.
2. Change the permissions on the script to make it executable:
   
   $ chmod u+x agent_snap.sh

Running the Script

1. Make sure the $ORACLE_HOME environment variable is set to the agent oracle home before running the script:
   
   $ export ORACLE_HOME=<path to agent home>



2. Make sure that the agent is running:
   
   $ORACLE_HOME/bin/emctl status agent



3. Run the script in the background:
   
   $ cd $ORACLE_HOME/bin
$ ./agent_snap.sh &



4. Verify that the agent_snap.log file is created under <AGENT_HOME>/sysman/log or <AGENT_HOME>/<nodename>/sysman/log (for RAC agents)
   and populated with the expected information.
5. The script is set to collect data by default every 120 seconds. You can change the interval as required by modifying the following line of the script:
   
   sleep 120

   Leave the script running for as long as you need to gather relevant information for the problem you are diagnosing.
   Note that you can customize the script as needed to run other commands to capture additional information every 120 seconds.
   


6. To stop the script - identify the process id and kill it:
   
   $ ps -ef | grep agent_snap.sh
$ kill -9 <pid of agent_snap.sh>


NOTE: The script was tested on Linux, Oracle Solaris & AIX. It may need customization on other Unix platforms.

Caution

This script is provided for educational purposes only and not supported us. It has been tested internally, however, and works as documented.
We do not guarantee that it will work for you, so be sure to test it in your environment before relying on it.
Proofread this script before using it! Due to the differences in the way text editors, e-mail packages and operating systems
handle text formatting (spaces, tabs and carriage returns), this script may not be in an executable state when you first receive it.
Check over the script to ensure that errors of this type are corrected.

Scripts

1. agent_snap.sh for Linux (Link to shell script)
   
   #!/bin/sh

emhome=`$ORACLE_HOME/bin/emctl getemhome | tail -1 | awk 'BEGIN {FS="="} {print $2}'`
echo -e "date\t\tcpu%\tvmem\trmem\tthreads\trunning\tready\tscheduled" >> $emhome/sysman/log/agent_snap.log
while [ 2 -gt 1 ]
do

pid=`$ORACLE_HOME/bin/emctl status agent|grep "Agent Process ID" | awk '{print $5}'`
dateStr=`date "+%m/%d %H:%M:%S"`
cpu=`ps -p $pid -o "pcpu"| tail -1 | sed 's/^[ ]*//'`
threads=`ps -p $pid -o "thcount"| tail -1 | sed 's/^[ ]*//'`
vmem=`ps -o vsz $pid | tail -1 | sed 's/^[ ]*//'`
rmem=`ps -o rss $pid | tail -1 | sed 's/^[ ]*//'`

$ORACLE_HOME/bin/emctl status agent scheduler > /tmp/snapschedule
running_line=`grep -n "Running entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}'`
ready_line=`grep -n "Ready entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}'`
scheduled_line=`grep -n "Scheduled entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}'`
final_line=`grep -n "Agent is" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}'`
running=`expr $ready_line - $running_line - 1`;
ready=`expr $scheduled_line - $ready_line - 1`;
scheduled=`expr $final_line - $scheduled_line - 2`;

echo -e "$dateStr\t$cpu\t$vmem\t$rmem\t$threads\t$running\t$ready\t$scheduled" >> $emhome/sysman/log/agent_snap.log

sleep 120
done



2. agent_snap.sh for Oracle Solaris (Link to shell script)
   
   #!/bin/sh

emhome=`$ORACLE_HOME/bin/emctl getemhome | tail -1 | awk 'BEGIN {FS="="} {print $2}'`
echo "date\t\tcpu\tvmem\trmem\tthreads\trun\tready\tsched" >> $emhome/sysman/log/agent_snap.log
while [ 2 -gt 1 ]
do
pid=`$ORACLE_HOME/bin/emctl status agent|grep "Agent Process ID" | awk '{print $5}'`
dateStr=`date "+%m/%d %H:%M:%S"`
cpu=`ps -p $pid -o "pcpu"| tail -1 | sed 's/^[ ]*//'`
threads=`ps -p $pid -o "nlwp"| tail -1 | sed 's/^[ ]*//'`
vmem=`ps -p $pid -o "vsz" | tail -1 | sed 's/^[ ]*//'`
rmem=`ps -p $pid -o "rss" | tail -1 | sed 's/^[ ]*//'`
$ORACLE_HOME/bin/emctl status agent scheduler > /tmp/snapschedule

running_line=`grep -n "Running entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}'`
ready_line=`grep -n "Ready entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}'`
scheduled_line=`grep -n "Scheduled entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}'`
final_line=`grep -n "Agent is" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}'`
running=`expr $ready_line - $running_line - 1`;
ready=`expr $scheduled_line - $ready_line - 1`;
scheduled=`expr $final_line - $scheduled_line - 2`;

echo "$dateStr\t$cpu\t$vmem\t$rmem\t$threads\t$running\t$ready\t$scheduled" >> $emhome/sysman/log/agent_snap.log

sleep 120
done



3. agent_snap.sh for AIX (Link to shell script)
   
   #!/bin/sh

emhome=`$ORACLE_HOME/bin/emctl getemhome | tail -1 | awk 'BEGIN {FS="="} {print $2}'`
echo "date\t\tcpu\tvmem\trmem\tthreads\trun\tready\tsched" >> $emhome/sysman/log/agent_snap.log
while [ 2 -gt 1 ]
do
pid=`$ORACLE_HOME/bin/emctl status agent|grep "Agent Process ID" | awk '{print $5}'`
dateStr=`date "+%m/%d %H:%M:%S"`
cpu=`ps -p $pid -o "pcpu"| tail -1 | sed 's/^[ ]*//'`
threads=`ps -p $pid -o "thcount"| tail -1 | sed 's/^[ ]*//'`
vmem=`ps v $pid | tail -1 | sed 's/^[ ]*//' | awk '{print $6}'`
rmem=`ps v $pid | tail -1 | sed 's/^[ ]*//' | awk '{print $7}'`
$ORACLE_HOME/bin/emctl status agent scheduler > /tmp/snapschedule

running_line=`grep -n "Running entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}'`
ready_line=`grep -n "Ready entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}'`
scheduled_line=`grep -n "Scheduled entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}'`
final_line=`grep -n "Agent is" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}'`
running=`expr $ready_line - $running_line - 1`;
ready=`expr $scheduled_line - $ready_line - 1`;
scheduled=`expr $final_line - $scheduled_line - 2`;

echo "$dateStr\t$cpu\t$vmem\t$rmem\t$threads\t$running\t$ready\t$scheduled" >> $emhome/sysman/log/agent_snap.log

sleep 120
done


Script Output

This is a sample output:

date cpu vmem rmem threads run ready sched
02/12 13:06:50 0.1 84616 54692 30 0 0 73
02/12 13:08:54 0.1 84616 54608 23 0 0 73
02/12 13:10:58 0.1 84616 54036 29 0 0 73


The agent_snap.log file will show the timestamp, CPU usage, virtual memory usage, physical memory usage and number of threads
spawned by the agent process. It will also show the number of running, ready and scheduled collection items.
The latter are based on the 'emctl status agent scheduler' and can show interesting trends in the number of running metric collections
- that can then be linked to what we see in terms of resource consumption.

Sqlplus Utility is not Shipped with 11g Grid Control and 12c Cloud Control Installation

Modified 09-APR-2012

Applies to

Enterprise Manager Base Platform - Version: 11.1.0.1 to 12.1.0.1.0 - Release: 11.1 to 12.1
Information in this document applies to any platform.

Goal

SQL*Plus is no longer shipped from the version 11.1.0.1 Grid Control onwards in the OMS directory.

SQL*Plus has been replaced in 11.1.0.1 Grid Control and above versions by a new program called "rcuJDBCEngine",
which will be used when executing the post-installation steps of a one-off patch on the OMS server.

Solution

For an OMS patch that requires executing a sql file, use the <OMS_HOME>/bin/rcuJDBCEngine script.

The 'rcuJDBCEngine' script is used as follows:

1. Set MW_HOME to the Weblogic Home and ORACLE_HOME to the OMS Home.
2. Execute 'rcuJDBCEngine' to run the <post_install_script>.sql as follows:
   
   $ORACLE_HOME/bin/rcuJDBCEngine sys/<password for sys>@<host>:<port>:<SID> JDBC_SCRIPT <post_install_script>.sql $PWD $ORACLE_HOME
   
   For Example:

$ORACLE_HOME/bin/rcuJDBCEngine sys/welcome1@reposmachine.domain:1521:emrep JDBC_SCRIPT post_install_script.sql $PWD $ORACLE_HOME

   You should get the message "Completed SQL script execution normally." - as an indication that the script was run successfully.
   

12c Cloud Control: How to Verify the Connectivity from 12c OMS to Repository Database using rcuJDBCEngine

Modified 14-SEP-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Purpose

This document describes the steps for using rcuJDBCEngine in the 12c OMS home for testing the connectivity between the OMS and the Repository Database.
This can be useful in troubleshooting OMS startup failures due to connectivity problems with the Repository Database.

rcuJDBCEngine has to be used as the sqlplus utility is no longer shipped in the 12c Cloud Control installation as described
in our technical tip:

"Sqlplus Utility is not Shipped with 11g Grid Control and 12c Cloud Control Installation" on page 11

Solution

1. In OMS machine, set the below environment variables:
   
   export MW_HOME=<Location where the 10.3.5 Weblogic is software installed>
   export ORACLE_HOME=<path of the 12c OMS installation>
   


2. Create a file named: test.sql in the $ORACLE_HOME/bin with the following SQL statement:
   
   select sysdate from dual;



3. Usage for the rcuJDBCEngine is:
   
   
   1. rcuJDBCEngine <user>/<password>@<service> <JDBC_BLOCK|JDBC_SCRIPT|SQLSERVER_SCRIPT|DB2_SCRIPT> <SQL-script-file>
      
      The above usage will be deprecated soon, so the following usage is recommended.
      
   
   
   2. rcuJDBCEngine <user>@<service> <JDBC_BLOCK|JDBC_SCRIPT|SQLSERVER_SCRIPT|DB2_SCRIPT> <SQL-script-file> [-f < filename]
      
      In this, user will be prompted for the password. To pass the database password via a file, use -f < filename option as the last parameter
      The service should be in following format for the Repository DB:
      
      <hostname>:<port>:<service_name> or the entire connect descriptor
      


4. Execute the test.sql using rcuJDBCEngine:
   
   cd $ORACLE_HOME>/bin
   ./rcuJDBCEngine sysman@<Repository Hostname>:<Listener Port>:<Repository Database SID> JDBC_SCRIPT test.sql
   
   This will prompt you for the sysman password.
   
   For example:
   
   ./rcuJDBCEngine sysman/@reposmachine.domain:1521:emrep JDBC_SCRIPT test.sql
Please enter database password.
2012-04-09 21:10:59.309 rcu:Extracting Statement from File Name: 'test.sql'Line Number: 1
2012-04-09 21:10:59.310 rcu:Extracted SQL Statement: [select sysdate from dual]
2012-04-09 21:10:59.310 rcu:Statement Type: 'DDL Statement'
Completed SQL script execution normally.
1 scripts were processed

   The above output indicates that the connection to Repository Database is successful and the script was executed successfully but the sql output will not be shown on the terminal.
   


5. If the connection is not successful, corresponding errors will be shown:
   
   - If the Repository hostname/the Listener Port value is wrong/if Listener is not up:
   
   

   >A SQL Error occurred!: oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: Unable to connect to the database using the provided details.
   Please enter a valid hostname and port or check if the listener is up and running.
   at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.connect(JDBCEngine.java:532)
   at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.main(JDBCEngine.java:2345)
   

   
   - If the Repository Database is not up:
   
   

   >A SQL Error occurred!: oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: Invalid SID or Service name.
   Please enter valid SID or Service name.
   at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.connect(JDBCEngine.java:532)
   at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.main(JDBCEngine.java:2345)
   

   
   - If the SYSMAN Password is incorrect:
   
   

   >A SQL Error occurred!: oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: Invalid username/password.
   Please enter valid username/password
   at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.connect(JDBCEngine.java:532)
   at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.main(JDBCEngine.java:2345)
   

ADRCI PURGE Command Is Not Cleaning Up CDUMP_<TIMESTAMP> Folders In TRACE Directory

Modified 07-FEB-2013

Applies to

Oracle Server - Enterprise Edition - Version 11.1.0.6 to 11.1.0.6 [Release 11.1]
Information in this document applies to any platform.
Oracle Server Enterprise Edition - Version: 11.1.0.6

Symptoms

For this article there are no symptoms in terms of errors seen in alert or within the session.

Instead for this article to be applicable the user will be trying to delete folders using the ADRCI tool under Oracle11g 11.1.0.6,
the problem is only seen on this particular release and is resolved in the 11.1.0.7 patchset.

There is no clear documentation to show how such directories can be removed using ADRCI so to demonstrate the problem and solution an example is now provided.
The example is specific to a particular machine/instance so cannot be simulated by a testcase. The purpose of the example is simply to highlight syntax and usage of ADRCI:

1. navigate to the directory containing the instance trace information:
   
   cd /app/oracle/diag/rdbms/v1116u/V1116U/trace

   The path will depend on the installation. If unsure of on the location, issue the following query against the database instance:
   
   SELECT VALUE FROM V$DIAG_INFO WHERE NAME LIKE 'Diag cdmp%';



2. At the OS command prompt issue the directory listing command, like in:
   
   $ ls -ltr | grep cdmp

drwxr-xr-x 2 oracle dba 4096 Nov 11 11:54 cdmp_20081111115441/
drwxr-xr-x 2 oracle dba 4096 Nov 13 09:10 cdmp_20081113091031/
drwxr-xr-x 2 oracle dba 4096 Nov 20 11:23 cdmp_20081120112348/
drwxr-xr-x 2 oracle dba 4096 Nov 27 06:52 cdmp_20081127065231/
drwxr-xr-x 2 oracle dba 4096 Nov 27 06:52 cdmp_20081127065232/
drwxr-xr-x 2 oracle dba 4096 Nov 27 06:52 cdmp_20081127065235/



3. initiate the ADRCI utility:
   
   $ adrci


4. at the ADRCI command prompt display all Oracle Homes maintained by this ADR:
   
   adrci> show homes


5. If (d) shows more than 1 home, select the proper home for use with ADRCI:
   
   adrci> set homepath diag/rdbms/v1116u/V1116U
adrci> set home diag/rdbms/v1116u/V1116U

   The SET command here depends on (a). If (c) already shows diag/rdbms/v1116u/V1116U then no action needed.
   


6. find out what incidents are reported for this instance:
   
   adrci> show incident

INCIDENT_ID PROBLEM_KEY CREATE_TIME
----------- ----------------------------- ------------------------------
39828 ORA-4031 2008-11-27 06:52:26 +00:00
39820 ORA-4031 2008-11-27 06:52:28 +00:00
39796 ORA-4031 2008-11-27 06:52:27 +00:00
39780 ORA-4031 2008-11-27 06:52:32 +00:00
39748 ORA-4031 2008-11-27 06:52:29 +00:00
31411 ORA-7445 [mdagun_term()+1031] 2008-11-20 11:23:43 +00:00

6 rows fetched



7. try to purge the evidence of the incident 31411:
   
   adrci> purge -i 31411
   'show incident' will then show the 5 remaining rows but still 'ls -ltr' shows six CDMP_<timestamp> directories.

Cause

The cause of this problem has been identified as a bug. It is caused by the ADR of 11.1.0.6 not taking into account the purging if CDMP directories.

Solution

In Oracle 11g 11.1.0.7 and higher releases a new option 'UTSCDMP' has been added to ADRCI's PURGE command:

adrci> help purge

Usage: PURGE [[-i <id1> | <id1> <id2>] |
[-age [-type ALERT|INCIDENT|TRACE|CDUMP|HM|UTSCDMP]]]:
...

The UTSCDMP type has been added by the bug fix to allow purge of the CDMP related information; e.g.:

adrci> purge -age 1440 -type UTSCDMP
This will delete the CDMP_<timestamp> directories from the TRACE directory which are more than 1 day old (1440 minutes = 1 day).
When available, download Patch 6343743 from Oracle Support to resolve this issue.

After Upgrade To 11.2.0.2 ADRCI Purge Command Fails With DIA-48322

Modified 16-DEC-2011

Applies to

Oracle Server - Enterprise Edition - Version: 11.2.0.2 and later [Release: 11.2 and later]
Information in this document applies to any platform.

Symptoms

After upgrading the software to 11.2.0.2 patchset, ADRCI purge command starts to fail with errors:

adrci> purge
DIA-48322: Relation [INCIDENT] of ADR V[2] incompatible with V[2] tool
DIA-48210: Relation Not Found
DIA-48166: error with opening ADR block file because file does not exist [/grdbms/64bit/app/oracle/diag/tnslsnr/rmtdcsol1/listener/metadata/INCIDENT.ams] [0]

adrci> show incident
ADR Home = /grdbms/64bit/app/oracle/diag/tnslsnr/rmtdcsol1/listener:
*************************************************************************
0 rows fetched

Changes

Upgrade to 11.2.0.2 patchset.

Cause

These errors are raised when you run a purge command in the listener ADR Home without incidents to purge,
and after an upgrade to the patch set level is done.

Solution

This feature will be enhances in future releases.
At present the errors could be safely ignored, but to prevent them the following workaround can be used:

1. Stop listener
2. Backup listener's ADR directory
   
   Eg.
   % cp -pr <listener's ADR directory> <backupfile>
   


3. Remove listener's ADR directory
   
   Eg.
   % rm -fr <listener's ADR directory>
   


4. Start listener
   
   This will create listener's ADR directory automatically
   


5. Retry the purge command, which should return no errors:
   
   

   adrci> set home diag/tnslsnr/rmtdcsol1/listener
   adrci> purge
   adrci> show incident
   
   ADR Home = /grdbms/64bit/app/oracle/diag/tnslsnr/rmtdcsol1/listener:
   *************************************************************************
   0 rows fetched
   

Understanding Database Discovery in Enterprise Manager 12cCloud Control

Modified 16-FEB-2013

Applies to

Enterprise Manager for Oracle Database - Version: 12.1.0.1.0 and later
Information in this document applies to any platform.

Purpose

This document is to discuss the different methods of discovering a database in Enterprise Manager Cloud Control 12c,
relevent background information and the pre-requisites necessary to achieve this.

Scope

It is intended for Enterprise Manager Administrators who are famiilar with the previous versions of Enterprise Manager.

Details

1. Background Information
   
   There are three ways to discover a database in Enterprise Manager 12c Cloud Control.
   
   
   • Automatic discovery
   • Manual discovery via Console
   • Manual discovery via command line
   
   Database discovery in Enterprise Manager 12c Cloud Control, differs to the previous versions of Enterprise Manager
   (Grid control 11.1.0.1, Grid Control 10.2.0.X and 9.2.0 OEM), in the sense that database targets which are discovered
   automatically are not populated in the console. Database/listener targets are auto-discovered but not displayed in the
   console until the user chooses to "promote" that particular database/listener target.
   
   The concept of "target promotion" is new in 12c but involves selecting the databases to be displayed in the console,
   and configuring them for monitoring by entering the dbsnmp username/password (configuring them is similar to "configuring"
   a database in 11g grid control). This removes the possibility of the Administrator seeing unwanted or unconfigured targets
   in the Console. This means that when a new agent is added to Enterprise Manager Cloud Control 12c, the database targets which
   run on that agent machine will not be shown automatically in the console.
   
   As well as the automatic discovery method, there is also a possibility to add the database manually either using the console,
   or via emcli (enterprise manager command line interface). When adding a database manually via the console, only one database
   at a time can be added (so again, this differs from the "add database" mechanism in previous versions of Enterprise Manager).
   Only the automatic discover method requires target promotion, the manual methods do not require promotion because the targets
   are manually chosen and configured by the user and so the necessary dbsnmp connect info has already been supplied.
   
   PLUGINS
   
   Another new concept in 12c is the concept of the database 'plugin'. Any database which is monitored by Cloud Control needs to use the
   "database plugin". The database plugin (version 12.1.0.1.0 with no revisions) is automatically deployed on the OMS by default.
   The database plugin also needs to be deployed on any management agent which needs to manage a database. The database plugin can either
   be deployed by the user to the management agent (via the setup/extensibility/plugins screen) or it will be deployed automatically at the
   time when the first database target to be managed by an agent is added, as part of the add database/promote database process.
   
   If subsequently it is a requirement to have a later version of the plugin (eg. 12.1.0.2.0) installed on the agent.
   The existing plugin can be upgraded from the "Plugins" console.
   
   

   Note that if intending to deploy the latest plugin on an agent, this latest version of the plugin needs to first be deployed on the OMS,
   before it can be deployed on the remote agent.
   

   
   If a database is discovered by an agent that has already previously discovered a database, the database plugin will already be installed on
   that agent in which case, the version of the database plugin on the OMS, and the version of the database plugin on the monitored agent, may be different.
   The version of the plugin installed on an agent can be checked from:-
   
   

   Setup/Extensibility/Plugins
   expand "Databases" folder
   Highlight "Oracle Database"
   go to Actions:Information
   click "Management agent" tab
   

   
   

   The plugins deployed on the agent can also be checked by:-
   
   cd $ORACLE_HOME<of agent>
   
   /oracle/12cagent/core/12.1.0.2.0
   
   [oracle@machine1 12.1.0.2.0]$ emctl listplugins agent
   Oracle Enterprise Manager Cloud Control 12c Release 2
   Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
   ---------------------------------------------------------------
   oracle.sysman.emas 12.1.0.3.0 /oracle/12cagent/plugins/oracle.sysman.emas.agent.plugin_12.1.0.3.0
   oracle.sysman.emrep 12.1.0.2.0 /oracle/12cagent/plugins/oracle.sysman.emrep.agent.plugin_12.1.0.2.0
   oracle.sysman.beacon 12.1.0.2.0 /oracle/12cagent/plugins/oracle.sysman.beacon.agent.plugin_12.1.0.2.0
   oracle.sysman.oh 12.1.0.2.0 /oracle/12cagent/plugins/oracle.sysman.oh.agent.plugin_12.1.0.2.0
   oracle.sysman.csa 12.1.0.2.0 /oracle/12cagent/plugins/oracle.sysman.csa.agent.plugin_12.1.0.2.0
   oracle.sysman.db 12.1.0.2.0 /oracle/12cagent/core/12.1.0.2.0/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.2.0
   

   
   Note that the plugins are not platform dependent in the sense that there is not a separate plugin for each Operating System. However, multi OS support was not available until the 111221 revision of the 12.1.0.1.0 database plugin (12.1.0.1.0 [u111221].)
   
   Periodically later versions of the database plugin (eg. 12.1.0.2.0) or later revisions of the database plugin are released
   
   PLUGIN REVISIONS
   
   A revision of a database plugin indicates that there is a new revised version of the *same* plugin. Revisions only affect target certifications, so there would not be a need to update an existing remote agent with the latest revision. The plugin revisions are needed if database monitoring for a particular platform newly becomes available, and the plugin revision adds support for that platform, so the latest revision plugin would be needed on the OMS. However, existing agents would not need this revision, because they would already have been installed on their own (different) platform. A plugin revision is shown in the console with the revision number in square brackets for example: 12.1.0.1.0 [u111221].
   
   SELF UPDATE
   
   These new versions/revisions of the database plugin are obtained via EM Cloud Control's 'Self Update' feature (setup/extensibility/self update). Self update automatically runs a job that regularly checks for updates. The cloud control administrator will be emailed automatically (if email notification has been set up) when any new plugins are available (via the incident rule "Event management Ruleset for Self Update". It is also possible to run an on-demand search of the latest updates by clicking on "check updates" this will run a job called "SELFUPDATE" which does a 'RefreshFromEMStore'.
   


2. Pre-requisites for database discovery
   
   
   1. There must be a 12c Management Agent (eg. 12.1.0.1.0 or 12.1.0.2.0) on the remote host (where the database target resides). Previous versions of the agent are not compatible with Enterprise Manager 12c. If there is no agent, it can be deployed using instructions from Section C.
   2. The database plugin which currently exists on the OMS must support the OS platform and version of database of the database to be discovered. Check this by:
      
      

      setup/extensibility/plugins
      click on the arrow to the left of the "Databases" folder and expand it
      Highlight "Oracle Database"
      Pick "Actions:Information"
      view "General" tab
      

      
      This will show the versions of the databases which the plugin can manage, and the OS versions which are certified. Note that the default database plugin (12.1.0.1.0 with no revisions) does not support any other platform than Linux. If the platform desired is not shown, also check the certification information on 'My Oracle Support', to clarify whether that platform is certified:-
      
      'My Oracle Support'/Certifications/Enterprise Manager Base Platform - Agent: version 12.1.0.1.0
      
      If the platform is certified, it may be that a later version or revision of the database plugin is needed. Multi platform support (other than Linux) was introduced in database plugin revision 12.1.0.1.0 [u111221].
      This can be obtained from Cloud Control itself via the 'self update' feature (Setup/Extensibility/Self Update). Self update will always indicate the latest version of a plugin. See point B3 for more info.
      
   
   
   3. If the current database plugin installed on the OMS does not support the required platform, either a later version database plugin, or a revision of the database plugin is needed. This new revision/version of the plugin will first need to be downloaded and deployed on the OMS.
      

   See our technical tip: "How to Deploy the Latest Database Plugin to the OMS and the Agent in 12C Cloud Control" on page 44

   
   
3. Getting the Agent
   
   
   1. If there is already an agent running on the host where the remote database is go to Section D
   2. If there is no agent on the remote host, the agent can be deployed to the remote host as follows:-
      
      

      setup/add target/add target manually./click on "add host target".
      

      
      After agent has been added, go to section D.
      
      

      NOTE: To see the results of the agent deployment (whether successful or not, go to setup/add target/add targets manually/click "add hosts results" button. Click on the hyperlink under "session name".)

   
   
   3. If unable to "add host target" because the "Agent software unavailable" message is shown for the necessary platform.
      
      The "agent software unavailable" message may be seen because by default the OMS will have the agent software for the platform the OMS was installed on.
      If the agent to be deployed is on a different platform than the OMS, then it is necessary to obtain the agent software for that platform.
      
      

      The agent software can be obtained from Enterprise Manager Cloud Control itself via self update. (Setup/Extensibility/Self Update/ under folder "Agent Software").
      

      
      Click on "download" to download the software. This runs a job called " DNLDEMUPDATEDNLDEMUPDATE (Download Update From Oracle)". After the download, the software should be automatically applied via the job (APPLYEMUPDATEAPPLYEMUPDATE). To check this, click on checkup/extensibility/self update/expand "agent software" folder.
      Look at the status (either "applied, available or apply failed".
      
      What do the Status mean:
      
      Available: means that the software for that agent is available to download from the EMStore at Oracle, but has not yet been downloaded or applied.
      Clicking on "download" will download the software to the OMS. The "download" action will automatically try to apply the software as part of the process.
      
      Applied: means that the software for that agent has been downloaded and applied (therefore it should be possible to install an agent on that particular platform)
      
      Apply failed: means that there was an attempt to download and apply that software to the OMS Software Library, but the apply failed.
      
      

      If "apply failed" is seen, highlight it, at the bottom of the screen the "past activities" pane will show and there will be a status against the operations "available/download and apply".
      If any operation has a "failed" (eg. apply:failed) click on ""details", which should drill down to the error message.
      

      See also our technical tips:
      "12c: Plugin Deployment Fails With Plug-in is not certified for the Operating System" on page 42
      "How to Deploy the Latest Database Plugin to the OMS and the Agent in 12C Cloud" on page 44
      

   
   
4. Discovering the Database
   
   Once the agent exists on the remote machine (where the databases reside) the database can be discovered in one of the following ways:-
   
   
   • Guided Process
   • Manual Discovery
   • Automatic Discovery (and "run discovery now")
   • Manual via EMCLI
   
   
   1. Guide Process
      
      Discovery via Guided Process is done via:-
      
      Setup/Add targets/Add targets manually
Add Non-Host Targets Using Guided Process (Also Adds Related Targets)
Target types:Oracle Database, Listener and Automatic Storage Management
Click "add using guided discovery"

      Or the guided process discovery can be done via:-
      
      targets/databases/add
choose hostname
click on 'continue'

      This will find all databases and their listeners on a particular host. The databases will be added to the console, once they have been configured by the user.
      'configuring' involves entering the password for the dbsnmp account, and then finishing the dialogue.
      
      
   2. Manual Discovery
      
      Alternatively a complete manual discovery can be done. This option adds only one database. It is called "manually", because the necessary configuration information (such as database name, port number etc)
      must be provided by the user. This can be useful when needing to add an individual database or listener, but if multiple databases need to be discovered it would be much quicker to use guided process or automatic discovery.
      
      

      Adding a database manually is discussed in detail in our technical tip "How To Manually Add Database Targets in 12C Cloud Control" on page 45. This note includes screen shots and troubleshooting tips.

      
      

      Note that if the database/listener/asm to be added will be the first database/listener/asm target to be discovered by that management agent, and that management agent has not previously had the database plugin manually deployed to it, then the database plugin will be automatically deployed as part of the add database/listener/asm process. The database plugin will be deployed to $ORACLE_HOME/plugins/oracle.sysman.db.agent.plugin_12.1.0.1.0 This means that the first time a database/listener or asm target is added on a new agent, there could be a small delay whilst the database plugin is deployed.

   
   
   3. Automatic Discovery of Target
      
      Discover the Databases
      
      By default when the OMS discovers a new agent only the host,the agent and the Oracle Home target are automatically shown in the Enterprise Manager 12c Cloud Control Console.
      The other targets (eg. database/listener/ASM) need to be discovered. To discover the targets it's necessary to do a 'Run Discovery Now'. To do this:-
      
      In 12.1.0.1.0
      
      go to setup/addtarget/configure autodiscovery
click on 'multitype target discovery on single host'
Highlight the "Agent host Name"
check that the discovery module "Oracle Database, Listener and Automatic Storage Management" shows as an "Enabled Discovery Module"
highlight host but do not click on the hyperlink> choose 'run discovery now'
Dialogue box:"are you sure you want to run target discovery on <hostname>"
click on "yes"

      in 12.1.0.2.0
      
      go to setup/add target/configure auto discovery
click on "all discovery modules"
Highlight the "Agent host Name"
Click on "configure"
Tick the "enabled" box, in order to enable the relevant discovery modules
(in the very least "Oracle Home Discovery" and "Oracle Database, Listener and Automatic Storage Management" is needed)
Say "OK" to save changes
<highlight host> choose 'run discovery now'
Dialogue box:"are you sure you want to run target discovery on <hostname>"
click on "yes"

      The "run discovery now" forces the agent to run its database discovery algorithm ($ORACLE_HOME/plugins/oracle.sysman.db.discovery.plugin_12.1.0.1.0/discover/oracledb.pl).
      This "run discovery now" achieves a similar aim as to running agentca -d or "add database" in the previous versions of Grid Control .
      Notice that the targets will still not show up in the console nor will they be added to the agent's targets.xml file ($ORACLE_HOME<ofagent>/agent_inst/sysman/emd/targets.xml)
      until they have been promoted in Cloud Control.
      
      Promote the Databases
      
      The databases which have been found by in 'autodiscovery results/non host target', and which the user wishes to monitor by cloud control need to be promoted before they will show up as database targets in the Console . To do this:-
      
      

      After the "run discovery now" has completed, click on "refresh".
      The "Discovered Targets" column should now have a number by it.
      Click on the hyperlink number eg. discovered targets "4", and this will drill down to the newly found targets. These targets can then be promoted/ignored etc
      

      
      

      TIP: Another way to find targets which have been discovered is: -setup/add target/auto discovery results/"non host target"
      

      
      To promote a target:-
      
      
      • Highlight thedatabase/listener/asm target (in the 'Auto Discovery Results screen')
      • Click on "promote"
      • For database targets, there is a "configure" icon, to enter the dbsnmp password details, and test the connect string.
      • After the database (and listener) have been promoted and configured successfully, they will be seen in the console.
      
      Any targets that need to be marked as 'ignored' can be highlighted and then click "ignore". (To multiple highlight to ignore multiple targets in one go, do "shift/highlight").
      
      

      NOTE: If the management agent does not have the database plugin deployed to it (eg. a newly discovered agent where the database plugin has not been previously manually deployed), and is automatically discovering a database/listener/asm target for the first time, the database plugin will be automatically deployed at the time when the first database/listener/asm target is successfully promoted. The database plugin will be deployed to $ORACLE_HOME<ofagent>/plugins/oracle.sysman.db.agent.plugin_12.1.0.1.0. This means that the first time a database/listener or asm target is promoted on a new agent, there could be a small delay whilst the database plugin is deployed.

      
      Note - there is no "mass promotion" concept for a target. This means that each target must be promoted individually. An internal Enhancement Request has been raised requesting to allow the feature
of some kind of mass promotion.

      Configure Auto-discovery Time Period
      
      After the initial discovery of the database/listener/ASM targets, the OMS will periodically check for any new database/listeners which have been added.
      The period in which the OMS checks for new databases is set by:
      
      

      "setup/add target/configure auto discovery/ Discovery Module/"Oracle Database, Listener and Automatic Storage Management"/configure".
      Or by "setup/add target/configure auto discovery/ Discovery Module:"Multi Target-Type Discovery on a Single Host"/configure.
      

      
      Again any targets which have been newly added should be detected and added to the auto discovery results. By default auto discovery runs every 1 day. Remember that auto-discovery results go to:-
      
      

      setup/add target/auto discovery results/"non host target"
      

      
      Has the database been discovered?
      
      If the Desired Database is not Shown in setup/add target/auto discovery results/"non host target" then the following should be checked:
      
      
      1. Perform a "run discovery now"
         
         go to setup/addtarget/configure autodiscovery
click on 'multitype target discovery on single host' (12.1.0.1.0) or 'all discovery modules' (12.1.0.2.0)
Brings you to "Target Discovery (Agent Based)" screen
<highlight host but do not click on the hyperlink> choose 'run discovery now'
this should bring up a dialogue box "are you sure you want to run target discovery on <hostname>?
click on "yes"
A "Run Discovery Now - In Progress" box should show on the screen.

         

         TIP: Make sure that the pop up box "are you sure you want to run target discovery on <hostname>?" is shown.
         If the hyperlink by the hostname is clicked by mistake, it does not run the discovery, instead it presents the screen to configure discovery options. If the "Run Discovery Now -In Progress" box is not shown, see step 2 below.
         

      
      
      2. Ensure that the necessary discovery modules are showing in Target Discovery (Agent Based) screen.
         
         go to setup/addtarget/configure autodiscovery
click on 'multitype target discovery on single host'
Brings you to "Target Discovery (Agent Based)" screen
<click on the hyperlink for the agent host name> (or highlight and choose "configure")
Tick the Discovery Modules which are required: eg.

Oracle Cluster and High Availability Service
Oracle Database, Listener and Automatic Storage Manager
Oracle Fusion Middleware
Oracle Home Discovery
OracleSecureBackup

         after enabling the discovery modules, perform a "Run Discovery Now"
         
      
      
      3. Check that the /etc/oratab or var/opt/oracle/oratab (Solaris) contains the entry for the database
      4. In the agent home, on the machine where the database to be discovered resides check:-
         
         $ORACLE_HOME/agent_inst/sysman/log/emagent_perl.trc
         
         This should show the DB discovery 'INFO' information eg.
         
         DB_LISTENER_DISCOVERY: ***** Start of Database and Listener Discovery***

         this should indicate whether the listener was found eg. this will show the version of the listener, ipaddress/hostname, the port etc.
         It is also possible to change the 'INFO' level to 'DEBUG' by changing the $ORACLE_HOME<of agent>/agent_inst/sysman/config/emd.properties
         parameter 'EMAGENT_PERL_TRACE_LEVEL=INFO' to 'EMAGENT_PERL_TRACE_LEVEL=DEBUG' and reloading the agent.
         
         

         NOTE: If the info/debug information does not show in the emagent_perl.trc try manually deploying the database plugin from the Extensibility/plugins.
         setup/extensibility/plugins expand "databases" folder
         highlight "oracle database"
         Actions: Deploy on Management Agent"
         <select the relevant agent>
         

      
      
      5. View the script oracledb.pl in $ORACLE_HOME/plugins/oracle.sysman.db.discovery.plugin_12.1.0.1.0/discover/oracledb.pl which contains the discovery algorithm.
         
         
   4. Manual Database Discovery via EMCLI
      
      Enterprise Manager Cloud Control 12c also offers a further discovery option, which is to use emcli to manually script database discovery.
      Emcli is a command line extension to Enterprise Manager Cloud Control. It can be installed on any machine. It does not necessarily have to be on the machine running Cloud Control.
      In this example it's installed on the agent machine, but it could also be installed on a windows pc (see our technical tip "How to Install the 12c Enterprise Manager Cloud Control EMCLI on a Windows PC" on page 46).
      EMCLI is downloaded from the OMS and is quick to set up.
      To use emcli to discover the database target, the following properties must be known:- Name,SID, hostname, dbsnmp account password, listener port, 'target type' and database oracle home path.
      
      

      How to find the values for: Name,SID, hostname, dbsnmp account password, listener port, 'target type' and database oracle home path for the database target.
      
      - "Name" is the name which will be given to the database target in Cloud Control. (it can be anything, does not have to correspond with the actual target name)
      - SID can be determined from sql*plus command "show parameter instance_name"
      - Hostname can be determined from uname -a. Then nslookup <name returned from uname -a> to get the fully qualified hostname.
      - Dbsnmp account password can be checked by logging into the database as 'dbsnmp'. If password is unknown it can be changed at sql*plus using "alter user dbsnmp identified by <newpassword>"
      - Listener port can be determined from $TNS_ADMIN/listener.ora or $ORACLE_HOME/network/admin/listener.ora (if $TNS_ADMIN is not set)
      - Database oracle home path can be determined from: /etc/oratab or /var/opt/oracle/oratab
      - Target type for the database is: oracle_database (but for the other targets could be: oracle_listener,osm_instance,rac_database,osm_cluster)
      

      
      The emcli add_target command takes the form:-
      
      emcli add_target
-name="name"
-type="type"
-host="hostname"
[-properties="pname1:pval1;pname2:pval2;..."]
[-separator=properties="sep_string"]
[-subseparator=properties="subsep_string"]
[-credentials="userpropname:username;pwdpropnamepwdpropname:password;..."]
[-input_file="parameter_tag:file_path"]
[-display_name="display name"]
[-groups="groupname1:grouptype1;groupname2:grouptype2;..."]
[-timezone_region="gmt_offset"]
[-monitor_mode="monitor_mode"]
[-instances="rac_database_instance_target_name1:target_type1;..."]

      [ ] indicates that the parameter is optional
      
      An example of the emcli command to add a database is:-
      
      ./emcli add_target -name="<NametoAppearinConsole>" -type="oracle_database" -host="<host_name>" -credentials="UserName:dbsnmp;password:<dbsnmp_account_password>;Role:Normal" -properties="SID:<sid>;Port:<port>;OracleHome:<path_of_oracle_home>;MachineName:<hostname>"
      
      for example:-
      
      ./emcli add_target -name="sally" -type="oracle_database" -host="machine1.uk.oracle.com" -credentials="UserName:dbsnmp;password:dbsnmp;Role:Normal" -properties="SID:sally;Port:1528;OracleHome:/u01/orabase/11db;MachineName:machine1.uk.oracle.com"

      Listeners can be added by:-
      
      emcli add_target -name="<listenerName>" -type="oracle_listener" -host="<hostname>" -properties="LsnrName:<listenerName>;ListenerOraDir:<path to listener.ora file>;Port:<port>;OracleHome:<path_to_oracle_home>;Machine:<hostname>"
      
      for example:
      
      emcli add_target -name="LISTENERTEST" -type="oracle_listener" -host="machine1.uk.oracle.com" -properties="LsnrName:LISTENERTEST;ListenerOraDir:/u01/orabase/11107dbase/network/admin;Port:1525;OracleHome:/u01/orabase/11107dbase;Machine:machine1.uk.oracle.com"

      These commands can be put in a script and run - eg.
      
      [oracle@rachvm1 emcliKit]$ pwd
[oracle@rachvm1 emcliKit]$ /u01/emcliKit

vi adddatabase.sh
contents of "adddatabase.sh" are:-

emcli login -username=sysman -password=oracle12
emcli add_target -name="sally" -type="oracle_database" -host="machine1.uk.oracle.com" -credentials="UserName:dbsnmp;password:dbsnmp;Role:Normal" -properties="SID:sally;Port:1528;OracleHome:/u01/orabase/11db;MachineName:machine1.uk.oracle.com"
emcli add_target -name="test" -type="oracle_database" -host="machine1.uk.oracle.com" -credentials="UserName:dbsnmp;password:dbsnmp;Role:Normal" -properties="SID:test;Port:1525;OracleHome:/u01/orabase/1107dbase;MachineName:machine1.uk.oracle.com"
emcli add_target -name="LISTENERTEST" -type="oracle_listener" -host="machine1.uk.oracle.com" -properties="LsnrName:LISTENERTEST;ListenerOraDir:/u01/orabase/11107dbase/network/admin;Port:1525;OracleHome:/u01/orabase/11107dbase;Machine:machine1.uk.oracle.com"
emcli logout

[oracle@rachvm1 emcliKit]$ chmod +x adddatabase.sh
[oracle@rachvm1 emcliKit]$ . adddatabase.sh

Output is shown on the command line:-

Login successful

Target "sally:oracle_database" added successfully
Target "test:oracle_database" added successfully
Target "LISTENERTEST:oracle_listener" added successfully
Logout successful
[oracle@rachvm1 emcliKit]$

      If a database has been added via emcli, it will straight away be monitored by Cloud Control, this means it will be seen as a target in the console straightaway and will not need to be promoted. This is because the dbsnmp details have been provided in the emcli input.
      At this moment in time, there is no emcli promote command (for targets that have been auto discovered).
      
      

      NOTE: If the agent is a management agent without the database plugin (eg. a newly discovered agent without the database plugin manually deployed to it which has never discovered any targets of type database/listener or asm), and emcli is used to add the first database/listener or asm target, the database plugin will be automatically deployed as part of the discovery process. The database plugin will be deployed to $ORACLE_HOME<ofagent>/plugins/oracle.sysman.db.agent.plugin_12.1.0.1.0

12c Cloud Control: Steps to Locate and Manage the Various Logs/Trace files in a 12c OMS Installation

Modified 12-FEB-2013

Applies to

Enterprise Manager for Oracle Database - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Purpose

This document lists and describes the usage of various log/trace files in the 12c Cloud Control OMS home and steps to control their size/tracing levels.

For details of the directory structure and useful locations in the 12c OMS home, refer to our technical tip:
"12c Cloud Control: Details of the Directory Structure and Commonly Used Locations in a 12c OMS Installation" on page 19

For a description of the log/trace files updated specifically during the OMS startup, refer to our technical tip:
"12c Cloud Control: Understanding OMS Process Control (Start/Stop/Status)" on page 46

Scope

Enterprise Manager Cloud Control Administrators who need to familiarize themselves with the various log/trace files within a 12c OMS home and know how to manage these files.

Details

OMS (emgc and empbs) Log/Trace Files

Logging and Tracing Directory:

OMS logging and tracing information is written to files located in the <EM_INSTANCE_BASE>/em/<OMS_NAME>/sysman/log directory.
For example: /home/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log/

Where, <EM_INSTANCE_BASE> is the OMS Instance directory. This is named gc_inst by default and is present in the Middleware Home.
Description of the OMS log/trace files created:

1. These settings are case sensitive and the values must be entered in uppercase.
2. The <EM_INSTANCE_BASE>/em/<OMS_NAME>/sysman/config/emomslogging.properties file is no longer used to configure the tracing/logging parameters.
   The parameters can be set or modified using the 'emctl set property' command.
3. After modifying any of the log or trace parameters, it is not necessary to re-start the OMS. The change is made effective at run-time itself.
4. To modify the trace level to DEBUG, execute:
   
   cd <OMS_HOME>/bin
emctl set property -name log4j.rootCategory -value 'DEBUG, emlogAppender, emtrcAppender' -module logging

   - The above command will prompt you to confirm the change by displaying the below message:
   
   Setting log4j.rootCategory to "DEBUG, emlogAppender, emtrcAppender"
has performance impact on OMS due to heavy logging.
It is recommended to use category specific logging instead.
Please consult My Oracle Support Note 1330898.1 for additional information.
Do you want to continue? (yes/no):

   Type 'yes' or 'y' to proceed with the change.
   To cancel the change, type 'no'.
   
   - If you type 'yes' or 'y', the command will further prompt you for the password of the sysman user.
   - If the property change is successful, the following output will be returned on the terminal:
   
   Property log4j.rootCategory for oms omsmachine.domain:4889_Management_Service has been set to value DEBUG, emlogAppender, emtrcAppender
OMS restart is not required to reflect the new property value



5. To view the current tracing level:
   
   
   1. Query the value for OMS Property: log4j.rootCategory
      
      cd <OMS_HOME>/bin
emctl get property -name log4j.rootCategory
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
SYSMAN password:
Value for property log4j.rootCategory for oms smnath-lnx.idc.oracle.com:4890_Management_Service is DEBUG, emlogAppender, emtrcAppender

   
   OR
   
   
   2. List all the properties and find the value for log4j.rootCategory:
      
      cd <OMS_HOME>/bin
emctl list properties

      In the output, look for property:
      
      log4j.rootCategory=DEBUG, emlogAppender, emtrcAppender


For more details on setting / viewing / deleting OMS property values in 12c, refer to our technical note "12c Cloud Control : How To View and Update OMS Configuration Properties Using 'emctl' utility?" on page 33.

You can also enable DEBUG level only for a particular subsystem / category. For details, refer to our technical note "Impact of Setting Debug Mode for OMS and Steps to Enable Debug for Particular Subsystems" on page 32.

1. Oracle HTTP Server (OHS) Logs
   
   The logs are stored at <EM_INSTANCE_BASE>/<webtier_instance_name>/diagnostics/logs/OHS/<ohs_name>.
   For example: /home/oracle/Middleware/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1
   
   

   File	Usage
   access_log	This records information for all client requests processed by the HTTP server. Each entry contains the following information: Host name Remote log name Remote user and time Request HTTP Response code Number of transferred bytes The HTTP Response code is useful in identifying whether the client request was successful or not.
   ohs1.log	This is the error log for the OHS.
   console~OHS~1.log	This is the Console log for the OHS and is generated by the OPMN. It records the stdout and stderr messages and is useful for troubleshooting OHS startup issues.
   mod_wl_ohs.log	This is the log file for the WLS plugin (mod_wl_ohs) for OHS. This plugin allows requests to be proxied from OHS to the Oracle WebLogic Server.
   em_upload_http_access_log	Contains entries for Agent uploads in http mode.
   em_upload_https_access_log	Contains entries for Agent uploads in https mode.



2. OPMN Logs
   
   The logs are stored at <EM_INSTANCE_BASE>/<webtier_instance_name>/diagnostics/logs/OPMN/<opmn_name>
   For example: /home/oracle/Middleware/gc_inst/WebTierIH1/diagnostics/logs/OPMN/opmn1
   
   

   File	Usage
   opmn.out	This contains the standard output (stdout) and standard error (stderr) log entries of OPMN and is also referred to as the OPMN "console log". After a certain point in the OPMN initialization, nothing else is written to this file. Only a small set of messages ever appears in this file; therefore, this file may not be present or may be of 0 bytes.
   opmn.log	This tracks command execution and operation progress and contains messages useful for monitoring the OPMN server operations. Output written to the opmn.log file contains the exit status of a child OPMN process. A status code of 4 indicates a normal reload of OPMN. All other status codes indicate an abnormal termination of the child OPMN process.
   debug.log	This contains OPMN debug log messages (English only), which track command execution and operation progress for Oracle Notification Service (ONS) and Process Manager(PM). The PM portion of OPMN generates and outputs the error messages in this file.



3. EM NodeManager Logs
   
   The log is located at <EM_INSTANCE_BASE>/NodeManager/emnodemanager
   
   For example:
   /home/oracle/Middleware/gc_inst/NodeManager/emnodemanager
   
   

   File	Usage	Rotation Policy
   nodemanager.log	Main log file for the nodemanager and has entries for activities performed by the nodemanager. There is also a nodemanager.log.lck file which is a lock file for the nodemanager and will be present as long as the nodemanager is up and running.	By default, there is no rotation policy as this file is not expected to grow too big. If needed, below parameters can be set in the <EM_INSTANCE_BASE>/NodeManager/emnodemanager/nodemanager.properties: LogLimit:Maximum size of the Node Manager Log specified as an integer. When this limit is reached, a new log file is started. By default, this is set to Unlimited. LogCount:Maximum number of log files to create when LogLimit is exceeded. By default, this is set to 1.



4. Adminserver Logs
   
   The logs are located at <EM_INSTANCE_BASE>/user_projects/domains/<domain_name>/servers/<ADMIN_SERVER_NAME>/logs
   
   For example:
   /home/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/
   
   

   File	Usage	Rotation Policy
   <ADMIN_SERVER_NAME>.log or EMGC_ADMINSERVER.log	The EMGC_ADMINSERVER writes all messages from its subsystems and applications to this logfile.	The filesize is restricted to 5MB, after which it is rotated but the older logs are not deleted automatically. The log level is WARNING, by default.
   <ADMIN_SERVER_NAME>.out or EMGC_ADMINSERVER.out	sysout and syserr messages are saved in this file by the Nodemanager.	These files cannot be automatically rotated by size or time. They are rotated by the nodemanager process only at the time of the Admin server start, so older files need to be manually purged.
   access.log	Contains details of HTTP requests processed by this server.	The filesize is restricted to 5MB, after which it is rotated but the older logs are not deleted automatically.
   <domain_name>.log or GCDomain.log	The domain log file provides a central location from which to view the overall status of the domain. In addition to writing messages to its own log file, each server instance also forwards a subset of its messages to this domain-wide log file. By default, servers forward only messages of severity level of NOTICE or higher.	The filesize is restricted to 5MB, after which it is rotated but the older logs are not deleted automatically.
   <SERVER_NAME>-diagnostic.log or EMGC_ADMINSERVER-diagnostic.log	Each server instance in a domain writes security-related (OPSS-based) errors raised by its subsystems and applications to a server log file named: ServerName-diagnostic.logn. Server-related security errors, such as exceptions raised by issues with a subject or principal, and errors that may occur while migrating or reassociating domain security data get written in the administration server diagnostic log. This log uses ODL formatting and the logging configuration is specified in the <EM_INSTANCE_BASE>/user_projects/domains/GCDomain/config/ fmwconfig/servers/EMGC_ADMINSERVER/logging.xml	The log files are rotated when they reach 10 MB and the maximum size of all log files is 100 MB i.e the total number of files is restricted to 10.





For enabling automatic purge policy for the older GCDomain.log, EMGC_ADMINSERVER.log and access.log files, refer to technical note "12c Cloud Control: How to Enable Log Rotation Policy to Automatically Delete Older GCDomain.log, EMGC_ADMINSERVER.log and access.log Files?" on page 15.



5. EM Managed Server Logs
   
   The logs are located at <EM_INSTANCE_BASE>/user_projects/domains/<domain_name>/servers/<SERVER_NAME>/logs
   
   For example: /home/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.log
   
   

   File	Usage	Rotation Policy
   <SERVER_NAME>.log or EMGC_OMS1.log	The EMGC_OMSn instance writes all messages from its subsystems and applications to this logfile.	The number of files are restricted to 10, which are rotated at default size of 5MB. The log level is WARNING, by default./td>
   <SERVER_NAME>.out or EMGC_OMS1.out	The messages written to sysout and syserr are saved in this file by the nodemanager.	These files cannot be automatically rotated by size or time.They are rotated by the nodemanager only at the time of Managed server start, so older files need to be manually purged.
   access.log	Contains details of HTTP requests processed by this server.	The number of files are restricted to 10, which are rotated at default size of 5MB.
   <SERVER_NAME>-diagnostic.log or EMGC_OMS1-diagnostic.log	Application-related security errors, such as exceptions raised by application-specific policies or credentials, get written in this managed server diagnostic log. This log uses ODL formatting and the logging configuration is specified in the <EM_INSTANCE_BASE>/user_projects/ domains/GCDomain/config/fmwconfig/servers/EMGC_OMSn/logging.xml file.	The log files are rotated when they reach 10 MB and the maximum size of all log files is 100 MB i.e the total number of files is restricted to 10.

For a list of WLS log files which need to be manually purged periodically, refer to our technical tip "12c Cloud Control: Which WLS Log Files Can be Removed/Purged Manually at Regular Intervals in 12c OMS Installation for Space Considerations?" on page 4.

1. Installation: <oraInventory>/logs
2. EMPrereqkit:<oraInventory>/logs/emdbprereqs/LATEST/repository.log or emprereqkit.log
3. Configuration Tools: <OMS_HOME>/cfgtoollogs
4. Repository Creation: <OMS_HOME>/sysman/log/schemamanager

12c Cloud Control: How to Enable Log Rotation Policy to Automatically Delete Older GCDomain.log, EMGC_ADMINSERVER.log and access.log Files?

Modified 18-APR-2012

Applies to

Enterprise Manager for Oracle Database - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

In a 12c OMS home, the GCDomain.log, EMGC_ADMINSERVER.log and access.log files located in the directory:
<EM_INSTANCE_HOME>/user_projects/domains/<domain_name>/servers/EMGC_ADMINSERVER/logs

For example: /home/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs)

are rotated once they reach 5MB in size. But the older logs are not automatically deleted and need to be manually purged at regular intervals.
This document explains how the rotation policy can be modified so that the older logs can be deleted automatically without any manual intervention.

Fix

• Open the <OMS_HOME>/install/setupinfo.txt file and identify the "Admin Server URL", for example:

Admin Server URL: https://omsmachine.domain:7102/console

• From a browser, login to the WLS Admin Server Console as the 'weblogic' user.
• In the 'Change Center' click on the 'Lock & Edit' button.


• For GCDomain.log:
  
  
  • Under the 'Domain Structure', click on GCDomain → Logging.
    In the page shown, enable the option 'Limit number of retained files' as shown in the screenshot below:
  
  

  

  
  
  • Click on the 'Save' button at the bottom of the page to save the settings.

Once the changes are activated, the number of GCDomain.logn files are restricted to 7, which are rotated at default size of 5MB.


• For EMGC_ADMINSERVER.log:
  
  
  • Under the 'Domain Structure', expand 'Environment' and click on 'Servers'. Click on the EMGC_ADMINSERVER(admin)
    and navigate to the 'Logging' tab and then 'General' subatab.
  • In the page shown, enable the option 'Limit number of retained files' as shown in the screenshot below:
  
  

  

  
  
  • Click on the 'Save' button at the bottom of the page to save the settings.

Once the changes are activated, the number of EMGC_ADMINSERVER.logn files are restricted to 10, which are rotated at default size of 5MB.


• For Admin Server access.log:
  
  
  • Under the 'Domain Structure', expand 'Environment' and click on 'Servers'. Click on the EMGC_ADMINSERVER(admin) and navigate to the 'Logging' tab and then 'HTTP' subatab.
  • In the page shown, enable the option 'Limit number of retained files' as shown in the screenshot below:
  
  

  

  
  
  • Click on the 'Save' button at the bottom of the page to save the settings.
  
  Once the changes are activated, the number of access.logn files are restricted to 10, which are rotated at default size of 5MB.
  
  
• After saving the above, activate the settings by clicking on 'Activate Changes' in the 'Change Center'.

NOTE:
The older logs are not immediately purged after the above configuration change. When the existing GCDomain.log / EMGC_ADMINSERVER.log / access.log reaches 5MB in size,
Weblogic will automatically rollover the file. At this time, the number of files configured to be retained are maintained and the older set of files are deleted automatically.

How to Determine the List of Patch Set Update(PSU) Applied to the Enterprise Manager OMS and Agent Oracle Homes?

Modified 20-DEC-2012

Applies to

Enterprise Manager Base Platform - Version 10.2.0.5 to 11.1.0.1 [Release 10.2 to 11.1]
Information in this document applies to any platform.

Goal

Applying a Patch Set Update (PSU) to the OMS or Agent Oracle_Home changes the value of the fifth digit in the complete version number of the product => 'x' in the format number below:

10.2.0.5.x with possible values for x : 0,1,2,3,4,5,6
11.1.0.1.x with possible values for x : 0,1,2,3,4,5,6,7
12.1.0.2.x with possible values for x : 0,1

After applying a PSU, the value of 'x' increases but this new value is not reflected:

1. in the banners shown in the emctl command:
   
   emctl status agent
   or
   emctl getversion
   
   For example:
   
   - Version of the Agent and OMS remain unchanged in the emctl status agent output below
   
   cd <AGENT_HOME>/bin
emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0

   - The 'emctl getversion' executed in the OMS / Agent home does not indicate whether a PSU patch is applied or not.
   The output only displays the 4-digit version, for example:
   
   cd <OMS_HOME>/bin
emctl getversion
Oracle Enterprise Manager 10g Release 5 Grid Control
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
--- Standalone agent
Enterprise Manager 10g OMS Version 10.2.0.5.0
Enterprise Manager 10g AS Control Version 10.2.0.0
Enterprise Manager 10g Agent Version 10.2.0.5.0

   or
   
   cd <OMS_HOME>/bin
./emctl getversion
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
--- Standalone agent
Enterprise Manager 11g OMS Version 11.1.0.1.0
Enterprise Manager 11g Agent Version 11.1.0.1.0



2. in the version number of components shown in the output of 'opatch lsinventory -detail'
   
   For example - below output is form an OMS home where PSU2 Patch:10270073 is applied but the Grid Control version still remains 11.1.0.1.0:
   
   Invoking Patch 11.1.0.8.3

Oracle Home : /ora/product/oracle/11.1.1/oem/oms11g
Central Inventory : /ora/product/oraInventory from : /etc/oraInst.loc
Installed Top-level Products
Oracle Enterprise Manager Grid Console 11.1.0.1.0
...
Enterprise Manager Common Core Files 11.1.0.1.0
Enterprise Manager Common Files 11.1.0.1.0
...
Enterprise Manager Grid Control Core Files 11.1.0.1.0
Enterprise Manager Repository 11.1.0.1.0
Enterprise Manager Repository Core Files 11.1.0.1.0
...
Interim patches (15) :

Patch 10270073 : applied on Thu Jul 24 17:11:21 CET 2011
Unique Patch ID: 13349102
Created on 30 Nov 2010, 15:34:49 hrs PST8PDT


Fix

A PSU is identified with an unique patch number and for the OMS / Agent homes and the PSU number applied is visible only in the inventory. The list of PSU patches applied to the OMS and Agent homes can be identified using the 'opatch lsinventory' output, as per the below steps.:

Solution A

1. For the OMS HOME
   
   - Set the ORACLE_HOME to the OMS installation location:
   
   On Unix:
   
   $ export ORACLE_HOME=<path of the oms10g or oms11g location>
   
   For example:
   
   $ export ORACLE_HOME=/u01/OracleHomes/oms10g
$ export ORACLE_HOME=/u01/Middleware/oms11g

   On Windows:
   
   > set ORACLE_HOME=<path of the oms10g or oms11g location>
   
   For example:
   
   > set ORACLE_HOME=C:\OracleHomes\oms10g
> set ORACLE_HOME=C:\Middleware\oms11g

   - Execute the following commands:
   
   cd <OMS_HOME>/OPatch
opatch lsinventory -bugs_fixed | grep -i 'ENTERPRISE MANAGER OMS' | grep -i 'PSU'

   For example, the output in a 10.2.0.5 OMS home with the PSU5 patch will be:
   
   9786002 10168579 Mon Aug 01 09:24:20 IST 2011 enterprise manager oms 10.2.0.5.4 psu
10168579 10168579 Mon Aug 01 09:24:20 IST 2011 enterprise manager oms 10.2.0.5.5 psu
9174904 9702579 Tue Nov 30 20:04:01 IST 2010 ENTERPRISE MANAGER OMS 10.2.0.5.2 PSU
9282397 9702579 Tue Nov 30 20:04:01 IST 2010 ENTERPRISE MANAGER OMS 10.2.0.5.3 PSU
9702579 9702579 Tue Nov 30 20:04:01 IST 2010 ENTERPRISE MANAGER OMS 10.2.0.5.4 PSU EM CONTENT

   In a 11g OMS home with a PSU4 patch, the output will be:
   
   12423703 12423703 Wed Sep 07 09:48:36 IST 2011 ENTERPRISE MANAGER OMS 11.1.0.1.4 GC PSU (GENERIC)

   

   NOTE: Due to an internal bug, the above command does not display the PSU1, PSU2 and PSU3 patches applied to a 11g OMS home. The issue was fixed in 11g PSU4 to display the correct output.
   



2. For the Agent HOME
   
   - Set the ORACLE_HOME to the Agent installation location:
   
   On Unix:
   
   $ export ORACLE_HOME=<path of the agent10g or agent11g location>
   
   For example:
   
   $ export ORACLE_HOME=/u01/OracleHomes/agent10g
$ export ORACLE_HOME=/u01/Middleware/agent11g

   On Windows:
   
   > set ORACLE_HOME=<path of the agent10g or agent11g location>
   
   For example:
   
   > set ORACLE_HOME=C:\OracleHomes\agent10g
> set ORACLE_HOME=C:\Middleware\agent11g

   - Execute the following commands:
   
   cd <AGENT_HOME>/OPatch
opatch lsinventory -bugs_fixed | grep -i 'ENTERPRISE MANAGER AGENT'| grep -i 'PSU' | grep -i 'ENTERPRISE MANAGER AGENT'| grep -i 'PSU'

   Agent 10.2.0.5
   
   For example, the output in a 10.2.0.5 Agent home with the PSU3 patch will be:
   
   9162498 9282414 Tue Nov 30 19:55:11 IST 2010 enterprise manager agent 10.2.0.5.2 psu
9282414 9282414 Tue Nov 30 19:55:11 IST 2010 ENTERPRISE MANAGER AGENT 10.2.0.5.3 PSU

   Agent 11g
   
   Important : After applying PSU4 in a 11g AGENT_HOME, the command :
   
   ./opatch lsinventory -bugs_fixed |grep PSU

does not return any output.
   
   In the same AGENT_HOME, after applying PSU5, the output is shown correctly.
   
   $ ./opatch lsinventory -bugs_fixed |grep PSU
9345906 9345921 Fri Dec 02 16:51:46 IST 2011 ENTERPRISE MANAGER AGENT 11.1.0.1.3 GC PSU (UNIX)
9345913 9345921 Fri Dec 02 16:51:46 IST 2011 ENTERPRISE MANAGER AGENT 11.1.0.1.4 PSU
9345921 9345921 Fri Dec 02 16:51:46 IST 2011 ENTERPRISE MANAGER AGENT 11.1.0.1.5 PSU
10273607 9345921 Fri Dec 02 16:51:46 IST 2011 ENTERPRISE MANAGER AGENT 11.1.0.1.2 GC PSU (GENERIC)

   The same problem can be seen after applying PSU7. Solution here is to adapt the opatch command as follows
   
   Command which does not show correct output:
   
   <-AGENT-[]> /home/oracle
[oracle@ ~]$ opatch lsinventory -bugs_fixed | grep -i 'ENTERPRISE MANAGER AGENT'| grep -i 'PSU' | grep -i 'ENTERPRISE MANAGER AGENT'| grep -i 'PSU'
9345906 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.3 GC PSU (UNIX)
9345913 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.4 PSU
9345921 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.5 PSU
9346243 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.6 PSU
10273607 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.2 GC PSU (GENERI

   Command that does show the correct PSU7 version installed:
   
   <-AGENT-[]> /home/oracle
[oracle@lpgrid001a ~]$ opatch lsinventory -bugs_fixed | grep -i 'PSU'
9345906 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.3 GC PSU (UNIX)
9345913 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.4 PSU
9345921 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.5 PSU
9346243 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.6 PSU
11.1.0.1.7 PSU (UNIX)
10273607 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.2 GC PSU (GENERI




Solution B

When the above opatch command described in Solution A does not provide the list of PSU patches directly, compare the list of patches seen in the 'opatch lsinventory -detail' command with the table below and identify the PSU patch manually:



List of OMS and Agent PSU Patches



NOTE: When a cell is empty, that means that the patch is not applicable/available.





10.2.0.5	OMS	AGENT Unix	AGENT Windows
PSU1 (10.2.0.5.1)	8864918	-	-
PSU1 (10.2.0.5.2)	9174904	9162498	9223116
PSU1 (10.2.0.5.3)	9282397	9282414	9490898
PSU1 (10.2.0.5.4)	9786002	-	-
PSU1 (10.2.0.5.5)	10168579	-	-
PSU1 (10.2.0.5.6)	13701923	-	-
PSU1 (10.2.0.5.7)	14766621	-	-




11.1.0.1.1	OMS	AGENT Unix	AGENT Windows
PSU1 (11.1.0.1.1)	10065631	-	-
PSU1 (11.1.0.1.2)	10270073	10273607	10273607
PSU1 (11.1.0.1.3)	11727299	9345906	11778791
PSU1 (11.1.0.1.4)	12423703	9345913	12423714
PSU1 (11.1.0.1.5)	12833678	9345921	12833724
PSU1 (11.1.0.1.6)	13248190	9346243	13248202
PSU1 (11.1.0.1.7)	13711705	9346282	13711732
PSU1 (11.1.0.1.8)	14766609	-	-




12.1.0.2.1	OMS	AGENT Unix	AGENT Windows
PSU1 (12.1.0.2.1)	14840279	-	-

Enterprise Manager Cloud Control 12C, Target Database Instance Remains in Pending Status Indefinitely

Modified 09-AUG-2012

Applies to

Enterprise Manager for Oracle Database - Version 12.1.0.1.0 and later
Information in this document applies to any platform.


Symptoms

The Database Instance appears in pending status from Enterprise Manager Cloud Control 12c for an undetermined period of time, while it is actually up and running. The agent is started and is uploading correctly. This may happen for other target types as well (like cluster, ASM, etc...)

Cause

The response metric timed out for that particular database target, thus it was moved to a suspended state and the target appears as pending in EM console.

Solution

1. Stop the EM agent
2. Remove the SUSPENSION files in <AGENT-INST>/sysman/emd/state/statemgmt/oracle_emd/agent/SUSPENSION directory. For example:
   
   $ pwd
/home/oracle/agent12c/agent_inst/sysman/emd/state/statemgmt/oracle_emd/agent/SUSPENSION
ls -ltr
-rw-r----- 1 oracle users 11 Apr 5 11:45 oracle_database:emrep12c.us.oracle.com:Response

rm -rf oracle_database:emrep12c.fr.oracle.com:Response



3. Start back the Agent
   
   

   NOTE: if the 12c EM agent has BP1 applied, it would be enough to just restart the EM agent to clear the suspended metrics.
   

   
   To avoid this condition, the following options can be used:
   
   
   • Increase the Response metric schedule (from the default 15 seconds to 1 minute)
   
   or
   
   
   • add the following to emd.properties:
     
     _canceledThreadWait=210
     
   
   

   NOTE: another cause for this behavior is a left-over blackout that ended on the OMS side, but did not ended on the agent side (was not cleanly removed). To identify this condition as well, check if the <AGENT-INST>/sysman/emd/blackouts.xml contains anything more than the default "<Blackouts/>", remove the file and restart the EM agent.

How to Disable Automatic Statistics Collection in 11g

Modified 02-SEP-2012

Applies to

Oracle Server - Enterprise Edition - Version 11.1.0.6 and later
Information in this document applies to any platform.

Goal

How to disable automatic statistics collection in 11g.

Fix

The command to disable automatic optimizer statistics collection is:

BEGIN
DBMS_AUTO_TASK_ADMIN.DISABLE(
client_name => 'auto optimizer stats collection',
operation => NULL,
window_name => NULL);
END;
/

This can be verified with:

SQL> select client_name,status from DBA_AUTOTASK_CLIENT;



                      CLIENT_NAME                        STATUS
                      ---------------------------------- --------
                      auto optimizer stats collection    DISABLED
                      auto space advisor                 ENABLED
                      sql tuning advisor                 ENABLED
                      

For more details see our next technical tip on this same page

How Can We Find Out Status Of Task 'Auto Optimizer Stats Collection'

Modified 01-OCT-2012

Applies to

Oracle Server - Enterprise Edition - Version 11.1.0.6 to 11.2.0.3 [Release 11.1 to 11.2]
Information in this document applies to any platform.

Goal

Explain how to find the status of 'GATHER_STATS_JOB' or in fact 'auto optimizer stats collection' in 11g.

Fix

11g

In 11g, the automated maintenance task that collects statistics is called 'auto optimizer stats collection'. You can determine its' status by selecting from DBA_AUTOTASK_CLIENT:

SELECT CLIENT_NAME,STATUS
FROM DBA_AUTOTASK_CLIENT
WHERE CLIENT_NAME = 'auto optimizer stats collection'
/



                      CLIENT_NAME                                                      STATUS
                      ---------------------------------------------------------------- --------
                      auto optimizer stats collection                                  ENABLED
                      

"DBA_AUTOTASK_TASK and DBA_AUTOTASK_CLIENT Shows Different Status For Auto Optimizer Stats Collection" on page 18.

                      JOB_NAME          PROGRAM_NAME        SCHEDULE_NAME               JOB_CLASS               ENABLED
                      --------          --------            --------                    --------                --------
                      GATHER_STATS_JOB  GATHER_STATS_PROG   MAINTENANCE_WINDOW_GROUP    AUTO_TASKS_JOB_CLASS    TRUE
                      

NOTE: If the 10g method is used on 11g, then it does not return any information as the implementation is quite different in 11g:

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production
With the Partitioning and Real Application Testing options

SQL> SELECT job_name,job_type,program_name,schedule_name,job_class
FROM dba_scheduler_jobs
WHERE job_name = 'GATHER_STATS_JOB';

no rows selected


DBA_AUTOTASK_TASK and DBA_AUTOTASK_CLIENT Shows Different Status For Auto Optimizer Stats Collection

Modified 12-MAR-2012

Applies to

Oracle Server - Enterprise Edition - Version: 11.1.0.6 to 11.2.0.3 - Release: 11.1 to 11.2
Information in this document applies to any platform.

Symptoms

We are Trying to Disable the Auto Optimizer Stats Collection in 11G,DBA_AUTOTASK_CLIENT shows that the 'auto optimizer stats collection' is disabled.
But DBA_AUTOTASK_TASK shows that the Task is Enabled.

BEGIN
DBMS_AUTO_TASK_ADMIN.DISABLE(
client_name => 'auto optimizer stats collection',
operation => NULL,
window_name => NULL);
END;
/
PL/SQL procedure successfully completed.

Checking the Status from DBA_AUTOTASK_TASK & DBA_AUTOTASK_CLIENT

SQL> select client_name,status from DBA_AUTOTASK_TASK;



                      CLIENT_NAME                       STATUS 
                      -------------------------------   -------- 
                      auto optimizer stats collection   ENABLED  <=== 
                      auto space advisor                ENABLED 
                      sql tuning advisor                ENABLED  
                      

                      CLIENT_NAME                       STATUS 
                      --------------------------------  -------- 
                      auto optimizer stats collection   DISABLED  <=== 
                      auto space advisor                ENABLED 
                      sql tuning advisor                ENABLED
                      

Cause

This is expected Behaviour. The views assume that there is a one to many relationship between CLIENTS and TASKS.
Task can be used by different/multiple client. So even though if we disable the client, the DBA_AUTOTASK_TASK may still show the status as enabled.
In current version, the TASKS has only one CLIENT. But in future version of oracle, the TASKS can have multiple CLIENTS so the status in DBA_AUTOTASK_TASK will show as ENABLED.
So the correct way to check the status is through DBA_AUTOTASK_CLIENT.

Solution

Use DBA_AUTOTASK_CLIENT to check the status.

SQL> select client_name,status from DBA_AUTOTASK_CLIENT;


Why Auto Optimizer Statistics Collection May Appear to be "Stuck"?

Modified 12-FEB-2013

Applies to

Oracle Server - Enterprise Edition - Version: 11.1.0.7 and later
Information in this document applies to any platform.

Symptoms

• The 'auto optimizer stats collection' task is enabled in auto task
• STATISTICS_LEVEL has already been set to TYPICAL or FULL
• DBA_AUTOTASK_CLIENT_HISTORY is empty
• Statistics on tables are stale.
• Some scheduler window is active and LAST_START_DATE is several days ago.

                      SELECT window_name,last_start_date,enabled,active

                      FROM DBA_SCHEDULER_WINDOWS;

                      

                      WINDOW_NAME       LAST_START_DATE                     ENABLE  ACTIVE
                      ---------------   -----------------------------       ------  ------
                      SATURDAY_WINDOW   26-Mar-11 06.00.00.019249 am +08:00 TRUE    TRUE 
                      

• In Alert log, there is a shutdown immediate after the window has opened but before the window has closed:

• The 'auto optimizer stats collection' task is assigned to a window group:

                      CLIENT_NAME                       WINDOW_GROUP
                      -------------------------------   ------------------
                      auto optimizer stats collection   ORA$AT_WGRP_OS
                      

• In 11g, each day has its own window, but they are in the same window group.

                      OWNER     GROUP_NAME          MEMBER_NAME
                      -----     ----------          -------------
                      SYS       ORA$AT_WGRP_OS      "SYS"."MONDAY_WINDOW"
                      SYS       ORA$AT_WGRP_OS      "SYS"."TUESDAY_WINDOW"
                      SYS       ORA$AT_WGRP_OS      "SYS"."WEDNESDAY_WINDOW"
                      SYS       ORA$AT_WGRP_OS      "SYS"."THURSDAY_WINDOW"
                      SYS       ORA$AT_WGRP_OS      "SYS"."FRIDAY_WINDOW"
                      SYS       ORA$AT_WGRP_OS      "SYS"."SATURDAY_WINDOW"
                      SYS       ORA$AT_WGRP_OS      "SYS"."SUNDAY_WINDOW" 
                      

Changes

User restarted database using shutdown immediately while one scheduler window is running.

Cause

If a window in a window group is already open, and a new job is created that points to that window group, the job is not started until the next window in the window group opens.

If for some reason, SATURDAY_WINDOW was not closed after the database restarted and the status of SATURDAY_WINDOW remains 'ACTIVE' in dba views it will never be closed until you close it manually.
Other windows will also be stuck and statistics information collecting will not execute automatically.

The following test case shows how this happened:

SQL> EXECUTE DBMS_SCHEDULER.OPEN_WINDOW ('SATURDAY_WINDOW','');

PL/SQL procedure successfully completed.

select window_name,active from DBA_SCHEDULER_WINDOWS



                      WINDOW_NAME                   ACTIVE
                      ----------------------------  ------
                      MONDAY_WINDOW                 FALSE  
                      TUESDAY_WINDOW                FALSE  
                      WEDNESDAY_WINDOW              FALSE  
                      THURSDAY_WINDOW               FALSE  
                      FRIDAY_WINDOW                 FALSE  
                      SATURDAY_WINDOW               TRUE        <==SATURDAY_WINDOW started 
                      SUNDAY_WINDOW                 FALSE  
                      WEEKNIGHT_WINDOW              FALSE  
                      WEEKEND_WINDOW                FALSE  
                      

                      CLIENT_NAME                       JOB_NAME            JOB_SCHEDULER_STATUS
                      -------------------------------   ------------------- --------------
                      auto optimizer stats collection   ORA$AT_OS_OPT_SY_84 RUNNING <==optimizer stats collection
                      

                      WINDOW_NAME               ATIVE
                      ------------------------  ------
                      MONDAY_WINDOW             FALSE
                      TUESDAY_WINDOW            FALSE
                      WEDNESDAY_WINDOW          FALSE
                      THURSDAY_WINDOW           FALSE
                      FRIDAY_WINDOW             FALSE
                      SATURDAY_WINDOW           TRUE     <==SATURDAY_WINDOW started
                      SUNDAY_WINDOW             FALSE
                      WEEKNIGHT_WINDOW          FALSE
                      WEEKEND_WINDOW            FALSE  
                      

                      CLIENT_NAME       JOB_NAME    JOB_SCHEDULER_STATUS
                      ------------      -------     --------------------                       
                                                    <==SATURDAY_WINDOW is still active, but running job is gone.                       
                                                    <==This status will remain for days until you close it manually.
                      

                      ORA-06512: at line 1         <==next window cannot start up
 
                      

Solution

Solution is to close the active window manually:

EXECUTE DBMS_SCHEDULER.CLOSE_WINDOW ('SATURDAY_WINDOW');

Then ensure the window has been closed:

select window_name,active from DBA_SCHEDULER_WINDOWS;

                      WINDOW_NAME                       ACTIVE 
                      ------------------------------    ------ 
                      MONDAY_WINDOW                     FALSE  
                      TUESDAY_WINDOW                    FALSE  
                      WEDNESDAY_WINDOW                  FALSE  
                      THURSDAY_WINDOW                   FALSE  
                      FRIDAY_WINDOW                     FALSE  
                      SATURDAY_WINDOW                   FALSE  
                      SUNDAY_WINDOW                     FALSE  
                      WEEKNIGHT_WINDOW                  FALSE  
                      WEEKEND_WINDOW                    FALSE
                      

12c Cloud Control: Details of the Directory Structure and Commonly Used Locations in a 12c OMS Installation

Modified 15-FEB-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later [Release: 12.1 and later ]
Information in this document applies to any platform.

Goal

This document provides details about the Directory Structure and list of commonly used locations in a 12c Cloud Control Oracle Management Service (OMS) installation.

Solution

Directory Structure in 12c OMS Installation

Following are some of the important directories in a 12c OMS installation:

1. Middleware Weblogic home: Also referred to as <MW_HOME>, this is the directory under which 10.3.5 Weblogic software is installed.
2. Oracle Management Service Instance Base: Also referred to as <EM_INSTANCE_BASE>; or <EM_CONFIG_HOME>, this is a directory,
   by default named gc_inst under the <MW_HOME>.
3. Oracle Management Service home: Also referred to as <OMS_HOME>, this is the location where the 12c OMS software is installed and is a directory
   under the <MW_HOME>.
4. Oracle Management Service Instance home directory: Also referred to as <EM_INSTANCE_HOME>, this is location where the EM Instance is created
   and is a directory under the gc_inst directory.
5. Middleware Common home directory: This directory contains some of the common tools such as Opatch, local inventory etc.
6. Middleware WebTier Home: Location where the OPMN and HTTP Server software is installed under the Middleware Home.
7. Middleware WebTier Instance home: Location where the HTTP Server and OPMN instance related files are stored. This is a directory under the gc_inst directory.
8. Weblogic Domain home directory: Also referred to as the <DOMAIN_HOME>, this is the location where the WLS Domain related files are created.
   By default, the OMS installation will result in ADMIN_SERVER and EMGC_OMS1 managed server.
9. Admin Server Location: Location where the ADMIN_SERVER files are created under the DOMAIN_HOME.
10. Managed Server Location: Location where the Managed server files are created under the <DOMAIN_HOME>.
11. EM NodeManager directory: This is the location where the EM specific Nodemanager is installed and is a directory under the gc_inst.
12. Oracle Management Agent Base directory: Also referred to as <AGENT_BASE>, this is the root of the 12c Agent installation and is a directory under the <MW_HOME>.
    This inturn contains multiple Homes for the Agent software bits, plugins, setuid binaries and the Agent State Home.
13. Oracle Management Agent Instance Home: Also referred to as the Agent State home or <AGENT_INSTANCE_HOME> or <EMSTATE>,
    this is the location where the 12c Agent instance files are created.
14. Oracle Management Service plugins directory: This is the location containing the plugins installed at the OMS side.
15. Oracle Management Agent plugins directory: This is the location containing the plugins installed at the Agent side.

NOTE: The above directory structure is applicable to a 12c Cloud Control setup where EM software is used to install the JDK and WLS and Simple Installation method is chosen.

- If JDK and Weblogic are pre-installed prior to the EM installation, then the JDK location would differ as per the installation.

- If Advanced Installation option is chosen in the OUI, user can choose to create the gc_inst in a different location.

12c: How to Check which OPatch version was used to apply the Bundle Patch 1 or One-off patches

Modified 08-MAR-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Goal

How to determine which OPatch version was used to apply Bundle Patch 1 or one-off patches to the ORACLE_HOMEs.

Solution

1. Locate the opatch_history.txt file which is normally in:
   
   
   • OMS: <OMS_HOME>/cfgtoollogs/opatch
     
     Example: /u01/middleware/oms/cfgtoollogs/opatch
     
   
   
   • Agent:
     
     Example: ../../agent/core/12.1.0.1.0/cfgtoollogs/opatch
     
   
   
2. View the file and look for the version used:
   
   Date & Time : Tue Mar 06 14:53:51 IST 2012
Oracle Home : /u01/middleware/oms
OPatch Ver. : 11.1.0.9.4
Current Dir : /u02/bundle_patch1/13470978
Command : apply -invPtrLoc /u01/middleware/oms/oraInst.loc
Log File : /u01/middleware/oms/cfgtoollogs/opatch/13470978_Mar_06_2012_14_53_51/apply2012-03-06_14-53-50PM_1.log

From the above information we can understand that the Opatch version used was 11.1.0.9.4 to apply the Patch 13470978.


12c Cloud Control: Overview of the EMCTL Options Available for Managing the 12c OMS

Modified 16-OCT-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Goal

This article provides an easy access to the list of options available with the emctl utility for the 12c OMS.

Solution

Please set the ORACLE_HOME to your OMS_HOME location using

For UNIX:

export ORACLE_HOME=/oracle/product/Middleware/oms

Description of emctl options:

Agent upload fails with: ERROR-400|ORA-01465: invalid hex number

Modified 20-JAN-2013

Applies to

Enterprise Manager Base Platform - Version 10.2.0.1 to 11.1.0.1 [Release 10.2 to 11.1]
Information in this document applies to any platform.

Symptoms

Agent fails to upload to the OMS.

emagent.trc file contains the following errors repeatedly:

2008-07-16 11:04:47 Thread-1172 ERROR upload: Failed to upload file B0000002.xml, ret = -2
2008-07-16 11:04:47 Thread-1172 WARN upload: FxferSend: received http error in header from repository: https://hostname:1159/em/upload
ERROR-400|ORA-01465: invalid hex number

Cause

The issue can happen if the policy_guid value in the xml file the agent is trying to upload is corrupted.

We can see similar to the following corrupted data in the xml file which is failed to load (In this example, file is B0000002.xml)

<POLICY_GUID><![CDATA["~�p]]></POLICY_GUID>

A bug has been issued for this issue.

Solution

The BUG was not resolved. The issue was resolved by upgrading the agent to the later versions (10.2.0.3/10.2.0.4/10.2.0.5) or by using the following workaround:

1. Stop the agent with
   
   AGENT_HOME/bin>./emctl stop agent



2. Please remove the corrupted xml file from the following location.
   
   AGENT_HOME/sysman/emd/uploads/
   


3. Remove
   
   AGENT_HOME/sysman/emd/lastupld.xml
   


4. Start the agent with
   
   AGENT_HOME/bin>./emctl start agent



5. Issue the upload command.
   
   AGENT_HOME/bin>./emctl upload agent

How To Find Database Services Which Are Registered To A Listener Target In Enterprise Manager Cloud Control 12c

Modified 26-NOV-2011

Applies to

Enterprise Manager for Oracle Database - Version: 12.1.0.1.0 and later [Release: 12.1 and later]
Information in this document applies to any platform.

Goal

This document explains how to find database services which are registered to a Listener target?

Solution

1. From the home page, Navigate to: Enterprise → Configuration → Search
   
   

   

2. Click on create
   
   

   

3. Choose → Target Type: Listener
   
   
4. Click on the "Magnifier" icon in front of the "Target Name" label and pick a listener
5. Choose → Configuration Item: Listener services
   
   

   

6. Click on Search

EM 12c: Creating an Administrator Role in Enterprise Manager Cloud Control 12.1.0.1

Modified 19-MAR-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Goal

This article provides a quick reference for creating an administrator role in Enterprise Manager Cloud Control 12c.

Solution

One Super Administrator, sysman, is created by default during the installation of the Enterprise Manager. By logging into the Cloud Control 12c Console as the sysman Super Administrator, other administrator accounts can be created for daily administration work.

NOTE: It is recommended that the sysman account should only be used to perform infrequent system wide, global configuration tasks.

"EM 12c: Creating an Administrative User and Assigning an Administrative Role in Enterprise Manager Cloud Control 12.1.0.1" on page 33

Tips and Tricks on Debugging the 12c Harvester

Modified 03-JAN-2013

Applies to

Oracle Configuration Manager - Version 10.3.6.0.0 to 10.3.6.0.0 [Release 10.3]
Information in this document applies to any platform.

Purpose

The following compilation of errors focuses on the overall 12c GridControl Harvester feature.
In this note, we will attempt to record errors which under certain conditions may trigger.
Therefore, this note will be the main source of troubleshooting the 12c GC Harvester.

There will be 3 categories:

- CSI Assignment UI
- GC Harvester Upload
- GC Harvester Registration

Troubleshooting Steps

CSI Assignment UI

Scenario 1

GC Harvester has never run. Possible causes:

- The Harvester job is disabled.

- The OCM collector is not installed/configured in the OMS_Home location.

To Do : See errors in emoms_pbs.trc located in the <Middleware_Home>/gc_inst/em/EMGC_OMS1/sysman/log

Or you can also > egrep -i emoms_pbs.trc |grep harvester

Scenario 2

GC Harvester has not run in the last {0} days. Possible causes:

- The Harvester Job was running successfully for a period of time but now its not running.

- The Harvester job has been disabled.

- The OCM collector is not installed/configured in the OMS_Home location.

To Do : See errors in emoms_pbs.trc or mgmt_system_error_log table (Setup > Management Services and Repository > Repository Operations)

If there are no errors being reported in the trace files, then perhaps the GC Harvester needs to be started and scheduled.
In that case, you need to start the Harvester using the gcharvester_job.sql script found in the OMS_HOME/sysman/admin/emdrep/sql/core/latest/gcharvester directory:

This is what it looks like:

declare
l_already_registered NUMBER :=0;
l_job_targets MGMT_JOB_TARGET_LIST := MGMT_JOB_TARGET_LIST();
l_params MGMT_JOB_PARAM_LIST := MGMT_JOB_PARAM_LIST();
l_schedule MGMT_JOB_SCHEDULE_RECORD;
l_job_id_out RAW(16);
l_execution_id_out RAW(16);
l_version varchar2(32);

begin
-- first check if job is already registered (possibly in earlier version of EM)
BEGIN
SELECT 1 into l_already_registered
FROM MGMT_JOB
WHERE job_name='GCHARVESTERJOB' AND job_type='GCHarvesterRun';

EXCEPTION
WHEN NO_DATA_FOUND THEN
null;
END;

IF (l_already_registered = 1) THEN
RETURN;
END IF;

-- start a daily job, running at 2 am
l_schedule :=
MGMT_JOBS.get_job_schedule_record(p_frequency_code => MGMT_JOBS.DAILY_FREQUENCY_CODE,
p_start_time => SYSDATE + 1,
p_end_time => null,
p_execution_hours => 2,
p_execution_minutes => 0,
p_interval => 0,
p_months => null,
p_days => null,
p_timezone_info => MGMT_JOBS.TIMEZONE_REPOSITORY,
p_timezone_target_index => 0,
p_timezone_offset => 0,
p_timezone_region => null,
p_start_grace_period => 240);

MGMT_JOBS.submit_job(p_job_name => 'GCHARVESTERJOB',
p_description => 'GC Harvester Job',
p_job_type => 'GCHarvesterRun',
p_job_targets => l_job_targets,
p_job_params => l_params,
p_schedule => l_schedule,
p_job_id_out => l_job_id_out,
p_execution_id_out => l_execution_id_out,
p_owner => null,
p_system_job => MGMT_JOB_ENGINE.SYSTEM_JOB,
p_job_creds => null,
p_job_target_type => null,
p_job_notify_states => null,
p_running_time_limit => -1);

update MGMT_JOB set job_owner= MGMT_USER.GET_REPOSITORY_OWNER where job_name='GCHARVESTERJOB' AND job_type='GCHarvesterRun';
MGMT_LOG.register_logging_module('GCHARVESTER','GC Harvester', NULL);

commit;
end;
/

The two main arguments are : hour to run (execution_hours) and minute to run (execution_minutes).
As long as the time entered is not in the past, the script will run fine allowing the GC Harvester to function properly.

GC Harvester Upload

Scenario 1

Now in 12c, the GCHarvester uploads its data directly to the CCR repository. As part of this process flow, the collection files are deleted as soon as they are uploaded.
However in some cases where a collection is failing or there is a concern about targets and their timestamps,
You may have to set the following parameter to retain those collection files so they can be reviewed.

With a connection to the repository as a SYSDBA user, the delete_upload_file property in the mgmt_ocm_upl_props table needs to be set to False as follows:

insert into mgmt_ocm_upl_props(name, str_value) values('delete_upload_file','false')

NOTE: At this point the OMS needs to be restarted in order for that parameter to take affect.

1. Locate the OMS instance home.
2. In the $ORACLE_HOME/sysman/config/emInstanceMapping.properties file (where ORACLE_HOME is the Oracle Home of the OMS), there is an entry referencing a file called emgc.properties.
3. The directory in which the emgc.properties file is located is the "instance home" of the OMS. In the following example, /u01/app/oracle/product/gc_inst/em/EMGC_OMS1 is the instance home of the OMS:
   
   EMGC_OMS1=/u01/app/oracle/product/gc_inst/em/EMGC_OMS1/emgc.properties



4. Set the environment variable ORACLE_CONFIG_HOME to the directory of this emgc.properties file.
   
   Example:
   
   $export ORACLE_CONFIG_HOME=/u01/app/oracle/product/gc_inst/em/EMGC_OMS1

   
5. cd $ORACLE_HOME/ccr/bin
   emCCR register
   

The next scheduled time the harvester job runs, it will trigger a full collection.
In a multiple OMS environment, this 'emCCR register' will need to be completed in all locations.

Scenario 3

In an E-Business Suite environment, the Harvester is not 'EBS' aware. In other words, EBS is not currently a harvested target from Cloud Control and therefore,
the OCM standalone product will need to be used in order to upload the data to My Oraclce Support.

Scenario 4

During the scheduled harvester run (default is 2AM), the job fails and the following error is dumpted into the emoms.trc file :

2012-10-20 02:00:30,781 [JobWorker 73854412:Thread-86] ERROR
gcharvester.HarvesterJobUtils prepareTargetsForShipment.? - GC OCM Harvester: Failed to stage targets for shipment to Oracle: ORA-01031:
insufficient privileges ORA-06512: at line 9

At this point, we are trying to create a sequence object and failing. SYSMAN lacks the privilege to create sequence. This sequence is used to put all harvested targets in mgmt_ocm_upl_metrics_tmp and then process them in batches of 512 targets at a time.
First 1 to 512 will be processed, then 513 to 1024 and so on. The fix here is to assign the SYSMAN user the CREATE SEQUENCE priv :

GRANT CREATE SEQUENCE,DROP ANY SEQUENCE TO SYSMAN;

GC Harvester Registration

Some Backround First

At first run, the GCHarvester performs a GC Default registration using the registration information of the OCM collector in the OMS location. As part of this registration,the EM Repository URL is also sent for book keeping purposes on the CCR side of things.
All other Oracle_Homes which are considered to be a 'Proxy' uploaded by the GCHarvester, is uploaded using the GC Default registration (this is similar to 11GC).

When a Target is added, this Target Oracle Home registration is performed after a user goes to UI CSI assignment page and assigns a CSI to it.

The registration process itself has a 2 step approach.


1. Authenticate: Done as part of the UI which authenticates with the CCR database.
2. Register: The GCHarvester internal job picks up on any 'Pending' targets and registers itself against our CCR database.

Also new in 12c is that the GCHarvester now directly uploads information to CCR directly and no longer relies on OCM to perform this task. It also deletes the files as soon as they are uploaded.




Scenario 1

Problem in assigning a Customer Support Identifier(CSI). UI Error: "You don't have the privilege to do this operation". Possible cause:

- The User itself does not have the Operator privilege on one or more targets within a given Oracle_Home for that target.

Scenario 2

The CSI drop down listing does not show any CSI information. To do:

- There is a chance that the My Oracle Support(MOS) connectivity from the OMS location is not successful. The emoms.trc file will have to be reviewed for any possible communication issue.

Scenario 3

The CCR authentication has failed. Possible cause:

- Since we use the CSI selected from the UI and MOS preferred ID for the CCR authentication process, that needs to be verified to be correct.
Check if the OCM collector can be configured using these same credentials that is being used in the UI under the My Oracle Support > Preferred Credentials > My Oracle Support Preferred Credentials page

Installation and Configuration

Q. Why do I see duplicate Targets in My Oracle Support after upgrading from 11GC to 12c?

A. This can be a result of not supplying the same My Oracle Support credentials that was used in the 11GC OMS Home when OCM was configured. Make sure that in the 12c OMS home location,
the OCM collector is installed with the same credentials that was used in 11GC OCM location so that no duplicate target are created in MOS.

Q. Why don't I see the OMS and Agent as a target after upgrading, or installing a new 12c environment?
A. New in a 12c OCM / Harvester installation, the OMS and Agent targets are now collected via the GridControl Harvester and not OCM directly. Therefore,

Q. Why don't I need to install OCM in an Agent home in a 12c environment?

A. As of 12c, the Agent is considered to be a 'Converged' target. That means that the Harvester now treats both the OCM and 12c metric collections for this target type the same. This convergence is to avoid duplication in configuration metrics and piggy-back on the new Harvester functionality.

How to Change the Retention Period or Purge Policy for Enterprise Manager Cloud Control 12c Job History?

Modified 29-JAN-2013

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

The Enterprise Manager default purge policy for jobs deletes all finished jobs that are older than 30 days. This value cannot be changed using Enterprise Manager, but it can be changed using SQL*Plus.

The details in this document is applicable only to the Job history seen in the Jobs tab in the Enterprise Manager and not to the Database-side scheduler jobs.

NOTE: The default purging policies have been designed to provide the most available data for analysis while still providing the best performance and disk-space requirements for the Management Repository.
As a result, these should not be modified to improve performance or increase the available disk space. Increasing the retention time drastically can affect the performance of the Management Repository and have adverse reactions on the scalability of the Enterprise Manager installation.

Fix

The actual purging of completed job history is implemented via a DBMS job that runs once a day in the repository database. When the DBMS job runs in the repository database, it looks for finished Enterprise Manager Jobs that have a history which is 'n' number of days older than the current time
(value of sysdate in the repository database) and deletes the job history details. The value of 'n' is, by default, set to 30 days.

The default purge policy is called SYSPURGE_POLICY. To check the current values for this purge policies , log into the Management Repository database as sysman user and execute:

SQL>connect sysman/<sysman_pwd>;
SQL>Connected.
SQL>select * from mgmt_job_purge_policies order by policy_name;



                      POLICY_NAME                      TIME_FRAME

                      -------------------------------- ----------

                      CERTEOL_CERT_MD_JOBPURGEPOLICY            7

                      CERTEOL_EOL_MD_JOBPURGEPOLICY             7

                      CERTEOL_PLATCH_JOBPURGEPOLICY             7

                      DOWNLOADCVU_PURGEPOLICY                   7

                      OPATCHPATCHUPDATE_PAPURGEPOLICY           7

                      REFRESHFROMMETALINKPURGEPOLICY            7

                      SYSPURGE_POLICY                          30

                      

                      POLICY_NAME                      TIME_FRAME

                      -------------------------------- ----------

                      CERTEOL_CERT_MD_JOBPURGEPOLICY            7

                      CERTEOL_EOL_MD_JOBPURGEPOLICY             7

                      CERTEOL_PLATCH_JOBPURGEPOLICY             7

                      DOWNLOADCVU_PURGEPOLICY                   7

                      OPATCHPATCHUPDATE_PAPURGEPOLICY           7

                      REFRESHFROMMETALINKPURGEPOLICY            7

                      SYSPURGE_POLICY                          45

                      

NOTE: If you forgot to drop the policy, the create policy command will fail with:

SQL>execute MGMT_JOBS.register_purge_policy('SYSPURGE_POLICY', 60, null);
SQL>BEGIN MGMT_JOBS.register_purge_policy('SYSPURGE_POLICY', 45, null); END;

ERROR at line 1:
ORA-20416: Purge policy exists
ORA-06512: at "SYSMAN.EM_JOB_PURGE", line 247
ORA-06512: at "SYSMAN.MGMT_JOBS", line 688
ORA-06512: at line 1


How To Create A New ACFS Volume and Filesystem And Set The ACFS Filesystem Ownership To A Non-Grid/Oracle OS User?

Modified 29-JAN-2013

Applies to

Oracle Server - Enterprise Edition - Version: 11.2.0.1 to 11.2.0.4 - [Release 11.2]
Information in this document applies to any platform.

Goal

The present document provides an example about how to create a new ACFS volume and filesystem and set the ACFS filesystem ownership to a non-grid/oracle OS User.

Fix

You can set a different OS user to the ACFS filesystem and mount it with that different OS user as follow:

1. Connect to the ASM instance & create the ACFS diskgroup:
   
   SQL> CREATE DISKGROUP ACFSTEST EXTERNAL REDUNDANCY
DISK 'ORCL:ASMDISK18' SIZE 4157 M
DISK 'ORCL:ASMDISK19' SIZE 4157 M
ATTRIBUTE 'compatible.asm' = '11.2', 'compatible.advm' = '11.2';

Diskgroup created.



2. Create the ACFS volume:
   
   SQL> ALTER DISKGROUP ACFSTEST ADD VOLUME ACFSTESTVOL SIZE 7G;

Diskgroup altered.



3. Verify the volume was created and obtain the new volume name:
   
   [grid@dbaasm grid]$ asmcmd
ASMCMD> volinfo -a

Diskgroup Name: ACFSTEST

Volume Name: ACFSTESTVOL
Volume Device: /dev/asm/acfstestvol-76
State: ENABLED
Size (MB): 7168
Resize Unit (MB): 256
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:



4. Connect as root OS user and create the mount point directory ( e.g. "/goldengate"):
   
   [grid@dbaasm grid]$ su -
Password:

[root@dbaasm /]# mkdir /goldengate



5. Create the ACFS filesystem on the new volume ( e.g. "/dev/asm/acfstestvol-76"):
   
   [root@dbaasm ~]# /sbin/mkfs -t acfs -b 4k /dev/asm/acfstestvol-76
mkfs.acfs: version = 11.2.0.1.0.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume = /dev/asm/acfstestvol-76
mkfs.acfs: volume size = 7516192768
mkfs.acfs: Format complete



6. Register the ACFS filesystem on the new ACFS volume:
   
   [root@dbaasm ~]# /sbin/acfsutil registry -f -a /dev/asm/acfstestvol-76 /goldengate
acfsutil registry: mount point /goldengate successfully added to Oracle Registry



7. Create the desired OS group (e.g. "ggate"):
   
   [root@dbaasm ~]# groupadd ggate
[root@dbaasm ~]# grep ggate /etc/group
ggate:x:1302:



8. Create the desired OS user (e.g. "goldenguser") and associated it with the desired OS group(s) (e.g. "ggate"):
   
   [root@dbaasm ~]# useradd -u 1150 -g ggate goldenguser
[root@dbaasm ~]# id goldenguser
uid=1150(goldenguser) gid=1302(ggate) groups=1302(ggate)



9. Mount the new mount point ("/goldengate") on the new ACFS volume ("/dev/asm/acfstestvol-76"):
   
   [root@dbaasm ~]# /bin/mount -t acfs /dev/asm/acfstestvol-76 /goldengate

   or
   
   [root@dbaasm ~]# /sbin/mount.acfs -o all



10. Set the new OS ownership and set the new OS correct/desired permissions:
    
    [root@dbaasm ~]# chown goldenguser:ggate /goldengate
[root@dbaasm ~]# chmod 775 /goldengate
[root@dbaasm ~]# ls -ld /goldengate
drwxrwxr-x 2 goldenguser ggate 4096 May 16 09:45 /goldengate



11. Verify the new filesystem ("/goldengate") is mounted:
    
    [root@dbaasm ~]# df -k /goldengate
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/asm/acfstestvol-76 7340032 51672 7288360 1% /goldengate

[grid@dbaasm grid]$ asmcmd
ASMCMD> volinfo -a

Diskgroup Name: ACFSTEST

Volume Name: ACFSTESTVOL
Volume Device: /dev/asm/acfstestvol-76
State: ENABLED
Size (MB): 7168
Resize Unit (MB): 256
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage: ACFS
Mountpath: /goldengate

ASMCMD>



12. Test files creation (permissions & ownership) on the new filesystem:
    
    [root@dbaasm ~]# su - goldenguser
[goldenguser@dbaasm ~]$ pwd
/home/goldenguser
[goldenguser@dbaasm ~]$ cd /goldengate
[goldenguser@dbaasm goldengate]$ pwd
/goldengate
[goldenguser@dbaasm goldengate]$ touch test.txt
[goldenguser@dbaasm goldengate]$ ls -l
total 64
drwx------ 2 root root 65536 May 16 09:55 lost+found
-rw-r--r-- 1 goldenguser ggate 0 May 16 10:00 test.txt


NOTE: For the additional syntax used on Linux, Solaris, AIX & Windows see the attached document:

Automatic Storage Management Administrator's Guide

Check for the following directories:

# 13-1 Summary of Oracle ACFS commands for Linux and UNIX

# 13-2 Options for the Linux fsck command
# 13-3 Options for the Linux mkfs command
# 13-4 Options for the Linux mount command
# 13-5 Options for the Linux umount command

# 13-6 Summary of Oracle ACFS commands for Solaris

# 13-7 Options for the Solaris fsck command
# 13-8 Options for the Solaris mkfs command
# 13-9 Options for the Solaris mount command
# 13-10 Options for the Solaris umount command

# 13-11 Summary of Oracle ACFS commands for AIX

# 13-12 Options for the AIX fsck command
# 13-13 Options for the AIX mkfs command
# 13-14 Options for the AIX mount command
# 13-15 Options for the AIX umount command

The PDF version of this document (which includes additional references) can be obtained here

12c: How to Modify JVM Heap Size, MaxPermSize for 12c Enterprise Manager Agent via Enterprise Manager 12c Cloud Control Console?

Modified 11-DEC-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0
Information in this document applies to any platform.

Goal

How to Modify/Increase JVM Heap Size for 12.1.0.1 Agent via Enterprise Manager Cloud Console?

This is useful for cases where the agent restarts frequently or becomes unresponsive and the following messages are seen in $<install directory of agent>/agent_inst/sysman/log/gcagent.log

oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response:Response
java.lang.OutOfMemoryError: PermGen space
2012-02-02 01:54:24,409 [57:GC.Executor.1 (oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response)] ERROR - Critical error:
java.lang.OutOfMemoryError: PermGen space
2012-02-02 04:03:41,660 [64:GC.Executor.8 (oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response) (oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response:Response)] ERROR - oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response:Response
java.lang.OutOfMemoryError: PermGen space
2012-02-02 04:03:41,691 [64:GC.Executor.8 (oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response)] ERROR - Critical error:
java.lang.OutOfMemoryError: PermGen space


Fix

1. Log on to 12c Cloud Console as the Super Administrator user (eg. 'sysman')
2. Setup → Agents
   
   

   



3. Click on Agent link (agent_name:port#)
   
   

   



4. Expand the arrow next to "Agent" to show the drop down menu, choose 'Properties'
   
   

   



5. Show: All Properties
   
   

   



6. Expand RunTime Settings
   
   

   



7. Edit the "agentJavaDefines" parameter value , as below and click on 'Apply'
   
   

   Default value of "agentJavaDefines" parameter is:-
   
   -Xmx128M -XX:MaxPermSize=96M
   
   Change to:-
   
   -Xmx512M -XX:MaxPermSize=96M
   
   

   
   

   



8. Verify the changes in AGENT_INST_HOME/sysman/config/emd.properties.
   
   Before changing the agentJavaDefine value:
   
   $ grep -i XM emd.properties
agentJavaDefines=-Xmx128M -XX:MaxPermSize=96M

After changing the agentJavaDefine value:

$ grep -i XM emd.properties
agentJavaDefines=-Xmx512M -XX:MaxPermSize=96M


How to Configure Logging for the Enterprise Manager Cloud Control Management Agent12.1.0.1 (12c R1)

Modified 22-FEB-2013

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.1.0 [Release 12.1]
Information in this document applies to any platform.
Enterprise Manager Cloud Control 12.1

Goal

The goal of this article is to provide instruction on how to enable logging for the Enterprise Manager Management Agent Release 12.1.0.1 (12c R1).


Fix

In EM 12c, the mechanism for configuring agent logging has not changed, but the process and syntax for enabling agent logging has changed slightly.

The logging properties for the agent are stored in $EMSTATE/sysman/config/emd.properties file, same as in 11.1.
But the properties that control logging follow the log4j configuration syntax instead of the syntax supported by the 11.1 agent.

To active changes to the logging parameters, you can use either of the emctl commands:

• emctl reload agent
• emctl setproperty agent

Overview

Agent logging entries look similar to the standard log4j logging, with the only significant difference being the prefix 'Logger.' that you will see on each line. For example:

# logging properties
Logger.log4j.appender.Rolling=org.apache.log4j.RollingFileAppender
Logger.log4j.appender.Rolling.File=/orcl/core/12.1.0.1.0/agent_inst/sysman/log/gcagent.log
Logger.log4j.appender.Rolling.Append=true
Logger.log4j.appender.Rolling.MaxFileSize=5000000
Logger.log4j.appender.Rolling.MaxBackupIndex=10
Logger.log4j.appender.Rolling.layout=oracle.sysman.gcagent.util.logging.GCPattern

#
# Set root category priority to INFO and its appenders to Rolling (and Errors).
Logger.log4j.rootCategory=INFO, Rolling, Errors

If you examine all of the Logger.log4j entries in the emd.properties file, you will see different settings for an Errors appender and a Mdu appender.

• Errors appender is used to direct ERROR level logging entries to $EMSTATE/sysman/log/gcagent_errors.log
• Mdu appender is used to direct metadata update entries to $EMSTATE/sysman/log/gcagent_mdu.log.



NOTE: Neither of these two appenders' configurations should be modified.


Default logging level

Logger.log4j.rootCategory=INFO, Rolling, Errors, Test

The rootCategory property controls the default log level. It is by default set to INFO.

Changing the default logging level

To enable DEBUG level logging across the entire agent, set the rootCategory property entry to use DEBUG, similarly to what is shown below.

#
# Set root category priority to INFO and its appenders to Rolling and Errors.
# Change to DEBUG level for gcagent.log
Logger.log4j.rootCategory=DEBUG, Rolling, Errors

And then reload the agent with: $emctl reload agent

Alternatively, use can also use the emctl setproperty agent command:

$ emctl setproperty agent -name 'Logger.log4j.rootCategory' -value 'DEBUG,Rolling,Errors'



NOTE: Documentation shows the above command with spaces after each comma (ex: 'DEBUG, Rolling, Errors'). This will result in an error when you run the command.
Remove the spaces between 'DEBUG,Rolling,Errors'.



Setting the number of backup log files

When you set agent logging to DEBUG level, for anything but a most-trivial agent deployment, log entries have the possibility of rolling away after less than an hour.
For this reason, it is also sensible to increase the number of log file rolls if the entire agent is configured to log at the DEBUG level.
The number of rolls is set through the MaxBackupIndex parameter and it represents the number of backup log files to retain. It is recommended to keep no more than 100 files:

# Increase to 100 rolls of the log (gcagent.log, gcagent.log.1,...,gcagent.log.100)
Logger.log4j.appender.Rolling.MaxBackupIndex=100

And then reload the agent with: $emctl reload agent

Alternatively, you can issue the command:

$ emctl setproperty agent -name "Logger.log4j.appender.Rolling.MaxBackupIndex" -value 100

Example of Standard DEBUG settings

Append the following to $EMSTATE/sysman/config/emd.properties and reload the agent.

# Recommended settings to produce DEBUG level logging
Logger.log4j.rootCategory=DEBUG, Rolling, Errors
Logger.log4j.appender.Rolling.MaxBackupIndex=50

Example of Standard TRACE settings

Append the following to to $EMSTATE/sysman/config/emd.properties and reload the agent.

# Recommended settings to produce TRACE level logging
Logger.log4j.rootCategory=DEBUG, Rolling, Errors
Logger.log4j.appender.Rolling.MaxBackupIndex=100
Logger._enableTrace=true

Setting Log Level for Individual Classes and Packages

The logging level for individual class and/or packages can also be set. Here are couple of examples that are currently configured by default:

# Set the class loaders to level INFO
Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.ChainedClassLoader=INFO
Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.ReverseDelegationClassLoader=INFO
Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.PluginLibraryClassLoader=INFO
Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.PluginClassLoader=INFO

The above configuration changed the default level of logging for the four classes to be INFO.
When the default level of logging is INFO it doesn't make any difference but if the default log level will be set to DEBUG (when debugging the code)
it will prevent those four classes from logging at DEBUG level (as they are normally too verbose).
The reverse is also true, for example if the following configuration is added (not set by default):

Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.collection=DEBUG

It will cause all classes in the oracle.sysman.gcagent.metadata.impl.collection package to log at DEBUG level even if the default log level is INFO.

gcagent.log Configuration

Logger.log4j.appender.Rolling=org.apache.log4j.RollingFileAppender
Logger.log4j.appender.Rolling.File=/ade/ysarig_v6/oracle/work/agentStateDir/sysman/log/gcagent.log
Logger.log4j.appender.Rolling.Append=true
Logger.log4j.appender.Rolling.MaxFileSize=5000000
Logger.log4j.appender.Rolling.MaxBackupIndex=10

Logger.log4j.appender.Rolling.layout=oracle.sysman.gcagent.util.logging.GCPattern

The gcagent.log is configured using the rolling appender (properties that starts with Logger.log4j.appender.Rolling):
• File property controls the location and name of the main log, it is recommended not to change the setting for it.
• Append determine if the log file should be append (true) or overwritten (false) on agent startup
• MaxFileSize determine the size of each of the log segments.
• MaxBackupIndex determine the number of backup segments for the log (the total number of segments are this number plus one)

gcagent_error.log Configuration

Logger.log4j.appender.Errors=org.apache.log4j.FileAppender
Logger.log4j.appender.Errors.File=/ade/ysarig_v6/oracle/work/agentStateDir/sysman/log/gcagent_errors.log Logger.log4j.appender.Errors.Append=true
Logger.log4j.appender.Errors.Threshold=ERROR

The gcagent_errors.log contains log messages of ERROR and FATAL levels. The log is not segmented and it has no size limit. The gcagent_errors.log is a subset of the gcagent.log

Understanding the Management Agent Logs Files



To understand the Management Agent Logs Files, refer to our technical tip "EM 12c Cloud Control : Understanding the Management Agent Logs Files" on page 25.

EM 12c Cloud Control : Understanding the Management Agent Logs Files

Modified 01-OCT-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Purpose

This document describes how to locate and understand the various log/trace files of the Enterprise Manager Cloud Control Agent 12c.

Scope

Cloud Control Administrators who need to monitor the operation of the Agent.
Oracle Management Agent log and trace files store important and useful information in order to troubleshoot problems.

Details

About the Main Management Agent Log File : gcagent.log

The Agent main log file gcagent.log is located in the directory $EMSTATE/sysman/log.

where $EMSTATE is the Agent instance directory. For example:

EMSTATE=<Agent_Oracle_Home>/agent_inst

NOTE: to find the Agent instance directory refer the "Locating the Management Agent Log and Trace Files" section below.

12C Cloud Control: How to set the 'My Oracle Support' Preferred Credentials in 12c Cloud control?

Modified 29-JAN-2013

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - [Release: 12.1]
Enterprise Manager for My Oracle Support - Version: 10.2.0.5 to 12.1.0.1.0 [Release: 10.2 to 12.1]
Information in this document applies to any platform.

Goal

How to set the 'My Oracle Support' Preferred Credentials in 12c Cloud control?

Solution

Accessing My Oracle Support from Enterprise Manager Cloud Control:

My Oracle Support is Oracle's customer support portal.
My Oracle Support integrated with Enterprise Manager Cloud Control provides a more complete presentation of the management process for both proactive and reactive issue management.

Access the following My Oracle Support Pages from Enterprise → My Oracle Support:

* Service Requests
* Knowledge
* Certify
* Community

The My Oracle Support Patches & Updates page is available from Change Management > Patches & Updates.

My Oracle Support Preferred Credentials:

Need to set the preferred credentials to simplify access to My Oracle Support. If set, you will be automatically signed in to My Oracle Support when logged into Enterprise Manager.

1. There are two ways to set the MOS credentials:
   
   
   1. Log into the 12C Cloud Control
      select the Setup menu located at the top-right of the page
      → My Oracle Support → Set Credentials
      
      

      

   
   
   2. Log into the 12C Cloud Control
      select the Setup menu located at the top-right of the page
      → Security → Preferred Credentials
      From the preferred credentials page, scroll down to the section "My Oracle Support Preferred Credentials"
      Click on "Set MOS Credentials"
      
      

      

   
   
2. Fill in the fields Username and Password with your My Oracle Support account username and password.
   
   

   



3. Click on "Apply" you will get the following screen as confirmation:
   
   

   

To unset/remove the MOS credentails:

1. We have an option to remove the MOS credentials Which is new in 12C.
   
   

   

2. Click on Remove and continue to remove MOS credentials.

NOTE: if you are in Offline Mode, you cannot set the MOS Credentials.

You will get the following message:

My Oracle Support Preferred Credentials

No preferred credential is required because My Oracle Support is in offline mode.To go online, configure the connection setting.

You need to click on the link "connection setting" then you need to change the connection mode from Offline to Online.

You will then be able to set the MOS Credentials.

Cannot Connect From 12c Cloud Control To HTTPS://Updates.Oracle.Com

Modified 12-DEC-2012

Applies to

Enterprise Manager for My Oracle Support - Version 12.1.0.2.0 and later
Information in this document applies to any platform.
Applies to Enterprise manager for Grid Control 12.1.0.1.0

Symptoms

After installing 12c Cloud Control, the OMS is unable to connect to Oracle sites.

1. At minimum, the following URLs should be made available through the firewall:
   
   aru-akam.oracle.com
   ccr.oracle.com
   updates.oracle.com
   login.oracle.com
   support.oracle.com
   
   
2. Here are the results of the wget commands used to test the connection:
   
   wget https://ccr.oracle.com
--2012-03-12 08:53:42-- https://ccr.oracle.com/
Resolving ccr.oracle.com... 141.146.8.206
Connecting to ccr.oracle.com|141.146.8.206|:443... failed: Connection refused.
wget https://updates.oracle.com
--2012-03-12 08:54:10-- https://updates.oracle.com/
Resolving updates.oracle.com... 141.146.44.51
Connecting to updates.oracle.com|141.146.44.51|:443... failed: Connection refused.
wget https://login.oracle.com
--2012-03-12 08:54:36-- https://login.oracle.com/
Resolving login.oracle.com... 141.146.8.119
Connecting to login.oracle.com|141.146.8.119|:443... failed: Connection refused.
wget https://support.oracle.com
--2012-03-12 08:55:05-- https://support.oracle.com/
Resolving support.oracle.com... 141.146.54.16
Connecting to support.oracle.com|141.146.54.16|:443... failed: Connection refused.

Changes

A network firewall is configured in this environment.

Cause

When Enterprise Manager Cloud Control 12c Release 1 (12.1.0.1) was released Oracle started using new URLs for connections into some backend Oracle sites which it requires for certain product features like Patching.

Solution

When a firewall is in place, then the following URLs need to be opened at the OMS level in order to communicate successfully to Oracle:

• support.oracle.com
• login.oracle.com
• servicecentral.sun.com
• updates.oracle.com
• loginadc.oracle.com
• aru-akam.oracle.com
• ccr.oracle.com

"12C Cloud Control: How to set the 'My Oracle Support' Preferred Credentials in 12c Cloud control?" on page 25

12c Cloud Control Blackouts: Steps to Create Blackouts from Console UI/emctl/emcli

Modified 25-FEB-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later [Release: 12.1 and later]
Information in this document applies to any platform.

Purpose

This document describes different ways available to create blackouts in 12c Cloud Control:

1. Using Console UI.
2. Using emctl command.
3. Using emcli.

Scope and Application

This document is applicable to Enterprise Manager users who would like to configure blackouts in the 12c Cloud Control.

12c Cloud Control Blackouts: Steps to Create Blackouts from Console UI/emctl/emclion

Using Console UI:

1. Login to 12c Cloud Control
2. Navigate to Enterprise → Monitoring → Blackouts
3. Click on "Create" to create a blackout.
4. In the page displayed, provide the following information:

1. Name: Name of the blackout
2. Comments: comments if any (optional)
3. Reason: Reason for creating the blackout (optional)
4. Below options can be chosen depending on the requirement:
• Run jobs during the blackout. Jobs that are not agent-bound will continue to run during the blackout even if this option is unselected.
• Detect Service that are dependent on target that will be selected in blackout.
5. Add the required target that needs to be in blackout. If you choose a Host target,
   you can opt to create the blackout against all the targets in the host or against a few selected targets only.

NOTE: Agent targets are not shown in the Target list because they can only be blacked out as part of a full host blackout.

1. Enter the TimeZone in which blackout needs to be executed
2. Select the Start options(Immediately/Later)
3. Select the Duration(Indefinite/Length/Until specific time)
4. Provide the Repeating option

1. To create blackout:
1. Create blackout on a particular target:

<AGENT_HOME>/bin> emctl start blackout <Name of the blackout> [<Target_name>[:<Target_Type>]] [-d <Duration>]


2. Create a node level blackout:

<AGENT_HOME>/bin> emctl start blackout <Name of the blackout> -nodeLevel [-d <Duration>]



2. To check blackout status:

<AGENT_HOME>/bin> emctl status blackout [<Target_name>[:<Target_Type>]]


3. To stop a blackout:

<AGENT_HOME>/bin> emctl stop blackout <Name of the blackout>

"12c Cloud Control EM CLI: How to Install and Setup 12c emcli (Enterprise Manager Command Line Interface)?" on page 7

1. To create a blackout:
1. Create blackout on specific target:

emcli create_blackout -name=<Name of the blackout> -add_targets=<target_name>:<target_type> -schedule=" frequency:once|interval|weekly|monthly|yearly];duration:[HH...][:mm...]" -reason="<reason for blackout>"

Example:

emcli create_blackout -name=DBMaintenance -add_targets=dev:oracle_database -schedule="duration::45" -reason="maintenance"


2. Create a node level blackout:

emcli create_blackout -name=<Name of the blackout> -add_targets=<target_name>:host -schedule=" frequency:once|interval|weekly|monthly|yearly];duration:[HH...][:mm...]" -reason="<reason for blackout>"

Example:

emcli create_blackout -name=HostMaintenance -add_targets=asingara-pc:host -schedule="duration::45" -reason="node level blackout"


2. To get list of blackouts:
• Details of all blackouts: emcli get_blackouts
• Details of blackout about all targets on specific host: emcli get_blackouts -hostnames=<host name>
• Details of blackout on specific target: emcli get_blackouts -target=<target name>:<target type>


3. To stop blackout:

emcli stop_blackout -name="<name of the blackout>" [-createdby="<blackout owner>"]


4. To delete blackout:

emcli delete_blackout -name="name" [-createdby="<blackout owner>" (default is current user)]

How to use Enterprise Manager 12c Pre-Requisites checker script?

Modified 29-AUG-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Purpose

This document describes in detail how to use EM 12c Pre-Requistes checker scripts in following steps:

1. What is EM Prerequisite Kit
2. How to Run EM Prereq Kit
3. How to Run EM Prereq Kit with Additional Arguments
4. Viewing Prereq Check Results
5. Viewing Prereq Check Log Files

Scope

This EM Prereq Kit is executed automatically when Install or Upgrade wizard is invoked. This can also be executed manually to check whether all the prerequisites are met and can be used to take Corrective Actions also.

Details

What is EM Prerequisite Kit?

• The EM Prerequisite Kit is a command line utility that runs repository-related prerequisite checks in your environment to ensure that you meet all the repository requirements for installing or upgrading an Enterprise Manager system.
• The kit not only runs the prerequisite checks but also takes corrective action automatically, to the extent possible, when a prerequisite check fails. The kit also takes post requisite steps to revert the corrective actions taken and ensure that the system is back to how it was before installing or upgrading the Enterprise Manager system.

NOTE: You can download the latest version of EM Prerequisite Kit from the Self Update framework as follows:

1. In Cloud Control, from the Setup menu, select Extensibility and then click Self Update.
2. On the Self Update page, download the new version of XMLs under the entity EM Deployment Prerequisite Resources Updates, if there are any available.

NOTE: If you downloaded and unzipped the Enterprise Manager Cloud Control 12c software with the Bundle Patch 1 (BP1) included, you do not need to follow the steps 1 to 3 below. You can just run the emprereqkit from the unzipped location.

Example: E:\distrib\oracle\12c\em12_winx64\install\requisites\bin or /u02/distrib/oracle/12c/em12_linux64

1. Create a directory Disk1 on your file system.
2. Copy <DVD-location>/install/ to Disk1/.
3. Copy <DVD-location>/stage/ to Disk1/.

NOTE: Ensure that the user running the EM Prerequisite Kit has write permission to the central inventory.

1. See the results without taking any corrective actions:
   
   Disk1/install/requisites/bin/emprereqkit -executionType install -prerequisiteXMLLoc <prereq_xml_location> -connectString <connect_string> -dbUser SYS -dbPassword <db_password> -dbRole sysdba -showPrereqs

Examples:

$ Disk1/install/requisites/bin/emprereqkit -executionType install -prerequisiteXMLLoc Disk1/install/requisites/list -dbHost rephost.repdomain -dbPort 1521 -dbSid emrep -dbUser SYS -dbPassword Oracle12 -dbRole sysdba -showPrereqs
$ /u02/distrib/oracle/12c/em12_linux64/install/requisites/bin/emprereqkit -executionType install -prerequisiteXMLLoc /u02/distrib/oracle/12c/em12_linux64/install/requisites/list -dbHost rephost.repdomain -dbPort 1521 -dbSid emrep -dbUser SYS -dbPassword Oracle12 -dbRole sysdba -showPrereqs



2. With Taking Corrective Actions to meet the repository requirements:
   
   Disk1/install/requisites/bin/emprereqkit -executionType install -prerequisiteXMLLoc <prereq_xml_location> -connectString <connect_string> -dbUser SYS -dbPassword <db_password> -dbRole sysdba -runPrerequisites -runCorrectiveActions

Examples:

$ Disk1/install/requisites/bin/emprereqkit -executionType install -prerequisiteXMLLoc Disk1/install/requisites/list -dbHost rephost.repdomain -dbPort 1521 -dbSid emrep -dbUser SYS -dbPassword Oracle12 -dbRole sysdba -runPrerequisites -runCorrectiveActions
$ /u02/distrib/oracle/12c/em12_linux64/install/requisites/bin/emprereqkit -executionType install -prerequisiteXMLLoc /u02/distrib/oracle/12c/em12_linux64/install/requisites/list -dbHost rephost.repdomain -dbPort 1521 -dbSid emrep -dbUser SYS -dbPassword Oracle12 -dbRole sysdba -runPrerequisites -runCorrectiveActions


Running from Installed Location

After the Cloud Contol is installed, EM Preq Kit can be called from the following location:

$<OMS_HOME>/install/requisites/bin/emprereqkit

The default XML files related to the prerequisite checks, which are stored in the Disk1/install/requisites/list directory on the software kit (DVD, downloaded shiphome), are current at the time of the release of the product.
However, after the release of the product, if a new prerequisite check is introduced or if an existing prerequisite check is updated, then you must do one of the following:


1. Enable the Self Update functionality within Enterprise Manager Cloud Control so that the new or updated prerequisite XML files are automatically downloaded to the $<OMS_HOME>/install/requisites/list directory.
2. Manually download the new or updated prerequisite XML files from Oracle store to the $<OMS_HOME>/install/requisites/list directory.


Arguments for EM Prereq Kit:

Mandatory Parameters for EM Prereq Kit:


1. -executionType
2. Either "-prerequisiteXMLLoc" Or "-prerequisiteXMLRootDir"
3. Either "-connectString" Or "-dbHost <hostname> -dbPort <port> -dbSid <sid>"
4. -dbUser
5. -dbPassword
6. -dbRole
7. -prereqResultLoc (If not given, current directory is taken)
8. -showPrereqs
9. Any one from ( "-runPrerequisites", "-showCorrectiveActions", "-runCorrectiveActions", "-showPostCorrectiveActions" , "-runPostCorrectiveActions" )
Optional Parameters for EM Prereq Kit:


1. -logLoc
2. -runOnlyFor
3. -responseFile
4. -contextName
5. -componentVariables
6. -logInDB
7. -stopExecOnFirstError
8. -list
9. -export
10. -purge
11. -help ( This option gives details of various parameters which can be passed to EM Prerequisite Kit)

Viewing Prereq Check Results

Every time the EM Prerequisite Kit is run, the results of the prerequisite checks run for a particular component are stored in an instance XML file. The instance XML file has the file name <component>.xml.
The results are in the same format as the information stored in the prerequisite XML files. The only difference is the new column that indicates the actual result of the prerequisite check.

Path if manual Invoked :

<prereqResultLoc>/resultXMLs/<time-stamp>
<prereqResultLoc>/resultXMLs/LATEST/ => Previous Executions

Path if invoked Automatically (by Install or Upgrade):

<MW_HOME>/.gcinstall_temp/resultXMLs/<timestamp>
<MW_HOME>/.gcinstall_temp/resultXMLs/LATEST => Previous Executions

Viewing Prereq Check Log Files


1. emprereqkit.log - Information about every step taken or action performed by the kit
2. repository.log - About the repository-related prerequisite checks that are run
3. emprereqkit.err.log - Only the error and stacktrace of the exceptions occured
4. performance.log - Information about the repository-specific performance-related prerequisite checks that are run
5. emprereqkit.output - Contains information about the status (pass or fail) of all the prerequisite checks that are run. It also contains detailed information regarding each prerequisite check.

11g Grid Control: How to Change SSL Administration Ports for both WebLogic Admin Server and the Mid (Web) Tier (Apache)

Modified 08-FEB-2013

Applies to

Enterprise Manager Base Platform - Version: 11.1.0.1 to 11.1.0.1 - Release: 11.1 to 11.1
Information in this document applies to any platform.
The contents of this note apply to Enterprise Manager release 11G and up

Goal

The goal of this document is to provide specific instructions on how to perform the following activities:

1. Change the SSL port of the WebLogic Administration Server.
   
   

   NOTE: The WebLogic Administration server stores the container specific configuration details for the OMS application server (managed server) and performs all administrative functions for the domain through the console.
   By default, the SSL port for the WebLogic Administration server is 7101



2. Change the SSL port for the Web Tier which is used to access the Enterprise Manager Console
   
   

   NOTE: In Enterprise Manager 11, communication between a browser client and the OMS server typically has the following flow:
   
   Browser → Web Tier (Apache) → OMS Server (WebLogic Managed Server)
   These instructions are to change the SSL port for the Web Tier only (Apache).
   The default SSL port for the Web Tier is 7799

Solution

1. Change the WebLogic Administration Server SSL port
   
   
   • Log into the Administration Console:
   • https://hostname:port/console
   • Default is 7101, e.g.
     
     https://hostname:7101/console
     
   • Log in with the username and password specified at EM 11 installation time. The default username is: weblogic
   • Once in, locate the "domain structure" pane on the left column
   • Expand: GCDomain → Environment → Servers and click on Servers
   • In the right frame, you will see the server EMGC_ADMINSERVER(admin)
   • Click on the server name and a list of properties will appear.

   NOTE: There are two lists of tabs at the top.
   From the top list, choose "Configuration."
   From the bottom list, choose "General."
   

   
   One of the properties in the resulting list will be SSL Listen Port.
   Enter the new value (e.g. 1234) and click "Save"
   
   See screenshot for details:
   
   

   

   NOTE: Once you do this, your browser will likely show a "page cannot be displayed" message.
   This is because you've just changed the admin server port. You must update the URL in your browser to reflect the new value for the port (e.g. https://hostname:1234/console) in order to log in again.
   But, since we've changed the port already, there is no need to log in again - You should move to the next step.

   
   Optional: Users may validate the changes have been made by viewing:
   
   gc_inst/user_projects/domains/GCDomain/config/config.xml

   under the grid control installation root
   
   Search for the text EMGC_ADMINSERVER and check the port, e.g.
   
   <server>
<name>EMGC_ADMINSERVER</name>
<ssl>
<name>EMGC_ADMINSERVER</name>
<enabled>true</enabled>
<hostname-verification-ignored>true</hostname-verification-ignored>
<listen-port>1234</listen-port>
</ssl>
<listen-port-enabled>false</listen-port-enabled>
<listen-address>www.myHost.com</listen-address>
</server>

   

   NOTE: It is not necessary to restart the Administration server at any point during the above procedure.

   
   It is also necessary to update the port in three additional files:
   
   
   1. Under the install root (oracle base directory), locate the file
      
      em/EMGC_OMS1/emgc.properties
      
      Update the value for AS_HTTPS_PORT e.g.
      
      AS_HTTPS_PORT=1234

      
   2. Under the install root (oracle base directory) locate the file ./user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/startup.properties
      Update the value for AdminURL e.g.
      
      AdminURL=https\://www.myHost.com\:1234

   
   
   3. Under the install root (oracle base directory) locate the file
      
      ./user_projects/domains/GCDomain/init-info/tokenValue.properties
      
      Update the value for SSL_PORT e.g.
      
      SSL_PORT=1234

      At this point the Admin server is using the updated port. A restart of the OMS server is necessary in order to force it to begin communicating to the Admin server on the new port definition, e.g.
      
      cd <OMS_HOME>/bin
emctl stop oms
emctl start oms

   
   
2. Change the Webtier (OHS) SSL port used to access the Enterprise Manager Grid Console
   
   • Identify the new port number, e.g. 9999
   • Execute emctl set property as follows:
     
     $OMS_HOME/bin/> ./emctl set property -name oracle.sysman.emSDK.svlt.EMConsoleServerHTTPSPort -value 9999

     A sample run:
     
     [oracle.myHost bin]$ ./emctl set property -name oracle.sysman.emSDK.svlt.EMConsoleServerHTTPSPort -value 9999

Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
SYSMAN password:
Property oracle.sysman.emSDK.svlt.EMConsoleServerHTTPSPort for oms myHost:4889_Management_Service has been set to value 9999

     
   • Stop the OMS by issuing 'emctl stop oms -all', e.g.
     
     cd <OMS_HOME>/bin
emctl stop oms -all

   
   
   • Take a backup of following files in :
     
     gc_inst/WebTierIH1/config/OHS/ohs1/ssl.conf
     gc_inst/WebTierIH1/config/OHS/ohs1/ssl.conf/em/EMGC_OMS1/emgc.properties
     
   
   
   • Edit the above mentioned and change the reference of port 7799 to 9999.
   • Finally, start the OMS.
     
     cd <OMS_HOME>/bin
emctl start oms


12c Cloud Control: How to Modify the Password for SYSMAN and other Enterprise Manager Users at the OMS Level and Repository Database?

Modified 07-SEP-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

This article provides steps for modifying the password of SYSMAN and other Enterprise Manager users at the OMS/WLS level and the Repository database in a Cloud Control setup.
Simply changing the password in the repository database is not sufficient as the password is also stored in the WLS credential store.
The OMS uses sysman account to login into the repository database and if there is a mis-match in the password at the database level and the OMS configuration, the OMS cannot start and function properly.

NOTE: From 12c onwards, directly modifying the password for sysman or any other repository user at the Repository Database is not supported.
If the password is manually changed at the database, then it cannot be modified at the OMS level using 'emctl config oms'. This will render the Cloud Control setup unusable.
Hence, ensure that the passwords are changed only using one of the below listed methods.

Fix

If the current SYSMAN password is known

1. Stop all the OMS:
   
   cd <OMS_HOME>/bin
emctl stop oms

   Execute the same command on all the OMS machines including the primary OMS machine. Do not include '-all' as the Admin Server needs to be up during this operation.
   


2. Modify the SYSMAN password:
   
   cd <OMS_HOME>/bin
emctl config oms -change_repos_pwd

   

   NOTE:
   
   - The above command will prompt you for the current password of the SYSMAN user and the new password.
   - The password will be modified at the Repository Database as well as the WLS Credential store and the monitoring credentials for the 'OMS and Repository' target.
   - Along with the SYSMAN password, this command will modify the password for the ALL the EM users (SYSMAN_MDS, BIP, SYSMAN_OPSS, SYSMAN_APM, SYSMAN_RO, MGMT_VIEW) created in the Repository Database.
   

   
   Example output:
   
   emctl config oms -change_repos_pwd
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
Enter Repository User's Current Password :
Enter Repository User's New Password :

Changing passwords in backend ...
Passwords changed in backend successfully.
Updating repository password in Credential Store...
Successfully updated Repository password in Credential Store.
Restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
Successfully changed repository password.



3. Stop the Admin server on the primary OMS machine and re-start all the OMS:
   
   cd <OMS_HOME>/bin
emctl stop oms -all
emctl start oms

If the current SYSMAN password is unknown but the password has not been changed manually at the Database

1. Stop all the OMS:
   
   cd <OMS_HOME>/bin
emctl stop oms

   Execute the same command on all the OMS machines including the primary OMS machine. Do not include '-all' as the Admin Server needs to be up during this operation.
   


2. Modify the SYSMAN password:
   
   cd <OMS_HOME>/bin
emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd <sys user password> -new_pwd <new sysman password>

   

   NOTE:
   
   - The '-use_sys_pwd' is used to connect to the database as a SYS user and modify the sysman password in the Repository database.
   - The current sysman password is not prompted for and only the new password needs to be entered. This will allow the reset of the old password to the new password entered.
   - The password will be modified at the Repository Database as well as the WLS Credential store and the monitoring credentials for the 'OMS and Repository' target.
   - Along with the SYSMAN password, this command will modify the password for the ALL the EM users (SYSMAN_MDS, BIP, SYSMAN_OPSS, SYSMAN_APM, SYSMAN_RO, MGMT_VIEW) created in the Repository Database.
   - This command will work only if the sysman or any other repository user password has not been manually modified at the repository database.
   

   
   Example output:

emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd oracle123 -new_pwd oracle12
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.

Changing passwords in backend ...
Passwords changed in backend successfully.
Updating repository password in Credential Store...
Successfully updated Repository password in Credential Store.
Restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
Successfully changed repository password.



3. Stop the Admin server on the primary OMS machine and re-start all the OMS:
   
   cd <OMS_HOME>/bin
emctl stop oms -all
emctl start oms


Modifying the password for SYSMAN and other EM users on a Standby OMS setup

The steps below are applicable only if Standby OMS has seperate Admin Server than the Primary OMS, but the same Repository database.

1. Ensure that the password is changed on the Primary OMS setup as per the above steps
2. Stop the OMS on Standby site , if already running
   
   <OMS_HOME>/bin>emctl stop oms -all



3. Start the Admin server only
   
   <OMS_HOME>/bin>emctl start oms -admin_only



4. Logged into Admin server console as user 'weblogic'.
   
   
   • Navigate to Services → Data Sources → sysman-opss-ds → Connection Pool
   • Update the password field with new sysman password.
   • Save and Activate the changes
   
   
   • Now run the command below from Standby OMS HOME
     
     cd <OMS_HOME>/bin
emctl config oms -change_repos_pwd

     The above command will prompt you for the current password of the SYSMAN user and the new password.Provide the same value for both
     
   
   
   • Run the commands below to start the Standby OMS
     
     <OMS_HOME>/bin>emctl stop oms -all
<OMS_HOME>/bin>emctl start oms
<OMS_HOME>/bin>emctl status oms -details


11 Grid Control: Steps for Modifying the HTTP and HTTPS Upload Ports After Installation

Modified 08-FEB-2013

Applies to

Enterprise Manager Base Platform - Version 11.1.0.1 to 11.1.0.1 [Release 11.1]
Information in this document applies to any platform.

Goal

In 11.1.0.1 Enterprise Manager Grid Control, the management agent uses port 4900 (HTTPS) and port 4889 (HTTP) by default to upload the *.xml files to the Oracle Management Server. Sometimes users may require to change these ports due to firewall policies and restrictions.
This article explains how to change the HTTP/HTTPS upload ports configured on the OMS in 11.1.0.1 Grid Control. Note that after these ports have been changed on the OMS, it will be necessary to update the configuration of all the agents which upload to this OMS.

In this example, the original configuration is as follows:-

$ /emctl status oms -details
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host :<OMS-hostname>
HTTP Console Port : 7800
HTTPS Console Port : 7801
HTTP Upload Port : 4889
HTTPS Upload Port : 4900
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1

In this example, the original http upload port is 4889 and will be changed to 4890
In this example, the original https upload port is 4900 and will be changed to 1159
In this example the http console port (7800) will not be changed
In this example the https console port (7801) will not be changed

IMPORTANT NOTE: in this article the ports which are used to access the console ( https://omshostname.domain:7801/em in this example) will NOT be changed.
See also our technical tip:

"11g Grid Control: How to Change SSL Administration Ports for both WebLogic Admin Server and the Mid (Web) Tier (Apache)" on page 27 for further details to change the ports used to access the console.

Fix

1. Stop the OMS
   
   $./emctl stop oms


2. Use emctl set property command to set the following properties:
   
   oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort
oracle.sysman.emSDK.svlt.ConsoleServerPort
   Example:
   
   ./emctl set property -name oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort -value 1159
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
SYSMAN password:
Property oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort for oms <OMS-hostname>:4889_Management_Service has been set to value 1159

$./emctl set property -name oracle.sysman.emSDK.svlt.ConsoleServerPort -value 4890
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
SYSMAN password:
Property oracle.sysman.emSDK.svlt.ConsoleServerPort for oms <OMS-hostname>:4889_Management_Service has been set to value 4890

   

   IMPORTANT: Note that the "ConsoleServerPort" and "ConsleServerHTTPSPort" properties are NOT used for changing the port used to access the Grid Control Console on. There are different properties EMConsoleServerHTTPSPort and EMConsoleServerPort to do this.

   
   See our technical tip:
   
   

   "How to Change SSL Administration Ports for both WebLogic Admin Server and the Mid (Web) Tier (Apache)" on page 27 for further details.



3. Backup and Edit <INSTANCE_HOME>/emgc.properties.
   
   EM_UPLOAD_HTTP_PORT=4889
EM_UPLOAD_HTTPS_PORT=4900

To

EM_UPLOAD_HTTP_PORT=4890
EM_UPLOAD_HTTPS_PORT=1159

   Above step changes
   
   HTTP upload port to 4890 from 4889.
   HTTPS upload port to 1159 from 4900.
   


4. Update the "Listen" and "VirtualHost" directives in <INSTANCE_HOME>/WebTierIH1/config/OHS/ohs1/httpd_em.conf, replacing the previous port number with the new port number.
   
   <IfDefine SSL>
Listen 4900
<VirtualHost *:4900>
....
....
Listen 4889
<VirtualHost *:4889>

to

<IfDefine SSL>
Listen 1159
<VirtualHost *:1159>
....
....
Listen 4890
<VirtualHost *:4890>



5. Start the OMS and verify the new upload ports.
   
   $./emctl start oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server Successfully Started
Oracle Management Server is Up

$ /emctl status oms -details
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host :<OMS-hostname>
HTTP Console Port : 7800
HTTPS Console Port : 7801
HTTP Upload Port : 4890
HTTPS Upload Port : 1159
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1



6. Because the upload ports have been changed at OMS side, the agents emd.properties file also needs to be updated with the modified HTTP/S upload ports. This step needs to be carried out for all agents which upload to this OMS.
   • On the agent machine, backup and edit the following parameter in AGENT_HOME/sysman/config/emd.properties file
     
     REPOSITORY_URL=https://<OMS-HOSTNAME>:4900/em/upload
emdWalletSrcUrl=https://<OMS-HOSTNAME>:4900/em/wallets/emd

to

REPOSITORY_URL=https://<OMS-HOSTNAME>:1159/em/upload
emdWalletSrcUrl=https://<OMS-HOSTNAME>:1159/em/wallets/emd

   
   
   • Stop and Restart the agent
     
     $<AGENT_HOME>/bin/emctl stop agent
$<AGENT_HOME>/bin/emctl start agent
$<AGENT_HOME>/bin/emctl upload agent (to confirm that the upload is successful)


How To Delete Multiple Incidents In Enterprise Manager Cloud Control 12c

Modified 08-OCT-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

How to delete multiple incidents in Enterprise Manager Cloud Control 12c ?

Is there a way to clear multiple incidents rather than clear each incident individually?

For Example:

You have 222 incidents opened in 12c Cloud Control console related to invalid logins which need to be cleared.

Fix

You can clear multiple incidents via the command line with the command 'emcli clear_stateless_alerts'.
Clears the stateless alerts associated with the specified target.

Only the user can clear these stateless alerts; the Enterprise Manager Agent does not automatically clear these alerts.

To find the metric internal name associated with a stateless alert, use the get_metrics_for_stateless_alerts verb.

NOTE: Before running any emcli command, you need to log in to emcli first if not already done.
It may also be needed to synchronize the emcli and the OMS.

Example:

$ emcli login -username=SYSMAN
$ emcli sync


• older_than
  
  Specify the age of the alert in days. (Specify 0 for currently open stateless alerts.)
  


• target_type
  
  Internal target type identifier, such as host, oracle_database, and emrep.
  
  To get a list of valid target types in your repository database, you may do the following:
  
  
  1. Run SQL Plus and log in as sysman user to your repository database
  2. Run the following query:
     
     SQL> select distinct target_type from mgmt_targets;

     Sample output:
     
     SQL> select distinct target_type from mgmt_targets;

TARGET_TYPE
----------------------------------------------------------------
oracle_listener
oracle_ias_farm
weblogic_j2eeserver
j2ee_application
metadata_repository
oracle_csa_collector
host
oracle_database
oracle_emd
oracle_emrep
weblogic_domain



• target_name
  
  Name of the target.
  


• include_members
  
  Applicable for composite targets to examine alerts belonging to members as well.
  


• metric_internal_name
  
  Metric to be cleaned up. Use the get_metrics_for_stateless_alerts verb to see a complete list of supported metrics for a given target type.
  
  Example for the target type oracle_database
  
  $ emcli get_metrics_for_stateless_alerts -target_type=oracle_database



• unacknowledged_only
  
  Only clear alerts if they are not acknowledged.
  


• ignore_notifications
  
  Use this option if you do not want to send notifications for the cleared alerts. This may reduce the notification sub-system load.
  


• preview
  
  Shows the number of alerts to be cleared on the target(s).
  

How to Connect to the Weblogic Server Administration Console which ships with Enterprise Manager 11g Grid Control and Enterprise Manager 12c Cloud Control

Modified 09-NOV-2012

Applies to

Enterprise Manager Base Platform - Version 11.1.0.1 to 12.1.0.1.0 [Release 11.1 to 12.1]
Information in this document applies to any platform.

Goal

How to Connect to the Weblogic Server Administration Console (admin server console) which ships with Enterprise Manager 11g and Enterprise Manager 12c.

Fix

The weblogic server administration console should be accessible on the same hostname as the OMS, but using a different port.

By default the port is 7101

The full URL to use is:-

https://hostname.domain:<port>/console

https://machine1.uk.oracle.com:7799/em (Cloud/Grid Control URL)

sysman/oracle1

Then:-

https://machine1.uk.oracle.com:7101/console (Admin server URL)

weblogic/oracle1 (weblogic password is the same as the 'sysman' account password)

1. Check the <OMS_HOME>/install/setupinfo.txt file and check for "Admin Server Port"
   
   eg./oracle/12cloud/oms/install or /u01/app/Middleware/oms11g/install

   

   Use the following URL to access:
   
   1. Enterprise Manager Cloud Control URL: https://machine1.uk.oracle.com:7799/em
   2. Admin Server URL: https://machine1.uk.oracle.com:7101/console
   
   The following details need to be provided during the additional OMS install:
   
   1. Admin Server Hostname: machine1.uk.oracle.com
   2. Admin Server Port: 7101
   

   
   or
   


2. Go to all targets/choose "EMGC_ADMINSERVER" in Enterprise Manager. On the right hand side under "tools"(12c) or "summary"(11g) it says "To configure and manage this web logic server use the "Weblogic Server Administration Console". This is a hyperlink to the admin console.
   
   or
   


3. In Enterprise Manager 12c:-
   Click on all targets/Middleware
   
   

   

   
   Expand "EMGC_GCDomain" expand "GCDomain" highlight "EMGC_ADMINSERVER"
   
   

   

   
   Right click , choose "target setup"/"monitoring configuration"
   
   

   

   
   View "SSL Listen Port". There is also a hyperlink to the admin console.
   
   

   



4. In Enterprise Manager 11g:-
   
   • targets/Middleware
   • expand secFarm_GCDomain, expand secFarm_GCDomain/GC_Domain
   • click on /secFarm_GCDomain/GCDomain/EMGC_ADMINSERVER
   • right click , choose "target setup"/"monitoring configuration"
   • view "SSL Listen Port"
   
   or (if it is not possible to access the EMGC_ADMINSERVER target page)
   


5. Go to setup/agents. Choose the agent which monitors the OMS. Scroll down to "monitored targets". Highlight the target EMGC_ADMINSERVER. Click on "configure" which will show:
   
   SSL Listen Port
<port value>

   or (if Enterprise Manager cannot be started and so nothing can be checked in the console)
   


6. Check:- ..gc_inst/user_projects/domains/GCDomain/config/config.xml
   /oracle/12cloud/gc_inst/user_projects/domains/GCDomain/config
   
   <listen-port>7101</listen-port>
</ssl>
<log>
<name>EMGC_ADMINSERVER</name>

EM 12c: Agent Install Fails at pre-requisite check wth Error message "SSH daemon (sshd) is not running on port 22"

Modified 04-SEP-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

Enterprise Manager 12c Agent Install Fails at pre-requisite step with Error message "SSH daemon (sshd) is not running on port "22"

+ oms/sysman/prov/agentpush/<TIMESTAMP>/applogs/<hostname>_deploy.log

2012-06-13_13-46-32:INFO: Jsch Valdation Failed Problem :SSH server check failed Recommendation: Verify the value of SSH_PORT in
the /oms/oui/prov/resources/Paths.properties file. Ensure that it is the same as the port on which the sshd is running on the remote host.
2012-06-13_13-46-32:INFO:Updating Action SSHValidationswith Status FAILED and error Message :SSH daemon (sshd) is not running on port "22" and problem
SSH server check failed and recommendation Verify the value of SSH_PORT in the /var/app/oracle/em/oms/oui/prov/resources/Paths.properties file.
Ensure that it is the same as the port on which the sshd is running on the remote host.


Cause

This error can occur due to either of the below reasons:

• SSH port 22 is blocked by proxy / firewall on the Target machine
• SSH is not running on the Target machine
• SSH daemon is running on non-default port (other than 22)

Solution

This error can occur due to either of the below reasons:

1. Verify that the SSH port 22 is not blocked from the OMS machine using
   
   telnet <target host> 22



2. Check SSH is running on the target machine using
   
   ps -ef | grep ssh
netstat -anp | grep 22

   Prerequisites for 12c Agent installation clearly mentions that the SSH daemon should be running on the default port (that is, 22) on all the destination hosts.
   


3. If SSH on target machine is running on a non-default port, then update the SSH_PORT property in $<OMS_HOME>/oui/prov/resources/Paths.properties as below:
   
   SSH_PORT=<port number>

   • If you are installing on target A where SSH is running on port 99; then you would need to make the above change. If the next Agent Deployment is on target B where SSH is running on default port 22; then you will need to revert the changes to set SSH_PORT= or SSH_PORT=22 in $<OMS_HOME>/oui/prov/resources/Paths.properties file.
   • You cannot install Agents on target X and target Y in one-go, if they have SSH running on different ports.

NOTE: This error is also commonly encountered on Windows target host when Cygwin is not installed.

12c Cloud Control: How to Perform Control Operations on Single/Multiple Agents From Console UI?

Modified 16-FEB-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Goal

This document lists the steps for performing Control Operations (Stop, Start, Restart, Secure, Unsecure, Resecure) against a Single/Multiple Agents from the 12c Console UI.

Solution

Controlling a Single Agent

Control operations for a single Agent can be performed on the Agent home page for that Agent.

1. Navigate to Setup → Agents and click on the desired Agent name.
2. From the Agent drop down menu, choose Control and then one of the control operations (Start Up/Shut Down, or Restart)
   
   

   



3. Upon choosing any of the above control menu options, a pop-up dialog requesting the credentials of the user displays. These operations require the credentials of the OS user who owns the Agent. At this point, you can either choose from a previously stored username/password, preferred or named credential. You also have the option of choosing a new set of credentials which could also be saved as the preferred credential or as a named credential for future use. When storing a named credential or creating a new set of credentials, a Test button is shown, thus allowing you to verify if the credential information you just entered is valid. Once you are authenticated, the chosen control operation begins and continues even if the pop-up dialog is closed. Any message of failure/success of the task is displayed in the pop-up dialog.
   When choosing the Secure/Resecure/Unsecure options, you must provide the necessary Registration Password.
   

   NOTE: The EM user must have at least operator privilege on the Agent target in order to perform Agent control operations.

1. From the Setup menu, choose Agents. The Management Agent setup page displays.
2. Select multiple Agents from the list.
   
   

   

3. Click one of the control operation buttons (Start Up/Shut Down/Restart/Secure/Resecure/Unsecure). When you click on any of the control operations, you are taken to the Job creation wizard where you schedule a new job to perform the action on the selected Agents. Below screenshot shows 'Restart operation'
   
   

   



4. In the Jobs page, you can view the chosen Agents in Target section in the General tab. You can add more Agents by clicking the Add button. You then provide the parameters for the operation in the Parameters tab, if needed. The credentials must be specified in the Credentials tab where you can either choose from a previously stored username/password, preferred, or named credential. You also have the option of choosing a new set of credentials which could also be saved as the preferred credential or as a named credential for future use. You are given the option to start the job immediately or schedule the job for a later time. At this point, you can also create a repeating job by specifying the job start time, the frequency, and the end time.
   The Access tab displays the Administrator details and the access levels they have to the job. You can then add a new administrator or modify the access level to View or Full, if you have the necessary privileges.
   
   

   



5. Click on ''Submit", it will create a job as shown in the below screenshot:
   

   



6. Click on the job to view the status and click on "Log Report" to view the results of the job:
   

   

How to Setup the Patch Plan Prerequisites in Enterprise Manager Base Platform 12c

Modified 12-MAR-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Purpose

The Patch-Plan feature

Starting with Enterprise Manager Cloud Control 12c, the application of one-off patches for the 12c Management Agents will now be done via the Patch Plan feature.

One of the main benefits of applying patches this way, is that all the pre-checks, patch validation, and ORACLE_HOME validation will be performed prior to incurring target downtime.
Any roadblocks will be reported and can be fixed before attempting to install the patch.

The prerequisites for this feature are:

1. A Software Library that is configured and operational
2. The latest version of Opatch must be staged in the Software Library

How to Setup the Patch Plan Prerequisites in Enterprise Manager Base Platform 12c

Instructions for Setting Up the Software Library:

• Log in to the EM Cloud Control Console as a superuser
• Navigate to Setup → Provisioning and Patching → Software Library page
• If there is nothing configured yet, 'Add' an 'OMS Shared Filesystem' location. This is a location on a shared file system, that is 'visible' on all the OMS's
  
  

  



• When the location gets added, a background job will start which populates the software library with the components needed for Provisioning and Cloud Management features.
  

• Navigate to the Enterprise → Patching and Provisioning → Saved Patches page








• Make sure that the Patch 6880880 is available for every platform that is running an Agent (Linux, Solaris, HP, AIX, ....). The minimal version required to perform Agent patching is 11.1.0.9.4
• If this patch is not available, you will need to download it from My Oracle Support or upload it manually (Offline mode) to the Software Library.