In Cloud Control 12c The Job Refresh From My Oracle Support fails with Error ORA-01882: Timezone Region Not Found When Setting My Oracle Support Preferred Credentials

Modified 20-FEB-2013

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later
Enterprise Manager for My Oracle Support - Version: 10.2.0.5 to 12.1.0.1.0 [Release: 10.2 to 12.1]
Generic Linux
Generic Unix

Symptoms

When setting My Oracle Support Preferred Credentials in the EM 12c Cloud Control console, by applying the following steps:

• Setup → Security → Preferred Credentials
• Then click on the "My Oracle Support Preferrred Credentials"
• Under "My Oracle Support Preferred Credentials" click on "Set MOS Credentials" throws following error:

 Refresh From My Oracle Support - Error: ORA-01882: timezone region not found ORA-06512: at "SYSMAN.EM_JOB_SCHEDULER", line 822 ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 3409 ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 3602 ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 4615 ORA-01403: no data found ORA-06512: at "SYSMAN.EM_JOB_OPS", line 235 ORA-06512: at "SYSMAN.GC_JOB", line 29 ORA-06512: at line 1 
This error may also be seen in the status of the 'Refresh From My Oracle Support' job.

This error may also be seen in the status of the 'Opatch Update' job.

Cause

The OS Time zone is set to a non-standard setting.

The problem is that the 'MYT' timezone shown with the 'date' command does not match any of those in the repository database.

 $date Sat Dec 17 03:29:22 MYT 2011  The following query run against the Repository database returns no rows for the 'MYT' time zone:  SQL> SELECT TZname, TzAbbrev FROM V$TIMEZONE_NAMES where TZname like '%MYT%' ORDER BY TzAbbrev; no rows selected 
The OS Timezone needs to match one of those returned by the following query against the repository database:

 SQL> SELECT TO_CHAR(SYSTIMESTAMP,'TZR') FROM DUAL UNION SELECT DISTINCT tzname FROM v$timezone_names;  Solution 1. Please run this query against the Repository Database:  SQL> SELECT TO_CHAR(SYSTIMESTAMP,'TZR') FROM DUAL UNION SELECT DISTINCT tzname FROM v$timezone_names; 

2. Coordinate with your sysadmins to adjust the operating system timezone to match one of the above accordingly, the one that your database is set to.
The operating system timezone and database timezone need to be the same. Indeed the database and operating system Timezone entries were not in sync.

To achieve it, the files touched/referenced are:

/usr/share/zoneinfo
/etc/sysconfig/clock
/etc/localtime

Create a symbolic link for the Date/time:

Fix

Ensure Grid Control is patched to PSU3 or Higher (Latest PSU would be preferable)

Apply patch 10297881 to the agent.

Then go to the metrics and thresholds page and add the key_value for resource_ora.gsd.type_ora.gsd. Set threshold to "not defined".

If required add resource_ora.oc4j.type_ora.oc4j and set threshold to "not defined".

Apply the changes

Then EM will not alert only for these resources.

EM 12c: Installing the Enterprise Manager Cloud Control 12.1.0.1 Agent through the Add Target Host Feature Results in a Directory Creation Error

Modified 06-SEP-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1 and later
Information in this document applies to any platform.

Symptoms

Installing the Cloud Control 12c agent through the Add Target Host Feature fails at agent start with the error:

Transferring Agent Software to Destination Host. The error is "Creation of directory /oracle/agent12c on host xxx Failed. Error Message:Not Available Exit Code : 1. Recommendation Not Available.

 FileName ---------------- ui.log 2012-03-22_14-11-25:INFO: HOST : abc.com Intialization : FAILED Remote Prereq status : NULL INSTALL status : NULL 2012-03-22_14-11-25:INFO:Inside setDetailsBean :groupid group1 hostname dbtst3.uark.edu 2012-03-22_14-11-25:INFO:Dep vo map for host dbtst3.uark.eduis {Initialization= [AdaDeploymentVo: status: FAILED [AdaActionVo: actionName: SSHValidations recommendation: null errorStream: null status: SUCCESS outStream: null AdaActionVo] [AdaActionVo: actionName: CopyAgentImage recommendation: Not Available errorStream: Error Message:Not Available Exit Code :1 status: FAILED outStream: null AdaActionVo] AdaDeploymentVo]} 2012-03-22_14-11-25:INFO:Inside Build Progress Bar : Session Ended true Refresh Frequency 30000 Progress Bar Message Initialization Failed 2012-03-22_14-11-25:INFO:local prereqs failed/images/title_error.png Failed 2012-03-22_14-11-25:INFO:Since all nodes failed. Disable Continue ignoring failed host 2012-03-22_14-11-25:INFO:==================END populateCurrentSessionData ==================:2012-03-22_14-04-25-PM 
Deploy.log:

 2012-07-13_15-18-44:INFO:/bin/sh -c 'if [[ -d /oracle/agent_inst ]] ; then exit 0; else exit 1; fi' execution failed on node155.213.223.133 2012-07-13_15-18-44:INFO: ACTION Execution of command /bin/sh -c 'if [[ -d /oracle/agent_inst ]] ; then exit 0; else exit 1; fi' on host 155.213.223.133 2012-07-13_15-18-44:INFO: OUT 2012-07-13_15-18-44:INFO: ERR 2012-07-13_15-18-44:INFO: EXIT CODE1 2012-07-13_15-18-49:INFO: EXIT CODE2 2012-07-13_15-18-49:INFO: ERR /bin/ls: /oracle/agent_inst: No such file or directory 2012-07-13_15-18-49:INFO: EXIT CODE2 2012-07-13_15-18-49:INFO:Printing Exception :CommandException: err: /bin/ls: /oracle/agent_inst: No such file or directory out: exitcode: 2 stacktrace: null 2012-07-13_15-19-09:INFO:LOGGING XML LOC:/oracle/EM/middleware/oms/oui/prov/resources 2012-07-13_15-19-09:INFO:RemoteInterfaces startup called with REMOTENODES_PLATFORM=226 and REMOTE_TEMP_LOCATION=/tmp 2012-07-13_15-19-10:INFO:checkSharedPath:Init RI with REMOTE_PATH_PROPERTIES_LOC= /oracle/EM/middleware/oms/oui/prov/resources 2012-07-13_15-19-10:INFO:RUN AS not set during startup 2012-07-13_15-19-10:INFO:CLUSTER EXCEPTION found in startup 2012-07-13_15-19-10:INFO:Printing Exception :oracle.sysman.prov.remoteinterfaces.exception.ClusterException: Check for remote command execution setup failed on nodes 155.213.223.133. at oracle.sysman.prov.remoteinterfaces.nativesystem.NativeSystem.startup(NativeSystem.java:538) at oracle.sysman.prov.remoteinterfaces.clusterops.ClusterBaseOps.startup(ClusterBaseOps.java:425) at oracle.sysman.prov.remoteinterfaces.clusterops.ClusterBaseOps.startup(ClusterBaseOps.java:338) 2012-07-13_15-19-12:INFO:=========Command Exception :Error Message:Not Available Exit Code :1 2012-07-13_15-19-12:INFO:Updating Action CopyAgentImagewith Status FAILED and error Message :Error Message:Not Available Exit Code :1 and problem Creation of directory /oracle on host 155.213.223.133 Failed and recommendation Not Available 

Cause

Checking the ssh connectivity between the OMS and the target server using this command

<OMS_HOME>/oui/prov/resources/scripts/sshUserSetup.sh -setup -user oracle -hosts <hostname>

Shows this result:

 sshUserSetup logs clearly says that "Checking if the remote hosts are reachable" 
From OMS host

 + ping <Agent Machine Name> >>oracle@oem ~ $ping abc.domain.com PING abc.domain.com (10.10.10.10) 56(84) bytes of data. --- abc.domain.com ping statistics --- 1361 packets transmitted, 0 received, 100% packet loss, time 1361060ms  Solution Agent deployment from the Cloud Control 12c console requires effective interaction through ssh as described in this reference: Oracle(R) Enterprise Manager Cloud Control Basic Installation Guide 12c Release 1 (12.1.0.1) Part Number E22624-09 Chapter 8 Installing Oracle Management Agent Section titled: Before You Begin and Table 8-1 Prerequisites for Installing Oracle Management Agent Download here Ensure ssh is configured for effective OMS host interaction with the agent hosts as described there. EM 12c Emctl Start OMS Returns Error Occurred: Cannot Run Program "/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not Enough Space Modified 25-APR-2012 Applies to Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later Information in this document applies to any platform. Symptoms When trying to start or stop OMS , It fails with the following errors:  ./emctl start oms Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. Starting WebTier... Error Occurred: Cannot run program "/u01/app/oracle/product/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space Please check /u01/app/oracle/product/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details  The following errors are found in <Middleware_Home>/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log:  2011-07-23 08:58:32,447 [main] ERROR wls.OMSController main.259 - OMSController failed for start oms 2011-07-23 08:58:32,449 [main] ERROR wls.OMSController main.260 - OMSController Error: Cannot run program "/u01/app/oracle/product/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space java.io.IOException: Cannot run program "/u01/app/oracle/product/gc_inst/WebTierIH1/bin/opmnctl": error=12, Not enough space at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at java.lang.Runtime.exec(Runtime.java:593) at java.lang.Runtime.exec(Runtime.java:431) at java.lang.Runtime.exec(Runtime.java:369) at oracle.sysman.emctl.util.WebTierUtil.execCmd(WebTierUtil.java:136) at oracle.sysman.emctl.util.WebTierUtil.stopWebTier(WebTierUtil.java:117) at oracle.sysman.emctl.util.WebTierUtil.startWebTier(WebTierUtil.java:122) at oracle.sysman.emctl.wls.OMSController.startOMS(OMSController.java:521) at oracle.sysman.emctl.wls.OMSController.main(OMSController.java:214) Caused by: java.io.IOException: error=12, Not enough space  Cause The Swap space was configured less.  %swap -s total: 6708280k bytes allocated + 0k reserved = 6708280k used, 1680328k available  Less than 2Gb was not enough for stopping/starting the oms stack. Solution Make sure following things are taken care: Allocate more Swap space. At least minimum of 4 to 5Gb should be available. So that:  %swap -s total: 6565136k bytes allocated + 0k reserved = 6565136k used, 6017776k available  And: Make sure the location where OMS Instance Home resides (<Middleware_Home>/gc_inst), has enough free space. 12c Cloud Control: Which WLS Log Files Can be Removed / Purged Manually at Regular Intervals in 12c OMS Installation for Space Considerations? Modified 19-APR-2012 Applies to Enterprise Manager for Oracle Database - Version 12.1.0.1.0 and later Information in this document applies to any platform. Goal In a 12c OMS installation, automatic log rotation has been enabled for all the OMS and WLS logs after they reach a certain size. Hence, creation of very large log/trace files should not be a concern. But, there are certain logs which are not purged automatically after rotation and these files have to be manually deleted at regular intervals. This document lists the log files which need to be purged manually on a periodic basis in a 12c OMS installation. Fix On the OMS machine, the following log files need to be manually purged: • Webtier Logs in <EM_INSTANCE_HOME>/WebTierIH1/diagnostics/logs/OHS/ohs1: • em_upload_http_access_log.* • access_log.* • em_upload_https_access_log.* • ohs1-*.log • console~OHS~1.log* • mod_wl_ohs.log* • Domain Logs: • <EM_INSTANCE_HOME>/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs/GCDomain.log* • EMGC_ ADMINSERVER Logs in <EM_INSTANCE_HOME>/users_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs: • EMGC_ADMINSERVER.out* • EMGC_ADMINSERVER.log* • access.log* • Managed Server Logs: • <EM_INSTANCE_HOME>/users_projects/domains/GCDomain/servers/EMGC_OMSn/logs/EMGC_OMS*.out* where <EM_INSTANCE_HOME> refers to the location of <gc_inst> directory. NOTE: 1. In an additional OMS installation, there are no Domain logs or EMGC_ADMINSERVER logs as the Adminserver runs on the primary OMS machine. So only the Webtier and Managed server out files need to be purged. 2. Automatic purging of older Domain logs and EMGC_ADMINSERVER logs can be enabled using the steps in our next technical tip: "How to Enable Log Rotation Policy to Automatically Delete Older GCDomain.log, EMGC_ADMINSERVER.log and access.log Files?" on page 15 Timeout Setting Mos Credentials in Oracle em12c Modified 16-SEP-2012 Applies to Enterprise Manager for My Oracle Support - Version 12.1.0.1.0 and later This problem can occur on any platform. Symptoms Cloud Control is unable to connect against My Oracle Support. The test against https://updates.oracle.com succeeds. But setting the preferred credentials for My Oracle Support will fail with: Timeout occurred while reading data from My Oracle Support Cause Proxy settings.are needed for connecting to My Oracle Support The settings in Setup → Proxy Settings are only done for the HTTP proxy, but no settings are done for HTTP settings Unlike the tip showing on this page the HTTPS proxy must be configured, too. Even if it is the same proxy server and port Solution Edit the proxy settings in Setup → Proxy Settings for Manual proxy configuration. • Let the settings for the HTTP proxy untouched • Add settings for HTTPS proxy Apply the changes. Do the settings of the My Oracle Support Credentials in Setup → My Oracle Support → Credentials again. Automatic Diagnostic Cleanup - Auto purge Modified 10-JAN-2012 Applies to Oracle Server - Enterprise Edition - Version: 11.1.0.7 and later [Release: 11.1 and later ] Information in this document applies to any platform. Goal How to increase the frequency of log auto purges in adrci? Solution Diagnostic data Purging in ADR is controlled by two attributes: • SHORTP_POLICY which is used for automatically purging short-lived files, i.e. dump files, expressed in hours and defaults to 30 days. • LONGP_POLICY which is used for automatically purging long-lived files, i.e. incidents, expressed in hours and defaults to 1 year. Viewing Purging policies: You can view Purging policies for short-lived and long-lived ADR contents using the SHOW CONTROL command, from within ADRCI.  adrci> show control  ADR Home = /u01/app/oracle/diag/rdbms/orc11107/orc11107: ************************************************************************* ADRID SHORTP_POLICY LONGP_POLICY LAST_MOD_TIME LAST_AUTOPRG_TIME LAST_MANUPRG_TIME ADRDIR_VERSION ADRSCHM_VERSION ADRSCHMV_SUMMARY ADRALERT_VERSION CREATE_TIME -------------------- -------------------- -------------------- ---------------------------------------- ---------------------------------------- --------------- 3963614641 720 8760 2010-08-06 13:18:00 +03:00 2010-08-27 03:25:21 +03:00 1 2 65 1 2010-08-06 13:18:00 +03:00 1 rows fetched Changing purging policies: You can change Purging policies using the SET CONTROL command, from within ADRCI:  adrci> set control (SHORTP_POLICY = 360) //Sets short-lived policy to 15 Days adrci> set control (LONGP_POLICY = 2160) //Sets long-lived policy to 3 Months  Some Oracle products, such Oracle Database, automatically purge diagnostic data at the end of its lifecycle. Other products and components require you to purge diagnostic data manually with the Purge command. Manual purging: You can use the PURGE command to purge data that is due to be automatically purged, or to purge all diagnostic data based on type, incident numbers and age. Syntax: purge [[-i {id | start_id end_id}] | [-age mins [-type {ALERT|INCIDENT|TRACE|CDUMP|HM}]]] Examples  This example purges all diagnostic data in the current ADR home based on the default purging policies: adrci> purge This example purges all diagnostic data for all incidents between 123 and 456: adrci> purge -i 123 456 This example purges all incident data from the last hour: adrci> purge -age 60 -type incident  11g Understanding Automatic Diagnostic Repository Modified 27-MAY-2012 Applies to Oracle Server - Enterprise Edition - Version 11.1.0.6 and later This problem can occur on any platform. Purpose Beginning with Release 11g, Oracle Database includes an advanced fault diagnosability infrastructure for preventing, detecting, diagnosing, and resolving problems. This note helps to understand the new 11g infrastructure known as Automatic Diagnostic Repository. Scope Audience: Oracle 11g Users and DBAs who want to know about ADR. Details What is New in 11g? In 11g, RDBMS diagnostic data has been reorganized and is stored inside a common directory structure, the Automatic Diagnostic Repository (ADR). An ADR is a centralized directory structure where one can find trace files, alert messages, incident dumps, core files, etc. Automatic Diagnostic Repository (ADR): All trace files, core files, and the alert files are now organized into a directory structure comprising the Automatic Diagnostic Repository (ADR). The ADR is a file-based repository for database diagnostic data. It has a unified directory structure across multiple instances and multiple products. Beginning with Release 11g, the database, Automatic Storage Management (ASM), Cluster Ready Services (CRS), and other Oracle products or components store all diagnostic data in the ADR. Each instance of each product stores diagnostic data underneath its own ADR home directory. For example, in an Oracle Real Application Clusters environment with shared storage and ASM, each database instance and each ASM instance has a home directory within the ADR. ADR's unified directory structure, consistent diagnostic data formats across products and instances, and a unified set of tools enable customers and Oracle Support to correlate and analyze diagnostic data across multiple instances. Problems and Incidents: Problem: is a critical error in the database Eg : ORA-600 , ORA-7445 , ORA-4031 etc. Problem key : Every problem has a problem key, which is a text string that includes an error code (such as ORA-600) and in some cases, one or more error parameters. Eg : ORA-600, ORA-4030 Incident : is a single occurance of a problem . Each incident has a numeric incident id. Where is ADR located: The location of the ADR is controlled by the Oracle "diagnostic_dest" parameter. Path specified in the "diagnostic_dest" parameter defines the ADR root directory,ADR BASE. The first subdirectory inside an ADR (under the <ADR_BASE> directory) is always named "diag" For example, if the "diagnostic_dest" and thus the <ADR_BASE> is specified as "$ORACLE_HOME/log", then expect to find the subdirectory "$ORACLE_HOME/log/diag". Below this will be <ADR_HOME>. Any number of instances/components can share same ADR BASE. Under ADR BASE there will be individual ADR HOMES. Under ADR BASE ,the address of an <ADR_HOME> will be similar to: "diag/<product_type>/<prod_id>/<instance_id>". Inside each ADR home, you can find several subdirectories, each for storing a specific type of diagnostic data. Among the subdirectories, you should be able to find TRACE, ALERT, INCIDENT, CDUMP etc. ADR HOME contents: You will find the following directories under ADR HOME - ALERT: The alert directory contains the XML alert log CDUMP: core dumps are stored in this directory TRACE: Process trace files and Alert.log are stored in the trace directory. "Background_dump_dest" and "user_dump_dest" are now ignored in 11g. Now all the trace files will be generated in "trace" directory. INCIDENT: The incident directory stores dump files created when critical errors are encountered. Each occurrence of a critical error (incident) is given its own incident directory, with the incident ID used to form the directory name. METADATA: The metadata directory stores a series of files that contain diagnostic metadata. HM: The HM directory stores reports for health checks INCPKG,IR ,LCK, SWEEP : These directories contain internal diagnosability framework state. DIAGNOSTIC_DEST - Default value: If environment variable ORACLE_BASE is set, DIAGNOSTIC_DEST is set to the directory designated by ORACLE_BASE. If environment variable ORACLE_BASE is not set, DIAGNOSTIC_DEST is set to ORACLE_HOME/log. V$DIAG_INFO:

For each database , you can query V$DIAG_INFO to check its ADR locations. This shows us the ADR BASE , ADR home , trace file locations , XML Alert location , incident dump locations , core dump and health monitor reports. Also gives us the default session trace (for the current session ) and number of problems , incidents reported in the database. The potential issues are categorized into problem groups and a complete overview of particular errors can quickly be obtained. Each problem is associated with a single error, such as ORA-00600 (with specific arguments), ORA-04031, etc., and can consist of one or more so-called incidents (occurrences of the problem). Details can be retrieved for both a problem and individual occurrences of a problem. ADR Command Interpreter (ADRCI): ADR is accessible via either a User Interface called Support Workbench from the Enterprise Manager UI, or via a command line utility called ADRCI. The following example focuses on the usage of the command line utility ADRCI to collect data about a specific problem. When ADRCI is started, the utility needs to be informed which Oracle diagnostic information should be used to analyze an issue. As indicated, several Oracle products are using the ADR and for each product one or more subdirectories exist in the ADR, such as individual directories for the various instances running on the system, a subdirectory for the listener, etc. The SHOW HOMES and SET HOME commands can be used to select which data subset should be accessed: $ adrci ADR base = "/u01/app/oracle" adrci> show homes ADR Homes: diag/rdbms/v1120/v1120 diag/tnslsnr/oel5/listener adrci> set home diag/rdbms/v1120/v1120 
To get an overview of the various issues reported in the ADR for that environment, the SHOW PROBLEM command can be used:

 adrci> show problem ADR Home = /u01/app/oracle/diag/rdbms/v1120/v1120: ************************************************************************************* PROBLEM_ID       PROBLEM_KEY       LAST_INCIDENT      LASTINC_TIME ---------------- ----------------- -------------- ---------------------------------- 1               ORA 600 [16613]     3818            2012-02-01 09:23:09 +01:00 1 rows fetched 
The PROBLEM_KEY column (or PROBLEM_ID, but the key is more explanatory) is used to obtain details about various occurrences
of the issue using the SHOW INCIDENT command:

 adrci> show incident -p "problem_key='ORA 600 [16613]'" ADR Home = /u01/app/oracle/diag/rdbms/v1120/v1120: ************************************************************************** INCIDENT_ID       PROBLEM_KEY         CREATE_TIME ---------------- -------------------- ------------------------------------ 3817               ORA 600 [16613]      2012-02-01 09:20:34 +01:00 3818               ORA 600 [16613]      2012-02-01 09:23:09 +01:00 2 rows fetched 
In this case the ORA-600 [16613] was raised twice. Each incident has trace files and associated incident trace files generated.
To gather information about particular incidents for use in analysis of a Service Request, the following command can be used to
pack this information in a single archive using the command IPS PACK:

 adrci> ips pack incident 3817 Generated package 1 in file /home/oracle/database/ORA600166_20120201093056_COM_1.zip, mode complete 
The generated package ORA600166_20120201093056_COM_1.zip contains all relevant information required to start analyzing the issue:

• the trace file(s)
• the associated incident trace file(s)
In case the package should be created in a different directory other than the current directory, use the "IN" clause:

 adrci> ips pack incident 3817 in /tmp Generated package 2 in file /tmp/ORA600166_20120201093743_COM_1.zip, mode complete 
The traces of an entire issue can also be packed using:

 adrci> ips pack problem 1 in /tmp Generated package 4 in file /tmp/ORA600166_20120201094023_COM_1.zip, mode complete 

NOTE: the ORA-600 [16613] issue in this example has PROBLEM_ID 1 (as listed in the previous SHOW PROBLEM output).

These generated packages can be uploaded as an attachment to a Service Request via My Oracle Support for
detailed analysis of the issue. Whenever opening a Service Request with Oracle Global Software Support,
providing the package which contains all relevant trace information is highly recommended to avoid delays in resolution.

12c Cloud Control: How to Change the Default Login Timeout Value for the Enterprise Manager Cloud Console Connections

Modified 01-JAN-2013

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.1.0 [Release 12.1]
Information in this document applies to any platform.

Goal

This document explains how to change the Default Login Timeout Value which defines the maximum idle time after
which you will have to log in again to the Cloud Control console.

Solution

This configuration task is described in the following documentation:
Oracle Enterprise Manager Cloud Control Advanced Installation and Configuration Guide
Topic "Modifying the Default Login Timeout Value"

For your convenience, the steps are provided below:

To prevent unauthorized access to the Cloud Control console, Enterprise Manager will automatically log you out of the
Cloud Control console when there is no activity for a predefined period of time.
For example, if you leave your browser open and leave your office, this default behavior prevents unauthorized users from using your

By default, if the system is inactive for 45 minutes or more, and then you attempt to perform an Enterprise Manager action,

CAUTION: As stated in the previous paragraphs, the timeout value for logging in to the Cloud Control console is defined in order to protect your system from unauthorized logins.
If you make changes to the login timeout value, be sure to consider the security implications of leaving your session open for other than the default timeout period.

To increase or decrease the default timeout period:

1. Set the value for the OMS ORACLE_HOME environment variable
Example:

 $export ORACLE_HOME=/u01/app/oracle/product/Middleware/oms  2. Navigate to the directory OMS$ORACLE_HOME/bin
3. Issue the following command to increase the timeout to 60 minutes for example:

 $./emctl set property -name oracle.sysman.eml.maxInactiveTime -value 60 -sysman_pwd oracle12  4. Stop and restart the OMS to reflect the new property value $ ./emctl stop oms 
 $./emctl start oms  12c Cloud Control EM CLI: How to Install and Setup 12c EMCLI (Enterprise Manager Command Line Interface)? Modified 08-OCT-2012 Applies to Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.1.0 [Release 12.1] Information in this document applies to any platform. Goal This document lists the steps for installing and setting up Enterprise Manager Command Line Interface (emcli) in 12c Cloud Control. The EM CLI enables you to access Enterprise Manager Grid Control functionality from text-based consoles (shells and command windows) for a variety of operating systems. You can call Enterprise Manager functionality using custom scripts, such as SQL*Plus, OS shell, Perl, or Tcl, thus easily integrating Enterprise Manager functionality with a company's business process. Using EM CLI, you can perform Enterprise Manager Grid Control console-based operations, such as monitoring and managing targets, jobs, groups, blackouts, notifications, and alerts. EM CLI is intended for use by enterprise or system administrators writing scripts, such as shell/batch files, Perl, Tcl, or PHP, that provide workflow in the customer's business process. You can also use EM CLI commands interactively from an operating system console. EM CLI is fully integrated with Enterprise Manager's security and user administration functions, enabling you to carry out operations using EM CLI with the same security and confidentiality as the Enterprise Manager Grid Control console. NOTE: - You should always use the latest version of EM CLI Client corresponding to the OMS version and not a lower version of EM CLI. - EM CLI does not allow OS Script jobs to be run against database targets. The Enterprise Manager Cloud Control console, however, does allow this. - EM CLI has only been certified for submitting OS Script and SQL Script jobs. - To avoid an uncommon occurrence in which multiple EM CLI sessions are created on the OMS, Oracle recommends that you enter the login command before running a script containing EM CLI commands. Fix The EM CLI Client can be installed on any machine within your managed network and provides a subset of the functionality available in the Enterprise Manager Cloud Control console. Prerequisites Before installing EMCLI, we need to have the following in place: • An existing 12c Cloud Control setup. • Workstation where EM CLI will be installed should have JDK version1.6.0_25 or higher. • Workstation where EM CLI will be installed should be running on the following platforms - Linux, Solaris, HPUX, Tru64, AIX, Windows with NTFS (client installation). Steps to Install 12c EM CLI Follow the steps provided below: • Download the 12c EM CLI Kit to your workstation using the following URL: https://omsmachine.domain:port/em/console/emcli/download  For example: https://omsmachine.domain:7801/em/console/emcli/download  • Set the JAVA_HOME environment variable and make sure that it is part of your PATH variable: Make sure that this variable is set to the home of a JDK 1.6.0_25 or greater.  For example: export JAVA_HOME=/u03/jdk16/jdk1.6.0_27 export PATH=$JAVA_HOME/bin:$PATH  • Install the EM CLI client in any directory: java -jar emclikit.jar client -install_dir=<emcli client directory>  For example: java -jar emclikit.jar client -install_dir=/home/oracle/emcli  NOTE: By default, emclikit.jar file available under OMS_HOME/sysman/jlib We can user the same jar file as another option. Setting up the 12c EM CLI Client After the EM CLI Client is installed, you are ready to begin using EM CLI. At this point, you can run the EM CLI Client out of the installation directory location, or alternatively, you can add it to your PATH. Immediately after installation, only basic operational verbs are installed: help: Access command-line help for EM CLI verbs. login: Log in and establish a session with the OMS. logout: Log out of EM CLI client from Enterprise Manager. setup: Configure EM CLI to function with a specific OMS. status: Show EM CLI setup details. sync: Synchronize the EM CLI Client with an OMS. version: List EM CLI verb versions or the EM CLI client version. argfile: Execute an EM CLI verb where the verb and any arguments are contained in a file. add_group_to_mpa: Add a Management Plug-in group to a Management Plug-in Archive. add_mp_to_mpa: Create (or add to) the Management Plug-in Archive. The Management Plug-in Archive is available for adding new target types to Enterprise Manager. You must run setup to connect the EM CLI Client to the OMS running the EM CLI Management Services. Running the setup verb installs all available verb-associated command-line help from the EM CLI Management Service. You must run setup each time you want to connect to a different OMS. • Setup the EM CLI client using the following command: emcli setup -dir=<directory of emcli> -url=<OMS URL> -user=<EM username for OMS on client machine>  For example: emcli setup -dir=/u03 -url=https://omsmachine.domain:7799/em -user=SYSMAN  • EMCLI Client can be configured to work with multiple OMS by doing multiple setups as shown below: • Setup EMCLI client for OMS1 at location dir1 emcli setup -dir= -url=<URL of OMS1> -user=<EM Username for OMS1> • Setup EMCLI client for OMS2 at location dir2 emcli setup -dir=<dir2> -url=<URL of OMS2> -user=<EM Username for OMS2> • Set the environment variable EMCLI_STATE_DIR to point to the setup dir for the OMS1 setenv EMCLI_STATE_DIR <dir1> This will allow the EM CLI Client to work with OMS1 • Set the environment variable EMCLI_STATE_DIR to point to the setup dir for the OMS2 setenv EMCLI_STATE_DIR <dir2> This will allow the EM CLI Client to work with OMS2. Known Issues: 1. After setting up the EM CLI client, trying to verify the emcli version shows the EMCLI version as 11.1.0.1.0 as shown below: This is because, though the 12.1.0.1.0 EM CLI version was setup, as the 11.1 emcli client is in the PATH and is getting picked up. To prevent this, remove the 11.1 emcli from the PATH environment variable and verify the EM CLI version, it should provide the expected output: 2. The 'emcli sync' command fails with:  Error: Session expired. Run emcli login to establish a session.  This can occur if the emcli was not setup correctly. Refer to our other technical tip: "Cloud Control 12c emcli sync fails with SEVERE: oracle.sysman.emCLI.omsbrowser.OMSBrowserException" on page 7 Cloud Control 12c emcli sync fails with SEVERE: oracle.sysman.emCLI.omsbrowser.OMSBrowserException Modified 05-DEC-2011 Applies to Enterprise Manager Base Platform - Version 12.1.0.1.0 and later [Release 12.1 and later] Information in this document applies to any platform. Symptoms OMS Version: 12.1.0.1 The command emcli login completes successfully. The command emcli sync fails with:  Error: Session expired. Run emcli login to establish a session.  The file MIDDLEWARE_HOME/gc_inst/em/EMGC_OMS1/sysman/emcli/setup/.emcli/.emcli.log shows:  Nov 4, 2011 12:44:21 PM oracle.sysman.emCLI.verb.SetupVerb execute INFO: SETUP: localdir=/u01/app/oracle/product/Middleware/gc_inst/em/EMGC_OMS1/sysman/emcli/log, omsurl=https://<hostname>:7801/em, username=SYSMAN, trustall=true, validate=true, autologin=false, certvalidate=false [...] Nov 14, 2011 2:13:22 PM oracle.sysman.emCLI.verb.SynchronizeVerb execute SEVERE: oracle.sysman.emCLI.omsbrowser.OMSBrowserException at oracle.sysman.emCLI.omsbrowser.LoginSystem.establishSession(LoginSystem.java:218) at oracle.sysman.emCLI.omsbrowser.OMSBrowser.getPageInternal(OMSBrowser.java:653) at oracle.sysman.emCLI.omsbrowser.OMSBrowser.getPage(OMSBrowser.java:595) at oracle.sysman.emCLI.verb.SynchronizeVerb.synchronize(SynchronizeVerb.java:280) at oracle.sysman.emCLI.verb.SynchronizeVerb.execute(SynchronizeVerb.java:162) at oracle.sysman.emSDK.emCLI.CLIController.execute(CLIController.java:540) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at oracle.sysman.emSDK.emCLI.CLIController.go(CLIController.java:343) at oracle.sysman.emSDK.emCLI.CLIController.main(CLIController.java:157)  Cause emcli was not setup correctly at the Cloud Control installation time. The first lines of the file emcli.log show that emcli was configured with the short name of the host where the OMS has been installed.  INFO: SETUP: localdir=/u01/app/oracle/product/Middleware/gc_inst/em/EMGC_OMS1/sysman/emcli/log, omsurl=https://<hostname>:7801/em, username=SYSMAN, trustall=true, validate=true, autologin=false, certvalidate=false  emcli should have instead been setup with the fully qualified host name: hostname.domainName. The file emcli.log should have:  INFO: SETUP: localdir=/u01/app/oracle/product/Middleware/gc_inst/em/EMGC_OMS1/sysman/emcli/log, omsurl=https://<hostname.domainname>:7801/em, username=SYSMAN, trustall=true, validate=true, autologin=false, certvalidate=false   Example: hostname.domainname = mycloudhost.mydomain.com. INFO: SETUP: localdir=/u01/app/oracle/product/Middleware/gc_inst/em/EMGC_OMS1/sysman/emcli/log, omsurl=https://mycloudhost.mydomain.com:7801/em, username=SYSMAN, trustall=true, validate=true, autologin=false, certvalidate=false  Solution To fix the problem, please execute: From the host where the OMS is running: Setup emcli with the fully qualified name of the host where the OMS is running $ cd $OMS_ORACLE_HOME/bin$ ./emcli setup -url=https://<OMS Fully Qualified Host Name>:<Console Port>/em -username="sysman" -password="<your password>" -trustall 
Example:

 $cd /u01/app/oracle/product/Middleware/oms/bin$ ./emcli setup -url="https://myhostname.mydomainname:7801/em" -username=sysman -password=XXXX -trustall 

12c Cloud Control EM CLI: 12c: How to delete Agent/targets using EMCLI Delete Verb

Modified 01-JAN-2013

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

How to delete an Agent (and all its monitoried targets) or a specific target in EM 12c using EMCLI verb?

Fix

1. To delete an Agent and all its targets, we can use the following command:

 $<EMCLI_HOME>/emcli delete_target -name="hostname:3872" -type="oracle_emd" -delete_monitored_targets -async  NOTE: The above deletes the Agent and ALL the monitored targets from the repository/console. "-delete_monitored_targets" : Delete the targets monitored by the specified Management Agent. Applicable only with the oracle_emd target type. 2. To delete any specific target:$<EMCLI_HOME>/emcli delete_target -name="<target_name>" -type="<target_type>" -delete_monitored_targets -async

 Example: $./emcli get_targets -targets="DBname:oracle_database"$./emcli delete_target -name="DB1" -type="oracle_database" -delete_monitored_targets -async 
3. How to De-install the Enterprise Manager Cloud Control 12c Agent

Modified 22-FEB-2013

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.2.0 [Release 12.1]
Information in this document applies to any platform.

Purpose

This document provides Steps to Deintall Oracle Management Agent (Intelligent Agent) using various methods. It includes the following:

• Prerequisites
• Deinstalation Steps
• Post Deinstallation Steps

Scope

This document describes in detail the Steps for Deinstalling Oracle Enterprise Manager Cloud Control 12c and Oracle Management Agent (Management Agent) 12c using a Graphical and Silent methods.

Details

1. Before you deinstall a Management Agent, do the following:

1. Stop the Agent using command from Management Agent home:

$emctl stop agent  Example: ../agent12c/agent_inst/bin/emctl stop agent  2. Wait for the Management Agent to go to the unreachable state in the Cloud Control console. 3. It is mandatory to delete the Management Agent and their monitored targets using EMCLI as explained in our previous technical tip on this page.  Example:$ emcli login -username=SYSMAN $emcli sync$ emcli delete_target -name="example.com:1836" -type="oracle_emd" -delete_monitored_targets  Or

Manually remove an agent from 12C Cloud Control as explained in our next technical tip:

"EM 12C: How to Manually Remove an Agent From 12C Cloud Control" on page 9

2. Deinstalling using Graphical Method:

1. Invoke the installer from Management Agent home by running the following command:

$<AGENT_HOME>/oui/bin/runInstaller -deinstall ORACLE_HOME=<absolute_path_to_agent_home> [-removeallfiles]  Example:$ /u01/app/oracle/agent/core/12.1.0.1.0/oui/bin/runInstaller -deinstall ORACLE_HOME=/u01/app/oracle/agent/core/12.1.0.1.0 -removeallfiles 

2. In the Installation wizard click on "Installed Products" button.
3. On the Inventory screen, select the plug-in homes, and click "Remove" button.
4. On the Inventory screen, select the sbin home, and click "Remove" button.
5. On the Inventory screen, select the Management Agent, and click "Remove" button.

6. Or

3. Deinstalling using Silent Method:

1. Deinstall the plug-in homes:

$/oui/bin/runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={absolute_path_to_plug-in_home}" -invPtrLoc  Example:$ ../agent/core/12.1.0.1.0/oui/bin/runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={../agent/plugins/oracle.sysman.emas.oms.plugin_12.1.0.1.0, ../agent/plugins/oracle.sysman.emct.oms.plugin_12.1.0.1.0}" -invPtrLoc /home/oracle/oraInst.loc 

2. Deinstall the sbin home:

$/oui/bin/runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={absolute_path_to_sbin_directory}" -invPtrLoc  Example:$ ../agent/agent_inst/oui/bin/runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={../agent/sbin}" -invPtrLoc /home/oracle/oraInst.loc 

3. Deinstall the Management Agent:

$/oui/bin/runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={absolute_path_to_agent_oracle_home}" -invPtrLoc  Example:$ ../agent/core/12.1.0.1.0/oui/bin/runInstaller -silent -deinstall -removeallfiles "REMOVE_HOMES={../agent/core/12.1.0.1.0}" -invPtrLoc /home/oracle/oraInst.loc 

NOTE: Parameter -invPtrLoc is optional
For Windows: Instead of runInstaller use setup.exe

4. (Only for Graphical Mode) Verify whether the Oracle Homes and other directories were successfully deinstalled. To do so, follow these steps:

1. Invoke the installation wizard by running the following command from the Management Agent Home:

$/oui/bin/runInstaller  Example:$ ../agent/core/12.1.0.1.0/oui/bin/runInstaller 

2. In the installation wizard, on the My Oracle Support Details screen, click on "Installed Products" button.
3. On the Inventory screen, check whether or not the Oracle Homes and other directories you deinstalled appears. If the deinstallation was successful then those Oracle Homes and directories should not appear.

5. Open inventory.xml and check Agent plugin home and Agent sbin home entries are removed or not. These entries should be removed during deinstallation.

NOTE: inventory.xml would be located under ContentsXML folder under the inventory location in oraInst.loc file.

6. Remove the Cloud Control Management Agent base directory:

For UNIX platforms:

$rm -rf <absolute_path_to_agent_base_dir>  Example:$ rm -rf /u01/app/oracle/product/agent12c 
For Microsoft Windows platforms:

C:\app\oracle> del <absolute_path_to_agent_base_dir>

 Example: C:\app\oracle> del c:\app\oracle\product\agent12c 

EM 12C: How to Manually Remove an Agent From 12C Cloud Control

Modified 29-JAN-2013

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

How to manually remove a 12.1.0.1 Agent in 12C Cloud Control.

Fix

Perform the following steps to delete an Agent from 12C Cloud Control:

NOTE: For Cloud Control 12c Release 12.1.0.1.0 - Before Deleing the Agent from Console, ensure that you delete the targets monitored by this agent.

1. Ensure that the agent on the target machine is not running:

 <AGENT_HOME>/bin> emctl status agent <AGENT_HOME>/bin> emctl stop agent 

2. Remove the Agent target manually from the console as follows:

1. Login to 12C Cloud Control
2. Navigate to Setup → Manage Cloud Control → Agents (Setup → Agents In cloud Control 12c Release 12.1.0.1)
3. Go to the Home page of the Agent that is to be deleted
4. Expand the drop-down menu near the "Agent"
5. Expand the "Target Setup" option
6. Select "Remove Target"
7. Following confirmation dialogue box will be displayed (may be slightly different for different Cloud Control 12c Releases):

"You have chosen to remove : (Agent). Do you want to proceed?"

Select "yes" to delete the agent.

In Cloud Control Release 12.1.0.2, the following dialog box will be displayed if the Agent is still monitoring targets:

Click Continue. You can then remove all targets (using the "Select All" link) in the next page.
You will then be able to remove the host/agent itself.

After Deleting the Agent and its targets from console, refer to our technical tip:

"12C Cloud Control: How to De-install 12C Agent" for deinstallation of Agent" on page 8.

NOTE: Ensure that all targets monitored by the Agent are removed including the host target; before proceeding with the Agent Deletion.
If the Agent is still monitoring any other targets, then the following error will be displayed in the console:

Unknown error.java.sql.SQLException: ORA-20242: Target myhost.oracle.com:3875:oracle_emd is monitoring other targets. It cannot be deleted
ORA-06512: at line 1

12c Cloud Control: Installation Fails with oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: java.sql.SQLException: ORA-04030: out of process memory

Modified 14-JUN-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

When installing 12c Cloud Control,but it failed at repository configuration stage.

Getting following error in rcu.log

 Caused by: oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: java.sql.SQLException: ORA-04030: out of process memory when trying to allocate 103504 bytes (QERHJ hash-joi,kllcqc:kllcqslt) ORA-06512: at "SYS.DBMS_AQADM", line 81 ORA-06512: at line 15 ORA-01403: no data found 

 Errors in diag/rdbms/omsrepo/omsrepo/trace/omsrepo_j001_2375780.trc (incident=7086): ORA-04030: out of process memory when trying to allocate 123416 bytes (QERHJ hash-joi,kllcqas:kllsltba)  Use ADRCI or Support Workbench to package the incident.

Cause

This error

 java.sql.SQLException: ORA-04030: out of process memory when trying to allocate 103504 bytes (QERHJ hash-joi,kllcqc:kllcqslt) 
caused by OS limit has been reached preventing oracle processes from allocating needed memory.

This can be checked by the following

ulimit -a
ulimit -ha

Sample out puts

 oracle@abc:/home/oracle$ulimit -a time(seconds) unlimited file(blocks) unlimited data(kbytes) 31072 stack(kbytes) 32768 memory(kbytes) 32768 coredump(blocks) 2097151 nofiles(descriptors) 4096   oracle@abc:/home/oracle$ ulimit -Ha time(seconds) unlimited file(blocks) unlimited data(kbytes) unlimited stack(kbytes) 4194304 memory(kbytes) unlimited coredump(blocks) unlimited nofiles(descriptors) 4096 

Solution

1. Increase the ulimits for stack to unlimited ulimit -s unlimited.
2. Make sure the ulimit -Hs is also set to unlimited.
3. Be sure to shutdown and restart the oracle server and listener so that the new ulimits take effect.
4. Re Try Instalation.

OCM fails to discover the Oracle Management System (OMS) 11.1.0.1.0

Modified 04-JAN-2012

Applies to

Oracle Configuration Manager - Version: 10.3.2.0 and later [Release: and later]
Oracle Configuration Manager - Version: 10.3.2.0 to 10.3.6.0.0
Information in this document applies to any platform.

Symptoms

You have installed successfully Oracle Configuration Manager 10.3.3.1.0 in the Grid Control Oracle Management Service (OMS) 11.1.0.1.0 ORACLE_HOME.
You check the xml files collected in the directory under the OMS $ORACLE_HOME/ccr/host/<host_name>/state/review. You notice that there are only the following files : • BB4D4FBF880D5CF02365405AA59D32D3-oracle_home_config.xml • 949EAACD431225800C0BA2CBFA7D0BB5-livelink_config.xml • DADDD6B3873D0957DD406B28E51DB598-ll_host_config.xml • targetMap.xml The file <config_id>-oracle_oms_config.xml is missing. If you view the file targetMap.xml, you notice that there is no target of type "Oracle Management Server" discovered as shown in the example below:  <?xml-stylesheet type="text/xsl" href="/u01/app/oracle/product/Middleware/oms11g/ccr/admin/xsl/preview_targets.xsl"?> <Targets collection_time="2010-06-10 09:54:25" host_name="myhost.mydomain.com"> <Target name="em2.us.oracle.com" type="Host"> <Collection name="ll_host_config" file="DADDD6B3873D0957DD406B28E51DB598-ll_host_config.xml" collection_timestamp="2010-06-10 09:54:25 America/New_York"/> </Target> <Target name="Oracle Configuration Manager" type="Oracle Configuration Manager"> <Collection name="livelink_config" file="949EAACD431225800C0BA2CBFA7D0BB5-livelink_config.xml" collection_timestamp="2010-06-10 09:54:40 America/New_York"/> </Target> <Target name="oms11g1" type="Oracle Home"> <Collection name="oracle_home_config" file="BB4D4FBF880D5CF02365405AA59D32D3-oracle_home_config.xml" collection_timestamp="2010-06-10 09:54:49 America/New_York"/> </Target> </Targets>  Cause Beginning with the Grid Control release 11.1.0.1.0, the configuration properties of the OMS are stored in the file emgc.properties. OCM needs to browse this file in order to discover the OMS and collects data. OCM will look for this file in the directory specified by the environment variable ORACLE_CONFIG_HOME. If this environment variable is not set when OCM is configured executing the command setupCCR, OCM will fail to discover the OMS. Solution In order to fix the issue, you need to follow the steps below: 1. Uninstall OCM. You need to uninstall OCM as the previous configuration removed the file setupCCR. 1. From the OMS$ORACLE_HOME/ccr/bin directory execute:

 $deployPackages -d$ORACLE_HOME/ccr/inventory/core.jar 

2. Remove the ccr directory and all directories and files under this directory

 $rm -rf$ORACLE_HOME/ccr (on Unix) 
 c> rmdir /s/q %ORACLE_HOME%\ccr (On Windows) 
2. Re-install OCM. Unzip the OCM zip file to the OMS $ORACLE_HOME 3. Look for the location of the file emgc.properties The location of the file emgc.properties is stored in the file OMS$ORACLE_HOME/sysman/config/emInstanceMapping.properties.

 Example: EMGC_OMS1=/u01/app/oracle/product/gc_inst/em/EMGC_OMS1/emgc.properties 

4. Set the environment variable ORACLE_CONFIG_HOME.

 Example: $export ORACLE_CONFIG_HOME=/u01/app/oracle/product/gc_inst/em/EMGC_OMS1  5. Configure OCM from the OMS$ORACLE_HOME/ccr/bin directory:

 $./setupCCR  NOTE: In a multiple OMS Home environment, steps 1-5 above needs to be completed on each location also. This keeps all OMS locations in sync. The data collected are stored in xml files which you can review from the following directory:$ORACLE_CONFIG_HOME/ccr/state/review

 Example: /u01/app/oracle/product/gc_inst/em/EMGC_OMS1 
Content of this directory:

• <config_id>-oracle_home_config.xml
• <config_id>-oracle_oms_config.xml
• <config_id>-ll_host_config.xml
• targetMap.xml

The file targetMap.xml file now looks like:

 <?xml-stylesheet type="text/xsl" href="/u01/app/oracle/product/Middleware/oms11g/ccr/admin/xsl/preview_targets.xsl"?> <Targets collection_time="2010-06-14 15:31:25" host_name="myhost.mydomain"> <Target name="em2.us.oracle.com" type="Host"> <Collection name="ll_host_config" file="E780DEE9DC7A6E46C680686B476370E4-ll_host_config.xml" collection_timestamp="2010-06-14 15:31:25 America/New_York"/> </Target> <Target name="Oracle Configuration Manager" type="Oracle Configuration Manager"> <Collection name="livelink_config" file="31BDD5A4EE9482F150569E40ABC262FE-livelink_config.xml" collection_timestamp="2010-06-14 15:31:39 America/New_York"/> </Target> <Target name="oms11g1" type="Oracle Home"> <Collection name="oracle_home_config" file="2DE8CB83B0EEF9AE220B13DE03D19337-oracle_home_config.xml" collection_timestamp="2010-06-14 15:31:46 America/New_York"/> </Target> <Target name="myhost.mydomain:4889_Management_Service" type="Oracle Management Server"> <Collection name="oracle_oms_config" file="584933EC9E3C084CBF8F2D15536ADB04-oracle_oms_config.xml" collection_timestamp="2010-06-14 15:31:47 America/New_York"/> </Target> 
You will need to set the environment variable ORACLE_CONFIG_HOME prior launching any ccr command, otherwise you will get an error message like:

Running the Script

1. Make sure the $ORACLE_HOME environment variable is set to the agent oracle home before running the script: $ export ORACLE_HOME=<path to agent home> 

2. Make sure that the agent is running:

 $ORACLE_HOME/bin/emctl status agent  3. Run the script in the background: $ cd $ORACLE_HOME/bin$ ./agent_snap.sh & 

4. Verify that the agent_snap.log file is created under <AGENT_HOME>/sysman/log or <AGENT_HOME>/<nodename>/sysman/log (for RAC agents)
and populated with the expected information.
5. The script is set to collect data by default every 120 seconds. You can change the interval as required by modifying the following line of the script:

 sleep 120 
Leave the script running for as long as you need to gather relevant information for the problem you are diagnosing.
Note that you can customize the script as needed to run other commands to capture additional information every 120 seconds.

6. To stop the script - identify the process id and kill it:

 $ps -ef | grep agent_snap.sh$ kill -9 <pid of agent_snap.sh> 

NOTE: The script was tested on Linux, Oracle Solaris & AIX. It may need customization on other Unix platforms.

Caution

This script is provided for educational purposes only and not supported us. It has been tested internally, however, and works as documented.
We do not guarantee that it will work for you, so be sure to test it in your environment before relying on it.
Proofread this script before using it! Due to the differences in the way text editors, e-mail packages and operating systems
handle text formatting (spaces, tabs and carriage returns), this script may not be in an executable state when you first receive it.
Check over the script to ensure that errors of this type are corrected.

Scripts

1. agent_snap.sh for Linux (Link to shell script)

 #!/bin/sh emhome=$ORACLE_HOME/bin/emctl getemhome | tail -1 | awk 'BEGIN {FS="="} {print$2}' echo -e "date\t\tcpu%\tvmem\trmem\tthreads\trunning\tready\tscheduled" >> $emhome/sysman/log/agent_snap.log while [ 2 -gt 1 ] do pid=$ORACLE_HOME/bin/emctl status agent|grep "Agent Process ID" | awk '{print $5}' dateStr=date "+%m/%d %H:%M:%S" cpu=ps -p$pid -o "pcpu"| tail -1 | sed 's/^[ ]*//' threads=ps -p $pid -o "thcount"| tail -1 | sed 's/^[ ]*//' vmem=ps -o vsz$pid | tail -1 | sed 's/^[ ]*//' rmem=ps -o rss $pid | tail -1 | sed 's/^[ ]*//'$ORACLE_HOME/bin/emctl status agent scheduler > /tmp/snapschedule running_line=grep -n "Running entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}' ready_line=grep -n "Ready entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print$1}' scheduled_line=grep -n "Scheduled entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}' final_line=grep -n "Agent is" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print$1}' running=expr $ready_line -$running_line - 1; ready=expr $scheduled_line -$ready_line - 1; scheduled=expr $final_line -$scheduled_line - 2; echo -e "$dateStr\t$cpu\t$vmem\t$rmem\t$threads\t$running\t$ready\t$scheduled" >> $emhome/sysman/log/agent_snap.log sleep 120 done  2. agent_snap.sh for Oracle Solaris (Link to shell script)  #!/bin/sh emhome=$ORACLE_HOME/bin/emctl getemhome | tail -1 | awk 'BEGIN {FS="="} {print $2}' echo "date\t\tcpu\tvmem\trmem\tthreads\trun\tready\tsched" >>$emhome/sysman/log/agent_snap.log while [ 2 -gt 1 ] do pid=$ORACLE_HOME/bin/emctl status agent|grep "Agent Process ID" | awk '{print$5}' dateStr=date "+%m/%d %H:%M:%S" cpu=ps -p $pid -o "pcpu"| tail -1 | sed 's/^[ ]*//' threads=ps -p$pid -o "nlwp"| tail -1 | sed 's/^[ ]*//' vmem=ps -p $pid -o "vsz" | tail -1 | sed 's/^[ ]*//' rmem=ps -p$pid -o "rss" | tail -1 | sed 's/^[ ]*//' $ORACLE_HOME/bin/emctl status agent scheduler > /tmp/snapschedule running_line=grep -n "Running entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print$1}' ready_line=grep -n "Ready entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}' scheduled_line=grep -n "Scheduled entries" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print$1}' final_line=grep -n "Agent is" /tmp/snapschedule | awk 'BEGIN {FS=":"} {print $1}' running=expr$ready_line - $running_line - 1; ready=expr$scheduled_line - $ready_line - 1; scheduled=expr$final_line - $scheduled_line - 2; echo "$dateStr\t$cpu\t$vmem\t$rmem\t$threads\t$running\t$ready\t$scheduled" >>$emhome/sysman/log/agent_snap.log sleep 120 done 

3. agent_snap.sh for AIX (Link to shell script)

 For Example: $ORACLE_HOME/bin/rcuJDBCEngine sys/welcome1@reposmachine.domain:1521:emrep JDBC_SCRIPT post_install_script.sql$PWD $ORACLE_HOME  You should get the message "Completed SQL script execution normally." - as an indication that the script was run successfully. 12c Cloud Control: How to Verify the Connectivity from 12c OMS to Repository Database using rcuJDBCEngine Modified 14-SEP-2012 Applies to Enterprise Manager Base Platform - Version 12.1.0.1.0 and later Information in this document applies to any platform. Purpose This document describes the steps for using rcuJDBCEngine in the 12c OMS home for testing the connectivity between the OMS and the Repository Database. This can be useful in troubleshooting OMS startup failures due to connectivity problems with the Repository Database. rcuJDBCEngine has to be used as the sqlplus utility is no longer shipped in the 12c Cloud Control installation as described in our technical tip: "Sqlplus Utility is not Shipped with 11g Grid Control and 12c Cloud Control Installation" on page 11 Solution 1. In OMS machine, set the below environment variables: export MW_HOME=<Location where the 10.3.5 Weblogic is software installed> export ORACLE_HOME=<path of the 12c OMS installation> 2. Create a file named: test.sql in the$ORACLE_HOME/bin with the following SQL statement:

 select sysdate from dual; 

3. Usage for the rcuJDBCEngine is:

The above usage will be deprecated soon, so the following usage is recommended.

2. rcuJDBCEngine <user>@<service> <JDBC_BLOCK|JDBC_SCRIPT|SQLSERVER_SCRIPT|DB2_SCRIPT> <SQL-script-file> [-f < filename]

In this, user will be prompted for the password. To pass the database password via a file, use -f < filename option as the last parameter
The service should be in following format for the Repository DB:

<hostname>:<port>:<service_name> or the entire connect descriptor

4. Execute the test.sql using rcuJDBCEngine:

cd $ORACLE_HOME>/bin ./rcuJDBCEngine sysman@<Repository Hostname>:<Listener Port>:<Repository Database SID> JDBC_SCRIPT test.sql This will prompt you for the sysman password. For example:  ./rcuJDBCEngine sysman/@reposmachine.domain:1521:emrep JDBC_SCRIPT test.sql Please enter database password. 2012-04-09 21:10:59.309 rcu:Extracting Statement from File Name: 'test.sql'Line Number: 1 2012-04-09 21:10:59.310 rcu:Extracted SQL Statement: [select sysdate from dual] 2012-04-09 21:10:59.310 rcu:Statement Type: 'DDL Statement' Completed SQL script execution normally. 1 scripts were processed  The above output indicates that the connection to Repository Database is successful and the script was executed successfully but the sql output will not be shown on the terminal. 5. If the connection is not successful, corresponding errors will be shown: - If the Repository hostname/the Listener Port value is wrong/if Listener is not up: >A SQL Error occurred!: oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: Unable to connect to the database using the provided details. Please enter a valid hostname and port or check if the listener is up and running. at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.connect(JDBCEngine.java:532) at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.main(JDBCEngine.java:2345) - If the Repository Database is not up: >A SQL Error occurred!: oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: Invalid SID or Service name. Please enter valid SID or Service name. at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.connect(JDBCEngine.java:532) at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.main(JDBCEngine.java:2345) - If the SYSMAN Password is incorrect: >A SQL Error occurred!: oracle.sysman.assistants.common.dbutil.SQLFatalErrorException: Invalid username/password. Please enter valid username/password at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.connect(JDBCEngine.java:532) at oracle.sysman.assistants.common.dbutil.jdbc.JDBCEngine.main(JDBCEngine.java:2345) ADRCI PURGE Command Is Not Cleaning Up CDUMP_<TIMESTAMP> Folders In TRACE Directory Modified 07-FEB-2013 Applies to Oracle Server - Enterprise Edition - Version 11.1.0.6 to 11.1.0.6 [Release 11.1] Information in this document applies to any platform. Oracle Server Enterprise Edition - Version: 11.1.0.6 Symptoms For this article there are no symptoms in terms of errors seen in alert or within the session. Instead for this article to be applicable the user will be trying to delete folders using the ADRCI tool under Oracle11g 11.1.0.6, the problem is only seen on this particular release and is resolved in the 11.1.0.7 patchset. There is no clear documentation to show how such directories can be removed using ADRCI so to demonstrate the problem and solution an example is now provided. The example is specific to a particular machine/instance so cannot be simulated by a testcase. The purpose of the example is simply to highlight syntax and usage of ADRCI: 1. navigate to the directory containing the instance trace information:  cd /app/oracle/diag/rdbms/v1116u/V1116U/trace  The path will depend on the installation. If unsure of on the location, issue the following query against the database instance:  SELECT VALUE FROM V$DIAG_INFO WHERE NAME LIKE 'Diag cdmp%'; 

2. At the OS command prompt issue the directory listing command, like in:

 $ls -ltr | grep cdmp drwxr-xr-x 2 oracle dba 4096 Nov 11 11:54 cdmp_20081111115441/ drwxr-xr-x 2 oracle dba 4096 Nov 13 09:10 cdmp_20081113091031/ drwxr-xr-x 2 oracle dba 4096 Nov 20 11:23 cdmp_20081120112348/ drwxr-xr-x 2 oracle dba 4096 Nov 27 06:52 cdmp_20081127065231/ drwxr-xr-x 2 oracle dba 4096 Nov 27 06:52 cdmp_20081127065232/ drwxr-xr-x 2 oracle dba 4096 Nov 27 06:52 cdmp_20081127065235/  3. initiate the ADRCI utility: $ adrci 

4. at the ADRCI command prompt display all Oracle Homes maintained by this ADR:

 adrci> show homes 

5. If (d) shows more than 1 home, select the proper home for use with ADRCI:

 adrci> set homepath diag/rdbms/v1116u/V1116U adrci> set home diag/rdbms/v1116u/V1116U 
The SET command here depends on (a). If (c) already shows diag/rdbms/v1116u/V1116U then no action needed.

6. find out what incidents are reported for this instance:

 adrci> show incident INCIDENT_ID PROBLEM_KEY CREATE_TIME ----------- ----------------------------- ------------------------------ 39828 ORA-4031 2008-11-27 06:52:26 +00:00 39820 ORA-4031 2008-11-27 06:52:28 +00:00 39796 ORA-4031 2008-11-27 06:52:27 +00:00 39780 ORA-4031 2008-11-27 06:52:32 +00:00 39748 ORA-4031 2008-11-27 06:52:29 +00:00 31411 ORA-7445 [mdagun_term()+1031] 2008-11-20 11:23:43 +00:00 6 rows fetched 

7. try to purge the evidence of the incident 31411:

 adrci> purge -i 31411 
'show incident' will then show the 5 remaining rows but still 'ls -ltr' shows six CDMP_<timestamp> directories.

Cause

The cause of this problem has been identified as a bug. It is caused by the ADR of 11.1.0.6 not taking into account the purging if CDMP directories.

Solution

In Oracle 11g 11.1.0.7 and higher releases a new option 'UTSCDMP' has been added to ADRCI's PURGE command:

 adrci> help purge Usage: PURGE [[-i <id1> | <id1> <id2>] | [-age [-type ALERT|INCIDENT|TRACE|CDUMP|HM|UTSCDMP]]]: ... 
The UTSCDMP type has been added by the bug fix to allow purge of the CDMP related information; e.g.:

 adrci> purge -age 1440 -type UTSCDMP 
This will delete the CDMP_<timestamp> directories from the TRACE directory which are more than 1 day old (1440 minutes = 1 day).
When available, download Patch 6343743 from Oracle Support to resolve this issue.

Modified 16-DEC-2011

Applies to

Oracle Server - Enterprise Edition - Version: 11.2.0.2 and later [Release: 11.2 and later]
Information in this document applies to any platform.

Symptoms

After upgrading the software to 11.2.0.2 patchset, ADRCI purge command starts to fail with errors:

DIA-48322: Relation [INCIDENT] of ADR V[2] incompatible with V[2] tool
DIA-48166: error with opening ADR block file because file does not exist [/grdbms/64bit/app/oracle/diag/tnslsnr/rmtdcsol1/listener/metadata/INCIDENT.ams] [0]

when trying to purge the listener Home.
This home does not contain any activity to be purged:

*************************************************************************
0 rows fetched

Cause

These errors are raised when you run a purge command in the listener ADR Home without incidents to purge,
and after an upgrade to the patch set level is done.

Solution

This feature will be enhances in future releases.
At present the errors could be safely ignored, but to prevent them the following workaround can be used:

1. Stop listener

Eg.
% cp -pr <listener's ADR directory> <backupfile>

Eg.
% rm -fr <listener's ADR directory>

4. Start listener

This will create listener's ADR directory automatically

5. Retry the purge command, which should return no errors:

*************************************************************************
0 rows fetched

Understanding Database Discovery in Enterprise Manager 12cCloud Control

Modified 16-FEB-2013

Applies to

Enterprise Manager for Oracle Database - Version: 12.1.0.1.0 and later
Information in this document applies to any platform.

Purpose

This document is to discuss the different methods of discovering a database in Enterprise Manager Cloud Control 12c,
relevent background information and the pre-requisites necessary to achieve this.

Scope

It is intended for Enterprise Manager Administrators who are famiilar with the previous versions of Enterprise Manager.

Details

1. Background Information

There are three ways to discover a database in Enterprise Manager 12c Cloud Control.

• Automatic discovery
• Manual discovery via Console
• Manual discovery via command line

Database discovery in Enterprise Manager 12c Cloud Control, differs to the previous versions of Enterprise Manager
(Grid control 11.1.0.1, Grid Control 10.2.0.X and 9.2.0 OEM), in the sense that database targets which are discovered
automatically are not populated in the console. Database/listener targets are auto-discovered but not displayed in the
console until the user chooses to "promote" that particular database/listener target.

The concept of "target promotion" is new in 12c but involves selecting the databases to be displayed in the console,
and configuring them for monitoring by entering the dbsnmp username/password (configuring them is similar to "configuring"
a database in 11g grid control). This removes the possibility of the Administrator seeing unwanted or unconfigured targets
in the Console. This means that when a new agent is added to Enterprise Manager Cloud Control 12c, the database targets which
run on that agent machine will not be shown automatically in the console.

As well as the automatic discovery method, there is also a possibility to add the database manually either using the console,
or via emcli (enterprise manager command line interface). When adding a database manually via the console, only one database
at a time can be added (so again, this differs from the "add database" mechanism in previous versions of Enterprise Manager).
Only the automatic discover method requires target promotion, the manual methods do not require promotion because the targets
are manually chosen and configured by the user and so the necessary dbsnmp connect info has already been supplied.

PLUGINS

Another new concept in 12c is the concept of the database 'plugin'. Any database which is monitored by Cloud Control needs to use the
"database plugin". The database plugin (version 12.1.0.1.0 with no revisions) is automatically deployed on the OMS by default.
The database plugin also needs to be deployed on any management agent which needs to manage a database. The database plugin can either
be deployed by the user to the management agent (via the setup/extensibility/plugins screen) or it will be deployed automatically at the
time when the first database target to be managed by an agent is added, as part of the add database/promote database process.

If subsequently it is a requirement to have a later version of the plugin (eg. 12.1.0.2.0) installed on the agent.
The existing plugin can be upgraded from the "Plugins" console.

Note that if intending to deploy the latest plugin on an agent, this latest version of the plugin needs to first be deployed on the OMS,
before it can be deployed on the remote agent.

If a database is discovered by an agent that has already previously discovered a database, the database plugin will already be installed on
that agent in which case, the version of the database plugin on the OMS, and the version of the database plugin on the monitored agent, may be different.
The version of the plugin installed on an agent can be checked from:-

Setup/Extensibility/Plugins
expand "Databases" folder
Highlight "Oracle Database"
go to Actions:Information
click "Management agent" tab

The plugins deployed on the agent can also be checked by:-

cd $ORACLE_HOME<of agent> /oracle/12cagent/core/12.1.0.2.0 [oracle@machine1 12.1.0.2.0]$ emctl listplugins agent
Oracle Enterprise Manager Cloud Control 12c Release 2
---------------------------------------------------------------
oracle.sysman.emas 12.1.0.3.0 /oracle/12cagent/plugins/oracle.sysman.emas.agent.plugin_12.1.0.3.0
oracle.sysman.emrep 12.1.0.2.0 /oracle/12cagent/plugins/oracle.sysman.emrep.agent.plugin_12.1.0.2.0
oracle.sysman.beacon 12.1.0.2.0 /oracle/12cagent/plugins/oracle.sysman.beacon.agent.plugin_12.1.0.2.0
oracle.sysman.oh 12.1.0.2.0 /oracle/12cagent/plugins/oracle.sysman.oh.agent.plugin_12.1.0.2.0
oracle.sysman.csa 12.1.0.2.0 /oracle/12cagent/plugins/oracle.sysman.csa.agent.plugin_12.1.0.2.0
oracle.sysman.db 12.1.0.2.0 /oracle/12cagent/core/12.1.0.2.0/../../plugins/oracle.sysman.db.agent.plugin_12.1.0.2.0

Note that the plugins are not platform dependent in the sense that there is not a separate plugin for each Operating System. However, multi OS support was not available until the 111221 revision of the 12.1.0.1.0 database plugin (12.1.0.1.0 [u111221].)

Periodically later versions of the database plugin (eg. 12.1.0.2.0) or later revisions of the database plugin are released

PLUGIN REVISIONS

A revision of a database plugin indicates that there is a new revised version of the *same* plugin. Revisions only affect target certifications, so there would not be a need to update an existing remote agent with the latest revision. The plugin revisions are needed if database monitoring for a particular platform newly becomes available, and the plugin revision adds support for that platform, so the latest revision plugin would be needed on the OMS. However, existing agents would not need this revision, because they would already have been installed on their own (different) platform. A plugin revision is shown in the console with the revision number in square brackets for example: 12.1.0.1.0 [u111221].

SELF UPDATE

These new versions/revisions of the database plugin are obtained via EM Cloud Control's 'Self Update' feature (setup/extensibility/self update). Self update automatically runs a job that regularly checks for updates. The cloud control administrator will be emailed automatically (if email notification has been set up) when any new plugins are available (via the incident rule "Event management Ruleset for Self Update". It is also possible to run an on-demand search of the latest updates by clicking on "check updates" this will run a job called "SELFUPDATE" which does a 'RefreshFromEMStore'.

2. Pre-requisites for database discovery

1. There must be a 12c Management Agent (eg. 12.1.0.1.0 or 12.1.0.2.0) on the remote host (where the database target resides). Previous versions of the agent are not compatible with Enterprise Manager 12c. If there is no agent, it can be deployed using instructions from Section C.
2. The database plugin which currently exists on the OMS must support the OS platform and version of database of the database to be discovered. Check this by:

setup/extensibility/plugins
click on the arrow to the left of the "Databases" folder and expand it
Highlight "Oracle Database"
Pick "Actions:Information"
view "General" tab

This will show the versions of the databases which the plugin can manage, and the OS versions which are certified. Note that the default database plugin (12.1.0.1.0 with no revisions) does not support any other platform than Linux. If the platform desired is not shown, also check the certification information on 'My Oracle Support', to clarify whether that platform is certified:-

'My Oracle Support'/Certifications/Enterprise Manager Base Platform - Agent: version 12.1.0.1.0

If the platform is certified, it may be that a later version or revision of the database plugin is needed. Multi platform support (other than Linux) was introduced in database plugin revision 12.1.0.1.0 [u111221].
This can be obtained from Cloud Control itself via the 'self update' feature (Setup/Extensibility/Self Update). Self update will always indicate the latest version of a plugin. See point B3 for more info.

3. If the current database plugin installed on the OMS does not support the required platform, either a later version database plugin, or a revision of the database plugin is needed. This new revision/version of the plugin will first need to be downloaded and deployed on the OMS.

See our technical tip: "How to Deploy the Latest Database Plugin to the OMS and the Agent in 12C Cloud Control" on page 44

3. Getting the Agent

1. If there is already an agent running on the host where the remote database is go to Section D
2. If there is no agent on the remote host, the agent can be deployed to the remote host as follows:-

After agent has been added, go to section D.

NOTE: To see the results of the agent deployment (whether successful or not, go to setup/add target/add targets manually/click "add hosts results" button. Click on the hyperlink under "session name".)

3. If unable to "add host target" because the "Agent software unavailable" message is shown for the necessary platform.

The "agent software unavailable" message may be seen because by default the OMS will have the agent software for the platform the OMS was installed on.
If the agent to be deployed is on a different platform than the OMS, then it is necessary to obtain the agent software for that platform.

The agent software can be obtained from Enterprise Manager Cloud Control itself via self update. (Setup/Extensibility/Self Update/ under folder "Agent Software").

Look at the status (either "applied, available or apply failed".

What do the Status mean:

Available: means that the software for that agent is available to download from the EMStore at Oracle, but has not yet been downloaded or applied.

Applied: means that the software for that agent has been downloaded and applied (therefore it should be possible to install an agent on that particular platform)

Apply failed: means that there was an attempt to download and apply that software to the OMS Software Library, but the apply failed.

If "apply failed" is seen, highlight it, at the bottom of the screen the "past activities" pane will show and there will be a status against the operations "available/download and apply".
If any operation has a "failed" (eg. apply:failed) click on ""details", which should drill down to the error message.

"12c: Plugin Deployment Fails With Plug-in is not certified for the Operating System" on page 42
"How to Deploy the Latest Database Plugin to the OMS and the Agent in 12C Cloud" on page 44

4. Discovering the Database

Once the agent exists on the remote machine (where the databases reside) the database can be discovered in one of the following ways:-

• Guided Process
• Manual Discovery
• Automatic Discovery (and "run discovery now")
• Manual via EMCLI

1. Guide Process

Discovery via Guided Process is done via:-

 Setup/Add targets/Add targets manually Add Non-Host Targets Using Guided Process (Also Adds Related Targets) Target types:Oracle Database, Listener and Automatic Storage Management Click "add using guided discovery" 
Or the guided process discovery can be done via:-

 targets/databases/add choose hostname click on 'continue' 
This will find all databases and their listeners on a particular host. The databases will be added to the console, once they have been configured by the user.
'configuring' involves entering the password for the dbsnmp account, and then finishing the dialogue.

2. Manual Discovery

Alternatively a complete manual discovery can be done. This option adds only one database. It is called "manually", because the necessary configuration information (such as database name, port number etc)
must be provided by the user. This can be useful when needing to add an individual database or listener, but if multiple databases need to be discovered it would be much quicker to use guided process or automatic discovery.

Adding a database manually is discussed in detail in our technical tip "How To Manually Add Database Targets in 12C Cloud Control" on page 45. This note includes screen shots and troubleshooting tips.

Note that if the database/listener/asm to be added will be the first database/listener/asm target to be discovered by that management agent, and that management agent has not previously had the database plugin manually deployed to it, then the database plugin will be automatically deployed as part of the add database/listener/asm process. The database plugin will be deployed to $ORACLE_HOME/plugins/oracle.sysman.db.agent.plugin_12.1.0.1.0 This means that the first time a database/listener or asm target is added on a new agent, there could be a small delay whilst the database plugin is deployed. 3. Automatic Discovery of Target Discover the Databases By default when the OMS discovers a new agent only the host,the agent and the Oracle Home target are automatically shown in the Enterprise Manager 12c Cloud Control Console. The other targets (eg. database/listener/ASM) need to be discovered. To discover the targets it's necessary to do a 'Run Discovery Now'. To do this:- In 12.1.0.1.0  go to setup/addtarget/configure autodiscovery click on 'multitype target discovery on single host' Highlight the "Agent host Name" check that the discovery module "Oracle Database, Listener and Automatic Storage Management" shows as an "Enabled Discovery Module" highlight host but do not click on the hyperlink> choose 'run discovery now' Dialogue box:"are you sure you want to run target discovery on <hostname>" click on "yes"  in 12.1.0.2.0  go to setup/add target/configure auto discovery click on "all discovery modules" Highlight the "Agent host Name" Click on "configure" Tick the "enabled" box, in order to enable the relevant discovery modules (in the very least "Oracle Home Discovery" and "Oracle Database, Listener and Automatic Storage Management" is needed) Say "OK" to save changes <highlight host> choose 'run discovery now' Dialogue box:"are you sure you want to run target discovery on <hostname>" click on "yes"  The "run discovery now" forces the agent to run its database discovery algorithm ($ORACLE_HOME/plugins/oracle.sysman.db.discovery.plugin_12.1.0.1.0/discover/oracledb.pl).
This "run discovery now" achieves a similar aim as to running agentca -d or "add database" in the previous versions of Grid Control .
Notice that the targets will still not show up in the console nor will they be added to the agent's targets.xml file ($ORACLE_HOME<ofagent>/agent_inst/sysman/emd/targets.xml) until they have been promoted in Cloud Control. Promote the Databases The databases which have been found by in 'autodiscovery results/non host target', and which the user wishes to monitor by cloud control need to be promoted before they will show up as database targets in the Console . To do this:- After the "run discovery now" has completed, click on "refresh". The "Discovered Targets" column should now have a number by it. Click on the hyperlink number eg. discovered targets "4", and this will drill down to the newly found targets. These targets can then be promoted/ignored etc TIP: Another way to find targets which have been discovered is: -setup/add target/auto discovery results/"non host target" To promote a target:- • Highlight thedatabase/listener/asm target (in the 'Auto Discovery Results screen') • Click on "promote" • For database targets, there is a "configure" icon, to enter the dbsnmp password details, and test the connect string. • After the database (and listener) have been promoted and configured successfully, they will be seen in the console. Any targets that need to be marked as 'ignored' can be highlighted and then click "ignore". (To multiple highlight to ignore multiple targets in one go, do "shift/highlight"). NOTE: If the management agent does not have the database plugin deployed to it (eg. a newly discovered agent where the database plugin has not been previously manually deployed), and is automatically discovering a database/listener/asm target for the first time, the database plugin will be automatically deployed at the time when the first database/listener/asm target is successfully promoted. The database plugin will be deployed to$ORACLE_HOME<ofagent>/plugins/oracle.sysman.db.agent.plugin_12.1.0.1.0. This means that the first time a database/listener or asm target is promoted on a new agent, there could be a small delay whilst the database plugin is deployed.

 Note - there is no "mass promotion" concept for a target. This means that each target must be promoted individually. An internal Enhancement Request has been raised requesting to allow the feature of some kind of mass promotion. 
Configure Auto-discovery Time Period

After the initial discovery of the database/listener/ASM targets, the OMS will periodically check for any new database/listeners which have been added.
The period in which the OMS checks for new databases is set by:

"setup/add target/configure auto discovery/ Discovery Module/"Oracle Database, Listener and Automatic Storage Management"/configure".
Or by "setup/add target/configure auto discovery/ Discovery Module:"Multi Target-Type Discovery on a Single Host"/configure.

Again any targets which have been newly added should be detected and added to the auto discovery results. By default auto discovery runs every 1 day. Remember that auto-discovery results go to:-

setup/add target/auto discovery results/"non host target"

Has the database been discovered?

If the Desired Database is not Shown in setup/add target/auto discovery results/"non host target" then the following should be checked:

1. Perform a "run discovery now"

 go to setup/addtarget/configure autodiscovery click on 'multitype target discovery on single host' (12.1.0.1.0) or 'all discovery modules' (12.1.0.2.0) Brings you to "Target Discovery (Agent Based)" screen <highlight host but do not click on the hyperlink> choose 'run discovery now' this should bring up a dialogue box "are you sure you want to run target discovery on <hostname>? click on "yes" A "Run Discovery Now - In Progress" box should show on the screen. 

TIP: Make sure that the pop up box "are you sure you want to run target discovery on <hostname>?" is shown.
If the hyperlink by the hostname is clicked by mistake, it does not run the discovery, instead it presents the screen to configure discovery options. If the "Run Discovery Now -In Progress" box is not shown, see step 2 below.

2. Ensure that the necessary discovery modules are showing in Target Discovery (Agent Based) screen.

 go to setup/addtarget/configure autodiscovery click on 'multitype target discovery on single host' Brings you to "Target Discovery (Agent Based)" screen <click on the hyperlink for the agent host name> (or highlight and choose "configure") Tick the Discovery Modules which are required: eg. Oracle Cluster and High Availability Service Oracle Database, Listener and Automatic Storage Manager Oracle Fusion Middleware Oracle Home Discovery OracleSecureBackup 
after enabling the discovery modules, perform a "Run Discovery Now"

3. Check that the /etc/oratab or var/opt/oracle/oratab (Solaris) contains the entry for the database
4. In the agent home, on the machine where the database to be discovered resides check:-

$ORACLE_HOME/agent_inst/sysman/log/emagent_perl.trc This should show the DB discovery 'INFO' information eg.  DB_LISTENER_DISCOVERY: ***** Start of Database and Listener Discovery***  this should indicate whether the listener was found eg. this will show the version of the listener, ipaddress/hostname, the port etc. It is also possible to change the 'INFO' level to 'DEBUG' by changing the$ORACLE_HOME<of agent>/agent_inst/sysman/config/emd.properties

NOTE: If the info/debug information does not show in the emagent_perl.trc try manually deploying the database plugin from the Extensibility/plugins.
setup/extensibility/plugins expand "databases" folder
highlight "oracle database"
Actions: Deploy on Management Agent"
<select the relevant agent>

5. View the script oracledb.pl in $ORACLE_HOME/plugins/oracle.sysman.db.discovery.plugin_12.1.0.1.0/discover/oracledb.pl which contains the discovery algorithm. 4. Manual Database Discovery via EMCLI Enterprise Manager Cloud Control 12c also offers a further discovery option, which is to use emcli to manually script database discovery. Emcli is a command line extension to Enterprise Manager Cloud Control. It can be installed on any machine. It does not necessarily have to be on the machine running Cloud Control. In this example it's installed on the agent machine, but it could also be installed on a windows pc (see our technical tip "How to Install the 12c Enterprise Manager Cloud Control EMCLI on a Windows PC" on page 46). EMCLI is downloaded from the OMS and is quick to set up. To use emcli to discover the database target, the following properties must be known:- Name,SID, hostname, dbsnmp account password, listener port, 'target type' and database oracle home path. How to find the values for: Name,SID, hostname, dbsnmp account password, listener port, 'target type' and database oracle home path for the database target. - "Name" is the name which will be given to the database target in Cloud Control. (it can be anything, does not have to correspond with the actual target name) - SID can be determined from sql*plus command "show parameter instance_name" - Hostname can be determined from uname -a. Then nslookup <name returned from uname -a> to get the fully qualified hostname. - Dbsnmp account password can be checked by logging into the database as 'dbsnmp'. If password is unknown it can be changed at sql*plus using "alter user dbsnmp identified by <newpassword>" - Listener port can be determined from$TNS_ADMIN/listener.ora or $ORACLE_HOME/network/admin/listener.ora (if$TNS_ADMIN is not set)
- Database oracle home path can be determined from: /etc/oratab or /var/opt/oracle/oratab
- Target type for the database is: oracle_database (but for the other targets could be: oracle_listener,osm_instance,rac_database,osm_cluster)

The emcli add_target command takes the form:-

 emcli add_target -name="name" -type="type" -host="hostname" [-properties="pname1:pval1;pname2:pval2;..."] [-separator=properties="sep_string"] [-subseparator=properties="subsep_string"] [-credentials="userpropname:username;pwdpropnamepwdpropname:password;..."] [-input_file="parameter_tag:file_path"] [-display_name="display name"] [-groups="groupname1:grouptype1;groupname2:grouptype2;..."] [-timezone_region="gmt_offset"] [-monitor_mode="monitor_mode"] [-instances="rac_database_instance_target_name1:target_type1;..."] 
[ ] indicates that the parameter is optional

An example of the emcli command to add a database is:-

for example:-

 ./emcli add_target -name="sally" -type="oracle_database" -host="machine1.uk.oracle.com" -credentials="UserName:dbsnmp;password:dbsnmp;Role:Normal" -properties="SID:sally;Port:1528;OracleHome:/u01/orabase/11db;MachineName:machine1.uk.oracle.com" 

for example:

 emcli add_target -name="LISTENERTEST" -type="oracle_listener" -host="machine1.uk.oracle.com" -properties="LsnrName:LISTENERTEST;ListenerOraDir:/u01/orabase/11107dbase/network/admin;Port:1525;OracleHome:/u01/orabase/11107dbase;Machine:machine1.uk.oracle.com" 
These commands can be put in a script and run - eg.

 [oracle@rachvm1 emcliKit]$pwd [oracle@rachvm1 emcliKit]$ /u01/emcliKit vi adddatabase.sh contents of "adddatabase.sh" are:- emcli login -username=sysman -password=oracle12 emcli add_target -name="sally" -type="oracle_database" -host="machine1.uk.oracle.com" -credentials="UserName:dbsnmp;password:dbsnmp;Role:Normal" -properties="SID:sally;Port:1528;OracleHome:/u01/orabase/11db;MachineName:machine1.uk.oracle.com" emcli add_target -name="test" -type="oracle_database" -host="machine1.uk.oracle.com" -credentials="UserName:dbsnmp;password:dbsnmp;Role:Normal" -properties="SID:test;Port:1525;OracleHome:/u01/orabase/1107dbase;MachineName:machine1.uk.oracle.com" emcli add_target -name="LISTENERTEST" -type="oracle_listener" -host="machine1.uk.oracle.com" -properties="LsnrName:LISTENERTEST;ListenerOraDir:/u01/orabase/11107dbase/network/admin;Port:1525;OracleHome:/u01/orabase/11107dbase;Machine:machine1.uk.oracle.com" emcli logout [oracle@rachvm1 emcliKit]$chmod +x adddatabase.sh [oracle@rachvm1 emcliKit]$ . adddatabase.sh Output is shown on the command line:- Login successful Target "sally:oracle_database" added successfully Target "test:oracle_database" added successfully Target "LISTENERTEST:oracle_listener" added successfully Logout successful [oracle@rachvm1 emcliKit]$ If a database has been added via emcli, it will straight away be monitored by Cloud Control, this means it will be seen as a target in the console straightaway and will not need to be promoted. This is because the dbsnmp details have been provided in the emcli input. At this moment in time, there is no emcli promote command (for targets that have been auto discovered). NOTE: If the agent is a management agent without the database plugin (eg. a newly discovered agent without the database plugin manually deployed to it which has never discovered any targets of type database/listener or asm), and emcli is used to add the first database/listener or asm target, the database plugin will be automatically deployed as part of the discovery process. The database plugin will be deployed to$ORACLE_HOME<ofagent>/plugins/oracle.sysman.db.agent.plugin_12.1.0.1.0

12c Cloud Control: Steps to Locate and Manage the Various Logs/Trace files in a 12c OMS Installation

Modified 12-FEB-2013

Applies to

Enterprise Manager for Oracle Database - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Purpose

This document lists and describes the usage of various log/trace files in the 12c Cloud Control OMS home and steps to control their size/tracing levels.

For details of the directory structure and useful locations in the 12c OMS home, refer to our technical tip:
"12c Cloud Control: Details of the Directory Structure and Commonly Used Locations in a 12c OMS Installation" on page 19

For a description of the log/trace files updated specifically during the OMS startup, refer to our technical tip:
"12c Cloud Control: Understanding OMS Process Control (Start/Stop/Status)" on page 46

Scope

Enterprise Manager Cloud Control Administrators who need to familiarize themselves with the various log/trace files within a 12c OMS home and know how to manage these files.

Details

OMS (emgc and empbs) Log/Trace Files

Logging and Tracing Directory:

OMS logging and tracing information is written to files located in the <EM_INSTANCE_BASE>/em/<OMS_NAME>/sysman/log directory.
For example: /home/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log/

Where, <EM_INSTANCE_BASE> is the OMS Instance directory. This is named gc_inst by default and is present in the Middleware Home.
Description of the OMS log/trace files created:

File Usage Number of Files
emoms.log.n Main log file for the OMS Console (emgc) application. log4j.appender.emlogAppender.MaxBackupIndex + 1
emoms.trc.n Main trace file for the OMS Console (emgc) application. og4j.appender.emtrcAppender.MaxBackupIndex + 1
emoms_pbs.log.n Main log file for the OMS Console (emgc) application. log4j.appender.emlogAppender.MaxBackupIndex + 1
emoms_pbs.trc.n Main trace file for the OMS pbs (empbs) application. og4j.appender.emtrcAppender.MaxBackupIndex + 1
secure.log Contains output from the 'emctl secure oms' commands. 1
emctl.log Created by the emctl utility, when it is executed in the OMS home. This file also contains OMS startup / shutdown details. 2 - emctl.log gets rolled over to emctl.log.1 after it reaches 10MB in size.
emctl.msg Created / written to by the OMS HealthMonitor thread, when it re-starts the OMS due to a critical error. 1
oms_diag_info_<timestamp>.msg This file gets created when the OMS is re-started by the HealthMonitor thread and contains a stack trace of the thread causing the restart, profiling data and full thread dump of the OMS. 1 file created for each OMS re-start by HealthMonitor thread
repo_dump_<timestamp>.html This file gets created when the OMS is re-started by the HealthMonitor thread and contains output of pre-defined queries executed in the Repository Database just before the re-start. 1 file created for each OMS re-start by HealthMonitor thread

Logging and Tracing Parameters:

The logging mechanism used for the OMS is based on the 'RollingFileAppender', a dynamic LOG4J logging mechanism, which allows the log files to get archived automatically as soon as they reach a specified size.
This avoids creation of large log and trace files that can consume too much disk space.

The parameters for LOG4J logging (emoms.log, emoms_pbs.log) and tracing (emoms.trc, emoms_pbs.trc) are:

Purpose Property and example Description
Maximum Log File Size (emoms.log and emoms_pbs.log) log4j.appender.emlogAppender.MaxFileSize=20000000 When the log file reaches this size (in bytes), the OMS copies the data to a new rollover/archive file (for eg: emoms.log.1 or emoms_pbs.log.1) and creates new files - emoms.log or emoms_pbs.log.
Maximum Trace File Size (emoms.trc and emoms_pbs.trc) log4j.appender.emtrcAppender.MaxFileSize=5000000 When the trace file reaches this size (in bytes), the OMS copies the data to a new rollover / archive file (for eg: emoms.trc.1 or emoms_pbs.trc.1) and creates new files - emoms.trc or emoms_pbs.trc.
Maximum Number of Log Files (Rotation) log4j.appender.emlogAppender.MaxBackupIndex=1 This optional property indicates how many times OMS will rollover the log file to a new file name before deleting logging data. When set to 1, there will be 1 archive log files plus the current log file, for eg: emoms.log and emoms.log.1. Similarly the empbs application will have emoms_pbs.log and emoms_pbs.log.1
Maximum Number of Trace Files (Rotation) log4j.appender.emtrcAppender.MaxBackupIndex=10 This property indicates how many times the OMS will rollover the trace file to a new file name before deleting tracing data. When set to 10, there will be 10 archive trace files plus the current trace file, for eg: i.e emoms.trc, emoms.trc.1...<upto> emoms.trc.10. Similarly for the empbs application, we'll have emoms_pbs.trc, emoms_pbs.trc.1, ...<upto> emoms_pbs.trc.10. As soon as more information is generated, the oldest archive file is removed.
File Location Tracing:
log4j.appender.emtrcAppender.File=<path to the emoms.trc and emoms_pbs.trc files>
Logging:
log4j.appender.emlogAppender.File=<path to the emoms.log and emoms_pbs.log files>
Specifies the fully qualified directory location where the OMS log and trace files will be created. By default, this is set to <EM_INSTANCE_BASE>/em/<OMS_NAME>/sysman/log directory. It is recommended to retain the path to the default location.
Layout Tracing:
log4j.appender.emtrcAppender.layout=org.apache.log4j.PatternLayout
Logging:
log4j.appender.emlogAppender.layout=org.apache.log4j.PatternLayout
Defines the java class used by the OMS for creating the log/trace files. This parameter should not be modified by the user.
Layout Conversion Pattern Tracing:
og4j.appender.emtrcAppender.layout.ConversionPattern=%d [%t] %-5p %c{2} %M.%L - %m
Logging:
log4j.appender.emlogAppender.layout.ConversionPattern=%d [%t] %-5p %c{2} %M.%L - %m
This defines the format in which the log/trace file entries will be created.

This parameter should not be modified by the user.
The variables definitions are:
%c : Logging Category
%C : Fully qualified JAVA Class name
%d : Timestamp (Date + Time)
%F : JAVA source file (Slow - Only to debug)
%I : Location of calling routine (Slow - Only to debug)
%L : Linenumber in the JAVA source (Slow - Only to debug)
%M : Module/Function which requests the log
%m : Message to be written
%n : Line separator
%p : Priority of message (ERROR,WARN,INFO,DEBUG)
%r : Time in miliseconds since start of application
%t : Thread which reports this message
%x : Nested Diagnostics Context
%X : Mapped Diagnostics Context

Modifying OMS Trace Level:

By default, the OMS will save all critical and warning messages to the emoms.trc and emoms_pbs.trc files. The LOG4J mechanism allows the level of logging to be controlled by the end user by specifying the priority of messages that should be shown in the trace file.
The trace level can be one of the below:

Trace Level Description
ERROR Report only critical errors
WARN Report critical errors and warning. Default Setting.
INFO Include informational messages.
DEBUG Full debug trace and useful for investigation of problems.

NOTE:

1. These settings are case sensitive and the values must be entered in uppercase.
2. The <EM_INSTANCE_BASE>/em/<OMS_NAME>/sysman/config/emomslogging.properties file is no longer used to configure the tracing/logging parameters.
The parameters can be set or modified using the 'emctl set property' command.
3. After modifying any of the log or trace parameters, it is not necessary to re-start the OMS. The change is made effective at run-time itself.
4. To modify the trace level to DEBUG, execute:

 cd <OMS_HOME>/bin emctl set property -name log4j.rootCategory -value 'DEBUG, emlogAppender, emtrcAppender' -module logging 
- The above command will prompt you to confirm the change by displaying the below message:

 Setting log4j.rootCategory to "DEBUG, emlogAppender, emtrcAppender" has performance impact on OMS due to heavy logging. It is recommended to use category specific logging instead. Please consult My Oracle Support Note 1330898.1 for additional information. Do you want to continue? (yes/no): 
Type 'yes' or 'y' to proceed with the change.
To cancel the change, type 'no'.

- If you type 'yes' or 'y', the command will further prompt you for the password of the sysman user.
- If the property change is successful, the following output will be returned on the terminal:

 Property log4j.rootCategory for oms omsmachine.domain:4889_Management_Service has been set to value DEBUG, emlogAppender, emtrcAppender OMS restart is not required to reflect the new property value 

5. To view the current tracing level:

1. Query the value for OMS Property: log4j.rootCategory

 cd <OMS_HOME>/bin emctl get property -name log4j.rootCategory Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. SYSMAN password: Value for property log4j.rootCategory for oms smnath-lnx.idc.oracle.com:4890_Management_Service is DEBUG, emlogAppender, emtrcAppender 

2. OR

3. List all the properties and find the value for log4j.rootCategory:

 cd <OMS_HOME>/bin emctl list properties 
In the output, look for property:

 log4j.rootCategory=DEBUG, emlogAppender, emtrcAppender 

For more details on setting / viewing / deleting OMS property values in 12c, refer to our technical note "12c Cloud Control : How To View and Update OMS Configuration Properties Using 'emctl' utility?" on page 33.

You can also enable DEBUG level only for a particular subsystem / category. For details, refer to our technical note "Impact of Setting Debug Mode for OMS and Steps to Enable Debug for Particular Subsystems" on page 32.

Oracle WebLogic Server Logs

Oracle Management Service is a J2EE application deployed on an Oracle WebLogic Server and different components of the Oracle WebLogic Server generate their own log files.
These files contain important information that are required for troubleshooting.

1. Oracle HTTP Server (OHS) Logs

The logs are stored at <EM_INSTANCE_BASE>/<webtier_instance_name>/diagnostics/logs/OHS/<ohs_name>.
For example: /home/oracle/Middleware/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1

File Usage
access_log This records information for all client requests processed by the HTTP server. Each entry contains the following information:

• Host name
• Remote log name
• Remote user and time
• Request
• HTTP Response code
• Number of transferred bytes
The HTTP Response code is useful in identifying whether the client request was successful or not.
ohs1.log This is the error log for the OHS.
console~OHS~1.log This is the Console log for the OHS and is generated by the OPMN. It records the stdout and stderr messages and is useful for troubleshooting OHS startup issues.
mod_wl_ohs.log This is the log file for the WLS plugin (mod_wl_ohs) for OHS. This plugin allows requests to be proxied from OHS to the Oracle WebLogic Server.

2. OPMN Logs

The logs are stored at <EM_INSTANCE_BASE>/<webtier_instance_name>/diagnostics/logs/OPMN/<opmn_name>
For example: /home/oracle/Middleware/gc_inst/WebTierIH1/diagnostics/logs/OPMN/opmn1

File Usage
opmn.out This contains the standard output (stdout) and standard error (stderr) log entries of OPMN and is also referred to as the OPMN "console log". After a certain point in the OPMN initialization, nothing else is written to this file. Only a small set of messages ever appears in this file; therefore, this file may not be present or may be of 0 bytes.
opmn.log This tracks command execution and operation progress and contains messages useful for monitoring the OPMN server operations. Output written to the opmn.log file contains the exit status of a child OPMN process. A status code of 4 indicates a normal reload of OPMN. All other status codes indicate an abnormal termination of the child OPMN process.
debug.log This contains OPMN debug log messages (English only), which track command execution and operation progress for Oracle Notification Service (ONS) and Process Manager(PM). The PM portion of OPMN generates and outputs the error messages in this file.

3. EM NodeManager Logs

The log is located at <EM_INSTANCE_BASE>/NodeManager/emnodemanager

For example:
/home/oracle/Middleware/gc_inst/NodeManager/emnodemanager

File Usage Rotation Policy
nodemanager.log Main log file for the nodemanager and has entries for activities performed by the nodemanager. There is also a nodemanager.log.lck file which is a lock file for the nodemanager and will be present as long as the nodemanager is up and running. By default, there is no rotation policy as this file is not expected to grow too big. If needed, below parameters can be set in the <EM_INSTANCE_BASE>/NodeManager/emnodemanager/nodemanager.properties:

LogLimit:Maximum size of the Node Manager Log specified as an integer. When this limit is reached, a new log file is started. By default, this is set to Unlimited.

LogCount:Maximum number of log files to create when LogLimit is exceeded.
By default, this is set to 1.

The logs are located at <EM_INSTANCE_BASE>/user_projects/domains/<domain_name>/servers/<ADMIN_SERVER_NAME>/logs

For example:

File Usage Rotation Policy
<ADMIN_SERVER_NAME>.log or EMGC_ADMINSERVER.log The EMGC_ADMINSERVER writes all messages from its subsystems and applications to this logfile. The filesize is restricted to 5MB, after which it is rotated but the older logs are not deleted automatically. The log level is WARNING, by default.
<ADMIN_SERVER_NAME>.out or EMGC_ADMINSERVER.out sysout and syserr messages are saved in this file by the Nodemanager. These files cannot be automatically rotated by size or time. They are rotated by the nodemanager process only at the time of the Admin server start, so older files need to be manually purged.
access.log Contains details of HTTP requests processed by this server. The filesize is restricted to 5MB, after which it is rotated but the older logs are not deleted automatically.
<domain_name>.log or GCDomain.log The domain log file provides a central location from which to view the overall status of the domain. In addition to writing messages to its own log file, each server instance also forwards a subset of its messages to this domain-wide log file. By default, servers forward only messages of severity level of NOTICE or higher. The filesize is restricted to 5MB, after which it is rotated but the older logs are not deleted automatically.
<SERVER_NAME>-diagnostic.log or EMGC_ADMINSERVER-diagnostic.log Each server instance in a domain writes security-related (OPSS-based) errors raised by its subsystems and applications to a server log file named: ServerName-diagnostic.logn. Server-related security errors, such as exceptions raised by issues with a subject or principal, and errors that may occur while migrating or reassociating domain security data get written in the administration server diagnostic log. This log uses ODL formatting and the logging configuration is specified in the <EM_INSTANCE_BASE>/user_projects/domains/GCDomain/config/ fmwconfig/servers/EMGC_ADMINSERVER/logging.xml The log files are rotated when they reach 10 MB and the maximum size of all log files is 100 MB i.e the total number of files is restricted to 10.

5. For enabling automatic purge policy for the older GCDomain.log, EMGC_ADMINSERVER.log and access.log files, refer to technical note "12c Cloud Control: How to Enable Log Rotation Policy to Automatically Delete Older GCDomain.log, EMGC_ADMINSERVER.log and access.log Files?" on page 15.

6. EM Managed Server Logs

The logs are located at <EM_INSTANCE_BASE>/user_projects/domains/<domain_name>/servers/<SERVER_NAME>/logs

For example: /home/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.log

File Usage Rotation Policy
<SERVER_NAME>.log or EMGC_OMS1.log The EMGC_OMSn instance writes all messages from its subsystems and applications to this logfile. The number of files are restricted to 10, which are rotated at default size of 5MB. The log level is WARNING, by default./td>
<SERVER_NAME>.out or EMGC_OMS1.out The messages written to sysout and syserr are saved in this file by the nodemanager. These files cannot be automatically rotated by size or time.They are rotated by the nodemanager only at the time of Managed server start, so older files need to be manually purged.
access.log Contains details of HTTP requests processed by this server. The number of files are restricted to 10, which are rotated at default size of 5MB.
<SERVER_NAME>-diagnostic.log or EMGC_OMS1-diagnostic.log Application-related security errors, such as exceptions raised by application-specific policies or credentials, get written in this managed server diagnostic log. This log uses ODL formatting and the logging configuration is specified in the <EM_INSTANCE_BASE>/user_projects/ domains/GCDomain/config/fmwconfig/servers/EMGC_OMSn/logging.xml file. The log files are rotated when they reach 10 MB and the maximum size of all log files is 100 MB i.e the total number of files is restricted to 10.

For a list of WLS log files which need to be manually purged periodically, refer to our technical tip "12c Cloud Control: Which WLS Log Files Can be Removed/Purged Manually at Regular Intervals in 12c OMS Installation for Space Considerations?" on page 4.

BI Publisher Logs

The logs are located at <EM_INSTANCE_BASE>/user_projects/domains/GCdomain/servers/BIP/logs

Example: /u01/app/oracle/product/Middleware/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs

File Usage Rotation Policy
BIP.log The BI Publisher processes write all messages to this logfile.These log uses ODL formatting. The number of files are restricted by default to 10, and are rotated at a default size of 5MB. The log level is WARNING by default. The retention can also be time based.
BIP.out The messages written to sysout and syserr are saved in this file by the nodemanager. These files cannot be automatically rotated by size or time. They are rotated by the nodemanager only at the time of Managed server start, so older files need to be manually purged.
access.log Contains details of HTTP requests processed by this server. The number of files are restricted to 10, which are rotated at default size of 5
BIP-diagnostic.log Application-related security errors, such as exceptions raised by application-specific policies or credentials, get written in this managed server diagnostic log. This log uses ODL formatting. The log files are rotated when they reach 10 MB and the maximum size of all log files is 100 MB i.e the total number of files is restricted to 10.

OMS Installation and Configuration Logs

1. Installation: <oraInventory>/logs
2. EMPrereqkit:<oraInventory>/logs/emdbprereqs/LATEST/repository.log or emprereqkit.log
3. Configuration Tools: <OMS_HOME>/cfgtoollogs
4. Repository Creation: <OMS_HOME>/sysman/log/schemamanager

12c Cloud Control: How to Enable Log Rotation Policy to Automatically Delete Older GCDomain.log, EMGC_ADMINSERVER.log and access.log Files?

Modified 18-APR-2012

Applies to

Enterprise Manager for Oracle Database - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

In a 12c OMS home, the GCDomain.log, EMGC_ADMINSERVER.log and access.log files located in the directory:

 For example: /home/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_ADMINSERVER/logs) 
are rotated once they reach 5MB in size. But the older logs are not automatically deleted and need to be manually purged at regular intervals.
This document explains how the rotation policy can be modified so that the older logs can be deleted automatically without any manual intervention.

Fix

• Open the <OMS_HOME>/install/setupinfo.txt file and identify the "Admin Server URL", for example:

•  Admin Server URL: https://omsmachine.domain:7102/console 
• From a browser, login to the WLS Admin Server Console as the 'weblogic' user.
• In the 'Change Center' click on the 'Lock & Edit' button.

• For GCDomain.log:

• Under the 'Domain Structure', click on GCDomain → Logging.
In the page shown, enable the option 'Limit number of retained files' as shown in the screenshot below:

• Click on the 'Save' button at the bottom of the page to save the settings.

• Once the changes are activated, the number of GCDomain.logn files are restricted to 7, which are rotated at default size of 5MB.

• Under the 'Domain Structure', expand 'Environment' and click on 'Servers'. Click on the EMGC_ADMINSERVER(admin)
and navigate to the 'Logging' tab and then 'General' subatab.
• In the page shown, enable the option 'Limit number of retained files' as shown in the screenshot below:

• Click on the 'Save' button at the bottom of the page to save the settings.

• Once the changes are activated, the number of EMGC_ADMINSERVER.logn files are restricted to 10, which are rotated at default size of 5MB.

• Under the 'Domain Structure', expand 'Environment' and click on 'Servers'. Click on the EMGC_ADMINSERVER(admin) and navigate to the 'Logging' tab and then 'HTTP' subatab.
• In the page shown, enable the option 'Limit number of retained files' as shown in the screenshot below:

• Click on the 'Save' button at the bottom of the page to save the settings.

Once the changes are activated, the number of access.logn files are restricted to 10, which are rotated at default size of 5MB.

• After saving the above, activate the settings by clicking on 'Activate Changes' in the 'Change Center'.

NOTE:
The older logs are not immediately purged after the above configuration change. When the existing GCDomain.log / EMGC_ADMINSERVER.log / access.log reaches 5MB in size,
Weblogic will automatically rollover the file. At this time, the number of files configured to be retained are maintained and the older set of files are deleted automatically.

How to Determine the List of Patch Set Update(PSU) Applied to the Enterprise Manager OMS and Agent Oracle Homes?

Modified 20-DEC-2012

Applies to

Enterprise Manager Base Platform - Version 10.2.0.5 to 11.1.0.1 [Release 10.2 to 11.1]
Information in this document applies to any platform.

Goal

Applying a Patch Set Update (PSU) to the OMS or Agent Oracle_Home changes the value of the fifth digit in the complete version number of the product => 'x' in the format number below:

10.2.0.5.x with possible values for x : 0,1,2,3,4,5,6
11.1.0.1.x with possible values for x : 0,1,2,3,4,5,6,7
12.1.0.2.x with possible values for x : 0,1

After applying a PSU, the value of 'x' increases but this new value is not reflected:

1. in the banners shown in the emctl command:

emctl status agent
or
emctl getversion

For example:

- Version of the Agent and OMS remain unchanged in the emctl status agent output below

 cd <AGENT_HOME>/bin emctl status agent Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0 Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 11.1.0.1.0 OMS Version : 11.1.0.1.0 Protocol Version : 11.1.0.0.0 
- The 'emctl getversion' executed in the OMS / Agent home does not indicate whether a PSU patch is applied or not.
The output only displays the 4-digit version, for example:

 cd <OMS_HOME>/bin emctl getversion Oracle Enterprise Manager 10g Release 5 Grid Control Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved. --- Standalone agent Enterprise Manager 10g OMS Version 10.2.0.5.0 Enterprise Manager 10g AS Control Version 10.2.0.0 Enterprise Manager 10g Agent Version 10.2.0.5.0 
or

 cd <OMS_HOME>/bin ./emctl getversion Oracle Enterprise Manager 11g Release 1 Grid Control Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved. --- Standalone agent Enterprise Manager 11g OMS Version 11.1.0.1.0 Enterprise Manager 11g Agent Version 11.1.0.1.0 

2. in the version number of components shown in the output of 'opatch lsinventory -detail'

For example - below output is form an OMS home where PSU2 Patch:10270073 is applied but the Grid Control version still remains 11.1.0.1.0:

 Invoking Patch 11.1.0.8.3 Oracle Home : /ora/product/oracle/11.1.1/oem/oms11g Central Inventory : /ora/product/oraInventory from : /etc/oraInst.loc Installed Top-level Products Oracle Enterprise Manager Grid Console 11.1.0.1.0 ... Enterprise Manager Common Core Files 11.1.0.1.0 Enterprise Manager Common Files 11.1.0.1.0 ... Enterprise Manager Grid Control Core Files 11.1.0.1.0 Enterprise Manager Repository 11.1.0.1.0 Enterprise Manager Repository Core Files 11.1.0.1.0 ... Interim patches (15) : Patch 10270073 : applied on Thu Jul 24 17:11:21 CET 2011 Unique Patch ID: 13349102 Created on 30 Nov 2010, 15:34:49 hrs PST8PDT 

The OMS or Agent version number is always een as 10.2.0.5.0 / 11.1.0.1.0 in both places a) and b) despite any PSU applied.

This document provides the steps for finding the list of PSU patches applied to the Enterprise Manager OMS and Grid Agent Oracle homes.

Fix

A PSU is identified with an unique patch number and for the OMS / Agent homes and the PSU number applied is visible only in the inventory. The list of PSU patches applied to the OMS and Agent homes can be identified using the 'opatch lsinventory' output, as per the below steps.:

Solution A

1. For the OMS HOME

- Set the ORACLE_HOME to the OMS installation location:

On Unix:

$export ORACLE_HOME=<path of the oms10g or oms11g location> For example: $ export ORACLE_HOME=/u01/OracleHomes/oms10g $export ORACLE_HOME=/u01/Middleware/oms11g  On Windows: > set ORACLE_HOME=<path of the oms10g or oms11g location> For example:  > set ORACLE_HOME=C:\OracleHomes\oms10g > set ORACLE_HOME=C:\Middleware\oms11g  - Execute the following commands:  cd <OMS_HOME>/OPatch opatch lsinventory -bugs_fixed | grep -i 'ENTERPRISE MANAGER OMS' | grep -i 'PSU'  For example, the output in a 10.2.0.5 OMS home with the PSU5 patch will be:  9786002 10168579 Mon Aug 01 09:24:20 IST 2011 enterprise manager oms 10.2.0.5.4 psu 10168579 10168579 Mon Aug 01 09:24:20 IST 2011 enterprise manager oms 10.2.0.5.5 psu 9174904 9702579 Tue Nov 30 20:04:01 IST 2010 ENTERPRISE MANAGER OMS 10.2.0.5.2 PSU 9282397 9702579 Tue Nov 30 20:04:01 IST 2010 ENTERPRISE MANAGER OMS 10.2.0.5.3 PSU 9702579 9702579 Tue Nov 30 20:04:01 IST 2010 ENTERPRISE MANAGER OMS 10.2.0.5.4 PSU EM CONTENT  In a 11g OMS home with a PSU4 patch, the output will be:  12423703 12423703 Wed Sep 07 09:48:36 IST 2011 ENTERPRISE MANAGER OMS 11.1.0.1.4 GC PSU (GENERIC)  NOTE: Due to an internal bug, the above command does not display the PSU1, PSU2 and PSU3 patches applied to a 11g OMS home. The issue was fixed in 11g PSU4 to display the correct output. 2. For the Agent HOME - Set the ORACLE_HOME to the Agent installation location: On Unix:$ export ORACLE_HOME=<path of the agent10g or agent11g location>

For example:

 $export ORACLE_HOME=/u01/OracleHomes/agent10g$ export ORACLE_HOME=/u01/Middleware/agent11g 
On Windows:

> set ORACLE_HOME=<path of the agent10g or agent11g location>

For example:

 > set ORACLE_HOME=C:\OracleHomes\agent10g > set ORACLE_HOME=C:\Middleware\agent11g 
- Execute the following commands:

 cd <AGENT_HOME>/OPatch opatch lsinventory -bugs_fixed | grep -i 'ENTERPRISE MANAGER AGENT'| grep -i 'PSU' | grep -i 'ENTERPRISE MANAGER AGENT'| grep -i 'PSU' 
Agent 10.2.0.5

For example, the output in a 10.2.0.5 Agent home with the PSU3 patch will be:

 9162498 9282414 Tue Nov 30 19:55:11 IST 2010 enterprise manager agent 10.2.0.5.2 psu 9282414 9282414 Tue Nov 30 19:55:11 IST 2010 ENTERPRISE MANAGER AGENT 10.2.0.5.3 PSU 
Agent 11g

Important : After applying PSU4 in a 11g AGENT_HOME, the command :

 ./opatch lsinventory -bugs_fixed |grep PSU  does not return any output.

In the same AGENT_HOME, after applying PSU5, the output is shown correctly.

 $./opatch lsinventory -bugs_fixed |grep PSU 9345906 9345921 Fri Dec 02 16:51:46 IST 2011 ENTERPRISE MANAGER AGENT 11.1.0.1.3 GC PSU (UNIX) 9345913 9345921 Fri Dec 02 16:51:46 IST 2011 ENTERPRISE MANAGER AGENT 11.1.0.1.4 PSU 9345921 9345921 Fri Dec 02 16:51:46 IST 2011 ENTERPRISE MANAGER AGENT 11.1.0.1.5 PSU 10273607 9345921 Fri Dec 02 16:51:46 IST 2011 ENTERPRISE MANAGER AGENT 11.1.0.1.2 GC PSU (GENERIC)  The same problem can be seen after applying PSU7. Solution here is to adapt the opatch command as follows Command which does not show correct output:  <-AGENT-[]> /home/oracle [oracle@ ~]$ opatch lsinventory -bugs_fixed | grep -i 'ENTERPRISE MANAGER AGENT'| grep -i 'PSU' | grep -i 'ENTERPRISE MANAGER AGENT'| grep -i 'PSU' 9345906 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.3 GC PSU (UNIX) 9345913 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.4 PSU 9345921 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.5 PSU 9346243 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.6 PSU 10273607 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.2 GC PSU (GENERI 
Command that does show the correct PSU7 version installed:

 <-AGENT-[]> /home/oracle [oracle@lpgrid001a ~]$opatch lsinventory -bugs_fixed | grep -i 'PSU' 9345906 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.3 GC PSU (UNIX) 9345913 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.4 PSU 9345921 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.5 PSU 9346243 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.6 PSU 11.1.0.1.7 PSU (UNIX) 10273607 9346282 Wed Aug 29 13:37:55 CEST 2012 ENTERPRISE MANAGER AGENT 11.1.0.1.2 GC PSU (GENERI  3. Solution B When the above opatch command described in Solution A does not provide the list of PSU patches directly, compare the list of patches seen in the 'opatch lsinventory -detail' command with the table below and identify the PSU patch manually: List of OMS and Agent PSU Patches NOTE: When a cell is empty, that means that the patch is not applicable/available. 10.2.0.5 OMS AGENT Unix AGENT Windows PSU1 (10.2.0.5.1) 8864918 - - PSU1 (10.2.0.5.2) 9174904 9162498 9223116 PSU1 (10.2.0.5.3) 9282397 9282414 9490898 PSU1 (10.2.0.5.4) 9786002 - - PSU1 (10.2.0.5.5) 10168579 - - PSU1 (10.2.0.5.6) 13701923 - - PSU1 (10.2.0.5.7) 14766621 - - 11.1.0.1.1 OMS AGENT Unix AGENT Windows PSU1 (11.1.0.1.1) 10065631 - - PSU1 (11.1.0.1.2) 10270073 10273607 10273607 PSU1 (11.1.0.1.3) 11727299 9345906 11778791 PSU1 (11.1.0.1.4) 12423703 9345913 12423714 PSU1 (11.1.0.1.5) 12833678 9345921 12833724 PSU1 (11.1.0.1.6) 13248190 9346243 13248202 PSU1 (11.1.0.1.7) 13711705 9346282 13711732 PSU1 (11.1.0.1.8) 14766609 - - 12.1.0.2.1 OMS AGENT Unix AGENT Windows PSU1 (12.1.0.2.1) 14840279 - - Enterprise Manager Cloud Control 12C, Target Database Instance Remains in Pending Status Indefinitely Modified 09-AUG-2012 Applies to Enterprise Manager for Oracle Database - Version 12.1.0.1.0 and later Information in this document applies to any platform. Symptoms The Database Instance appears in pending status from Enterprise Manager Cloud Control 12c for an undetermined period of time, while it is actually up and running. The agent is started and is uploading correctly. This may happen for other target types as well (like cluster, ASM, etc...) Cause The response metric timed out for that particular database target, thus it was moved to a suspended state and the target appears as pending in EM console. Solution 1. Stop the EM agent 2. Remove the SUSPENSION files in <AGENT-INST>/sysman/emd/state/statemgmt/oracle_emd/agent/SUSPENSION directory. For example: $ pwd /home/oracle/agent12c/agent_inst/sysman/emd/state/statemgmt/oracle_emd/agent/SUSPENSION ls -ltr -rw-r----- 1 oracle users 11 Apr 5 11:45 oracle_database:emrep12c.us.oracle.com:Response rm -rf oracle_database:emrep12c.fr.oracle.com:Response 

3. Start back the Agent

NOTE: if the 12c EM agent has BP1 applied, it would be enough to just restart the EM agent to clear the suspended metrics.

To avoid this condition, the following options can be used:

• Increase the Response metric schedule (from the default 15 seconds to 1 minute)

• or

• add the following to emd.properties:

• NOTE: another cause for this behavior is a left-over blackout that ended on the OMS side, but did not ended on the agent side (was not cleanly removed). To identify this condition as well, check if the <AGENT-INST>/sysman/emd/blackouts.xml contains anything more than the default "<Blackouts/>", remove the file and restart the EM agent.

How to Disable Automatic Statistics Collection in 11g

Modified 02-SEP-2012

Applies to

Oracle Server - Enterprise Edition - Version 11.1.0.6 and later
Information in this document applies to any platform.

Goal

How to disable automatic statistics collection in 11g.

Fix

The command to disable automatic optimizer statistics collection is:

 BEGIN DBMS_AUTO_TASK_ADMIN.DISABLE( client_name => 'auto optimizer stats collection', operation => NULL, window_name => NULL); END; / 
This can be verified with:

 SQL> select client_name,status from DBA_AUTOTASK_CLIENT; 

                      CLIENT_NAME                        STATUS
---------------------------------- --------
auto optimizer stats collection    DISABLED

 
to re-enable again:

 BEGIN DBMS_AUTO_TASK_ADMIN.ENABLE( client_name => 'auto optimizer stats collection', operation => NULL, window_name => NULL); END; / 

For more details see our next technical tip on this same page

How Can We Find Out Status Of Task 'Auto Optimizer Stats Collection'

Modified 01-OCT-2012

Applies to

Oracle Server - Enterprise Edition - Version 11.1.0.6 to 11.2.0.3 [Release 11.1 to 11.2]
Information in this document applies to any platform.

Goal

Explain how to find the status of 'GATHER_STATS_JOB' or in fact 'auto optimizer stats collection' in 11g.

Fix

11g

In 11g, the automated maintenance task that collects statistics is called 'auto optimizer stats collection'. You can determine its' status by selecting from DBA_AUTOTASK_CLIENT:

 SELECT CLIENT_NAME,STATUS FROM DBA_AUTOTASK_CLIENT WHERE CLIENT_NAME = 'auto optimizer stats collection' / 

                      CLIENT_NAME                                                      STATUS
---------------------------------------------------------------- --------
auto optimizer stats collection                                  ENABLED

 
so you can see this historical information using a query like:

 SELECT CLIENT_NAME,STATUS, MEAN_INCOMING_TASKS_7_DAYS, MEAN_INCOMING_TASKS_30_DAYS FROM DBA_AUTOTASK_CLIENT WHERE CLIENT_NAME = 'auto optimizer stats collection' / 
How to enable auto stats collection?

If for some reason automatic optimizer statistics collection is disabled, you can enable it using the ENABLE procedure in the DBMS_AUTO_TASK_ADMIN package:

 exec DBMS_AUTO_TASK_ADMIN.ENABLE( client_name => 'auto optimizer stats collection', operation => NULL, window_name => NULL);  How to disable the auto stats collection?

In situations when you want to disable automatic optimizer statistics collection, you can disable it using the DISABLE procedure in the DBMS_AUTO_TASK_ADMIN package:

 exec DBMS_AUTO_TASK_ADMIN.DISABLE( client_name => 'auto optimizer stats collection', operation => NULL, window_name => NULL); 
See our technical tip:

10g

The following is the 'old method' for checking the current and past automated maintenance tasks:

 SELECT j.job_name,j.program_name,j.schedule_name,j.job_class,p.enabled FROM dba_scheduler_programs p,dba_scheduler_jobs j WHERE p.program_name=j.program_name; JOB_NAME PROGRAM_NAME SCHEDULE_NAME JOB_CLASS ENABLED -------- -------- -------- -------- -------- GATHER_STATS_JOB GATHER_STATS_PROG MAINTENANCE_WINDOW_GROUP AUTO_TASKS_JOB_CLASS TRUE 
How do you disable the GATHER_STATS_JOB for 10g?

The most direct approach is to disable the GATHER_STATS_JOB as follows:

 SQL> exec sys.dbms_scheduler.disable ('GATHER_STATS_JOB'); 

How do you enable the GATHER_STATS_JOB for 10g?

This is enabled by default. If you have disabled it, then you can re-enable it as

 SQL> exec sys.dbms_scheduler.enable ("SYS"."GATHER_STATS_JOB"); 

NOTE: If the 10g method is used on 11g, then it does not return any information as the implementation is quite different in 11g:

 Connected to: Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production With the Partitioning and Real Application Testing options SQL> SELECT job_name,job_type,program_name,schedule_name,job_class FROM dba_scheduler_jobs WHERE job_name = 'GATHER_STATS_JOB'; no rows selected 

Modified 12-MAR-2012

Applies to

Oracle Server - Enterprise Edition - Version: 11.1.0.6 to 11.2.0.3 - Release: 11.1 to 11.2
Information in this document applies to any platform.

Symptoms

We are Trying to Disable the Auto Optimizer Stats Collection in 11G,DBA_AUTOTASK_CLIENT shows that the 'auto optimizer stats collection' is disabled.

 BEGIN DBMS_AUTO_TASK_ADMIN.DISABLE( client_name => 'auto optimizer stats collection', operation => NULL, window_name => NULL); END; / PL/SQL procedure successfully completed. 

 SQL> select client_name,status from DBA_AUTOTASK_TASK; 

                      CLIENT_NAME                       STATUS
-------------------------------   --------
auto optimizer stats collection   ENABLED  <===

 SQL> select client_name,status from DBA_AUTOTASK_CLIENT; CLIENT_NAME STATUS -------------------------------- -------- auto optimizer stats collection DISABLED <=== auto space advisor ENABLED sql tuning advisor ENABLED 

Cause

This is expected Behaviour. The views assume that there is a one to many relationship between CLIENTS and TASKS.
Task can be used by different/multiple client. So even though if we disable the client, the DBA_AUTOTASK_TASK may still show the status as enabled.
In current version, the TASKS has only one CLIENT. But in future version of oracle, the TASKS can have multiple CLIENTS so the status in DBA_AUTOTASK_TASK will show as ENABLED.
So the correct way to check the status is through DBA_AUTOTASK_CLIENT.

Solution

Use DBA_AUTOTASK_CLIENT to check the status.

 SQL> select client_name,status from DBA_AUTOTASK_CLIENT; 

Why Auto Optimizer Statistics Collection May Appear to be "Stuck"?

Modified 12-FEB-2013

Applies to

Oracle Server - Enterprise Edition - Version: 11.1.0.7 and later
Information in this document applies to any platform.

Symptoms

• The 'auto optimizer stats collection' task is enabled in auto task
• STATISTICS_LEVEL has already been set to TYPICAL or FULL
• Statistics on tables are stale.
• Some scheduler window is active and LAST_START_DATE is several days ago.
 SELECT window_name,last_start_date,enabled,active FROM DBA_SCHEDULER_WINDOWS; WINDOW_NAME LAST_START_DATE ENABLE ACTIVE --------------- ----------------------------- ------ ------ SATURDAY_WINDOW 26-Mar-11 06.00.00.019249 am +08:00 TRUE TRUE 
• In Alert log, there is a shutdown immediate after the window has opened but before the window has closed:

 Sat Mar 26 23:33:23 2011 Stopping background process SMCO Stopping background process FBDA Shutting down instance: further logons disabled Sat Mar 26 23:33:26 2011 Stopping background process CJQ0 Stopping background process QMNC Stopping background process MMNL Stopping background process MMON Shutting down instance (immediate) <==system shutdown immediate 
• The 'auto optimizer stats collection' task is assigned to a window group:

Solution

Solution is to close the active window manually:

 EXECUTE DBMS_SCHEDULER.CLOSE_WINDOW ('SATURDAY_WINDOW'); 
Then ensure the window has been closed:

select window_name,active from DBA_SCHEDULER_WINDOWS;

 

                      WINDOW_NAME                       ACTIVE
------------------------------    ------
MONDAY_WINDOW                     FALSE
TUESDAY_WINDOW                    FALSE
WEDNESDAY_WINDOW                  FALSE
THURSDAY_WINDOW                   FALSE
FRIDAY_WINDOW                     FALSE
SATURDAY_WINDOW                   FALSE
SUNDAY_WINDOW                     FALSE
WEEKNIGHT_WINDOW                  FALSE
WEEKEND_WINDOW                    FALSE

 
At the start time for the next window, the auto task will run automatically again.

12c Cloud Control: Details of the Directory Structure and Commonly Used Locations in a 12c OMS Installation

Modified 15-FEB-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later [Release: 12.1 and later ]
Information in this document applies to any platform.

Goal

This document provides details about the Directory Structure and list of commonly used locations in a 12c Cloud Control Oracle Management Service (OMS) installation.

Solution

Directory Structure in 12c OMS Installation

Following are some of the important directories in a 12c OMS installation:

1. Middleware Weblogic home: Also referred to as <MW_HOME>, this is the directory under which 10.3.5 Weblogic software is installed.
2. Oracle Management Service Instance Base: Also referred to as <EM_INSTANCE_BASE>; or <EM_CONFIG_HOME>, this is a directory,
by default named gc_inst under the <MW_HOME>.
3. Oracle Management Service home: Also referred to as <OMS_HOME>, this is the location where the 12c OMS software is installed and is a directory
under the <MW_HOME>.
4. Oracle Management Service Instance home directory: Also referred to as <EM_INSTANCE_HOME>, this is location where the EM Instance is created
and is a directory under the gc_inst directory.
5. Middleware Common home directory: This directory contains some of the common tools such as Opatch, local inventory etc.
6. Middleware WebTier Home: Location where the OPMN and HTTP Server software is installed under the Middleware Home.
7. Middleware WebTier Instance home: Location where the HTTP Server and OPMN instance related files are stored. This is a directory under the gc_inst directory.
8. Weblogic Domain home directory: Also referred to as the <DOMAIN_HOME>, this is the location where the WLS Domain related files are created.
By default, the OMS installation will result in ADMIN_SERVER and EMGC_OMS1 managed server.
9. Admin Server Location: Location where the ADMIN_SERVER files are created under the DOMAIN_HOME.
10. Managed Server Location: Location where the Managed server files are created under the <DOMAIN_HOME>.
11. EM NodeManager directory: This is the location where the EM specific Nodemanager is installed and is a directory under the gc_inst.
12. Oracle Management Agent Base directory: Also referred to as <AGENT_BASE>, this is the root of the 12c Agent installation and is a directory under the <MW_HOME>.
This inturn contains multiple Homes for the Agent software bits, plugins, setuid binaries and the Agent State Home.
13. Oracle Management Agent Instance Home: Also referred to as the Agent State home or <AGENT_INSTANCE_HOME> or <EMSTATE>,
this is the location where the 12c Agent instance files are created.
14. Oracle Management Service plugins directory: This is the location containing the plugins installed at the OMS side.
15. Oracle Management Agent plugins directory: This is the location containing the plugins installed at the Agent side.

The above locations can be graphically represented as below:

If the 10.3.5 Weblogic is installed at /home/oracle/Middleware, then:

Name Directory
1 MW_HOME /home/oracle/Middleware
2 EM_INSTANCE_BASE /home/oracle/Middleware/gc_inst
3 OMS_HOME /home/oracle/Middleware/oms
4 EM_INSTANCE_HOME <EM_INSTANCE_BASE>/em/EMGC_OMS1
5 Middleware Common Home /home/oracle/Middleware/oracle_common
6 Middleware WebTier Home /home/oracle/Middleware/Oracle_WT
7 MiddlewareWebtier Instance Home /home/oracle/gc_inst/WebTierIH1
8 Weblogic Domain Home <EM_INSTANCE_BASE>/user_projects/domains/GCDomain/servers
10 EM Managed Server Location <Domain Home>/EMGC_OMS1
11 EM NodeManager Location <EM_INSTANCE_BASE>/NodeManager/emnodemanager
12 Agent Base directory /home/oracle/Middleware/agent
13 AGENT_INSTANCE_HOME /home/oracle/Middleware/agent/agent_inst
14 OMS Plugin Home /home/oracle/Middleware/plugins
15 Agent Plugin Home /home/oracle/Middleware/agent/plugins

To identify the above locations:

1. Open the 12c <OMS_HOME>/sysman/config/emInstanceMapping.properties file. This has a single entry of the type:

 EMGC_OMS1=/home/oracle/Middleware/gc_inst/em/EMGC_OMS1/emgc.properties 
In the above entry, the OMS Instance name=EMGC_OMS1

2. The emgc.properties file listed in the emInstanceMapping.properties will contain the above listed locations.

NOTE: The above directory structure is applicable to a 12c Cloud Control setup where EM software is used to install the JDK and WLS and Simple Installation method is chosen.

- If JDK and Weblogic are pre-installed prior to the EM installation, then the JDK location would differ as per the installation.

- If Advanced Installation option is chosen in the OUI, user can choose to create the gc_inst in a different location.

12c: How to Check which OPatch version was used to apply the Bundle Patch 1 or One-off patches

Modified 08-MAR-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Goal

How to determine which OPatch version was used to apply Bundle Patch 1 or one-off patches to the ORACLE_HOMEs.

Solution

1. Locate the opatch_history.txt file which is normally in:

• OMS: <OMS_HOME>/cfgtoollogs/opatch

Example: /u01/middleware/oms/cfgtoollogs/opatch

• Agent:

Example: ../../agent/core/12.1.0.1.0/cfgtoollogs/opatch

2. View the file and look for the version used:

 Date & Time : Tue Mar 06 14:53:51 IST 2012 Oracle Home : /u01/middleware/oms OPatch Ver. : 11.1.0.9.4 Current Dir : /u02/bundle_patch1/13470978 Command : apply -invPtrLoc /u01/middleware/oms/oraInst.loc Log File : /u01/middleware/oms/cfgtoollogs/opatch/13470978_Mar_06_2012_14_53_51/apply2012-03-06_14-53-50PM_1.log From the above information we can understand that the Opatch version used was 11.1.0.9.4 to apply the Patch 13470978. 

12c Cloud Control: Overview of the EMCTL Options Available for Managing the 12c OMS

Modified 16-OCT-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Solution

For UNIX:

 export ORACLE_HOME=/oracle/product/Middleware/oms 
Description of emctl options:

Command Description
Commands for Repository credentials/connectivity:
emctl config oms -list_repos_details Lists the Management Service repository details i.e. gives the information about Connect Descriptor repository host, SID along with the "Repository User"

emctl config oms -store_repos_details (-repos_host -repos_port -repos_sid | -repos_conndesc ) -repos_user [-repos_pwd ] [-no_check_db] Changes and stores Management Repository Details in the OMS.
emctl config oms -change_repos_pwd [-change_in_db] [-old_pwd ] [-new_pwd ] [-use_sys_pwd [-sys_pwd ]] Change the repository password.

-change_in_db

This option will change in repository too. If not specified, only Credential Store will be updated.

"12c Cloud Control: How to Modify the Password for SYSMAN and other Enterprise Manager Users at OMS and Repository?" on page 28

emctl config oms -change_view_user_pwd [-sysman_pwd ] [-user_pwd] [-auto_generate] To change the password of the MGMT_VIEW user.

-auto_generate

This option will generate a random password.
emctl config repos -host -oh -conn_desc "<TNS connect descriptor>" Relocate the repository database target to the Agent running on host "B" by running the following command from the OMS.
emctl config emrep -conn_desc "<TNS connect descriptor>" Change the monitoring configuration for the OMS and Repository target: by running the following command from the OMS.
emctl config emrep [-sysman_pwd <sysman password>][-agent <new agent>] [-conn_desc [<jdbc connect descriptor>]] In case of Management Repository Switch to a Primary/Standby host or in case the Management Repository is restored to a different hots, this command relocates the Management Repository to a new monitoring Management Agent and provides the new Connect Descriptor.
emctl resync repos -full -name "<resync name>" Initiate Repository Resync from one of the OMS Oracle Homes.
emctl abortresync repos (-full|-agentlist "agent names") -name "resync name" [-sysman_pwd "sysman password"] Aborts the currently running repository resynchronization operation. Use the full option to stop a full repository resynchronization. Use the agentlist option to stop resync on a list of agent.
emctl statusresync repos -name "resync name" Lists the status of given repository resynchronization operation.
OMS Start | Stop | Status commands:
emctl start oms 1. Start OPMN and OHS if not already up.
2. Start the Node Manager if not running.
3. Check if running on Admin Server. If yes, start it, if already not up.
(Recognizes using emgc.properties file)
4. Start the Managed server through Node Manager.
emctl stop oms 1. Stop the OHS.
2. Stop the OPMN.
3. Stop the Managed server.
emctl stop oms -all 1. Stop the OHS.
2. Stop the OPMN.
3. Stop the Managed server.
4. Stop the Admin server if running.
6. Stop the Node Manager.
emctl stop oms -all -force
OR
emctl stop oms -force
If the emctl stop oms commands do not shutdown the relevant processes, using -force option will forcefully stop the relevant processes.
emctl status oms Checks the status of the OMS and provide the current status.

-details

Lists the status of the Management Service in detail i.e. provides the secure status and prots used.
emctl list oms Gives the OMS name configured in that local oracle home:

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
OMS Instance(s) associated with current Oracle Home:
EMGC_OMS1*
emctl dump oms Gives the facility of capturing all node manager logs, admin server logs, managed server logs, oms sysman logs and stack trace files.

"EM 12c: Use of the Enterprise Manager Cloud Control 12.1.0.1 .../oms/bin/emctl dump Options to Collect OMS Log Files" on page 32

emkey commands:

NOTE: The emkey is an encryption key that is used to encrypt and decrypt sensitive data in Enterprise Manager such as host passwords, database passwords and others.

emctl status emkey [-sysman_pwd] This command shows the health or status of the emkey i.e. whether the EM key is configured properly or not. Depending on the status of the emkey, the following messages are displayed:
1. When the emkey has been correctly configured in the Credential Store, the following message is displayed.

The EmKey is configured properly, but is not secure. Secure the EmKey by running "emctl config emkey -remove_from_repos

2. When the emkey has been correctly configured in the Credential Store and has been removed from the Management Repository, the following message is displayed.

The EMKey exists in the Management Repository, but is not configured properly or is corrupted in the credential store.
Configure the EMKey by running "emctl config emkey -copy_to_credstore.

3. When the emkey is corrupted in the Credential Store and removed from the Management Repository, the following message is displayed.

The EMKey is not configured properly or is corrupted in the credential store and does not exist in the Management Repository.
To correct the problem:

1) Get the backed up emkey.ora file.
2) Configure the emkey by running "emctl config emkey -copy_to_credstore_from_file"
emctl config emkey -copy_to_credstore [-sysman_pwd] This command copies the emkey from the Management Repository to the Credential Store.
emctl config emkey -copy_to_repos [-sysman_pwd] This command copies the emkey from the Credential Store to Management Repository.
emctl config emkey -remove_from_repos [-sysman_pwd] This command removes the emkey from the repository.

NOTE: If the emkey is corrupted, you cannot remove it from the Management Repository.

emctl config emkey -copy_to_file_from_credstore -admin_host -admin_port -admin_user [-admin_pwd] [-repos_pwd] -emkey_file This command copies the emkey from the Credential Store to a specified file.
emctl config emkey -copy_to_file_from_repos (-repos_host -repos_port -repos_sid | -repos_conndesc) -repos_user [-repos_pwd] [-admin_pwd] -emkey_file This command copies the emkey from the Management Repository to a specified file.
emctl config emkey -copy_to_credstore_from_file -admin_host -admin_port -admin_user [-admin_pwd] [-repos_pwd] -emkey_file This command copies the emkey from a specified file to the Credential Store.
emctl config emkey -copy_to_repos_from_file (-repos_host -repos_port -repos_sid | -repos_conndesc) -repos_user [-repos_pwd] [-admin_pwd] -emkey_file This command copies the emkey from a specified file to the repository.
Secure Commands
emctl secure oms [-sysman_pwd] [-reg_pwd] [-host] [-slb_port] [-slb_console_port] [-reset] [-console] [-lock] [-lock_console] [-secure_port] [-upload_http_port]
[-root_dc] [-root_country] [-root_email] [-root_state] [-root_loc] [-root_org] [-root_unit] [-wallet -trust_certs_loc] [-wallet_pwd] [-key_strength] [-cert_validity] [-protocol]
Enables the OMS to accept upload requests from Agents and Console requests in https mode.
adminCredsWallet directory. This command is to recreate Administrator Credentials wallet on each machine on which the Management Service is running

To prevent Management Agents from uploading data to the Management Service over HTTP.
-console

To allow the OMS to accept uploads from unsecure Management Agents.
-console

emctl secure console -wallet To configure the third party certificate for HTTPS WebTier Virtual Host.

Note: Only single-sign-on wallets are supported.
emctl secure setpwd [authpasswd] [newpasswd] Creates a new Agent registration password.

"12c Cloud Contol: How to Create / Edit the Agent Registration Password for Agent to OMS Secure Communication" on page 32

Miscellaneous:
emctl list properties [-sysman_pwd "sysman password"] [-module] Lists all the properties set from emoms.properties file.
emctl get property [-sysman_pwd "sysman password"] -name Gets the property value from emoms.properties file for the given property name.
emctl set property [-sysman_pwd "sysman password"] -name -value[-module (default emoms)] Sets the property values in emoms.properties file.

-name

This option requires a property name for which a new value needs to be set.

-value

This option requires a property value to be set.

"12c Cloud Control : How To View and Update OMS Configuration Properties Using 'emctl' utility?" on page 33

emctl delete property [-sysman_pwd "sysman password"] -name Removes the property from the emoms.properties file for the given property name.
emctl importconfig oms -file Imports the exported configuration on the standby Management Service.
emctl exportconfig oms [-sysman_pwd ] [-dir ] [-keep_host] Exports the configuration from the primary Management Service.

-dir

Specifies directory to store backup file.

-keep_host

Specify this parameter if the OMS was installed using a virtual hostname.
emctl statusresync repos -name "resync name" Lists the status of given repository resynchronization operation.
emctl enroll oms -as_host -as_port port> Re-enroll the additional OMS, if any, with the recovered Administration Server.

Description of Available Properties:

Property Description
-sysman_pwd Oracle Management Repository user password.
- repos_user The Management Repository user name. The default value is SYSMAN.
- repos_pwd Oracle Management Repository user password.
-repos_conndesc The Management Repository Oracle Net Connect String for the repository database.
The values specified for properties -repos_sid, -repos_host, and -repos_port must be
the same as that of HOST, PORT, and SERVICE_NAME in the connect string. If this
property is not specified, then -repos_sid, -repos_host, and -repos_port properties
are used to construct the connect descriptor.

NOTE: Connect descriptor should be enclosed in quotes, for example,'""' or "''"

-repos_sid The System Identifier (SID) for the database where the Management Repository schema resides.
-repos_host The name of the server or host computer where the repository database resides.
-repos_port The port number for the repository database.
-emkey_file The emkey is an encryption key. By default, the emkey is stored in the <OMS_HOME>/sysman/config/emkey.ora file.
The location of this file can be changed.
Secure command options:
-host Host name to be used in the certificate used by the Oracle Management Service.
You may need to use the SLB host name if there is an SLB before the Management Service.
-reset A new certificate authority will be created. All the Agents and Oracle Management Services need to be resecured.
-secure_port port to be used for secure communication.
-slb_port This parameter is required when Server Load Balancer is used. It specifies the secure upload port configured in the Server Load Balancer.
-slb_console_port This parameter is required when Server Load Balancer is used. It specifies the secure console port configured in the Server Load Balancer.
-root_dc The domain component used in the root certificate. The default value is com.
-root_country The country to be used in the root certificate. The default value is US.
-root_state The state to be used in the root certificate. The default value is CA.
-root_loc The location to be used in the root certificate. The default value is EnterpriseManager ON.
-root_org The organization name to be used in the root certificate. The default value is EnterpriseManager ON.
-root_unit The organizational unit to be used in the root certificate. The default value is EnterpriseManager on.
-root_email The email address to be used in the root certificate. The default value is EnterpriseManager@.
-wallet This is the location of the wallet containing the third party certificate.
This parameter should be specified while configuring third party certificates.
-trust_certs_loc The location of the trusted_certs.txt (required when third party certificates are used).
-key_strength The strength of the key to be used. Valid values are 512, 1024, 2048, and 4096.
-cert_validity The number of days for which the self-signed certificate is valid. The valid range is between 1 to 3650.
-protocol This parameter is used to configure Oracle Management Service in
TLSv1-only or SSLv3-only or mixed mode (default). Valid values are the allowed
values as per Apache's SSLProtocol directive.

Agent upload fails with: ERROR-400|ORA-01465: invalid hex number

Modified 20-JAN-2013

Applies to

Enterprise Manager Base Platform - Version 10.2.0.1 to 11.1.0.1 [Release 10.2 to 11.1]
Information in this document applies to any platform.

Symptoms

Agent fails to upload to the OMS.

emagent.trc file contains the following errors repeatedly:

ERROR-400|ORA-01465: invalid hex number

Cause

The issue can happen if the policy_guid value in the xml file the agent is trying to upload is corrupted.

We can see similar to the following corrupted data in the xml file which is failed to load (In this example, file is B0000002.xml)

 <POLICY_GUID><![CDATA["~ï¿½p]]></POLICY_GUID> 
A bug has been issued for this issue.

Solution

The BUG was not resolved. The issue was resolved by upgrading the agent to the later versions (10.2.0.3/10.2.0.4/10.2.0.5) or by using the following workaround:

1. Stop the agent with

 AGENT_HOME/bin>./emctl stop agent 

2. Please remove the corrupted xml file from the following location.

3. Remove

AGENT_HOME/sysman/emd/lastupld.xml

4. Start the agent with

 AGENT_HOME/bin>./emctl start agent 

 AGENT_HOME/bin>./emctl upload agent 

How To Find Database Services Which Are Registered To A Listener Target In Enterprise Manager Cloud Control 12c

Modified 26-NOV-2011

Applies to

Enterprise Manager for Oracle Database - Version: 12.1.0.1.0 and later [Release: 12.1 and later]
Information in this document applies to any platform.

Goal

This document explains how to find database services which are registered to a Listener target?

Solution

1. From the home page, Navigate to: Enterprise → Configuration → Search

2. Click on create

3. Choose → Target Type: Listener

4. Click on the "Magnifier" icon in front of the "Target Name" label and pick a listener
5. Choose → Configuration Item: Listener services

6. Click on Search

EM 12c: Creating an Administrator Role in Enterprise Manager Cloud Control 12.1.0.1

Modified 19-MAR-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Goal

This article provides a quick reference for creating an administrator role in Enterprise Manager Cloud Control 12c.

Solution

One Super Administrator, sysman, is created by default during the installation of the Enterprise Manager. By logging into the Cloud Control 12c Console as the sysman Super Administrator, other administrator accounts can be created for daily administration work.

NOTE: It is recommended that the sysman account should only be used to perform infrequent system wide, global configuration tasks.

Steps to create an administrator role in Cloud Control 12c:

From the Setup menu, select Security, then select Roles.

Select Create in the Roles page to launch the Create Role wizard.

In step 1 of 6, the Properties page, provide a name and description (example.Test_Role) for the role and select Next.

Provide a name and description (Example.Test_Role_Viewer) for the role and select Next.

In step 2 of 6, the Roles page, select the role as required and Next.

Select privileges as required in step 3 of 6, the Target Privileges page.

Select the "Add" button in the Target Privileges section and add the Targets in the provided menu as required.

In step 4 of 6, the Resource Privileges page, you can manage the Privilege Grants by clicking on the pencil icon.

In step 5 of 6 skip the Create Role: Administrators step and select "Next".

In step 6 of 6 review the role and select "Finish".

Please refer to our technical tip:

"EM 12c: Creating an Administrative User and Assigning an Administrative Role in Enterprise Manager Cloud Control 12.1.0.1" on page 33

Tips and Tricks on Debugging the 12c Harvester

Modified 03-JAN-2013

Applies to

Oracle Configuration Manager - Version 10.3.6.0.0 to 10.3.6.0.0 [Release 10.3]
Information in this document applies to any platform.

Purpose

The following compilation of errors focuses on the overall 12c GridControl Harvester feature.
In this note, we will attempt to record errors which under certain conditions may trigger.
Therefore, this note will be the main source of troubleshooting the 12c GC Harvester.

There will be 3 categories:

- CSI Assignment UI
- GC Harvester Registration

Troubleshooting Steps

CSI Assignment UI

Scenario 1

GC Harvester has never run. Possible causes:

- The Harvester job is disabled.

- The OCM collector is not installed/configured in the OMS_Home location.

To Do : See errors in emoms_pbs.trc located in the <Middleware_Home>/gc_inst/em/EMGC_OMS1/sysman/log

Or you can also > egrep -i emoms_pbs.trc |grep harvester

Scenario 2

GC Harvester has not run in the last {0} days. Possible causes:

- The Harvester Job was running successfully for a period of time but now its not running.

- The Harvester job has been disabled.

- The OCM collector is not installed/configured in the OMS_Home location.

To Do : See errors in emoms_pbs.trc or mgmt_system_error_log table (Setup > Management Services and Repository > Repository Operations)

If there are no errors being reported in the trace files, then perhaps the GC Harvester needs to be started and scheduled.
In that case, you need to start the Harvester using the gcharvester_job.sql script found in the OMS_HOME/sysman/admin/emdrep/sql/core/latest/gcharvester directory:

This is what it looks like:

 declare l_already_registered NUMBER :=0; l_job_targets MGMT_JOB_TARGET_LIST := MGMT_JOB_TARGET_LIST(); l_params MGMT_JOB_PARAM_LIST := MGMT_JOB_PARAM_LIST(); l_schedule MGMT_JOB_SCHEDULE_RECORD; l_job_id_out RAW(16); l_execution_id_out RAW(16); l_version varchar2(32); begin -- first check if job is already registered (possibly in earlier version of EM) BEGIN SELECT 1 into l_already_registered FROM MGMT_JOB WHERE job_name='GCHARVESTERJOB' AND job_type='GCHarvesterRun'; EXCEPTION WHEN NO_DATA_FOUND THEN null; END; IF (l_already_registered = 1) THEN RETURN; END IF; -- start a daily job, running at 2 am l_schedule := MGMT_JOBS.get_job_schedule_record(p_frequency_code => MGMT_JOBS.DAILY_FREQUENCY_CODE, p_start_time => SYSDATE + 1, p_end_time => null, p_execution_hours => 2, p_execution_minutes => 0, p_interval => 0, p_months => null, p_days => null, p_timezone_info => MGMT_JOBS.TIMEZONE_REPOSITORY, p_timezone_target_index => 0, p_timezone_offset => 0, p_timezone_region => null, p_start_grace_period => 240); MGMT_JOBS.submit_job(p_job_name => 'GCHARVESTERJOB', p_description => 'GC Harvester Job', p_job_type => 'GCHarvesterRun', p_job_targets => l_job_targets, p_job_params => l_params, p_schedule => l_schedule, p_job_id_out => l_job_id_out, p_execution_id_out => l_execution_id_out, p_owner => null, p_system_job => MGMT_JOB_ENGINE.SYSTEM_JOB, p_job_creds => null, p_job_target_type => null, p_job_notify_states => null, p_running_time_limit => -1); update MGMT_JOB set job_owner= MGMT_USER.GET_REPOSITORY_OWNER where job_name='GCHARVESTERJOB' AND job_type='GCHarvesterRun'; MGMT_LOG.register_logging_module('GCHARVESTER','GC Harvester', NULL); commit; end; / 
The two main arguments are : hour to run (execution_hours) and minute to run (execution_minutes).
As long as the time entered is not in the past, the script will run fine allowing the GC Harvester to function properly.

Scenario 1

Now in 12c, the GCHarvester uploads its data directly to the CCR repository. As part of this process flow, the collection files are deleted as soon as they are uploaded.
However in some cases where a collection is failing or there is a concern about targets and their timestamps,
You may have to set the following parameter to retain those collection files so they can be reviewed.

With a connection to the repository as a SYSDBA user, the delete_upload_file property in the mgmt_ocm_upl_props table needs to be set to False as follows:

 insert into mgmt_ocm_upl_props(name, str_value) values('delete_upload_file','false') 

NOTE: At this point the OMS needs to be restarted in order for that parameter to take affect.

By Default, the files will be created in the $INSTANCE_HOME/sysman/ocm/harvester/tmp location. The files themselves will be incremental in nature. Scenario 2 In some cases, the following error may be noticed in the emoms_pbs.trc file:  2012-03-29 02:00:04,261 [JobWorker 208183:JobThreadPool:Long-System:Thread-641692] ERROR gcharvester.HarvesterJobUtils performHistoryBasedUpdates.? - GC OCM Harvester: Failed to populate OCM metrics based on ECM history: ORA-01878: specified field not found in datetime or interval ORA-06512: at line 358  This is a clear indication that you are hitting Bug 13972728 which has the following workaround as of OCM 10.3.7.0 1. Locate the OMS instance home. 2. In the$ORACLE_HOME/sysman/config/emInstanceMapping.properties file (where ORACLE_HOME is the Oracle Home of the OMS), there is an entry referencing a file called emgc.properties.
3. The directory in which the emgc.properties file is located is the "instance home" of the OMS. In the following example, /u01/app/oracle/product/gc_inst/em/EMGC_OMS1 is the instance home of the OMS:

 EMGC_OMS1=/u01/app/oracle/product/gc_inst/em/EMGC_OMS1/emgc.properties 

4. Set the environment variable ORACLE_CONFIG_HOME to the directory of this emgc.properties file.

Example:

 $export ORACLE_CONFIG_HOME=/u01/app/oracle/product/gc_inst/em/EMGC_OMS1  5. cd$ORACLE_HOME/ccr/bin
emCCR register

6. The next scheduled time the harvester job runs, it will trigger a full collection.
In a multiple OMS environment, this 'emCCR register' will need to be completed in all locations.

Scenario 3

In an E-Business Suite environment, the Harvester is not 'EBS' aware. In other words, EBS is not currently a harvested target from Cloud Control and therefore,
the OCM standalone product will need to be used in order to upload the data to My Oraclce Support.

Scenario 4

During the scheduled harvester run (default is 2AM), the job fails and the following error is dumpted into the emoms.trc file :

 2012-10-20 02:00:30,781 [JobWorker 73854412:Thread-86] ERROR gcharvester.HarvesterJobUtils prepareTargetsForShipment.? - GC OCM Harvester: Failed to stage targets for shipment to Oracle: ORA-01031: insufficient privileges ORA-06512: at line 9 
At this point, we are trying to create a sequence object and failing. SYSMAN lacks the privilege to create sequence. This sequence is used to put all harvested targets in mgmt_ocm_upl_metrics_tmp and then process them in batches of 512 targets at a time.
First 1 to 512 will be processed, then 513 to 1024 and so on. The fix here is to assign the SYSMAN user the CREATE SEQUENCE priv :

 GRANT CREATE SEQUENCE,DROP ANY SEQUENCE TO SYSMAN; 

GC Harvester Registration

Some Backround First

At first run, the GCHarvester performs a GC Default registration using the registration information of the OCM collector in the OMS location. As part of this registration,the EM Repository URL is also sent for book keeping purposes on the CCR side of things.
All other Oracle_Homes which are considered to be a 'Proxy' uploaded by the GCHarvester, is uploaded using the GC Default registration (this is similar to 11GC).

When a Target is added, this Target Oracle Home registration is performed after a user goes to UI CSI assignment page and assigns a CSI to it.

The registration process itself has a 2 step approach.

1. Authenticate: Done as part of the UI which authenticates with the CCR database.
2. Register: The GCHarvester internal job picks up on any 'Pending' targets and registers itself against our CCR database.

Also new in 12c is that the GCHarvester now directly uploads information to CCR directly and no longer relies on OCM to perform this task. It also deletes the files as soon as they are uploaded.

Scenario 1

Problem in assigning a Customer Support Identifier(CSI). UI Error: "You don't have the privilege to do this operation". Possible cause:

- The User itself does not have the Operator privilege on one or more targets within a given Oracle_Home for that target.

Scenario 2

The CSI drop down listing does not show any CSI information. To do:

- There is a chance that the My Oracle Support(MOS) connectivity from the OMS location is not successful. The emoms.trc file will have to be reviewed for any possible communication issue.

Scenario 3

The CCR authentication has failed. Possible cause:

- Since we use the CSI selected from the UI and MOS preferred ID for the CCR authentication process, that needs to be verified to be correct.
Check if the OCM collector can be configured using these same credentials that is being used in the UI under the My Oracle Support > Preferred Credentials > My Oracle Support Preferred Credentials page

Installation and Configuration

Q. Why do I see duplicate Targets in My Oracle Support after upgrading from 11GC to 12c?

A. This can be a result of not supplying the same My Oracle Support credentials that was used in the 11GC OMS Home when OCM was configured. Make sure that in the 12c OMS home location,
the OCM collector is installed with the same credentials that was used in 11GC OCM location so that no duplicate target are created in MOS.

Q. Why don't I see the OMS and Agent as a target after upgrading, or installing a new 12c environment?
A. New in a 12c OCM / Harvester installation, the OMS and Agent targets are now collected via the GridControl Harvester and not OCM directly. Therefore,

Q. Why don't I need to install OCM in an Agent home in a 12c environment?

A. As of 12c, the Agent is considered to be a 'Converged' target. That means that the Harvester now treats both the OCM and 12c metric collections for this target type the same. This convergence is to avoid duplication in configuration metrics and piggy-back on the new Harvester functionality.

How to Change the Retention Period or Purge Policy for Enterprise Manager Cloud Control 12c Job History?

Modified 29-JAN-2013

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

The Enterprise Manager default purge policy for jobs deletes all finished jobs that are older than 30 days. This value cannot be changed using Enterprise Manager, but it can be changed using SQL*Plus.

The details in this document is applicable only to the Job history seen in the Jobs tab in the Enterprise Manager and not to the Database-side scheduler jobs.

NOTE: The default purging policies have been designed to provide the most available data for analysis while still providing the best performance and disk-space requirements for the Management Repository.
As a result, these should not be modified to improve performance or increase the available disk space. Increasing the retention time drastically can affect the performance of the Management Repository and have adverse reactions on the scalability of the Enterprise Manager installation.

Fix

The actual purging of completed job history is implemented via a DBMS job that runs once a day in the repository database. When the DBMS job runs in the repository database, it looks for finished Enterprise Manager Jobs that have a history which is 'n' number of days older than the current time
(value of sysdate in the repository database) and deletes the job history details. The value of 'n' is, by default, set to 30 days.

The default purge policy is called SYSPURGE_POLICY. To check the current values for this purge policies , log into the Management Repository database as sysman user and execute:

 SQL>connect sysman/<sysman_pwd>; SQL>Connected. SQL>select * from mgmt_job_purge_policies order by policy_name; 

                      POLICY_NAME                      TIME_FRAME
-------------------------------- ----------
CERTEOL_CERT_MD_JOBPURGEPOLICY            7
CERTEOL_EOL_MD_JOBPURGEPOLICY             7
CERTEOL_PLATCH_JOBPURGEPOLICY             7
OPATCHPATCHUPDATE_PAPURGEPOLICY           7
SYSPURGE_POLICY                          30

 
To change the retention for the policy SYSPURGE_POLICY, you need to drop and recreate the policy with the new retention period.

Example to change the retention policy from the default value of 30 days to the new value of 45 days:

 SQL>connect sysman/<sysman_pwd>; SQL>Connected. SQL>execute MGMT_JOBS.drop_purge_policy('SYSPURGE_POLICY'); SQL>execute MGMT_JOBS.register_purge_policy('SYSPURGE_POLICY', 45, null); SQL>COMMIT; 
 SQL>select * from mgmt_job_purge_policies order by policy_name; 
 POLICY_NAME TIME_FRAME -------------------------------- ---------- CERTEOL_CERT_MD_JOBPURGEPOLICY 7 CERTEOL_EOL_MD_JOBPURGEPOLICY 7 CERTEOL_PLATCH_JOBPURGEPOLICY 7 DOWNLOADCVU_PURGEPOLICY 7 OPATCHPATCHUPDATE_PAPURGEPOLICY 7 REFRESHFROMMETALINKPURGEPOLICY 7 SYSPURGE_POLICY 45 

NOTE: If you forgot to drop the policy, the create policy command will fail with:

 SQL>execute MGMT_JOBS.register_purge_policy('SYSPURGE_POLICY', 60, null); SQL>BEGIN MGMT_JOBS.register_purge_policy('SYSPURGE_POLICY', 45, null); END; ERROR at line 1: ORA-20416: Purge policy exists ORA-06512: at "SYSMAN.EM_JOB_PURGE", line 247 ORA-06512: at "SYSMAN.MGMT_JOBS", line 688 ORA-06512: at line 1 

If you want to check when the purge job will be executed and the command run, execute:

Enterprise Manager Cloud Control 12.1.0.1:

 SQL> connect sysman/<sysman_pwd>; Connected. SQL> select job_name, job_action, next_run_date from user_scheduler_jobs where job_name = 'EM_JOB_PURGE_POLICIES'; JOB_NAME JOB_ACTION NEXT_RUN_DATE ----------------------- --------------------------------------- ------------------------------------------> EM_JOB_PURGE_POLICIES MGMT_JOB_ENGINE.apply_purge_policies(); 06-JUN-12 05.00.00.500000 AM EUROPE/ZURICH 

If you want to execute the purge job immediately, execute:

 SQL> exec MGMT_JOB_ENGINE.apply_purge_policies(); PL/SQL procedure successfully completed. 

Enterprise Manager Cloud Control 12.1.0.2:

 SQL> connect sysman/<sysman_pwd>; Connected. SQL> select job_name, job_action, next_run_date from user_scheduler_jobs where job_name = 'EM_JOB_PURGE_POLICIES'; JOB_NAME JOB_ACTION NEXT_RUN_DATE ----------------------- --------------------------------------- ------------------------------------------> EM_JOB_PURGE_POLICIES EM_JOB_PURGE.APPLY_PURGE_POLICIES(); 06-JUN-12 05.00.00.500000 AM EUROPE/ZURICH 

If you want to execute the purge job immediately, execute:

 SQL> exec MGMT_JOB_ENGINE.apply_purge_policies(); PL/SQL procedure successfully completed. 

How To Create A New ACFS Volume and Filesystem And Set The ACFS Filesystem Ownership To A Non-Grid/Oracle OS User?

Modified 29-JAN-2013

Applies to

Oracle Server - Enterprise Edition - Version: 11.2.0.1 to 11.2.0.4 - [Release 11.2]
Information in this document applies to any platform.

Goal

The present document provides an example about how to create a new ACFS volume and filesystem and set the ACFS filesystem ownership to a non-grid/oracle OS User.

Fix

You can set a different OS user to the ACFS filesystem and mount it with that different OS user as follow:

1. Connect to the ASM instance & create the ACFS diskgroup:

 SQL> CREATE DISKGROUP ACFSTEST EXTERNAL REDUNDANCY DISK 'ORCL:ASMDISK18' SIZE 4157 M DISK 'ORCL:ASMDISK19' SIZE 4157 M ATTRIBUTE 'compatible.asm' = '11.2', 'compatible.advm' = '11.2'; Diskgroup created. 

2. Create the ACFS volume:

 SQL> ALTER DISKGROUP ACFSTEST ADD VOLUME ACFSTESTVOL SIZE 7G; Diskgroup altered. 

3. Verify the volume was created and obtain the new volume name:

 [grid@dbaasm grid]$asmcmd ASMCMD> volinfo -a Diskgroup Name: ACFSTEST Volume Name: ACFSTESTVOL Volume Device: /dev/asm/acfstestvol-76 State: ENABLED Size (MB): 7168 Resize Unit (MB): 256 Redundancy: UNPROT Stripe Columns: 4 Stripe Width (K): 128 Usage: Mountpath:  4. Connect as root OS user and create the mount point directory ( e.g. "/goldengate"):  [grid@dbaasm grid]$ su - Password: [root@dbaasm /]# mkdir /goldengate 

5. Create the ACFS filesystem on the new volume ( e.g. "/dev/asm/acfstestvol-76"):

 [root@dbaasm ~]# /sbin/mkfs -t acfs -b 4k /dev/asm/acfstestvol-76 mkfs.acfs: version = 11.2.0.1.0.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/acfstestvol-76 mkfs.acfs: volume size = 7516192768 mkfs.acfs: Format complete 

6. Register the ACFS filesystem on the new ACFS volume:

 [root@dbaasm ~]# /sbin/acfsutil registry -f -a /dev/asm/acfstestvol-76 /goldengate acfsutil registry: mount point /goldengate successfully added to Oracle Registry 

7. Create the desired OS group (e.g. "ggate"):

 [root@dbaasm ~]# groupadd ggate [root@dbaasm ~]# grep ggate /etc/group ggate:x:1302: 

8. Create the desired OS user (e.g. "goldenguser") and associated it with the desired OS group(s) (e.g. "ggate"):

 [root@dbaasm ~]# useradd -u 1150 -g ggate goldenguser [root@dbaasm ~]# id goldenguser uid=1150(goldenguser) gid=1302(ggate) groups=1302(ggate) 

9. Mount the new mount point ("/goldengate") on the new ACFS volume ("/dev/asm/acfstestvol-76"):

 [root@dbaasm ~]# /bin/mount -t acfs /dev/asm/acfstestvol-76 /goldengate 
or

 [root@dbaasm ~]# /sbin/mount.acfs -o all 

10. Set the new OS ownership and set the new OS correct/desired permissions:

 [root@dbaasm ~]# chown goldenguser:ggate /goldengate [root@dbaasm ~]# chmod 775 /goldengate [root@dbaasm ~]# ls -ld /goldengate drwxrwxr-x 2 goldenguser ggate 4096 May 16 09:45 /goldengate 

11. Verify the new filesystem ("/goldengate") is mounted:

 [root@dbaasm ~]# df -k /goldengate Filesystem 1K-blocks Used Available Use% Mounted on /dev/asm/acfstestvol-76 7340032 51672 7288360 1% /goldengate 
 [grid@dbaasm grid]$asmcmd ASMCMD> volinfo -a Diskgroup Name: ACFSTEST Volume Name: ACFSTESTVOL Volume Device: /dev/asm/acfstestvol-76 State: ENABLED Size (MB): 7168 Resize Unit (MB): 256 Redundancy: UNPROT Stripe Columns: 4 Stripe Width (K): 128 Usage: ACFS Mountpath: /goldengate ASMCMD>  12. Test files creation (permissions & ownership) on the new filesystem:  [root@dbaasm ~]# su - goldenguser [goldenguser@dbaasm ~]$ pwd /home/goldenguser [goldenguser@dbaasm ~]$cd /goldengate [goldenguser@dbaasm goldengate]$ pwd /goldengate [goldenguser@dbaasm goldengate]$touch test.txt [goldenguser@dbaasm goldengate]$ ls -l total 64 drwx------ 2 root root 65536 May 16 09:55 lost+found -rw-r--r-- 1 goldenguser ggate 0 May 16 10:00 test.txt 

NOTE: For the additional syntax used on Linux, Solaris, AIX & Windows see the attached document:

Check for the following directories:

# 13-1 Summary of Oracle ACFS commands for Linux and UNIX

# 13-2 Options for the Linux fsck command
# 13-3 Options for the Linux mkfs command
# 13-4 Options for the Linux mount command
# 13-5 Options for the Linux umount command

# 13-6 Summary of Oracle ACFS commands for Solaris

# 13-7 Options for the Solaris fsck command
# 13-8 Options for the Solaris mkfs command
# 13-9 Options for the Solaris mount command
# 13-10 Options for the Solaris umount command

# 13-11 Summary of Oracle ACFS commands for AIX

# 13-12 Options for the AIX fsck command
# 13-13 Options for the AIX mkfs command
# 13-14 Options for the AIX mount command
# 13-15 Options for the AIX umount command

How To Create A New ACFS Volume & Filesystem And Set The ACFS Filesystem Ownership To A Non-Grid/Oracle OS User (White Paper)

12c: How to Modify JVM Heap Size, MaxPermSize for 12c Enterprise Manager Agent via Enterprise Manager 12c Cloud Control Console?

Modified 11-DEC-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0
Information in this document applies to any platform.

Goal

How to Modify/Increase JVM Heap Size for 12.1.0.1 Agent via Enterprise Manager Cloud Console?

This is useful for cases where the agent restarts frequently or becomes unresponsive and the following messages are seen in $<install directory of agent>/agent_inst/sysman/log/gcagent.log  oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response:Response java.lang.OutOfMemoryError: PermGen space 2012-02-02 01:54:24,409 [57:GC.Executor.1 (oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response)] ERROR - Critical error: java.lang.OutOfMemoryError: PermGen space 2012-02-02 04:03:41,660 [64:GC.Executor.8 (oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response) (oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response:Response)] ERROR - oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response:Response java.lang.OutOfMemoryError: PermGen space 2012-02-02 04:03:41,691 [64:GC.Executor.8 (oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response)] ERROR - Critical error: java.lang.OutOfMemoryError: PermGen space  Fix 1. Log on to 12c Cloud Console as the Super Administrator user (eg. 'sysman') 2. Setup → Agents 3. Click on Agent link (agent_name:port#) 4. Expand the arrow next to "Agent" to show the drop down menu, choose 'Properties' 5. Show: All Properties 6. Expand RunTime Settings 7. Edit the "agentJavaDefines" parameter value , as below and click on 'Apply' Default value of "agentJavaDefines" parameter is:- -Xmx128M -XX:MaxPermSize=96M Change to:- -Xmx512M -XX:MaxPermSize=96M 8. Verify the changes in AGENT_INST_HOME/sysman/config/emd.properties. Before changing the agentJavaDefine value: $ grep -i XM emd.properties agentJavaDefines=-Xmx128M -XX:MaxPermSize=96M After changing the agentJavaDefine value: $grep -i XM emd.properties agentJavaDefines=-Xmx512M -XX:MaxPermSize=96M  9. How to Configure Logging for the Enterprise Manager Cloud Control Management Agent12.1.0.1 (12c R1) Modified 22-FEB-2013 Applies to Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.1.0 [Release 12.1] Information in this document applies to any platform. Enterprise Manager Cloud Control 12.1 Goal The goal of this article is to provide instruction on how to enable logging for the Enterprise Manager Management Agent Release 12.1.0.1 (12c R1). Fix In EM 12c, the mechanism for configuring agent logging has not changed, but the process and syntax for enabling agent logging has changed slightly. The logging properties for the agent are stored in$EMSTATE/sysman/config/emd.properties file, same as in 11.1.
But the properties that control logging follow the log4j configuration syntax instead of the syntax supported by the 11.1 agent.

To active changes to the logging parameters, you can use either of the emctl commands:

• emctl setproperty agent

Overview

Agent logging entries look similar to the standard log4j logging, with the only significant difference being the prefix 'Logger.' that you will see on each line. For example:

 # logging properties Logger.log4j.appender.Rolling=org.apache.log4j.RollingFileAppender Logger.log4j.appender.Rolling.File=/orcl/core/12.1.0.1.0/agent_inst/sysman/log/gcagent.log Logger.log4j.appender.Rolling.Append=true Logger.log4j.appender.Rolling.MaxFileSize=5000000 Logger.log4j.appender.Rolling.MaxBackupIndex=10 Logger.log4j.appender.Rolling.layout=oracle.sysman.gcagent.util.logging.GCPattern # # Set root category priority to INFO and its appenders to Rolling (and Errors). Logger.log4j.rootCategory=INFO, Rolling, Errors 
If you examine all of the Logger.log4j entries in the emd.properties file, you will see different settings for an Errors appender and a Mdu appender.
• Errors appender is used to direct ERROR level logging entries to $EMSTATE/sysman/log/gcagent_errors.log • Mdu appender is used to direct metadata update entries to$EMSTATE/sysman/log/gcagent_mdu.log.

NOTE: Neither of these two appenders' configurations should be modified.

Default logging level

 Logger.log4j.rootCategory=INFO, Rolling, Errors, Test 
The rootCategory property controls the default log level. It is by default set to INFO.

Changing the default logging level

To enable DEBUG level logging across the entire agent, set the rootCategory property entry to use DEBUG, similarly to what is shown below.

 # # Set root category priority to INFO and its appenders to Rolling and Errors. # Change to DEBUG level for gcagent.log Logger.log4j.rootCategory=DEBUG, Rolling, Errors 
And then reload the agent with: $emctl reload agent Alternatively, use can also use the emctl setproperty agent command:$ emctl setproperty agent -name 'Logger.log4j.rootCategory' -value 'DEBUG,Rolling,Errors'

NOTE: Documentation shows the above command with spaces after each comma (ex: 'DEBUG, Rolling, Errors'). This will result in an error when you run the command.
Remove the spaces between 'DEBUG,Rolling,Errors'.

Setting the number of backup log files

When you set agent logging to DEBUG level, for anything but a most-trivial agent deployment, log entries have the possibility of rolling away after less than an hour.
For this reason, it is also sensible to increase the number of log file rolls if the entire agent is configured to log at the DEBUG level.
The number of rolls is set through the MaxBackupIndex parameter and it represents the number of backup log files to retain. It is recommended to keep no more than 100 files:

 # Increase to 100 rolls of the log (gcagent.log, gcagent.log.1,...,gcagent.log.100) Logger.log4j.appender.Rolling.MaxBackupIndex=100  And then reload the agent with: $emctl reload agent Alternatively, you can issue the command:$ emctl setproperty agent -name "Logger.log4j.appender.Rolling.MaxBackupIndex" -value 100

Example of Standard DEBUG settings

Append the following to $EMSTATE/sysman/config/emd.properties and reload the agent.  # Recommended settings to produce DEBUG level logging Logger.log4j.rootCategory=DEBUG, Rolling, Errors Logger.log4j.appender.Rolling.MaxBackupIndex=50  Example of Standard TRACE settings Append the following to to$EMSTATE/sysman/config/emd.properties and reload the agent.

 # Recommended settings to produce TRACE level logging Logger.log4j.rootCategory=DEBUG, Rolling, Errors Logger.log4j.appender.Rolling.MaxBackupIndex=100 Logger._enableTrace=true  Setting Log Level for Individual Classes and Packages

The logging level for individual class and/or packages can also be set. Here are couple of examples that are currently configured by default:

 # Set the class loaders to level INFO Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.ChainedClassLoader=INFO Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.ReverseDelegationClassLoader=INFO Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.PluginLibraryClassLoader=INFO Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.PluginClassLoader=INFO 
The above configuration changed the default level of logging for the four classes to be INFO.
When the default level of logging is INFO it doesn't make any difference but if the default log level will be set to DEBUG (when debugging the code)
it will prevent those four classes from logging at DEBUG level (as they are normally too verbose).
The reverse is also true, for example if the following configuration is added (not set by default):

 Logger.log4j.category.oracle.sysman.gcagent.metadata.impl.collection=DEBUG 
It will cause all classes in the oracle.sysman.gcagent.metadata.impl.collection package to log at DEBUG level even if the default log level is INFO.

gcagent.log Configuration

 Logger.log4j.appender.Rolling=org.apache.log4j.RollingFileAppender Logger.log4j.appender.Rolling.File=/ade/ysarig_v6/oracle/work/agentStateDir/sysman/log/gcagent.log Logger.log4j.appender.Rolling.Append=true Logger.log4j.appender.Rolling.MaxFileSize=5000000 Logger.log4j.appender.Rolling.MaxBackupIndex=10 
Logger.log4j.appender.Rolling.layout=oracle.sysman.gcagent.util.logging.GCPattern

The gcagent.log is configured using the rolling appender (properties that starts with Logger.log4j.appender.Rolling):
• File property controls the location and name of the main log, it is recommended not to change the setting for it.
• Append determine if the log file should be append (true) or overwritten (false) on agent startup
• MaxFileSize determine the size of each of the log segments.
• MaxBackupIndex determine the number of backup segments for the log (the total number of segments are this number plus one)

gcagent_error.log Configuration

 Logger.log4j.appender.Errors=org.apache.log4j.FileAppender Logger.log4j.appender.Errors.File=/ade/ysarig_v6/oracle/work/agentStateDir/sysman/log/gcagent_errors.log Logger.log4j.appender.Errors.Append=true Logger.log4j.appender.Errors.Threshold=ERROR 
The gcagent_errors.log contains log messages of ERROR and FATAL levels. The log is not segmented and it has no size limit. The gcagent_errors.log is a subset of the gcagent.log

Understanding the Management Agent Logs Files

To understand the Management Agent Logs Files, refer to our technical tip "EM 12c Cloud Control : Understanding the Management Agent Logs Files" on page 25.

EM 12c Cloud Control : Understanding the Management Agent Logs Files

Modified 01-OCT-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Purpose

This document describes how to locate and understand the various log/trace files of the Enterprise Manager Cloud Control Agent 12c.

Scope

Cloud Control Administrators who need to monitor the operation of the Agent.
Oracle Management Agent log and trace files store important and useful information in order to troubleshoot problems.

Details

About the Main Management Agent Log File : gcagent.log

The Agent main log file gcagent.log is located in the directory $EMSTATE/sysman/log. where$EMSTATE is the Agent instance directory. For example:

EMSTATE=<Agent_Oracle_Home>/agent_inst

NOTE: to find the Agent instance directory refer the "Locating the Management Agent Log and Trace Files" section below.

The gcagent.log is the log for the Agent framework. This log file contains trace, debug, information, error, or warning messages from the Agent.
It can be used for debugging agent framework issues.

The log is segmented by default to 11 segments, 5MB each.
The segments are named gcagent.log and gcagent.log.# → where # is a number in the range of 1-10.
These settings are controlled by properties in $EMSTATE/sysman/config/emd.properties as explained in the following sections. The latest segment is always gcagent.log and the oldest is the gcagent.log.X where X is the highest number. The log contains individual log messages with the following format:  YYYY-MM-DD HH:MM:SS,### [<tid>:<thread code or code:name>] <level> -<the message>  * YYYY-MM-DD HH:MM:SS,### is a timestamp (in 24 hours format and ### is the fraction in msec) * <tid> is the thread id (as a decimal number) * <thread name or code> is the thread full name or an abbreviated hexadecimal code (see the following example) * <level> is the logging level that can be one of (in ascending order of importance): DEBUG, INFO, WARN, ERROR, FATAL. * <the message> is the free text message that is being logged. The message can contain new lines and spawn multiple lines.  For example: 2011-06-07 15:00:00,016 [1:3305B9:main] DEBUG - ADR_BASE='/ade/example_user/oracle/example/agentStateDir' 2011-06-07 15:00:01,883 [1:3305B9] INFO - Agent is starting up  Logging Configuration The Agent can be configured to log at DEBUG or TRACE level to yield more complete diagnostics. To change the logging level of that file, refer our previous note: How to Configure Logging for EM 12c Management Agent Complete List of the Agent Log Files Log File Description gcagent.log The gcagent.log is the log for the Agent framework. This log file contains trace, debug, information, error, or warning messages from the Agent. It can be used for debugging Agent framework issues. The Agent can be configured to log at DEBUG or TRACE level to yield more complete diagnostics.(see above for all details). gcagent_errors.log The gcagent_errors.log contains log messages of ERROR and FATAL levels. The log is not segmented and it has no size limit. The gcagent_errors.log is a subset of the gcagent.log. emagent.nohup Watchdog launches the Agent process and dumps startup information in this file. After the agent is launched, the stdout and stderr of the Agent daemon is redirected to emagent.nohup. This log file does not provide logging levels that can be controlled. Its always enabled by default. Log entries generated by watchdog have the following format : TimeStamp? ::PID::Message Note: The agent process will not start if the agent user does not have rw permission on this file. This log will have entries that help determine the following : - Times when the agent was shutdown and when it was brought back up. - JVM startup flags. - Cause for an abnormal failure of the agent (if any). Abnormal failures can be: JVM Coredump, Agent process initialization failures, Out-of-Memory, Agent process killed etc. gcagent_mdu.log This log tracks the metadata updates to the Agent. The log is of the form : <date> [<thread>] - <metadata update request description> followed generally shortly thereafter by one or more of the following entries <date> [<thread>] - <metadata update result> The list of metadata update requests that are instrumented to write the mdu log are : • Save targets • Remove targets • Set collection items (SET_ITEMS, REPLACE_ALL, UPDATE_THRESHOLDS) • Set blackout • Remove blackout • Deploy metric extension • Undeploy metric extension • Attach metric extension • Detach metric extension gcagent_sdk.trc The gcagent_sdk.trc file contains log messages from each of the plugins for the various targets. It is set to DEBUG level by default. The file is located in$EMSTATE/sysman/log. gcagent_sdk.trc could be used for debugging plugin related issues.

The file contains individual log messages with the following format :
YYYY-MM-DD HH:MM:SS,### [<tid>:<thread code or code:name>] <level> -<the message>

Where :
YYYY-MM-DD HH:MM:SS,### is a timestamp (24H format and ### is the fraction in msec). <tid> is the thread id (as a decimal number) <thread name or code> is the thread full name or an abbreviated hexadecimal code (see below). <level> is the logging level that can be one of (in ascending importance order): DEBUG, INFO, WARN, ERROR, FATAL. <the message> is the free text message that is being logged. The message can contain new lines and spawn multiple lines.
emctl.log emctl.log is the entry point for emctl command line interface and provides logging for emctl commands. This file does not support tracelevels.

This log file can provide details about user CLI operations that were performed at the agent and their exit codes.

The format of the entries is :
PID :: Timestamp :: Message
emdctlj.log emdctlj.log contains logging from the agent-side implementation of emctl commands.
It provides the entry point and the exit codes of the emctl command.
Tracelevels or entries format can be controlled by clientlogging.properties file under $EMSTATE/sysman/config. All the parameters in that configuration file to control the log4j appender for emdctlj starts with "EmdctljLogger.log4j." For example : EmdctljLogger.log4j.rootCategory=DEBUG, Emdctlj secure.log This log provides additional logs for "emctl secure agent" command. This log can provide additional information to diagnose failures during securing of the Agent. emagent_perl.trc Trace file for the PERL scripts. This includes the PERL metrics and the targets discovery. Log level can be modified by updating$EMSTATE/sysman/config/emd.properties file and by setting this property : EMAGENT_PERL_TRACE_LEVEL=DEBUG
user_perl.trc This trc file contains log messages generated by PERL scripts used by jobs. These are operations that are executed as a user other than the agent user and may not have any access to the agents logging area.

By default log messages generated by the PERL logging module, provided by the agent, are suppressed for PERL scripts executed by jobs.

To enable logging the user would have to do the following:

1. Set a new property in $EMSTATE/sysman/config/emd.properties : userPerlTraceDir=<complete path to a valid directory> The value for this property can be chosen by the user. It must point to a directory that already exists and has "rwx" permission for the job users. Note: • The agent does not monitor this location and so does not clean up any files in this location. • Once debugging session is complete, this property should be removed from the agent. 2. Set the property in$EMSTATE/sysman/config/emd.properties:

EMAGENT_PERL_TRACE_LEVEL=DEBUG

The following command can be used to set the property at the agent:

emctl setproperty agent -allow_new -name userPerlTraceDir -value "/scratch"

Locating the Management Agent Log and Trace Files

The log and trace files for the Agent are written in the Agent instance directory. You can find the Agent instance directory by using this command :

$emctl getemhome The log and trace files will be located at$EMSTATE/sysman/log directory.

Example:

 [oracle@host bin]$./emctl getemhome Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. EMSTATE=/DATA/oracle/Middleware/agent/agent_inst  Locating the Management Agent Installation Log Files They can be found in several places : <AGENT_ORACLE_HOME>/cfgtoollogs Example :  .../agent/core/12.1.0.1.0/cfgtoollogs  <AGENT_INST>/install/logs Example :  .../agent/agent_inst/install/logs  12C Cloud Control: How to set the 'My Oracle Support' Preferred Credentials in 12c Cloud control? Modified 29-JAN-2013 Applies to Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - [Release: 12.1] Enterprise Manager for My Oracle Support - Version: 10.2.0.5 to 12.1.0.1.0 [Release: 10.2 to 12.1] Information in this document applies to any platform. Goal How to set the 'My Oracle Support' Preferred Credentials in 12c Cloud control? Solution Accessing My Oracle Support from Enterprise Manager Cloud Control: My Oracle Support is Oracle's customer support portal. My Oracle Support integrated with Enterprise Manager Cloud Control provides a more complete presentation of the management process for both proactive and reactive issue management. Access the following My Oracle Support Pages from Enterprise → My Oracle Support: * Service Requests * Knowledge * Certify * Community The My Oracle Support Patches & Updates page is available from Change Management > Patches & Updates. My Oracle Support Preferred Credentials: Need to set the preferred credentials to simplify access to My Oracle Support. If set, you will be automatically signed in to My Oracle Support when logged into Enterprise Manager. 1. There are two ways to set the MOS credentials: 1. Log into the 12C Cloud Control select the Setup menu located at the top-right of the page → My Oracle Support → Set Credentials 2. Log into the 12C Cloud Control select the Setup menu located at the top-right of the page → Security → Preferred Credentials From the preferred credentials page, scroll down to the section "My Oracle Support Preferred Credentials" Click on "Set MOS Credentials" 2. Fill in the fields Username and Password with your My Oracle Support account username and password. 3. Click on "Apply" you will get the following screen as confirmation: 4. To unset/remove the MOS credentails: 1. We have an option to remove the MOS credentials Which is new in 12C. 2. Click on Remove and continue to remove MOS credentials. NOTE: if you are in Offline Mode, you cannot set the MOS Credentials. You will get the following message: My Oracle Support Preferred Credentials No preferred credential is required because My Oracle Support is in offline mode.To go online, configure the connection setting. You need to click on the link "connection setting" then you need to change the connection mode from Offline to Online. You will then be able to set the MOS Credentials. Cannot Connect From 12c Cloud Control To HTTPS://Updates.Oracle.Com Modified 12-DEC-2012 Applies to Enterprise Manager for My Oracle Support - Version 12.1.0.2.0 and later Information in this document applies to any platform. Applies to Enterprise manager for Grid Control 12.1.0.1.0 Symptoms After installing 12c Cloud Control, the OMS is unable to connect to Oracle sites. 1. At minimum, the following URLs should be made available through the firewall: aru-akam.oracle.com ccr.oracle.com updates.oracle.com login.oracle.com support.oracle.com 2. Here are the results of the wget commands used to test the connection:  wget https://ccr.oracle.com --2012-03-12 08:53:42-- https://ccr.oracle.com/ Resolving ccr.oracle.com... 141.146.8.206 Connecting to ccr.oracle.com|141.146.8.206|:443... failed: Connection refused. wget https://updates.oracle.com --2012-03-12 08:54:10-- https://updates.oracle.com/ Resolving updates.oracle.com... 141.146.44.51 Connecting to updates.oracle.com|141.146.44.51|:443... failed: Connection refused. wget https://login.oracle.com --2012-03-12 08:54:36-- https://login.oracle.com/ Resolving login.oracle.com... 141.146.8.119 Connecting to login.oracle.com|141.146.8.119|:443... failed: Connection refused. wget https://support.oracle.com --2012-03-12 08:55:05-- https://support.oracle.com/ Resolving support.oracle.com... 141.146.54.16 Connecting to support.oracle.com|141.146.54.16|:443... failed: Connection refused.  Changes A network firewall is configured in this environment. Cause When Enterprise Manager Cloud Control 12c Release 1 (12.1.0.1) was released Oracle started using new URLs for connections into some backend Oracle sites which it requires for certain product features like Patching. Solution When a firewall is in place, then the following URLs need to be opened at the OMS level in order to communicate successfully to Oracle: • support.oracle.com • login.oracle.com • servicecentral.sun.com • updates.oracle.com • loginadc.oracle.com • aru-akam.oracle.com • ccr.oracle.com In addition to this, it's good practice to also make sure that the My Oracle Support Credentials have been also set, which is explained in one of our previous technical tips: "12C Cloud Control: How to set the 'My Oracle Support' Preferred Credentials in 12c Cloud control?" on page 25 12c Cloud Control Blackouts: Steps to Create Blackouts from Console UI/emctl/emcli Modified 25-FEB-2012 Applies to Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later [Release: 12.1 and later] Information in this document applies to any platform. Purpose This document describes different ways available to create blackouts in 12c Cloud Control: 1. Using Console UI. 2. Using emctl command. 3. Using emcli. Scope and Application This document is applicable to Enterprise Manager users who would like to configure blackouts in the 12c Cloud Control. 12c Cloud Control Blackouts: Steps to Create Blackouts from Console UI/emctl/emclion Using Console UI: 1. Login to 12c Cloud Control 2. Navigate to Enterprise → Monitoring → Blackouts 3. Click on "Create" to create a blackout. 4. In the page displayed, provide the following information: Step 1: 1. Name: Name of the blackout 2. Comments: comments if any (optional) 3. Reason: Reason for creating the blackout (optional) 4. Below options can be chosen depending on the requirement: • Run jobs during the blackout. Jobs that are not agent-bound will continue to run during the blackout even if this option is unselected. • Detect Service that are dependent on target that will be selected in blackout. 5. Add the required target that needs to be in blackout. If you choose a Host target, you can opt to create the blackout against all the targets in the host or against a few selected targets only. NOTE: Agent targets are not shown in the Target list because they can only be blacked out as part of a full host blackout. Step 2: Schedule 1. Enter the TimeZone in which blackout needs to be executed 2. Select the Start options(Immediately/Later) 3. Select the Duration(Indefinite/Length/Until specific time) 4. Provide the Repeating option Review the details provided and click on submit. Using emctl command The 'emctl' utility in an Agent installation can be used to create blackouts against the targets monitored by that particular Agent only. 1. To create blackout: 1. Create blackout on a particular target: 2. <AGENT_HOME>/bin> emctl start blackout <Name of the blackout> [<Target_name>[:<Target_Type>]] [-d <Duration>] 3. Create a node level blackout: 4. <AGENT_HOME>/bin> emctl start blackout <Name of the blackout> -nodeLevel [-d <Duration>] 2. To check blackout status: 3. <AGENT_HOME>/bin> emctl status blackout [<Target_name>[:<Target_Type>]] 4. To stop a blackout: 5. <AGENT_HOME>/bin> emctl stop blackout <Name of the blackout> Using EMCLI The 'emcli' utility can be executed from the OMS machine or any client machine where it has been installed and configured. To install and setup emcli on a client machine, refer to our previous technical tip: "12c Cloud Control EM CLI: How to Install and Setup 12c emcli (Enterprise Manager Command Line Interface)?" on page 7 1. To create a blackout: 1. Create blackout on specific target: 2. emcli create_blackout -name=<Name of the blackout> -add_targets=<target_name>:<target_type> -schedule=" frequency:once|interval|weekly|monthly|yearly];duration:[HH...][:mm...]" -reason="<reason for blackout>" Example:  emcli create_blackout -name=DBMaintenance -add_targets=dev:oracle_database -schedule="duration::45" -reason="maintenance"  3. Create a node level blackout: 4. emcli create_blackout -name=<Name of the blackout> -add_targets=<target_name>:host -schedule=" frequency:once|interval|weekly|monthly|yearly];duration:[HH...][:mm...]" -reason="<reason for blackout>" Example:  emcli create_blackout -name=HostMaintenance -add_targets=asingara-pc:host -schedule="duration::45" -reason="node level blackout"  2. To get list of blackouts: • Details of all blackouts: emcli get_blackouts • Details of blackout about all targets on specific host: emcli get_blackouts -hostnames=<host name> • Details of blackout on specific target: emcli get_blackouts -target=<target name>:<target type> 3. To stop blackout: 4. emcli stop_blackout -name="<name of the blackout>" [-createdby="<blackout owner>"] 5. To delete blackout: 6. emcli delete_blackout -name="name" [-createdby="<blackout owner>" (default is current user)] How to use Enterprise Manager 12c Pre-Requisites checker script? Modified 29-AUG-2012 Applies to Enterprise Manager Base Platform - Version 12.1.0.1.0 and later Information in this document applies to any platform. Purpose This document describes in detail how to use EM 12c Pre-Requistes checker scripts in following steps: 1. What is EM Prerequisite Kit 2. How to Run EM Prereq Kit 3. How to Run EM Prereq Kit with Additional Arguments 4. Viewing Prereq Check Results 5. Viewing Prereq Check Log Files Scope This EM Prereq Kit is executed automatically when Install or Upgrade wizard is invoked. This can also be executed manually to check whether all the prerequisites are met and can be used to take Corrective Actions also. Details What is EM Prerequisite Kit? • The EM Prerequisite Kit is a command line utility that runs repository-related prerequisite checks in your environment to ensure that you meet all the repository requirements for installing or upgrading an Enterprise Manager system. • The kit not only runs the prerequisite checks but also takes corrective action automatically, to the extent possible, when a prerequisite check fails. The kit also takes post requisite steps to revert the corrective actions taken and ensure that the system is back to how it was before installing or upgrading the Enterprise Manager system. NOTE: You can download the latest version of EM Prerequisite Kit from the Self Update framework as follows: 1. In Cloud Control, from the Setup menu, select Extensibility and then click Self Update. 2. On the Self Update page, download the new version of XMLs under the entity EM Deployment Prerequisite Resources Updates, if there are any available. How to Run EM Prereq Kit? The EM Prerequisite Kit is run internally by the Enterprise Manager Installation Wizard while installing or upgrading an Enterprise Manager system. However, if any of the repository prerequisite checks fail for some reason, you can manually run them by invoking this kit. Running from Software Kit If you want to run the EM Prerequisite Kit standalone from the DVD, do the following: NOTE: If you downloaded and unzipped the Enterprise Manager Cloud Control 12c software with the Bundle Patch 1 (BP1) included, you do not need to follow the steps 1 to 3 below. You can just run the emprereqkit from the unzipped location. Example: E:\distrib\oracle\12c\em12_winx64\install\requisites\bin or /u02/distrib/oracle/12c/em12_linux64 1. Create a directory Disk1 on your file system. 2. Copy <DVD-location>/install/ to Disk1/. 3. Copy <DVD-location>/stage/ to Disk1/. NOTE: Ensure that the user running the EM Prerequisite Kit has write permission to the central inventory. Now, the EM Prerequisite Kit is available in the following location of the software kit (DVD, downloaded shiphome):  Disk1/install/requisites/bin/emprereqkit  To run the EM Prerequisite Kit, do one of the following: 1. See the results without taking any corrective actions:  Disk1/install/requisites/bin/emprereqkit -executionType install -prerequisiteXMLLoc <prereq_xml_location> -connectString <connect_string> -dbUser SYS -dbPassword <db_password> -dbRole sysdba -showPrereqs Examples:$ Disk1/install/requisites/bin/emprereqkit -executionType install -prerequisiteXMLLoc Disk1/install/requisites/list -dbHost rephost.repdomain -dbPort 1521 -dbSid emrep -dbUser SYS -dbPassword Oracle12 -dbRole sysdba -showPrereqs $/u02/distrib/oracle/12c/em12_linux64/install/requisites/bin/emprereqkit -executionType install -prerequisiteXMLLoc /u02/distrib/oracle/12c/em12_linux64/install/requisites/list -dbHost rephost.repdomain -dbPort 1521 -dbSid emrep -dbUser SYS -dbPassword Oracle12 -dbRole sysdba -showPrereqs  2. With Taking Corrective Actions to meet the repository requirements:  Disk1/install/requisites/bin/emprereqkit -executionType install -prerequisiteXMLLoc <prereq_xml_location> -connectString <connect_string> -dbUser SYS -dbPassword <db_password> -dbRole sysdba -runPrerequisites -runCorrectiveActions Examples:$ Disk1/install/requisites/bin/emprereqkit -executionType install -prerequisiteXMLLoc Disk1/install/requisites/list -dbHost rephost.repdomain -dbPort 1521 -dbSid emrep -dbUser SYS -dbPassword Oracle12 -dbRole sysdba -runPrerequisites -runCorrectiveActions $/u02/distrib/oracle/12c/em12_linux64/install/requisites/bin/emprereqkit -executionType install -prerequisiteXMLLoc /u02/distrib/oracle/12c/em12_linux64/install/requisites/list -dbHost rephost.repdomain -dbPort 1521 -dbSid emrep -dbUser SYS -dbPassword Oracle12 -dbRole sysdba -runPrerequisites -runCorrectiveActions  3. Running from Installed Location After the Cloud Contol is installed, EM Preq Kit can be called from the following location: $<OMS_HOME>/install/requisites/bin/emprereqkit 
The default XML files related to the prerequisite checks, which are stored in the Disk1/install/requisites/list directory on the software kit (DVD, downloaded shiphome), are current at the time of the release of the product.
However, after the release of the product, if a new prerequisite check is introduced or if an existing prerequisite check is updated, then you must do one of the following:

1. Enable the Self Update functionality within Enterprise Manager Cloud Control so that the new or updated prerequisite XML files are automatically downloaded to the $<OMS_HOME>/install/requisites/list directory. 2. Manually download the new or updated prerequisite XML files from Oracle store to the$<OMS_HOME>/install/requisites/list directory.

Arguments for EM Prereq Kit:

Mandatory Parameters for EM Prereq Kit:

1. -executionType
2. Either "-prerequisiteXMLLoc" Or "-prerequisiteXMLRootDir"
3. Either "-connectString" Or "-dbHost <hostname> -dbPort <port> -dbSid <sid>"
4. -dbUser
6. -dbRole
7. -prereqResultLoc (If not given, current directory is taken)
8. -showPrereqs
9. Any one from ( "-runPrerequisites", "-showCorrectiveActions", "-runCorrectiveActions", "-showPostCorrectiveActions" , "-runPostCorrectiveActions" )
Optional Parameters for EM Prereq Kit:

1. -logLoc
2. -runOnlyFor
3. -responseFile
4. -contextName
5. -componentVariables
7. -stopExecOnFirstError
8. -list
9. -export
10. -purge
11. -help ( This option gives details of various parameters which can be passed to EM Prerequisite Kit)

Viewing Prereq Check Results

Every time the EM Prerequisite Kit is run, the results of the prerequisite checks run for a particular component are stored in an instance XML file. The instance XML file has the file name <component>.xml.
The results are in the same format as the information stored in the prerequisite XML files. The only difference is the new column that indicates the actual result of the prerequisite check.

Path if manual Invoked :

<prereqResultLoc>/resultXMLs/<time-stamp>
<prereqResultLoc>/resultXMLs/LATEST/ => Previous Executions

Path if invoked Automatically (by Install or Upgrade):

<MW_HOME>/.gcinstall_temp/resultXMLs/<timestamp>
<MW_HOME>/.gcinstall_temp/resultXMLs/LATEST => Previous Executions

Viewing Prereq Check Log Files

1. emprereqkit.log - Information about every step taken or action performed by the kit
2. repository.log - About the repository-related prerequisite checks that are run
3. emprereqkit.err.log - Only the error and stacktrace of the exceptions occured
4. performance.log - Information about the repository-specific performance-related prerequisite checks that are run
5. emprereqkit.output - Contains information about the status (pass or fail) of all the prerequisite checks that are run. It also contains detailed information regarding each prerequisite check.

11g Grid Control: How to Change SSL Administration Ports for both WebLogic Admin Server and the Mid (Web) Tier (Apache)

Modified 08-FEB-2013

Applies to

Enterprise Manager Base Platform - Version: 11.1.0.1 to 11.1.0.1 - Release: 11.1 to 11.1
Information in this document applies to any platform.
The contents of this note apply to Enterprise Manager release 11G and up

Goal

The goal of this document is to provide specific instructions on how to perform the following activities:

1. Change the SSL port of the WebLogic Administration Server.

NOTE: The WebLogic Administration server stores the container specific configuration details for the OMS application server (managed server) and performs all administrative functions for the domain through the console.
By default, the SSL port for the WebLogic Administration server is 7101

2. Change the SSL port for the Web Tier which is used to access the Enterprise Manager Console

NOTE: In Enterprise Manager 11, communication between a browser client and the OMS server typically has the following flow:

Browser → Web Tier (Apache) → OMS Server (WebLogic Managed Server)
These instructions are to change the SSL port for the Web Tier only (Apache).
The default SSL port for the Web Tier is 7799

Solution

1. Change the WebLogic Administration Server SSL port

• https://hostname:port/console
• Default is 7101, e.g.

 https://hostname:7101/console 
• Once in, locate the "domain structure" pane on the left column
• Expand: GCDomain → Environment → Servers and click on Servers
• Click on the server name and a list of properties will appear.

NOTE: There are two lists of tabs at the top.
From the top list, choose "Configuration."
From the bottom list, choose "General."

One of the properties in the resulting list will be SSL Listen Port.
Enter the new value (e.g. 1234) and click "Save"

See screenshot for details:

NOTE: Once you do this, your browser will likely show a "page cannot be displayed" message.
This is because you've just changed the admin server port. You must update the URL in your browser to reflect the new value for the port (e.g. https://hostname:1234/console) in order to log in again.
But, since we've changed the port already, there is no need to log in again - You should move to the next step.

Optional: Users may validate the changes have been made by viewing:

 gc_inst/user_projects/domains/GCDomain/config/config.xml 
under the grid control installation root

Search for the text EMGC_ADMINSERVER and check the port, e.g.

 <server> <name>EMGC_ADMINSERVER</name> <ssl> <name>EMGC_ADMINSERVER</name> <enabled>true</enabled> <hostname-verification-ignored>true</hostname-verification-ignored> <listen-port>1234</listen-port> </ssl> <listen-port-enabled>false</listen-port-enabled> <listen-address>www.myHost.com</listen-address> </server> 

NOTE: It is not necessary to restart the Administration server at any point during the above procedure.

It is also necessary to update the port in three additional files:

1. Under the install root (oracle base directory), locate the file

em/EMGC_OMS1/emgc.properties

Update the value for AS_HTTPS_PORT e.g.

 AS_HTTPS_PORT=1234 
2. Under the install root (oracle base directory) locate the file ./user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/startup.properties
Update the value for AdminURL e.g.

 AdminURL=https\://www.myHost.com\:1234 

3. Under the install root (oracle base directory) locate the file

./user_projects/domains/GCDomain/init-info/tokenValue.properties

Update the value for SSL_PORT e.g.

 SSL_PORT=1234 
At this point the Admin server is using the updated port. A restart of the OMS server is necessary in order to force it to begin communicating to the Admin server on the new port definition, e.g.

 cd <OMS_HOME>/bin emctl stop oms emctl start oms 

2. Change the Webtier (OHS) SSL port used to access the Enterprise Manager Grid Console
• Identify the new port number, e.g. 9999
• Execute emctl set property as follows:

 $OMS_HOME/bin/> ./emctl set property -name oracle.sysman.emSDK.svlt.EMConsoleServerHTTPSPort -value 9999  A sample run:  [oracle.myHost bin]$ ./emctl set property -name oracle.sysman.emSDK.svlt.EMConsoleServerHTTPSPort -value 9999 Oracle Enterprise Manager 11g Release 1 Grid Control Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved. SYSMAN password: Property oracle.sysman.emSDK.svlt.EMConsoleServerHTTPSPort for oms myHost:4889_Management_Service has been set to value 9999 
• Stop the OMS by issuing 'emctl stop oms -all', e.g.

 cd <OMS_HOME>/bin emctl stop oms -all 

• Take a backup of following files in :

gc_inst/WebTierIH1/config/OHS/ohs1/ssl.conf
gc_inst/WebTierIH1/config/OHS/ohs1/ssl.conf/em/EMGC_OMS1/emgc.properties

• Edit the above mentioned and change the reference of port 7799 to 9999.
• Finally, start the OMS.

 cd <OMS_HOME>/bin emctl start oms 

12c Cloud Control: How to Modify the Password for SYSMAN and other Enterprise Manager Users at the OMS Level and Repository Database?

Modified 07-SEP-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

This article provides steps for modifying the password of SYSMAN and other Enterprise Manager users at the OMS/WLS level and the Repository database in a Cloud Control setup.
Simply changing the password in the repository database is not sufficient as the password is also stored in the WLS credential store.
The OMS uses sysman account to login into the repository database and if there is a mis-match in the password at the database level and the OMS configuration, the OMS cannot start and function properly.

NOTE: From 12c onwards, directly modifying the password for sysman or any other repository user at the Repository Database is not supported.
If the password is manually changed at the database, then it cannot be modified at the OMS level using 'emctl config oms'. This will render the Cloud Control setup unusable.
Hence, ensure that the passwords are changed only using one of the below listed methods.

If the current SYSMAN password is known

1. Stop all the OMS:

 cd <OMS_HOME>/bin emctl stop oms 
Execute the same command on all the OMS machines including the primary OMS machine. Do not include '-all' as the Admin Server needs to be up during this operation.

 cd <OMS_HOME>/bin emctl config oms -change_repos_pwd 

NOTE:

- The above command will prompt you for the current password of the SYSMAN user and the new password.
- The password will be modified at the Repository Database as well as the WLS Credential store and the monitoring credentials for the 'OMS and Repository' target.
- Along with the SYSMAN password, this command will modify the password for the ALL the EM users (SYSMAN_MDS, BIP, SYSMAN_OPSS, SYSMAN_APM, SYSMAN_RO, MGMT_VIEW) created in the Repository Database.

Example output:

 emctl config oms -change_repos_pwd Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. Enter Repository User's Current Password : Enter Repository User's New Password : Changing passwords in backend ... Passwords changed in backend successfully. Updating repository password in Credential Store... Successfully updated Repository password in Credential Store. Restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'. Successfully changed repository password. 

3. Stop the Admin server on the primary OMS machine and re-start all the OMS:

 cd <OMS_HOME>/bin emctl stop oms -all emctl start oms 

If the current SYSMAN password is unknown but the password has not been changed manually at the Database

1. Stop all the OMS:

 cd <OMS_HOME>/bin emctl stop oms 
Execute the same command on all the OMS machines including the primary OMS machine. Do not include '-all' as the Admin Server needs to be up during this operation.

 cd <OMS_HOME>/bin emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd <sys user password> -new_pwd <new sysman password> 

NOTE:

- The '-use_sys_pwd' is used to connect to the database as a SYS user and modify the sysman password in the Repository database.
- The current sysman password is not prompted for and only the new password needs to be entered. This will allow the reset of the old password to the new password entered.
- The password will be modified at the Repository Database as well as the WLS Credential store and the monitoring credentials for the 'OMS and Repository' target.
- Along with the SYSMAN password, this command will modify the password for the ALL the EM users (SYSMAN_MDS, BIP, SYSMAN_OPSS, SYSMAN_APM, SYSMAN_RO, MGMT_VIEW) created in the Repository Database.
- This command will work only if the sysman or any other repository user password has not been manually modified at the repository database.

 Example output: emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd oracle123 -new_pwd oracle12 Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. Changing passwords in backend ... Passwords changed in backend successfully. Updating repository password in Credential Store... Successfully updated Repository password in Credential Store. Restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'. Successfully changed repository password. 

3. Stop the Admin server on the primary OMS machine and re-start all the OMS:

 cd <OMS_HOME>/bin emctl stop oms -all emctl start oms 

Modifying the password for SYSMAN and other EM users on a Standby OMS setup

The steps below are applicable only if Standby OMS has seperate Admin Server than the Primary OMS, but the same Repository database.

1. Ensure that the password is changed on the Primary OMS setup as per the above steps
2. Stop the OMS on Standby site , if already running

 <OMS_HOME>/bin>emctl stop oms -all 

3. Start the Admin server only

 <OMS_HOME>/bin>emctl start oms -admin_only 

4. Logged into Admin server console as user 'weblogic'.

• Navigate to Services → Data Sources → sysman-opss-ds → Connection Pool
• Save and Activate the changes

• Now run the command below from Standby OMS HOME

 cd <OMS_HOME>/bin emctl config oms -change_repos_pwd 
The above command will prompt you for the current password of the SYSMAN user and the new password.Provide the same value for both

• Run the commands below to start the Standby OMS

 <OMS_HOME>/bin>emctl stop oms -all <OMS_HOME>/bin>emctl start oms <OMS_HOME>/bin>emctl status oms -details 

Known Issues

Problem: Modifying the sysman password fails with "java.io.IOException":

 cd <OMS_HOME>/bin emctl config oms -change_repos_pwd Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. Enter Repository User's Current Password : Enter Repository User's New Password : java.io.IOException Error occurred. Check the log 
Cause: The above error occurs if the Admin server is not running when the sysman password is being modified.

Solution: As advised in the sections above, the Admin Server needs to be up and running when the sysman password is being modified.
So, the OMS should not be stopped using 'emctl stop oms -all'.

 cd <OMS_HOME>/bin emctl start oms emctl stop oms emctl config oms -change_repos_pwd 

11 Grid Control: Steps for Modifying the HTTP and HTTPS Upload Ports After Installation

Modified 08-FEB-2013

Applies to

Enterprise Manager Base Platform - Version 11.1.0.1 to 11.1.0.1 [Release 11.1]
Information in this document applies to any platform.

Goal

In 11.1.0.1 Enterprise Manager Grid Control, the management agent uses port 4900 (HTTPS) and port 4889 (HTTP) by default to upload the *.xml files to the Oracle Management Server. Sometimes users may require to change these ports due to firewall policies and restrictions.
This article explains how to change the HTTP/HTTPS upload ports configured on the OMS in 11.1.0.1 Grid Control. Note that after these ports have been changed on the OMS, it will be necessary to update the configuration of all the agents which upload to this OMS.

In this example, the original configuration is as follows:-

 $/emctl status oms -details Oracle Enterprise Manager 11g Release 1 Grid Control Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Console Server Host :<OMS-hostname> HTTP Console Port : 7800 HTTPS Console Port : 7801 HTTP Upload Port : 4889 HTTPS Upload Port : 4900 OMS is not configured with SLB or virtual hostname Agent Upload is locked. OMS Console is locked. Active CA ID: 1  In this example, the original http upload port is 4889 and will be changed to 4890 In this example, the original https upload port is 4900 and will be changed to 1159 In this example the http console port (7800) will not be changed In this example the https console port (7801) will not be changed IMPORTANT NOTE: in this article the ports which are used to access the console ( https://omshostname.domain:7801/em in this example) will NOT be changed. See also our technical tip: "11g Grid Control: How to Change SSL Administration Ports for both WebLogic Admin Server and the Mid (Web) Tier (Apache)" on page 27 for further details to change the ports used to access the console. Fix 1. Stop the OMS $./emctl stop oms 

2. Use emctl set property command to set the following properties:

 oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort oracle.sysman.emSDK.svlt.ConsoleServerPort 
Example:

 ./emctl set property -name oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort -value 1159 Oracle Enterprise Manager 11g Release 1 Grid Control Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved. SYSMAN password: Property oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort for oms <OMS-hostname>:4889_Management_Service has been set to value 1159 $./emctl set property -name oracle.sysman.emSDK.svlt.ConsoleServerPort -value 4890 Oracle Enterprise Manager 11g Release 1 Grid Control Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved. SYSMAN password: Property oracle.sysman.emSDK.svlt.ConsoleServerPort for oms <OMS-hostname>:4889_Management_Service has been set to value 4890  IMPORTANT: Note that the "ConsoleServerPort" and "ConsleServerHTTPSPort" properties are NOT used for changing the port used to access the Grid Control Console on. There are different properties EMConsoleServerHTTPSPort and EMConsoleServerPort to do this. See our technical tip: "How to Change SSL Administration Ports for both WebLogic Admin Server and the Mid (Web) Tier (Apache)" on page 27 for further details. 3. Backup and Edit <INSTANCE_HOME>/emgc.properties.  EM_UPLOAD_HTTP_PORT=4889 EM_UPLOAD_HTTPS_PORT=4900 To EM_UPLOAD_HTTP_PORT=4890 EM_UPLOAD_HTTPS_PORT=1159  Above step changes HTTP upload port to 4890 from 4889. HTTPS upload port to 1159 from 4900. 4. Update the "Listen" and "VirtualHost" directives in <INSTANCE_HOME>/WebTierIH1/config/OHS/ohs1/httpd_em.conf, replacing the previous port number with the new port number.  <IfDefine SSL> Listen 4900 <VirtualHost *:4900> .... .... Listen 4889 <VirtualHost *:4889> to <IfDefine SSL> Listen 1159 <VirtualHost *:1159> .... .... Listen 4890 <VirtualHost *:4890>  5. Start the OMS and verify the new upload ports. $./emctl start oms Oracle Enterprise Manager 11g Release 1 Grid Control Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved. Starting WebTier... WebTier Successfully Started Starting Oracle Management Server... Oracle Management Server Successfully Started Oracle Management Server is Up $/emctl status oms -details Oracle Enterprise Manager 11g Release 1 Grid Control Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Console Server Host :<OMS-hostname> HTTP Console Port : 7800 HTTPS Console Port : 7801 HTTP Upload Port : 4890 HTTPS Upload Port : 1159 OMS is not configured with SLB or virtual hostname Agent Upload is locked. OMS Console is locked. Active CA ID: 1  6. Because the upload ports have been changed at OMS side, the agents emd.properties file also needs to be updated with the modified HTTP/S upload ports. This step needs to be carried out for all agents which upload to this OMS. • On the agent machine, backup and edit the following parameter in AGENT_HOME/sysman/config/emd.properties file  REPOSITORY_URL=https://<OMS-HOSTNAME>:4900/em/upload emdWalletSrcUrl=https://<OMS-HOSTNAME>:4900/em/wallets/emd to REPOSITORY_URL=https://<OMS-HOSTNAME>:1159/em/upload emdWalletSrcUrl=https://<OMS-HOSTNAME>:1159/em/wallets/emd  • Stop and Restart the agent $<AGENT_HOME>/bin/emctl stop agent $<AGENT_HOME>/bin/emctl start agent$<AGENT_HOME>/bin/emctl upload agent (to confirm that the upload is successful) 

How To Delete Multiple Incidents In Enterprise Manager Cloud Control 12c

Modified 08-OCT-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

How to delete multiple incidents in Enterprise Manager Cloud Control 12c ?

Is there a way to clear multiple incidents rather than clear each incident individually?

 For Example: You have 222 incidents opened in 12c Cloud Control console related to invalid logins which need to be cleared. 

Fix

You can clear multiple incidents via the command line with the command 'emcli clear_stateless_alerts'.
Clears the stateless alerts associated with the specified target.

Only the user can clear these stateless alerts; the Enterprise Manager Agent does not automatically clear these alerts.

To find the metric internal name associated with a stateless alert, use the get_metrics_for_stateless_alerts verb.

NOTE: Before running any emcli command, you need to log in to emcli first if not already done.
It may also be needed to synchronize the emcli and the OMS.

Example:

 $emcli login -username=SYSMAN$ emcli sync 

Format (One line)

 $emcli clear_stateless_alerts -older_than=number_in_days -target_type=target_type -target_name=target_name [-include_members] [-metric_internal_name=target_type_metric:metric_name:metric_column] [-unacknowledged_only] [-ignore_notifications] [-preview] [ ] indicates that the parameter is optional.  Options • older_than Specify the age of the alert in days. (Specify 0 for currently open stateless alerts.) • target_type Internal target type identifier, such as host, oracle_database, and emrep. To get a list of valid target types in your repository database, you may do the following: 1. Run SQL Plus and log in as sysman user to your repository database 2. Run the following query:  SQL> select distinct target_type from mgmt_targets;  Sample output:  SQL> select distinct target_type from mgmt_targets; TARGET_TYPE ---------------------------------------------------------------- oracle_listener oracle_ias_farm weblogic_j2eeserver j2ee_application metadata_repository oracle_csa_collector host oracle_database oracle_emd oracle_emrep weblogic_domain  • target_name Name of the target. • include_members Applicable for composite targets to examine alerts belonging to members as well. • metric_internal_name Metric to be cleaned up. Use the get_metrics_for_stateless_alerts verb to see a complete list of supported metrics for a given target type. Example for the target type oracle_database $ emcli get_metrics_for_stateless_alerts -target_type=oracle_database 

• unacknowledged_only

Only clear alerts if they are not acknowledged.

• preview

Shows the number of alerts to be cleared on the target(s).

Examples

The following example clears alerts generated from the database alert log over a week old. In this example, no notifications are sent when the alerts are cleared.

 $emcli clear_stateless_alerts -older_than=7 -target_type=oracle_database -tar get_name=database -metric_internal_name=oracle_database:alertLog:genericErrStack -ignore_notifications  Example for a cluster database If the target type for cluster database is "rac_database". , execute the following command $ ./oms/bin/emcli clear_stateless_alerts -older_than=1 -target_type=rac_database -target_name=oracle.rac.ccc.kk 

How to Connect to the Weblogic Server Administration Console which ships with Enterprise Manager 11g Grid Control and Enterprise Manager 12c Cloud Control

Modified 09-NOV-2012

Applies to

Enterprise Manager Base Platform - Version 11.1.0.1 to 12.1.0.1.0 [Release 11.1 to 12.1]
Information in this document applies to any platform.

Goal

How to Connect to the Weblogic Server Administration Console (admin server console) which ships with Enterprise Manager 11g and Enterprise Manager 12c.

Fix

The weblogic server administration console should be accessible on the same hostname as the OMS, but using a different port.

By default the port is 7101

The full URL to use is:-

https://hostname.domain:<port>/console

https://machine1.uk.oracle.com:7799/em (Cloud/Grid Control URL)

sysman/oracle1

Then:-

How to find out which port/hostname the admin server console is using, if it is not using the default?

1. Check the <OMS_HOME>/install/setupinfo.txt file and check for "Admin Server Port"

 eg./oracle/12cloud/oms/install or /u01/app/Middleware/oms11g/install 

Use the following URL to access:

1. Enterprise Manager Cloud Control URL: https://machine1.uk.oracle.com:7799/em

The following details need to be provided during the additional OMS install:

or

2. Go to all targets/choose "EMGC_ADMINSERVER" in Enterprise Manager. On the right hand side under "tools"(12c) or "summary"(11g) it says "To configure and manage this web logic server use the "Weblogic Server Administration Console". This is a hyperlink to the admin console.

or

3. In Enterprise Manager 12c:-
Click on all targets/Middleware

Expand "EMGC_GCDomain" expand "GCDomain" highlight "EMGC_ADMINSERVER"

Right click , choose "target setup"/"monitoring configuration"

View "SSL Listen Port". There is also a hyperlink to the admin console.

4. In Enterprise Manager 11g:-
• targets/Middleware
• expand secFarm_GCDomain, expand secFarm_GCDomain/GC_Domain
• right click , choose "target setup"/"monitoring configuration"
• view "SSL Listen Port"

or (if it is not possible to access the EMGC_ADMINSERVER target page)

5. Go to setup/agents. Choose the agent which monitors the OMS. Scroll down to "monitored targets". Highlight the target EMGC_ADMINSERVER. Click on "configure" which will show:

 SSL Listen Port <port value> 
or (if Enterprise Manager cannot be started and so nothing can be checked in the console)

6. Check:- ..gc_inst/user_projects/domains/GCDomain/config/config.xml
/oracle/12cloud/gc_inst/user_projects/domains/GCDomain/config

 <listen-port>7101</listen-port> </ssl> <log> <name>EMGC_ADMINSERVER</name> 

EM 12c: Agent Install Fails at pre-requisite check wth Error message "SSH daemon (sshd) is not running on port 22"

Modified 04-SEP-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

Enterprise Manager 12c Agent Install Fails at pre-requisite step with Error message "SSH daemon (sshd) is not running on port "22"

+ oms/sysman/prov/agentpush/<TIMESTAMP>/applogs/<hostname>_deploy.log

 2012-06-13_13-46-32:INFO: Jsch Valdation Failed Problem :SSH server check failed Recommendation: Verify the value of SSH_PORT in the /oms/oui/prov/resources/Paths.properties file. Ensure that it is the same as the port on which the sshd is running on the remote host. 2012-06-13_13-46-32:INFO:Updating Action SSHValidationswith Status FAILED and error Message :SSH daemon (sshd) is not running on port "22" and problem SSH server check failed and recommendation Verify the value of SSH_PORT in the /var/app/oracle/em/oms/oui/prov/resources/Paths.properties file. Ensure that it is the same as the port on which the sshd is running on the remote host. 

Cause

This error can occur due to either of the below reasons:

• SSH port 22 is blocked by proxy / firewall on the Target machine
• SSH is not running on the Target machine
• SSH daemon is running on non-default port (other than 22)

Solution

This error can occur due to either of the below reasons:

1. Verify that the SSH port 22 is not blocked from the OMS machine using

 telnet <target host> 22 

2. Check SSH is running on the target machine using

 ps -ef | grep ssh netstat -anp | grep 22 
Prerequisites for 12c Agent installation clearly mentions that the SSH daemon should be running on the default port (that is, 22) on all the destination hosts.

3. If SSH on target machine is running on a non-default port, then update the SSH_PORT property in $<OMS_HOME>/oui/prov/resources/Paths.properties as below:  SSH_PORT=<port number>  • If you are installing on target A where SSH is running on port 99; then you would need to make the above change. If the next Agent Deployment is on target B where SSH is running on default port 22; then you will need to revert the changes to set SSH_PORT= or SSH_PORT=22 in$<OMS_HOME>/oui/prov/resources/Paths.properties file.
• You cannot install Agents on target X and target Y in one-go, if they have SSH running on different ports.

NOTE: This error is also commonly encountered on Windows target host when Cygwin is not installed.

12c Cloud Control: How to Perform Control Operations on Single/Multiple Agents From Console UI?

Modified 16-FEB-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Goal

This document lists the steps for performing Control Operations (Stop, Start, Restart, Secure, Unsecure, Resecure) against a Single/Multiple Agents from the 12c Console UI.

Solution

Controlling a Single Agent

Control operations for a single Agent can be performed on the Agent home page for that Agent.

1. Navigate to Setup → Agents and click on the desired Agent name.
2. From the Agent drop down menu, choose Control and then one of the control operations (Start Up/Shut Down, or Restart)

3. Upon choosing any of the above control menu options, a pop-up dialog requesting the credentials of the user displays. These operations require the credentials of the OS user who owns the Agent. At this point, you can either choose from a previously stored username/password, preferred or named credential. You also have the option of choosing a new set of credentials which could also be saved as the preferred credential or as a named credential for future use. When storing a named credential or creating a new set of credentials, a Test button is shown, thus allowing you to verify if the credential information you just entered is valid. Once you are authenticated, the chosen control operation begins and continues even if the pop-up dialog is closed. Any message of failure/success of the task is displayed in the pop-up dialog.
When choosing the Secure/Resecure/Unsecure options, you must provide the necessary Registration Password.

NOTE: The EM user must have at least operator privilege on the Agent target in order to perform Agent control operations.

Controlling Multiple Agents

In order to perform control operations on multiple Agents, Enterprise Manager makes use of the Job system to automate repetitive tasks.
Therefore, you must have Job privileges for controlling multiple Agents through a single action. To access

1. From the Setup menu, choose Agents. The Management Agent setup page displays.
2. Select multiple Agents from the list.

3. Click one of the control operation buttons (Start Up/Shut Down/Restart/Secure/Resecure/Unsecure). When you click on any of the control operations, you are taken to the Job creation wizard where you schedule a new job to perform the action on the selected Agents. Below screenshot shows 'Restart operation'

4. In the Jobs page, you can view the chosen Agents in Target section in the General tab. You can add more Agents by clicking the Add button. You then provide the parameters for the operation in the Parameters tab, if needed. The credentials must be specified in the Credentials tab where you can either choose from a previously stored username/password, preferred, or named credential. You also have the option of choosing a new set of credentials which could also be saved as the preferred credential or as a named credential for future use. You are given the option to start the job immediately or schedule the job for a later time. At this point, you can also create a repeating job by specifying the job start time, the frequency, and the end time.
The Access tab displays the Administrator details and the access levels they have to the job. You can then add a new administrator or modify the access level to View or Full, if you have the necessary privileges.

5. Click on ''Submit", it will create a job as shown in the below screenshot:

6. Click on the job to view the status and click on "Log Report" to view the results of the job:

How to Setup the Patch Plan Prerequisites in Enterprise Manager Base Platform 12c

Modified 12-MAR-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Purpose

The Patch-Plan feature

Starting with Enterprise Manager Cloud Control 12c, the application of one-off patches for the 12c Management Agents will now be done via the Patch Plan feature.

One of the main benefits of applying patches this way, is that all the pre-checks, patch validation, and ORACLE_HOME validation will be performed prior to incurring target downtime.
Any roadblocks will be reported and can be fixed before attempting to install the patch.

The prerequisites for this feature are:

1. A Software Library that is configured and operational
2. The latest version of Opatch must be staged in the Software Library

How to Setup the Patch Plan Prerequisites in Enterprise Manager Base Platform 12c

Instructions for Setting Up the Software Library:

• Log in to the EM Cloud Control Console as a superuser
• Navigate to Setup → Provisioning and Patching → Software Library page
• If there is nothing configured yet, 'Add' an 'OMS Shared Filesystem' location. This is a location on a shared file system, that is 'visible' on all the OMS's

• When the location gets added, a background job will start which populates the software library with the components needed for Provisioning and Cloud Management features.

To Validate the Content of the Software Library:

• Navigate to the Enterprise → Patching and Provisioning → Saved Patches page

• Make sure that the Patch 6880880 is available for every platform that is running an Agent (Linux, Solaris, HP, AIX, ....). The minimal version required to perform Agent patching is 11.1.0.9.4
• If this patch is not available, you will need to download it from My Oracle Support or upload it manually (Offline mode) to the Software Library.

Setup in Online mode (with My Oracle Support Connectivity)

• If the Patch 6880880 is not showing up in the output, this means the 'Opatch Update' job hasn't executed yet), schedule a one-time 'Opatch Update' job:
• From the 'Create Job' drop down list, select 'Opatch Update', and click 'Go'
• Enter a name (the name of this job), and a description (optional)
• On the 'Schedule' tab, make sure the 'Type' is set to 'One Time (Immediately)')
• Click submit
• Once the job finishes, go back to the 'Saved Patches' page and verify whether the Patch 6880880 is available now and with a version equal or higher to 11.1.0.9.4:

• A repeating job will be submitted in Enterprise Manager, to make daily checks for Opatch updates. To see if this job is still running:
• Go to the Enterprise → Job → Job Activity page
• Click on 'Advanced Search', select the 'Opatch Update' Job Type, choose the 'Targetless' option from the Target Type drop-down list and click 'Go'
• There should be one (and only one) repeating job be scheduled in the output, to make that daily check
• To check the repository directly to see if the job is there, use this SQL:

•  SELECT * FROM mgmt$jobs WHERE job_type = 'OpatchPatchUpdate_PA' AND interval = 1 AND is_library = 0 AND start_time IS NOT NULL AND end_time IS NULL;  • There should also be daily jobs (and again only 1 per type) of the types 'CVUPackDownload', 'Refresh Updates' and 'Refresh From My Oracle Support' scheduled Setup in Offline Mode (without My Oracle Support Connectivity) • If there is no connectivity to My Oracle Support, the patches will have to get manually uploaded to the Software Library. • To upload the patch: • Log in to https://support.oracle.com from a machine/laptop • Click on the 'Patches & Updates' tab • In the 'Patch Search' region, enter the Opatch patch number (6880880) • From the platform drop-down list, select the OS/Platform needed for the OPatch update, and click 'Search' • When the list appears with the found patches, select the one for the '11.1.0.0.0' release (This is the version the Cloud Control 12c Agents are using) • Hover over the 'Uploaded' column of that selected row, and write down the release date of the patch (Dec-03 2011 for the 11.1.0.9.4 release) • Click on the 'Readme' button from the selection option bar, to get the exact version number of the OPatch update that will get downloaded Click on the 'Download' button from the selection option bar. A pop up will appear with the option to download the patch At the bottom of the pop up window, first select the 'Download Patch Metadata' button, and download that small XML file first After that has been downloaded, click on the 'Download' button again, and download the main patch ZIP file • Now that all the necessary pieces are locally available, go back to the Enterprise Manager console, and navigate to the Enterprise → Patching and Provisioning → Saved Patches page • Use 'Upload' to upload the Opatch patch to the software library • On the top of the page, select Product Family ('Oracle System Management Products') and Product 'Universal Installer', with the type of 'Patch' • Use the 'Upload from local host' radio button, to be able to specify the downloaded My Oracle Support files using the browser. • Using the browse buttons for the first and the second column, select the Patch Metadata XML file for the 1st column, and the ZIP file for the 2nd column • In the optional column 'Patch Number' enter the bug number for the Opatch release 6880880 • The Description column is optional - A user-friendly description can be put here for this. • If you want to make this consistent with the online-mode, enter 'Opatch patch of version for Oracle software releases 11.1.0.x ()' • The Created On column is the release date of the patch - This the date we got from the Patch selection from My Oracle Support (Dec-03 2011 for the 11.1.0.9.4 release) • From the Platform drop-down list, specify the version of the patch that you downloaded • The language can be kept to 'American English' • And finally, click the 'Upload' button on the bottom of the page Repeat these steps for every platform where the Agents are running (Linux, Solaris, HP, AIX, etc). Use of the Enterprise Manager Cloud Control 12.1.0.1 .../oms/bin/emctl dump Options to Collect OMS Log Files Modified 24-AUG-2012 Applies to Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.1.0 [Release 12.1] Information in this document applies to any platform. Purpose Listed here are various options available with the emctl dump command for compiling log files into a .zip file for upload to Oracle. Scope Log file uploads are useful when diagnosing OMS restart, crash or performance issues. Details Command emctl dump oms [-heap] [-include_top -include_netstat -s <num of samples> -n <interval> -destdir <destination directory>] <OMS_HOME>/bin/emctl dump oms  This command will capture the standard assortment of logs: node manager logs admin server logs managed server logs oms sysman logs stack trace files Sample output: [oracle@vboddeti-pc bin]$ ./emctl dump oms Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.0.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. dump has been generated at /tmp/oms_dump_11.10.23.02:20:12.zip 
How it works with each option

1. <OMS_HOME>/bin/emctl dump oms -heap

 This will provide heap dump files in addition to the standard dump file collection: node manager logs admin server logs managed server logs oms sysman logs stack trace files heap dump files 

2. <OMS_HOME>/bin/emctl dump oms -heap -include_top

 This will provide a top command output file in addition to the output from #1: node manager logs admin server logs managed server logs oms sysman logs stack trace files heap dump files top command output files 

3. <OMS_HOME>/bin/emctl dump oms -heap -include_top -include_netstat

 This will provide a netstat output file in addition to the output of #2: node manager logs admin server logs managed server logs oms sysman logs stack trace files heap dump files top command output files netstat output files 

4. <OMS_HOME>/bin/emctl dump oms -heap -include_top -include_netstat -s <num of samples> -n < interval>

 This will provide the same output as #3: node manager logs admin server logs managed server logs oms sysman logs stack trace files heap dump files top command output files netstat output files except that the number of netstat and top files generated will be limited by the <num of samples> associated with -s option. For example: emctl dump oms -heap -include_top -include_netstat -s2 In this case, 2 samples of top and netstat files will be generated. 

5. <OMS_HOME>/bin/emctl dump oms -heap -include_top -include_netstat -s <num of samples> -n <interval>

 This will provide the same output as #4: node manager logs admin server logs managed server logs oms sysman logs stack trace files heap dump files top command output files netstat output files but the timeperiod of the file generation for netstat and top will be limited to a specific time interval with the -n option. For example: emctl dump oms -heap -include_top -include_netstat -s2 -n15 In this case, 2 samples of top and netstat files will be generated with 15 seconds time gap. 

By default, all these files comes in a compressed .zip file, and this file will be created in the /tmp directory.
The -destdir option will allow selection of an alternate destination directory.

For example:
emctl dump oms -heap -include_top -include_netstat -s2 -n15 -destdir /home/logs

In this case, the dump .zip file will be generated in /home/logs directory.

NOTE: If the OMS is not running, using the emctl dump command will only generate node manager logs, admin server logs, managed server logs and OMS sysman logs. The heap, top and netstat options will not provide useful information when the OMS is not operating..

12c Cloud Control Security: How to Create/Edit the Agent Registration Password for Agent to OMS Secure Communication

Modified 16-FEB-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Goal

Enterprise Manager uses the Agent Registration password to validate that installations of Oracle Management Agents are authorized to load their data into the Oracle Management Service.
The Agent Registration password is created during installation when security is enabled for the Oracle Management Service.
This document provides the steps for managing (add/edit/delete) the Agent registration password.

From 12c Cloud Control UI

1. Login to 12C Cloud Control as sysman or any other super-administrator.
2. Navigate to Setup → Security → Registration Passwords

When you create or edit an Agent Registration Password, the following need to be chosen:
• Persistent or One-time: 'Persistent' can be used for multiple agents. 'One-time' will be deleted once an agent uses it for registration.
• Expiry date for the password.

From Command Line using emctl

This option can be used to create a new Agent registration password:

cd <OMS_HOME>/bin
emctl secure setpwd [sysman pwd] [new registration pwd]

 For example: emctl secure setpwd Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root Password : <Please enter the sysman password > Enter New Agent Registration Password : <Please enter a new agent registration password> Registration Password added successfully. 

NOTE: If you change the Agent Registration Password, you must communicate the new password to other Enterprise Manager administrators who need to install new Management Agents, enable Enterprise Manager Framework Security for existing Management Agents, or install additional Management Services.

• The above command will prompt for the sysman user's password, if not provided at the command line.
• Creating the registration password from the command line does not mandate an expiry date to be chosen.
• The password is 'Persistent' with 'No Expire Date'.
• There is no option to edit / delete the registration password from the command line, use the Cloud Console UI for these activities.

11g Grid Control: Impact of Setting Debug Mode for OMS and Steps to Enable Debug for Particular Subsystems

Modified 14-SEP-2012

Applies to

Enterprise Manager Grid Control - Version: 11.1.0.1 and later [Release: 11.1 and later]
Information in this document applies to any platform.

Purpose

The purpose of this note is to explain what is the impact of turning on DEBUG level on the OMS side with the command:

emctl set property -name log4j.rootCategory -module LOGGING -value "DEBUG, emlogAppender, emtrcAppender"

The note also explains how to turn on debugging for individual OMS subsystems, that are being investigated, rather than the entire OMS.

Scope and Application

The intended audience for this document are Grid Control Administrators.

11g Grid Control: Impact of Setting Debug Mode for OMS and Steps to Enable Debug for Particular Subsystems

Setting DEBUG on the OMS side is sometimes needed to troubleshoot some issues. Given that this generate more data in the log files, this may have some significant impact on the OMS performance. One could see in particular peaks in the Disk I/O metrics.
Furthermore, generating more logging information may may result in faster rollover of the debug logs and the useful entries may be overwritten.

Given that DEBUG level can be turned on and off dynamically in 11GC without bouncing the OMS, Oracle recommends to turn on DEBUG only to capture debug traces for a particular issue, and let the OMS running in regular WARN mode otherwise.
In addition, we can enable the debug mode only for selective components, based on the issue being investigated.
Suppose we have to set debug mode for OMS receiver, we need to run this command:

 cd <OMS_HOME>/bin emctl set property -name log4j.category.oracle.sysman.emdrep.receiver -value "DEBUG" -module logging 
The following specific debug levels are a subset of the ones available for Grid Control 11.1:

OMS Subsystem Tracing Parameter Description
Jobs 1. log4j.category.oracle.sysman.emdrep.jobs
2. log4j.category.oracle.sysman.em.jobs
3. log4j.category.oracle.sysman.eml.jobs
4. log4j.category.oracle.sysman.emSDK.comm
1. The job system engine (Dispatcher and Job Worker).
2. Job System Commands and the Job Receiver
3. Job System Console (UI).
4. OMS-agent communication (this is essential for debugging agent-bound operations)
Reports log4j.category.oracle.sysman.eml.ip -
Templates log4j.category.oracle.sysman.eml.metrics.template
log4j.category.oracle.sysman.eml.mntr.metrics.UserDefinedMetricLauncherDataObject
log4j.category.oracle.sysman.eml.mntr.metrics.UserDefinedMetricsDataObject
log4j.category.oracle.sysman.eml.mntr.metrics.UserDefinedMetricsDeleteDataObject
log4j.category.oracle.sysman.eml.mntr.metrics.UserDefinedMetricsTable
-
EM Servlet log4j.category.em.console -
Servlet Filters/Listeners log4j.category.oracle.sysman.eml.app For debugging Context Filters, Session Listeners, etc

12c Cloud Control : How To View and Update OMS Configuration Properties Using 'emctl' utility?

Modified 14-FEB-2012

Goal

In the 12c Cloud Control, the OMS/Repository configuration and logging properties are now stored in the Repository Database itself. This document explains how the 'emctl' utility can be used to view and update these properties.

Solution

Useful Locations / Environment Variables:

Refer also to our technical tip:

"12c Cloud Control: Details of the Directory Structure and Commonly Used Locations in a 12c OMS Installation" on page 19

The 'emctl' utility found in the <OMS_HOME> is used for viewing / updating the OMS configuration properties.
The command will prompt for the sysman password, if it is not entered as an argument with the emctl options.

To View / List Properties

1. To view all the OMS properties:

cd <OMS_HOME>/bin
emctl list properties [-sysman_pwd "sysman password"] [-module <emoms | logging>]

The -module option allows a further break-up of the properties depending on whether they are related to logging or only configuration. If this option is not mentioned, then all the properties will be displayed.

For example:

 $emctl list properties Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. SYSMAN password: LargeRepository=false conn_leak_allowance_time=45 em.ip.ui.enable=true em.loader.maxRecvThreads=20 em.loader.perfUpdatePeriod=60000 em.oms.dumpModules=omsThread,repos em.security.xsrf_check_enabled=false emConfigMBeanPort=0 ..etc  2. To view a specific OMS property: cd <OMS_HOME>/bin emctl get property [-sysman_pwd "sysman password"] -name <property name> To execute this command, the exact property_name should be known. This can be picked up from the output of the 'emctl list properties' command. For example: $ emctl get property -name oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. SYSMAN password: Value for property oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort for oms omsmachine.domain:4890_Management_Service is 4901 

To Set / Update Properties

1. To set / update a specific OMS property:

cd <OMS_HOME>/bin
emctl set property [-sysman_pwd "sysman password"] -name <property name> -value <property value> [-module <emoms | logging>]

For example:

 $emctl set property -name log4j.rootCategory -value 'DEBUG, emlogAppender, emtrcAppender' -module logging Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. SYSMAN password: Property log4j.rootCategory for oms omsmachine.domain:4890_Management_Service has been set to value DEBUG, emlogAppender, emtrcAppender  You can use the 'emctl list properties' or 'emctl get property' to verify that the change is successful. 2. To set / update multiple OMS properties using a file: cd <OMS_HOME>/bin emctl set property [-sysman_pwd "sysman password"] -file <absolute path of file containing properties> [-module <emoms | logging>] The file should contain the property name and value in this format: log4j.rootCategory=WARN, emlogAppender, emtrcAppender log4j.appender.emtrcAppender.MaxBackupIndex=12 For example: $ emctl set property -file omsprop.txt Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. SYSMAN password: Properties in file omsprop.txt has been updated for oms omsmachine.domain:4890_Management_Service 

You can use the 'emctl list properties' or 'emctl get property' to verify that the change has occurred.

To Delete Properties

To delete a specific OMS property:

cd <OMS_HOME>/bin
emctl delete property [-sysman_pwd "sysman password"] -name <property name>

NOTE: This command should be used with utmost caution as deleting necessary OMS properties may affect the operations.

To View / Update Repository Connection Details

1. To view the connection details to the Grid Control Repository Database:

cd <OMS_HOME>/bin
emctl config oms -list_repos_details

For example:

 $emctl config oms -list_repos_details Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. Repository Connect Descriptor : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat-pc.idc.oracle.com)(PORT=1522)))(CONNECT_DATA=(SID=orcl))) Repository User : SYSMAN  2. To update / modify the connection details to the Grid Control Repository Database: cd <OMS_HOME>/bin emctl config oms -store_repos_details (-repos_host <host> -repos_port <port> -repos_sid <sid> | -repos_conndesc <connect descriptor>) -repos_user <username> [-repos_pwd <pwd>] [-no_check_db] To Modify Password for SYSMAN and other Repository Users To modify the password for the SYSMAN and other repository users in the Weblogic credential store and in the Repository Database: cd <OMS_HOME>/bin emctl config oms -change_repos_pwd [-old_pwd <old_pwd>] [-new_pwd <new_pwd>] [-use_sys_pwd [-sys_pwd <sys_pwd>]] For details, refer to our technical tip: "12c Cloud Control: How to Change the SYSMAN Password at OMS and Repository?" on page 28 EM 12c: Creating an Administrative User and Assigning an Administrative Role in Enterprise Manager Cloud Control 12.1.0.1 Modified 02-JUL-2012 Applies to Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.1.0 [Release 12.1] Information in this document applies to any platform. Goal This article provides a quick reference for creating an administrative user and assign an administrative role in Enterprise Manger Cloud Control 12c. Fix One Super Administrator, sysman, is created by default during the installation of the Enterprise Manager. By logging into the Cloud Control Console as the sysman Super Administrator, other administrator accounts can be created for daily administration work. NOTE: It is recommended that the sysman account should only be used to perform infrequent system wide, global configuration tasks. Please refer to our technical tip for steps to create an administrative role: "EM 12c: Creating an Administrator Role in Enterprise Manager Cloud Control 12.1.0.1" on page 22 Steps to create an administrative user and assign an administrative role in Enterprise Manager Cloud Control 12c: • Log into Enterprise Manager as a Super Administrator user. • From the Setup menu, select 'Security', then select 'Administrators'. • Select 'Create' in the Administrators page to launch the 'Create Administrator' wizard. • In step 1 of 5 enter the name and password for the role and select 'Next'. • In step 2 of 5 from the list of Available Roles, select the required role to be assigned to the new user and move it to the Selected Roles table and select 'Next'. • Select privileges as required in step 3 of 5, the Target Privileges page. • Select 'Add' in the Target Privileges section and add the Targets from the provided menu as required. • In step 4 of 5, the Resource Privileges page, Privilege Grants can be managed by selecting the pencil icon. • In step 5 of 5, review the changes and select Finish. A confirmation message will show in the next screen when the user is created. Enterprise Manager Cloud control 12c Plug-ins Frequently Asked Questions Modified 18-OCT-2012 Applies to Enterprise Manager Base Platform - Version 12.1.0.1.0 and later Enterprise Manager for Oracle Database - Version 12.1.0.1.0 and later Information in this document applies to any platform. Purpose This notes helps to understand the Enterprise Manager Cloud Control 12 Plug-ins Questions and Answers What is a Plug-in? • A plug-in is a group of files (such as target definition files, collection scripts to collect metrics from targets, and any custom UI components to customize user interfaces) that has been added to a plug-in archive using the Enterprise Manager Extensibility Development Kit (EDK). The plug-in files must be added to an archive before they become a plug-in officially. A plug-in archive file (*.opar) associates the files together as an official plug-in. • Each plug-in defines a new type or types of target that can be monitored by Enterprise Manager. A target, or more specifically, a target instance, can be defined as any entity that can be monitored within an enterprise. This entity can be an application running on a server, the server itself, the network, or any of its constituent parts. • Enterprise Manager makes managing target instances simple by enabling you to add new target instances to the management framework from the Enterprise Manager console. Then you can take advantage of Enterprise Manager's monitoring and administrative features. • A plug-in is like any other Oracle software which have their own Oracle Home, and Opatch inventory. What is the Plug-in Life Cycle? 1. The Plugin-in developer uploads the Plug-in Metadata and Plug-in Archive to EM Oracle store. 2. Using Self Update feature in Enterprise Manager 12c, Plugin-in Metadata is downloaded from EM Oracle store. 3. EM administrators are notified with any Plug-in updates or any new Plug-ins available. 4. Em administrators can initiate the download of the Plug-in. 5. Plug-in Archive is download to the local EM system. 6. EM administrator can deploy the Plug-in to the Management Servers and Management Agents. How to check if any Plug-in updates are available As new versions of a plug-in are released, the EM administrator can import the new plug-in by using the self-update feature of Enterprise Manager to download the new version from the Oracle Store, or by importing the new plug-in archive if it is a private distribution. How to download a Plug-in in Online mode? Plug-ins and Plug-ins updates Online through EM 12c console: 1. From EM console Home page → Setup → Extensibility → Self Update → Plugins 2. Choose a Plug-in in "Available" status. 3. Click on "Download" How to download a Plug-in in Offline mode? 1. Download the updates catalog: • From EM console → Extensibility → Self Update → Click "Check Updates" • Use the provided URL to download the latest updates catalog. 2. Place the downloaded file on Management Server or any managed host. 3. Use emcli to import the catalog into EM. • emcli import_update_catalog -file="file" -omslocal • emcli import_update_catalog -file="file" -host="hostname" [-credential_set_name="setname"] | -credential_name="name" -credential_owner="owner"  Examples:$ emcli import_update_catalog -file="/u01/common/p9984818_121000_Generic.zip" -omslocal $emcli import_update_catalog -file="/u01/common/p9984818_121000_Generic.zip" -host="host1.example.com" -credential_set_name="HostCredsNormal"$ emcli import_update_catalog -file="/u01/common/p9984818_121000_Generic.zip" -host="host1.example.com" -credential_name="host1_creds" -credential_owner="admin1" 
6. Place the downloaded Plug-in on Management Server or any managed host.
7. Use emcli to import the update archive into EM:
• emcli import_update -file="file" -omslocal
• emcli import_update -file="file" -host="hostname" [-credential_set_name="setname"] | -credential_name="name" -credential_owner="owner"

•  Examples: emcli import_update -file="/u01/common/update1.zip" -omslocal emcli import_update -file="/u01/common/update1.zip" -host="host1.example.com" -credential_set_name="HostCredsNormal" emcli import_update -file="/u01/common/update1.zip" -host="host1.example.com" -credential_name="host1_creds" -credential_owner="admin1" 

NOTES:

- After importing the catalog using "import_update_catalog", the Plug-in status will show as Available in EM console
- After imported the Plug-in to EM using "emcli import_update", the Plug-in status will show as Downloaded in EM console
- It's not possible to download the Plug-in using offline mode and further deploy it on online mode, the whole download/deploy process should be performed in one mode either offline or online.

What is the difference between Plug-in version and Plug-in revision?

• A new Plug-in version indicates that there the Plug-in can be "upgraded" to the newer version, however a Plug-in revision indicates that there is a new revision of the same Plug-in version.
• A new revision of the same Plug-in version will contains updates regarding the certification matrix of the Plugin. i.e. Which operating systems are certified with this Plug-in revision.
• The Plug-in upgrade it done out of place, means the newer version will be installed on a new Plug-in Oracle Home.
• When downloading a new revision of the same Plug-in version, there is no need to deploy it on a Management Agent which has an older revision is already deployed. Otherwise it will be required to un-deploy the Plug-in from the Management Agent and further deploy the new Plug-in revision.

How to deploy Plug-ins on Management Server using EM console?

1. From EM console Home page → Setup → Extensibility → Plug-ins
2. Choose a Plug-in to deploy.
3. Deploy on the Management Server
• Repository SYS password is required to start deploying the Plug-in
• It is recommended to take a backup of EM repository and configurations before starting any backup on Management servers.
• Deployment of Plug-in on Management server will require downtime.

How to deploy Plug-ins on Management Server using EMCLI?

•  $emcli deploy_plugin_on_server -plugin=oracle.sysman.db:12.1.0.1.0  How to track the Plug-in deploy status? • As long as the Management server was not shutdown due to Plug-in deployment, you can use the emcli command to track the deployment status • $ emcli get_plugin_deployment_status [-plugin_id="plugin_id"] 
• Once the Management server is down, you can track the deployment status using emctl command

•  $<OMS_HOME>/bin/emctl status oms -details  How to deploy Plug-ins on Management Agent using EM console? 1. From EM console Home page → Setup → Extensibility → Plug-ins 2. Choose a Plug-in to deploy. 3. Deploy on the Management Agent NOTE: The Plug-in should be deployed on the Management Servers before attempting to deploy it on Management Agents. How to deploy Plug-ins on Management Agent using EMCLI? • emcli deploy_plugin_on_agent -agent_names="agent1;agent2" -plugin="plug-in_id[:version"] [-discovery_only] •  emcli deploy_plugin_on_agent -plugin="oracle.sysman.db2" -agent_names="myhost1.example.com:1838"  How to un-deploy Plug-ins from Management Agent using EM console? 1. From EM console Home page → Setup → Extensibility → Plug-ins 2. Choose a Plug-in to unplug. 3. Undeploy on the Management Agent • Agent side un-deployment will delete all the targets of those target types which are brought in by the Plug-in on that agent. • Un-deployment of the Plug-in does not mean a rollback. For instance, if the Plug-in was upgraded from a previous version, using un-deployment the Plug-in will be completely removed and not rolled back to the previous version. How to un-deploy Plug-ins from Management Servers using EMCLI? • emcliemcli undeploy_plugin_from_server -plugin="plug-inId"[:"pluginVersion"] [-sys_password="sys_password"] • $ emcli undeploy_plugin_from_server -plugin="oracle.sysman.db2" 

What is Plug-in Mismatch?

• Agent must report same list of Plug-ins as reported by Management server.
• There are some cases where list on Agent side may not match with list of Management Server. i.e. when agent are installed from a cloned image.
• In this case, Agent is blocked to prevent further damage to repository data.
• Resynchronize the Agent to resolve the issue.

How to find a Plug-in Oracle Home?

• For Management Server: <Middleware_Home>/plugins/<plugin_version>
• For Management Agent: <Agent_Home>/plugins/<plugin_version>

Where to find Plug-ins log location?

• For Management Server: <OMS_HOME>/cfgtoollogs/pluginca
• For Management Agent: <AGENT_HOME>/agent_inst/install/logs

How to get the list of all deployed Plug-ins on Management Server side?

• Use the following emcli command.

•  $emcli list_plugins_on_server  How to get the list of all deployed Plug-ins on Management Agent side? • Use the following emcli command, this command can be executed under any emcli client. • It gets the information for repository inventory: • $ emcli list_plugins_on_agent [-agent_names="agent1,agent2"] 
• Use the following emctl command.
• It gets the information for repository inventory.

•  $<AGENT_HOME>/agent_inst/bin/emctl listplugins agent  What is the difference between Primary OMS and Non-Primary OMS? • Primary OMS is the Management Server resides on host which also have the Weblogic Admin Server installed. • Non-Primary OMS is any Management Server resides on host which the Weblogic Admin Server is not installed. • To verify whether a Management Server is a Primary OMS or not, execute the following command: • $ cat <Middleware_Home>/gc_inst/em/EMGC_OMS1/emgc.properties | grep IS_ADMIN_HOST IS_ADMIN_HOST=true 

What is a Central Agent?

• It is the Agent which resides on the host where a Management Server is running.

Some Possible Failure Scenarios for Plug-in deployment on Management server side

1. 'emcli import_update' fails:
• This command will fail if the software library is not configured.

To configure the software library see the documentation which can be downloaded here:

• Errors are reported in emoms.log and emoms.trc

2. Prerequisites check failure:
• At least one of central agent is unreachable: All Central Agents should be up and running.

• Plug-in deployment locked: As EM 12c does not support parallel plug-in deployment, thus if one plug-in deployment is initiated no other plug-in deployments can be initiated.

3. Problem with EM Job System or DBMS jobs are down:
• Plug-in deployment job depends on EM job system, and a possible failure causes of Plug-in deployment job is either there is a problem in the job system itself (which can be verified by submitting another job type test the job system i.e. OS job), or the DBMS jobs are down.
• To verify the status of the DBMS jobs: From EM console => Setup => Management Service and Repository => Repository Operations => "Repository Scheduler Job Status"
• If the DBMS jobs are down, they can be resubmitted by running the following PL/SQL block on EM repository using SYSMAN user:
 Begin emd_maintenance.submit_em_dbms_jobs(); End; / 
4. Failure while stopping OMS:
• Usually occurs because of stuck database sessions.
• Another possible reason is that a second plug-in deployment has been started too soon. i.e. If a plug-in deployment is just concluded and another deployment has been started immediately, it may fails because there may be some plug-in life cycle callbacks are being executed from the first deployment operation.
• For both cases the solution is to retry the deployment again after some time.

How to Configure Email Notification in Cloud Control 12c

Modified 28-JUN-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later
Information in this document applies to any platform.

Purpose

This document describes the steps involved in configuring the Email Notification Method available in EM Following steps are involved to configure email notifications in 12C Cloud Control

1. Configure Mail Server

It is also necessary to define a Notification Rule. The Notification Rule provides an option to choose the targets and conditions under which the OMS should be sending notifications.

The Notification Rules can be configured with the severities/violations of metrics/policies and various states of the Jobs for which the notification has to be sent for.

Scope

This document is applicable to Enterprise Manager users who would like to configure their Enterprise Manager.

Details

1. Configuring Mail Server

1. Login to 12C Cloud Control
2. Navigate to Setup → Notifications

4. Click on "Test Mail Servers" to verify the Outgoing Mail(SMTP) Server settings.

-- If the connection is successful, it returns the following message:

 Test Results test:25: Test succeeded - You will also need to verify that a test e-mail has been received by email@abc.com [ SMTP Server: test:25, User name: email@abc.com, Secure Connection: SSL] 
-- If the test connection fails, the following message is displayed:

 Test Results test:25: Test failed with message: "Could not connect to SMTP host: test, port: 25" [SMTP Server: test:25, User name: email@abc.com, Secure Connection: SSL] 

5. When you click on 'Test Mail Servers', a test mail is sent to the Email Id specified for 'Sender's E-mail Address' :

 Email Header : subject : EM Test Message From : <value for Identify Sender As> <Sender's E-mail Address> Email Body : This test e-mail message from Oracle Enterprise Manager indicates a successful configuration of your e-mail address and mail server. 

6. Once the "Test Mail Server" is successful, then click on "Apply". This will save the Mail Server Configuration.

2. Navigate to the logged user icon right to the Setup Upper Right menu

3. Click on "Enterprise Manager Password & Email"

4. Click on "Add Another Row" to add E-mail address of the user. Enter the email address and click on "Test". The following message is displayed:

 Test Results email@abc.com: Test succeeded - You will also need to verify that a test e-mail has been received 

The content of the email is as below :

 Email Header : subject : EM Test Message From : <value for Identify Sender As> <Sender's E-mail Address> Email Body : This test e-mail message from Oracle Enterprise Manager indicates a successful configuration of your e-mail address and mail server. 

5. If the email is received to the mail box, click on "Apply" to save the email address for the user.

1. Before defining the notification schedule, you must add the Email Addresses using the steps listed above When trying to "Define Schedule" without adding an Email Address for the administrator will result in the error :

 No E-mail Addresses User <EM Administrator Name> does not have any e-mail addresses defined. The schedule cannot be edited without e-mail addresses. Click here to set e-mail addresses. 

• Login to 12C Cloud Control
• Navigate to Setup → Notifications
• Click on "My Notification Schedule".

This schedule can be modified as required by that particular administrator or as the SYSMAN user.

IMPORTANT: If an another address is added like in step 2, the definition of the schedule is not automatically done, like it is done for the first email address entered.

In that case, the following steps should be achieved :

On the "My Notification Schedule" page, click on "Edit Schedule Definition" button, then :

- Choose "Edit existing Schedule", click on "Continue"
- Ensure that the new address is well mentioned in the "Email Address" field
- Choose the required hours, then click to "Batch Fill-in" button, or manually update only the required cells of the scheduler.
- Click on "Finish" to save the schedule for the new mail address.

EM12c How to Subscribe or Unsubscribe for Email Notification for an Incident Rule Set ?

Modified 07-FEB-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later [Release: 12.1 and later]
Information in this document applies to any platform.

Goal

How to Subscribe or Unsubscribe to get Email Notification for an Incident Rule in Enterprise Manager 12c?

Solution

1. On the 12c Cloud Control Home page, navigate to "Setup" → "Incidents" → "Incident Rules" page.
2. If you want to "subscribe/unsubscribe" for a Rule Set, select the Rule Set name or just select a single Rule.
3. On the same page navigate to "Actions" → "Email" as below :

4. Click on "Subscribe Me" if you want to receive Email Notification for that Rule/Rule Set, else click on "Unsubscribe Me", if you want to disable email notification from that Rule.

5. NOTE: If "subscribed/unsubscribed" for a Rule Set, then the same will be applicable for all the single rules that belong to that Rule Set.

If you are Owner of the Rule Set, You can "Email" Subscribe the other Administrator(s)

There are two different ways to achieve it from the Incident Rules page used in previous steps.

Either

• Select the required Rule Set in the list.
Or

• By Editing the required Incident Rule Set in the list
• Click on the Rules tab. Select the desired rule, and click "Edit".
• Click on "Next" button. Click on "Add Action", or "Edit" the existing Action.
• Enter the Administrator name on either "E-mail To" or "E-mail Cc" fields in the Basic Notification region. As shown on that screenshot :

• Click "Continue", click "Next", click "Next", click "Continue".

A dialog box appears with that message :

 Rule <RuleName> has been successfully modified. Changes are not saved until "Save" button is clicked. 

• Finally click "Save".

NOTE: You can only edit your own 'created' Incident Rule Sets and cannot edit Incident Rule Sets that are System Generated.

How To Configure Notification Rules in 12c Enterprise Manager Cloud Control ?

Modified 06-AUG-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later
Information in this document applies to any platform.

Purpose

This document explains the steps to configure Notification Rules in the 12C Enterprise Manager Cloud Control. Configuring notification rules in 12C is different from earlier releases. The concept and function of notification rules has been replaced with a two-tier system consisting of Incident Rules and Incident Rule Sets.

1. Incident Rules: Operate at the lowest level granularity (on discrete events) and performs the same role as notification rules from earlier releases.
2. Incident Rule Set: An incident rule set is a set of one or more rules that apply to a common set of objects such as targets (hosts, databases, groups), jobs, or Web applications. The set of objects to which the rule set applies do not have to be of the same type. A rule set allows you to logically combine different rules relating to the common set of objects (such as jobs, targets, applications) into a single manageable unit.

Scope

This document is applicable to Enterprise Manager users who would like to configure their Enterprise Manager 12C Cloud Control, so that they can receive notifications for Events (Metric Alerts, Policy Violations... ), Incidents, or Problems in the console.

Details

To configure Incident Rule Sets and Incident Rules in EM 12c, follow the below steps :

Rules are created in the context of a rule set. We need first to create a Rule Set, then Rules that belong to that Rule Set.

1. Navigate to the "Incident Rules" page in 12C Cloud Control.
• Login to 12C Cloud Control
• Navigate to "Setup → Incidents"
• Select "Incident Rules"

2. On that page, click on "Create a Rule Set". In the page displayed, enter the following details:
• Name: Name of the Rule Set
• Description: Description of the Rule Set
• Applies To: Select any of the options provided in the drop down menu. The available options are Targets/Job/Metric Extensions/Self Update

3. In the Rules tab of the "Create/Edit Rule Set" page, click on "Create". Select the type of rule to create (Event, Incident, Problem) on the" Select Type of Rule to Create" page. Click "Continue".

4. In the "Create New Rule" wizard, provide the required information. For example, when selecting "Add Event Rule"
• Select the "Event Type" the rule will apply to, for example, Metric Alert.
Metric Alert is available for rule sets of the type Targets.
There are different type of events available like:
Type/Severity/Category/Target Type/Associated with incident/Event name....

• Select "Specific Metrics" option to add metrics. The table for selecting metric alerts displays.
Click the "Add" button to launch the metric selector.
A list of relevant metrics display. Select the ones in which you are interested.
You also have the option to select the severity and corrective action status.

• Click "OK".

• Once you have provided the initial information, click "Next".

• Click "+Add" to add the actions to occur when the event is triggered.
Enter the administrators in the Notifications section in the appropriate fields in order to activate the notification. Ensure that Mail Notification has been well and completely configured for the Administrators selected in the "Notifications" section when creating the rules. One of the actions is to "Create Incident". As part of creating an incident, you can assign the incident to a particular user, set the priority, and create a ticket. Once you have added all the conditional actions, click "Continue".

• After you have provided all the information on the "Add Actions" page, click "Next" to specify the name and description for the rule.

• Click "Next". Once on the Review page, verify that all the information is correct. Click "Back" to make corrections; click "Continue" to return to the "Edit/Create Rule Set" page.

• Important: Click on "Save" to ensure that the changes to the rule set and rules are saved.

How to Add agent and host after removing it from 12c Cloud Control

Modified 21-FEB-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Goal

How to Add agent and host targets after removing it from 12c Cloud Control.

Solution

1. Start the agent by issuing following command:

 <AGENT_INSTANCE_HOME>/bin/emctl start agent 

2. Issue following command:

 <AGENT_INSTANCE_HOME>/bin/emctl config agent addinternaltargets 

3. This would list the agent target in Setup → Add Target → Auto Discovery Results → Non-Host Targets

4. Secure the agent by issuing following command:

 <AGENT_INSTANCE_HOME>/bin/emctl secure agent 
This would show the host and agent in pending status under Targets → All targets.

5. Do the following steps to show the status of host and agent as up:

 <AGENT_INSTANCE_HOME>/bin/emctl stop agent <AGENT_INSTANCE_HOME>/bin/emctl clearstate agent <AGENT_INSTANCE_HOME>/bin/emctl start agent 
Once done, the host and agent status would be shown as up.

6. Do a resync of agent Now rest of the targets can be discovered.
7. See our technical tip 'How to Perform Agent Resynchronization in 12C Cloud Control' on page 2.

NOTE: <INSTANCE_HOME> would represent full path to agent_inst. If agent is installed in /u01/app/agent, then /u01/app/agent/agent_inst would represent <INSTANCE_HOME>

Some of EM 12c Reports Return No Data For All Target Databases

Modified 15-NOV-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

In 12c Cloud Control, running reports that based on the "oracle_security" database metric returns no data for all target databases

Example of affected reports:

• DBA Group Membership
• Permission and Ownership
• Privileges Granted to PUBLIC
• Privileges Granted with Administrative or Grant Options
• System Packages with Public Execute Privileges

Cause

The "oracle_security" database metric collection is disabled out of the box.

Solution

To enable oracle_security metric collection for one database:

• Go to the DB home page → Oracle Database → Monitoring → Metric and Collection Settings → Other collected items
• Locate "Security Configuration for Oracle Database (Reports)". Many entries will be found.
• For any entry, click on the "Disable" Link
• Click On Enable → Continue → OK → OK
• Go back to Metric and Collection Settings page and make sure that it is enabled now
• Wait for the next metric collection, or force the collection by issuing this command at target side:

emctl control agent runCollection <target_name>:oracle_database oracle_security

NOTE: To enable this metric for multiple databases, use monitoring templates.

Collecting The Required Information For Support To Troubleshoot ASM/ASMLIB Issues.

Modified 10-JAN-2012

Applies to

Oracle Server - Enterprise Edition - Version: 10.1.0.2 to 11.2.0.2 - Release: 10.1 to 11.2
Linux x86
Linux x86-64
Linux Itanium

Goal

The present document provides a list of steps to collect the required information to troubleshoot & diagnostic ASM/ASMLIB Issues required for support. Obtain the most recent ASMLIB & ASM state from your current environment.

Solution

1. In order to check if the ASMLIB API is correctly configured, please execute the next commands and provide us the output (from each node if this is RAC):

 $> cat /etc/*release$> uname -a $> rpm -qa |grep oracleasm$> df -ha $>/usr/sbin/oracleasm configure$> /sbin/modinfo oracleasm 

2. Check the discovery path (from each node if this is RAC):

 $> /etc/init.d/oracleasm status$> /usr/sbin/oracleasm-discover $> /usr/sbin/oracleasm-discover 'ORCL:*'  3. Please check if the ASMLIB devices can be accessed (from each node if this is RAC): $> /etc/init.d/oracleasm scandisks $> /etc/init.d/oracleasm listdisks$> /etc/init.d/oracleasm querydisk -p <each disk from previous output> $> ls -l /dev/oracleasm/disks$> /sbin/blkid 

4. Upload the next files from each node if this is RAC:

 => /var/log/messages* => /var/log/oracleasm => /etc/sysconfig/oracleasm 

5. Please show us the partition table (from each node if this is RAC):

 $> cat /proc/partitions  6. If you are using multipath devices (mapper devices or emcpower) then show me the output of: $> ls -l /dev/mpath/* $> ls -l /dev/mapper/*$> ls -l /dev/dm-* $> ls -l /dev/emcpower*  Or if you have another multipath configuration then list the devices: $> ls -l /dev/<multi path device name>* 

7. Finally connect to your ASM instance, execute the next script and upload me the output file (from each node if this is RAC):

 spool asm<#>.html SET MARKUP HTML ON set echo on set pagesize 200 alter session set nls_date_format='DD-MON-YYYY HH24:MI:SS'; select 'THIS ASM REPORT WAS GENERATED AT: ==> ' , sysdate " " from dual; select 'HOSTNAME ASSOCIATED WITH THIS ASM INSTANCE: ==> ' , MACHINE " " from v$session where program like '%SMON%'; select * from v$asm_diskgroup; SELECT * FROM V$ASM_DISK ORDER BY GROUP_NUMBER,DISK_NUMBER; SELECT * FROM V$ASM_CLIENT; select * from V$ASM_ATTRIBUTE; select * from v$asm_operation; select * from gv$asm_operation; select * from v$version; show parameter asm show parameter cluster show parameter instance_type show parameter instance_name show parameter spfile show sga spool off exit 

8. Also, if this is not a new ASM/ASMLIB implementation, please describe in detail what has changed since this last worked (OS patches, OS kernel upgrade, SAN migration, etc.)?

Tips On Installing and Using ASMLib on Linux

Modified 21-NOV-2012

Applies to

Linux OS - Version 1 and later
Information in this document applies to any platform.

Goal

ASMLib is a support library for the Automatic Storage Management feature of Oracle Database 10g. This document is a set of tips for installing the library and its supporting driver.

This document describes the steps required to install the Linux specific ASM library and its assocated driver. This library is provide to enable ASM I/O to Linux disks without the limitations of the standard Unix I/O API. The steps below are steps that the system administrator must follow.

Fix

Locating the ASMLib Packages

You will see 4-6 packages for your Linux platform. The oracleasmlib package provides the actual ASM library. The oracleasm-support package provides the utilities used to get the ASM driver up and running. Both of these packages need to be installed.

The remaining packages provide the kernel driver for the ASM library. Each package provides the driver for a different kernel. You must install the appropriate package for the kernel you are running. Use the "uname -r command to determine the version of the kernel. The oracleasm kerel driver package will have that version string in its name. For example, if you were running Red Hat Enterprise Linux 4 AS, and the kernel you were using was the 2.6.9-5.0.5.ELsmp kernel, you would choose the oracleasm-2.6.9-5.0.5-ELsmp package.

Installing the ASMLib Packages

So, to install these packages on RHEL4 on an Intel x86 machine, you might use the command:

 rpm -Uvh oracleasm-support-2.0.0-1.i386.rpm \ oracleasm-lib-2.0.0-1.i386.rpm \ oracleasm-2.6.9-5.0.5-ELsmp-2.0.0-1.i686.rpm 
If you were on a different machine, the package names would be slightly different, replacing 'i686' with the appropriate architecture. Use the package names relevant for your distribution

NOTE: Distributions with the Linux 2.4 kernel still use the 1.0 kernel driver, while distributions based on the Linux 2.6 kernel use the 2.0 kernel driver. All distributions use version 2.0 of the support and library packages.

Once the command completes, ASMLib is now installed on the system.

Making the ASM Driver Available

Now that the ASMLib software is installed, a few steps have to be taken by the system administrator to make the ASM driver available. The ASM driver needs to be loaded, and the driver filesystem needs to be mounted. This is taken care of by the initialization script, /etc/init.d/oracleasm.

Run the /etc/init.d/oracleasm script with the 'configure' option. It will ask for the user and group that default to owning the ASM driver access point. If the database was running as the 'oracle' user and the 'dba' group, the output would look like this:

 [root@ca-test1 /]# /etc/init.d/oracleasm configure Configuring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting without typing an answer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: oracle Default group to own the driver interface []: dba Start Oracle ASM library driver on boot (y/n) [n]: y Fix permissions of Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration [ OK ] Creating /dev/oracleasm mount point [ OK ] Loading module "oracleasm" [ OK ] Mounting ASMlib driver filesystem [ OK ] Scanning system for ASM disks [ OK ] 
This should load the oracleasm.o driver module and mount the ASM driver filesystem. By selecting enabled = 'y' during the configuration, the system will always load the module and mount the filesystem on boot.

The automatic start can be enabled or disabled with the 'enable' and 'disable' options to /etc/init.d/oracleasm:

 [root@ca-test1 /]# /etc/init.d/oracleasm disable Writing Oracle ASM library driver configuration [ OK ] Unmounting ASMlib driver filesystem [ OK ] Unloading module "oracleasm" [ OK ] [root@ca-test1 /]# /etc/init.d/oracleasm enable Writing Oracle ASM library driver configuration [ OK ] Loading module "oracleasm" [ OK ] Mounting ASMlib driver filesystem [ OK ] Scanning system for ASM disks [ OK ] 
Creating/Deleting/Querying/Scanning ASM Disks

The system administrator has one last task. Every disk that ASMLib is going to be accessing needs to be made available. This is accomplished by creating an ASM disk. The /etc/init.d/oracleasm script is again used for this task:

 [root@ca-test1 /]# /etc/init.d/oracleasm createdisk VOL1 /dev/sdg1 Creating Oracle ASM disk "VOL1" [ OK ] 
Disk names are ASCII capital letters, numbers, and underscores. They must start with a letter.

Disks that are no longer used by ASM can be unmarked as well:

 [root@ca-test1 /]# /etc/init.d/oracleasm deletedisk VOL1 Deleting Oracle ASM disk "VOL1" [ OK ] 
Any operating system disk can be queried to see if it is used by ASM:

 [root@ca-test1 /]# /etc/init.d/oracleasm querydisk /dev/sdg1 Checking if device "/dev/sdg1" is an Oracle ASM disk [ OK ] [root@ca-test1 /]# /etc/init.d/oracleasm querydisk /dev/sdh1 Checking if device "/dev/sdh1" is an Oracle ASM disk [FAILED] 
Existing disks can be listed and queried:

 [root@ca-test1 /]# /etc/init.d/oracleasm listdisks VOL1 VOL2 VOL3 [root@ca-test1 /]# /etc/init.d/oracleasm querydisk VOL1 Checking for ASM disk "VOL1" [ OK ] 
When a disk is added to a RAC setup, the other nodes need to be notified about it. Run the 'createdisk' command on one node, and then run 'scandisks' on every other node:

 [root@ca-test1 /]# /etc/init.d/oracleasm scandisks Scanning system for ASM disks [ OK ] 
ASMLib uses discovery strings to determine what disks ASM is asking for. The generic Linux ASMLib uses glob strings. The string must be prefixed with "ORCL:". Disks are specified by name. A disk created with the name "VOL1" can be discovered in ASM via the discovery string "ORCL:VOL1". Similarly, all disks that start with the string "VOL" can be queried with the discovery string "ORCL:VOL*".

Disks cannot be discovered with path names in the discovery string. If the prefix is missing, the generic Linux ASMLib will ignore the discovery string completely, expecting that it is intended for a different ASMLib. The only exception is the empty string (""), which is considered a full wildcard. This is precisely equivalent to the discovery string "ORCL:*".

NOTE: Once you mark your disks with Linux ASMLib, Oracle Database 10g R1 (10.1) OUI will not be able to discover your disks. It is recommended that you complete a Software Only install and then use DBCA to create your database (or use the custom install).

ACFS Technical Overview and Deployment Guide

Modified 10-JAN-2012

Applies to

Oracle Server - Enterprise Edition - Version 11.2.0.2 to 11.2.0.2 [Release 11.2]
Information in this document applies to any platform.

Abstract

There is a big movement in the industry towards Consolidation, Grid and Cloud Computing. At the core of all this is centralized-clustered storage which requires a centralized file store, such as a filesystem.

In Oracle Grid Infrastructure 11g Release 2, Oracle extends the capability of ASM by introducing the ASM Cluster Filesystem, or ACFS. ACFS is feature-rich, scalable, POSIX-compliant filesystem that is built from the traditional ASM disk groups. ACFS is built on top of the standard vnode/VFS filesystem interface ubiquitous in the Unix/Linux world, thus ACFS uses standard file related system calls like other filesystems in the market. However, most filesystems on the market are platform specific; this is especially true of cluster filesystems. ACFS is a multi-platform filesystem that runs on any platform that currently runs ASM.

This document discusses ACFS and the underlying architecture. It is assumed that the reader has some familiarity with ASM and has read the ASM Technical Overview and Best Practices paper, which is attached here.

***Please review the attached ASM Best Practices Paper ****

Also, this document mainly covers the deployment of ACFS in a Linux environment. For Windows configurations please see the Oracle Storage Administrator's Guide for the appropriate commands.

***Please review the attached ACFS Deployment Guide ****

How to Manage Cloud Control Agent 12c Release 12.1.0.2 and Later Logging and Tracing

Modified 22-FEB-2013

Applies to

Enterprise Manager Base Platform - Version 12.1.0.2.0 and later
Information in this document applies to any platform.

Purpose

This document explains how to manage the Cloud Control Agent 12.1.0.2 and later Logging and Tracing.

Details

The logging and tracing mechanism has slightly changed in the Cloud Control Agent 12.1.0.2 compared to the Cloud Control Agent 12.1.0.1.

For Logging and Tracing of the Cloud Control Agent Release 12.1.0.1 please read the following technical note:

"How to Configure Logging for the Enterprise Manager Cloud Control Management Agent12.1.0.1 (12c R1)" on page 24

The Enterprise Manager cloud Control Management Agent Logging and Tracing is fully documented in:

Oracle Enterprise Manager Cloud Control Administrator's Guide 12c Release 2 (12.1.0.2).

Check the chapter 24.2 Locating Management Agent Log and Trace Files

• About the Management Agent Log and Trace Files

• Structure of Agent Log Files

• Locating the Management Agent Log and Trace Files

• Setting Oracle Management Agent Log Levels

• Modifying the Default Logging Level

• Setting gcagent.log

• Setting gcagent_error.log

• Setting the Log Level for Individual Classes and Packages

• Setting gcagent_mdu.log

• Setting the TRACE Level

Note for Windows Platforms: the command emctl setproperty agent -name needs that the property name is enclosed with " instead of ' on Unix and Linux platforms.
Example for linux:

 $emctl setproperty agent -name 'Logger.log.level' -value 'DEBUG'  Equivalent on Windows:  C> emctl setproperty agent -name "Logger.log.level" -value 'DEBUG'  If you use 'Logger.log.level' on Windows Platforms, the command will succeed. However if you check the agent file %INSTANCE_HOME%\sysman\config\emd.properties, you will notice that the property value has not been changed. How To Configure Enterprise Manager 12c Account To Manage GoldenGate Instances Only? Modified 10-JAN-2012 Applies to Management Pack for Oracle GoldenGate - Version 11.2.1.0.1 and later Information in this document applies to any platform. Goal The goal of this document is to show how to create a separate Oracle Enterprise Manager 12c user to manage only GoldenGate instances/targets. Once this user account is created and the user logs in to Enterprise Manager 12c the account will only be able to see GoldenGate instances/targets assigned to it. This document assumes GoldenGate Monitor Plug-In for Oracle Enterprise Manager 12c is fully deployed configured and all GoldenGate instances/targets are already discovered/promoted. Fix • Login to Enterprise Manager 12 as SYSMAN user Click on "Setup" → "Security" → "Administrators" Click "Create" button to create the new account • On the "Create Administrator: Properties" page (Step 1 of 5) fill in the fields with the relevant information. Click "Next" button to continue. • On the "Create Administrator [name]: Roles" page (Step 2 of 5) the default roles granted are EM_USER and PUBLIC. Click "Next" button to accept default roles and continue. • On the "Create Administrator [name]: Target Privileges" page (Step 3 of 5) under "Target Privileges" section Click "Add" button to select and assign GoldenGate targets to this account. On the "Search and "Add: Targets" page → "Search" section select Target Type Oracle GoldenGate Click "Go" button to search for GoldenGate target types. Once the search results are shown, click "Select All" link to select all GoldenGate target types. Then Click "Select" button to go back to main "Target Privileges" screen Repeat the same procedure to add more GoldenGate target types to this account as needed. Click "Add" button to select and assign GoldenGate targets to this account. On the "Search and "Add: Targets" page → "Search" section select Target Type Oracle GoldenGate Manager Click "Go" button to search for GoldenGate target types. Once the search results are shown, click "Select All" link to select all GoldenGate Manager target types. Then Click "Select" button to go back to main "Target Privileges" screen Click "Add" button to select and assign GoldenGate targets to this account. On the "Search and "Add: Targets" page → "Search" section select Target Type Oracle GoldenGate Extract Click "Go" button to search for GoldenGate target types. Once the search results are shown, click "Select All" link to select all GoldenGate Manager target types. Then click "Select" button to go back to main "Target Privileges" screen • Once all GoldenGate targets are selected for this new account the "Create Administrator [name]: Target Privileges" page (Step 3 of 5) will look similar to this Click on "Next" button to continue. • On the "Create Administrator [name]: Resource Privileges" page (Step 4 of 5) click "Next" button to continue with no changes made. • On "Create Administrator [name]: Review" page (Step 5 of 5) click "Finish" button to create the account. • Verifying Account Access only to GoldenGate targets Logout SYSMAN account from Oracle Enterprise Manager 12c. Login to Enterprise Manager 12c with the newly created user. The "Enterprise Summary" page will display a count of "Targets Monitored". These targets should only be the GoldenGate targets previously configured for this account. Clicking on the "target monitored" (13 in this example) will display a screen like Clicking on "Targets" → Oracle GoldenGate Will display "Oracle GoldenGate Home" Page Understanding Host Credentials in 12C Cloud Control Modified 21-FEB-2012 Applies to Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later [Release: 12.1 and later] Information in this document applies to any platform. Purpose This document helps in understanding the different types of host credentials and its usage in 12C Cloud Control. Scope and Application This document is applicable for the 12C Cloud Control users and administrators working on jobs, patching.. that uses credentials. Understanding Host Credentials in 12C Cloud Control Host Credentials Credentials are used to simply access to the managed targets by storing the target login credentials in repository database. These credentials can be used for executing the jobs, patching and other management tasks. The credentials can be set for Normal user or Privileged user. Types of Credentials There are three different types of host credentials in 12C Cloud Control. Credential Type Description Preferred Credential With preferred credentials set, users can access an Enterprise Manager target that recognizes those credentials without being prompted to log into the target. Preferred credentials are set on a per user basis, thus ensuring the security of the managed enterprise environment. Named Credential Named Credential specifies a users authentication information on a system. Named Credential can be a username/password, or a public key-private key pair. New Credential New Credential is to create a new set of credentials that will override the preferred credentials or the named credentials. If you want to register the new set of credentials with Enterprise Manager, then click Save As, and either accept the default profile name or enter a custom name for it. Click Test to verify the credentials entered. Usage 1. Preferred Credentials • The Preferred Credentials are of two types: 1. Default Preferred Credentials 2. Target Preferred Credentials • To set the Preferred Credentials: 1. Login to 12C Cloud Control 2. Navigate to Setup --> Security 3. Select "Preferred Credentials" 4. Select the Target Type "host" and Click on "Manage Preferred Credentials" It will display the page as shown below to set the default preferred credentials and target preferred credentials. Depending on requirement, can use either default preferred credentials or target preferred credentials. Below is Tabular column which explains the precedence of credentials in different scenarios: Default Preferred Credentials Target Preferred Credentials Usage Set for the host Not set for any of the host Default Credentials will be used Normal Credentials set for the host Privileged Credentials are set for the host target For operations where normal credentials are required, default credentials are used. For operations where privileged credentials are required, target preferred credentials are used Privileged Credentials set for the host Normal Credentials are set for the host target For operations where normal credentials are required, target preferred credentials are used. For operations where normal credentials are required, default preferred credentials are used Normal and Privileged Credentials are set for the host Normal and Privileged Credentials are set for the host target Target Preferred Credentials will take precedence over default credentials 2. Named Credentials 3. Named Credential can be a username/password, or a public key-private key pair, or an X509v3 certificate and use these for performing operations such as running jobs, patching and other system management tasks. For example, a user can store the username and password they want to use for patching as â€œMyPatchingCredsâ€?. They can later submit a patching job that uses â€œMyPatchingCredsâ€? to patch the production databases Named Credentials are shareable. Owner of named credentials controls access for others. Owner will be able to grant View/Edit/Full Privileges to the other administrators. Even Named Credentials can be set as target based or global. To set the Named Credentials: 1. Login to 12C Cloud Control 2. Navigate to Setup → Security 3. Select "Named Credentials" 4. In the page displayed, Click on "Create" 5. Provide the required fields and Click on "Save" If you want to test connecting to the host with those credentials, click Test and Save. Known Issues 1. Deleting Named Credentials used as Preferred Credentials for the host target throws error as below: Clicking on "Delete" button gives the following error message: To delete such named credentials, delete the preferred credentials set for the host and then delete the named credentials. 2. Creating Preferred Credentials with the same user as existing gives a warning message as below: 12c: Plugin Deployment Fails With "Plug-in is not certified for the Operating System" Modified 03-JUL-2012 Applies to Enterprise Manager for Oracle Database - Version 12.1.0.1.0 to 12.1.0.1.0 [Release 12.1] Information in this document applies to any platform. Symptoms Deploying plugin's using 12c Cloud Control fails with following error  Error: Plug-in is not certified for the Operating System  gcagent.log generated under target machine$AGENT_INST/sysman/log/ shows following errors:

 oracle.sysman.emSDK.agent.client.exception.GetMetricDataException: Unable to find discovery home for plugin id:oracle.sysman.db at oracle.sysman.gcagent.dispatch.cxl.GetMetricDataAction.satisfyRequest(GetMetricDataAction.java:268) at oracle.sysman.gcagent.dispatch.ProcessRequestAction._call(ProcessRequestAction.java:128) at oracle.sysman.gcagent.dispatch.ProcessRequestAction.call(ProcessRequestAction.java:93) at oracle.sysman.gcagent.dispatch.InlineDispatchCoordinator.dispatchRequest(InlineDispatchCoordinator.java:220) at oracle.sysman.gcagent.dispatch.DispatchRequestsAction.call(DispatchRequestsAction.java:76) at oracle.sysman.gcagent.dispatch.DispatchRequestsAction.call(DispatchRequestsAction.java:45) 

emoms.trc on the OMS Server under $gc_inst/em/EMGC_OMS1/sysman/log/ shows following errors:  2011-12-22 03:19:52,769 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN Util.class logp.251 - oracle.sysman.core.discovery.uimodel: MissingResourceException: nlsid=OSB_DISCOVERY_TEXT, rb=oracle.sysman.db.rsc.ob 2011-12-22 05:36:46,620 [[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR agent.EMPluginAgentDeploymentHelper logp.251 - Setting NOT_CERTIFIED_FOR_PLATFORM for aruId 23  Deploying Fusion Middleware Plugin Fails with  "Discovery Module Oracle Fusion Middleware is not supported on selected monitoring agent <agent name>: Plug-in is not certified for Operating System."  Cause Required patches are missing from the OMS side. For Database Plugin Patch 13086659 is required on OMS before deploying the plugin (SOLARIS ONLY) For Fusion Middleware Plugin Patch 13092872 is required on OMS before deploying the plugin (SOLARIS ONLY) OS's AIX, HP-IA, HP-PA Need's Patch 13426642 on top of OMS. (For DB Plugins ) OS's AIX, HP-IA, HP-PA Need's Patch 13426710 on top OMS. (For FMW Plugins) Solution Install required, OMS Patch 13426642 and then - For Database Plugin Apply the Patch 13086659 to OMS before deploying plugin For Fusion Middleware Plugin Patch 13092872 is required on OMS before deploying the plugin  Apply the patch as per Readme file. Plugin home should be$MW_HOME/plugins/<plugin name> For example : PLUGIN_HOME=/u01/app/oracle/MW/plugins/oracle.sysman.emas.oms.plugin_12.1.0.1.0 DB Plugin is = oracle.sysman.db.oms.plugin_12.1.0.1.0 FMW Plugin is = oracle.sysman.emas.oms.plugin_12.1.0.1.0 

Login to 12c Cloud Control as Super User and navigate to:

"Setup" >> "Extensibility" >> "Plugins"

Once the Plugin is downloaded successfully, navigate to "Setup" >> "Extensibility" >> "Plugins" >> Select the Plugin "Oracle Fusion Middleware" and click " Deploy on Management Servers"

NOTE : Deploying Plugin to OMS will bring down the OMS.

Once the plugin get deployed successfully to the OMS server.

Deploy the plugin to the Target Agents

As a Super User navigate to:

"Setup" >> "Extensibility" >> "Plugins" >> Select the plugin and click "Deploy on Management Agent"

EM12c: How To Upgrade Plugin On 12.1 Management Agent

Modified 19-OCT-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Goal

How to upgrade plugin on 12.1 agent

Fix

On the Agent-side, the Plugin Upgrade can only happen across version which means we can Upgrade from 12.1.0.1 [any revision]
to 12.1.0.2 [any revision] but not 12.1.0.2 [lower revision] to 12.1.0.2 [higher revision]

This is because we have 3 revisions for each Plugin version as below:

12.1.0.2 Base which support Linux
12.1.0.2 [20120427] which supports Linux + Windows
12.1.0.2 [20120704] which supports all platforms

We would have 12.1.0.2 Base installed on Linux.

It is not required to upgrade from 12.1.0.2 to 12.1.0.2 [u20120704]
This is because the latest revision would only contain platform-specific Fix for the supported platform.

Because of the above reasons; in the current version; there is no provision to Upgrade at a revision level on the Agent-side. However, this is still required on OMS-side; as you need the latest plugin on OMS to support Agents on Solaris, HP and AIX.

Plugin on Management Server Fails With Error "java.io.FileNotFoundException: OMS archive not found"

Modified 31-JAN-2012

Applies to

Enterprise Manager for Oracle Database - Version: 12.1.0.1.0 and later [Release: 12.1 and later]
Information in this document applies to any platform.

Symptoms

Deploying database plugin to Management server in 12c Cloud Control fails with following error:

2. Navigate to "Setup" → "Extensibility"
3. Click on "Plug-ins"
4. Expand Databases
5. Select "Oracle Database" and Choose the option "Deply On Management Servers"
6. Provide the SYS password in the page displayed and Click on "Next"

The prerequisites fail with the error:

 java.io.FileNotFoundException: OMS archive not found 

Cause

2. Navigate to "Setup" → "Extensibility"
3. Click on "Plug-ins"
4. Verify the Oracle Database Plugin version

The latest database plugin version available is u111221 but it is not yet downloaded.
The latest downloaded plugin version shows u111220 as shown in figure below

Solution

Perform the following steps to resolve the issue:

1. Ensure that the latest database plugin software is downloaded as per the following note:

"How to Deploy Latest Database Plugin to OMS and Agent in 12C Cloud Control" on page 44

2. Perform the plugin deployment on management server

1. Delete the plugin revision(u111221) for oracle database as mentioned below:
• Navigate to Setup → Extensibility
• Select the option "Self Update" and Click on "Plug-ins"
• Select the latest revision version(u111221) for "Oracle Database"
• Click on "Delete"

2. Deploy the latest version of plugin(u111221) as per the following note:

"How to Deploy Latest Database Plugin to OMS and Agent in 12C Cloud Control" on page 44

How to Deploy the Latest Database Plugin to the OMS and the Agent in 12C Cloud Control

Modified 29-NOV-2012

Applies to

Enterprise Manager for Oracle Database - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Purpose

This document describes how to deploy the latest database plugin to the OMS and the Agent using 12C Cloud Control.

Scope

Intended Audience: 12C Cloud Control administrators managing database targets.

Details

To deploy the latest database Plug-in on the OMS and the Agent, you need to perform the following steps:

1. Acquire the latest database Plug-in software using Self Update
2. Deploy the Plug-in to the management server
3. Deploy the Plug-in to the management agent

Prerequisites

1. Following are the database plugins released till date:

12.1.0.2.0[u20120804]
12.1.0.2.0[u20120704]
12.1.0.2.0[u20120427]
12.1.0.2.0
12.1.0.1.0[u20111121] 12.1.0.1.0[u20111120]
12.1.0.1.0

The latest avaialble database plugin is 12.1.0.2.0[u20120804]

2. Required patches to deploy the database plugins:

 12.1.0.2.0[u20120804] Database plugin home patch 14340329 12.1.0.2.0[u20120704] OMS patch 14040891 Database plugin home patch 14089634 12.1.0.2.0[u20120427] OMS patch 13707704 Database plugin home patch 13713877 12.1.0.2.0 OMS Patch 13242773 12.1.0.1.0[u20111121] OMS Patch 13426571 Database plugin home patch 13426642 12.1.0.1.0[u20111120] OMS Patch 12989982 Database plugin home patch 13086659 12.1.0.1.0 Default database plugin available with 12c install

3. The Admin server should be up and running when deploying plugins to the management server.
Check beforehand the weblogic user credentials by logging in to the Admin server console.

A. Acquire the latest database plugin software

When trying to acquire plugin software using self update, there are two phases:

1. Get the latest software using "Check Updates" to import the catalog

The above steps can be performed either using online mode(using console) or offline mode(using emcli).

If "Check Updates" is performed using offline mode(using emcli) and then try to download the plugin using online mode( plugins page) in the console, the plugin will not get deployed resulting in errors.

Using Online mode:

1. Login to 12C Cloud Control
2. Navigate to "Setup" → "Extensibility"
3. Select the option "Self Update"
4. In the page displayed, Select "Plugins" and Click on "Check Updates". This will execute a job and once the job is finished, go back to the "Self Update" Page. Click on "Plug-in" to navigate to Plugin home page.

Using Offline mode:

1. Login to 12C Cloud Control
2. Navigate to "Setup" → "Extensibility"
3. Select the option "Self Update"
4. In the page displayed, Select Plugins and Click on "Check Updates"

For example:

Once catalog is downloaded, it can be imported to Enterprise Manager in one of the following two ways:

Transfer the catalog to the Management Server host and run following command to import to Enterprise Manager

 emcli import_update_catalog -file=<catalog file name with full path> -omslocal 
Transfer the catalog to any Managed Host in your environment and run following command to import to Enterprise Manager

 emcli import_update_catalog -file=<catalog file name with full path> -host=<host name> <host credential options> 

Once the above steps are performed then Click on "Plug-in" to navigate to Plugin home page.
The plugin software for "Oracle Database" is shown as "Available"

5. Choose the Plugin "Oracle Database" and Click on "Download", this will give a pop up with set of emcli commands to download the software.

For example:

Use the following URL(s) on a server with connection to My Oracle Support to download and follow the instructions

Once update is downloaded, it can be imported into Enterprise Manager in one of the following two ways:

Transfer the downloaded file to the Management Server host and run following command to import into Enterprise Manager

 emcli import_update -omslocal -file=p13528532_112000_Generic.zip 
Transfer the downloaded file to any Managed Host in your environment and run following command to import into Enterprise Manager

 emcli import_update -host=<hostname> -file=p13528532_112000_Generic.zip <host credential options> 
Now the database plugin shown as downloaded in the 12c Cloud Control.

B. Deploy the Plug-in to the Management Server

To deploy to the database plugin to OMS,ensure that the required patches are applied to OMS HOME and PLUGIN HOME as mentioned in the prerequisites section.

To verify the patches applied to plugin home, for example to check for patch 13086659:

 $echo$ORACLE_HOME /DATA/oracle/Middleware/oms $echo$PLUGIN_HOME /DATA/oracle/Middleware/plugins/oracle.sysman.db.oms.plugin_12.1.0.1.0 $opatch lsinventory -oh$PLUGIN_HOME | grep 13086659 Patch 13086659 : applied on Thu Dec 22 10:25:29 IST 2011 13086659 
Follow the below mentioned steps to deploy plugin to OMS:

1. Login to 12C Cloud Control
2. Navigate to "Setup" → "Extensibility"
3. Select the option "Plug-ins"
4. Expand the option "Databases"
5. Select "Oracle Database"
6. Expand the option "Deploy On" and Choose "Management Servers"
7. In the page displayed, provide the SYS user password
8. The deployment performs the following:

Step 1: Following validations are performed:
• Initialize the plugin
• Validate plug-in home
• Check mandatory patches

Once the above three steps are done, Click on "next".

Step 2: In the "Review" page, it gives warning as below. Click on "Deploy" to proceed with deployment.

Step 3: Confirmation message about deployment of database plugin on Management Server.

This will complete the deployment of database plugin to management server.

C. Deploy the Plug-in to the Management Agent

Its a prerequisite to deploy the latest database plugin to OMS before deploying to Agent.

Ensure the plugin deployment to OMS is successful and then perform the following to deploy plugin to agent:

1. Login to 12C Cloud Control
2. Navigate to "Setup" → "Extensibility"
3. Select the option "Plugins"
4. Expand the option "Databases"
5. Select "Oracle Database"
6. Expand the option "Deploy On" and Choose "Management Agent"
7. In the page displayed, Click on "Add" to add the Agent on which plugin needs to be deployed.
8. After adding the agent name, it will run the prerequisite check, after successful completion of prerequisite check, click on "Next".
9. A Confirmation message about deployment of database plugin on Management Agent will be displayed.
This will complete the deployment of database plugin to management agent.

To deploy to the database plugin to OMS,ensure that the required patches are applied to OMS HOME and PLUGIN HOME as mentioned in the prerequisites section.

NOTE: On the Agent-side, the Plugin Upgrade can only happen across versions which means, we can Upgrade from 12.1.0.1 [any revision] to 12.1.0.2 [any revision] but not 12.1.0.2 [lower revision] to 12.1.0.2 [higher revision].

For more details, see our technical tip "EM12c: How To Upgrade Plugin On 12.1 Management Agent" on page 43

Known Issues

1. How to identify whether the latest database plugin is available/downloaded or not.

1. Login to 12C Cloud Control
2. Navigate to "Setup" → "Extensibility"
3. Select the option "Plugins"
4. Expand the option "Databases"
5. Select "Oracle Database"

The latest version of the Plug-in will have the version like 12.1.0.1[u111120].

2. How to check on which agents the database plugin is deployed along with the version of plugin.

1. Login to 12C Cloud Control
2. Navigate to "Setup" → "Extensibility"
3. Select the option "Plugins"
4. Expand the option "Databases"
5. Select "Oracle Database"
6. Expand the option "Actions" and Click on "Information"

It will display the information of agents with deployed database plugin version.

3. Error: "Plug-in is not certified for the Operating System"

Deploying database plugin on Solaris Management Agent, getting the following error:

Perform the above mentioned steps (acquire latest database software and then deploy to OMS)
Retry the deployment of plugin to agent

See our technical tip "Plug-in is not certified for the Operating System" on page 42

4. Error: " The available version of plug-in is not yet downloaded."
Deploying database plugin to Management Server, getting the following error:

Need to download the latest version of database plugin to the local EM store as mentioned in stepa above and then deploy the plugin to Management Server.

Deploying database plugin to Management Server, getting the following error:

Ensure that the following patch is applied to the PLUGIN home:

Patch 13086659 - TRACKING BUG:N-APPLY PATCH DB PLUG-IN OMS HOME FOR BUCKET 1-SPARC,SOLARIS X64

Perform the plugin deployment to Management Server.

6. Patch for bugs 13086659 needs to be applied on Management Service <oms host> before deploying this plugin revision

Ensure that the patch 13086659 is already applied to the Plug-in home.

Apply the following patch to OMS:

Patch 12989982 - TRACKING BUG:N-APPLY PATCH FOR PLATFORM OMS HOME FOR BUCKET 1-SPARC,SOLARIS X64

Perform the Plug-in deployment to Management Server.

7. Patch for bugs 13426642 needs to be applied on Management Service <oms host> before deploying this plugin revision

Ensure that the patch 13426642 is already applied to the Plug-in home.

Apply the following patch to OMS:

Patch 13426571 - TRACKING BUG:N-APPLY PATCH FOR PLATFORM OMS HOME FOR BUCKET 2- AIX, HP-IA, HP-PA

Perform the Plug-in deployment to the Management Server.

Deploying database plugin to Management Server, getting the following error:

 "java.io.FileNotFoundException: OMS archive not found in ../Middleware/oms/sysman/install/plugins/oracle.sysman.db/12.1.0.1.0 for plugin" 
Need to download the latest version of database plugin to the local EM store as mentioned in stepa above and then deploy the plugin to Management Server.

"Error occurred while deploying AGENT plug-in: null" Is returned When Adding Target in Enterprise Manager Cloud Control 12C

Modified 27-JUL-2012

Applies to

Enterprise Manager for Oracle Database - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Symptoms

When adding a target in Enterprise Manager Cloud Control 12c, the errors listed below have been reported. In all cases, it does not matter how the target is being added (either from "autodiscovery results"/non-host targets/promote target or from trying to add the target manually).
The error is returned on the "add target" screen.

This error is reported when trying to add database target:-

oracle.sysman.emSDK.core.extensibility.plugin.deployment.EMPluginDeploymentException: Error occurred while deploying AGENT plug-in: null

This error has been reported when adding a cluster target:-

Saving cluster Cluster01 to host machine1.uk.oracle.com....Error occurred while deploying AGENT plug-in:null
Saving cluster Cluster01 to host machine1.uk.oracle.com...Error occurred while deploying AGENT plug-in:null Finished

This error has been reported when trying to add a listener target:-

oracle.sysman.emSDK.core.extensibility.plugin.deployment.EMPluginDeploymentException: Error occurred while deploying AGENT plug-in: null

LISTENER_listenertest.oracle.com:
LISTENER_listenertest.oracle.com:Failed to add target - oracle.sysman.emSDK.core.extensibility.plugin.deployment.EMPluginDeploymentException: Error occurred while deploying AGENT plug-in: null - Error occurred while deploying AGENT plug-in: null

This error has been reported when trying to add an ASM target:-

+ASM_asmtest01.oracle.com +ASM_asmtest01.oracle.com: Failed to add target - oracle.sysman/emSDK.core.extensibility.plugin.deployment.EMPluginDeploymentException: Error occurred while deploying AGENT plug-in:null - Error occurred while deploying AGENT plug-in:null

Solution

The Operating System of the target (which fails to be added) is not certified.

1. To check whether the platform is supported:-

Go to certify:-

 https://support.us.oracle.com/oip/faces/secure/ml3/certify/SearchCertification.jspx?mc=true Choose Product: "Enterprise Manager Base Platform - Agent" Release: 12.1.0.1.0 Platform: <leave the default value of "any"> click on "search" button 

2. To check the OS release/version on the target machine:-

Connect to the machine of the target which cannot be added

 uname -a 
This will indicate whether 64 or 32 bit - eg.

Linux testmachine1 2.6.18-238.0.0.0.1.el5xen #1 SMP Tue Jan 4 09:38:01 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

Then to find out the release:-

 For OEL (Oracle Enterprise Linux) cat /etc/enterprise-release For RHEL (Red Hat Enterprise Linux) cat /etc/redhat-release for SLES (Suse Linux Enterprise Server) cat /etc/SuSE-release 

How To Manually Add Database Targets in 12C Cloud Control

Modified 31-JAN-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 and later [Release: 12.1 and later ]
Information in this document applies to any platform.

Purpose

This document describes different methods to manually add a database target in 12C Cloud Control.

Scope and Application

This document is applicable for the 12C Cloud Control administrators monitoring database targets.

How To Manually Add Database Targets in 12C Cloud Control

There are different ways of adding database target in 12C Cloud control:

1. Configuring Automatic Discovery

Database target can be manually added in two ways:

1. Login to 12C Cloud Control
2. Navigate to Setup → Add Target
3. Click on Add Targets Manually

There are two different options for adding non host target:

• Select Target Type "Oracle Database, Listener and Automatic Storage Management"
• Click on "Add Usuing Guided Discovery"
• In the page displayed, select the hostname and Click on "Continue"
• Displays a page with list of databases and listeners that has to be discovered
• Select the databases that needs to be monitored and Click on "Next"
• Finish the remaining steps and click on "Save"

2. Add Non-Host Targets by Specifying Target Monitoring Properties

• Select the Target Type "Database Instance"
• Select the Monitoring Agent
• Click on "Add Manually" button
• The page displays database configuration page with the parameters required for database discovery
• Fill in the required details and click on "OK"; The target will be added in 12C Cloud Control

1. Login to 12C Cloud Control
2. Navigate to Targets → Databases
4. In the page displayed, select the hostname and Click on "Continue"
5. Displays a page with list of databases and listeners that has to be discovered
6. Select the databases that needs to be monitored and Click on "Next"
7. Finish the remaining steps and click on "Save"; The target will be added in 12C Cloud Control

Known Issues

1. If all the required database configuration details are not provided then adding the database target will fail with "Validation Error" as shown below:

Solution:

Ensure that all the database configuration details are provided and then add the database target

2. If the database target is not reachable either if the database or listener is down, test connection to the database will fal with "The Network Adapter could not establish the connection"

Solution:

Ensure that the database and listener is up. Database is registered with istener.
Also verify that monitoring user can connect to database with the connect descriptor specified in error message.

For example:

3. If agent on the database machine is not running/reachable then following warnings will be displayed on the console:

1. <hostname> - unable to connect to the agent at <agent URL> [Connection refused]

2. <hostname> - unable to connect to the agent at <agent URL> [No route to host]

3. <hostname> - unable to connect to the agent at <agent URL> [Failed to agree on an SSL cipher suite]

4. <hostname> - unable to connect to the agent at <agent URL> [peer not authenticated]

5. <hostname> - agent key mismatch

Solution:

Ensure that the agent on database machine is up and running, upload is succesful.
Also verify the agent status in 12C Cloud Control, status should be up.

4. Database discovery will fail with following error as:

"oracle.sysman.emSDK.core.extensibility.plugin,deployment.EMPluginDeployment:Error occurred while deploying AGENT plug-in:null"

Solution:

Refer to our technical tip on this same page 45:

"Error occurred while deploying AGENT plug-in: null" Is returned When Adding Target in Enterprise Manager Cloud Control 12C"

5. Database is not getting displayed in the discovery phase

Solution:

Database is not added in /etc/oratab file. Add database to oratab file and perform the discovery.

6. Database discovery fails with following warning:
"Unable to find discovery home for plugin id:oracle.sysman.db"

Solution: Deploy the latest database plugin to OMS and Agent as per our technical tip:

"How to Deploy Latest Database Plugin to OMS and Agent in 12C Cloud Control" on page 44.

Once the latest plugin is deployed, add the database target as per the steps mentioned above.

How to Install the 12c Enterprise Manager Cloud Control EMCLI on a Windows PC

Modified 29-JAN-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Goal

How to install the Emcli (Enterprise Manager Command Line Interface) on a windows pc.
The following example downloads and installs the emcli to a windows pc.

Solution

The Enterprise Manager Command Line Interface (emcli) is a command line interface , which can be used to run certain commands in Enterprise Manager Cloud Control.
It is designed for Cloud Control Administrators who would like to run command line commands or scripting.
Emcli can be installed on the same machine as the OMS or a remote machine (which does not necessarily have to run any of the Cloud Control Components).
An emcli OMS extension runs in the OMS to accept the commands, this is automatically configured and needs no set up.

To install emcli client, the following is needed:

• java 1.6.0_25 or higher
• machine to install the emclient on (can be the same machine as OMS, or can be a completely separate machine. Supported OS are solaris, linux, hpux, tru64 or windows with ntfs)
• a connection to the Enterprise Manager 12c Console (in order to download the emclikit.jar file)

1. Make sure the windows pc has jdk 1.6.0_25 or higher. If not, the latest jdk can be installed from:-

• Select first option: "Java SE Development Kit 7u2"
• Choose platform, eg. Select:windows x86 platform

After the download has completed, install the jdk. By default it will be installed to c:\program files\java\jdk1.7_0_02

To test that it is working:-
• Launch the windows command prompt
cd to C:\Program Files\Java\jdk1.7.0_02\jre\bin>
• issue the command:- java -version

Example:

 C:\Program Files\Java\jdk1.7.0_02\jre\bin>java -version java version "1.7.0_02" Java(TM) SE Runtime Environment (build 1.7.0_02-b13) Java HotSpot(TM) Client VM (build 22.0-b10, mixed mode, sharing) 

2. Download the emcli client kit from 12c Cloud Control console, go to:-

• save it to a temporary directory - eg. c:\temp

3. Install the emcli as follows:-

At the command prompt, cd to the jre/bin directory where java was installed and run the command:-

java -jar emclikit.jar client -install_dir=<emcli client dir>

For example:-

 C:\Program Files\Java\jdk1.7.0_02\jre\bin>java -jar c:\temp\emclikit.jar client -install_dir=c:\temp\emclikit Oracle Enterprise Manager 12c Release 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved. EM CLI client-side install completed successfully. 

The emcli has been installed into a directory called c:\temp\emclikit

4. Setup the emcli using:-

For example:-

 emcli setup -url=https://machine1.uk.oracle.com:4900/em -username=sysman C:\Program Files\Java\jdk1.7.0_02\jre\bin>c:\temp\emclikit\emcli setup -url=https://machine1.uk.oracle.com:4900/em -username=sysman Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0 Copyright (c) 1996, 2011 Oracle Corporation and/or its affiliates. All rights reserved. Enter password Warning: This certificate has not been identified as trusted in the local trust store -------------------------------------- <some certificate information> -------------------------------------- Warning: This certificate could be compromised Warning: This certificate has not been identified as trusted in the local trust store -------------------------------------- <some certificate information> -------------------------------------- Do you trust the certificate chain? [yes/no] yes Emcli setup successful 

5. Now test a command:-

1. query the setup info:-

C:\Program Files\Java\jdk1.7.0_02\jre\bin>c:\temp\emclikit\emcli setup

 Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0. Copyright (c) 1996, 2011 Oracle Corporation and/or its affiliates. All rights reserved. Instance Home : C:\Documents and Settings\rachel\.emcli EM URL : https://machine1.uk.oracle.com:4900/em EM user : sysman Trust all certificates : false 
2. run a simple metric collection:-

emcli collect metric -target_type=<target_type> -target_name=<target_name> -metric_name=<metric_name>

For example:-

 C:\Program Files\Java\jdk1.7.0_02\jre\bin>c:\temp\emclikit\emcli collect_metric -target_type=host -target_name=machine1 -metric_name=Load Metric Load was collected at repository successfully. 
Emcli commands and further reference (including unix instructions) can be found here

12c Cloud Control: Understanding OMS Process Control (Start/Stop/Status)

Modified 13-APR-2012

Applies to

Enterprise Manager Base Platform - Version: 12.1.0.1.0 to 12.1.0.1.0 - Release: 12.1 to 12.1
Information in this document applies to any platform.

Purpose

This document provides details about the commands that can be used for controlling (start/stop/status) the 12c OMS and the OS level processes which get started.

For details of the directory structure and useful locations in the 12c OMS home, refer to our technical tip "12c Cloud Control: Details of the Directory Structure and Commonly Used Locations in a 12c OMS Installation" on page 19.

Scope and Application

12c Cloud Control Administrators who need to understand the steps involved in the OMS process control.

Pre-requisites

The 12c Enterprise Manager Cloud Control setup consists of these components:

• Oracle Management Services (OMS)
• Oracle Management (Cloud Control) Repository, hosted in a certified Oracle Database.
• Oracle Management (Cloud) Agent, installed on the target machines (including the OMS machine as well), which need to be monitored from Grid Console.

Before starting the 12c OMS:

• The Repository Database and the Listener servicing this database should be up and running.
• These user accounts: sysman, sysman_opss, sysman_mds in the Repository database should be open:

•  select username, account_status from dba_users where username in ('SYSMAN','SYSMAN_MDS','SYSMAN_OPSS'); USERNAME ACCOUNT_STATUS --------------- --------------- SYSMAN OPEN SYSMAN_OPSS OPEN SYSMAN_MDS OPEN 
• The passwords for the sysman, sysman_opss, sysman_mds should not have been manually modified in the Repository database.

Any password change should be performed via the emctl utility using the steps in our technical tip
"12c Cloud Control: How to Modify the Password for SYSMAN and other Enterprise Manager Users at the OMS Level and Repository Database?" on page 28

• If the Repository Database is running on a remote machine:
• From the OMS machine, the hostname lookup must succeed and IP address of the database machine must be reachable.
• The two-way communication between OMS machine to Database Listener port should be open (ie. not blocked by any kind of firewall).
• The OMS should be started by the same OS user who has installed the Cloud Control software in the machine.

Starting, Stopping, and Checking the Status of the 12c OMS

The OMS must be started using the Enterprise Manager Control utility - emctl located in <OMS_HOME>/bin. emctl is a command-line utility to start, stop, or check the status of the EM components. It is a shell script which sets all the right environment (to launch perl, java, etc.) for starting the OMS and then invokes the emctl.pl script with appropriate arguments (start/stop/status).

Set these environment variables:

 export ORACLE_HOME=<path of the 12c OMS installation> export PATH=$ORACLE_HOME/bin:$PATH 
Or navigate to the <OMS_HOME>/bin directory and execute emctl:

 cd <OMS_HOME>/bin ./emctl start|stop|status oms 
Command Options

Usage and Typical Output Description
emctl start oms

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server Successfully Started
Oracle Management Server is Up
• Starts the opmn and HTTP_SERVER (OHS), if not already up.
• Starts the Node Manager, if not running.
• Checks if the Admin Server is configured on the machine. If yes, starts the Admin Server, if already not up.
• Starts the Managed server (EMGC_OMSn) through the Node Manager.

• Note:

• The components started by this command depend on the command which was earlier used to stop the OMS components: 'emctl stop oms' OR 'emctl stop oms -all'.
• emctl utility is capable of recognizing which components have to be started and no user intervention required.
• Value of the below parameter in the <EM_INSTANCE_BASE>/em/EMGC_OMS1/emgc.properties file is used to determine whether the Admin Server is on the local machine or not.

• The 12c EM Managed Server has two applications: emgc and empbs. Both of these should start up successfully for the OMS startup to be successful and to be shown as Up in the command output.
emctl status oms

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
WebTier is Up
Oracle Management Server is Up
Checks the status of the Webtier (http_server, opmn) and the OMS and provides the current status. OMS is shown as up only if both the emgc and empbs applications are accessible.
emctl status oms -details

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host : omsmachine.domain
HTTP Console Port : 7789
HTTPS Console Port : 7801
OMS is not configured with SLB or virtual hostname
OMS Console is locked.
Active CA ID: 1
Console URL: https://omsmachine.domain:7801/em

WLS Domain Information
Domain Name : GCDomain

Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: omsmachine.domain
This command is very useful in checking details of the ports and secure access for the Agent upload and console access:

• Prompts for the SYSMAN password.
• Provides details of the HTTP (s) Console ports and the Agent Upload Ports.
• Reports whether OMS is configured to an SLB / Virtual hostname or not.
• Reports whether the Agent Upload and Console Access are locked in secure mode or not.
• Reports the number of active Certificate Authorities in the Repository.
• Displays the actual Console and Upload URL's for the Cloud Control setup.
• Provides the WLS Domain name and Admin Server machine name
• Provides details about the EM Managed Server.

NOTE: This command can be executed even if the OMS is down, to view the setup details.

emctl stop oms

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
• Stops the Webtier (http_server, opmn)
• Stops the EM Managed server.

NOTE: This command does not stop the Node Manager and the Admin Server.

emctl stop oms -all

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
• Stops the Webtier (http_server, opmn)
• Stops the EM Managed server.
• Stops the Admin server, if the OMS is running on Admin Server host.
• Stops the Node Manager.
• Also stops the ADP Manager, JVMD Manager and BIP processes, if running.
emctl stop oms -force

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
• Kills all the processes forcefully, should be used only when the normal stop does not work.
• Can be used with/without -all option
emctl list oms

Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
OMS Instance(s) associated with current Oracle Home:
EMGC_OMS1*
• Provides the EM Managed Server name configured in that local ORACLE home.
• If the Admin server is configured on the same host, it displays a '*' next to OMS name e.g. EMGC_OMS1*.

NOTE:

1. emctl can control/manage local processes on that machine only, there is no option to start/stop an OMS on a remote machine.

2. Any additional products that are installed with 12c Cloud Console like the ADP Manager, JVMD Manager, BIP need to be started manually using the WLS Admin Console.

3. There is no need to modify or execute any Weblogic scripts such as startManagedWebLogic.sh and startWebLogic.sh for starting up the components as all the necessary components are started by the 'emctl start oms' itself.

4. It is also not necessary to set any WLS related environment variables.

5. If any environment or Java related parameters need to be set for the OMS, they can be added to the <EM_INSTANCE_BASE>/user_projects/domains/GCDomain/servers/bin/startEMServer.sh file.

6. The 12c OMS installation configures a new EM specific nodemanager at <EM_INSTANCE_BASE>/em/NodeManager/emnodemanager. Hence, the default nodemanager at the <MW_HOME>/wlserver_10.3/common/nodemanager should not be configured/started.

7. The <EM_INSTANCE_BASE>/em/NodeManager/emnodemanager/nodemanager.properties has the following set:

StartScriptName=startEMServer.sh
StartScriptEnabled=true

These properties should not be modified.

8. The emctl script controls the Webtier (opmn and http_server) components by using the opmnctl utility.

- It starts the Webtier using:

<EM_INSTANCE_BASE>/WebTierIH1/bin/opmnctl startall

This command can be executed independently at the command line if only the Webtier needs to be started.

- To stop only the Webtier:

<EM_INSTANCE_BASE>/WebTierIH1/bin/opmnctl stopall

- To check the Webtier status:
<EM_INSTANCE_BASE>/WebTierIH1/bin/opmnctl status

Processes in Instance: instance1
---------------------------------+--------------------+---------+---------
ias-component | process-type | pid | status
---------------------------------+--------------------+---------+---------
ohs1 | OHS | 28197 | Alive

List of Operating System Processes Started by 12c OMS

There are two main processes: opmn and Java (NodeManager) both with a ppid of 1. These start other processes as listed below:

1. opmn
||
opmn
||
|| ---> httpd.worker (for HTTP server)

• The opmn process spawns another opmn process which is responsible for starting and monitoring the HTTP server. Can be viewed on Unix using:

$ps -aef | grep -i opmn • One process for HTTP_Server: <MW_HOME>/Oracle_WT/ohs/bin/httpd.worker -DSSL This in turn spawns many other 'httpd.worker -DSSL', rotatelogs, odl_rotatelogs processes as needed. Can be viewed on Unix using:$ ps -aef | grep -i ohs

2. Java (Nodemanager)
||
||
||---> startEMServer.sh
||                                    ||
||                                    || ---> startWebLogic.sh
||                                                                        ||
||
||---> startEMServer.sh
||
|| ---> startWebLogic.sh
||
|| ---> EMGC_OMSn

• The NodeManager java process is responsible for starting and monitoring the Admin Server and the EM Managed Server. Can be viewed on Unix using:

$ps -aef | grep -i NodeManager • This further spawns two shells, each running the startEMServer.sh script. • Both the shells further invoke startWebLogic.sh script. • One of the shell running startWebLogic.sh script starts the EMGC_ADMINSERVER. This is present only on the primary OMS machine. Can be viewed on Unix using:$ ps -aef | grep EMGC_ADMINSERVER

• The other shell running the startWebLogic.sh script starts the EM Managed Server. Can be viewed on Unix using:

$ps -aef | grep EMGC_OMS • You can also view the entire list of processes using other OS tools like:$ pstree -capG <OS user>

where <OS user> is the Operating system user who has installed and started the OMS.

Log / Trace files Generated during the 12c OMS Startup

The following log/trace files are updated during the OMS startup:

Component Files Location
emctl emctl.log <EM_INSTANCE_BASE>/em/EMGC_OMSn/sysman/log
OPMN opmn.log <EM_INSTANCE_BASE>/WebTierIH1/diagnostics/logs/OPMN/opmn
HTTP_SERVER access_log, console~OHS~1.log <EM_INSTANCE_BASE>/WebTierIH1/diagnostics/logs/OHS/ohs1
EM Node Manager (only, if the node manager is started) nodemanager.log <EM_INSTANCE_BASE>/NodeManager/emnodemanager
EM Managed Server EMGC_OMS1.out, EMGC_OMS1.log <EM_INSTANCE_BASE>/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs

To troubleshoot the OMS startup/shutdown, emctl.log file is the starting point to look at. Based on each component that is being started, you can then look at the Webtier, Nodemanager, Admin server and EM Managed server logs.

Understanding and Configuring Software Library In 12C Cloud Control

Modified 19-JUN-2012

Applies to

Enterprise Manager Base Platform - Version 12.1.0.1.0 and later
Information in this document applies to any platform.

Purpose

This note is to help in understanding , configuring and verifying software library in 12c Cloud Control.

Scope

Oracle Software Library (Software Library) is one of the core features offered by Enterprise Manager Cloud Control. Technically, it is a repository that stores software entities such as software patches, virtual appliance images, reference gold images, application software, and their associated directive scripts. The software library stores it's repository on a file system accessible by the OMS

Details

Pre-requisistes:

1. Depending on the features or software required for provisioning and patching, you must decide and configure the Software Library storage space.A minimum storage of 2 GB is required to store all the Oracle shipped files , but going forward it is advisable to have approximately 50gb space allotted since this is our centralized storage.
2. On UNIX systems, Oracle recommends that you configure at least one OMS Shared File System location that is accessible from all the OMS hosts.On Windows systems, Oracle recommends configuring OMS Agent File System storage.
3. Each OMS host must have a preferred normal host credential set before configuring the location. For OMS Agent File System location configuration, a credential (preferred or named) has to be specified.
4. The user configuring the Software Library must have view privilege on all the OMS, and the agent targets running on the host machine. As per the accessibility verification, you must be able to view and edit these newly configured locations using the credentials described in point 4.
5. All the credentials used while configuring the locations, must be viewable by all the Software Library designers, as this will enable the designers to upload the files to the Software Library storage. Additionally, designers can also stage the uploaded files to other hosts for provisioning or patching activities.
6. In addition to local file system additional supported file systems : NFS, OCFS2 and ACFS

Software library storage:

A storage location in software library represents a repository of files. These files are either uploaded by software library , or generated and saved by some user-owned process. To start using software library , you must add at least one upload file storage location (OMS Shared location or OMS Agent location)

Available types of storage locations:

2. Referenced File Location

Upload File Locations support two storage options:

1. OMS Shared File System
2. OMS Agent File System

Steps to Configure OMS Shared File System Storage Location:

1. In Cloud Control, from Setup menu, select Provisioning and Patching and then, click Software Library.
2. On Software Library: Actions-->Administration page, select OMS Shared Filesystem.
4. In the Add OMS Shared File System location dialog box, provide a unique name, and location on the OMS host, where you want to set up the upload location.

5. Click on OK , configuring upload location for the first time, a metadata registration job is submitted as shown below

NOTE: Ensure that the configured storage location is a shared location that is accessible by all the OMS instances. For a Multi OMS setup, set the Normal Preferred Credentials for all the OMS(s).

When you configure an upload location for the first time, a metadata registration job is submitted which imports all the metadata information of all the installed plugins from the Oracle home of the OMS.

To track the progress of the job, from Enterprise menu, select Job, and then click Activity. On the Job Activity Page, in the Advanced Search region, enter the name of the job, choose Targetless as the Target Type, and then click Search. Typically, the name of the job starts with SWLIBREGISTERMETADATA_*.

Steps to Configure OMS Agent Storage Location:

1. In Cloud Control, from setup menu, select Provisioning and Patching, and then click Software Library.
2. On Software Library:Actions → Administration page, select OMS Agent Filesystem.
3. Click +Add, in the Add OMS Agent File System Location dialog box, enter the following details:

• In the Name field, enter a unique name for the storage.
• In the Host field, click the magnifier icon. From the Search and Select: Hosts dialog box, select a host where the OMS is running, and click Select. For example, xyz.acme.com
• In the Location field, click the magnifier icon. In the Remote File Browser dialog box, click Login As to log into the host machine with either Preferred, Named or New credentials. Navigate to the location on the host where you want to create the Agent File System, and click OK.

4. The selected credential is saved along with the host and selected file system path. The saved credential is used to upload files and stage the uploaded files to a host target as part of some provisioning or patching activity.

Click OK to create a new entry for the storage location in the table

Common Issues:

1. If software library is not configured in 12C cloud control we may encounter any of the following warning messages while navigating different pages of 12C cloud control

Case A.

WARNING: Software Library does not have an upload file location configured. At least one upload file location should be configured. You can configure the file location from Setup > Provisioning and Patching > Software Library

Case B.

Software Library setup is incomplete. Deployment Procedures will not work if the Software Library is not configured properly. To configure the Software Library, go to Grid Menu and navigate to Provisioning and Patching and then Software Library. Then add a location from Administration in the Actions menu. For more information on using the Software Library, review the online help or refer to the Chapter "Using a Software Library" in the Oracle Enterprise Manager Administrator's Guide for Software and Server Provisioning and Patching Guide.

2. If the preferred credentials are not configured , the following error message is displayed.

"Understanding Host Credentials in 12C Cloud Control" on page 41 to configure preferred credentials.

Verifying the software Library:

When ever the software library is being configured a background job gets submitted , the following steps helps to know whether the software library job is executed successfully or not:

• Click on Enterprise button → Job → Activity → click on advanced search
• In Name section enter %SWLIBREGISTERMETADATA%
• In status section select the option of all for status section (in drop down)
• In scheduled Start section select all from the drop down
• In target type section select "targetless" from the drop down
• Click on 'Go'.

Ideally you will get a job with name something like SWLIBREGISTERMETADATA_xxxxxxxx
If this job status should show succeeded it implies the software library is configured properly.

Database 11g: Quick Steps to Package and Send Critical Error Diagnostic Information to Support [Video]

Modified 20-DEC-2012

Applies to

Oracle Server - Enterprise Edition - Version 11.1.0.6 and later
Information in this document applies to any platform.

Goal

How to Run SQL Testcase Builder from ADRCI.

Starting with Oracle11g, alert and trace information is generated in a new format and stored in the Automatic Diagnostic Repository (ADR).

Refer also to our technical tip:

"11g Understanding Automatic Diagnostic Repository" on page 6 to know the ADR in detail.

This article explains how to find and send the alert.log and relevant trace files to Oracle Global Software Support when a critical error occurs in database, using the ADRCI tool.

Fix

Oracle11g provides tools and methods to automatically identify and gather trace files related to a problem (a critical error in the database environment) or an incident (an occurrence of a problem).
This facility is known as the Incident Packaging Services (IPS). The Support Workbench (GUI) and ADRCI (command line) utilities are the two interfaces to the Incident Packaging Services.
Every occurrence of a critical error (problem) in the database creates an incident.

Steps in identifying and sending files to support.

1. You are notified of a critical error reported in the database environment:

 SQL> select * from atab; select * from atab * ERROR at line 1: ORA-01578: ORACLE data block corrupted (file # 6, block # 11) ORA-01110: data file 6: '/opt/oracle/oradata/db11g/tt.dbf' 
2. Check for the error reported in the instance alert file and in the ADR.

To check for the error reported in the ADR:

With the 11g environment being set, from the OS command prompt invoke the ADRCI utility:

 $adrci  Let ADRCI know which Oracle environment to investigate:  adrci> show home --show home will list all the ADR homes. Set the proper database ADR home for use with ADRCI. adrci> set homepath database_home  Check for the problems reported:  adrci> show problem ADR Home = /opt/oracle/diag/rdbms/db11g/db11g: ************************************************************************* PROBLEM_ID PROBLEM_KEY LAST_INCIDENT LASTINC_TIME ------------ -------------- --------------- --------------------------------- 1 ORA 1578 18104 2009-06-01 22:06:19.501207 +10:00 1 rows fetched  You can have multiple problems reported. Create a package for the one which needs to be investigated. 3. Check for the incidents reported for the problem which needs to be packaged Each occurrence of the problem is called an 'incident'. The instance alert file will show all the incidents reported. Each incident will be associated with a unique incident id in the ADR. From the ADRCI prompt, execute the SHOW INCIDENT command for the problem to be investigated, by specifying the associated problem key. You can get the problem key from the SHOW PROBLEM command:  adrci> show incident -p "problem_key='ORA 1578'" ADR Home = /opt/oracle/diag/rdbms/db11g/db11g: ************************************************************************* INCIDENT_ID PROBLEM_KEY CREATE_TIME --------------- ------------------------ ---------------------------------- 18147 ORA 1578 2009-06-01 22:02:08.805002 +10:00 1 rows fetched  You can see multiple incidents reported for the same problem. Note the incident number for which you need to create a package. 4. Package the trace files invoking IPS. Generate the package using the IPS PACK command and store the generated package in a directory of your choice. The example below shows the package for the incident will be created in the /tmp directory.  adrci> ips pack incident 18147 in /tmp Generated package 9 in file /tmp/ORA1578_20090602113045_COM_1.zip, mode complete  If at this step the ADRCI utility complains that the incident was flood-controlled and that no package can be generated for it, then instead of choosing the most recent incident to be packaged, choose the first incident that occurred after an instance startup. Examples of IPS PACK include:  ips pack problem 100 in /tmp --Generates the package for the problem id 100 in /tmp ips pack incident 6439 in /tmp --Generates the package for the incident id 6439 in /tmp ips pack problemkey "ORA 1578" --Generates the package for the problem with the problem_key 'ORA 1578' ips pack seconds 8 --Generates the package with the incidents occurred in last 8 seconds. ips pack time '2007-05-01 10:00:00.00' to '2007-05-01 23:00:00.00' --Generates the package with the incidents occurred between the times '2007-05-01 10:00:00.00' and '2007-05-01 23:00:00.00'  Upload the ZIP file generated to the Service Request you created on the My Oracle Support website . This file will include all the trace files, instance alert file and other diagnostic information for the critical error. Make sure that the ZIP file is uploaded WITHOUT any post processing (such as TAR). How to Run SQL Testcase Builder from ADRCI Modified 20-MAR-2012 Applies to Oracle Server - Enterprise Edition - Version: 11.1.0.6 to 11.2.0.1 - Release: 11.1 to 11.2 Information in this document applies to any platform. Goal How to Run SQL Testcase Builder from ADRCI [Video]. This note is for the DBAs who have read and understand ADR,ADRCI & IPS Packaging and SQL Testcase builder. See also our other technical tips: "11g Understanding Automatic Diagnostic Repository" on page 6 "11g Quick Steps to Package and Send Critical Error Diagnostic Information to Support" on this page. Solution 1. With Oracle environment set, invoke adrci 2.  Example: ]$ adrci ADRCI: Release 11.1.0.7.0 - Production on Thu Aug 12 08:49:00 2010 Copyright (c) 1982, 2007, Oracle. All rights reserved. ADR base = "/u01/Oracle" 
3. Set the ADR Home for your database using the command 'set homepath'

4.  Example: adrci> set homepath diag/rdbms/orcl11g/ORCL11G 
5. Execute 'show problem' and 'show incident' to check the errors reported in the database.

6.  Example: adrci> show problem ADR Home = /u01/Oracle/diag/rdbms/orcl11g/ORCL11G: ************************************************************************* PROBLEM_ID PROBLEM_KEY LAST_INCIDENT LASTINC_TIME ----------------------------------------------------------- 1 ORA 600 [qksdie - feature:QKSFM_CVM] 10979 2010-08-09 11:47:18.461403 +10:00 adrci> show incident ADR Home = /u01/Oracle/diag/rdbms/orcl11g/ORCL11G: ************************************************************************* INCIDENT_ID PROBLEM_KEY CREATE_TIME ----------------------------------------------------------- 10979 ORA 600 [qksdie - feature:QKSFM_CVM] 2010-08-09 11:47:18.461403 +10:00 10947 ORA 600 [qksdie - feature:QKSFM_CVM] 2010-08-09 10:41:02.855703 +10:00 
7. The command 'DDE SHOW ACTIONS' will show if there are any recommended actions for the incident reported in the database. Execute this command to check the recommended actions.

8.  Example: adrci> dde show actions --------------------------------------------------------- ACTION INFORMATION: INCIDENT_ID 10947 ACTION_NAME SQLTCB INVOCATION 1 STATUS Success SOURCE Recommended FLAGS 0 INVOCATION_TIME 2010-08-09 11:08:18.055121 +10:00 FINISH_TIME 2010-08-09 11:10:15.110794 +10:00 ---------------------------------------------------------- ---------------------------------------------------------- ACTION INFORMATION: INCIDENT_ID 10979 ACTION_NAME SQLTCB INVOCATION 1 STATUS Ready SOURCE Recommended FLAGS 0 INVOCATION_TIME FINISH_TIME ------------------------------------------------------------- 
Second Incident 10979 has the action status 'Ready' which means there is a recommended action SQLTCB ( SQL Testcase Builder) ready to be executed.

9. To run the action, the command is 'DDE EXECUTE ACTION'.

10.  Syntax: DDE EXECUTE ACTION INCIDENT <incident_id> ACTIONNAME <action_name> INVOCATION <invocation_id> 
The invocation ID will be 1 (one), unless the same action name has been recommended more than once for the same incident.

You can get the correct invocation id from the output of 'dde show actions'.

 Example: adrci> dde execute action incident 10979 actionname SQLTCB invocation 1 Action requires login Username: system Password: manager adrci> 
The action SQLTCB requires login to the database to run dbms_sqldiag package.

Running the 'dde show actions' after the execution of action, will show the action status 'SUCCESS' which means the recommended action is successfully run on the incident.

 Example: adrci> dde show actions ACTION INFORMATION: INCIDENT_ID 10979 ACTION_NAME SQLTCB INVOCATION 1 STATUS Success SOURCE Recommended FLAGS 0 INVOCATION_TIME 2010-08-09 11:55:01.172856 +10:00 FINISH_TIME 2010-08-09 11:56:12.464153 +10:00 
11. All the sql testcase files will be stored in the incident directory of the respective incident under ADR HOME.

12.  Example: ]$cd u01/Oracle/diag/rdbms/orcl11g/ORCL11G ]$ cd incident ]$cd incdir_10979  Along with incident dumps, there will be a directory created by name SQLTCB. Under this directory all the testcase files will be stored.  ]$ cd SQLTCB_1 ]\$ ls -lrt README.txt oratcb1_009A00A40001ol.xml oratcb1_009A00A40001sql.xml oratcb1_009A00A40001dpexp.sql oratcb1_009A00A40001dpexp.log oratcb1_009A00A40001dpexp.dmp oratcb1_009A00A40001xpls.sql oratcb1_009A00A40001ssimp.sql oratcb1_009A00A40001dpimp.sql oratcb1_009A00A40001xpl.txt oratcb1_009A00A40001xplo.sql oratcb1_009A00A40001xplf.sql oratcb1_009A00A40001main.xml 
13. These files will be included in the package created for this incident, along with incident dumps, trace files and alert.log.

14.  Example: adrci> ips pack incident 10979 in /tmp Generated package ORA600qks_20100809120430_COM_1.zip 
Package for incident 10979 is created in /tmp.

15. Upload the package created to Oracle Support to diagnose the issue.

Valencia, Spain
Phone : +34 667 60 77 33