Cloud Control collects a ton of metrics. With default thresholds applied and having 1000+ targets, there is the possibility of having false alerts being triggered. This isn’t necessarily a problem, but when you want to be notified of critical issues, it pays to optimize the metric settings.
I was triggered by the blog post EM12c- Managing Incidents, Stopping the Insanity, Part I  by @dbakevkar , which describes a quite similar case.
In my case I wanted to get rid of all bogus heap usage notifications by applying monitorig templates.
subject: EM Incident: Critical:New: - The heap usage is 91%. Host=weblogic.vanesch.org Target type=Oracle WebLogic Server Target name=/soa_domain/soa_domain/soa_server1 Incident creation time=Nov 26, 2013 3:05:25 PM CET Last updated time=Nov 26, 2013 3:05:25 PM CET Message=The heap usage is 91%. Severity=Critical Incident ID=22101 Event count=1 Incident Status=New Escalated=No Priority=None Incident owner= Incident Acknowledged By Owner=No Categories=Capacity Rule Name=Notification of critical alerts ECM targets,Email on incidents to Tony Rule Owner=TESCH Update Details: Incident created by rule (Name = Incident management rule set for all targets, Create incident for critical metric alerts [System generated rule]).
This is where monitoring templates come into play. Out of the box Oracle supplies a significant number of ‘certified’ monitoring templates, which are applied to newly discovered targets. Monitoring templates manage metrics per target type, in my case Weblogic server. The best way is to start with a copy of the default Oracle certified template and adjust it accordingly.
Make a copy of the relevant default template.
Check the default template checkbox to mark our custom template as the new default template.
Go to the Metric Thresholds tab and click the edit pencil for advanced settings.
Adjust the number of occurrences to prevent false alerts. I upped it from 2 to 4.
Next we need to apply this template to all of our discovered Weblogic Servers.
Now all current and future targets will be monitored with our custom monitoring template.