This post is to understand how to find the root cause when a target status is down on Oracle OEM (Oracle Enterprise Manager) – when in fact it is up.
What OEM reports
The targets are actually up. This is a new cluster that is being added to OEM.
Doing a Test Connection is successful. You can do this by selecting the Target and then Target Setup -> Monitoring Configuration. Ensure the form fields have the correct values, then press the Test Connection button.
How to figure out the actual issue
Use the EM agent to get an idea of what it sees and why it is reporting the target as down.
First, get the target name and target type for the target you need to check.
> emctl config agent listtargets |grep -i asm [ASMNET1LSNR_ASM_rac7db01.domain.com, oracle_listener] [+ASM_cluster7, osm_cluster] [+ASM1_rac7db01.domain.com, osm_instance]
Here, the target name is +ASM1_rac7db01.domain.com and target type is osm_instance
Next, get the status of the particular target along with this particular metric – ‘response’.
Use the target name and type from the previous output. Note, there should be no space after the comma (,) in your query.
> emctl status agent target +ASM1_rac7db01.domain.com,osm_instance|grep -i response Response Status n/a CRITICAL Fri Mar 19 16:12:32 GMT 2020
The current status of the ‘response’ metric is CRITICAL – relates to the target status being ‘Down’ in OEM.
EM Agent submits the following query to get the ‘response’ metric. If a ‘1’ is returned, then the ASM instance is considered to be up.
select 1,'' from v$instance;
In our case, we should not be getting a ‘1’. Let’s find the status and error that the EM Agent gets when trying to evaluate the ‘response’ metric – which is used to determine the status of the target.
> emctl getmetric agent +ASM1_rac7db01.domain.com,osm_instance,Response Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. Status,oraerr 0,Failed to connect: java.sql.SQLException: ORA-28040: No matching authentication protocol
The root cause of the problem, in this case, is the oraerr that is reported.
> oerr ora 28040 28040, 0000, "No matching authentication protocol" // *Cause: No acceptible authentication protocol for both client and server // *Action: Administrator should set SQLNET_ALLOWED_LOGON_VERSION parameter // on both client and servers to values that matches the minimum // version supported in the system.
This error occurs when the client (EM Agent) authentication protocol version is lower than the server’s (GI/ASM) authentication protocol. We have 12.5 OEM (client) and 18c GI/ASM (server).
The workaround, in this case, is to add the below parameters to the
sqlnet.ora file on the server. This is basically setting the authentication protocol version to allow for backward compatibility when a client with an older authentication protocol tries to connect to the server using the newer protocol. You can read more about the protocol enhancements in the references at the bottom of this post.
Your situation may be different but this post illustrates how we go about finding the root cause when a target status is down on Oracle OEM but the target is actually up. I hope this helps!
If you get a different output when checking the ‘response’ metric, leave a comment so we can understand other situations that could cause such issues.