Configuring and Tuning the Solstice CMIP Stack and
Enterprise Manager's CMIP MPA for Best Performance

By Mark R. Bowyer, Laurence Caullet, Milos Muzik, and Bruce Rothermal

(August, 2006)

1. Summary

This article explains the configuration options for both the Common Management Information Protocol (CMIP) stack and the Enterprise Manager CMIP Management Protocol Adapter (MPA), to assist in the tuning of these products for different performance criteria. The following descriptions are aimed at the most recent versions of these products, which at time of this writing were CMIP 8.2.1P12 or CMIP9.0P5, and Solstice Enterprise Manager (SEM) 3.0P11, SEM4.0P7 or SEM4.1P2. SEM5.0 was just being released as this was being published. Some of the maximum values given here were lower on previous patch releases of these products. Therefore, this document should be seen as an update to the documentation supplied with these products.

This article is not an attempt to explain what CMIP is or how it works. This is covered at some length in the documentation for these products. The intent here is just to cover all the available configuration options with respect to performance for both products and the Solaris Operating Environment (OE) kernel in one place, including the changes and additions made to them since the original product documentation was printed.

2. CMIP Stack Tunables

The following parameters can be found in /etc/opt/SUNWconn/cmip/cmip.conf, and can be set by using the /opt/SUNWconn/cmip/sbin/cmiptool configuration tool or by editing the file itself.

Maximum Number of Outstanding Operations

MAX_OUTSTANDING_OPS <integer>

This parameter defines the maximum number of outstanding asynchronous operations that the XMP library can handle simultaneously. This parameter overrides the value of the constant MP_MAX_OUTSTANDING_OPERATIONS, defined in the file xmp.h. This parameter can be set in the range of 10 to 1024. The default value is 20.

Maximum Number of Confirmed Indications

MAX_CONFIRMED_INDS <integer>

This parameter defines the maximum number of confirmed indications that the osimcsd daemon can handle simultaneously. Limiting the number of simultaneous confirmed indications limits the amount of memory required to store outstanding indications. However, if the osimcsd daemon is servicing a large number of applications, increasing the maximum number of confirmed indications reduces the amount of time for which indications are queued. This parameter can be set in the range of 1 to 1024. The default value is 20.

Maximum Number of Confirmed Requests

MAX_CONFIRMED_REQS <integer>

This parameter defines the maximum number of confirmed requests that the osimcsd daemon can handle simultaneously. Limiting the number of simultaneous confirmed requests limits the amount of memory required to store outstanding requests. However, if the osimcsd daemon is servicing a large number of applications, increasing the maximum number of confirmed requests reduces the amount of time for which requests are queued. This parameter can be set in the range of 1 to 1024. The default value is 20.

Inactivity Timer

INACTIVITY_TIMER <integer>

This parameter defines how long, in multiples of 10 seconds, an association can remain idle before it is closed automatically by the osimcsd daemon. If the inactivity timer is set to zero, the association remains open until it is explicitly closed. This parameter can be set in the range of 0 to 255. The default value is 12.

Request Timer

REQUEST_TIMER <integer>

This parameter defines how long, in multiples of 10 seconds, the osimcsd daemon will wait for a response to a confirmed request. If a response is not received before the request timer expires, the osimcsd daemon issues an error response to the requestor. If the response timer is set to zero, the osimcsd daemon will wait indefinitely for a response. For CMIP 8.2.1, this parameter can be set in the range of 0 to 127, with a default value of 5. For CMIP 9.0, the parameter can be set in the range of 0 to 255, with a default value of 10.

User Window Size

USER_WND_SIZE <integer>

This parameter defines the number of simultaneous interactions that the osimcsd daemon will handle for a single user application, before that user is blocked. The osimcsd daemon maintains a record of all outstanding interactions and refuses any interactions requested by a user that exceeds its window size. This imposes a form of flow control, which prevents a single user from dominating the resources of the osimcsd daemon. The user window size should not exceed the maximum number of confirmed requests. This parameter can be set in the range of 1 to 255. The default value is 20. For patch revisions prior to 111619-11, the limits were 1 to 20 and the default value was 5. The errors generated in the log files when the window size is exceeded looks like:

@(#) OSIMCSD MCS <CMIP patch number> User error : User credit (<type of request>)


where <type of request> may be:
	Abt-Rq:	User Abort Request
	Rel-Rq:	Release Request
	Ass-Rq:	Association Request
	OP-Rq:	Other Request (that is, different from the above)

Note that, in the case of a single CMIP user process (that is, PMI application in the case of Enterprise Manager) it would be quite possible to reach the user window size limit well before the other three maximum settings come into effect. The window in question here actually resides at the MCS layer; this is limiting the number of interactions that are being queued within the stack on that side of the connection, before they are sent over the association.

Non-blocking Flow Control

NONBLOCK_FLOW_CTRL ON|OFF

This parameter can be used to switch on the non-blocking flow control feature. When it is set to ON then user window flow control does not block. By default NONBLOCK_FLOW_CTRL is set to OFF. Both the feature and the NONBLOCK_FLOW_CTRL parameter have been introduced in CMIP patch 111619-11. In earlier versions the CMIP stack behaves as if NONBLOCK_FLOW_CTRL were set to OFF. The parameter NONBLOCK_FLOW_CTRL is not accessible from cmiptool, it has to be tuned manually in cmip.conf.

The non-blocking flow control feature affects these primitives/calls in asynchronous mode:

• Event-Report-Request / mp_event_report_req()
• Get-Request / mp_get_req()
• Set-Request / mp_set_req()
• Action-Request / mp_action_req()
• Create-Request / mp_create_req()
• Delete-Request / mp_delete_req()
• Cancel-Get-Request / mp_cancel_get_req()

When the user window in MCS layer becomes full (flow control is triggered) in asynchronous mode and NONBLOCK_FLOW_CTRL is set to ON, then an attempt to send one of the aforementioned request fails. The XMP API call returns immediately (it does not block) and the error returned is as follows:

	Error class :	MPPC_COMMUNICATIONS_ERROR
	Returned error :	MP_E_COMMUNICATIONS_PROBLEM
	Additional error :	MP_E_ADD_MCS_SEND_USER_CREDIT
	Other return values (result_return and invoke_id_return) are not valid.

The user application should resend requests (but not necessarily the same one which failed). When the flow control condition ends, then an API call will return with success and the request will be sent down the stack.

User window flow control is applied per a bind session. It means that when one association causes a flow control, then also all other associations within the same session are unable to send requests until the flow control is over. However, associations from other sessions can continue sending. Receiving incoming indications/confirmations is possible from all associations regardless of flow control.

Memory Pool Size

MEMORY_POOL_SIZE <integer>

This parameter defines the amount of shared memory in KBytes, allocated internally by the osimcsd daemon. The memory pool size limits the maximum number of concurrent associations allowed. Increasing the value of this parameter allows more associations to be created, but also increases the amount of memory used by Solstice CMIP. The default value of this parameter is chosen to maximize the number of associations allowed without incurring excessive memory usage. This parameter can be set in the range of 100 to 10240. The default value is 200. If MEMORY_POOL_SIZE is set to a large value, it may become necessary to increase the value set for shminfo_shmmax in /etc/system. The line looks like:
set shmsys:shminfo_shmmax=<value>

This is a kernel parameter that defines the maximum size of a shared segment allowed, and is already configured for the expected use of Enterprise Manager (EM) when EM is installed. If problems are seen with large MEMORY_POOL_SIZE values, shminfo_shmmax should be increased to match. The system has to be rebooted for the kernel to be able to implement this change.

Note that some of the memory-related problems with large MEMORY_POOL_SIZE values don't show up until after a reboot anyway. The manner in which shared memory is handled by osimcsd and other OSI deamons, such as FTAM, is very complex. One effect that crops up here is if you stop osimcsd, increase MEMORY_POOL_SIZE and then restart it, it will simply find and reuse the "old" shared memory segment it had before. Therefore, no problems will be seen with setting MEMORY_POOL_SIZE larger than the available shared memory. Only after a reboot, when no shared segments have been assigned, will it try to grab one of the size set and possibly report an error.

XMP MCS_Ready_Indication Timer

export XMP_MCS_READY_IND_TIMER=<integer>

Configuring this internal timer helps avoid window timeouts when the user application interactions have to be stored in the MCS layer due to a triggered network flow control or a very slow remote machine. This is the same window controlled by USER_WND_SIZE.
If XMP_MCS_READY_IND_TIMER is unset, it defaults to 60 seconds.
If XMP_MCS_READY_IND_TIMER is set < 60, it also defaults to 60 seconds.
If XMP_MCS_READY_IND_TIMER is set > 60, XMP sets it to that value.
If XMP_MCS_READY_IND_TIMER is set > 100000000, XMP sets it to 100000000.


This configuration variable should be set as an environment variable in the shell from which the XMP application is to be run, as shown above for Bourne or Korn shells. It is read from within the XMP library linked by the application at run time. Thus, to change this value for the em_cmip process of Enterprise Manager, you can add this line to the /etc/rc2.d/S98cmipmpa startup script. But be aware that this file may get replaced when upgrading or patching Enterprise Manager later.

Care should be taken when setting this value for em_cmip when running it as an Agent to a remote manager. Should this timer expire, a Session Abort signal is sent both to the remote manager, and back up to em_cmip. This causes the session to be closed, but since em_cmip is acting as an Agent, it can't reestablish the session, and all communication breaks down. Before SEM4.1P7 and CMIP9.0P5 a bug existed whereby when this happened, em_cmip would fall into a poll() loop and became unusable. These two patches contain fixes that cause em_cmip to exit on receiving a Session Abort, so that it can be restarted by a wrapper script (not provided by the SEM distribution) where necessary. This then allows the Manager application to reconnect to it and processing continues uneffected. However, it is still strongly recommended that this environment variable is set high enough that it doesn't time out in normal use.

3. CMIP MPA Tunables

The following tunables can be found in /opt/SUNWconn/em/build/acct/EM-config and are read during em_services startup. The default values for each release of Enterprise Manager can be seen in this file, and there is no upper limit.

CMIP MPA Platform Connect Timeout

EM_CMIP_MPA_PLATFORM_CONNECT_TIMEOUT: <integer>

When the CMIP MPA connects to the MIS using a Plaform::connect(), this value, in seconds, is used for the Timeout value. This is used by a ping-like keep-alive function between a Platform and the MIS, to tell if the MIS is still there and alive.

CMIP MPA Reconnect Poll Interval

EM_CMIP_MPA_RECONNECT_POLL_INTERVAL: <integer>

Should the connection between the CMIP MPA and MIS be lost, this value is used to set the time, in seconds, that the CMIP MPA will wait to reconnect so that it doesn't deplete resources by retrying too often.

MIS to MPA Request Timeout

EM_MIS_TO_MPA_REQ_TIMEOUT: <integer>

When a message is sent from the MIS to any MPA, this timer is the amount of time in seconds it will wait for a response before generating a timeout response back to the MIS. A value of 0 means wait indefinitely.

MIS to MPA Association Timeout

EM_MIS_TO_MPA_ASSOC_TIMEOUT: <integer>

This timer determines how long an association is allowed to sit idle before it is closed and resources are released between the MIS and any MPA. A value of 0 means wait indefinitely.

CMIP MPA Retry Control

SEM em_cmip also takes two arguements/parameters which effect the retrying of messages being sent:

EM_CMIP_MPA_ASSOC_RETRY_DELAY: <integer>
or
EM_CMIP [-retry_delay <milliseconds>]
and

EM_CMIP_MPA_ASSOC_MAX_RETRIES: <integer>
or
EM_CMIP [-max_retrys <max-retrys>]

By default, requests sent from em_cmip to the cmip stack will retry one time after 30 seconds of no response. If an error/response is received for the request within the 30 seconds, it dequeues and deletes the request sending the error/response back to the MIS. These default values can be changed using the preceding options.

Association Initiator in Agent Role

This new option has been added to SEM3.0 Patch 11 and SEM4.0 Patch 5, based on a customer RFE. The environment variable:

EM_CMIP_MPA_ASSOC_INITIATOR_IN_AGENT_ROLE: <"YES"|"NO">


which defaults to YES to continue normal operation, can be set to NO to allow the SEM em_cmip to work in a subordinate agent mode. When this option is thus enabled, the SEM system can not initiate the communication association to the manager. Any request to be sent out results in a lookup of the remote system. If an association is already established, the message can be sent. If it is not already established em_cmip will discard the message and provide a processing failure response for confirmed requests.


The following table covers the the complete set of CMIP MPA parameters available in SEM 5.0 for reference: