We are current experiencing a problem whereby a job that executes on a AIX 5.3 with CTM Agent 6.1.03, appears to be successfully submitted by CTRLM Server, however although the job only executes for a few seconds, in EM the job continues to "run" for up to an hour or more.
This job has now been edited to only perform the following shell
command:
echo 1
When looking at the sysout, this appears to execute OK:
/usr/bmc/ctm/sysout:>cat F09ASROCHK3.LOG_00buoz_00002
+ /usr/local/bin/xi.sroCHK3.job F09A
1
/usr/bmc/ctm/sysout:>
Looking at the "status" the following is apparent:
/usr/bmc/ctm/status:>cat 0buoz_0002.dsect
DSECT_HEADER_KEY_VERSION 06
DSECT_HEADER_KEY_APPID CM
DSECT_HEADER_KEY_SID X
DSECT_HEADER_KEY_MID R
DSECT_HEADER_KEY_LAST T
DSECT_HEADER_KEY_CTMS_HOSTNAME MSHSRMSAPPC0026
DSECT_HEADER_KEY_TASK_CLASS
DSECT_HEADER_KEY_APPL_TYPE OS
DSECT_HEADER_KEY_APPL_VERSION
DSECT_TYPE 7
MEMNAME F09ASROCHK3
JOBNAME F09ASROCHK3
TASKTYPE C
CMDLINE /usr/local/bin/xi.sroCHK3.job 'F09A'
MEMLIB
OVERLIB
OWNER pnf1122
ORDERNO 0buoz
RUNNO 00002
DSECT_ON_COUNT 000
START_TIME 20080830150326
JOBSTATUS 5
REQTYPE TRACK
PROCID 389218
EXITSTATUS Y
EXITCODE 0
EXITMSG
SYSOUTSIZE 39
CPUUSED 0
ELAPSEDTIME 4
END_TIME 20080830162630
usr/bmc/ctm/status:>
BMC Support suggested that we run the command outside of CTRLM:
I have edited the mentioned script to now display the system time before and after performing the shell command, and you will notice that it completes in less than a second, so clearly the command is executed as expected. :
$
$ /usr/local/bin/xi.sroCHK3.job F09A
Wed Sep 3 06:54:56 GMT 2008
1
Wed Sep 3 06:54:56 GMT 2008
$
When then running the script from CTRLM, I can see the same sysout as above in the "current run" sysout, however the job is still running.
Further investigation would suggest that there might be a problem with Agent-to-Server comms. It would appear that all utilities that perform "server" actions such as:
Ctmvar
Ag_ping
Ag_comms_diag
all seem to hang, i.e. executes but does not return any output, not even failure.
In response, I set TCP/IP Timeout, Comms Trace and Timeout of Agent Utilities to lower values, in an attempt to induce at least an error message, as follows:
Agent Configuration Utility
1) Server-to-Agent Port Number . . . : [7006]
2) Agent-to-Server Port Number . . . : [7005]
3) TCP/IP Timeout. . . . . . . . . . : [30]
For items 4 and 5 do not use IP address
4) Primary CONTROL-M/Server Host . . : [MSHSRMSAPPC0026]
5) Authorized CONTROL-M/Server Hosts : [MSHSRMSAPPC0026]
6) Tracker Polling Interval. . . . . : [120]
7) Diagnostic Level. . . . . . . . . : [0]
AutoEdit Inline. . . . . . . (Y|N): [N]9) Comm Trace. . . . . .(0-OFF|1-ON) : [1]
10) CTMS Address Mode . . . . . (IP|) : []
11) Days To Retain Log Files. . . . . : [1]
12) Daily Log File Enabled. . . (Y|N): [Y]
13) Timeout for Agent utilities . . . : [30]
14) Locale. . . . . . . . . . . . . . : []
15) Common Event mechanism. . . . . . : [Y]
16) Tracker Event Port. . . . . . . . : [7016]
s) Save and Restart CONTROL-M/Agent Services
q) Quit
Comm traces reveal nothing spectacular:
Synchronising the agent from the CTM Server
$ cat trace_323798
0903 08:53:28
SIZE: a 91
0903 08:53:28
DATA: <<<0101022008090308581606CMT RT00000000094000000139Foodscate2.unix.<ourdomain>.com15MSHSRMSAPPC00260000160031T0000110001906290994C0bwsq00001>>>
0903 08:53:28
SIZE: a 007c
0903 08:53:28
DATA: <<<0100752008090308532806CMT>>>
Pinging agent from CTM Server
foodscate2:/usr/bmc/ctm/proclog >cat trace_319630
0903 08:58:27
SIZE: a 74
0903 08:58:27
DATA: <<<0101022008090309052006CMP RT00000100000100000139foodscate2.unix.<ourdomain>.com15MSHSRMSAPPC0026000009000200>>>
0903 08:58:27
SIZE: a 0051
0903 08:58:27
DATA: <<<0100752008090308582706CMP>>>
I would appreciate any help...
.