CICS PROBLEM RESOLUTION

CICS PROBLEM RESOLUTION

[Updated 02/06/2012 Source at W:\actj2\htdocs\staff\lgilik\CicsProblem.html]

This page is intended to help with solving CICS problems in the UCSD ACT environment. Click on the condition that applies:

Need info about CICS regions
Performance issues
Transaction abends AEI0, AEY9, AICA, AICG, AKC3, APCT, ASRA, ATND, ATNI, BBCP, DSNC, -nnn etc.
No response for any user in the region (region hung).
No response for a single user only.
Logon/logoff problems.
CICS fails to shut down .

For other problems, try browser command Find and additional problem resolution steps .

List of CICS Regions

Production	Production	Training	QA	Test	Description
NET LPAR	PROD LPAR		DEV LPAR
n/a	CICFFISP	CICEFISE	CICQFISQ	CICTFIST	IFIS
n/a	CICSSISP	n/a	CICQSISQ	CICTSIST	ISIS
n/a	CICXXISP	n/a	CICQXISQ	CICTXIST	ISIS - Extension
n/a	CICNIVYP	n/a	CICNIVYQ	n/a	Ivory Server
CICNSECP	n/a	n/a	CICNSECQ	CICNSECT	For Passtickets
In any LPAR: CICITS1T, CICITS2T, CICITS3T, CICITS4T					Sys. Prog. Regions

Product Notes

CICS monitor

The easiest way to monitor CICS regions is by using the Mainview SUMMARY DISPLAY. In PROD01 ISPF, invoke Mainview by command =L.M, then enter option C4 (not on the menu), then option =P.1. In the top right corner, enter the region name, e.g. CICSSISP as shown below.

 BMC Software ------------ SUMMARY DISPLAY --------------------- TIME: 08:26:58
 COMMAND ===>                                                CICS ===> CICSSISP

   PERFORMANCE INDICATOR      TREND   CURRENT   CURRENT ENVIRONMENT
   TOTAL CPU PERCENTAGE        1.28      4.34
   USER CPU PERCENTAGE         1.21      4.91    TERMINALS
   PAGEIN  / SECOND RATE       0.00      0.00      SIGNED ON              248
   TRANSACTIONS / SECOND       0.80      1.98      PROCESSING               1
   TERMINAL TRANS. / SECOND    0.63      1.77      OUT OF SERVICE           0
   AVERAGE RESPONSE TIME       0.07      0.07      ACQUIRED               247
   FC I/O WAIT / SECOND        0.03      0.38
   PERCENT CDSA IN USE         6.41      6.40    FILES
   PERCENT UDSA IN USE         0.00      0.00      OPENED                   2
   PERCENT ECDSA IN USE        4.05      6.00      CLOSED                  15
   PERCENT EUDSA IN USE        0.32      0.40      DISABLED                 3
   MVS STORAGE < 16M          91.87     92.00      ENABLED                 14
   MVS STORAGE > 16M          37.97     37.90
            -PREVIOUS HIGH VALUE-RECENT HIGH-    TRANSACTIONS
   CPUREAL  08:10:16 S403   0.89  S486   0.81      EXECUTING NOW            4
   PAGING                    N/A          N/A      TOTAL EXECUTED       8,139
   RESPONSE 08:10:16 S403   4.75  S403   3.53      UNIQUE IDS USED        123
   I/O WAIT 08:10:16 S403   4.71  S403   3.51
   ***************************** END OF DATA ***********************************

If TOTAL CPU PERCENTAGE stays > 30%, use option =1.7 to doublecheck that there are no transaction numbers that persist and that use noticeable CPU. That would likely mean an infinite loop in the transaction. (Transactions like that should eventually automatically get killed by Mainview). Exception: Transactions JNL2, BCRT, and AAON stay in the system under normal circumstances.
PAGEIN should be between 0-1 page per second.
TRANSACTIONS / SECOND over 15 transactions/second probably means extremely heavy load on the system.
If the region is busy, TERMINAL TRANS. / SECOND should comprise majority of the TRANSACTIONS / SECOND.
Most important value to check is the AVERAGE RESPONSE TIME ! It is possible that the CURRENT value will spike, but TREND response time over 0.2 seconds is unusual. Response time trend over 0.5 seconds sounds like trouble.
FC I/O WAIT / SECOND may form major part of the AVERAGE RESPONSE TIME.
MVS STORAGE < 16M, and MVS STORAGE > 16M are difference between REGION value set for the CICS region in JCL, and between DSALIMIT(MVS<16M), or EDSALIMIT(MVS>16M) on DSAS DISPLAY - see below. They are used when CICS opens files, for Java, etc. The values there seem to be relatively stable.
To really see what is happenning with storage, instead of xxDSA IN USE on the SUMMARY DISPLAY, use DSAS DISPLAY, option =1.10 to check storage status. The EDSALIMIT over 80% in the PCT USED column should be an unusual occurence. DSALIMIT is tighter, but the use of the storage is less dependent on the load. xDSA storage areas are dynamically allocated up to the DSALIMIT value, and ExDSA storage areas are dynamically allocated up to the EDSALIMIT value, so xxDSA values nearing even 100% do not necessarily mean danger.
EDSALIMIT can be changed dynamically using CICS command CEMT SET EDSALIMIT. MVS>16M value will be diminished by the same amount.

 BMC Software ------------ DSAS DISPLAY ------------------------ TIME: 08:19:19
 COMMAND ===>                                                CICS ===> CICSSISP
                                    PCT    LARGEST  SOS  CUSHION     HWM    HWM
                  SIZE      USED   USED      FREE  COUNT   SIZE      USED   PCT
   CDSA           768K      464K   60.4      512K     0      64K     656K   85.4
   UDSA         1,024K        0K    0.0    2,048K     0      64K      64K    6.2
   SDSA           256K       36K   14.0      440K     0      64K      36K   14.0
   RDSA           256K      176K   68.7      160K     0      64K     176K   68.7

   ECDSA       40,960K   38,608K   94.2      944K     0     128K  38,648K   94.3
   EUDSA       77,824K    2,048K    2.6    8,192K     0       0K  77,824K  100.0
   ESDSA        1,024K      180K   17.5    1,008K     0     128K     516K   50.3
   ERDSA       44,032K   43,624K   99.0      360K     0     256K  43,624K   99.0

   DSALIMIT     7,168K      676K    9.4
   EDSALIMIT  655,360K   84,460K   12.8

   MVS <16M     9,128K    7,824K   85.7    1,300K
   MVS >16M  1770,496K  672,636K   37.9 1079,580K

   ***************************** END OF DATA ***********************************

For further info, such as which line commands to use, see operators CICS documentation and B&B Mainview help panels.

SYSVIEW

SYSVIEW shows JES2 output and other information. SYSVIEW is invoked from ISPF as option =S. For currently running regions use DA option, then select the appropriate CICS started task. In the CICS output you'll find the following DDs:

JESMSGLG, JESJCL, JESYSMSG: MVS messages and JCL
MSGUSR, DFHCXRF : CICS messages
CEEMSG,CEEOUT: LE/370 messages
S000nnn: Most likely the application reports sent to spool.

Fault Analyzer

Fault Analyzer is invoked from ISPF as an option =P.1. Fault Analyzer shows formatted transaction dumps, DB2 info, etc.

Problem: Performance issues

Use one of the following sources of information to find the bottleneck:

To get the current region status, use BMC Mainview monitor.
To get data about earlier transactions, use the monitor, history option, then point by cursor to the transaction and hit Enter to find which part of the transaction response time is the largest.
To get region and transaction statistics from earlier times in IBM format, use IBM supplied programs DFH$STUP,DFH$MOLS to access our SMF data. Examples of the JCL are in dataset SYSP.ZEKE.JCL(SYSPDHWM) and SYSLNG.JCL.CNTL(DFHMOLS) but use the CICS Operations and Utilities Guide to construct your job.
Daily job SYSP.ZEKE.JCL(SYSPDHWM) generates performance graphs at http://www-act.ucsd.edu/actonly/performance/cics/cicsstat.all.html. (Note: To get the html pages to the appropriate web server, FTP to 132.239.224.38 is used. The server runs on a "Paybase" PC located in the computer room. It stores all pages in directory W:\actj2\htdocs\actonly\performance\cics\*.html[.add], from which the web servers update the live pages. In case that the pages do not get updated, and the mainframe files CLP1.CICSSTAT.** contain the correct data, check that the "Paybase" FTP server is running and that the W: drive is working correctly.)

Problem: Transaction abends

Solution: Use Fault Analyzer to look at the dumps. Notable Abend codes are:

AEI0 - Unknown program. Programs are being automatically installed the first time they are requested. Thus the fact that PPT was not created indicates that the program is not in the loadlib, that it is not executable (as a result of compile or link errors) or that something else prevented the automatic install. Ivory remote definitions are an exception. These have to be created before the program is invoked by Ivory.
AEY9 - CICS/DB2 interface probably down. If DB2 is up then try DSNC STRT and look for possible error messages on CICS screen and using SYSVIEW.
AICA - program timed out. Probably loop which does not involve any EXEC CICS call.
AICG, AKC3, ATND, or ATNI NC-ACCESS or actual user disconnected a terminal session - this is currently a normal situation.
APCT - Similar as AEI0 above: something prevented the automatic install of a mapset or program.
ASRA - equivalent of S0C4, S0C7 etc.
BBCP - transaction abended by BMC Mainview monitor after consuming too many CPU cycles (for majority of transactions after 30 sec of CPU).
DSNC or -922 - Problem accessing DB2. See output of Fault analyzer. If RC=-922 with PLAN ACCESS then:
- Check that the failing transaction is set up to use the right plan.
- Check for any other problems in DB2 RDO setup.
- Check for any security problems, especially when -922 occurs without abend DSNC.
- Check with DBAs that the authid in question has been granted necessary privileges to access the plan and that the plan has been bound as necessary.
-nnn (sign and a three digit number) - probably an application generated abend. -nnn should correspond to the SQL return code.

If Fault Analyzer does not point out the problem, please perform additional problem resolution steps .

Problem: No user in the region gets any response

Solution: Try the following steps in sequence:

Use monitor option "CICS Tasks" to find what task is holding things up and cancel that task if necessary.
Use SYSVIEW to check region messages. If "Short on Storage" messages are constantly re-occurring, then add DSA using command CEMT SET DSALIMIT which takes immediate effect. Then update EDSALIM in CLP1.PARMLIB(CIC*) and RGN parm in SYS3.PROCLIB(CIC*) to make the change permanent.

If these steps do not resolve the problem, please perform additional problem resolution steps .

Problem: Single user gets no response from CICS

Solution: try the following steps in sequence:

Have user try Reset key on 3270 and retry requested action.
Use monitor option "CICS Tasks" to see if user's task is looping or waiting. Record the name of the application program that has control and any other requested information and then cancel that user's task.
Use monitor option "Terminals" to see that user's terminal and put that terminal out of service.

If these steps do not resolve the problem, please perform additional problem resolution steps .

Problems with logon or logoff

To logon into CICS, please select appropriate region from the NC-ACCESS menu. To logoff, enter CICS transaction BYE or CESF LOGOFF.

If user gets the "UCSD CICS" logo screen with the bottom line specifying USER-ID SIGNED ON. LOG OFF THE OTHER SESSION OR TRY AGAIN, this means that that user is logging on while their id still exists in the region. The Autooperator should then end the older session and user should be able to log on. If this is not the case, use monitor, Option "Terminals" to verify whether that userid is still logged on and if yes, then put the corresponding session out of service [line command 0].

If during logon some user gets a message Logon proceeding instead of proceeding to CICS then try the following steps in sequence:

Try to logon from the terminal in the back of the computer room. By entering e.g. "LOGON APPLID(CICSTEST) you can bypass NC-ACCESS and hopefully see messages indicating the problem.
When in the NC-ACCESS "Logon proceeding" screen, hit the 3270 "Reset" key (Esc on PC), then PA2. You should get the NC-ACCESS application menu. Enter H on the line corresponding to the problem region and hit Enter. Then enter line command S to see the error message.
Use SYSVIEW to look for error messages that start with AUTOIN2 or AUTOIN3.
Check all links in the chain, e.g. NC-Access, TCP/IP, VTAM, CICS, RACF.

Problem: CICS shutdown is unsuccessful

Startup and shutdown of CICS is normally performed automatically by ZEKE. If CICS shutdown fails then try the following steps in sequence:

Wait for 5 minutes after the shutdown is initiated. It is normal for the region to take some time waiting for active tasks to finish.
Use monitor option "CICS Tasks" to find what task is holding things up and cancel that task if necessary.
CEMT PERFORM SHUTDOWN IMMED may be tried at this point in time.

If these steps do not resolve the problem, please perform additional problem resolution steps .

Additional Problem Resolution Steps

For how to questions, use Google and the following information:

Collect messages.

Messages on user CICS screen.
Messages in JES2 output - use SYSVIEW.
COBOL messages - may be found in CICS Temporary storage queue CEBRtermid.

Use monitor to investigate problems that affect CICS at the moment.
Use Fault Analyzer to investigate any related transaction abends.
Use IPCS to investigate any related SVC dumps. See IBM manual "CICS Problem Determination Guide", chapter 5, "Dealing with CICS system abends" and chapter 15, "Using dumps in problem determination" for basic dump reading information.

verbexit dfhpd660 'APS=< TASKID=nnnn>'

If possible, check if shutdown and restart clears the problem (this will impact users!). Make sure that you collect error messages and other available data first. Continue with problem resolution after this test. Also, please note that some problems can be cleared only by a cold (or even initial) start.
If problem is still not resolved then collect all the information, and report the problem using IBM Link - ETR.

--- End of Document ---