IKJ56602I COMMAND SYSTEM RESTARTING DUE TO ERROR

[Munken]

We recently switched from Zowe CLI to Zowe Java SDK.
We are using Zowe to execute multiple TSO commands in the same address space.
However, after switching we sporadically observe

IKJ56602I COMMAND SYSTEM RESTARTING DUE TO ERROR 

Looking at the logs on the mainframe, we are quite certain that the problem is related to this stacktrace

.ERROR   . CWWKO0801E: The SSL connection cannot be initialized from the 10.4.15.13 host and 50,443 port on the remote client to the 10.10.39.107 host and 65,5
.at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
.at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:383)
.at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
.at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
.at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:134)
.at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:737)
.at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:692)
.at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:507)
.at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:483)
.at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
.at com.ibm.ws.channel.ssl.internal.SSLUtils.handleHandshake(SSLUtils.java:910)
.at .internal classes.
Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
.at java.base/sun.security.ssl.SSLCipherÅT13GcmReadCipherGeneratorÅGcmReadCipher.decrypt(SSLCipher.java:1898)
.at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239)
.at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196)
.at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159)
.at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
.... 7 more

We have tried to upgrade from 7.0.0 to 7.0.3. However, the problem persists.

Our Zowe "driver"

```` @slf4j public class ZoweExecutor implements AutoCloseable {

public static final String READY = "READY";

private static final ObjectMapper MAPPER = new ObjectMapper();
private static final String TSO_DATA = "tsoData";
private static final String TSO_MESSAGE = "TSO MESSAGE";
private static final String TSO_PROMPT = "TSO PROMPT";
private static final String DATA = "DATA";
private static final String HIDDEN = "HIDDEN";
private static final String CODE_PAGE = "277";

private final String account;
private final TsoStart tsoStart;
private final TsoSend tsoSend;
private final TsoReply tsoReply;
private final TsoStop tsoStop;
private final TsoPing tsoPing;
private final String sessionId;

public ZoweExecutor(String host, String user, String password, String account, String port) {
    this(account, ZosConnectionFactory.createBasicConnection(host, Integer.parseInt(port), user, password));
}

private ZoweExecutor(String account, ZosConnection connection) {
    this(
            account,
            new TsoStart(connection),
            new TsoSend(connection),
            new TsoReply(connection),
            new TsoStop(connection),
            new TsoPing(connection));
}

ZoweExecutor(
        String account, TsoStart tsoStart, TsoSend tsoSend, TsoReply tsoReply, TsoStop tsoStop, TsoPing tsoPing) {
    this.account = account;
    this.tsoStart = tsoStart;
    this.tsoSend = tsoSend;
    this.tsoReply = tsoReply;
    this.tsoStop = tsoStop;
    this.tsoPing = tsoPing;
    this.sessionId = openAddressSpace();
}

public static ZoweExecutor fromConfig(ZoweConnectionConfigs.ZoweConnectionConfig config) {
    return new ZoweExecutor(config.host(), config.user(), config.password(), config.account(), config.port());
}

public void runCommandAndCheckReadyResponse(String command) {
    final var response = run(command);
    if (!isReadyResponse(response)) {
        throw new UnexpectedZoweResponse(command, READY, response);
    }
}

public void runCommandAndCheckResponseContains(String command, String expectedResponse) {
    final var response = run(command);
    if (!response.contains(expectedResponse)) {
        throw new UnexpectedZoweResponse(command, expectedResponse, response);
    }
}

public static boolean isReadyResponse(String response) {
    return READY.equals(response);
}

public String run(String command) {
    log.debug("Running TSO command: {}", command);
    final var commandResponse = sendAndCollect(command);
    log.debug("Command response: {}", commandResponse);
    return commandResponse;
}

@SneakyThrows
public boolean ping() {
    final TsoCommonResponse response = tsoPing.ping(sessionId);
    log.debug("Ping response: {}", response);
    return response != null && !Boolean.TRUE.equals(response.getTimeout());
}

@Retry(name = "zowe")
@SneakyThrows
private String sendAndCollect(String command) {
    final List<String> messages = new ArrayList<>();
    var response = tsoSend.sendCommand(sessionId, command);
    var promptReceived = collectMessages(response, messages);
    while (!promptReceived) {
        response = tsoReply.reply(sessionId);
        promptReceived = collectMessages(response, messages);
    }
    return String.join("\n", messages).trim();
}

@SneakyThrows
private boolean collectMessages(String responseStr, List<String> messages) {
    final JsonNode tsoData = MAPPER.readTree(responseStr).path(TSO_DATA);
    if (!tsoData.isArray()) {
        return false;
    }
    var promptReceived = false;
    for (final JsonNode item : tsoData) {
        final JsonNode message = item.get(TSO_MESSAGE);
        if (message != null && message.hasNonNull(DATA)) {
            messages.add(message.get(DATA).asText());
        }
        final JsonNode prompt = item.get(TSO_PROMPT);
        if (prompt != null && prompt.hasNonNull(HIDDEN)) {
            promptReceived = true;
        }
    }
    return promptReceived;
}

@Override
public void close() {
    try {
        stopAddressSpace();
    } catch (Exception e) {
        log.warn("Failed to close Zowe address space: {}", sessionId, e);
    }
}

@SneakyThrows
private String openAddressSpace() {
    final var input = new StartTsoInputData();
    input.setAccount(account);
    input.setCodePage(CODE_PAGE);
    final TsoStartResponse response = tsoStart.start(input);
    if (!response.isSuccess()) {
        throw new IllegalStateException("Failed to start TSO address space: " + response.getResponse());
    }
    return response.getSessionId();
}

@SneakyThrows
private void stopAddressSpace() {
    tsoStop.stop(sessionId);
}

}

</details>

[Munken]

I ran an LLM investigation. It claims

Is this something that you have encountered?

[frankgiordano]

I have never come across this error before. I have not seen any consumers use TsoSend and TsoReply separately. These methods can be used separately, but extra care is needed. Preferably, for TSO command requests, use the main TsoCmd class, which automates starting the session, issuing the command, handling tsoReply loops until PROMPT, and terminating the TSO session safely.

If tsoReply.reply throws a transient exception (such as a network glitch or HTTP socket timeout), Resilience4j @Retry re-invokes sendAndCollect. This sends tsoSend.sendCommand(...) again to the same active sessionId.

Because the TSO address space on z/OS was still mid-execution or expecting a reply request rather than a new sendCommand request, sending a new command out of sequence violates the z/OSMF TSO state machine. The TSO command processor detects an invalid state and restarts the session environment, raising: IKJ56602I COMMAND SYSTEM RESTARTING DUE TO ERROR.

Zowe Java SDK uses Unirest, which maintains a persistent HTTP connection pool and reuses TCP/TLS connections to WebSphere Liberty (z/OSMF). When a stream is interrupted while z/OSMF is still transmitting response bytes, unread data remains in the socket buffer or the TCP connection is closed abruptly.

When subsequent HTTP requests reuse the pooled connection or z/OSMF attempts to process remaining TLS record bytes after a dirty disconnect, the TLS record framing becomes desynchronized. Decryption fails at the GCM tag verification step, triggering WebSphere Liberty to log CWWKO0801E and terminate the SSL connection.

Zowe Java SDK operates inside a long-running JVM process with HTTP connection pooling and keep-alive connections. Retrying requests on an active TSO session over pooled connections exposes state and stream synchronization issues if network timeouts occur.

Ways to proceed:

Do not retry operations on an active sessionId Remove @Retry from sendAndCollect. Once a request fails on a TSO session, that session's state must be considered invalid. If you want to use @Retry, apply it at a higher level so each retry creates a fresh ZoweExecutor (and a fresh TSO session).

Use the SDK's built-in TsoCmd class (Recommended) Instead of managing TsoStart, TsoSend, TsoReply, and TsoStop manually, use TsoCmd. It automates session lifecycle management, handles the reply loop until a PROMPT is received, and closes the session safely:

TsoCmd tsoCmd = new TsoCmd(connection, account);
List output = tsoCmd.issueCommand("YOUR COMMAND");

Ensure strict single-threading per sessionId. A TSO address space is strictly single-threaded. Ensure each thread uses its own dedicated ZoweExecutor instance / sessionId. If an instance must be shared across threads, synchronize access to run(...).

HTTP Socket Read Timeout Configuration If long-running TSO commands trigger socket timeouts, timeouts should be increased. Currently, the SDK encapsulates its internal HTTP client, so timeouts cannot be modified via ZosConnection. I may be adding a configurable timeout property feature to the SDK project in an upcoming release to support this directly.

[frankgiordano]

I believe the your LLM analysis aligns with what I discussed and explains the exact mechanics behind the SSL error:

The 2-byte fragment: The stack trace specifically mentions fragment (2), which is a 2-byte TLS 1.3 KeyUpdate message sent by Java to rotate encryption keys on long-lived connections.

When WebSphere Liberty mistimes this message with incoming request data, it fails decryption and logs CWWKO0801E.

In Zowe CLI, it uses short-lived processes that close connections before key rotations happen. The Zowe Java SDK runs in a long-lived JVM where connections stay open longer, allowing key updates and connection pool reuse to occur.

How to address it alongside the TSO retry fix:

Fix the @Retry scope or use TsoCmd (as mentioned in my previous message) so requests aren't interrupted mid-stream on active TSO sessions or move the @Retry to runCommandAndCheckReadyResponse method.

Force TLS 1.2 if needed: If the TLS 1.3 key update issue persists on long-lived connections, you can pass -Djdk.tls.client.protocols=TLSv1.2 to the JVM to bypass TLS 1.3 KeyUpdate messages entirely.

[Munken]

Thanks for taking the time to reply to this Frank.

I double checked the code and the @Retry is not actually active since it is called within the same class.
I'll switch to the TsoCmd class to simplify.

Also note that the access is fully single threaded.

With regards to the TLS issue. As far as I can see we have the following options:

1. Downgrade to TLS 1.2 (-Djdk.tls.client.protocols=TLSv1.2 )
2. Disable connection pooling (-Djdk.httpclient.keepalive.timeout=0)

We will try option 1 first and see whether this have any effect.
However, it seems that option 2 is the long term solution.

[Munken]

Actually look at TsoCmd, the reason that we did not go with that is the overhead.
We are issuing > 20 TSO commands in rapid succession. As such we would like to avoid
the overhead of opening an address space for each command.

[frankgiordano]

I see. Yes then do as you are and try both options individually and see how that works. Let me know how they worked too. I am interested to hear.

Seeing that the Retry is not in play, the issue may be related to premature next command.

Ensure the loop strictly waits for the full command completion before sending the next tsoSend will prevent z/OSMF from restarting the address space.

When you are doing the TSO flow outside TsoCmd yourself it needs extra care.

In your code, if z/OSMF returns a prompt where HIDDEN is missing or null (for example, a prompt containing only "DATA": "READY"), promptReceived stays false.

Sending a tsoReply in this state may trigger 56602I COMMAND SYSTEM RESTARTING DUE TO ERROR

I would add the following in collectMessages in the beginning portion:

// 1. Check if z/OSMF returned an error/msgData response instead of tsoData
if (rootNode.has("msgData") && !rootNode.get("msgData").isNull()) {
log.warn("z/OSMF returned msgData: {}", responseStr);
// Treat error responses as complete so we don't keep sending tsoReply
return true;
}

Maybe with these changes you do not need the workaround options too.

Let me know how it goes. Regards

[frankgiordano]

any news?

[Munken]

We have been running with a Tls 1.2 downgrade since and have not seen the problem. But it is low frequency so nothing definitive.

[frankgiordano]

Okay. I think it may work with TLS 1.3 using the code changes I recommended. In either case, would you like to keep this issue open for further investigation, or can I close it for now?

[Munken]

We just hit this issue again.
It guess we have two options:

• switch to TsoCmd
• disable keepalive

[Munken]

Also worth noting did not see any errors on the zOSMF side this time. But I saw the same error on the zowe side.

Best long term fix would be switching to TsoCmd or update collectMessage

[frankgiordano]

I was able to reproduce the IKJ566021 error by sending a TsoReply when the Tso prompt completed and expected a TsoSend command next.

I suggest you do the following in your code. In collectMessages method should change that loop block to simply check for the existence of the prompt item itself, completely lifting the HIDDEN restriction

final JsonNode prompt = item.get(TSO_PROMPT);
if (prompt != null && !messages.isEmpty()) {
promptReceived = true;
}

It is possible that the JSON payload gets broken down into network packets and misses to see the HIDDEN chain response with initial Tso Ready data type response.

Due to network packet fragmentation or z/OSMF buffering behaviors, the initial TSO PROMPT data block (such as a standard READY state) can arrive in a payload without the HIDDEN field packaged inside it.

Because the Java loop checks only for the presence of the HIDDEN field, it misses this valid prompt signal. The code incorrectly assumes the mainframe isn't ready, skips past its collection logic, and sends a premature tsoReply.reply().

The official Zowe Node.js SDK implementation explicitly avoids this trap. It treats the presence of any valid TSO_PROMPT block (once data is accumulated) as the absolute marker that the operation is finished, regardless of whether inner attributes like HIDDEN are present in that exact payload segment.

I am going to my updating my TsoCmd implementation the same way.

[frankgiordano]

If the issue reoccurs with my suggestion, then the next step is to use disable keepalive configuration.

[Munken]

@frankgiordano wouldn’t it make sense that TsoCmd supported an adress space parameter and then reused this?

This is what it is possible with the zowe node cli.