Deploy model goes into Unknown State

[gauravAggrahari]

Hi,
I am trying to deploy a model using openshift ai. I have created the model server successfully. But, when I deploy the model it goes into Unknown state and never comes to the normal state.

My cluster configuration is : 3 nodes * 8vCPU * 32 GM mem.

Could you please help us getting this resolved ?

Note: When we had only two nodes, the deploy model goes into critical state and complains that Insufficient Resources. Whereas, according to doc the resources we have are just fine.
Could you please conform why this is complaining about insufficient resources ?

[jiridanek]

Whereas, according to doc the resources we have are just fine.

Can you link the docs you are using?

My cluster configuration is : 3 nodes * 8vCPU * 32 GM mem.

What else besides the model server is running on the cluster? The docs cannot account for other admin-specified or user-specified clusters that are also running on the cluster. For example, if there is an HA deployment of Strimzi (https://strimzi.io/) running on your cluster, then it makes sense that there's not enough resources for OpenShift AI remaining, despite the cluster being able to run OpenShift AI alone.

deploy model goes into critical state and complains that Insufficient Resources

please provide complete logs, everything relevant that you discovered during your own investigations

Could you please confirm why this is complaining about insufficient resources ?

Is this single model serving or multi model serving? That is, does deploying a model result in creating new OpenShift resources on the cluster? If resources need to be created, then please investigate the specific resources and see why they aren't getting scheduled.

investigate how much available resources there is on your nodes, oc describe nodes should tell you how much more resource requests are your cluster nodes available to accomodate, for example, for each node you'd see something like this in the output

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                3170m (79%)   5200m (130%)
  memory             4404Mi (27%)  11398Mi (71%)
  ephemeral-storage  0 (0%)        0 (0%)
  hugepages-1Gi      0 (0%)        0 (0%)
  hugepages-2Mi      0 (0%)        0 (0%)
  hugepages-32Mi     0 (0%)        0 (0%)
  hugepages-64Ki     0 (0%)        0 (0%)

If you are using multimodel server, then investigate the relevant pod logs for a message like this (the example is something I saved for the fun of it, I don't have a deployment handy to provide more realistic listing to illustrate what I have in mind)

mm service failed, exiting
mm {"instant":{"epochSecond":1710686708,"nanoOfSecond":787936069},"thread":"ll-svc-events-3","level":"INFO","loggerName":"com.ibm.watson.litelinks.server.ProbeHttpServer","message":"Stopping litelinks health probe http server on port 8089","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":20,"threadPriority":5}
mm Wrote termination message to /dev/termination-log: "Service startup failed: Reported model capacity -0.125GiB too small relative to advertised default model size of 1MiB"
mm java.lang.IllegalStateException: Expected the service modelmesh-serving to be RUNNING, but the service has FAILED
mm     at com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:384)
mm     at com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:324)
mm     at com.ibm.watson.litelinks.server.LitelinksService.run(LitelinksService.java:699)
mm     at com.ibm.watson.litelinks.server.LitelinksService.launch(LitelinksService.java:144)
mm     at com.ibm.watson.litelinks.server.LitelinksService.main(LitelinksService.java:108)
mm Caused by: java.lang.Exception: Reported model capacity -0.125GiB too small relative to advertised default model size of 1MiB
mm     at com.ibm.watson.modelmesh.ModelMesh.setupMemory(ModelMesh.java:1275)
mm     at com.ibm.watson.modelmesh.ModelMesh.initialize(ModelMesh.java:735)
mm     at com.ibm.watson.litelinks.server.ThriftService.doStart(ThriftService.java:123)
mm     at com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:252)
mm     at com.ibm.watson.litelinks.server.DefaultThriftServer.lambda$doStart$0(DefaultThriftServer.java:390)
mm     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
mm     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
mm     at java.base/java.lang.Thread.run(Thread.java:840)
mm     at com.ibm.watson.litelinks.ThreadPoolHelper$3$1.run(ThreadPoolHelper.java:91)
Stream closed EOF for modelserving-inference-test/modelmesh-serving-some-serving-runtime-599dcdf75c-5phnc (mm)

/label community-support/provided status/waiting-for-community-member

[gauravAggrahari]

Thank you @jiridanek for quick response.

• We dont have any other workload running on the openshift cluster
• As per the above screenshot its multi-model server.
• Can you help me with resources where I can look for more information ? That is, what pod logs do you want me to look into ?
• I am also interested in know why this is going in Unknown State.
• Can we get more info on the Openshift AI page itself. Provided, most of the users for openshift ai are Data Scientists or AI developers who may not have much idea on the openshift.

[SugandhaAgrawal]

@jiridanek

1. We are following this tutorial https://docs.redhat.com/en/documentation/red_hat_openshift_ai_cloud_service/1/html-single/openshift_ai_tutorial_-_fraud_detection_example/index#introduction-tutorial

2. oc describe nodes result

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests          Limits
  --------           --------          ------
  cpu                6190m (78%)       11530m (145%)
  memory             11230739Ki (38%)  33780000Ki (117%)
  ephemeral-storage  0 (0%)            0 (0%)
  hugepages-1Gi      0 (0%)            0 (0%)
  hugepages-2Mi      0 (0%)            0 (0%)

(PS: In order for the model server status to move out of Error/Insufficient resources phase, we added another CPU 8x32 hence we have 3 CPUs now on the cluster)

3. During Model Server deployment we did not see the option to pick between Single or Multi-model server. It might due to available resources but it might be nice to have some info on UI.

4. Modelmesh server pod shows 5 containers and all in running state. No error for any of the containers. I looked at the logs.

[gauravAggrahari]

Hi @jiridanek ,
Any updates ?

[jiridanek]

@red-hat-data-services/model-serving can you please help, either here or in private chat at https://redhat.enterprise.slack.com/archives/C03UGJY6Z1A/p1742036901738849?

I'm out of ready-to-try ideas

[jiridanek]

@sutaakar @jstetina @RomanFilip does this ring a bell, w.r.t. resources stuff? Any insight would be appreciated.

[sutaakar]

No idea from my side

[RomanFilip]

I think I had a similar problem, it was caused by failing operator installation

[spolti]

The last time we saw this issue on modelmesh:

 Caused by: java.lang.Exception: Reported model capacity -0.125GiB too small relative to advertised default model size of 1MiB

IIRC, the model was somehow corrupted.