Skip to content

超时问题:Timeout request,NettyRPC server request timeout!  #28

Description

@zanhonglei

架构如下:
Jstorm实时拉源数据,数据交给bolt处理,bolt处理完的数据同步该框架把数据回写到远程,当处理20W数据的时候,会出现超时的情况,无论同步的RPC或者异步的RPC请求,超时时间30s,网络状态良好,我把博主的代码copy到项目中,报错信息如下:
cn.com.jit.pmi2.collection.netty.rpc.exception.InvokeTimeoutException: Timeout request,NettyRPC server request timeout!
at cn.com.jit.pmi2.collection.netty.rpc.core.MessageCallBack.await(MessageCallBack.java:86)
at cn.com.jit.pmi2.collection.netty.rpc.core.MessageCallBack.start(MessageCallBack.java:48)
at cn.com.jit.pmi2.collection.netty.rpc.netty.MessageSendProxy.handleInvocation(MessageSendProxy.java:43)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:84)
at com.sun.proxy.$Proxy98.addOrUpdateUser(Unknown Source)
at cn.com.jit.pmi2.collection.topology.bolt.user.AddUserBolt.execute(AddUserBolt.java:71)
at com.alibaba.jstorm.task.execute.BoltExecutors.processTupleEvent(BoltExecutors.java:248)
at com.alibaba.jstorm.task.execute.BoltExecutors.onEvent(BoltExecutors.java:184)
at backtype.storm.utils.DisruptorQueueImpl.consumeBatchToCursor(DisruptorQueueImpl.java:261)
at backtype.storm.utils.DisruptorQueueImpl.consumeBatchWhenAvailable(DisruptorQueueImpl.java:226)
at backtype.storm.utils.DisruptorQueueImpl.consumeBatchWhenAvailable(DisruptorQueueImpl.java:185)
at com.alibaba.jstorm.task.execute.BoltExecutors.run(BoltExecutors.java:148)
at com.alibaba.jstorm.callback.AsyncLoopRunnable.run(AsyncLoopRunnable.java:95)
at java.lang.Thread.run(Thread.java:748)


java.lang.ClassCastException: io.netty.util.concurrent.ScheduledFutureTask cannot be cast to io.netty.channel.EventLoop at cn.com.jit.pmi2.collection.netty.rpc.netty.MessageSendInitializeTask$1.operationComplete(MessageSendInitializeTask.java:62) at cn.com.jit.pmi2.collection.netty.rpc.netty.MessageSendInitializeTask$1.operationComplete(MessageSendInitializeTask.java:55) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:683) at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:604) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:564) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:425) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:276) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:292) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:545) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:485) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:399) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:748)
java.lang.NullPointerException: null at cn.com.jit.pmi2.collection.netty.rpc.netty.RpcServerLoader.unLoad(RpcServerLoader.java:128) at cn.com.jit.pmi2.collection.netty.rpc.netty.MessageSendExecutor.stop(MessageSendExecutor.java:49) at cn.com.jit.pmi2.collection.netty.rpc.event.ClientStopEventListener.listen(ClientStopEventListener.java:34) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:91) at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:150) at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:76) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399) at com.google.common.eventbus.Subscriber.dispatchEvent(Subscriber.java:71) at com.google.common.eventbus.Dispatcher$PerThreadQueuedDispatcher.dispatch(Dispatcher.java:116) at com.google.common.eventbus.EventBus.post(EventBus.java:217) at cn.com.jit.pmi2.collection.netty.rpc.spring.NettyRpcReference.destroy(NettyRpcReference.java:63) at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:272) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:954) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523) at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:961) at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1033) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1009) at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:961) at org.springframework.context.support.AbstractApplicationContext.destroy(AbstractApplicationContext.java:948) at cn.com.jit.pmi2.collection.topology.bolt.user.AddUserBolt.execute(AddUserBolt.java:144) at com.alibaba.jstorm.task.execute.BoltExecutors.processTupleEvent(BoltExecutors.java:248) at com.alibaba.jstorm.task.execute.BoltExecutors.onEvent(BoltExecutors.java:184) at backtype.storm.utils.DisruptorQueueImpl.consumeBatchToCursor(DisruptorQueueImpl.java:261) at backtype.storm.utils.DisruptorQueueImpl.consumeBatchWhenAvailable(DisruptorQueueImpl.java:226) at backtype.storm.utils.DisruptorQueueImpl.consumeBatchWhenAvailable(DisruptorQueueImpl.java:185) at com.alibaba.jstorm.task.execute.BoltExecutors.run(BoltExecutors.java:148) at com.alibaba.jstorm.callback.AsyncLoopRunnable.run(AsyncLoopRunnable.java:95) at java.lang.Thread.run(Thread.java:748)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions