Skip to content

Entity-server crashes due to running out of JavaScript Heap memory #2364

Description

@JulianGro

This happens on the Hub every couple of hours.

Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]: <--- Last few GCs --->
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]: 0.[943733:0x7f919800d000]  8078400 ms: Mark-Compact (reduce) 1399.3 (1424.2) -> 1398.7 (1424.8) MB, pooled: 0 MB, 1821.78 / 0.00 ms  (+ 87.8 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2668 ms) (average mu = 0.[943733:0x7f919800d000]  8081853 ms: Mark-Compact (reduce) 1399.7 (1424.8) -> 1399.1 (1425.5) MB, pooled: 0 MB, 2034.97 / 0.00 ms  (+ 67.1 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2910 ms) (average mu = 0.
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]: <--- JS stacktrace --->
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]: #
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]: # Fatal JavaScript out of memory: Reached heap limit
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]: #
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]: ==== C stack trace ===============================
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::base::debug::StackTrace::StackTrace()+0x16) [0x7f91ce0e7206]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(+0x140ab3b) [0x7f91cd20ab3b]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::base::FatalOOM(v8::base::OOMType, char const*)+0x3b) [0x7f91ce0d3f0b]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(+0x181e085) [0x7f91cd61e085]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&)+0x355) [0x7f91cd61e3e5]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(+0x1a46dd7) [0x7f91cd846dd7]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)+0x2bd) [0x7f91cd85e89d]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment)+0xb1b) [0x7f91cd8371db]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment)+0x25) [0x7f91cd837875]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment)+0xcb) [0x7f91cd81450b]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawFixedArray(int, v8::internal::AllocationType)+0x2c) [0x7f91cd80577c]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::internal::Factory::CopyPropertyArrayAndGrow(v8::internal::Handle<v8::internal::PropertyArray>, int)+0x37) [0x7f91cd81f3d7]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::internal::JSObject::MigrateToMap(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Map>, int)+0x17c0) [0x7f91cda38670]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::internal::LookupIterator::ApplyTransitionToDataProperty(v8::internal::Handle<v8::internal::JSReceiver>)+0x181) [0x7f91cda97951]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::internal::Object::TransitionAndWriteDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin)+0x71) [0x7f91cdaafd51]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::internal::Runtime::SetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::StoreOrigin, v8::Maybe<v8::internal::ShouldThrow>)+0x1ea) [0x7f91cdc284aa]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libnode.so.127(v8::Object::Set(v8::Local<v8::Context>, v8::Local<v8::Value>, v8::Local<v8::Value>)+0x114) [0x7f91cd6326f4]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libscript-engine.so(ScriptValueV8Wrapper::setProperty(QString const&, ScriptValue const&, QFlags<ScriptValue::PropertyFlag> const&)+0x24f) [0x7f91d279886f]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libentities.so(EntityItemProperties::copyToScriptValue(ScriptEngine*, bool, bool, bool, std::bitset<15ul>) const+0x3aa8) [0x7f91d303e2c8]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libentities.so(EntityItemNonDefaultPropertiesToScriptValue(ScriptEngine*, EntityItemProperties const&)+0x28) [0x7f91d2ff07c8]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libentities.so(RecurseOctreeToJSONOperator::processEntity(std::shared_ptr<EntityItem> const&)+0x7b) [0x7f91d2fdb37b]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libentities.so(RecurseOctreeToJSONOperator::postRecursion(std::shared_ptr<OctreeElement> const&)+0xa7) [0x7f91d2fdb767]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::recurseElementWithOperator(std::shared_ptr<OctreeElement> const&, RecurseOctreeOperator*, int)+0x52) [0x7f91d2d43792]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/libentities.so(EntityTree::writeToJSON(QString&, std::shared_ptr<OctreeElement> const&)+0x1f3) [0x7f91d2f8df53]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::toJSONString(QString&, std::shared_ptr<OctreeElement> const&)+0x136) [0x7f91d2d449b6]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::toJSON(QByteArray*, std::shared_ptr<OctreeElement> const&, bool)+0x3c) [0x7f91d2d44d3c]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::writeToJSONFile(char const*, std::shared_ptr<OctreeElement> const&, bool)+0x4a) [0x7f91d2d493ea]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(Octree::writeToFile(char const*, std::shared_ptr<OctreeElement> const&, QString)+0x3a7) [0x7f91d2d49a47]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(OctreePersistThread::persist()+0x1d6) [0x7f91d2d62176]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /opt/overte/lib/liboctree.so(OctreePersistThread::process()+0xac) [0x7f91d2d6276c]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libQt5Core.so.5(+0x31d357) [0x7f91d1d1d357]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libQt5Core.so.5(QObject::event(QEvent*)+0x18b) [0x7f91d1d0c03b]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libQt5Core.so.5(QCoreApplication::notifyInternal2(QObject*, QEvent*)+0xf5) [0x7f91d1cde425]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libQt5Core.so.5(QTimerInfoList::activateTimers()+0x3d1) [0x7f91d1d3a821]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libQt5Core.so.5(+0x33b174) [0x7f91d1d3b174]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x5c3c5) [0x7f91cb3033c5]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x5e5f7) [0x7f91cb3055f7]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_iteration+0x30) [0x7f91cb305d60]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libQt5Core.so.5(QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)+0x6d) [0x7f91d1d3b4ad]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libQt5Core.so.5(QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)+0x132) [0x7f91d1cdcdc2]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libQt5Core.so.5(QThread::exec()+0x60) [0x7f91d1adda90]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libQt5Core.so.5(+0xded21) [0x7f91d1aded21]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libc.so.6(+0x92b7b) [0x7f91d129eb7b]
Jul 11 20:39:34 domain-beta.overte.org assignment-client[943733]:     /lib/x86_64-linux-gnu/libc.so.6(+0x1107f8) [0x7f91d131c7f8]
Jul 11 20:39:36 domain-beta.overte.org systemd[1]: overte-entity-server@Overte_Hub.service: Main process exited, code=killed, status=5/TRAP
Jul 11 20:39:36 domain-beta.overte.org systemd[1]: overte-entity-server@Overte_Hub.service: Failed with result 'signal'.
Jul 11 20:39:36 domain-beta.overte.org systemd[1]: overte-entity-server@Overte_Hub.service: Consumed 35min 53.712s CPU time, 4.1G memory peak, 4.8G memory swap peak.
Jul 11 20:39:36 domain-beta.overte.org systemd[1]: overte-entity-server@Overte_Hub.service: Scheduled restart job, restart counter is at 222.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions