Will the data of the local node be updated when the computing power of the main network expires?

[CTryChu]

At 14:00 on 04/15 (utc+0), it was found that the local Fil node had an oom situation. After checking the heap, it was found that GetPowerTable occupied about 96.5% of the memory. At that time, the mainnet had more than 600 P of expired computing power. Will this trigger the need to update the local computing power data of the node?

There are a lot of these logs.
INFO pubsub go-libp2p-pubsub@v0.13.0/pubsub.go:984 Can't deliver message to subscription for topic /f3/granite/0.0.3/filecoin/49; subscriber too slow

We are using v1.32.2.

Thank you .

[Kubuxu]

Thanks for the report.
Can you confirm how you collected that profile? It might be a memory allocation amount (not retained), not heap usage. It shouldn't be a direct cause of OOM although that depends on your setup. We will verify the memory usage of that code path.

[CTryChu]

Thanks for the report. Can you confirm how you collected that profile? It might be a memory allocation amount (not retained), not heap usage. It shouldn't be a direct cause of OOM although that depends on your setup. We will verify the memory usage of that code path.

In the first few seconds of the outage, a program was generating a heap report.

[Kubuxu]

Which program did you use (exact invocation would be useful)? Could you provide the full report it generated?

[CTryChu]

Several of our local nodes went down after 04/15 14:00 utc+0. cpu,disk, and mem were 100% at that point.
Also, after upgrading to 1.32.2, at 04/15 6:30 utc+0, are there any background processes? cpu,disk, and mem all went up at that time.

[masih]

Thanks.

• Double confirming that the CPU percentage is not per core?
• What does "Disk Io" percent represent? Percentage of IOPS or?
• Can you check if a splitstore compaction was happening around the same time?
  • I think the logs should be able to indicate it.

[CTryChu]

Thanks.

• Double confirming that the CPU percentage is not per core?

• What does "Disk Io" percent represent? Percentage of IOPS or?

• Can you check if a splitstore compaction was happening around the same time?

  • I think the logs should be able to indicate it.

1. All cpu 32 accounts are taken up
2. Disk iops is 9000, which is also full
   The problem happened around 14:00. I will send you the log of one hour before and after.

fil.tar.gz

[Kubuxu]

At 14:16Z a split store compaction has started which is consistent with high IO usage:

fil.log.1:2025-04-15T14:16:50.942Z	INFO	splitstore	splitstore/splitstore_compact.go:856	preparing compaction transaction
fil.log.1:2025-04-15T14:16:50.942Z	INFO	splitstore	splitstore/splitstore_compact.go:153	compacting splitstore
fil.log.1:2025-04-15T14:16:50.942Z	INFO	splitstore	splitstore/splitstore_compact.go:527	waiting for active views to complete
fil.log.1:2025-04-15T14:16:50.943Z	INFO	splitstore	splitstore/splitstore_compact.go:530	waiting for active views done	{"took": 0.000280158}
fil.log.1:2025-04-15T14:16:50.943Z	INFO	splitstore	splitstore/splitstore_compact.go:559	running compaction	{"currentEpoch": "4880673", "baseEpoch": "4876141", "boundaryEpoch": "4877073", "inclMsgsEpoch": "4877073", "compactionIndex": 666}
fil.log.1:2025-04-15T14:16:50.989Z	INFO	splitstore	splitstore/splitstore_compact.go:580	protecting references with registered protectors
fil.log.1:2025-04-15T14:16:50.990Z	INFO	splitstore	splitstore/splitstore_compact.go:508	protected 6 references through 1 protectors
fil.log.1:2025-04-15T14:16:50.990Z	INFO	splitstore	splitstore/splitstore_compact.go:588	marking reachable objects

[Kubuxu]

Additionally around 14:24 the node looks to be out of sync in chain, it gets new blocks in quick succession.

[CTryChu]

At 14:16Z a split store compaction has started which is consistent with high IO usage:

sorry, the graph I posted is in utc+8 and the log is in Utc+0. The time of the anomaly is 22:00 in the picture and 14:00 in the log.

[masih]

@CTryChu We think we might have found an edge case that could have caused what you observed. It is fixed as part of a PR here and will be released in the next upcoming lotus release.

Thank you for reporting this.