An Icinga DB crash was reported in the Icinga Community Forum, https://community.icinga.com/t/icingadb-max-allowed-packet-exceeded/14414. After a temporary Icinga DB downtime due to a version update, the Redis contained too much data for the state_history table, exceeding MySQL's max_allowed_packet value of 64 MiB.
Dec 22 10:13:58 skinner2.backbone.admin icingadb[1507116]: Error 1105 (HY000): Parameter of prepared statement which is set through mysql_send_long_data() is longer than 'max_allowed_packet' bytes
can't perform "INSERT INTO \"state_history\" (\"state_type\", \"hard_state\", \"previous_soft_state\", \"check_attempt\", \"soft_state\", \"long_output\", \"object_type\", \"host_id\", \"check_source\", \"endpoint_id\", \"service_id\", \"event_time\", \"previous_hard_state\", \"output\", \"max_check_attempts\", \"id\", \"scheduling_source\", \"environment_id\") VALUES (:state_type,:hard_state,:previous_soft_state,:check_attempt,:soft_state,:long_output,:object_type,:host_id,:check_source,:endpoint_id,:service_id,:event_time,:previous_hard_state,:output,:max_check_attempts,:id,:scheduling_source,:environment_id) ON DUPLICATE KEY UPDATE \"id\" = VALUES(\"id\")"
github.com/icinga/icinga-go-library/database.CantPerformQuery
github.com/icinga/icinga-go-library@v0.4.0/database/utils.go:16
github.com/icinga/icinga-go-library/database.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2.1
github.com/icinga/icinga-go-library@v0.4.0/database/db.go:535
github.com/icinga/icinga-go-library/retry.WithBackoff
github.com/icinga/icinga-go-library@v0.4.0/retry/retry.go:65
github.com/icinga/icinga-go-library/database.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2
github.com/icinga/icinga-go-library@v0.4.0/database/db.go:530
golang.org/x/sync/errgroup.(*Group).Go.func1
golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78
runtime.goexit
runtime/asm_amd64.s:1700
retry deadline exceeded
github.com/icinga/icinga-go-library/retry.WithBackoff
github.com/icinga/icinga-go-library@v0.4.0/retry/retry.go:100
github.com/icinga/icinga-go-library/database.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2
github.com/icinga/icinga-go-library@v0.4.0/database/db.go:530
golang.org/x/sync/errgroup.(*Group).Go.func1
golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78
runtime.goexit
runtime/asm_amd64.s:1700
The reporter solved this issue by performing a FLUSHALL in the Redis.
However, as I wrote in the thread, "Icinga DB always sets max_allowed_packet to 64 MiB, as this is what the underlying MySQL client library does, https://github.com/go-sql-driver/mysql/blob/4395c45fd098a81c5251667cda111f94c693ab14/dsn.go#L88. However, if I read the following MySQL bug - https://bugs.mysql.com/bug.php?id=83958 - correctly, prepared statements may easily exceed this limit, as it seems to be the case for your crash."
Without having looked too closely, the DB.NamedBulkExec method allows bulking. Maybe we can adjust something over there. I have mostly created this issue here to not lose this over in the community forum.
An Icinga DB crash was reported in the Icinga Community Forum, https://community.icinga.com/t/icingadb-max-allowed-packet-exceeded/14414. After a temporary Icinga DB downtime due to a version update, the Redis contained too much data for the
state_historytable, exceeding MySQL'smax_allowed_packetvalue of 64 MiB.The reporter solved this issue by performing a
FLUSHALLin the Redis.However, as I wrote in the thread, "Icinga DB always sets
max_allowed_packetto 64 MiB, as this is what the underlying MySQL client library does, https://github.com/go-sql-driver/mysql/blob/4395c45fd098a81c5251667cda111f94c693ab14/dsn.go#L88. However, if I read the following MySQL bug - https://bugs.mysql.com/bug.php?id=83958 - correctly, prepared statements may easily exceed this limit, as it seems to be the case for your crash."Without having looked too closely, the
DB.NamedBulkExecmethod allows bulking. Maybe we can adjust something over there. I have mostly created this issue here to not lose this over in the community forum.