WORKER TIMEOUT on application start in AWS EC2 instance

[dboterho]

Hello,

We have been trying to set up an instance of the application on an AWS EC2 instance. Everything seems to be setup correctly; EC2 instance is running, we can telnet to the ports, attached volume is OK, and AWS Elasticsearch instance appears accessible and running. When we hit the /ping URL it times out. And looking at the docker logs we can see the below messages:

[2018-12-04 16:50:33 +0000] [1946] [INFO] Booting worker with pid: 1946
[2018-12-04 16:50:33 +0000] [1947] [INFO] Booting worker with pid: 1947
[2018-12-04 16:50:33 +0000] [1950] [INFO] Booting worker with pid: 1950
[2018-12-04 16:50:33 +0000] [1954] [INFO] Booting worker with pid: 1954
[2018-12-04 16:51:33 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:1946)
[2018-12-04 16:51:33 +0000] [1946] [INFO] Worker exiting (pid: 1946)
[2018-12-04 16:51:33 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:1947)
[2018-12-04 16:51:33 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:1950)
[2018-12-04 16:51:33 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:1954)
[2018-12-04 16:51:33 +0000] [1954] [INFO] Worker exiting (pid: 1954)
[2018-12-04 16:51:33 +0000] [1947] [INFO] Worker exiting (pid: 1947)
[2018-12-04 16:51:33 +0000] [1950] [INFO] Worker exiting (pid: 1950)

We have ruled out memory and cache issues and tried using different versions. The issue appears to originate from the dsys/match image. Does anybody know what our issue is?

AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20180912 (ami-07a3bd4944eb120a0)
EC2 instance type: t2.medium
Elastic Search version: 6.3

Thanks

[cucomans]

Same problem here. Did you get with a solution?

[dboterho]

@cucomans no, didn't find a workround and went with an alternate solution (which didn't include phash).

[cucomans]

It was working fine untill last monday and then boom this error started to occurr.

[lklic]

I had the same problem, from the logs I was getting the following:
Elasticsearch: Max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Following this stackoverflow i did the following:
https://stackoverflow.com/questions/51445846/elasticsearch-max-virtual-memory-areas-vm-max-map-count-65530-is-too-low-inc

edit /etc/sysctl.conf and added vm.max_map_count = 262144 to the end of the file

ran sysctl -w vm.max_map_count=262144
then systemctl restart docker