Deployment fails when cloudfoundry-for-logsearch is not co-deployed

[voelzmo]

Hi there! I'm having trouble deploying the latest logsearch boshrelease. When the update gets to the ingestor nodes, bosh-healthcheck will fail with

==> bosh_dns_health.stdout.log <==
[healthcheck] 2020/10/14 09:15:01 INFO - Initializing
[healthcheck] 2020/10/14 09:15:01 ERROR - failed parsing jobs: open /var/vcap/jobs/parser-config-lfc/.bosh/links.json: permission denied

This is because this directory is different from the other job directories

ingestor/7b226d96-23d1-402a-bcdc-3ab4768b4dc7:/var/vcap/jobs# ls -la
total 28
drwxr-x---  3 root vcap 4096 Oct 14 09:06 .
drwxr-xr-x 11 root root 4096 Oct 14 09:07 ..
lrwxrwxrwx  1 root root   69 Oct 14 09:06 bosh-dns -> /var/vcap/data/jobs/bosh-dns/6fad4c7d8547cbc7f2f6e4d0b9b6eddfac190de6
lrwxrwxrwx  1 root root   64 Oct 14 09:06 bpm -> /var/vcap/data/jobs/bpm/50c9c73827dc7174f54e77960ad93d070a0d261c
lrwxrwxrwx  1 root root   74 Oct 14 09:06 elasticsearch -> /var/vcap/data/jobs/elasticsearch/18e5ab69eb9ea678e5a20760a509d92433022e65
lrwxrwxrwx  1 root root   76 Oct 14 09:06 ingestor_syslog -> /var/vcap/data/jobs/ingestor_syslog/571810ace7aede5e4d31b01af70fffb9dd64dbca
drwx------  3 root root 4096 Oct 14 08:52 parser-config-lfc

And that seems to be the case because it is not actually a job I'm using, but just a bpm-generated mount:

logsearch-boshrelease/jobs/ingestor_syslog/templates/config/bpm.yml.erb

Lines 8 to 14 in 7e56a1e

	additional_volumes:
	- path: /var/vcap/sys/tmp/ingestor_syslog
	writable: true
	allow_executions: true
	- path: /var/vcap/jobs/ingestor_syslog/config
	writable: true
	- path: /var/vcap/jobs/parser-config-lfc/config

Steps to reproduce

• Used bbl up with newest bosh-deployment and jumpbox-deployment from github
• deployed logsearch with bosh deploy -d logsearch ~/workspace/logsearch-boshrelease/deployment/logsearch-deployment.yml -o ~/workspace/logsearch-boshrelease/deployment/operations/enable-router.yml -o ~/workspace/logsearch-boshrelease/deployment/operations/enable-tls.yml -o ~/workspace/logsearch-boshrelease/deployment/operations/disable-post-start.yml -o ~/workspace/logsearch-boshrelease/deployment/operations/change-azs.yml -v="azs=[z1,z2]", which worked, and then afterwards with bosh deploy -d logsearch ~/workspace/logsearch-boshrelease/deployment/logsearch-deployment.yml -o ~/workspace/logsearch-boshrelease/deployment/operations/enable-router.yml -o ~/workspace/logsearch-boshrelease/deployment/operations/enable-tls.yml -o ~/workspace/logsearch-boshrelease/deployment/operations/change-azs.yml -v="azs=[z1,z2]", which failed, because the bosh-dns-healthcheck job keeps failing on the ingestor VM

ingestor/7b226d96-23d1-402a-bcdc-3ab4768b4dc7              -                     failing        z1  10.0.1.15  logsearch  
~                                                          bosh-dns              failing        -   -          -  
~                                                          bosh-dns-healthcheck  unknown        -   -          -  
~                                                          bosh-dns-resolvconf   running        -   -          -  
~                                                          elasticsearch         running        -   -          -  
~                                                          ingestor_syslog       running        -   -          -  

It seems that for some reason the mechanism to indicate that there is no links.json to be parsed cannot work because of incorrect permissions for this directory:
https://github.com/cloudfoundry/bosh-dns-release/blob/e94bb010822b25959eba2feb0de7ad47abfa4a33/src/bosh-dns/healthconfig/jobs_parser.go#L73-L82

[Infra-Red]

@axelaris Do you know why writable: true has not been configured for parser-config-lfc job in ingestor_syslog bpm config?

logsearch-boshrelease/jobs/ingestor_syslog/templates/config/bpm.yml.erb

Line 14 in 7e56a1e

- path: /var/vcap/jobs/parser-config-lfc/config