Generate hosts file with all IPs and hostnames for all containers

[mitar]

I would like to generate an alternative /etc/hosts file with all IPs and hostnames for all containers. I will then use dnsmasq's --addn-hosts to read this file in and provide resolving of containers.

Currently, my template is:

{{ range $index, $container := $ }}
{{ $container.IP }} {{ $container.Hostname }}
{{ end }}

But with recent Docker additions I am not sure if this really lists all IPs for all containers and their hostnames. Doesn't container have multiple IPs in multiple networks? How could I list all IP from all networks and then for each of those IPs all hostnames? So hostname and also hostname.network_name?

[thaJeztah]

Note that the IP-address of the container is the (are the) IP-address(es) on the internal network. Also, containers are already reachable by their name by other containers if they're in the same network, and that alternative names will be supported in docker 1.10 trough this feature; moby/moby#18699

[mitar]

Note that the IP-address of the container is the (are the) IP-address(es) on the internal network.

Here I would like to generate a list of IPs from all networks. Or are you saying that containers cannot access each other between networks even if they would know the IP of a container in another network?

Then, the question is how to list all containers in the current network? So that I could at least list them?

Also, containers are already reachable by their name by other containers if they're in the same network

But not the default bridge. :-(

Also, some of us are stuck with old Docker versions (using distributions like Debian). And were also bitten by /etc/hosts being updated automatically in one version of Docker but then removed in the next version of Docker. It is hard to have stable deployments against a moving target. So solutions like docker-hosts and docker-gen seems a good abstractions to use because docker-gen seems to be more stable than Docker itself in features it supports.

[thaJeztah]

Or are you saying that containers cannot access each other between networks even if they would know the IP of a container in another network?

Yes, networks are isolated, and IP-addresses are only unique per network, so a container can have the same IP-address on different networks. So to reach a container, both containers should be attached to the same network.

But not the default bridge. :-(

That is by design, to keep backwards compatibility with older versions. For the "default" bridge, use --link to connect containers.

Also, some of us are stuck with old Docker versions (using distributions like Debian).

For those distributions, I'd recommend installing docker from the official repository (apt.dockerproject.org). Those are actively maintained. If you want to pin to a specific version, some information can be found here; moby/moby#17935

For the default bridge, your approach could work to add support for discovery. For backwards compatibility, the Networksettings.IpAddress is still present (see #129), and will always be propagated with the IP-address for the default bridge network. Information for other networks is in the Networksettings.Networks.[networkname].IpAddress field, but only accessible to other containers in the same network, and (as described) already discoverable by default (by container-name)

[mitar]

Thanks for all this information. Really helpful.

So I made a Nginx reverse proxy image based on the example docker-gen template, but also allowing custom static sites. To make it work on all versions of Docker I am trying to use docker-gen also to generate a list of IPs.

But from what I understand, docker-gen uses all containers, not just containers from the network where a container with docker-gen is running? So if I am generating nginx (or hosts) configuration using docker-gen, I might be generating entries for containers in other networks? This could make things misbehave, because I would create a reverse proxy entry for a container in the other network my reverse proxy container could not access then.

So I think the question is then: how to list only containers in the same network (or networks) as docker-gen container is running, and how to then get IPs for them?

This ticket was more about syntax though. It is unclear how groupBy could work over $container.Networks which is an array, not just dict. But I see that I do not need that now.

[thaJeztah]

Hm, good points (okay, we're diverting from the original question, but let me see if I can add some info);

But from what I understand, docker-gen uses all containers, not just containers from the network where a container with docker-gen is running?

Yes, that's probably right; docker-gen listens to docker-events, and those will be fired for each start of a container. If you're only interested in the default "bridge" network, checking if Networksettings.IpAddress is filled (or Networksettings.Networks.bridge.IpAddress) should tell you if a container has an IP-address on the default bridge network.

Docker 1.10 will bring and improved events system, with more information available, and also events for network connect/disconnect, see: moby/moby#18888 (documentation is in this commit; moby/moby@851fe00). These events would enable you to listen for containers joining a specific network, which may be even more interesting than listening for container start/stop. I expect that changes will be needed in docker-gen to support this, but it may be an interesting thought.

[mitar]

Hm, I think some function for docker-gen templating language is needed which would limit containers only to those accessible from the docker-gen instance currently running. I would really not like to hard-code into templates which bridge/network an docker-gen instance is in, especially because it can be added dynamically to multiple networks.

[thaJeztah]

I can imagine an option that allows specifying a filter for the events in docker-gen.cfg or as a command-line parameter

[mitar]

Hm, different templates might need different rule here? So this should be per config-section parameter, if something, not a global parameter.

[jwilder]

If you need to make your templates a little more flexible, you might be able to use an env var from the docker-gen environment in the template as well. E.g.

NETWORK=1.2.3.4 docker-gen foo.tmpl

and in the template use:

{{ .Env.NETWORK }}

[unclev]

I was not aware of this issue, and did a template with IP addresses, container host names, names and networks. Please don't beat me for hardcoded domain name and address format, - I did this primarily for myself.