build: speed up setup.sh for non-merged devel spaces

[wjwwood]

Currently the setup.sh file which is generated by catkin build is very slow because it must execute setup.sh --extend for each leaf package and each of those leaf package's run depends recursively (excluding overlap from build depends). On a large workspace, like hydro-desktop_full, 250 packages means sourcing ~188 setup files.

Because of the way the devel space is created we can theoretically create a faster single setup.sh which builds the combined devel space in one invocation.

Related: #24

[jack-oquin]

I just started experimenting with this fine tool.

Even after reading the doc fairly carefully, I am still unclear about what setup.bash to source.

[jbohren]

Even after reading the doc fairly carefully, I am still unclear about what setup.bash to source.

I currently always run with --merge-devel which, honestly, might make sense to be made default. The real issue with catkin is build-time collisions and side-effects, and having identical globally-scoped target filenames is only a secondary concern, and something which is pretty easy to detect / debug.

[wjwwood]

@jack-oquin I don't understand what you mean, could you explain your workflow a little and at which point it is not clear to you which setup file should be sourced?

[jack-oquin]

I was trying to follow the directions in the docs:

• http://catkin-tools.readthedocs.org/en/latest/commands/catkin_build.html

I created a new workspace for experimentation:

$ source /opt/ros/hydro/setup.bash
$ cd ~/ros
$ mkdir -p ws/src
$ cd ws/src
$ wstool init . ../../rocon/src/.rosinstall

Then I built one package, followed by the rest (~100):

$ cd ..
$ catkin build camera1394
$ catkin build

That all worked fine, but now I need to source something to add all those packages to my ROS environment. It looks like there is a separate devel/setup.bash for every package I built, but no top-level one that gathers them all together.

So, what is the next step? How do I run unit tests? How do I launch ROS nodes?

[jack-oquin]

@jbohren: that --merge-devel option might be the answer I am looking for. Where is it documented?

UPDATE: I found it under catkin build --help. Looks like what I need, so I'll give it a try.

[jack-oquin]

That works fine, mostly as I had guessed. My $ROS_PACKAGE_PATH is over 5KB long, but it seems functional.

Based only on this brief, naive experience, I agree with Jon's suggestion that --merge-devel be made the default. While I feel sure there must be good use cases for "unmerged" devel spaces, perhaps they mostly appeal to more-experienced users.

I realize that the alias mechanism would allow me to change that default, but beginners are unlikely to do that right off. Personally, I avoid using custom tool aliases, because they make it hard to work on uncustomized lab or robot systems.

[wjwwood]

I agree that it should probably be the default too. I just haven't gotten around to changing it. This ticket is meant to address performance concerns related to sourcing the non-merged devel spaces.

[jbohren]

That works fine, mostly as I had guessed. My $ROS_PACKAGE_PATH is over 5KB long, but it seems functional.

Yeah, Orocos also complains about the LD_LIBRARY_PATH being too long when using an isolated develspace.

I agree that it should probably be the default too. I just haven't gotten around to changing it. This ticket is meant to address performance concerns related to sourcing the non-merged devel spaces.

I created a new ticket to track this here: #44

[jack-oquin]

I understand that merging involves more variables than just $ROS_PACKAGE_PATH, but that one is particularly crucial when doing anything with ROS.

After working with catkin_make spaces, it feels particularly inconvenient having to fiddle with $ROS_PACKAGE_PATH so often. When adding a new repository to my workspace, I have to catkin build it, re-source the setup.bash in every shell, then shutdown and restart emacs. All that feels like a drag on productivity.

Since $ROS_PACKAGE_PATH descends the file tree correctly, in most cases all I would need to prepend is ~/ws/src. Technically, I suppose there could be other packages in that part of the tree which are not intended as part of the workspace, although I can't think why anyone would want to do that.

Should catkin build add a --merge-src option signalling that setting RPP to ~/ws/src is acceptable and desired? Or, perhaps that should be the default, with an --isolate-src option to override it.

[wjwwood]

Currently those values are set by each package, and catkin build doesn't have anything to do with it.

It might be possible to have the setup file for catkin build which after sourcing all the packages, simplifies the RPP where possible, assuming that having just the source space would be equivalent to having each package within the workspace.

I have some hesitations about this though. For example, I'm not sure how this works with respect to symlink'ed folders in the source space. Also, I regard this as more magic which has to be done.

I can think of one workaround which might be possible, you could set the RPP to your source space in the terminals before source'ing the setup files, that way it is always on the RPP.

For emacs, I'm not sure why it needs the RPP in the first place, but if it is using the RPP, then maybe it should have a command to re-source the setup file.

I don't have time to work on this right now, but if you can come up with a reliable way of doing this, I'll help implement/merge this where I can.

[jack-oquin]

For emacs, I'm not sure why it needs the RPP in the first place, but if it is using the RPP, then maybe it should have a command to re-source the setup file.

It's for rosemacs, which can find files in packages, etc. Very useful with catkin_make.

[tkruse]

Correct me if I am wrong, aren't the values from devel/.catkin used for generation of the RPP? So only that file would need to be sanitized.

[wjwwood]

I'd have to look at the code which sets the RPP, but that may be the case.