Various improvements: Multi sources support, patches, variables, auto generated packages

[smuzaffar]

This PR proposes following updates which make building/configuring packages easy. All these changes should be backward compatible and should not change/rebuilt existing packages (if package recipe is not changed). See [a] for a example package recipe

• Support for non-git based sources. This also allows to have multiple sources for a package e.g. RocM has around 30 binary sources which one can not push to github. Having sources: in the package recipe will override default source:. For each source bits build will set SOURCEn env variable (where n is source number starting with 0 for first source). In build recipe one can use $SOURCEDIR/$SOURCE0 to access the first source. SOURCE_COUNT env will be set to point to total number of sources. Changes in expanded sources will result in changed hash of package and should trigger a re-build

• Support for providing patches from the local build recipe config directory. patches should be available in the recipe_dir/patches e.g. general.bit/patches. bits build will set PATCHn env variable (where n is patch number starting with 0 for first patch). In build recipe one can use $SOURCEDIR/$PATCH0 to access the first patch. PATCH_COUNT env will be set to point to total number of patches. Change in the patch content will result in the change of package hash and should re-trigger the build

• One can add configuration parameters via variables: . One can use these to substitute values within variables, sources, patches and prepend_path variables using %(variable_name)s. It can also recursively expand variables e.g. %%(%(v1)s_%(v2)s)s will be first expanded to %(<v1_value>_<v2_value>)s and then it will be expanded to the value of <v1_value>_<v2_value>. This is helpful when there are different sources based on arch or some configuration parameters. variables do not change the hash of package unless used by the package recipe/sources/patches etc. Also added root_dir in the known variables which expands to ${<PACKAGE_NAME>_ROOT}. this is helpful for adding prepend_path e.g. instead of

    prepend_path:
      LID_LIBRARY_PATH: ${PACKAGE_ROOT}/lib64

one can do

    prepend_path:
      LID_LIBRARY_PATH: %(root_dir)s/lib64

it helps when one copy/paste the package recipes

• Added support to auto-generate packages without having their <package>.sh file. This is helpful for generating python's pip based packages using the requirements.txt file or generate data packages which has the same build recipe but different package repos and tags.

• Newly generated init.sh file now do something like the following i.e. check if directory exists

  [ ! -d "${PACKAGE_ROOT}/lib" ] || export LD_LIBRARY_PATH="$PACKAGE_ROOT/lib${LD_LIBRARY_PATH+:$LD_LIBRARY_PATH}"

also added lib64 in the default LD_LIBRARY_PATH of package

• Expanding variables in build recipe: If package has variables or explicitly has added expand_recipe: true then bits will expand the build recipe too e.g. it will replace all %(var_name)s with the value of var_name

[a]

package: foo
version: "1.0"
variables:
  cxxflags: -g -Os -Wall
  ldflags: -lfoo
  aarch64_src: -arm
  x86_64_src: ""
  selected_src: %%(%(platform_machine)s_src)s
sources:
  - https://foo.bar/%(package)s-%(version)s%(selected_src)s.tar.gz
patches:
 - foo-make.patch
 - bar-make.patch
 ---
tar -xzvf ${SOURCEDIR}/${SOURCE0}
cd ${PKGNAME}-${PKGVERSION}
if [ "${PATCH_COUNT}" -gt 0 ] ; then
  for (( num=0; num<${PATCH_COUNT}; num++ )) ; do
    patch_name="PATCH${num}"
    patch  --verbose -p1 < $SOURCEDIR/${!patch_name}
  done
fi
make CXXFLAGS="%(cxxflags)s" LDFLAGS="%(ldflags)s"
make install

[smuzaffar]

FYI @akritkbehera