RELEASE_NOTES

===============================
Release Notes for Pegasus 4.7.4
===============================

We are pleased to announce Pegasus 4.7.4. Pegasus 4.7.4 is a minor
release of Pegasus and includes improvements and bug fixes to the
4.7.4 release. 

Bugs Fixed
--------------
1) [PM-1148] - kickstart should print a more helpful error message if
the executable is missing 

2) [PM-1160] - Dashboard is not recording the hostname correctly

3) [PM-1163] - Confusing error message in pegasus-kickstart

4) [PM-1164] - worker package in submit directory gets deleted during
workflow run  

===============================
Release Notes for Pegasus 4.7.3
===============================

We are happy to announce the release of Pegasus 4.7.3. Pegasus 4.7.3
is a minor release of Pegasus and includes improvements and bug fixes
to the 4.7.2 release. It has a bug fix without which monitoring will
break for users running with HTCondor 8.5.8 or higher. 

Improvements
--------------
1) [PM-1109] – dashboard to display errors if a job is killed instead
   of exiting with non zero exitcode 
   
   pegasus-monitord did not pass signal information from the kickstart
   records to the monitoring database. If a job fails and because of a
   signal, it will now create an error message indicating the signal
   information, and populate it. 

2) [PM-1129] – dashboard should display database and pegasus version 

3) [PM-1138] – Pegasus dashboard pie charts should distinguish between
   running and unsubmitted 

4) [PM-1155] – remote cleanup jobs should have file url’s if possible 

Bugs Fixed
--------------

1) [PM-1132] – Hashed staging mapper doen’t work correctly with sub
   dax generation jobs 

   For large workflows with dax generation jobs, the planning broke
   for sub workflows if the dax was generated in a hashed directory
   structure. It is now fixed. 

   Note: As a result of this fix, pegasus-plan prescripts for sub
   workflows in all cases, are now invoked by pegasus-lite 

2) [PM-1135] – pegasus.transfer.bypass.input.staging breaks symlinking
   on the local site 

3) [PM-1136] – With bypass input staging some URLs are ending up in
   the wrong site 

4) [PM-1147] – pegasus-transfer should check that files exist before
   trying to transfer them 

   In case where the source file url’s don’t exist, pegasus-transfer
   used to still attempt multiple retries resulting in hard to read
   error messages. This was fixed, whereby pegasus-transfer does not
   attempt retries on a source if a source file does not exist. 

5) [PM-1151] – pegasus-monitord fails to populate stampede DB
   correctly when workflow is run on HTCondor 8.5.8 

6) [PM-1152] – pegasus-analyzer not showing stdout and stderr of
   failed transfer jobs 

   In case of larger stdout+stderr outputted by an application, we
   store only first 64K in the monitoring database combined for a
   single or clustered job. There was a bug whereby if a single task
   outputted more than 64K nothing was populated. This is fixed 

7) [PM-1153] – Pegasus creates extraneous spaces when replacing <file
   name=”something” /> 

   DAX parser was updated to not add extraneous spaces when
   constructing the argument string for jobs 

8) [PM-1154] – regex too narrow for GO names with dashes

9) [PM-1157] – monitord replay should work on submit directories that
   are moved

   pegasus generated submit files have absolute paths. However, for
   debugging purposes where a submit directory might be moved to a
   different host, where the paths don’t exist. monitord now searches
   for files based on relative paths from the top level submit
   directory. This enables users to repopulate their workflow
   databases easily. 

===============================
Release Notes for Pegasus 4.7.2
===============================

We are happy to announce the release of Pegasus 4.7.2. Pegasus 4.7.2
is a minor release of Pegasus and includes improvements and bug fixes
to the 4.7.1 release.

Improvements in 4.7.2 are

 - [PM-1141] - The commit to allow symlinks in pegasus-transfer broke
               PFN fall through
 - [PM-1142] - Do not set LD_LIBRARY_PATH in job env
 - [PM-1143] - R DAX API
 - [PM-1144] - pegasus lite prints the wrong hostname for non-glidein
               jobs


===============================
Release Notes for Pegasus 4.7.1
===============================

We are happy to announce the release of Pegasus 4.7.1. Pegasus 4.7.1
is a minor release of Pegasus and includes improvements and bug fixes
to the 4.7.0 release.

Improvements in 4.7.1 are

- Fix for stage in jobs with repeated portion of LFN [PM-1131]
- Fix for pegasus.transfer.bypass.input.staging breaks symlinking on
  the local site [PM-1135]
- Capture the execution site information in pegasus lite [PM-1134]
]
- Added ability to check CVMFS for worker package


===============================
Release Notes for Pegasus 4.7.0
===============================
We are happy to announce the release of Pegasus 4.7.0.  Pegasus 4.7.0
is a major release of Pegasus and includes all the bug fixes and
improvements in the 4.6.2 release 

New features and Improvements in 4.7.0 are

    - automatic submit directory organization
    - improved directory management on staging site in nonsharedfs mode
    - R DAX API
    - pegasus-analyzer reports information about held jobs
    - check for cyclic dependencies in DAG

New Features
--------------
1) [PM-833] – Pegasus should organize submit files of workflows in
   hierarchal data structure 

   Pegasus now automatically distributes the files in HTCondor submit
   directory for all workflows in 2 level directory structure. This is
   done to prevent having too many workflow and condor submit files in
   one directory for a large workflow.  The behavior of submit
   directory organization can be controlled by the following
   properties 

   pegasus.dir.submit.mapper.hashed.levels         the number of
   directory levels used to accomodate the files. Defaults to 2. 
   pegasus.dir.submit.mapper.hashed.multiplier      the number of
   files associated with a job in the submit directory. defaults to    5. 

   Note that this is enabled by default.  If you want to have pre
   4.7.0 behavior you can 
   	 pegasus.dir.submit.mapper Flat
   
   Submit mapper properties are documented in the user guide here
   https://pegasus.isi.edu/docs/4.7.0/properties.php#site_dir_props 

2) [PM-833] – Pegasus should manage directory structure on the staging
site

    For non sharedfs mode, Pegasus will now automatically manage the
    directory structure on the staging site in a hierarchal directory
    structure via use of staging mappers. The staging mappers
    determine what sub directory on the staging site a job will be
    associated with. Before, the introduction of staging mappers, all
    files associated with the jobs scheduled for a particular site
    landed in the same directory on the staging site. As a result, for
    large workflows this could degrade filesystem performance on the
    staging servers. More information can be found in the
    documentation at
    https://pegasus.isi.edu/docs/4.7.0/ref_staging_mapper.php 

3) [PM-1036] – R DAX API

   Pegasus now includes an R API for generating DAXes of complex and 
   large workflows in R environments. The API follows the Google' R 
   style guide, and all objects and methods are defined using the S3 
   OOP system. The source package can be obtained by running 
   *pegasus-config --r* or from the Pegasus' downloads page. A tutorial
   workflow can be generated using *pegasus-init*, and an example
   workflow is provided in the examples folder. More information can
   be found in the documentation at
   https://pegasus.isi.edu/docs/4.7.0/dax_generator_api.php#api-r

   Related JIRA item: [PM-1074] – Add R example to pegasus-init

4) [PM-1126] – pegasus-analyzer should report information about held jobs

   Pegasus monitoring daemon now populates the reason for held jobs in
   it’s database. Both pegasus-analyzer and dashboard were updated to
   show this information. 
   Related JIRA items:
   	   [PM-1121] – Store reasons for workflow failure in stampede database
	   [PM-1122] – update dashboard to display reasons for workflow state and jobstate
	   [PM-1058] – Create homebrew tap for pegasus

5) Pegasus is now also available via home-brew on MACOSX via a tap
 repository
   github.com/pegasus-isi/homebrew-tools

   It contains formulas for pegasus and htcondor.Users can do:$ brew
   tap pegasus-isi/tools 

$ brew install pegasus htcondor
$ brew tap homebrew/services
$ brew services start htcondor

6) [PM-928] – pegasus-exitcode should write its output to a log file

   pegasus-exticode is now set to write to a workflow global log file
   ending in exitcode.log that captures pegasus-exitcode stdout and
   stderr as json messages.  This allows users to check
   pegasus-exitcode messages, which otherwise would have been set to
   /dev/null by condor dagman. 

7) [PM-1115] – Pegasus to check for cyclic dependencies in the DAG
   Pegasus now explicitly checks for cyclic dependencies and reports
   one of the edges making up the cycle. 

8) [PM-1054] – Add option to ignore files in libinterpose

   kickstart now has support for environment variables
   KICKSTART_TRACE_MATCH and KICKSTART_TRACE_IGNORE that determine
   what file accesses are captured via lib interpose. The MATCH
   version only traces files that match the patterns, and the IGNORE
   version does NOT trace files that match the patterns. Only one of
   the two can be specified. 

9) [PM-915] – modify kickstart to collect and aggregate runtime
metadata

10) [PM-1004] – update metrics server ui to display extra planner configuration metrics

11) [PM-1111] – pegasus planner and api’s should have support for
ppc64 as architecture type

12) [PM-1117] – Support for tutorial via pegasus-init on bluewaters

Improvement
--------------

1) [PM-1125] – Disable builds for older platforms

2) [PM-1116] – pass task resource requirements as environment
variables for job wrappers to pick up 

3) [PM-1112] – enable variable expansion for regex based replica
catalog

4) [PM-1105] – Mirror job priorities to DAGMan node priorities

   HTCondor ticket 5749 . We can assign DAG priorities only if
   detected condor version is greater than 8.5. 

5) [PM-1094] – pegasus-dashboard file browser should load on demand

6) [PM-1079] – pegasus-statistics should be able to skip over failures
when generating particular type of stats 

7) [PM-1073] – condor_q changes in 8.5.x will affect pegasus-status

8) [PM-1023] – KIckstart stdout/stderr as CDATA

9) [PM-749] – Store job held reasons in stampede database

10) [PM-1088] – Move to relative paths in dagman and condor submit
files

11) [PM-900] – site catalog for XSEDE

12) [PM-901] – site for OSG

Bugs Fixed
--------------
1) [PM-1061] – pegasus-analyzer should detect and report on failed job
submissions

2) [PM-1118] – database changes to jobstate and workflow state tables

3) [PM-1124] – Hashed Output Mappers throw unable to instantiate error

4) [PM-1127] – Wf adds worker package staging even though it has already placed a worker package in place
 

===============================
Release Notes for Pegasus 4.6.2
===============================


We are happy to announce the release of Pegasus 4.6.2.  Pegasus 4.6.2
is a minor release of Pegasus and includes improvements and bug fixes
to the 4.6.1 release. 
 
New features and Improvements in 4.6.2 are

- support for kickstart wrappers that can setup a user environment 
- support for Cobalt and SLURM schedulers via the Glite interfaces 
- ability to do local copy of files in PegasusLite to staging site, if
  the compute and staging site is same 
- support for setting up Pegasus Tutorial on Bluewaters using
  pegasus-init 

New Features
--------------

1) [PM-1095] - pegasus-service init script

2) [PM-1101] - Add support for gsiscp transfers
   These will work like the scp ones, but with x509 auth instead of
   ssh public keys. 

3) [PM-1110] - put in support for cobalt scheduler at ALCF
   Pegasus was updated to use the HTCondor Blahp support. ALCF has a
   cobalt scheduler to schedule jobs to the BlueGene system. The
   documentation has details on how the pegasus task requirement
   profiles map to Cobalt
   parameters. https://pegasus.isi.edu/docs/4.6.2/glite.php#glite_mappings 
   
   To use HTCondor on Mira, please contact the HTCondor team to point
   you to the latest supported HTCondor installation on the system. 

4) [PM-1096] - Update Pegasus' glite support to include SLURM

5) [PM-1115] - Pegasus to check for cyclic dependencies in the DAG

   Pegasus now checks for cyclic dependencies that may exist in the
   DAX or are as a result of adding edges automatically based on data
   depedencies 

6) [PM-1116] - pass task resource requirements as environment
   variables for job wrappers to pick up 
   The task resource requirements are also passed as environment
   variables for the jobs in the GLITE style. This ensures that job
   wrappers can pick up task requirement profiles as environment
   variables. 

Improvements
--------------

1) [PM-1078] - pegasus-statistics should take comma separated list of
   values for -s option 

2) [PM-1105] - Mirror job priorities to DAGMan node priorities

   The job priorities associated with jobs in the workflow are now
   also associated as DAGMan node priorities, provided that HTCondor
   version is 8.5.7 or higher. 

3) [PM-1108] - Ability to do local copy of files in PegasusLite to
    staging site, if the compute and staging site is same 
    
    The optimization implemented is implemented in the Planner's
    pegasus lite generation code, where when constructing the
    destination URL's for the output site it checks for 
    a) symlinking is turned on
    b) compute site for the job and staging site for job are same.

    This means that the shared-scratch directory used on the staging
    site is locally accessible to the compute nodes. So we can go
    directly via the filesystem to copy the file. So instead of
    creating a gsiftp url , will create a file url in pegasuslite
    wrappers for the jobs running on local site. 

4) [PM-1112] - enable variable expansion for regex based replica
   catalog 

   Variable expansion for Regex based replica catalogs was not
   supported earlier. This is fixed now. 

5) [PM-1117] - Support for tutorial via pegasus-init on Bluewaters
   pegasus-init was updated to support running tutorial examples on
   Bluewaters. To use this, users need to logon to the bleaters login
   node and run pegasus-init. The assumption is that HTCondor is
   running on the login node either in user space or root. 

6) [PM-1111] - pegasus planner and api's should have support for ppc64
as architecture type 

Bugs Fixed
--------------

1) [PM-1087] - dashboard and pegasus-metadata don't query for sub
   workflows 

2) [PM-1089] - connect_by_submitdir should seek for braindump.txt in
   the workflow root folder 

3) [PM-1093] - disconnect in site catalog and DAX schema for specifying OSType

4) [PM-1099] - x509 credentials should be transferred using
   x509userproxy 

5) [PM-1100] - Typo in rsquot, ldquot and rdquot

6) [PM-1106] - pegasus-init should not allow (or should handle) spaces
   in site name 

7) [PM-1107] - pegasuslite signal handler race condition 

8) [PM-1113] - make planner directory options behavior more consistent

===============================
Release Notes for PEGASUS 4.6.1
===============================
We are happy to announce the release of Pegasus 4.6.1.  Pegasus 4.6.1
is a minor release of Pegasus and includes improvements and bug fixes
to the 4.6.0 release 

New features and Improvements in 4.6.1 are 

    - support for MOAB submissions via glite. A new tool called
      pegasus-configure-glite helps users setup their HTCondor GLite
      directory for use with Pegasus 
    - pegasus-s3 now allows for downloading and uploading folders to
      and from S3
    - initial support for globus online in pegasus-transfer
      planner automatically copies the user catalog files into a
      directory called catalogs in the submit directory. 
    - changes to how worker package staging occurs for compute jobs.

New Features
--------------
1) [PM-1045] – There is a new command line tool
   pegasus-configure-glite that automatically installs the Pegasus
   shipped glite local attributes script to the condor glite
   installation directory 

2) [PM-1044] – Added glite scripts for moab submissions via the Glite
   interface 

3) [PM-1054] – kickstart has an option to ignore files in lib
   interpose.

   This is triggered by setting the environment variables
   KICKSTART_TRACE_MATCH and KICKSTART_TRACE_IGNORE. The MATCH version
   only traces files that match the patterns, and the IGNORE version
   does NOT trace files that match the patterns. Only one of the two
   can be specified. 

4) [PM-1058] -pegasus can be now installed via homebrew on MACOSX 
   For details refer to documentation at
   https://pegasus.isi.edu/documentation/macosx.php 

5) [PM-1075] – pegasus-s3 to be able to download all files in a folder 

   pegasus-s3 has a –recursive option to allow users to download all
   files from a folder in S3 or upload all files from a local
   directory to S3 bucket. 

6) [PM-680] – Add support for GlobusOnline to pegasus-transfer
   Details on how to configure can be found at
   https://pegasus.isi.edu/docs/4.6.1/transfer.php#transfer_globus_online 

7) [PM-1043] – Improve CSV file read for Storage Constraints algorithm

8) [PM-1047] – Pegasus saves all the catalog files in submit dir in a
   directory named catalogs. This enables for easier debugging later on
   as everything is saved in the submit directory. 

Improvements
--------------
1) [PM-1043] – Improve CSV file read for Storage Constraints algorithm

2) [PM-1057] – PegasusLite worker package download improvements
   Pegasus exposes two additional properties to control behavior of
   worker package staging for jobs. Users can use these to control
   whether a PegasusLite job downloads a worker package from the
   pegasus website or not , in case the shipped worker package does
   not match the node architecture. 

   pegasus.transfer.worker.package.strict – enforce strict checks
   against provided worker package. if a job comes with worker package
   and it does not match fully with worker node architecture , it
   falls down to pegasus download website. Default value is true. 
   pegasus.transfer.worker.package.autodownload – a boolean property
   to indicate whether a pegasus lite job is allowed to download from
   pegasus website. Defaults to true. 

3) [PM-1059] – Implement backup for MySQL databases

4) [PM-1060] – expose a way to turn off kickstart stat options

5) [PM-1063] – improve performance for inserts into database replica catalog

6) [PM-1067] – pegasus-cluster -R should report the finish time and
   duration, not the start time and duration 

7) [PM-1078] – pegasus-statistics should take comma separated list of
   values for -s option 

8) [PM-1073] – condor_q changes in 8.5.x will affect pegasus-status

   pegasus-status was updated to account for changes in the condor_q
   output in the 8.5 series 

Bugs Fixed
--------------
1) [PM-1077] – pegasus-remove on hierarchal workflows results in jobs
   from the sub workflows still in the condor queue

   DAGMan no longer condor_rm jobs in a workflow itself. Instead it
   relies on condor schedd to do it. Pegasus generated sub workflow
   description files did not trigger this . As a result,
   pegasus-remove on a top level workflow still resulted in jobs from
   the sub workflows to be in the condor queue. This is now
   fixed. Pegasus generated dagman submit files have the right
   expressions specified. 

2) [PM-997] – pyOpenSSL v0.13 does not work with new version of
   openssl (1.0.2d) and El Captain 

3) [PM-1048] – PegasusLite should do a full version check for
   pre-installed worker packages 

   PegasusLite does a full check ( including the patch version) with
   the pegasus version installed on the node, when determining whether
   to use the preinstalled version on the node or not. 

4) [PM-1050] – pegasus-plan should not fail if -D arguments don’t
   appear first

5) [PM-1051] – Error missing when nodes, cores, and ppn are all
   specified 

   In 4.6.0 release, there was a bug where the error message thrown (
   when user specified an invalid combination of task requirements)
   was incorrect. This is fixed, and error messages have been improved
   to also indicate a reason 

6) [PM-1053] – pegasus-cluster does not know about new Kickstart
   arguments

7) [PM-1055] – Interleaved libinterpose records

8) [PM-1061] – pegasus-analyzer should detect and report on failed job
   submissions 

   pegasus-monitord did not populate the stampede workflow database
   with information about job submission failures. As a result,
   pegasus-analyzer for the cases where a job failed because of job
   submission errors did not report any helpful information as to why
   the job failed. This is now fixed. 

9) [PM-1062] – pegasus dashboard shows some workflows twice

   In the case where HTCondor crashes on a submit node, DAGMan logs
   may miss a workflow end event. When monitord detects consecutive
   start events, it creates and inserts a workflow end event. The end
   event had the same timestamp as the new start event, because of
   which underlying dashboard query retrieved multiple rows.  This was
   fixed by setting the timestamp for the artificial end event to be
   one second less than the second start event. 

10) [PM-1064] – pegasus-transfer prepends to PATH

    pegasus-transfer used to prepend the system path with other
    internal determined lookup directories based on environment
    variables such as GLOBUS_LOCATION. As a result, in some cases,
    user preferred copy of executables were not picked up. This is now
    fixed. 

11) [PM-1066] – wget errors because of network issues
    pegasus-transfer now sets the OSG_SQUID_LOCATION/http_proxy
    setting only for the first wget attempt

12) [PM-1068] – monitord fails when trying to open a job error file in
    a workflow with condor recovery 

    monitord parses the job submit file whenever it notices job
    submission log by DAGMan. This is done to avoid the case, where
    because of HTCondor recovery a job may not have a ULOG_ job
    submission event, because of which the internal state of the job
    maybe uninitialized. 

13) [PM-1069] – Dashboard invocation page gives an error if the task
    has no invocation record 

    Dashboard did not display invocation records for Pegasus added
    auxiliary jobs in the workflow. This was due to a bug in the query
    that is now fixed. 

14) [PM-1070] – monitord should handle case where jobs have missing
    JOB_FAILURE/JOB_TERMINATED events 

15) [PM-1072] – Worker package staging issues on OSX

16) [PM-1081] – pegasus-plan complains if output dir is set but site
     catalog entry for local site does not storage directory specified 

     pegasus-plan complained if a storage directory was not specified
     in the site catalog entry for site “local”, even if a user
     specified a –output-dir option. This is now fixed. The planner
     will create a default file server based entry for this case. 


17) [PM-1082] – transfer jobs don’t have symlink destination URL even
    though symlink is enabled 

    In the case, where there are multiple candidate replica locations
    ( some on preferred site and some on other sites), the destination
    URL for the transfer jobs did not have a symlink URL. As a result
    the data was never symlinked even though it was available locally
    on the preferred site. 

18) [PM-1083] – dashboard user home page requires a trailing /

    To access a user home page on the dashboard, a trailing / needs to
    be specified after the username in the URL. Dashboard was updated
    to handle URL’s without trailing names. 

19) [PM-1084] – credential handling for glite jobs

    As part of credential handling the environment variable for the
    staged credential was as environment key instead of the
    +remote_environment classed key.  As a result transfer jobs
    running via Glite submission failed as the appropriate environment
    variable was not set. This is fixed now.

20) [PM-1085] – -p 0 options for condor_dagman sub dax jobs result in
    dagman ( 8.2.8) dying 

    Pegasus updated to generate the dagman submit files for sub
    workflows to be compatible with 8.5.x series.  However, the new
    arguments added resulted in breaking workflows running with old
    HTCondor versions. The offending argument is now set only if
    condor version is more than 8.3.6

21) [PM-1086] – Never symlink executables
    
    Pegasus adds chmod jobs to explicitly set the x bit of the
    executables staged. If the executable is a symlinked executable,
    then chmod fails. Symlinking is never triggered for staged
    executables now. 
 

===============================
Release Notes for PEGASUS 4.6.0
===============================

 We are happy to announce the release of Pegasus 4.6.0.  Pegasus 4.6.0
 is a major release of Pegasus and includes all the bug fixes and
 improvements in the 4.5.4 release 
 
 New features and Improvements in 4.6.0  are
     - metadata support
     - support for variable substitution 
     - constraints based cleanup algorithm
     - common pegasus profiles to specify task requirements
     - new command line client pegasus-init to configure pegasus and
 pegasus-metadata to query workflow database for metadata 
     - support for fallback PFN's

 Migration guide available at
 http://pegasus.isi.edu/wms/docs/4.6.0dev/useful_tips.php#migrating_from_leq45 
 
 Debian and Ubuntu users: Please note that the Apt repository GPG key
 has changed. To continue to get automatic updates, please follow the
 instructions on the download page on how to install the new key. 

New Features
--------------

1)  Metadata support in Pegasus
    Pegasus allows users to associate metadata at
     - Workflow Level in the DAX
     - Task level in the DAX and the Transformation Catalog
     - File level in the DAX and Replica Catalog
 
    Metadata is specified as a key value tuple, where both key and
    values are of type String. 
 
    All the metadata ( user specified and auto-generated) gets
    populated into the workflow database ( usually in the workflow
    submit directory) by pegasus-monitord. The metadata in this
    database can be be queried for using the pegasus-metadata command
    line tool, or is also shown in the Pegasus Dashboard. 
 
    Documentation: https://pegasus.isi.edu/wms/docs/4.6.0/metadata.php
 
    Relevant JIRA items
    [PM-917] - modify the workflow database to associate metadata with
    workflow, job and files 
    [PM-918] - modify pegasus-monitord to populate metadata into
    stampede database 
    [PM-919] - pegasus-metadata command line tool
    [PM-916] - identify and generate the BP events for metadata
    [PM-913] - kickstart support for stat command line options
    [PM-1025] - Document the metadata capability for 4.6
    [PM-992] - automatically capture file metadata from kickstart and record it
    [PM-892] - Add metadata to DAX schema
    [PM-893] - Add metadata to Python DAX API
    [PM-894] - Add metadata to site catalog schema
    [PM-895] - Add metadata to transformation catalog text format
    [PM-902] - support for metadata to JAVA DAX API
    [PM-903] - add metadata to perl dax api
    [PM-904] - support for parsing DAX 3.6 documents
    [PM-978] - Update JDBCRC with the new schema
    [PM-925] - support for 4.1 new site catalog schema with metadata extensions
    [PM-991] - pegasus dashboard to display metadata stored in workflow database
 
2) Support for Variable Substitution

   Pegasus Planner supports notion of variable expansions in the DAX
   and the catalog files along the same lines as bash variable
   expansion works. This is often useful, when you want paths in your
   catalogs or profile values in the DAX to be picked up from the
   environment. An error is thrown if a variable cannot be expanded.  

   Variable substitution is supported in the DAX, File Based Replica
   Catalog, Transformation Catalog and the  Site Catalog. 

   Documentation: https://pegasus.isi.edu/wms/docs/4.6.0/variable_expansion.php

   Relevant JIRA items
   [PM-831] - Add better support for variables

3) Constraints based Cleanup Algorithm

   The planner now has support for a new cleanup algorithm called
   constraint. The algoirthm adds cleanup nodes to constraint the
   amount of storage space used by a workflow. The nodes remove files
   no longer required during execution. The added cleanup node
   guarantees limits on disk usage. The leaf cleanup nodes are also
   added when this is selected. 
 
   [PM-850] - Integrate Sudarshan's cleanup algorithm
 
4) Common Pegasus Profiles to indicate Resource Requirements for jobs

   Users can now specify Pegasus profiles to indicate resource
   requirements for jobs. Pegasus will automatically, translate these
   to the approprate condor, globus or batch system keys based on how
   the job is executed.  

   The profiles are documented in the configuration chapter at ask
   requirement profiles are documented here
   https://pegasus.isi.edu/wms/docs/4.6.0/profiles.php#pegasus_profiles 
 
   [PM-962] - common pegasus profiles to indicate resource requirements for job

 
5) New client pegasus-init
 
   A new command line client called "pegasus-init" that generates a
   new workflow configuration based by asking the user a series of
   questions. Based on the responses to these questions,
   *pegasus-init* generates a workflow configuration including a DAX
   generator, site catalog, properties file, and other artifacts that
   can be edited to meet the user's needs. 
 
   [PM-1019] - pegasus-init client to setup pegasus on a machine
 
6) Support for automatic fallover to fallback file locations

   Pegasus, now during Replica Selection orders all the candidate
   replica's instead of selecting the best replica. This replica's are
   ordered based on the strategy selected, and the ordered list is
   passed to pegasus-transfer invocation. This allows users to specify
   failover, or preferred location for discovering the input files. 

   By default, planner employs the following logic for the ordering of replicas 
      - valid file URL's . That is URL's that have the site attribute
   matching the site where the executable pegasus-transfer is
   executed.  
      - all URL's from preferred site (usually the compute site) 
      - all other remotely accessible ( non file) URL's 
 
   If a user wants to specify their own order preference , then they
   should use the Regex Replica Selector and specify a ranked order
   list of regular expressions in the properties.  
 
   Documentation:
   https://pegasus.isi.edu/wms/docs/4.6.0/data_management.php#replica_selection

   Relevant JIRA items: 
   [PM-1002] - Support symlinking against compute site datasets in
   nonsharedfs mode with bypass of input file staging 
   [PM-1014] - Support for Fallback PFN while transferring raw input files
 
7) Support SGE via the HTCondor Glite/Batch GAHP support 
 
   Pegasus now has support for submitting to a local SGE cluster via
   the HTCondor Glite/Blahp interfaces. More details can be found in
   the documentation at
   https://pegasus.isi.edu/wms/docs/4.6.0/glite.php  

   [PM-955] - Support for direct submission through SGE using
   Condor/Glite/Blahp layer 
 
8) Glite Style improvements

   Users don't need to set extra pegasus profiles to enable jobs to
   run correctly on glite style sites. By default, condor quoting for
   jobs  on glite style sites is disabled. Also, the -w option to
   kickstart is always as batch gahp does not support specification of
   a remote execution directory directly.  
 
   If the user knows that a compute site shares a file system with the
   submit host, then they can get Pegasus to run the auxillary jobs in
   local universe. This is especially helpful , when submitting to
   local campus clusters using Glite and users don't want the pegasus
   auxillary jobs to run through the cluster PBS|SGE queue. 

   Relevant JIRA items
   [PM-934] - changed how environment is set for jobs submitted via
   HTCondor Glite / Blahp layer 
   [PM-1024] - Use local universe for auxiliary jobs in glite/blahp mode
   [PM-1037] - Disable Condor Quoting for jobs run on glite style
   execution sites 
   [PM-960] - Set default working dir to scratch dir for glite style jobs
 
 
9)  Support for PAPI CPU counters in kickstart
    [PM-967] - Add support for PAPI CPU counters in Kickstart
 
10) Changes to worker package staging

    Pegasus now by default, attempts to use the worker package out of
    the Pegasus submit host installation unless a user has specified
    finer grained attributes for the compute sites in the site catalog
    or an entry is specified in the transformation catalog.  

    Relevant JIRA items
    [PM-888] - Guess which worker package to use based on the submit host
 
11) [PM-953] - PMC now has the ability to set CPU affinity for multicore tasks.
 
12) [PM-954] - Add useful environment variables to PMC
 
13) [PM-985] - separate input and output replica catalogs 

    Users can specify a different output replica catalog optionally by
    specifying the property with prefix pegasus.catalog.replica.output 

    This is useful when users want to separate the replica catalog
    that they use for discovery of input files and the catalog where
    the output files generated are registered. For example use a
    Directory backed replica catalog backend to discover file
    locations, and a file based replica catalog to catalog the
    locations of the output files. 

14) [PM-986] - input-dir option to pegasus-plan should be a comma
separated list 
 
15) [PM-1031] - pegasus-db-admin should have an upgrade/dowgrade
option to update all databases from the dashboard database to current
pegasus version 

16) [PM-882] - Create prototype integration between Pegasus and Aspen
 
17) [PM-964] - Add tips on how to use CPU affinity on condor

Improvements
--------------

1) [PM-924] - Merge transfer/cleanup/create-dir into one client

2) [PM-610] - Batch scp transfers in pegasus-transfer

   pegasus-transfer now batches 70 transfers in a single scp
   invocation against the same host. 
 
3) [PM-611] - Batch rm commands in scp cleanup implementation

   scp rm are now batched together at a level of 70 per group so that
   we can keep the command lines short enough.
 
4) [PM-856] - pegasus-cleanup should use pegasus-s3's bulk delete
feature

    s3 removes are now batched and passed in a temp file to pegasus-s3
 
5) [PM-890] - pegasus-version should include a Git hash
 
6) [PM-899] - Handling of database update versions from different branches
 
7) [PM-911] - Use ssh to call rm for sshftp URL cleanup
 
8) [PM-929] - Use make to build externals to make python development easier
 
9) [PM-937] - Discontinue support for Python 2.4 and 2.5
 
10) [PM-938] - Pegasus DAXParser always validates against latest supported DAX version
 
11) [PM-958] - Deprecate "gridstart" names in Kickstart
 
12) [PM-963] - Add support for wrappers in Kickstart

     Kickstart supports an environment variable, KICKSTART_WRAPPER
     that contains a set of command-line arguments to insert between
     Kickstart and the application 
 
13) [PM-965] - monitord amqp population
 
14) [PM-979] - Update documentation for new DB schema
 
15) [PM-984] - condor_rm on a pegasus-kickstart wrapped job does not
return stdout back 

      When a user condor_rm's their job, Condor sends the job a
      SIGTERM. Previously this would cause Kickstart to die. This
      commit changes Kickstart so that it catches the SIGTERM and
      passes it on to the child instead. That way the child dies, but
      not Kickstart, and Kickstart can report an invocation record
      forthe job to provide the user with useful debugging info. This
      same logic is also applied to SIGINT and SIGQUIT. 
 
16) [PM-1018] - defaults for pegasus-plan to pick up properties and
other catalogs 

      pegasus will default the --conf option to pegasus-plan to
      pegasus.properties in the current working directory.  
      In addition, the default locations for the various catalog files
      now point to current working directory ( rc.txt, tc.txt,
      sites.xml )  
 
 
17) [PM-1038] - Update tutorial to reflect the defaults for Pegasus 4.6 release
 
Bugs Fixed
--------------

1) [PM-653] - pegasus.dagman.nofity should be removed in favor of
Pegasus level notifcaitons 
 
2) [PM-897] - kickstart is reporting misleading permission error when
it is really a file not found 
 
3) [PM-906] - Add Ubuntu apt repository
 
4) [PM-910] - Cleanup jobs should ignore "file not found" errors, but
not other errors 
 
5) [PM-920] - Bamboo / title.xml problems
 
6) [PM-922] - Dashboard and monitoring interface contain Python that
is not valid for RHEL5 
 
7) [PM-923] - Debian packages rebuild documentation
 
8) [PM-931] - For Subworkflows Monitord populates host.wf_id to be
wf_id of root_wf and not wf_id of sub workflow 
 
9) [PM-944] - Make it possible to build Pegasus on SuSE (openSUSE and SLES)
 
10) [PM-1029] - Planner should ensure that local aux jobs run with the same Pegasus install as the planner
 
11) [PM-1035] - pegasus-analyzer fails when workflow db has no
entries

===============================
Release Notes for PEGASUS 4.5.4
===============================

We are happy to announce the release of Pegasus 4.5.4. Pegasus 4.5.4
is a minor release, which contains minor enhancements and fixes
bugs. This will most likely be the last release in the 4.5 series, and
unless you have specific reasons to stay with the 4.5.x series, we
recommend to upgrade to 4.6.0. 

New Features
--------------
1) [PM-1003] - planner should report information about what options were
used in the planner 

   Planner now reports additional metrics such as command line
   options, whether PMC was used and number of deleted tasks to the
   metrics server.   

2) [PM-1007] - "undelete" or attach/detach for pegasus-submitdir

   pegasus-submit dir has two new commands : attach, which adds the
   workflow to the dashboard (or corrects the path), and detach, which
   removes the workflow from the dashboard.  

3) [PM-1030] - pegasus-monitord should parse the new dagman output
that reports timestamps from condor user log 

   Starting 8.5.2 , HTCondor DAGMan record sthe condor job log
   timestamps in the ULOG event messages in the end of the log
   message. monitord was updated to prefer these timestamps for the
   job events if present in the DAGMan logs. 
 
 
Improvements
--------------
1) [PM-896] - Document events that monitord publishes

   The netlogger messages generated by monitord that are used for
   populated the workflow database and master database, are now
   documented at
   https://pegasus.isi.edu/wms/docs/4.5.4cvs/stampede_wf_events.php  

2) [PM-995] - changes to Pegasus tutorial 

   Pegasus tutorial was reorganized and simplified to focus more on
   the pegasus-dashboard, and debugging exercises 

3) [PM-1033] - update monitord to handle updated log messages in dagman.out file

   Starting 8.5.x series, some of the dagman log messages in
   dagman.out file were updated to have HTCondor instead of
   Condor. This broke the monitord parsing regex's and hence it was
   not able to parse information from the dagman.out file. This is now
   fixed. 

4) [PM-1034] - Make it more difficult for users to break pegasus-submitdir archive

   Adding locking mechanism internally, to make pegasus-submitdir more
   robust , when a user accidently kills an archive operation .  

5) [PM-1040] - pegasus-analyzer should be able to handle cases where the workflow failed to start

   pegasus-analyzer now detects if a workflow failed to start because
   of DAGMan fail on NFS error setting, and also displays any errors
   in *.dag.lib.err files. 


Bugs Fixed
--------------

1)  [PM-921] - Specified env is not provided to monitord

    The environment for pegasus-monitord is now set in the dagman.sub
    file. The following order is used: pick system environment,
    override it with env profiles in properties and then from the
    local site entry in the site catalog. 

2)  [PM-999] - pegasus-transfer taking too long to finish in case of retries

    pegasus-transfer has moved to a exponential back-off: min(5 **
    (attempt_current + 1) + random.randint(1, 20), 300)  
    That means that failures for short running transfers will still
    take time, but is necessary to ensure scalability of real world
    workflows .

3) [PM-1008] - Dashboard file browser file list breaks with sub-directories

   Dashboard filebrowser broke when there were sub directories in the
   submit directory. this is now fixed. 

4) [PM-1009] - File browser just says "Error" if submit_dir in workflow db is incorrect

   File browser gives a more informative message when submit directory
   recorded in the database does not actually exist. 

5) [PM-1011] - OSX installer no longer works on El Capitan

   El Capitan has a new "feature" that disables root from modifying
   files in /usr with some exceptions (e.g. /usr/local). Since the
   earlier installer installed Pegasus in /usr, it no longer
   worked. Installer was updated to install Pegasus in /usr/local
   instead. 

6) [PM-1012] - pegasus-gridftp fails with "no key" error

   The SSL proxies jar was updated . The error was triggered because
   of following JGlobus issue:
   https://github.com/jglobus/JGlobus/issues/146  

7) [PM-1017] - pegasus-s3 fails with [SSL: CERTIFICATE_VERIFY_FAILED]

   s3.amazonaws.com has a cert that was issued by a CA that is not in
   the cacerts.txt file bundled with boto 2.5.2. Boto bundled with
   Pegasus was updated to 2.38.0 

8) [PM-1021] - kickstart stat for jobs in the workflow does not work for clustered jobs

   kickstart stat did not work for clustered jobs. This is now fixed.

9) [PM-1022] - dynamic hierarchy tests failed randomly

   The DAX jobs were not considered for cleanup. Because of this, if
   there was a compute job that generated the DAX the subdax job
   required, sometimes the cleanup of the dax file happened before the
   subdax job finished. This is now fixed. 

10) [PM-1039] - pegasus-analyzer fails with: TypeError: unsupported operand type(s) for -: 'int' and 'NoneType'

    pegasus-analyzer threw a stacktrace when a workflow did not start
    because of DAGMan NFS settings.  This is now fixed. 

11) [PM-1041] - pegasus-db-admin 4.5.4 gives a stack trace when run on pegasus 4.6 workflow submit dir

    A clean error is displayed, if pegasus-db-admin from 4.5.4 is run
    against a workflow submit directory from a higher Pegasus
    version. 

===============================
Release Notes for PEGASUS 4.5.3
===============================

We are happy to annouce the release of Pegasus 4.5.3. Pegasus 4.5.3 is
a minor release, which contains minor enhancements and fixes bugs in
the Pegasus 4.5.2 release.

The following issues were addressed and more information can be found
in the Pegasus Jira (https://jira.isi.edu/)

Bug Fixes:

  [PM-980] - pegasus-plots fails with "-p all"
  [PM-982] - MRC replica catalog backend does not work
  [PM-987] - noop jobs created by Pegasus don't use DAGMan NOOP keyword
  [PM-996] - Pegasus Statistics transformation stats columns getting
             larger ad larger with more sub workflows
  [PM-997] - pyOpenSSL v0.13 does not work with new version of openssl
             (1.0.2d) and El Captain

Improvements:

  [PM-976] - ignore register and transfer flags for input files
  [PM-981] - register only based names for output files with deep LFN's
  [PM-983] - data reuse algorithm should consider file locations while
             cascading deletion upwards
  [PM-984] - condor_rm on a pegasus-kickstart wrapped job does not
             return stdout back
  [PM-988] - pegasus-transfer should handle file://localhost/ URL's
  [PM-989] - pegasus-analyzer debug job option should have a hard check
             for output files
  [PM-993] - Show dax/dag planning jobs in
             failed/succesfull/running/failing tabs in dashboard
  [PM-1000] - turn off concurrency limits by default

New Features:

  [PM-985] - separate input and output replica catalog
  [PM-986] - input-dir option to pegasus-plan should be a comma
             separated list


===============================
Release Notes for PEGASUS 4.5.2
===============================

We are happy to annouce the release of Pegasus 4.5.2. Pegasus 4.5.2 is
a minor release, which contains minor enhancements and fixes bugs in
the Pegasus 4.5.1 release.  The release addresses a critical fix for
systems running HTCondor 8.2.9 , whereby all dagman jobs for Pegasus
workflows fail on startup. 
 
Enhancements
-------------- 

1) File locations in the DAX treated as a Replica Catalog
 
   By default, file locations listed in the DAX override entries
   listed in the Replica Catalog. Users can now set the boolean
   property   pegasus.catalog.replica.dax.asrc to treat the dax
   locations along with the entries listed in the Replica Catalog for
   Replica Selection. 

   Associated JIRA item   https://jira.isi.edu/browse/PM-973
 
 2) Pegasus auxillary tools now have support for  iRods 4.x
 
Bugs Fixed
--------------

1) pegasus-dagman setpgid fails under HTCondor 8.2.9
 
   Starting with version 8.2.9, HTCondor sets up the process group
   already to match the pid, and hence the setpgid fails in the
   pegasus-dagman wrapper around condor-dagman. Because of this all
   Pegasus workflows fail to start on submit nodes with  HTCondor
   8.2.9 .  

   If you cannot upgrade to Pegasus version 4.5.2 and are running
   HTCondor 8.2.9, you can set you can turn off HTCondor's setsid'ing
   by setting the following in your condor configuration 
 
    USE_PROCESS_GROUPS = false 
 
    The pegasus-dagman wrapper now does not fatally fail, if setpgid
    fails. More details at
    https://jira.isi.edu/browse/PM-972
 
2) nonshareddfs execution does not work for Glite if auxiliary jobs
are planned to run remotely 
 
    For nonsharedfs execution to a local PBS|SGE cluster using the
    GLite interface, Pegasus generated auxillary jobs had incorrect
    paths to pegasus-kickstart in the submit files, if a job was
    mapped to run on the remote ( non local ) site.  
 
    This is now fixed.
    https://jira.isi.edu/browse/PM-971

===============================
Release Notes for PEGASUS 4.5.1
===============================

We are happy to annouce the release of Pegasus 4.5.1. Pegasus 4.5.1 is
a minor release, which contains minor enhancements and fixes bugs to
Pegasus 4.5.0 release.   

Enhancements
--------------

1) pegasus-statistics reports workflow badput

   pegasus-statistics now reports the workflow badput time, which is
   the sum of all failed kickstart jobs. More details at
   https://pegasus.isi.edu/wms/docs/4.5.1/plotting_statistics.php 

   Associated JIRA item   https://jira.isi.edu/browse/PM-941

2) fast start option for pegasus-monitord

   By default, when monitord starts tracking a live dagman.out file,
   it sleeps intermittently, waiting for new lines to be logged in the
   dagman.out file.  

   This behavior, however causes monitord to lag considerably 
   	- when starting for large workflows 
	- when monitord gets restarted due to some failure by
   pegasus-dagman, or we submit a rescue dag.  

   Users can now set the property pegasus.monitord.fast_start property
   to enable it. For a future release, it will be the default
   behavior. 

   Associated JIRA item   https://jira.isi.edu/browse/PM-947


3) Support for throttling jobs across workflows using HTCondor
concurrency limits  

   Users can now throttle jobs across worklfows using HTCondor
   concurrency limits. However, this only applies to vanilla universe
   jobs. 

   Documentation at
   https://pegasus.isi.edu/wms/docs/4.5.1sjob_throttling.php#job_throttling_across_workflows  

   Associated JIRA item   https://jira.isi.edu/browse/PM-933 

4) Support for submissions to local SGE cluster using the GLite interfaces

   Prelimnary support for SGE clusters has been added in Pegasus. To
   use this you need to copy the sge_local_submit_attributes.sh from
   the Pegasus share directory and place it in your condor
   installation.  

   The list of supported keys can be found here
   https://pegasus.isi.edu/wms/docs/4.5.1/glite.php

   Associated JIRA item   https://jira.isi.edu/browse/PM-955 


5) PEGASUS_SCRATCH_DIR set in the job environment for sharedfs deployment

   Pegasus not sets an environment variable for the job that indicates
   the PEGASUS scratch directory the job is executed in  , in the case
   of sharedfs deployments. This is the directory that is created by
   the create dir job on the execution site for the workflow.  

   Associated JIRA item   https://jira.isi.edu/browse/PM-961


6) New properties to control read timeout while setting up connections
to the database 

   User can now set pegasus.catalog.*.timeout to set the timeout value
   in seconds. This should be set only if you encounter database
   locked errors for your installation. 

   Associated JIRA item   https://jira.isi.edu/browse/PM-943

7) Ability to prepend to system path before launcing an application
executable 

   Users can now associate an env profile named KICKSTART_PREPEND_PATH
   with their jobs, to specify the PATH where application specific
   modules are installed. kickstart will take this value and prepend
   it to system path before launching the executable 

   Associated JIRA item   https://jira.isi.edu/browse/PM-957

8) environment variables in condor submit files are specified using the newer condor syntax 

   For GLITE jobs the environment is specified using the key
   +remote_environment. For all other jobs, the environment is
   specified using the environment key but the value is in newer
   format ( i.e key=value separated by whitespace) 

   Associated JIRA item   https://jira.isi.edu/browse/PM-934

9) pass options to pegasus-monitord via properties

   Users can now specify pegasus.monitord.arguments to pass extra
   options with which pegasus-monitord is launched for the workflow at
   runtime. 

   Associated JIRA item   https://jira.isi.edu/browse/PM-948

10) pegasus-transfer support OSG stashcp

    pegasus-transfer has support for the latest version of stashcp
    
    Associated JIRA item   https://jira.isi.edu/browse/PM-948

11) pegasus-dashboard improvements

    pegasus-dashboard now loads the directory listing via a AJAX
    calls. Makes the loading of the workflow details page much faster
    for large workflows. 

    Show working dir. for a job_instance, and invocation in job
    details and invocation details page. 

    Displays an appropriate error message if pegasus-db-admin update
    of a database fails. 

    Added a HTML error page for DB Migration error. 

    Configure logging so Flask log messages show up in Apache logs

    Associated JIRA item   https://jira.isi.edu/browse/PM-940

12) PEGASUS_SITE environment variable is set in job's environment

    https://jira.isi.edu/browse/PM-907 


Bugs Fixed
--------------

1) InPlace cleanup failed if an intermediate file when used as input had transfer flag set to false

   If an intermediate file ( an output file generated by a parent job)
   was used as an input file to a child job with the transfer flag set
   to false, then the associated cleanup job did not have a dependency
   to the child job. As a result, the cleanup job  could run before
   the child job  (that required it as input) could be run. 

   This is now fixed.
   https://jira.isi.edu/browse/PM-969


2) Incorrect ( malformed) rescue dag submitted in case planner dies
because of memory related issues 

   For hieararchal workflows, if a sub worklfow fails then a rescue
   dag for the sub workflow gets submitted on the job retry. The .dag
   file for the sub workflow is generated by the planner. If the
   planner fails during code generation an incoplete .dag file can be
   submitted. 

   This is now fixed. The planner now writes the dag to a tmp file
   before renaming it to the .dag extension when code completion is
   done. 

    https://jira.isi.edu/browse/PM-966

3) Mismatched memory units in kickstart records

   kickstart now reports all memory values in KB. Earlier the procs
   element in the machine entry was reporting the value in bytes,
   while the maxrss etc values in the usage elments were in KB. 

   This is now fixed. 
   https://jira.isi.edu/browse/PM-959

4) pegasus-analyzer did not work for sub workflows

   There was a bug in the 4.5.0 release where pegasus-analyzer did not
   pick up the stampede database for the sub workflows correctly. This
   is now fixed. 

   https://jira.isi.edu/browse/PM-956

5) Rescue DAGS not submitted correctly for dag jobs

   There was a bug in the 4.5.0 release as a result of the
   .dag.condor.sub file was generated. As a result of that, the force
   option was propogated for the dag jobs in the DAX ( dag jobs are
   sub workflows that are not planned by Pegasus). 

    https://jira.isi.edu/browse/PM-949

6) nonsharedfs configuration did not work with Glite style submissions

   In case of nonsharedfs, transfer_executable is set to true to
   transfer the PegasusLite script. However, in the Glite case, that
   was explicity disabled, which was preventing the workflows from
   running successfully. 

   https://jira.isi.edu/browse/PM-950

7) pegasus-analyzer catches error for wrong directory instead of
listing the traceback  

   https://jira.isi.edu/browse/PM-946

8) pegasus-gridftp fails with: Invalid keyword "POSTALCODE"

   pegasus-gridftp was failing against the XSEDE site stampede,
   because of change in certificates at TACC. This was fixed by
   udpating to the latest jglobus jars. 

   https://jira.isi.edu/browse/PM-945

9) pegasus-statistics deletes directories even if -o option is specified

   By default pegasus-statistics deletes the statistics directory in
   which the statistics files are generated. However, this had the
   side affect of deleting user specified directories set by the -o
   option. that is no longer the case. 

   https://jira.isi.edu/browse/PM-932

10) pegasus-exitcode ignores errors when it gets "-r 0"

    pegasus-exitcode now only ignores invocation records exitcodes ,
    but does all the other checks specified when the -r option is
    specified. 

    https://jira.isi.edu/browse/PM-927

12) pegasus-statistics displays a workflow not found error in case of throwing SqlAlchemy error

    This also happens if pegasus-admin creates an empty workflow
    database for a new workflow, and nothing is populated ( because
    events population is turned off).  

    Associated JIRA item   https://jira.isi.edu/browse/PM-942

13) InPlace cleanup did not work correctly with mutlisite runs

    InPlace cleanup did not work correctly with inter site transfer
    jobs. This is now fixed. 

     https://jira.isi.edu/browse/PM-936




===============================
Release Notes for PEGASUS 4.5.0
===============================

We are happy to announce the release of Pegasus 4.5.0.  Pegasus 4.5.0
is a major release of Pegasus and includes all the bug fixes and
improvements in the minor releases 4.4.1 and 4.4.2 . 
 
New features and Improvements in 4.5.0 are
 - ensemble manager for managing collections of workflows
 - support for job checkpoint files
 - support for Google storage
 - improvements to pegasus-dashboard
 - data management improvements
 - new tools pegasus-db-admin, pegasus-submitdir , pegasus-halt and pegasus-graphviz

Migration guide available
http://pegasus.isi.edu/wms/docs/4.5.0/useful_tips.php#migrating_from_leq44 
 
NEW FEATURES
--------------

1) Ensemble manager for managing collections of workflows
 
   The ensemble manager is a service that manages collections of
   workflows called ensembles. The ensemble manager is useful when you
   have a set of workflows you need to run over a long period of
   time. It can throttle the number of concurrent planning and running
   workflows, and plan and run workflows in priority order. A typical
   use-case is a user with 100 workflows to run, who needs no more
   than one to be planned at a time, and needs no more than two to be
   running concurrently. 
 
   The ensemble manager also allows workflows to be submitted and
   monitored programmatically through its RESTful interface. 
   Details about ensemble manager can be found at
   https://pegasus.isi.edu/wms/docs/4.5.0/service.php 
 
2) Support for Google Storage
 
   Pegasus now supports running of workflows in the Google cloud. When
   running workflows in Google cloud, users can specify  Google
   storage to act as the staging site. More details on how to
   configure Pegasus to use google storage can be found at
   pegasus.isi.edu/wms/docs/4.5.0/cloud.php#google_cloud. All the
   pegasus auxillary clients ( pegasus-transfer, pegasus-create-dir
   and pegasus-cleanup) were updated to handle google storage URL's (
   starting with gs://). The tools call out to google command line
   tool called gsutils. 
 
3) Support for job checkpoint files
 
   Pegasus now supports checkpoint files created by jobs. This allows
   users to run long running jobs ( where the runtime of a job exceeds
   the maxwalltime supported on a compute site) to completion,
   provided the jobs generate a checkpoint file periodically. To use
   this, checkpoint files with link as checkpoint need to be specified
   for the jobs in the DAX . Additionally, the jobs need to specify
   the pegasus profile checkpoint.time that indicates the number of
   minutes after which pegasus-kickstart sends a TERM signal to the
   job, signalling it to start the generation of the checkpoint file .  
 
   Details on this can be found in the userguide
   https://pegasus.isi.edu/wms/docs/4.5.0/transfer.php#staging_job_checkpoint_files  
 
4) Pegasus Dashboard Improvements
 
   Pegasus dashboard can now be deployed in multiuser mode. It is now
   started by the pegasus-service command.  Instructions for starting
   the pegasus service can be found at
   https://pegasus.isi.edu/wms/docs/4.5.0/service.php#idp2043968 

   Changed the look and feel of the dashboard. Users can now track all
   job instances ( retries ) of a job through the dashboard. Earlier
   it was only the latest job retry. 

   There is a new tab called failing jobs on the workflows page. The
   tab lists jobs that have failed at least once and are currently
   being retried. 

   The submit host is displayed on the workflow's main page.

   The job details page now shows information about the Host where the
   job ran, and all the states that the job has gone through. 

   The dashboard also has a file browser which allows users to view
   files in the worklfow submit directory directly from the
   dashboard. 

5) Data configuration is now supported per site

   Starting with the 4.5.0 release, users now can associate a pegasus
   profile key data.configuration per site in the site catalog to
   specify the data configuration mode (sharedfs, nonsharedfs or
   condorio) to use for jobs executed on that site.  Earlier this was
   a global configuration, that applied to the whole workflow and had
   to be specified in the properties file.  
 
   More details at 
   https://jira.isi.edu/browse/PM-810
 
6) Support for sqlite JDBCRC

   Users can now specify a sqlite backend for their JDBCRC replica
   catalog. To create the database for the sqlite based replica
   catalog, use the command pegasus-db-admin 

   pegasus-db-admin create jdbc:sqlite:/shared/jdbcrc.db

   To setup Pegasus to use sqlite JDBCRC set the following properties
 
   pegasus.catalog.replica JDBCRC
   pegasus.catalog.replica.db.driver sqlite
   pegasus.catalog.replica.db.url  jdbc:sqlite:/shared/jdbcrc.db

   Users can use the tool pegasus-rc-client to insert, query and
   delete entires from the catalog.
 
7) New database management tool called pegasus-db-admin

   Depending on configuration, Pegasus can refer to three different
   types of databases during the various stages of workflow planning
   and execution. 

   master - Usually a sqlite database located at
   $HOME/.pegasus/workflow.db. This is always populated by
   pegasus-monitord and is used by pegasus-dashboard to track users
   top level workflows. 

   workflow - Usually a sqlite database created by pegasus-monitord in
   the workflow submit directory. This contains detailed information
   about the workflow execution. 

   jdbcrc   - if a user has configured a JDBCRC replica catalog.

   The tool is automatically invoked by the planner to check for
   comaptibility and updates the master database if required. The
   jdbcrc is checked if a user has it configured at planning time or
   when using the pegasus-rc-client command line tool.  

   This tool should be used by users, when setting up new database
   catalogs, or to check for compatibility. For more details refer to
   the migration guide at
   https://pegasus.isi.edu/wms/docs/4.5.0cvs/useful_tips.php#migrating_from_leq44 

8) pegasus-kickstart allows for system calls interposition

   pegasus-kickstart has new options -z and -Z that get enabled for
   linux platforms. When enabled, pegasus-kickstart captures
   information about the files opened and I/O for user applications
   and includes it in the proc section of it's output. This -z flag
   causes kickstart to use ptrace() to intercept system calls and
   report a list of files accessed and I/O performed. The -Z flag
   causes kickstart to use LD_PRELOAD to intercept library calls and
   report a list of files accessed and I/O performed.  
 
9) pegasus-kickstart now captures condor job id and LRMS job ids

   pegasus-kickstart now captures both the condor job id and the local
   LRMS  ( the system through which  the job is executed) in the
   invocation record for the job. 
 
	https://jira.isi.edu/browse/PM-866
	 https://jira.isi.edu/browse/PM-867
 
10) pegasus-transfer has support for SSHFTP 

   pegasus-transfer now has support for GridFTP over SSH . More
   details at
   https://pegasus.isi.edu/wms/docs/4.5.0/transfer.php#idp17066608 
 
11) pegasus-s3 has support for bulk deletes

    pegasus-s3 now supports batched deletion of keys from a S3
    bucket. This improves the performance for deleting keys from a
    large bucket.  
 
    https://jira.isi.edu/browse/PM-791

 
12) DAGMan metrics reporting enabled 
 
    Pegasus workflows now have DAGMan metric reporting capability
    turned on. Details on Pegasus usage tracking policy can be found
    at https://pegasus.isi.edu/wms/docs/4.5.0/usage_statistics.php 

    As part of this effort the planner now invokes condor_submit_dag
    at planning time to generate the DAGMan submit file, that is then
    modified to enable metrics reporting.  
 
    More details at https://jira.isi.edu/browse/PM-797

 
13) Planner reports file distribution counts in metrics report

    The planner now reports file distribution counts ( number of
    input, intermediate and output files) in it's metrics report . 
 
 
14) Notion of scope for data reuse
 
    Users can now enable partial data reuse, where only output files
    of certain jobs are checked for existence in the replica catalog,
    to trigger data reuse. Three scopes are supported 

       full - full data reuse as is implemented in 4.4 
       none - no data reuse i.e same as --force option to the planner 
       partial - in this case, only certain jobs ( those that have
       pegasus profile key enable_for_data_reuse set to true )are
       checked for presence of output files in the replica catalog 
 
15) New tool called pegasus-submitdir
 
    There is a new tool called pegasus-submitdir that allows users to
    archive, extract , move and delete a workflow submit
    directory. The tool ensures that master database ( usually in
    $HOME/.pegasus/workflow.db) is updated accordingly. 
 
16) New tool called pegasus-halt

    There is a new tool called pegasus-halt , that allows users to
    gracefully halt running workflows. The tool places DAGMan .halt
    files
    (http://research.cs.wisc.edu/htcondor/manual/v8.2/2_10DAGMan_Applications...)
    for all dags in a workflow. 
 
    More details at https://jira.isi.edu/browse/PM-702
 
17) New tool called pegasus-graphviz
 
    Pegasus now has a tool called pegasus-graphviz that allows you to
    visualize the DAX and DAG files. It creates a dot file as output .  
 
18) New canonical executable pegasus-mpi-keg

    New executable called pegasus-mpi-keg that can be compiled from
    source. Useful for creating synthetic workflows containing MPI
    jobs. It is similar to pegasus-keg and accepts the same command
    line arguments. The only difference is that it is MPI code. 
 
19) Change in default values

    By default, pegasus-transfer now launches maximum of 8 threads to
    manage the transfers of multiple files. 
    The default  job retries for a job in case of failure is now 1
    instead of 3.  
    The time for removing the job after has entered the HELD state has
    been reduced from 1 hour to 30 minutes now. 
 
20) Support for DAGMan ABORT-DAG-ON feature

    Pegasus now supports a dagman profile key named ABORT-DAG-ON ,
    that can be associated with a job. This job can then cause the
    whole workflow to be aborted if it fails or exits with a specific
    value.  
    
    More details at https://jira.isi.edu/browse/PM-819 
 
21) Deprecated pool attribute in replica catalog
    
    Users now can associate a site attribute in their file based
    replica catalogs to indicate the site where a file resides. The
    old attribute pool has been deprecated. 

    More details at https://jira.isi.edu/browse/PM-813
 
22) Support for pegasus profile glite.arguments
 
    Users can now specify a pegasus profile key glite.arguments that
    gets added to corresponding PBS qsub file that is generated by the
    Glite layer in HTCondor. For e.g you can set the value to "-N
    testjob -l walltime=01:23:45 -l nodes=2" . This will get
    translated to the following in the PBS file  
    #PBS -N testjob -l walltime=01:23:45 -l nodes=2 
 
    The values specified for this profile,  override  any other
    conflicting directives that are created on the basis of the globus
    profiles associated with the jobs. 
     
    More details at https://jira.isi.edu/browse/PM-880
 
23) Reorganized documentation

    The userguide has been reorganized to make it easier for users to
    identify the right chapter they want to navigate to. The
    configuration documentation has been streamlined and put into a
    single chapter, rather than having a separate chapter for profiles
    and properties. 
 
24) Support for hints namespace

    Users can now specify the following hints profile keys to control
    the behavior of the planner. 
 
     execution.site   - the execution site where a job should execute
     pfn                    - the path to the remote executable picked up
     grid.jobtype      - the job type to be used while selecting the
     gridgateway / jobmanager for the job 

     More details at https://jira.isi.edu/browse/PM-828
 
25) Added support for HubZero Distribute job wrapper

    Added support for HubZero specific job launcher Distribute, that
    submits jobs to a remote PBS cluster. The compute jobs are setup
    by Pegasus to run in local universe, and are wrapped with
    Distribute job wrapper, that takes care of the submission and
    monitoring of the job. More details at
    https://jira.isi.edu/browse/PM-796  
 
26) New classad populated for dagman jobs

    Pegasus now popualtes a +pegasus_execution_sites classad in the
    dagman submit file. The value is the list of execution sites for
    which the workflow was planned for. 
 
    More details at https://jira.isi.edu/browse/PM-846

 
27) Python DAX API now bins the file by link type when rendering the
workflow

    Python DAX API now groups the jobs by their link type before
    rendering them to XML.  This improves the readability of the 
    generated DAX. 
 
    More details at https://jira.isi.edu/browse/PM-874

 
28) Better demarcation of various stages in PegasusLite logs
 
   The jobs .err file in PegasusLite modes captures the logs from the
   PegasusLite wrapper that launches users jobs on remote nodes. This
   log is now clearly demarcated to  identify the various stages of a
   job execution by PegasusLite. 

29) Dropped support for Globus RLS replica catalog backends

30) pegasus-plots is deprecated and will be removed in 4.6

Bugs Fixed
-------------

1) Fixed kickstart handling of environment variables with quotes 
 
   If an environment variable has quotes, then invalid XML output was
   produced by pegasus-kickstart. This is now fixed. More details at  

   https://jira.isi.edu/browse/PM-807
 
2) Leaking file descriptors for two stage transfers  
 
   pegasus-transfer opens a temp file for each two stage transfer it
   has to execute. It was not closing them explicitly.  
 
3) Disabling of chmod jobs triggered an exception

   Disabling the chmod jobs results in creation of noop jobs instead
   of the chmod jobs. However, that resulted in planner exceptions
   when adding create dir and leaf cleanup nodes. This is now fixed.  
 
   More details at  https://jira.isi.edu/browse/PM-845
 
4) Incorrect binning of file transfers amongst transfer jobs 

   By default, pair only considered the destination URL of a transfer
   pair to determine whether the associated transfer job has to run
   locally on the submit host or on the remote staging site. However,
   this logic broke when user had input files catalogued in the
   replica catalog with file urls for files on the submit site and
   remote execution sites. The logic has now been updated to take into
   account source URL's also.  
 
   More details at  https://jira.isi.edu/browse/PM-829

5) pegasus auxillary jobs are never lauched with pegasus-kickstart invoke capability
 
   For compute jobs with long command line arguments , the planner
   triggers the pegasus invoke capability in addition to the -w
   option. However, this cannot be applied to pegasus auxillary jobs
   as that interferes with the credential handling.  
 
   More details at  https://jira.isi.edu/browse/PM-851

 
6) Everything in the remote job directory gets staged in condorio
mode, if a job has no output files 
 
   If a job has no output files asscociated with it in the DAX, then
   in condorio data configuration mode the planner added an empty
   value for classad key transfer_output_files in the job submit
   file. This results in Condor staging back all the inputs ( all the
   contents in remote jobs directory) back to the submit host. This is
   now fixed as the planner now adds a special key +TransferOutput=""
   , that prevents Condor from staging everything back. 
 
   More details at  https://jira.isi.edu/browse/PM-820

7) Setting multiple strings for exitcode.successmsg and exitcode.failuremsg
 
   Users can now specify multiple pegasus profiles with the key
   exitcode.successmsg or exticode.failuremsg. Each value gets
   translated to a corresponding -s or -f argument to pegasus-exitcode
   invocation for the job. 
 
   More details at  https://jira.isi.edu/browse/PM-826
 
8) pegasus-monitord failed when submission of job fails
 
   The events  SUBMIT_FAILED, GRID_SUBMIT_FAILED, GLOBUS_SUBMIT_FAILED
   were not handled correctly by pegasus-monitord. As a result,
   subsequent event insertions for the job resulted in integrity
   errors. This is now fixed. 
 
   More details at  https://jira.isi.edu/browse/PM-877
 
===============================
Release Notes for PEGASUS 4.4.2
===============================


We are happy to annouce the release of Pegasus 4.4.2. Pegasus 4.4.2 is
a minor release, which contains minor enhancements and fixes bugs to
Pegasus 4.4.1 release.  
 
Enhancements
--------------

1) Support for recursive clustering 

   Pegasus now supports recursive clustering, where users can employ
   multiple clustering techniques on the same graph. For example a
   user can do label based clustering on the graph, and then do a
   level based clustering. 

   More details at
   https://jira.isi.edu/browse/PM-817
 
2) Planner reports file breakdowns in the metrics sent to the metrics
   server. 

   The planner now send file breakdowns ( number of input,
   intermediate and output files ) as part of the metrics message to
   the metrics server. This is also reported in the metrics file left
   in the submit directory. 
 
3) pegasus-transfer does not hide scp errors. 
 
4) more helpul message is thrown if user does not set
pegasus.catalog.site.file property 

Bugs Fixed
--------------

1) work dir in job instance table was populated incorrectly

   The work directory in the job instance table of the monitoring
   database was populated by the submit directory instead of the
   directory in wihch the job was executed. This resulted in
   pegasus-analyzer displaying the submit directory for the failed job
   instead of the directory in which the job actually ran on the
   remote node. 
 
   More details at
   https://jira.isi.edu/browse/PM-817
 
2) pegasus-status showed master dag job as failure also

   When a job in a workflow fails, pegasus-status also includes the
   corresonding dag job as failed. This leads to it reporting 1 more
   than the actual number of user compute jobs that failed. this is
   now fixed. 

   More details at
   https://jira.isi.edu/browse/PM-811 
 
3) local-scratch directory not picked up for PegasusLite jobs

   Users can specify a local-scratch directory in the site catalog for
   a site, to designate the local directory on the worker node where a
   PegasusLite job should be run. However, this was not picked up by
   the planner and set for the jobs. This is now fixed. This only
   works when user is executing workflows in nonsharedfs mode. 
 
4) pegasus dashboard tables were not updated in real time. 
 
   Fixed off by 1 error in the flush command, where we decide whether
   we want to flush the event to the database or batch them up. This
   one off error affected the pegasus dashboard as the workflow start
   and end events were not updated to the database when they happened
   by pegasus-monitord. this is now fixed. 

 
5) Input files in the DAX where transfer flag is set to false, should
   not be considered for cleanup, as they are never staged to scratch 
   directory on the staging site by the stage in jobs 
 
6) pegasus.gridstart.arguments was not set for all clustered jobs. 

   User provided extra arguments for kickstart invocation were not
   passed to all the constitutent jobs making up a job cluster, in
   case of job clustering.  This is now fixed.  
 
   More details at https://jira.isi.edu/browse/PM-823
 
7) MPI_ERR_TRUNCATE: message truncated in PMC

   This error was encountered in certain conditions and was a result
   of  mismatched tag/source between an MPI_Probe/MPI_Iprobe and
   MPI_Recv. This is now fixed 

   More details at https://jira.isi.edu/browse/PM-848
 
8) Setting pegasus.catalog.site XML4 raised an error 

   Introduced backward compatibility for this.
   More details at https://jira.isi.edu/browse/PM-815
 
9) pegasus-plan --help resulted in metrics to be sent.
   
   This is now fixed
   More details at https://jira.isi.edu/browse/PM-816

===============================
Release Notes for PEGASUS 4.4.1
===============================

We are happy to annouce the release of Pegasus 4.4.1. Pegasus 4.4.1 is
a minor release, which contains minor enhancements and fixes bugs to
Pegasus 4.4.0 release.  
 
Enhancements 
--------------

1) Leaf cleanup job failures don't trigger workflow failures
 
2) Finer grained capturing of GridFTP errors 

   Moved to only ignore common failures of GridFTP removals, instead
   of ignoring all errors 
 
3) pegasus-transfer threading enhancements

   Allow two retrie with threading before falling back on
   single-threaded transfers. This prevents pegasus-transfer from
   overwhelming remote file servers when failures happen. 

4) Support for MPI Jobs when submitting using Glite to PBS

   For user specified MPI jobs in the DAX, the only way to ensure that
   the MPI job launches in the right directory through GLITE and blahp
   is to have a wrapper around the user mpi job and refer to that in
   the transformation catalog. The wrapper should cd in to the
   directory set by Pegasus in the job's environment. The following
   environment variable is set _PEGASUS_SCRATCH_DIR 
 
5) Updated quoting support for glite jobs
 
   Quoting in the blahp layer in Condor for glite jobs is
   broken. There were fixes made to the planner and
   pbs_loca_submit_attributes.sh files such that env. var values can
   contain spaces or double quotes.  
   
   The fix relies on users to put the pbs_local_submit_attributes.sh
   from the pegasus distribution to the condor glite bin directory.  

   More details at https://jira.isi.edu/browse/PM-802
 
6) pegasus-s3 now has support for copying objects larger than 5GB
 
7) pegasus-tc-converter code was cleaned up . support for database
    backed TC was dropped. 
 
8) The planner now complaisn for deep LFN's when using condor file transfers
 
9) The planner stack trace is enabled for errors with a single -v (
i.e INFO messagae level or higher)  
 
More details at https://jira.isi.edu/browse/PM-800
 

Bugs Fixed
--------------
 
1) Change in  how monitord parses job output and error files

   Earlier pegasus-monitord had a race condition, at it tried to parse
   the .out and .err file when a JOB_FAILURE or JOB_SUCCESS happened,
   instead of doing it at POST_SCRIPT_SUCCESS or POST_SCRIPT_FAILURE
   message, if a postscript was associated . This resulted in it
   detecting empty kickstart output files, as postscript might have
   moved it before monitord opened a file handle to it. The fix for
   this , changed the monitord logic to parse files on JOB_FAILURE  or
   JOB_SUCCESS only if postscript is not associated with the job 
 
   More details at https://jira.isi.edu/browse/PM-793

2) pegasus-monitord did not handle aborted jobs well

   For aborted jobs that failed with signal,  monitord did not parse
   the  job status . Because of that no corresponding JOB_FAILURE was
   recorded, and hence the exitcode for the inv.end event is not
   recorded.  

   https://jira.isi.edu/browse/PM-805

3) A set of portability fixes from the Debian packaging were
incorporated into pegasus builds. 
 
4) Clusters of size 1 should be allowed when using PMC

   An earlier fix for 4.4.0 allowed single jobs to be clustered using
   PMC. However, this resulted in regular MPI jobs that should not be
   clustered, to be clustered also using PMC. The logic was updated to
   only wrap a single job with PMC if label based clustering is turned
   on and the job is associated with a label.  

   More details at https://jira.isi.edu/browse/PM-745
 
5) Round robin site selector did not do correct distribution
 
   The selector was not distributing the jobs round robin at each
   level as it was suppposed to. 
 More details at https://jira.isi.edu/browse/PM-775
 
6) Based on user configuration, the leaf cleanup jobs tried to delete the
submit directory for the workflow  

   A user can configure a workflow such that the workflow submit
   directory and the workflow scratch directory are the same on local
   site. This can result in stuck workflows if the leaf cleanup jobs
   are enabled. The planner now throws an error during planning if it
   detects the directories are the same 
 
   More details at https://jira.isi.edu/browse/PM-773
 
7) pegasus-cleanup needs to add wildcards to s3:// URLs when
--recursive is used
   
   More details at https://jira.isi.edu/browse/PM-790
 
8) leaf cleanup jobs delete directory that a workflow corresponding to
dax job may require 

    For hierarchical workflows, there maybe a case where the jobs that
    make up the workflow referred to by the subdax job may run in a
    child directory of the scratch directory in whcih jobs of top
    level worklfow are running. With leaf cleanup enabled, the parent
    scratch directory maybe cleaned before the subdax job has been
    completed. Fix for this involved, putting in explicit dependencies
    between the leaf cleanup job and the subdax jobs. 
 
    More details at https://jira.isi.edu/browse/PM-795

9)  pegasus-analyzer did not show planner prescript log for failed subdax jobs 

    For prescript failures for sub dax jobs ( i.e the failure of
    planning operation on the sub workflow ), pegasus-analyzer never
    showed the content of the log. It only pointed to the location of
    the log in the submit directory. This is now fixed. 

    https://jira.isi.edu/browse/PM-808

10)  pegasus-analyzer shows job stderr for failed pegasus-lite jobs

    When a Pegasus Lite job fails, pegasus-analyzer showed stderr from
    both the Kickstart record and the job stderr. This was pretty
    confusing as stderr for those jobs are used to log all kinds of
    PegasusLite stuff, and has usually nothing to do with the
    failure. To make these jobs easier to debug for our users, we
    added logic to only show the Kickstart stderr in these cases.  
 
    More details at https://jira.isi.edu/browse/PM-798
 
11) Planner did not validate pegasus.data.configuration value.
 
    AS a result, because of a typo in the properties file planner failed with NPE. 
    More details at https://jira.isi.edu/browse/PM-799

12) pegasus-statistics output padding

    Value padding is done only for text output files so they are human
    readable. However, due to a bug the value padding computation were
    being done for CSV file as well at one point in code. This caused an
    exception when output filetype for job statistics was csv 


===============================
Release Notes for PEGASUS 4.4.0
===============================

We are happy to announce the release of Pegasus 4.4.0
 
Pegasus 4.4.0 is a major release of Pegasus which contains all the enhancements and bugfixes in 4.3.2
 
New features and Improvements in 4.4.0 include
    - substantial performance improvements for the planner for large workflows
    - leaf cleanup jobs in the workflow
    - new default transfer refiner
    - abitlity to automatically add data flow dependencies
    - new mode for runtime clustering
    - pegasus-transfer is now multithreaded
    - updates to replica catalog backends
 
 
New Features
--------------

1) Improved Planner Performance
 
   This release has major performance improvements to the planner that 
   should help in planning larger DAX'es than earlier. Additionally,
   the planner can now optionally log JAVA HEAP memory usage on the
   INFO log at the end of the planning process, if the property
   pegasus.log.memory.usage is set to true.  
 
2) Leaf Cleanup Jobs
 
   Pegasus now has a new cleanup option called Leaf that adds a leaf 
   cleanup jobs symmetric to the create dir jobs. The leaf cleanup
   jobs remove the directory from the staging site that the create dir
   jobs create at the end of the workflow. The leaf cleanup is turned
   on by passing --cleanup Leaf to pegasus-plan. 

   Care should be taken while enabling this option for hierarchal
   workflows. Leaf cleanup jobs will create problems, if there are data 
   dependencies between sub workflows in a hierarchal workflow. In
   that  case, the cleanup option needs to be explicitly set to None
   for the  pegasus-plan invocations for the dax jobs in the hierachal
   DAX.  
 
3) New Default Transfer Refiner
 
   This release has a new default transfer refiner called
   BalancedCluster that does round robin distribution at the file
   level instead of the job level, while creating clustered stagein
   and  stageout jobs. This refiner by default adds two stagein and two
   stageout jobs per level of the workflow. 
 
4) Planner can automatically infer and data flow dependencies in the DAG 
 
   The planner can now automatically add dependencies on the basis of 
   data dependencies implied by input and output files for jobs. For 
   example if Job A creates an output file X and job B consumes it,
   then  the planner should automatically add a dependency between A
   -> B if it does not exist already. 

   This feature is turned on by default and can be turned off by
   setting  the property pegasus.parser.dax.data.dependencies to
   false. More  details at https://jira.isi.edu/browse/PM-746  
 
5) Update to Replica Catalog Backends
 
   The replica catalog backends ( File, Regex and JDBCRC) have been 
   updated to consider lfn, pfn mapping but with different pool/handle  
    as different entries. 

   For the JDBCRC the database schema has been updated. To migrate
   your  existing JDBCRC backend, users are recommended to use the 
   alter-my-rc.py script located into 'share/pegasus/sql' to migrate
   the database. 
 
   Note that you will need to edit the script to update the database 
    name, host, user, and password. Details at
    https://jira.isi.edu/browse/PM-732  
 
6) Improved Credential Handing for data transfers

   In case of data transfer jobs, it is now possible to associate 
   different credentials for a single file transfer ( one for the 
   source server and the other for the destination server) . For
   example, when leveraging GridFTP transfers between two sides that 
   accept different grid credentials such as  XSEDE Stampede site and 
   NCSA Bluewaters. In that case, Pegasus picks up  the associated 
   credentials from the site catalog entries for the source   and the
   destination sites associated with the transfer. 

   Also starting 4.4, the credentials should be associated as Pegasus 
   profiles with the site entries in the site catalog, if you want
   them  transferred with the job to the remote site. 

   Details about credential handling in Pegasus can be found here
   https://pegasus.isi.edu/wms/docs/4.4.0cvs/reference.php#cred_staging
 
   Associated JIRA item for the improvement
   https://jira.isi.edu/browse/PM-731
 
   The credential handling support in pegasus-transfer,
   pegasus-createdir and pegasus-cleanup were also updated 
 
7) New mode for runtime clustering 
   
   This release has a new mode added for runtime clustering.
 
   Mode 1: The module groups tasks into clustered job such that no
   clustered job runs longer than the maxruntime input parameter to
   the module.  

   Mode 2(New): New mode now allows users to group tasks into a fixed
   number of clustered jobs. The module distributes tasks evenly
   (based on job runtime) across jobs, such that each clustered job
   takes approximately the same time. This mode is helpful when users
   are aware of the number of resources available to them at the time
   of execution. 
 
8) pegasus-transfer is now threaded

   pegasus-transfer is now multithreaded.  Pegasus exposes two knobs
   to control the number of threads pegasus-transfer can use depending
   on whether  you want to control standard transfer jobs, or you want
   to control transfers that happen as a part of a PegasusLite job
   . For the former, see the pegasus.transfer.threads property, and
   for the latter the pegasus.transfer.lite.threads property. For
   4.4.0 pegasus.transfer.threads defaults to 2 and
   pegasus.transfer.lite.threads defaults to 1.  
 
9) pegasus-analyzer recurses into subworkflows
 
   pegasus-analyzer has a --recurse option that sets it to
   automatically recurse into failed sub workflows.  By default, if a
   workflow has a sub workflow in it, and that sub workflow fails ,
   pegasus-analyzer reports that the sub workflow node failed, and
   lists a command invocation that the user must execute to determine
   what jobs in the sub workflow failed. If this option is set, then
   the analyzer automatically issues the command invocation and in
   addition displays the failed jobs in the sub workflow. 

   Details  at https://jira.isi.edu/browse/PM-730
 
10) Support for Fixed Output Mapper
 
   Using this output mapper, users can specify  an externally
   accessible URL in the properties file, pointing to a directory
   where the output files needs to be transferred to. To use this
   mapper, set the following  properties  
 
   pegasus.dir.storage.mapper Fixed
   pegasus.dir.storage.mapper.fixed.url  <url to the storage directory
   e.g. gsiftp://outputs.isi.edu/shared/outputs> 
 
11) Extra ways for user application to flag errors

   CondorG does not propogate exitcodes correctly from GRAM. As a
   result, a job in a Pegasus workflow that is not launched via
   pegasus-kickstart maynot have the right exitcode propogated from
   user application -> GRAM -> CondorG -> Workflow.  For example, in
   Pegasus MPI jobs are never launched using
   pegasus-kickstart. Usually ways of handling this error is to have a
   wrapper script that detects failure and then having the postscript
   fail on the basis of the message logged.  

   Starting 4.4.0, Pegasus provides a mechanism of logging something
   on stdout /stderr that can be used to designate failures. This
   obviates the need for users to have a wrapper script. Users can
   associate two pegasus profiles with the jobs 

   exitcode.failuremsg -The message string that pegasus-exitcode
   searches for in the stdout and stderr of the job to flag failures. 
   exitcode.successmsg - The message string that pegasus-exitcode
   searches for in the stdout and stderr of the job to determine
   whether a job logged it's success message or not. Note this value
   is used to check for whether a job failed or not i.e if this
   profile is specified, and pegasus-exitcode DOES NOT find the string
   in the job stdout or stderr, the job is flagged as failed. The
   complete rules for determining failure are described in the man
   page for pegasus-exitcode. 

   More details at http://jira.isi.edu/browse/PM-737
 
12) Updated examples for Glite submission directly to local PBS 
    
   The 4.4.0 release has improvements for the submission of workflows
   directly to local PBS using the Condor Glite interfaces. The
   documentation on how to use this through Pegasus is documented at 
 
   http://pegasus.isi.edu/wms/docs/4.4.0/execution_environments.php#glite 

   It is important to note that to use this, you need to use the
   pbs_local_attributes.sh file shipped with Pegasus in the
   share/pegasus/htcondor/glite directory and put in the glite bin
   directory of your condor installation. 
 
   Additionally, there is a new example in the examples directory that
   illustrates how to execute an MPI job using this submission
   mechanism through Pegasus. 
 
13) Finer grained specification of linux versions for worker package
    staging 
 
   Planner now has added logic for users to specify finer grained
   linux versions to stage the worker package for .  

   Users can now specify in the site catalog the osrelease and
   osversion attributes e.g. 

   <site handle="exec-site" arch="x86_64" os="LINUX" osrelease="deb" osversion="7">
 
   If a supported release version combination is not specified, then
   planner throws a warning and defaults to the default combination
   for the OS. 

   More details at https://jira.isi.edu/browse/PM-732
 
14) pegasus-kickstart can now copy all of applications stdio if -b all
is passed 
 
   Added an option to capture all stdio. This is a feature that
   HUBzero requested. Kickstart will now copy all stdout and stderr of
   the job to the invocation record if the user specifies '-B all'. 
 
15) Tutorial includes pegasus-dashboard

    The tutorial comes configured with pegasus-dashboard.
 
16) Improved formatting of extra long values for pegasus-statistics
   More details at  https://jira.isi.edu/browse/PM-744
 
17) Changed timeout parameters for pegasus-gridftp
 
   Increased the timeout parameter for GridFTPClient to 60
   seconds. The globus jar defaults to 30 seconds. The timeout was
   increased to ensure that transfers don't fail against heavliy
   loaded GridFTP servers. 

Bugs Fixed
--------------

1) IRODS support in pegasus-transfer , pegasus-createdir was broken

   irods mkdir command got the wrong path when invoked by
   pegasus-transfer. this is now fixed 
 
2) Data reuse algorithm does not cascade the deletion upwards 

   In certain cases, the cascading of deletion in data reuse did not
   happen completely. This is now fixed.  More details at
   https://jira.isi.edu/browse/PM-742 
 
3) Improved argument management for PMC
 
   This was done to address the case where a task has quoted arguments
   with spaces. 
 
4) Clusters of size 1 should be allowed when using PMC

   For label based clustering with PMC single node clusters are
   allowed. This is important as in some cases, PMC jobs might have
   been set to work with the relevant globus profiles. 
 
   https://jira.isi.edu/browse/PM-745
 
5) nonascii characters in application stdout broke parsing in monitord 
 
   The URL quoting logic was updated to encode unicode strings  as
   UTF-8 before the string was passed to the quote fuction. More
   details at  

    https://jira.isi.edu/browse/PM-757
 
6) Removing a workflow using pegasus-remove does not update the
stampede database 
   
   If you remove a running workflow, using pegasus-remove, the
   stampede database is not updated to reflect that the workflow
   failed. Changes were made to pegasus-dagman to ensure that
   pegasus-monitord gets 100 seconds to complete the population before
   sending a kill signal. 
 
7) Translation of values from days to years/days was borken in
pegasus-statistics 

   This is now fixed.
		 

===============================
Release Notes for PEGASUS 4.3.2
===============================
We are happy to annouce the release of Pegasus 4.3.2. Pegasus 4.3.2 is
a minor release, that has minor enhancements and fixes bugs to Pegasus
4.3.1 release.  
 
Enhancements
--------------

1) Better error recording for PegasusLite failures in the monitoring
   database

   If a  PegasusLite job failed because of an error encountered while
   retrieving hte input files from the staging site, then no kickstart
   record for the main user job is generated ( as the job is never
   launched). As a result, in the database no record is populated
   indicating a failure of the job. This was fixed to ensure that
   monitord now populates an invocation recording containing the error
   message from the err file of the PegasusLite job.  

   More details can be found at
   https://jira.isi.edu/browse/PM-733
 
2) PMC Changes

   By default PMC now clears the CPU mask using sched_setaffinity. If
   libnuma is available, it also resets the NUMA memory policy using
   set_mempolicy. If the user wants to keep the inherit
   affinity/policy, then they can use the --keep-affinity argument.

   More details can be found at
   https://jira.isi.edu/browse/PM-735
 
   The number of open files tracked in the internal file descriptor
   cache was decreased from 4096 to 256. Also if an error is
   encountered because, the fd limit is exceeded on a system then PMC
   logs the number of file descriptors it has open helping the user
   identify the number of FD's open by PMC. 
 
3) pegasus-transfer changes 

   pegasus-transfer checks in the local cp mode to ensure that src and
   dst is not the same file.  
   pegasus-transfer sets the -fast option by default to GUC  for 3rd
   party gsiftp transfers 
 
4) pegasus-status changes 
 
   Minor fix for when the parent dag disappears before the job (can
   happen for held jobs)  

 
5) Changes to java memory settings

   The pegasus-plan wrapper script takes into account ulimit -v
   settings while determining the java heap memory for the planner. 
 
 
Bugs Fixed
--------------

1) Symlinking in PegasusLite against SRM server

   In the case, where the data on the staging server is directly
   accessible to the worker nodes it is possible to enable symlinking
   in Pegasus that results in PegasusLite to symlink the data against
   the data on the staging site. When this was enabled, the source URL
   for the symlink transfer referred to a SRM URL resulting in pegasus
   lite doing a duplicate transfer. The planner needed to be changed
   to resolve the SRM URL to a file URL that is visible from the
   worker node.  

   Also the planner never symlinks the executable files in Pegasus
   Lite as it can create problems with the setting of the x bit on the
   executables staged. For executable staging to work, the executable
   need to be copied to the worker node filesystem. 

   More details can be found at
   https://jira.isi.edu/browse/PM-734
 
2) The input file corresponding to the DAX for the DAX jobs was not
   associated correctly when the planner figures out the transfers
   required for the DAX job. This happened, if the DAX job only referred
   to the DAX file as an input file and that was generated by a parent
   dax generation job in the workflow. 
 
3) File dependency between a compute job and a sub workflow job
 
   The planner failed while planning a dax job for an input file, that
   a parent job of the corresponding DAX job generated. This is now
   fixed as the cache file for the parent workflow is passed to the
   sub workflow planner invocations. 
 
   More details can be found at
   https://jira.isi.edu/browse/PM-736
 
4) Error when data reuse and cleanup is enabled

   The planner failed in the case, where cleanup was enabled in
   conjuction with data reuse, where jobs removed by the data reuse
   algorithm were the ones for which output files are required by the
   user on the output site. In that case, the planner adds stageout
   jobs to stage the data from the location in the replica catalog to
   the output site. The addition of this stageout job was resulting in
   an execption in the cleanup module. This is now fixed. 
 
   More details can be found at
   https://jira.isi.edu/browse/PM-739
 
5) pegasus-analyzer not reporting the planner prescript log for failed
sub workflows 
 
   In the case, where a workflow fails because the planner invoked in
   the prescript for the sub workflow failed, pegasus-analyzer did not
   point the user to the planner log file for the sub workflow. This
   is now fixed.  
 
   More details can be found at
   https://jira.isi.edu/browse/PM-704

===============================
Release Notes for PEGASUS 4.3.1
===============================
We are happy to announce the release of Pegasus 4.3.1 .  

Pegasus 4.3.1 is a minor release, that has minor enhancements and fixes bugs to Pegasus 4.3.0 release. 
 
Enhancements:
--------------

1) Support for Fixed Output Mapper

   Using this output mapper, users can specify  an externally
   accessible URL in the properties file, pointing to a directory
   where the output files needs to be transferred to. To use this
   mapper, set the following  properties  
 
	pegasus.dir.storage.mapper Fixed
	pegasus.dir.storage.mapper.fixed.url  <url to the storage directory e.g. gsiftp://outputs.isi.edu/shared/outputs>
 

Bugs Fixed:
-------------- 

1) pegasus-analyzer does not detect jobs that are condor_rm'ed if no
postscript is associated with the job 

   By default, each job has a postscript associated that detects an
   empty job stdout and flags it as a failure. However, if a job is
   not asscociated with a postscript and a user/system condor_rm's the
   job, the failure is not detected. This is now fixed, and a
   JOB_ABORTED event is logged in the stampede database, when a job is
   aborted. 
 
2) IRODS support in pegasus-transfer

   The IRODS support in pegasus-transfer was broken. This is now fixed.
 
3) pegasus-kickstart compilation warnings for character encodings

   kickstart maintains a table to escape characters correctly for
   putting them in a XML document. The non-ascii characters in the
   table were latin1 not UTF-8. This caused a warning on newer
   versions of gcc, which could not be disabled across all platforms.
   kickstart now writes out it;s output in UTF-8 encoding and the xml
   escaping was updated accordingly. 
 
4) Fix to URL handling in the planner

   Changed the regex in PegasusURL so that we can pass urls with just
   the hostnames specified. e.g http://isis.isi.edu Note: no trailing
   / . Before the fix, the planner was throwing an exception if a user
   specified an input URL with path names containing only one
   directory. 
 
5) planner had a rogue debug statement

   There was a rogue system.out statement in the planner output that
   led to a statement being logged for each job in the workflow. 
 
6) pegasus-statistics had 2.6 style code

   Pegasus is distributed as part of OSG software stack, and one of
   the supported platforms there is EL5 systems that come with Python
   2.4. pegasus-statistics had some code that was compatible with
   2.5. This is now fixed.

===============================
Release Notes for PEGASUS 4.3
===============================
We are happy to announce the release of Pegasus 4.3.  

Pegasus 4.3 is a major release of Pegasus which contains all the
enhancements and bugfixes in 4.2.2 release 

New features and Improvements in 4.3 include

    - improvements to pegasus lite and optimizations for input file
    staging in non shared filesystem deployments 
    - support for output mappers, allowing users finer grained control
    over where to place the outputs on an output site 
    - support for SSH based submissions on top of Condor BOSCO. 
    - substantial improvements to pegasus-kickstart including abiltiy
    to track peak memory usage for jobs 
    - improvements to pegasus-s3 and pegasus-transfer


NEW FEATURES
--------------

1) Support for bypassing transfer of input files via the staging site 

   In the non shared filesystem deployments (
   pegasus.data.configuration = nonsharedfs|condorio) users, can now
   setup pegasus to transfer the input files directly to the worker
   nodes without going through the staging site.  This can be done by
   setting the following property to true 
  
   pegasus.transfer.bypass.input.staging

   In the nonsharedfs case, if the input files are already present on
   a shared disk accessible from the worker nodes, pegasus lite can
   symlink instead of copying them over to the local directory on the
   worker node.  The cleanup algorithm was updated to ignore files
   that are directly pulled to the worker nodes from the input site
   locations. 
 
2) Support for DAX generation jobs in hierarchal workflows

   Pegasus now has support for having a dax generation job in the
   workflow. This allows users to add long running dax generation jobs
   as a compute job in the workflow, that can be run remotely. These
   dax generation jobs need to be a parent of the associated DAX
   job. Pegasus will ensure that the DAX generated on a remote site is
   brought to the local site for the associated sub workflow
   corresponding to the DAX job to be planned. 

   Earlier, the only way for hierarchal workflows was that the DAX'es
   for the sub workflows had to be pre generated and the paths to the
   dax files was specified in the DAX jobs.  Pegasus did not
   automatically handle separate DAX generations jobs out of the
   box. More details can be found at
   https://jira.isi.edu/browse/PM-667. There is an example checked in
   the share/pegasus/examples/dynamic-hierarchy directory. 
 
3) Support for output mappers
   
   Pegasus now has support for output mappers, that allow users fine
   grained control over how the output files on the output site are
   laid out.  The mappers can be configured by setting the following
   property  

    pegasus.dir.storage.mapper

   The following mappers are supported

   Flat: By default, Pegasus will place the output files in the
   storage directory specified in the site catalog for the output
   site. 

   Hashed: This mapper results in the creation of a deep directory
   structure on the output site, while populating the results. The
   base directory on the remote end is determined from the site
   catalog. Depending on the number of files being staged to the
   remote site a Hashed File Structure is created that ensures that
   only 256 files reside in one directory. To create this directory
   structure on the storage site, Pegasus relies on the directory
   creation feature of the Grid FTP server, which appeared in globus
   4.0.x  
 
   Replica: This mapper determines the path for an output file on the
   output site by querying an output replica catalog. The output site
   is one that is passed on the command line. The output replica
   catalog can be configured by specifiing the properties 
 
	pegasus.dir.storage.mapper.replica       Regex|File
	pegasus.dir.storage.mapper.replica.file  the RC file at the
	backend to use if using a file based RC 
 
4) Support for SSH based submissions

   Pegasus now exposes a ssh style to enable submission to remote
   sites using SSH. This builds upon the Condor BOSCO funtionality
   that allows for submission over ssh. 

   Check out the bosco-shared-fs example in the share/pegasus/examples
   directory for a sample site catalog and configuration. 

 
5) Support for JDBC based Replica Catalog
 
   Resurrected support for JDBC  backed Replica Catalog in
   Pegasus. Users can use pegasus-rc-client to interact with the JDBC
   backend. 
 
6) Reduced Dependencies for create dir jobs

   Pegasus earlier added edges between create dir jobs and all the
   compute jobs scheduled for that particular site. Pegasus now adds
   edges from the create dir job to a compute job only if a create dir
   job is not reachable from one of the parents of the job. This
   strategy is now the default for 4.3. 
 
7) New tool called pegasus-archive 

   Pegasus 4.3 has a new tool called pegasus-archive that compresses a
   workflow submit directory in a way that allows pegasus-dashboard,
   pegasus-statistics, pegasus-plots, and pegasus-analyzer to keep
   working. More information can be found in the manpage for the
   tool. 
 
8) pegasus-transfer enhancements 
 
   The internal pegasus-transfer tool was improved to do multi-hop
   staging in the case of two incompatible protocols being used for
   source and destination of a transfer. For example, if a workflow
   requires the transfer of a file from GridFTP to S3,
   pegasus-transfer will split the transfer up into two transfers:
   GridFPT->Local and Local->S3. This is transparent to the end-user
   and the Pegasus planner. 

 
9) pegasus-mpi-cluster enhancements
 
   Added a --maxfds to control size of FDCache. This argument to PMC
   that enables the user to set the maximum number of file descriptors
   that will be cached by PMC in I/O forwarding. This is to help SCEC
   accomplish coscheduling on BlueWaters. 
 
10) pegasus-kickstart can track peak memory usage for the jobs launched by it
 
   pegasus-kickstart now add per-pid I/O, memory and CPU usage.  These
   changes add one or more <proc> elements inside all of the <*job>
   elements. The new <proc> elements are only available on Linux
   systems with kernels >2.5.60 that support ptrace with exit
   events. The new <proc> element contains information about 

   	   - the peak memory usage of each child process,
	   - the start and end times of the processes,
	   - the number of characters and bytes read and written,
	   - the utime and stime, and the pid and parent pid. 
 
   This information can be used to compute the resource usage of a job.
 
11) pegasus-kickstart enhancements
 
   Added a -q option to reduce output . This new option omits the
   <data> part of the <statcall> recordsfor stdout and stderr if the
   job succeeds. If the job fails, then the <data> is included. 

   When kickstart is executed on a Linux kernel >= 3.0, logic in the
   machine extensions code of kickstart prevented the proc statistics
   gathering, because it was a reasonable assumption that the API
   might have changed (it did between 2.4 and 2.6). This restriction
   is now removed.  

   The behavior of the -B option was changed so that it grabs the last
   N bytes of stdio instead of the first N bytes of stdio if thesize
   of stdio is larger than the -B option. 

   The invocation record that kickstart writes out is now consistent
   with new invocation schema version 2.2 . This version adds the
   <proc> element under <*job>, and renames the <linux>/<proc> element
   to <linux>/<procs> to eliminate the name collision. 
   
   pegasus-kickstart now sets the max size of a single argument to
   128k instead of earlier 2048 characters, which appears to be the
   individual limit in Linux. If the total size of all the arguments
   is over the total limit, then execve will fail, so we don't try to
   catch that in the argument parser. 
 
12) pegasus-s3 enhancements
 
   The put -b/--create-bucket option was made more efficient. There is
   no need to check if the bucket exists before calling create_bucket
   because it is a noop if the bucket already exists.   
 
   pegasus-s3 does not rely on mmap for upload and download. This
   should reduces the memory usage of pegasus-s3 for large files. 
 
   Updated the boto version Boto 2.5.2 to better support multipart
   uploads.  

 
   Added upload rate info to put command output.

   pegasus-s3 now supports transfers from one s3 bucket to another.
 
13) pegasus-analyzer enhancements

   pegasus-analyzer earlier did not detect prescript failures. If a
   job's prescript failed ( for example the planner instance on a
   subworkflow for a hierarchal workflow ) , then that failure was
   not recorded in the monitoring database. This led pegasus-analyzer
   to not report the prescript failures.  Changes were made in the
   monitoring daemon to ensure those errors are detected and
   associated correctly in the database. More details can be found at
   https://jira.isi.edu/browse/PM-704 

   pegasus-analyzer can be used to debug pegasus lite workflows now
   using the --debug-job option.It facilitates the debugging of a
   failed pegasus lite job by creating a shell script that can be run
   locally.  The --debug-job option creates a shell script in a tmp
   directory that can be executed to pull in the input data and
   execute the job. It also now has a --local-executable option that
   can be used to pass to the local executable for the job that is
   being debugged. 
 
14) pegasus-statistics can generate statistics across multiple root
workflows

    pegasus-statistics now has a -m option to generate statitsics
    across multiple root workflows. User can pass either multiple
    workflow submit directories or workflow uuids separated by
    whitespace. 

    This feature is also useful is the runs for multiple root
    workflows are populated in the same database in mysql. 

    For e.g
    pegasus-statistics -Dpegasus.monitord.output=mysql://user:password@host:port/databasename  -s all -m 


15) pegasus-lite stages out output files in case of failure 
   In the nonsharedfs case, PegasusLite now always attempt to
   transfer the output files even if the main command of the script
   fails. 

   Details at https://jira.isi.edu/browse/PM-701
 
16) Directory backed Replica Catalog now supports flat lfn's
 
   By default the directory based replica catalog backend constructs
   deep lfn's while traversing an input directory.  

   For example, if input directory is points to a directory input then
   input/deep/f.a file results in LFN called deep/f.a   
 
   If a user sets,  pegasus.catalog.replica.directory.flat.lfn to true 
 
   then the leaf is only constructed for creating the lfn. 
   For example input/deep/f.a will result in lfn f.a
 
17) Updated jglobus jars and globus rls client jar

   Pegasus now ships with updated jglobus and globus rls client jars
   that allow us to use the proxies generated using newer certificates
   to authenticate against a RLS deployment. The RLS client jar
   shipped with pegasus  works with JGlobus 2.0.5. 
 
18) Updated proxy detection logic in the planner.

   pegasus.local.env property is no longer supported. To use it users
   need to just do env.VARIABLE_NAME in their properties.  The planner
   now uses GSSManager class from jglobus to determe the DN of the
   proxy for writing out in the braindump file. 
 
19) Support for SQLite 3.7.13 for the stampede statistics layer
 
   SQLIte 3.7.12 introduced a bug as to how the nested aggregate
   queries are handled. This is fixed in version 3.7.14 , but version
   3.7.13 is what Debian installs when one does it through apt. The
   query that generates the jobs.txt file was updated so as to not to
   fail . The update query works across all the recent SQLite versions 
 
20) Changes to tutorial VM image
 
   The tutorial image was updated so that the udev persistent rules
   for eth0 are disabled. Added a GNOME X desktop to the VM. The VM
   image can now grow to 8GB 
 
21) dax2dot now implements transitive reduction algorithm to reduce
extra edges to the workflow 
 
   The dax2dot now implments a transitive reduction algorithm to
   remove extra edges from the workflow. It also has Improved handling
   of -f option. This fixes PM-721 by treating files and jobs as
   equivalent Nodes so that transitive reduction works in the case
   where the DAG contains a mix of File Nodes and Job
   Nodes. Non-redundant Job-Job edges will still be rendered if the
   user specifies -f, but redundant edges will be removed. If the user
   specifies both -f and -s, then there will be many redundant edges
   in a typical workflow. Sometimes the -f option will cause cycles in
   the graph (e.g. files with "inout" linkage, or jobs with a file
   that is both an input and output). In those cases the -s option
   must also be specified. 

 
22) Better handling in monitord for submit host crashes
 
   monitord now detects consecutive workflow started events. In this
   case, it inserts an intervening workflow end event with status set
   to 2 to indicate unknown failure. This case can happen, when condor
   dies on the submit host, say because of power failure. The
   intervening workflow end event is inserted to ensure that the
   queries don't to the database don't fail because of mismatched
   start and end events. 

23) Application Metrics Reporting

   Applications can now enable the planer to pass application defined
   metrics to the metrics server. 
 
   This allows the metrics on the server to be grouped by application
   name. 

   In order to do that, please set the property 
   pegasus.metrics.app      application-name
 
   Users can also associated arbitary key value pairs that can be
   passed to the server. 
 
	pegasus.metrics.app.attribute1 value1 
 
 
 
24) Change of maxpre settings for hierarchal workflows
 
   Changed the default for maxpre from 2 to 1. More sensible in context
   of ensemble manager. 

 
BUGS FIXED
--------------


1) memory explosion for monitord when parsing large PMC workflows 

   For large SCEC workflows using PMC it was noticed that monitord
   memory usage exploded when parsing large hierarchal workflows with
   PMC enabled ( tens of thousands of jobs in one PMC cluster). This
   is now fixed . More details can be found at
   https://jira.isi.edu/browse/PM-712 
 
2) kickstart segfault in tracing

   If the job forks lots of children then the size of the buffer for
   the final invocation record fills up with <proc> tags and causes a
   segfault.  
 
3) kickstart segafault on missing argument

   kickstart segfaulted on missing arguments. This is now fixed.
 
4) pegasus-dagman used pegasus bindir in the search path for
    determining condor location 
 
   Fixed bug bringing in the location of Pegasus when determining
   which HTCondor to use 
 
5) JAVA DAX API stdout| stderr handling  
 
   Changed the handling for stdout | stderr | stdin files in the JAVA
   DAX API. Corresponding uses files are now only added when we are
   printing out the ADAG contents in the toXML method if not already
   specified by the user. This also removes the warning messages,
   where a user adds a uses section for a stdout file with different
   transfer and register flags explicitly in their DAX generators. 

 
6) Fix for heft site selector

   The Heft site selector was not correctly initialized if a user did
   not specify any execution sites on the command line. this is now
   fixed. 

===============================
Release Notes for PEGASUS 4.2.2
===============================
Pegasus 4.2.2 is a minor release, that has minor enhancements and
fixes some bugs to Pegasus 4.2.0 release.  Improvements include  

      - support for sever side pagination for pegasus-dashboard 
      - support for lcg-utils command line clients to retrieve and
        push data to SRM servers 
      - installation of Pegasus python libraries  in standard system
        locations 



IMPROVEMENTS
--------------

1) Rotation of monitord logs

   monitord is automatically launched by pegasus-dagman.  When
   launching monitord, pegasus-dagman sets up the monitord to a log
   file it initializes. However monitord also took a backup of the log
   when it started up as it detected the log file existed. This led to
   two monitord log files in the submit directory which was
   confusing. Now only pegasus-dagman setsup the monitord log. 
 
    More details can be found at
    https://jira.isi.edu/browse/PM-688
 
2) Monitord Recovery in case of SQLLite DB

   If a monitord gets killed on a currently running workflow, then it
   restarts from the start. The information in the recovery file it
   writes out is insufficient to recover gracefully. In case of
   SQLlite DB , monitord does not attempt to expunge the information
   from the database. Instead it takes a backup of the sqlite database
   in the submit directory.  
 
   More details can be found at
   https://jira.isi.edu/browse/PM-689
 
3) Support for lcg-utils for srm transfers 
   
   The pegasus-create-dir, pegasus-cleanup and pegasus-transfer
   clients were updated to include support for lcg utils to do
   operations against a SRM server 
 
    Note that lcg utils takes precedence if both lcg-cp and srm-copy
    are available. 
 
4) Improvements to the dashboard
   
   - Use Content Delivery Networks as source for jQuery, jQueryUI, and
     DataTables plugin. 
   - Most tables in dashboard now have server side pagination, to
      enable large workflows. 
   - Replaced radio buttons with jQuery buttons for a better look and
     feel. 
   - Made Statistics/Charts links more prominent.
   - Added a drop down to filter list of workflows run in last hour,
     day, month, or year. 
 
5) Newer examples added in the examples directory
   
   The release has new examples checked in that highlight 
     - how to use the nonshared fs against a remote staging site that
       has a scp server. 
     - use glite submission to a local PBS cluster using the sharedfs
       data configuration 
     - use the nonsharedfs case, where we use SRM as a staging site
       using CREAMCE submission 
 
6) Pegasus python libraries are installed in standard system locations
   
   The RPM and DEB packages now installs the Python modules in the
   standard system locations. Users should no longer have to set
   PYTHONPATH or add to the include paths in their DAX generators. 
 
7) Condor job logs are no longer in the /tmp directory

   pegasus.condor.logs.symlink now defaults to false. This is to
   ensure compatibility with condor 7.9.4 onwards and ticket
   https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=1419 DAGMAn
   will fail by default now if it detects that common log is in /tmp  
 
BUGS FIXED
--------------

1) Externally Accessible URL's for staged executables broken for SRM

   In certain cases, for SRM file servers in the site catalog, the URL
   constructed to a staged executable was incorrect. This is now
   fixed. 
    
   More details can be found at
   https://jira.isi.edu/browse/PM-686
 
2) pegasus-exitcode cluster-summary w/submitted=0
   
   If the output file has a cluster-summary record, and  the number of
   submitted tasks is 0, then the job succeeded. This fixes an error
   SCEC had that was  introduced when the "tasks" and "submitted"
   values in cluster-summary were separated for PMC. 
 
3) Pegasus Lite did not support jobs with stdin file tracked in the DAX

   In the pegasus lite case, support for jobs with their stdin tracked
   in the DAX was broken. This is now fixed. 

   More details can be found at
   https://jira.isi.edu/browse/PM-694
 
4) pegasus-cleanup did not support symlink deletion

   In case where symlinks to the input files are created in the
   scratch directory on the staging-site, the pegasus-cleanup job was
   created with symlink urls to be deleted. This led to the jobs
   failing as pegasus-cleanup did not support deletion of
   symlinks.This is now fixed .

   Additionally, the planner sets up the cleanup jobs to run on the
   remote if the url to b deleted is a file url or a symlink url 

   More details can be found at
   https://jira.isi.edu/browse/PM-696
 
5) pegasus-createdir and pegasus-transfer with S3 buckets

   pegasus-createdir and pegasus-transfer did translate the S3 bucket
   name correctly if it contained a -. This is now fixed. Also the
   clients don't fail if the bucket already exists. 
 
6) Bug fixes to the cleanup algorithm

   The planner exited with an index out of bounds exception when data
   reuse was triggered and an output file that needed to be staged was
   required to be deleted. This is fixed 

   Also, the clustering of the cleanup jobs resulted in not all the
   files to be deleted by the cleanup jobs. 
   
   Improvements were made how excess edges were removed from the
   graph. The edge removal was done per file instead of per cleanup
   job. This fix drastically reduces the runtime for workflows with
   lots of files that need to be cleaned up. 

   More details can be found  at 
   https://jira.isi.edu/browse/PM-699
 
7) pegasus-analyzer detects prescript failures in the DB mode

   Pegasus analyzer in the database mode was not detecting pre script
   failures for dax jobs as the associated job instance was not
   updated with the exitcode. Changed the way how monitord handles
   failures for sub workflows. In case of pre script failures, the
   prescript failure exitcode is recorded in addition to the stdout of
   the planner log. More details at 
 
   https://jira.isi.edu/browse/PM-704

 
8) monitord tracks non kickstarted  files with rotated stdout and stderr files

   monitord did not track the rotated stdout and stderr of jobs that
   were not launched by kickstart. Because of this the stdout and
   stderr was not populated. This is now fixed. More details at  
 
  https://jira.isi.edu/browse/PM-685

9) Planner fails on determining the DN from a proxy file

   The planner uses the Java COG jar to determine the DN from a proxy file. It
   was discovered that for proxies generated from  an X.509 end entity credential, 
   by a GSI-enabled OpenSSH server results in a NPE in the COG jar.

   The planner now catches all the exceptions while trying to determine the DN.
   There is never a FATAL error if unable to determine the DN.

10) pegasus-exitcode checks for the existence of .err file 

   The pegasuslite_failures function did not check for missing stderr files. As a 
   result, if exitcode was called in a scenario where there was no .err file, then
   it failed trying to determine if None is a valid path.  

===============================
Release Notes for PEGASUS 4.2.1
===============================
Pegasus 4.2.1 was tagged but never officially released. Users are advised to use
4.2.2 instead.  The difference between 4.2.2 and 4.2.1 is that it does not have
the fix for 
    - planner fails if there is NullPointerException in the underlying COG code
      when trying to determine the DN.
    - pegasus-exitcode checks for existense of .err file before trying to base the
      exitcode on it's contents.

===============================
Release Notes for PEGASUS 4.2.0
===============================

This a major release of Pegasus which contains
     - several improvements on data management capabilities
     - a new web based monitoring dashboard
     - job submission interfaces supported. CREAM CE is now supported 
     - new replica catalog backends. 
     - support for PMC only workflows and IO forwarding for PMC clustered jobs
     - anonymous usage metrics reporting.

The data management improvements include a new simpler site catalog
schema to describe the site layouts, and enables data to be
transferred to and from staging sites using different protocols. A
driving force behind this change was Open Science Grid, in which it is
common for the compute sites to have Squid caches available to the
jobs.  For example, Pegasus workflows can now be configured to stage
data into a staging site using SRM or GridFTP, and stage data out over
HTTP. This allows the compute jobs to automatically  use the Squid
caching mechanism provided by the sites, when pulling in data to the
worker nodes over HTTP. 

Also, with the release we include a beta version of a web based
monitoring dashboard (built on flask) that users can use to monitor and
debug their running workflows. The dashboard provides workflow overview,
graphs and job status/outputs.

Job submissions to the CREAM job management system has been implemented
and tested. 

New simpler replica catalog backends are included that allow the user
to specify the input directory where the input files reside instead of
specifying a replica catalog file that contains the mappings.

There is prototypical support for setting up Pegasus to generate the
executable workflow as a PMC task workflow instead of a Condor DAGMan
workflow. This is useful for environments, where Condor cannot be
deployed such as Blue Waters. I/O forwarding in PMC enables each task
in a PMC job to write data to an arbitrary number of shared files in a
safe way. This is useful for clustered jobs that contain lots of tasks
and each task only writes out a few KB of output data.

Starting with this release, Pegasus will send anonymous usage statistics 
to the Pegasus development team. Collecting this anonymous information
is mandated by the main Pegasus funding agency, NSF. Please refer to 
http://pegasus.isi.edu/wms/docs/latest/funding_citing_usage.php#usage_statistics 
for more details on our privacy policy and configuration.

NEW FEATURES
--------------

1) New Site Catalog Schema
   
   Pegasus 4.2 release introduces a version 4 for the site catalog
   schema. The schema represents a simpler way to describing a site
   and organizes the site information by various directories
   accessible on the site for the workflow to use.

   The schema is described in our user guide here
   http://pegasus.isi.edu/wms/docs/latest/creating_workflows.php#sc-XML4
   
   and examples in our distribution have been updated to use the new
   schema. Sample site catalog files in the newer format can also be
   found in the etc directory. 

   With 4.2, Pegasus will autoload the appropriate site catalog schema
   backend by inspecting the version number in the site catalog file
   at runtime. 

   Users can use the client pegasus-sc-converter to convert their
   existing site catalogs in XML3 format to the newer versions. Users
   can choose to specify pegasus.catalog.site as XML or leave it
   undefined. 

   The 4.2 release no longer supports the following old Site Catalog
   Implementations 
   		   - VORS
		   - MYOSG
		   - XML2
		   - Text
   
2) Improved Data Management Capabilities

   Users can now specify different protocols to push data to a
   directory on the staging site and retrieve data from the directory
   using another protocol.

   For example users can use a HTTP file server to retrieve data (
   pull ) data from a staging site to worker node and use another
   protocol say scp to push data back to the staging site after a job
   completes. This is particularly useful when you want to leverage a
   high throughput HTTP deployment backed by SQUID proxies when
   serving input data to compute nodes.

   Users can specify different file servers for a particular directory
   by specifying different operation attribute on the file
   servers. The operation attribute can take enumerated set of values
  
      - put  ( use the server only for put operations to the directory )
      - get ( use the server only for get operations to the directory)
      - all ( use it for both get and put operations)


3) Online Workflow Dashboard
   
   This release includes a beta version of a web based monitoring
   dashboard ( built on flask ) that users can use to monitor and
   debug their running workflows. 

   The dashboard is meant to be run per user, and lists all the
   workflows run by that user. The dashboard gets a list of running
   workflows by looking up a sqlite database in the users home
   directory ~HOME/.pegasus/workflow.db . This database is populated
   by pegasus-monitord everytime a new root workflow is
   executed. Detailed information for each workflow is retrieved from
   the stampede database for the each workflow.

   The workflow dashboard lists all the user workflows on the home
   page and are color coded. Green indicates a successful workflow,
   red indicates a failed workflow while blue indicates a running
   workflows. 

   Users can click on a workflow to drill down and get more
   information about the workflow that leads to a workflow page. The
   workflow page has identifying metadata about the workflow, and has
   a tabbed interface that can be used to traverse through the list of
   sub workflows, failed, running and successful jobs.

   Each job or sub workflow can be clicked to get further details
   about that entity .Clicking on a failed/successful job will lead to
   an invocation details page that will have the contents of the
   associated kickstart record displayed.

   The charts button can be clicked to generate relevant charts about
   the workflow execution such as the 
       - Workflow Gantt Chart
       - Job Distribution by Count/Time
       - Time Chart by Job/Invocation


   The statistics button can be clicked to display a page that lists
   the statistics for a particular workflow. The statistics page
   displays statistics similar to what the command line tool
   pegasus-statistics displays.

   
   The workflow dashboad can be started by a  a command line tool
   called pegasus-dashboard. 

   
4) Usage Statistics Collection

   Pegasus WMS is primarily a NSF funded project as part of the NSF
   SI2 track. The SI2 program focuses on robust, reliable, usable and
   sustainable software infrastructure that is critical to the CIF21
   vision. As part of the requirements of being funded under this
   program, Pegasus WMS is required to gather usage statistics of
   Pegasus WMS and report it back to NSF in annual reports. The
   metrics will also enable us to improve our software as they will
   include errors encountered during the use of our software. 

   More details about our policy and metrics collected can be found
   online at  
   http://pegasus.isi.edu/wms/docs/latest/funding_citing_usage.php#usage_statistics


5) Support for CREAMCE submissions

   CREAM is a webservices based job submission front end for remote
   compute clusters. It can be viewed as a replaced for Globus GRAM
   and is mainly popular in Europe. It widely used in the Italian
   Grid. 

   In order to submit a workflow to compute site using the CREAMCE
   front end, the user needs to specify the following for the site in
   their site catalog 

   	 - pegasus profile style with value set to cream
	 - grid gateway defined for the site with contact attribute
	   set to CREAMCE frontend and scheduler attribute to remote
	   scheduler. 
	 -  a remote queue can be optionally specified using globus
	  profile queue with value set to queue-name 

   More details can be found here

   http://pegasus.isi.edu/wms/docs/latest/execution_environments.php#creamce_submission	  

6) Initial Support for PMC only workflows

   Pegasus can now be configured to generate a workflow in terms of a
   PMC input workflow. This is useful to run on platforms where it not
   feasible to run Condor such as the new XSEDE machines such as Blue
   Waters. In this mode, Pegasus will generate the executable workflow
   as a PMC task workflow and a sample PBS submit script that submits
   this workflow .

   Users can modify the generated PBS script to tailor it to their
   particular cluster.

   To use Pegasus in this mode, set 
   
   pegasus.code.generator PMC

   In this mode, the workflow should be configured to submit to a
   single execution site.
   
6) New options --input-dir and output-dir for  pegasus-plan

   The planner now has --input-dir and --output-dir options. This
   allows the planner to read mappings for input files from an input
   directory and stage the results to an output directory.
 
   If , the output-dir option is set then the planner updates the storage
   directory for the output site specified by the user. If none is
   specified , then the local site entry is updated. 

7) Directory based Replica Catalog

   Users can now setup Pegasus to read the input file mappings from an
   input directory. Details on how to use and configure Pegasus in
   this mode can be found here

   http://pegasus.isi.edu/wms/docs/latest/creating_workflows.php#idp11375504

8) Regular Expressions support in File based Replica Catalog

   Users can now specify a regex expression in a file based replica
   catalog to specify paths for mulitple files/data sets.

   To use it you need to set
   pegasus.catalog.replica to Regex

   More details can be found here

   http://pegasus.isi.edu/wms/docs/latest/creating_workflows.php#idp11375504


9) Support for IO Forwarding in PMC ( pegasus-mpi-cluster )
   
   In workflows that have lots of small tasks it is common for the I/O
   written by those tasks to be very small. For example, a workflow
   may have 10,000 tasks that each write a few KB of data. Typically
   each task writes to its own file, resulting in 10,000 files. This
   I/O pattern is very inefficient on many parallel file systems
   because it requires the file system to handle a large number of
   metadata operations, which are a bottleneck in many parallel file 
   systems.

   In order to address this use case PMC implements a feature that we
   call "I/O Forwarding". I/O forwarding enables each task in a PMC
   job to write data to an arbitrary number of shared files in a safe
   way. It does this by having PMC worker processes collect data
   written by the task and send it over over the high-speed network
   using MPI messaging to the PMC master  process, where it is written
   to the output file. By having one process  (the PMC master process)
   write to the file all of the I/O from many parallel  tasks can be
   synchronized and written out to the files safely.    

   More details on how IO Forwarding works can be found in the manpage
   for PMC under the section I/O Forwarding

   http://pegasus.isi.edu/wms/docs/trunk/cli-pegasus-mpi-cluster.php


10) Clustering of cleanup jobs

    The InPlace cleanup algorithm that adds cleanup jobs to the
    executable workflow , now clusters the cleanup jobs by
    default for each level of the workflow . This keeps in check the
    number of cleanup jobs created for large workflows.

    The number of cleanup jobs added per level can be set by the
    following  property

    pegasus.file.cleanup.clusters.num  

    It defaults to 2.

11) Planner has support for SHIWA Bundles

    The planner can be take in shiwa bundles to execute workflows. For
    this to happen, the bundle need to be created in shiwa gui with
    the appropriate Pegasus Plugins

    More details at
    https://jira.isi.edu/browse/PM-638
    
12) Improvements to pegasus-statistics
    		 
    There is now a single API call executed to get the succeeded and
    failed count for job and sub workflows.

13) Improvements to planner performance

    The performance of the planner has been improved for large
    workflows. 

14) Renamed --output option to --output-site

   The --output option has been deprecated and replaced by a new
   option --output-site

15) Removed support for pegasus.dir.storage

   Pegasus no longer supports pegasus.dir.storage property. The
   storage directory can only be specified in the site catalog for a
   site. 


BUGS FIXED
--------------

1) Failure in Data Reuse if a deleted job had an output file that had
to be transferred

   There was a bug, where the planner failed in case of data reusue if
   any of the deleted jobs had output files that needed to be
   transferred. 

   More details at 
   https://jira.isi.edu/browse/PM-675


===============================
Release Notes for PEGASUS 4.1.0
===============================
This a major release of Pegasus that has support for PMC
(pegasus-mpi-cluster ) that can be used to run the tasks in a
clustered job in parallel on remote machines using MPI. As part of
this release, the support for submitting workflows using CondorC has
been updated. The Pegasus Tutorial has also been updated and is
available to run on  

   - Amazon EC2
   - Futuregrid
   - Local machine using Virtual Box
 

NEW FEATURES
--------------

1) pegasus-mpi-cluster

   Pegasus has support for a new clustering executable called
   pegasus-mpi-cluster (PMC) that allows users to run tasks in a clustered
   job in parallel using MPI on the remote node. The input format for
   PMC is a DAG based format similar to Condor DAGMan's. PMC follows
   the dependencies specified in the DAG to release the jobs in the
   right order and executes parallel jobs via the workers when
   possible. The input file for PMC is automatically generated by the
   Pegasus Planner when generating the executable workflow.

   In order to use PMC set in your properties
   pegasus.clusterer.job.aggregator  mpiexec

   Also, you may need to put an entry in your transformation catalog
   for pegasus::mpiexec to point to the location of the PMC executable
   on the remote side.

   More details can be found in the man page for pegasus-mpi-cluster
   and in the clustering chapter

   https://pegasus.isi.edu/wms/docs/4.1/reference.php#job_clustering
    
    There is a XSEDE example in the examples directory that shows how
    to use PMC on XSEDE

2) Use of new client pegasus-gridftp in pegasus-create-dir and
pegasus-cleanup 

    Starting with release 4.1, the pegasus create dir and cleanup
    clients use a java based client called pegasus-gridftp to create
    directories and remove files from against a gridftp server. 

    Pegasus by default now adds a dagman category named cleanup for
    all cleanup jobs in the workflow. The maxjobs for this category is
    set to 4 by default.

    This can be overriden by specifying the property
    dagman.cleanup.maxjobs

3) Support for CondorC

   The support for CondorC in Pegasus has been updated. Users can
   associate a pegasus profile named style with value condorc with a
   site in the site catalog to indicate that submission to the site
   has to be achieved using CondorC.

   The site catalog entry should mention the grid gateways to indicate
   the remote schedd to which the jobs need to be submitted, and the
   condor collector for the condorc site. It is optional to specify
   the condor collector. If not specified, Pegasus will use the
   contact mentioned in the grid gateway.

   Example snippet with relevant entries  below
    <site handle="isi-condorc" arch="x86" os="LINUX">
   
	 <grid type="condor" contact="ccg-testing1.isi.edu" scheduler="Condor" jobtype="compute" total-nodes="50"/>
  	   <grid type="condor" contact="ccg-testing1.isi.edu" scheduler="Condor" jobtype="auxillary" total-nodes="50"/>
    ....
         <!-- specify which condor collector to use -->
	 <profile namespace="condor" key="condor_collector">ccg-testing1.isi.edu</profile>
    </site>


4) Changed the default transfer refiner for Pegasus

   The default transfer refiner in Pegasus now clusters both stagein
   and stageout jobs per level of the workflow. The previous version
   used to cluster stagein jobs per workflow and the stageout jobs per
   level of the workflow.

   More details can be found at 

   https://pegasus.isi.edu/wms/docs/4.1/reference.php#id645300

5) Updated the Pegasus Tutorial

   The Pegasus Tutorial has now been updated and is available to run
   on 
   - Amazon EC2
   - Futuregrid
   - Local machine using Virtual Box

    https://pegasus.isi.edu/wms/docs/4.1/tutorial_vm.php
   
6) pegasus-statistics has a new -f option
   
   The -f option can be used to specify the output format for
   pegasus-statistics. Valid supported formats are txt and csv

7) Updated condor periodic_release and periodic_remove expressions

   Earlier, Pegasus used to set default periodic_release and
   periodic_remove expressions as follows

   periodic_release = (NumSystemHolds <= 3) 
   periodic_remove = (NumSystemHolds > 3) 

   This had the effect of removing the jobs as soon as they went to
   held state.

   Starting 4.1 the expressions have been updated to 
   periodic_release = False 
   periodic_remove = (JobStatus == 5) && ((CurrentTime -
   EnteredCurrentStatus) > 14400) 
   
   With this, the job remains in held state for 4 hours before being
   removed. The idea is that it is a long enough time for users to 
   debug held jobs.
   
   If users wish to use the previous expressions, they can do it by
   specifying the condor profile keys periodic_release and
   periodic_remove. 

8) Property to turn off registration jobs

   Pegasus now exposes a boolean property pegasus.register that can be
   used to turn off the registration of jobs.

9) More descriptive errors if incomplete site catalog specified

   Earlier, incomplete site catalog causes NPE's when running
   pegasus-plan. This has been replaced by more descriptive errors
   that will give user enough information to figure out the missing
   entries in the site catalog.

   More details at
   https://jira.isi.edu/browse/PM-590

10) Change in DAX schema
   
   The dax schema version is now 3.4. The schema now allows for
   specifying filesizes as a size attribute in the uses element that
   lists the input and output files for a job.

   The DAX Generator API's have been updated accordingly.

   This is useful for users extending the Pegasus Code for their
   specific research use cases.

11) Prototype support for Shiwa bundles
   
   pegasus-plan has a new option --shiwa-bundle that allows users to
   pass a pegasus SHIWA bundle for execution.  A Pegasus shiwa bundle,
   is a bundle that has been generated using the Pegasus Plugin for
   the Shiwa Desktop.

12) Improved performance for the expunge operation in against mysql
database 

    When monitord is run in a replay mode, the database is first
    expunged of all the information related to that workflow. In case,
    of mysql backend where the same database maybe used to track
    multiple hierarchal workflows, the expunge operation has to be
    careful to delete only the relevant entries for the various
    tables. 

    In earlier versions, this expunge operation was implemented at OR
    level in SQLAlchemy that led to lots of select and delete
    statements to be executed ( one per entry ). This blew up the
    memory footprint for monitord and prevented the workflow
    population in case of large databases. For 4.1, we changed the
    schema to add cascaded delete clauses, and set the passive delete
    option to true in SQLAlchemy.  

    More details
    https://jira.isi.edu/browse/PM-646

13 ) Runtime Clustering picks up pegasus profile key named runtime
   
   Starting 4.1, the runtime clustering  in Pegasus picks up  pegasus
   profile key runtime instead of job.runtime .  

   job.runtime is deprecated and a message is logged if a user has
   that specified. The planner picks up job.runtime only if runtime is
   not specified for a job. 


BUGS FIXED
----------

1) pegasus-lite-local.sh made assumptions on PATH

   pegasus-lite-local wrapper that is invoked if a pegasus lite jobs
   runs in local universe made assumption on PATH variable to
   determine the pegasus tools.

   This is now fixed. More details at 
   https://jira.isi.edu/browse/PM-636
   
2) Overwriting of entries with file based replica catalog

   pegasus-rc-client lfn pfn pool="local" # Inserts new entry in RC file 
   pegasus-rc-client lfn pfn pool="usc" # Overwrites pool="local" to pool="usc" 

   The uniqueness constraint in the File RC has been updated to
   consider the site attribute also.
   
   More details at
   https://jira.isi.edu/browse/PM-634

3) pegasus-statistics failed on workflows with large number of sub
workflows 

   pegasus-statistics failed if a workflow had more 1000 sub
   workflows. This was due to a SQL Alchemy issue

   More details at 
   https://jira.isi.edu/browse/PM-616

4) Properties propogation for sub workflows

   There was a bug with properties propogation for hierarchal
   workflows when using PegasusLite for some sub workflows and
   sharedfs for others

   This is partially fixed.
   https://jira.isi.edu/browse/PM-624
   
===============================
Release Notes for PEGASUS 4.0.1
===============================
This is a minor release, that fixes some bugs and has minor enhancements. 


NEW FEATURES
--------------

1) pegasus lite local wrapper is now used for local universe jobs in
   shared fs mode also, if condor io is detected. 

   Also  remote_initialdir is not implemented consistently across
   universes in Condor.  For vanilla universe condor file io does not
   transfer the file to the remote_initialdir.

2) task summary queries were reimplemented

   The task summary queries ( that list the number of successful and
   failed tasks )  in the Stampede Statisitcs API was
   reimplemented for better performance.


3) pegasus-monitord sets PEGASUS_BIN_DIR while calling out notfication
scripts . 
	
   https://jira.isi.edu/browse/PM-598

4) the default notification script can send out emails to multiple
recipients. 

5) Support for new condor keys 

   Pegasus allows users to specify the following condor keys as
   profiles in the Condor namespace. The new keys have been introduced
   in Condor 7.8.0

   request_cpus 
   request_memory 
   request_disk 
   
   https://jira.isi.edu/browse/PM-600


BUGS FIXED
----------
1) pegasus-kickstart does not collect procs and tasks statistics on kernels >= 3.0
   
   When kickstart is executed on a Linux kernel >= 3.0, logic in the
   machine extensions prevented the proc statistics gathering, because
   it was a reasonable assumption that the API might have changed (it
   did between 2.4 and 2.6). This is now fixed, as it is supported for
   kernels 3.0 through 3.2

   https://jira.isi.edu/browse/PM-571

2) scp transfer mode did not create remote directories
   
   When transferring to a scp endpoint, pegasus-transfer failed unless
   the remote directory already existed. This broke deep LFNs and
   staging to output sites.  This is now fixed.

   https://jira.isi.edu/browse/PM-579

3) Incorrect resolution of PEGASUS_HOME path in the site catalog for
remote sites  in some cases
       
   If a user specified a path to PEGASUS_HOME for remote sites in the
   site catalog and the directory also existed on the submit machine,
   the path was resolved locally. Hence if the local directory was a
   symlink, the symlink was resolved and that path was used for the
   remote site's PEGASUS_HOME.

   https://jira.isi.edu/browse/PM-577

4) pegasus-analyzer did not work correctly against the MySQL Stampede
DB
	
   pegasus-analyzer had problems querying MySQL stampede database
   because of a query aliasing error in the API underneath. This is
   now fixed.

   https://jira.isi.edu/browse/PM-580

5) Wrong timezone offsets for ISO timestamps 

   Pegasus python library was generating the wrong time zone offset
   for ISO 8601 time stamps. This was because of an underlying bug in
   python where %z does not work correctly across all platforms.

   https://jira.isi.edu/browse/PM-576

6) pegasus-analyzer warns about "exitcode not an integer!"
 
   pegasus-analyzer throwed a warning if a long value for an exitcode
   was detected.
 
    https://jira.isi.edu/browse/PM-584

7) Perl DAX generator uses 'out' instead of 'output' for stderr and
stdout linkage 

  The perl DAX generator API generated the wrong link attribute for
  stdout files. Instead of having link = output it generated link =
  out.

  https://jira.isi.edu/browse/PM-585

8) Updated Stampede Queries to handle both GRID_SUBMIT and
GLOBUS_SUBMIT events.

  Two of the queries ( get_job_statistics and get_job_state ) were
  broken for CondorG workflows when operating against a MySQL database
  backend. In that case,  both GRID_SUBMIT and GLOBUS_SUBMIT can be
  logged for the jobs. In that case, some of the subqueries were
  breaking against MySQL has MySQL has stricter checks on queries
  returning a single value.


9) Support for DAGMAN_COPY_TO_SPOOL Condor configuration parameter

   Condor has a setting DAGMAN_COPY_TO_SPOOL that if set to true,
   results in Condor copying the DAGMan binary to the spool directory
   before launching the workflow. In case of Pegasus, condor dagman is
   launched by a wrapper called pegasus-dagman. Because of this ,
   pegasus dagman was copied to the condor spool directory before
   being launched in lieu of condor dagman binary.

   This is now fixed whereby pegasus-dagman will copy condor_dagman
   binary to the submit directory for the workflow before launching
   the workflow.

   More details at 
   https://jira.isi.edu/browse/PM-595

===============================
Release Notes for PEGASUS 4.0.0
===============================

This is a major release of Pegasus that introduces new advanced data
handling capabilities, and contains improved support for running
workflows in non-shared filesystem scenarios such as clouds and Condor
pools. Pegasus now optionally separates the data staging site from the
workflow execution site for more flexible data management. A new
feature is PegasusLite - an autonomous lightweight execution
environment to manage jobs on the compute nodes and handles data
movement to/from such jobs against the workflow staging site. The RPM
and Debian packages conform to the Filesystem Hierarchy Standard
(FHS).


NEW FEATURES
--------------

1) PegasusLite

   Pegasus 4.0 has improved support for running workflows in a non
   shared fileysystem setup . This is useful for running in cloud
   environments and allows for more dynamic placement of jobs. 
   Pegasus 4.0 introduces the concept of a staging site for the
   worklfows, that can be different from an execution site. The
   planner places the data on the staging site for the workflow. When
   the jobs start on the remote compute nodes, they are launched by a
   lightweight component called PegasusLite, that stages in the data
   from the staging site to a local directory on the worker node
   filesystem . The output data generated by the compute job is
   similarly pushed back to the staging site, when a job completes.

   Users can now setup Pegasus for different environments by setting
   the property

   pegasus.data.configuration 

   More details can be found here

   http://pegasus.isi.edu/wms/docs/trunk/running_workflows.php#data_staging_configuration 

   

2) Move to FHS layout

   Pegasus 4.0 is the first release of Pegasus which is Filesystem
   Hierarchy Standard (FHS) compliant. The native packages no longer
   installs under /opt. Instead, pegasus-* binaries are in /usr/bin/
   and example workflows can be found under
   /usr/share/pegasus/examples/.

   To find Pegasus system components, a pegasus-config tool is
   provided. pegasus-config supports setting up the environment for

     - Python
     - Perl
     - Java
     - Shell

3) Improved Credential Handling
   
   Pegasus 4.0 has improved credential handling. The planner while
   planning the worklfow automatically associates the jobs with the
   credentials it may require. This is done by inspecting the URL's
   for the files a job requires. 

   More details on how the credentials are set can be found here
   http://pegasus.isi.edu/wms/docs/4.0/reference.php#cred_staging


4) New clients for directory creation and file cleanup

   Starting 4.0, Pegasus has changed the way how the scratch
   directories are created on the staging site. The planner now
   prefers to schedule the directory creation and cleanup jobs
   locally. The jobs refer to python based tools, that call out to
   protocol specific clients to determine what client is picked
   up. For protocols, where specific remote cleanup and directory
   creation clients don't exist ( for example gridftp ), the python
   tools rely on the corresponding transfer tool to create a directory
   by initiating a transfer of an empty file. The python clients used
   to create directories and remove files are called
      - pegasus-create-dir
      - pegasus-cleanup

    More details about the clients can be found in the transfers
    chapter
    
    http://pegasus.isi.edu/wms/docs/trunk/reference.php#transfer

5) Runtime based clustering

   Users can now do horizontal clustering based on job runtimes. If
   the jobs in the DAX are annotated with job runtimes ( use of
   pegasus profile key job.runtime ) , then Pegasus can horizontally
   cluster the jobs in such a way that the clustered job will not run
   more than a maxruntime ( specified by use of profile
   clusters.maxruntime). 

   More details can be found in the clustering chapter of the online
   guide. 
   http://pegasus.isi.edu/wms/docs/4.0/reference.php#job_clustering

6) pegasus-analyzer works with stampede database

   Starting with 4.0 release, by default pegasus analyzer queries the
   database generated in the workflow submit directory ( unless using
   mysql) to debug the workflow. If you want it to use files in the
   submit directory , use the --files option.

7) CSV formatted output files for pegasus statistcs
   
   pegasus-statistics now generates it's output in a csv formatted
   file also, in addition to the txt files it creates. This is useful,
   for importing statistics in tools like Mircrosoft Excel.

8) Tracking of job exitcode in the stampede schema
   
   The stampede database schema was updated to associate a job
   exitcode field with the job_instance table. This makes it easier
   for the user and the mining tools to determine whether a job
   succeeded for failed at the Condor level.

   Earlier that was handled at query time by looking up the last state
   in the jobstate table.

   The updated stampede schema can be found here

   http://pegasus.isi.edu/wms/docs/4.0/monitoring_debugging_stats.php#monitoring_pegasus-monitord
   
9) Change to how exitcode is stored in the stampede database
   
   Kickstart records capture raw status in addition to the exitcode
   . The exitcode is derived from the raw status. Starting with
   Pegasus 4.0 release, all exitcode columns ( i.e invocation and job
   instance table columns ) are stored with the raw status by
   pegasus-monitord. If an exitcode is encountered while parsing the
   dagman log files , the value is converted to the corresponding raw
   status before it is stored. All user tools, pegasus-analyzer and
   pegasus-statistics then convert the raw status to exitcode when
   retrieving from the database.

10) Accounting for MPI jobs in the stampede database

   Starting with the 4.0 release, there is a multiplier factor
   associated with the jobs in the job_instance table. It defaults to
   one, unless the user associates a Pegasus profile key named cores
   with the job in the DAX. The factor can be used for getting more
   accurate accounting statistics for jobs that run on multiple
   processors/cores or mpi jobs.

   Full details in the pegasus-monitord chapter.

11) Stampede database upgrade tool

   Starting with the 4.0 release, users can use the stampede database
   upgrade tool to upgrade a 3.1.x database to the latest version.

   All statistics and debugging tools will complain if they determine
   at runtime that the database is from an old version.

   More details can be found in the migration guide
   http://pegasus.isi.edu/wms/docs/4.0/useful_tips.php#id727831

BUGS FIXED
----------

1) pegasus-plots invocation breakdown

   In certain cases, it was found that the invocation breakdown piechart
   generated by pegasus-plots was broken. This is now fixed.

   More details at
   https://jira.isi.edu/browse/PM-566

2) UUID devices confuse pegasus-keg

   It is possible with Linux to describe devices in /etc/fstab with
   their UUID instead of the device name, i.e.

   UUID=xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx / ext3 defaults 1 1 
   instead of 
   /dev/sda1 / ext3 default 1 1 
   
   However, some logic in keg relies on the device starting with a
   slash to recognize a true-file device. 
   This is now fixed.
   

===============================
Release Notes for PEGASUS 3.1.1
===============================
This is a minor release, that fixes some bugs and has minor enhancements. 


NEW FEATURES
--------------

1) the jobs file created by pegasus-statistics includes hostname and
exitcode 
	 
   The hostname of the host on which the job ran and the exitcode with
   which job exited is now included in the jobs file created by
   pegasus-statistics .

   https://jira.isi.edu/browse/PM-487


2) pegasus-monitord truncates stdout/stderr

   pegasus-monitord truncates stdout/stderr if they exceed the maximum
   size allowed by the database. 

   The boolean property to disable stdout/stderr parsing is 

   pegasus.monitord.stdout.disable.parsing 
   
3) logging improvements for pegasus-monitord

   monitord adds logging information when exiting upon a signal. The
   netlogger DB backend code is now more verbose when errors happen.

   pegasus-monitord also prints the start and end of the DB flushing
   function in its logs.

   pegasus-monitord prints its pid whenever it outputs the
   starting/ending messages. This can be used to track if multiple
   pegasus-monitord instances are running at the same time.

4) Switched precedence for env loading for local site.

   On certain Unix systems like debian, java overrides the
   LD_LIBRARY_PATH on linux.

   Now a value specified for the local site in the site catalog is
   preferred over the one in the environment. This is to ensure, that
   a user can set LD_LIBRARY_PATH in the site catalog, and it
   overrides the the JAVA provided one.
   
   Details at
   https://jira.isi.edu/browse/PM-471

5) seqexec always fails on first job failure
   
   The default behaviour for the clustered jobs has been changed to
   make seqexec fail on first job that fails. 

   To change it , to run all the jobs in a cluster users need to set
   the following property to false

   pegasus.clusterer.job.aggregator.seqexec.firstjobfail

   


BUGS FIXED
----------
1) kickstart with stdout redirection
   
   The stdout redirection feature of kickstart failed on some systems
   failed, as newer versions of libc() don't allow for the realpath()
   function to point to a non existing filename. This prevented
   kickstart from redirecting the stdout of a job to a file.

   This is now fixed.

   More details at 
   https://jira.isi.edu/browse/PM-469   

2) seqexec incorrectly reported the summary section in its records
   
   seqexec incorrectly reported the summary section in its records,
   when launched through condor. It reported jobs succeeded even
   though jobs had failed.

   More details at
   https://jira.isi.edu/browse/PM-541


3) Fixed a bug in pegasus-monitord that was preventing the database
population of certain pre and post script events. 

4) Fixed some python 2.4 compatibility issues in pegasus-plots.

   More details at
   https://jira.isi.edu/browse/PM-534

5) SubDAG jobs referred in a DAX were launched by pegasus-dagman

   DAX can contain references to condor DAGS, that are not planned through
   Pegasus. They were launched by pegasus-dagman instead of directly
   being launched by condor_dagman. This is now fixed

   More details at
   https://jira.isi.edu/browse/PM-553

6)  invoke option for kickstart -I flag triggered incorrectly for  clustered jobs 

    There was a bug in how pegasus handled invoke to get around the 
    long arguments specified for the jobs in the DAX. In that case, a .arg
    created that is transferred with the job. This was not happening for
    clustered jobs.

    More details at
    https://jira.isi.edu/browse/PM-526


===============================
Release Notes for PEGASUS 3.1.0
===============================
This is a major release of Pegasus with support for notifications and
a redesigned backend database to store information about workflows and  jobs.
There are updated and redesigned auxillary tools in the release
namely
   - pegasus-status
   - pegasus-statistics
   - pegasus-plots

There has been a change on how locations of properties files are
passed to the planner and tools. From Pegasus 3.1 release onwards,
support has been dropped for the following properties that were used
to signify the location of the properties file 
  - pegasus.properties
  - pegasus.user.properties
Instead, users should use the --conf option for the tools.
More details at
http://pegasus.isi.edu/wms/docs/3.1/reference.php#Properties


The user guide has been reorganized and now has a new user walkthrough
and reference guide for all command line tools.
   
   http://pegasus.isi.edu/wms/docs/3.1/

NEW FEATURES
---------------

1) Support for Notifications
  
  This release of Pegasus has support for workflow level and job
  notifications. Currently, the user can annotate the DAX to specify
  what notifications they want associated with the workflow or/and
  individual jobs. Associating a notification with job entails
  specifying the condition when notification should be sent, the
  executable that needs to be invoked and the arguments with which it
  needs to be invoked. All notifications are invoked on the submit
  host by the monitoring daemon pegasus-monitord. The release comes
  bundled with default notification scripts that users can use for
  notifications. 

  The DAX API's have also been updated to allow for associating
  notifications with the jobs and the workflow.

  More details about how notifications can be found here
  http://pegasus.isi.edu/wms/docs/3.1/reference.php#notifications

2) Workflows and Jobs Database

   The backend database schema to which pegasus-monitord populates
   runtime information about jobs in the workflow has been
   redesigned. Now in addition to jobs in the executable workflow,
   information about tasks in the DAX is also tracked and can be
   connected to the corresponding kickstart records.

   Also, pegasus-monitord no longer dies on database related
   errors. The statistics and plotting tools have in built checks that
   will notify a user if a DB was not populated fully for a workflow
   run. 

    monitord now logs timestamps in monitord.log and monitord.done
    files to reflect the time monitord finishes processing specific
    sub-workflows.  

   Information about the updated database schema can be found here.
   http://pegasus.isi.edu/wms/docs/3.1/monitoring_debugging_stats.php#monitoring_pegasus-monitord

3) Updated pegasus-statistics and plots

   pegasus-statistics and pegasus-plots have been updated to retrive
   all information from the runtime stampede database. pegasus plots
   now generates plots using protoviz and generates charts showing
   invocation breakdown, workflow gantt chart, host over time chart
   that shows how jobs ran on various hosts and a Time chart shows job
   instance/invocation count and runtime of the workflow run over time 

   More information about updated statistics and tools can be found
   here
   http://pegasus.isi.edu/wms/docs/3.1/monitoring_debugging_stats.php#plotting_statistics

4) Updated pegasus-status tool
   
   The pegasus-status tool has been reimplemented for this
   release. The new tool shows the current state of a Condor Q and
   formats it better. For hierarichal workflows, the tool now displays
   jobs correctly grouped by sub workflows.

   More information can be found here
   http://pegasus.isi.edu/wms/docs/3.1/reference.php#pegasus-status

5) Improved support for S3

   With 3.1 release, there is a pegasus-s3 client that uses the amazon
   api to create buckets, put and retrieve files from buckets. This
   client has further been incorporated into pegasus-transfer . The
   pegasus-s3 looks up a configuration file to look up connection
   parameters and authentication tokens. The S3 config file is
   automatically transferred to the cloud with jobs when a
   workflow is configured to run in the S3 mode.

   In the S3 mode, jobs will run in the cloud without requiring a
   shared filesystem.

   More information about S3 mode in Pegasus can be found here
   http://pegasus.isi.edu/wms/docs/3.1/running_workflows.php#running_on_cloud
   

6) Tools now have a --conf option
   
   Most of the command line tools now have a --conf option that can be
   used to pass a properties file to the tools. Properties can no
   longer be passed to a tool using
   -Dpegasus.properties=/path/to/props or
   -Dpegasus.user.properties=/path/to/props 

 
7) Improved rescue dag semantics for hierarchal workflows

   In earlier releases, a rescue dag submission of a hierarchal
   workflow lead to re-planning of the sub workflows even though
   rescue dags were submitted for the sub workflows. This could create
   problems as the re-planning resulted in the braindump files being
   over-written and monitord attempting to load information into the
   stampede database with a new workflow uuid.

   In this release, this issue has been addressed. By default for sub
   workflows  rescue dags are always submitted unless a --force-replan
   option is provided to pegasus-plan. In case of replanning, now a
   new submit directory is created for the sub workflow. The submit
   directories for sub workflows are now symlinks that point to the
   current submit directory for a sub workflow. This ensures that
   there are no race conditions between monitord and the workflow
   while populating to the database.

8) Default categories for certain types of jobs.

   subdax, subdag , cleanup and registration jobs now have default
   DAGMan categories associated with them. 

   JOB TYPE 	 | CATEGORY NAME 
   --------------------------
    dax		 | subwf 
    dag          | subwf 
    cleanup      | cleanup 
    registration | registration

   This allows for a user to control maxjobs for these categories
   easily in properties by specifying
   dagman.[CATEGORY NAME].maxjobs property

    If a file based replica catalog is used, then maxjobs for
    registration jobs is set to 1. This is to ensure, that multiple
    registration jobs are not run at the same time. 

9) Automatic loading of DAXParser based on schema number
   
   Earlier the users needed to specify the pegasus.schemea.dax
   property to point to the corresponding DAX schema definition file
   to get Pegasus to load DAX'es with version < 3.2 and plan it.

   Pegasus now inspects the version number in the adag element to
   determine what parser will be loaded.

10) pegasus-tc-client

   pegasus-tc-client now displays output in the new multi line Text
   format, rather than the old File format.

   The support for the File format for the Transformation Catalog will
   be removed in the upcoming releases.    

11) Removed the requirements for specifying grid gateways in Site
Catalog
   
   Grid Gateways are associated with a site in a Site Catalog to
   designate the jobmanagers associated with a grid site. However, in
   the case where jobs were submitted in a pure condor enviornment or
   on a local sites ( where jobs are not submitted via jobmanagers),
   Pegasus still required users to associate dummy grid gateways with
   the site.  This is no longer required . The Grid Gateways need to
   be specified only for grid sites now.
	   
12)Workflow metrics file in the submit directory

   A workflow metrics file is created by the planner in the submit
   directory that gives a breakdown of various jobs in the executable
   workflow by type.
   

13) pegasus-plan is always niced

    Starting with this release, pegasus-plan always nice's the
    corresponding java invocation that launches the planner. This is
    helpful in keeping the load on the submit host in check.

14) Dropped support for VORS and MyOSG backends
    
    pegasus-sc-client now relies only on one backend ( OSGMM ) to
    generate a site catalog for OSG. VORS and MyOSG are no longer
    suppored by OSG.    

===============================
Release Notes for PEGASUS 3.0.3
===============================
This is a minor release, that fixes some bugs and has minor
enhancements. 


NEW FEATURES
--------------

1) job statistics file has a num field now

   The job statistics file has statistics about Condor jobs in the 
   workflow across retries. There is now a new field called num added 
   that indicates the number of times JOB_TERMINATED event is seen for
   a Condor Job.

2) improvements to pegasus-monitord

   When using MySQL, users no longer are required to create the database
   using the 'latin1' character encoding. Now, pegasus-monitord will 
   automatically create all tables using the 'latin1' encoding.

   When using MySQL, the database engine used by Pegasus-monitord is set
   to 'InnoDB'. This prevents certain database errors and allows for 
   improved performance.

3) inherited-rc-files option for pegasus-plan

   pegasus-plan has a new option --inherited-rc-files, that is used in
   hierarichal workflows to pass the file locations in the parent
   workflow's DAX to planner instances working on a subdax job. Locations
   passed via this option, have a lower priority than the locations of
   files mentioned in the DAX.

4) improved pegasus-status

   There is a slight change in default behavior of the tool, so please
   read the manpage or short information about it. By default, it will
   look both, into the current Condor Q and a workflow run directory 
   (if a valid one is the current one, or was specified.) Each of this
   behavior can be turned on and off separately.

   Improved output includes UTF-8 box drawing characters to show 
   dependencies, a color option (for white terminal backgrounds), and
   detection of the current terminal size. 

BUGS FIXED
----------
1) Fixed a bug in the code handling SIGUSR1 and SIGUSR2 that caused 
   pegasus-monitord to abort due to an out-of-bounds condition.

2) Fixed Python 2.4 compatibility issue that caused pegasus-monitord to
   abort when receiving a SIGUSR1 or SIGUSR2 to change its debugging level.

3) pegasus-transfer failed on scp if the destination URL was a file URL
   This is now fixed. More details at
   
   https://jira.isi.edu/browse/PM-375

4) pegasus transfer failed on scp if the destination host was not in users
   know_hosts. This is now fixed. More details at

   https://jira.isi.edu/browse/PM-374

5)  pegasus-plan had a potential stack overflow issue that could occur 
    while calling out to transformation selectors that return more than 
    one entry. 

6)  Destination file url's were not correctly replaced with symlink protocol
    scheme in the case where the destination site had a file server 
    ( url prefix file:/// ). 



===============================
Release Notes for PEGASUS 3.0.2
===============================
This is a minor release, that fixes some bugs and has minor
enhancements. 


NEW FEATURES
--------------

1) New Pegasus Properties for pegasus-monitord daemon

   The pegasus-monitord daemon is launched by pegasus-run while
   submitting the workflow, and by default parses the condor logs for
   the workflows and populates them in a sqllite DB in the workflow
   submit directory. 

   pegasus.monitord.events - A Boolean Property indicating whether to
   parse and generate log events or not.

   pegasus.monitord.output - This property can be used to specify the
   destination for generated log events in pegasus-monitord
	

2) Improvements to pegasus-monitord 

   pegasus-monitord now does batches evennts before popualating them
   in to the stampede backend.

3) New entries in braindump file
   
   The braindump file generated in the submit directory has two new
   keys. 

   The braindump file has two extra entries now

   properties - path to the properties file
   condor_log -  path to the condor log for the workflow

4) pegasus-transfer supports ftp transfers in unauthenticated mode.

BUGS FIXED
----------
1) Failure of rescue dags if submit directory on NFS.
   
   Pegasus creates a symlink in the submit directory to the condor log
   file for the workflow in /tmp . In case the workflow failed and the
   submit directory was on NFS, pegasus-run on rescue would take a
   backup of the symlink file in the submit directory. This resulted
   in the workflow failing on resubmission, as the condor log now
   pointed to a file in the submit directory that was on NFS.

   pegasus-submit-dag was fixed to copy the symlinked log while
   rotating instead of copying just the symlink.  


===============================
Release Notes for PEGASUS 3.0.1
===============================
This is minor release, that fixes some bugs discovered after 3.0
release and some  enhancements. 

NEW FEATURES
--------------

1) Pegasus VM Tutorial updated

   The Pegasus VM Tutorial has been updated to use a new smaller
   Virtual Image running Debian 6 on x86. The tutorial notes are
   available online at 

   http://pegasus.isi.edu/wms/docs/3.0/tutorial_vm.php

2) pegasus-plots improvements

   The worklfow gantt chart produced by pegasus-plots now has better
   legend placement. The legends are placed below the x axis so that
   they dont interfere with the chart in the plotting area. Also the
   gantt chart generated now uses a fixed pallete of colors to assign
   colors to the different compute jobs in the workflows. Earlier,
   random colors were assigned to the compute jobs.


Bugs FIXED
----------

1) In the case where symlinking was turned on and the pool attributes
   of the source URL ( discovered in the Replica Catalog ) matched with
   the compute site, the source URL was not converted to a file URL. This
   lead to the symlink job failing when the workflow executed. This is
   fixed now and the planner now converts the source URL's used for
   symlink jobs to a file URL scheme.


2) Metrics displayed by pegasus-statistics were not clear as to what
   the worfklow execution time was, as two metrics with similar meaning
   were displayed ( Workflow Execution Time and Workflow Walltime ). The
   metrics names have been changed and the meaning of the metrics are
   explained in detail in both the output of pegasus-statistics and in
   the online user guide. 
   http://pegasus.isi.edu/wms/docs/3.0/monitoring_debugging_stats.php#id2841404

   Additional clarifications have been made to as how the metrics
   calculations happen on hierarchal worklfows.

3) pegasus-transfer when in file copy mode complained if the
   destination file already existed. This is now fixed .

4) pegasus-bug-report was broken in 3.0 release. This is now fixed.

5) Added support in pegasus-monitord for handling SUBDAG EXTERNAL jobs 
   without the DIR option. 

   More details at This is related to JIRA issue PM-300.






===============================
Release Notes for PEGASUS 3.0
===============================
This is a major release that attempts to simplify configuration and
running workflows through Pegasus. 

Existing users please refer to the Migration Guide available at
http://pegasus.isi.edu/wms/docs/3.0/Migration_From_Pegasus_2_x.php .

A user guide is now available online at
http://pegasus.isi.edu/wms/docs/3.0/index.php

NEW FEATURES
--------------

1) Support for new DAX Schema Version 3.2

   Pegasus 3.0 by default  parses DAX documents conforming to the DAX
   Schema 3.2 and is explained online at
   http://pegasus.isi.edu/wms/docs/3.0/api.php . The same link has
   documentation about using the JAVA/Python and Perl API's to
   generate DAX3.2 compatible DAX'es. Users are encourage to use the
   API's in their DAX Generators.

2) New Multiline Text Format for Transformation Catalog

   The default format for Transformation Catalog is now Text which is
   a multiline textual format. It is explained in the Catalogs Chapter
   http://pegasus.isi.edu/wms/docs/3.0/catalogs.php#transformation 

3) Shell Code Generator

   Pegasus now has support for a Shell Code Generator that generates a
   shell script in the submit directory instead of Condor DAGMan and
   condor submit files. In this mode, all the jobs are run locally on
   the submit host.

   To use this set 
      - pegasus.code.generator Shell
      - make sure that --sites option passed to pegasus only has site
        local mentioned.

4) Profiles and Properties Simplification
   
   Starting with Pegasus 3.0 all profiles can be specified in the
   properties file. Profiles specified in the properties file have the
   lowest priority. 

   As a result of this a lot of existing Pegasus Properties were
   replaced by profiles. An exhaustive list of Properties replaced can
   be found at
   http://pegasus.isi.edu/wms/docs/3.0/Migration_From_Pegasus_2_x.php#id2765439

   All Profile Keys are  documented in the Profiles Chapter 
   http://pegasus.isi.edu/wms/docs/3.0/advanced_concepts_profiles.php

   The properties guide was broken into two parts and can be found in
   the installation in the etc directory
       - basic.properties   lists only the basic properties that need to
                            be set to use Pegasus. Sufficient for most
   			    users.
       - advanced.properties lists all the properties that can be used
       	 		     to configure Pegasus. 

    The properties documentation can be found online at
    http://pegasus.isi.edu/wms/docs/3.0/configuration.php
    http://pegasus.isi.edu/wms/docs/3.0/advanced_concepts_properties.php 



5) Transfers Simplification

   Pegasus 3.0 has a new default transfer client pegasus-transfer that
   is invoked by default for first level and second level staging. The
   pegasus-transfer client is a python based wrapper around various
   transfer clients like globus-url-copy, lcg-copy, wget, cp, ln
   . pegasus-transfer looks at source and destination url and figures
   out automatically which underlying client to use. pegasus-transfer
   is distributed with the PEGASUS and can be found in the bin
   subdirectory . 

   Also, the Bundle Transfer refiner has been made the default for
   pegasus 3.0. Most of the users no longer need to set any transfer
   related properties. The names of the profiles keys that control the
   Bundle Transfers have been changed . 

   To control the clustering granularity of stagein transfer jobs
   following Pegasus Profile Keys can be used

   - stagein.clusters
   - stagein.local.clusters
   - stagein.remote.clusters

   To control the clustering granularity of stageout transfer jobs

   - stageout.clusters
   - stageout.local.clusters
   - stageout.remote.clusters


6) New tools called pegasus-statistics and pegasus-plots

   There are new tools called pegasus-statistics and pegasus-plots that
   can be used to generate statistics and plots about a worklfow run. 
   
   The tools are documented in Monitoring and Debugging Chapter
   http://pegasus.isi.edu/wms/docs/3.0/monitoring_debugging_stats.php#id3083376 

7) Example Workflows

   Pegasus distribution comes with canned examples that users can use
   to run after installing Pegasus.

   The examples are documented in the Example Workflows Chapter
   http://pegasus.isi.edu/wms/docs/3.0/example_workflows.php

8) Support for GT5

   Pegasus now has support for GT5. 

   Users can use GT5 only if they are using the new site catalog
   schema ( XML3 ) , as only that has a place holder to specify grid
   type with the grid gateways element.  
   e.g 
   <site handle="isi_viz" arch="x86" os="LINUX" osrelease="" osversion="" glibc=""> 
               <grid type="gt5"  contact="viz-login.isi.edu/jobmanager-pbs" scheduler="PBS"
   jobtype="compute"/>  
               <grid type="gt2" contact="viz-login.isi.edu/jobmanager-fork" scheduler="Fork" jobtype="auxillary"/> 
          .... 
    </site> 

    Users can use pegasus-sc-converter to convert their site catalogs
    to the XML3 format.
    Sample Usage:
    sc-client -I XML -O XML3 -i sites.xml -o sites.xml3 

 
9) pegasus-analyzer can parse kickstart outputs
   
   pegasus-analyzer now parses kickstart outputs and prints meaningful
   information about the failed jobs.  The following information is
   printed for the failed jobs

   - the executable
   - the arguments
   - the site the job ran on
   - remote the working directory of the job
   - exitcode
   - node the job ran on
   - stdout | stderr

11) Logging improvements
    
    pegasus-plan now has a -q option to decrease the logging
    verbosity. The logging verbosity can be increased by using the -v
    options. Additionally, there are two new logging levels
    - CONSOLE ( enabled by default - messages like pegasus-run
    invocation are printed to this level )
    - TRACE  ( more verbose that DEBUG level ).

    The INFO level is no longer enabled by default. To see the INFO
    log level messages pass -v to pegasus-plan

    To turn on the TRACE level pass -vvvv . In the TRACE mode, in case
    of exceptions  the full stack trace for exceptions is printed.

12) Default condor priorities to jobs
    
    Pegasus assigns default Condor priorities to jobs now. The
    priorities come into play when running jobs in Condor vanilla /
    standard or local universe jobs. They dont apply for grid universe
    jobs.

    The default priority for various types of jobs is

    Cleanup     :  1000 
    Stage out   :  900 
    Dirmanager  :  800 
    Stage in    :  700 
    Compute jobs:  level * 10 where level is the level of the job in
    	           the workflow as compute from the root of the workflow.

    This priority assignment gives priority to workflows that are
    further along, but also overlap data staging and computation (data
    jobs have higher priority, but the assumptions is 
    that there are relatively few data staging jobs compared to compute
    jobs) .


13) Turning off Condor Log Symlinking to /tmp
    
    By default pegasus has the Condor common log -0.log in the submit
    file as a symlink to a location in /tmp . This is to ensure that
    condor common log does not get written to a shared filesystem. If
    the user knows for sure that the workflow submit directory is not
    on the shared filesystem, then they can opt to turn of the
    symlinking of condor common log file by setting the property
    pegasus.condor.logs.symlink to false. 


===============================
Release Notes for PEGASUS 2.4.3
===============================
This is minor release, that fixes some RPM and DEB packaging bugs. It
has improvements to the Data Reuse Algorithm and pegasus-analyzer.

NEW FEATURES
--------------

1) pegasus-analyzer has debug-job feature for pure condor environment

   pegasus-analyzer now has an option debug-job that generates a shell
   script for a failed job and allows users to run it later. This is
   only valid for pure condor environment where we relying on condor
   to do the file transfers. 
   The script will copy all necessary files to a local directory and
   invoke the job with the necessary command-line options. More
   details  in JIRA issue PM-92. Two options were added --debug-job,
   and  --debug-dir, where the first one indicated which job should
   be  debugged. The second option is used to specify where the user
   wants to debug this job (default is to create a temp dir).

2) New Implementation of Data Reuse Algorithm

   The data reuse algorithm reduces the workflow on the basis of
   existing  output files of the workflow found in the Replica
   Catalog. The algorithm works in two passes.

   In the first pass , we determine all the jobs whose output files
   exist  in the Replica Catalog. An output file with the transfer
   flag set to false is treated equivalent to the file existing in the
   Replica Catalog , if

   the output file is not an input to any of the children of the job X

   In the second pass, we remove the job whose output files exist in
   the Replica Catalog and try to cascade the deletion upwards to the
   parent jobs. We start the breadth first traversal of the workflow
   bottom up. 

   A node is marked for deletion if -

     ( It is already marked for deletion in pass 1
       OR
        ( ALL of it's children have been marked for deletion
       	  AND
           Node's output files have transfer flags set to false
      	 )
     )


3)  Workflow with NOOP Job is created when workflow is reduced fully

    In the case where the Data Reuse Algorithm reduces all the jobs in
    the workflow, a workflow with a single NOOP job is created. 

    This is to ensure that pegasus-run works correctly, and in case of
    workflows of workflows the empty sub workflows dont trigger an
    error in the outer level workflow. 


Bugs FIXED
----------

1) RPMs had mismatched permissions on the jars versus the setup.sh/
   setup.csh scripts. The result was empty CLASSPATHs after sourcing
   the setup scripts. The scripts have now been opened up to allow
   for the permissions of the files in the RPMs

2) A Debian packaging problem due to .svn files being left in the deb

3) The setup scripts can now guess JAVA_HOME using a common system
   install locations. This is useful for RPMs and DEBs which already
   have declared dependencies on certain JREs

4) pegasus-status incorrectly reported status of workflow when it
   starts up.

   pegasus-status incorrectly reported the number of workflows and %
   done when a workflow of workflows started. This is now fixed both
   in branch 2.4 and head.

5) NPE while label based clustering
   
   The logger object in the label based clusterer was incorrectly
   instantiated leading to a null pointer exception 

   This is fixed both in branch 2.4 and head

   More details athttps://jira.isi.edu/browse/PM-144

6) Incorrect -w option to kickstart for clustered jobs with Condor
   file staging. 

   When using condor file transfers and label-based clustering Pegasus
   generated the -w working directory option and sets it to a
   generated  path in /tmp. This broke the workflow because condor
   transfers  all the input files to the condor exec directory. This
   problem did not get triggered if the workflow was not clustered.

   More details athttps://jira.isi.edu/browse/PM-145

7) Condor stdout and stderr streaming

   Condor streaming is now supported in Condor for both grid and non
   grid universe jobs. We always put in the streaming keys. They
   default to false. 

   But can be overridden by properties.

   pegasus.condor.output.stream
   pegasus.condor.error.stream

8) Bug fix for exploding relative-dir option when using pdax

   There was a bug whereby the relative-dir option constructed for the
   sub workflows exploded when a user passed a pdax file to
   pegasus-plan. 
   
   Full details athttps://jira.isi.edu/browse/PM-142

9) Specifying SUNOS in old site catalog format

   The 2.4 branch was not translating OS'es correctly. This lead to a
   NPE if a user specified SUNOS in the old XML site catalog format.

   The conversion functions that convert internally to new format were
   expanded to support more OS

===============================
Release Notes for PEGASUS 2.4.2
===============================
This is minor release, that has some feature enhancements to pegasus-status
and pegasus-analyzer for working with Condor SUBDAG's.

NEW FEATURES
--------------

1) pegasus-status tracks workflows of workflows better now
  
   pegasus-status now always shows top most level dag last in the long
   output. It also displays the percentage  of work done on the outer
   most dag. It tracks SUBDAG correctly, even if the submit directory
   for those workflows is not rooted in the top level dag's submit
   directory.


2) pegasus-analyzer

   pegasus-analyzer now parses the VARS lines in the DAG file to perform
   variable substitution while figuring out the output and error files for
   the failed jobs. This is triggered when --strict option is passed to 
   pegasus-analyzer. In case of workflows of workflows all invocations
   of pegasus-analyzer for SUBDAGS have the --strict option included.

===============================
Release Notes for PEGASUS 2.4.1
===============================
This is minor release, that fixes some critical bugs discovered 
after 2.4.0 release and some feature enhancements.

NEW FEATURES
--------------

1) Specifying different relative submit and execution directories
   
   Earlier --relative-dir was used to determine the relative submit
   directory for a workflow and the relative execution directory on
   the remote site. Users now can optionally specify --relative-submit-dir
   if they want the relative submit directory to be different from
   the remtoe execution directory.

   This is useful when a user wants certain sub workflows in a 
   DAX 3.0 dax to be executed in the same execution directory , while 
   having the submit directories different.

   This feature addition was tracked in JIRA via PM-116

   http://jira.isi.edu/browse/PM-116

2) Python tools to visualize the DAX and DAG

   There are new tools dag2dot.py and dax2dot.py in the bin directory.
   - dag2dot.py reads a DAGMan .dag file (and the corresponding .sub files, if present). 
   - dax2dot.py reads a Pegasus DAX file.

   Notable features of the new scripts include:
   - Removal of redundant edges in the workflow to make the 
     resulting diagrams easier to look at.
   - Coloring by transformation type.
   - Removal of nodes by transformation type.

   The new scripts are in the $PEGASUS_HOME/bin directory.

3) Generating a site catalog for OSG for VO other than LIGO/Engage

   pegasus-get-sites can now generate a site catalog for OSG for any 
   VO using the OSGMM backend. Earlier a user could only generate
   a site catalog for LIGO and Engage VO.

   This feature addition was tracked in JIRA via PM-67
   http://jira.isi.edu/browse/PM-67

4) New features to pegasus-analyzer

   pegasus-analyzer can now run on other users submit directories.

   - Earlier pegasus-analyzer could not analyze workflow submit
     directories for which the user running pegasus-analyzer did 
     not have the write permissions. This was because pegasus-analyzer
     tried to generate a jobstate.log file in the workflow submit
     directory. This is now directed to a tmp file in the /tmp directory

   - pegasus-analyzer can be used to analyze SUBDAGS/workflows that
     are not planned via Pegasus

   - New command line option --print option that optionally can print
     the invocation and prescript for the failed jobs.

5) pegasus-status has a --long option

   If the --long option is set,  pegasus-status retrieves the last updated
   status of the workflow from the dagman out file, instead of displaying
   the running jobs in the condorq.

   For e.g here is how the sample output looks like

   $ pegasus-status --long .
   ./inspiral_hipe_datafind-0.dag
   03/25 04:24:22 Done Pre Queued Post Ready Un-Ready Failed
   03/25 04:24:22 === === === === === === ===
   03/25 04:24:22 5 0 50 0 43 0 0


BUGS FIXED
----------

1) Mismatched source and destination urls in input files for stagein jobs

   In certain cases, it was found that the source and destination pair in
   the input files for the stagein jobs were mismatched.

   The source url referred to one file and the destination file to another.

   This was triggered when pegasus detected circular symlinks . i.e a file 
   that was to be symlinked in the remote execution directory already existed
   ( accd to the entry in the replica catalog )

   More details athttp://jira.isi.edu/browse/PM-125

2) symlink jobs were not launched correctly via kickstart

   The symlink jobs while being launched via kickstart did not have the stdin 
   file transferred correctly from the submit host.

   This is now fixed.

   More details athttp://jira.isi.edu/browse/PM-124

3) Determining Condor Version
   
   Pegasus executes the condor_version command to determine the version of
   condor running on the submit host. This is used to create the correct 
   version of the dagman.sub files, when workflows of workflows are executed.

   Pegasus did not parse the following output correctly

   $CondorVersion: 7.4.1 Dec 17 2009 UWCS-PRE $

   The internal regex was updated to follow the following rule
   $CondorVersion: 7.4.1 Dec 17 2009 <ANY_ARBITRARY_STRING> $

   the version number and date will always be there.

   <ANY_ARBITRARY_STRING> may or may not be there, and can 
   include spaces but is really completely arbitrary. don't rely
   on that for the version information.just use the version and date.


===============================
Release Notes for PEGASUS 2.4.0
===============================
 
NEW FEATURES
--------------

1) Support for Pegasus DAX 3.0 

   Pegasus now also can accept DAX'es in Pegasus 3.0 format 

   Some salient features of the new format are
   - Users can specify locations of the files in the DAX
   - Users can specify what executables to use in the DAX
   - Users can specify sub dax in the DAX using the dax element. The
     dax jobs result in a separate subworkflow being launched with the 
     appropriate pegasus-plan command as the prescript
   - Users can specify condor DAG's in the DAX using the dag
     element. The dag job is passed on the Condor DAGMAN as a SUBDAG
     for execution.

   A sample 3.0  DAX can be found at 
   http://pegasus.isi.edu/mapper/docs/schemas/dax-3.0/two_node_dax-3.0_v6.xml

   In the next Pegasus release ( Pegasus 3.0 ) a JAVA DAX API will be
   made available. Certain more extensions will be added to the
   schema. For feature requests email pegasus@isi.edu

2) Support for running workflows on EC2 using S3 for storage

   Users while running on Amazon EC2 can use S3 for storage backend
   for the workflow execution. The details below assume that a user
   configures a condor pool on the nodes allocated from EC3

   To enable Pegasus for S3 the following properties need to be set.

   - pegasus.execute.*.filesystem.local = true
   - pegasus.transfer.*.impl = S3
   - pegasus.transfer.sls.*.impl = S3
   - pegasus.dir.create.impl = S3
   - pegasus.file.cleanup.impl = S3
   - pegasus.gridstart = SeqExec
   - pegasus.transfer.sls.s3.stage.sls.file = false

   For data stagein and creating S3 buckets for workflows pegasus
   relies on the amazon provided s3cmd command line client.
   
   Pegasus looks for a transformation with namespace amazon and
   logical name as s3cmd in the transformation catalog to figure out
   the location of the s3cmd client. for e.g in the File based
   Transformation Catalog the full name for transformation will be 
   amazon::s3cmd

   In order to enable stdtout and stderr streaming correctly from
   Condor on EC2 we recommend adding certain profiles in the site
   catalog for the cloud site.
   Here is a sample site catalog

   <site handle="ec2" sysinfo="INTEL32::LINUX">
      <profile namespace="env" key="PEGASUS_HOME">/usr/local/pegasus/default</profile>
      <profile namespace="env" key="GLOBUS_LOCATION">/usr/local/globus/default</profile>
      <profile namespace="env" key="LD_LIBRARY_PATH">/usr/local/globus/default/lib</profile>
   
       <!-- the directory where a user wants to run the jobs on the
   	nodes retrived from ec2 -->
       <profile namespace="env" key="wntmp">/mnt</profile>
     
       <profile namespace="pegasus" key="style">condor</profile>
    
       <!-- to be set to ensure condor streams stdout and stderr back
	to submit host -->	
       <profile namespace="condor" key="should_transfer_files">YES</profile>
       <profile namespace="condor" key="transfer_output">true</profile>
       <profile namespace="condor" key="transfer_error">true</profile>
       <profile namespace="condor" key="WhenToTransferOutput">ON_EXIT</profile>
       
       <profile namespace="condor" key="universe">vanilla</profile>

       <profile namespace="condor" key="requirements">(Arch==Arch)&amp;&amp;(Disk!=0)&amp;&amp;(Memory!=0)&amp;&amp;(OpSys==OpSys)&amp;&amp;(FileSystemDomain!="")</profile>
       <profile namespace="condor" key="rank">SlotID</profile>
   
       <lrc url="rls://example.com"/>
       <gridftp url="s3://" storage="" major="2" minor="4" patch="3"/>
       <jobmanager universe="vanilla" url="example.com/jobmanager-pbs" major="2" minor="4" patch="3"/>
       <jobmanager universe="transfer" url="example.com/jobmanager-fork" major="2" minor="4" patch="3"/>

       <!-- create a new bucket for each wf
           <workdirectory >/</workdirectory>
        -->
        <!-- use an existing bucket -->
   	<workdirectory>existing-bucket</workdirectory>
   </site>
   
   Relevant JIRA links
   http://jira.pegasus.isi.edu/browse/PM-68
   http://jira.pegasus.isi.edu/browse/PM-20
   http://jira.pegasus.isi.edu/browse/PM-85



3) pegasus-analyzer

   There is a new tool called pegasus-analyzer. It helps the users to
   analyze the workflows after the workflow has finished executing. 

   It is not meant to be run while the workflow is still running. To
   track the status of a running workflow for now, the users are
   recommended to use pegasus-status. 

   pegasus-analyzer looks at the workflow submit directory and parses
   the condor dagman logs and the job.out files to print a summary of
   the workflow execution. 

   The tool prints out the following summary of the workflow

   Total jobs
   jobs succeeded
   jobs failed
   jobs unsubmitted

   For all the failed jobs the tool prints out the contents of job.out
   and job.err file. 

   The user can use the --quiet option to display only the paths to
   the .out and .err files. This is useful when the job output is
   particularly big or when kickstart is used to launch the jobs. 

   For pegasus 3.0 the tool will be updated to parse kickstart output
   files and provide a concise view rather than displaying the whole
   output 

4) Support for Condor Glite

   Pegasus now supports a new style named glite for generating the submit
   files. This allows pegasus to create submit files for a glite
   environment where a glite blahp talks to the scheduler instead of
   GRAM. At a minimum the following profiles need to be associated with
   the job. 

   pegasus profile style - value set to glite
   condor profile grid_resource - value set to the remote scheduler to
   	  	  		  which glite blahp talks to .

    This style should only be used when the condor on the submit host
    can directly talk to scheduler running on the cluster. In Pegasus
    site  catalog there should be a separate compute site that has
    this style associated with it. This style should not be specified
    for the local site. 

    As part of applying the style to the job, this style adds the
    following classads expressions to the job description 

    +remote_queue - value picked up from globus profile queue
    +remote_cerequirements - See below

    The remote CE requirements are constructed from the following
    profiles associated with the job. The profiles for a job are
    derived from various sources 

    - user properties
    - transformation catalog
    - site catalog
    - DAX

    Note it is upto the user to specify these or a subset of them.

    The following globus profiles if associated with the job are picked up

    hostcount -> PROCS
    count -> NODES
    maxwalltime-> WALLTIME

    The following condor profiles if associated with the job are picked up

    priority -> PRIORITY

    All the env profiles are translated to MYENV

    For e.g. the expression in the submit file may look as

    +remote_cerequirements = "PROCS==18 && NODES==1 && PRIORITY==10 && WALLTIME==3600
       && PASSENV==1 && JOBNAME==\"TEST JOB\" && MYENV ==\"FOO=BAR,HOME=/home/user\""
 
    All the jobs that have this style applied dont have a remote
    directory specified in the submit directory. They rely on
    kickstart to change to the working directory when the job is
    launched on the remote node. 


5) Generating a site catalog for OSG using OSGMM
   The pegasus-get-sites tool has been modified to query the OSGMM (
   OSG Match Maker) to generate a site catalog for a VO 

   It builds upon the earlier Engage implementation. It has now been
   generalized and renamed to OSGMM 

   To pegasus-get-sites the source option now needs to be OSGMM
   instead of Engage 

   Some of the changes are

   The condor collector host can be specified at command line or in
   properties by specifying the property pegasus.catalog.site.osgmm.collector.host .
   It defaults to ligo-osgmm.renci.org 

   If a user is part of the Engage VO they should set 
   pegasus.catalog.site.osgmm.collector.host=engage-central.renci.org

   The default VO used is LIGO. Can be overriden by specifying the
   --vo option to pegasus-get-sites , or specifying the property
   pegasus.catalog.site.osgmm.vo 

   By default the implementation always returns validated sites. 
   To retrieve all sites for a VO set
   pegasus.catalog.site.osgmm.retrieve.validated.sites to false.

   In case of multiple gatekeepers are associated with the same osg
   site, multiple site catalog entries are created in the site
   catalog. A suffix is added to the extra sites (__index , where
   index starts from 1) 

   Sample Usage
   pegasus-get-sites --source OSGMM --sc osg-sites.xml --vo LIGO --grid OSG

   Tracked in JIRA athttp://pegasus.isi.edu/jira/browse/PM-67

   Currently, there is no way to filter sites according to the grid ( OSG|OSG-ITB ) in OSGMM

   The site catalog generated has storage directories that have a VO component in them.

 
6) Generating a site catalog for OSG using MYOSG
   pegasus-get-sites has now been modified to generate a site catalog by querying MyOSG

   To use MYOSG as the backend the source option needs to be set to MYOSG

   Sample usage

   pegasus-get-sites --source MYOSG --sc myosg-sites-new.xml -vvvvv --vo  ligo --grid osg
   
   This was tracked in JIRA
   http://pegasus.isi.edu/jira/browse/PM-61
   
   Pegasus Team recommends using OSGMM for generating a site catalog.

7) Separation of Symlink and Stagein Transfer Jobs

   The following transfer refiners
   - Default
   - Bundle
   - Cluster
   now support the separation of the symlink jobs from the stage in
   jobs. While using these refiners, the files that need to be
   symlinked against existing files on a compute site will have a
   separate symlink job. The files that need to be actually copied to
   a remote site, will appear in the stage_in_ jobs. 

   This distinction, allows for the users to stage in data using third
   party transfers that run on the submit host, and at the same time
   be able to symlink against existing datasets. 
   
   The symlink jobs run on the remote compute sites. Earlier this was
   not possible, and hence for a user to use symlinking they had to
   turn off third party transfers. This resulted in an increased load
   on the head node as the stage in jobs executed there. 

   By default, Pegasus will use the transfer executable shipped with
   the worker package to do the symbolic linking . 

   If the user wants to change the executable to use , they can set the following property

   pegasus.transfer.symlink.impl

   The above also allows us to use separate executables for staging in
   data and for symbolic linking. 
   For e.g. we can use GUC to stage in data by setting

   pegasus.transfer.stagein.impl GUC

   To control the symlinking granularity in the Bundle and Cluster
   refiners the following Pegasus profile keys can be associated 

   bundle.symlink
   cluster.symlink

   The feature implementation was tracked in JIRA at
   http://pegasus.isi.edu/jira/browse/PM-54

8) Bypassing First Level Staging of Files for worker node execution

   Pegasus now has  capability to bypass first level staging if the
   input files in the replica catalog have a pool attribute matching
   the site at which a job is being run. This applies in case of
   worker node execution. 

   The cache file generated in the submit directory is the transient
   replica catalog. It also now has locations of where the inpute
   files are staged on the remote sites. Earlier it was only the files
   that were generated by the workflow. 

   Tracked in JIRA here
   http://pegasus.isi.edu/jira/browse/PM-20
   http://pegasus.isi.edu/jira/browse/PM-62 

9) Resolving SRM URL's for file URL's on a filesystem
   
   There is now support to resolve the SRM urls in the replica catalog to
   the file url on a site. The user needs to specify the URL prefix
   and the mount point of the filesystem.

   This can be done by specifying the properties

   pegasus.transfer.srm.[sitename].service.url
   pegasus.transfer.srm.[sitename].service.mountpoint

   Pegasus will then map SRM URL's associate with site to a paht on
   the filesytem by replacing the service url component with the mount
   point. 

   For example if user has this specified

   pegasus.transfer.srm.ligo-cit.service.url          srm://osg-se.ligo.caltech.edu:10443/srm/v2/server?SFN=/mnt/hadoop
   pegasus.transfer.srm.ligo-cit.service.mountpoint   /mnt/hadoop/
   
   then url
   srm://osg-se.ligo.caltech.edu:10443/srm/v2/server?SFN=/mnt/hadoop/ligo/frames/S5/test.gwf
   will resolve to 

   /mnt/hadoop/ligo/frames/S5/test.gwf

10) New Transfer implementation Symlink

   Pegasus has now support for a perl executable called symlink
   shipped with the Pegasus worker package, that can be used to create
   multiple symlinks against input datasets in a single invocation 

   The Transfer implementation that uses the transfer executable also
   has the same functionality. 
   However, the transfer executable complains if it cannot find the
   Globus client libraries. 

   In order to use this executable for the symlink jobs, users need to
   set the following property 
   
   pegasus.transfer.symlink.impl Symlink

   Later on ( pegasus 3.0 release onwards ) this will be made the
   default executable to be used for symlinking jobs. 

11) Passing options forward to pegasus-run in pegasus-plan

   Users can now pass forward option to pegasus-run invocation that is
   used to submit the workflows in case of successful mapping.

   There is a  --forward option[=value] to pegasus-plan . This option
   allows a user to forward options to pegasus-run.
   For e.g. nogrid option can be passed to pegasus-run as follows
   pegasus-plan --forward nogrid

   The option can be repeated multiple times to forward multiple
   options to pegasus-run. The longopt version should always be
   specified for pegasus-run. 

12) Passing extra arguments SLS transfer implementations

   Users now can specify pegasus.transfer.sls.arguments to pass extra
   options at runtime to the SLS Implementations used by Pegasus. 
   The following SLS transfer implementations accept the above property.

   S3
   Transfer

13) Passing non standard java options to dax jobs in DAX 3.0

   The non standard jvm options (-X[option]) can now be specified for
   the sub workflows in the arguments section for the dax jobs. 

   For example for the DAX jobs , user can set the java max heap size
   to 1024m by specifying -X1024m in the arguments for the DAX job 



14) Location of Condor Logs directory on the submit host
   
   By default, pegasus designates the condor logs to be created in the
   /tmp directory. This is done to ensure that the logs are created in
   a local directory even though the submit directory maybe on NFS. 

   In the submit directory the symbolic link to the appropriate log
   file in the /tmp exists. However, since /tmp is automatically
   purged in most cases, users may want to preserve their condor logs
   in a directory on the local filesystem other than /tmp 

   The new property

   pegasus.dir.submit.logs

   allows a user to designate the logs directory on the submit host
   for condor logs. 

15) Removing profile keys as part of overriding profiles

   There is now a notion of empty profile key valus in Pegasus.  The
   default action on empty key value is to remove the key. Currently
   the following namespaces follow this convention 

       - Condor
       - Globus
       - Pegasus

   This allows a user to unset values as part of overriding
   profiles. Normally a user can only update a profile value i.e they
   can update the value of a key, but the key remains associated with
   the job This allows the user to remove the key from the profile
   namespace. 

   For e.g.

   A user may have a profile X set in the site catalog.

   Now for a particular job a user does not want that profile key to
   be used. He can now specify the same profile X with empty value in
   the transformation catalog for that job. This results in the
   profile key X being removed from the job. 

16) Constructing Paths to Condor DAGMan for recursive/hierarichal
   workflows

   The entry for condor::dagman is no longer required for site local
   in the transformation catalog. 
   Instead pegasus constructs path from the following environment
   variables. CONDOR_HOME, CONDOR_LOCATION

   The priority order is as follows

   1) CONDOR_HOME defined in the environment
   2) CONDOR_LOCATION defined in the environment
   3) entry for condor::dagman for site local

   This is useful when running workflows that refer to sub workflows
   as in the new DAX 3.0 format.

   This was tracked in JIRA
   http://pegasus.isi.edu/jira/browse/PM-50

17) Constructing path to kickstart

   By default the path to kickstart is determined on the basis of the
   environment variable PEGASUS_HOME associated with a site entry in
   the site catalog. 

   However, in some cases a user might want to use their own modified
   version of kickstart. 

   In order to enable that

   The path to kickstart will be constructed according to the following rule	
   1) pegasus profile gridstart.path specified in the site catalog for
      the site in question. 
   2) If 1 is not specified, then a path is constructed on the basis
   of the environment variable PEGASUS_HOME for the site in the site
   catalog. 

   The above was tracked in JIRA
   http://pegasus.isi.edu/jira/browse/PM-60

18) Bulk Lookups to Replica Catalog using rc-client
   
   rc-client now can do bulk lookups similar to how it does bulk
   inserts and deletes

   Details at
   http://jira.pegasus.isi.edu/browse/PM-75

19) Additions to show-job workflow visualization script

   show-job now has a --title option to list add a user provided title for the generated gantt chart.

   show-job can also visualize workflow of workflows

20) Absolute paths for certain properties in the properties file

   The properties file that is written out now in the submit directory
   has absolute paths specified for the following property values. 

   pegasus.catalog.replica.file
   pegasus.catalog.transformation.file
   pegasus.catalog.site.file

   This is even though user may have specified relative paths in properties file.


21) The default horizontal clustering factor

   Updated the default clustering factor as collapse with value = 1, instead of earlier value of 3

   This ensures, that users can cluster only jobs of certain types,
   and let others remain unclustered. Another way was to specify the
   collapse factor as 1 explicitly for jobs that users dont want
   clustering for. 

BUGS FIXED
----------
1) Handling of standard universe in condor style
   In Condor style , standard universe if specified for a job is ONLY
   associated for compute jobs. This ensures that pegasus auxillary
   jobs never execute in standard universe. 

2) Bug Fix for replica selection bug 43
   Checked in the fix for JIRA bug 43 http://pegasus.isi.edu/jira/browse/PM-43

   The ReplicaLocation class now has a clone method that does a
   shallow clone 
   This clone method is called in the selectReplica methods in the replica selectors.

3) rc-client did not implement  pegasus.catalog.replica.lrc.ignore property
   This is now fixed.
   This bug was tracked in JIRA http://pegasus.isi.edu/jira/browse/PM-42

4) DAX'es created while partitioning a workflow
   During the partitioning of the workflows , the DAX for a partition
   was created incorrectly as the register flags were not correctly
   parsed by the VDL DAX parser. 

   This was tracked in JIRA
   http://pegasus.isi.edu/jira/browse/PM-48

5) Handling of initialdir and remote_initialdir keys
   Changed the internal handling for the initialdir and
   remote_initialdir keys. The initialdir key is now only associated
   for standard universe jobs. For glidein and condor style we now
   associate remote_initialdir unless it is a standard universe job. 

   This was tracked in JIRA
   http://pegasus.isi.edu/jira/browse/PM-58

6) Querying RLI for non existent LFN using rc-client
   rc-client had inconsistent behavior when querying RLI for a LFN
   that does not exist in the RLS. 
   This affected the rc-client lookup command option.

   Details at
   http://jira.pegasus.isi.edu/browse/PM-74

7) Running clustered jobs on the cloud in directory other than /tmp
   There was a bug whereby the clustered jobs executing in worker node
   execution mode did not honor the wntmp environment variable
   specified in the Site Catalog for the site. 

   The bug fix was tracked through JIRA
   http://jira.isi.edu/browse/PM-83

8) Bug Fix for worker package deployment
   The regex employed to determine the pegasus version from a URL to a
   worker package was insufficient. It only took care of x86 builds. 

   For e.g. it could not parse the following
   urlhttp://pegasus.isi.edu/mapper/download/nightly/pegasus-worker-2.4.0cvs-ia64_rhas_3.tar.gz
   STATIC_BINARY INTEL64::LINUX NULL 

   This is now fixed.
   Related to JIRA PM-33

9) Destination URL construction for worker package staging
   Earlier the worker package input files always had third party
   URL's, even if the worker package deployment job executed on the
   remote site ( in push / pull mode ). 

   Now, the third party URL's are only constructed if the worker
   package deployment job is actually run in third party mode.		
   In push-pull mode, the destination URL's are file URLs	 

   Tracked in JIRA athttp://jira.isi.edu/browse/PM-89

Documentation
--------------

1) User Guides
   The Running on different Grids Guide now has information on how to
   run workflows using glite.   
   - Pegasus Replica Selection

   The guides are checked in $PEGASUS_HOME/doc/guides

   They can be found online at
   http://pegasus.isi.edu/mapper/doc.php
   
2) Property Document was updated with the new properties introduced.


===============================
Release Notes for PEGASUS 2.3.0
===============================
 
NEW FEATURES
--------------
1) Regex Based Replica Selection
   Pegasus now allows users to use regular expression based replica
   selection. To use this replica selector, users need to set the
   following property 
   
   pegasus.selector.replica  Regex 

   The Regex replica selector allows the user allows the user to
   specifiy the regex expressions to use for ranking various PFNs
   returned from the Replica Catalog for a particular LFN. This
   replica selector selects the highest ranked PFN i.e the replica
   with the lowest rank value. 

   The regular expressions are assigned different rank, that determine
   the order in which the expressions are employed. The rank values
   for the regex can expressed in user properties using the property. 

   pegasus.selector.replica.regex.rank.[value]

   The value is an integer value that denotes the rank of an
   expression with a rank value of 1 being the highest rank. 

   For example, a user can specify the following regex expressions
   that will ask Pegasus to prefer file URL's over gsiftp url's from
   example.isi.edu 
   
   pegasus.selector.replica.regex.rank.1 file://.*
   pegasus.selector.replica.regex.rank.2 gsiftp://example\.isi\.edu.*

   User can specify as many regex expressions as they want.
   Since Pegasus is in Java , the regex expression support is what
   Java supports. It is pretty close to what is supported by
   Perl. More details can be found at
   http://java.sun.com/j2se/1.5.0/docs/api/java/util/regex/Pattern.html 

   There is documentation about the new replica selector in the
   properties document . It can also be found at
   $PEGASUS_HOME/etc/sample.properties 

   To use this set pegasus.selector.replica Regex


2) Automatic Determination of pool attributes in RLS Replica Catalog

   Pegasus can now associate a pool attribute with the replica catalog
   entries returned from querying a LRC if the pool attribute is not
   already specified. 

   This is achieved by associating the site handles with corresponding
   LRC url's in the properties file. This mapping tells us what
   default pool attribute should be assigned while querying a
   particular LRC. For example

   pegasus.catalog.replica.lrc.site.llo rls://ldas.ligo-la.caltech.edu:39281
   pegasus.catalog.replica.lrc.site.lho rls://ldas.ligo-wa.caltech.edu:39281

   tells Pegasus that all results from LRC
   rls://ldas.ligo-la.caltech.edu:39281 are associated with site llo 

   Using this feature only makes sense, when a LRC *ONLY* contains
   mapping for data on one site, as in case of LIGO LDR deployment.

3) Pegasus auxillary jobs on submit host now execute in local universe

   All the scheduler universe jobs are now executed in local
   universe. Also any job planned for site local will by default run
   in local universe instead of scheduler universe.

   Additionally, extra checks were put in to handle the Condor File
   Transfer Mechansim issues in case local/scheduler universe. This
   was tracked in bugzilla at
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=40 

   A user can override the local universe generation by specifying the
   condor profile key universe and setting it to the value desired.

4) Python API for generating DAX and PDAX
   
   Pegasus now includes a Python API for generating DAXes and PDAXes.

   An example can be found online at
   http://vtcpc.isi.edu/pegasus/index.php/ChangeLog#Added_Python_API_for_DAX_and_PDAX 

   For more information on the DAX API type: pydoc Pegasus.DAX2
   For more information on the PDAX API type: pydoc Pegasus.PDAX2


5) Interface to Engage VO for OSG

   There is a new Site Catalog Implementation called Engage that
   interfaces with the Engage VO to discover resource information
   about OSG from the information published in RENCI glue classads.

   To use it set
   pegasus.catalog.site Engage

   To generate a site catalog using pegasus-get-sites set the source
   option to Engage 

   pegasus-get-sites --source Engage --sc engage.sc.xml

6) Gensim now reports Seqexec Times and Seqexec Delays


   Gensim script ($PEGASUS_HOME/contrib/showlog/gensim) now reports
   the seqexec time and the seqexec delay for the clustered jobs. 

   There are two new columns in the jobs file created by seqexec
   - seqexec
   - seqexec delay.

   The seqexec time is determined from the last line of the .out file
   of the clustered jobs. E.g format [struct stat="OK", lines=4,
   count=4, failed=0,
   duration=21.836,start="2009-02-20T16:14:56-08:00"] 

   The seqexec delay is the seqexec time - kickstart time. 
   
   This useful for analyzing large scale workflow runs.

7) Properties to turn on or off the seqexec progress logging

   The property  pegasus.clusterer.job.aggregator.seqexec.hasgloballog
   is now deprecated.

   It has been replaced by  two boolean properties
   - pegasus.clusterer.job.aggregator.seqexec.log whether to log
     progress or not 
   - pegasus.clusterer.job.aggregator.seqexec.log.global whether to
     log progress to global file or not. 

     The pegasus.clusterer.job.aggregator.seqexec.log.global only
     comes into effect when
     pegasus.clusterer.job.aggregator.seqexec.log is set to true 


8) Passing of the DAX label to kickstart invocation

   Now, the kickstart invocation for the jobs is always passed the dax
   label using the -L option. To disable the passing of the DAX label,
   user needs to set pegasus.gridstart.label to false

   Additionally, the basename option to pegasus-plan overrides the
   label value retrieved from the DAX. 

9) show-job works on MAC OSX platform

   $PEGASUS_HOME/contrib/showlog/show-job now does not fail on
   unavailability of convert program. It only logs a warning and
   creates the    EPS File , but not the png files. This allows us to
   run show-job on MAC OSX systems. 

10) Enabling InPlace cleanup in deferred planning

    By default in case of deferred planning cleanup is turned off as
    the cleanup algorithm does not work across partitions. 
    However, in scenarios where the partitions themseleves are
    independant ( i.e. dont share files ), user can safely turn on
    cleanup. 

    This can now be done by setting
    pegasus.file.cleanup.scope  deferred

    If the property is set to deferred, and the users wants to disable
    cleanup , they can still specify --nocleanup option on command
    line and that is honored. 

    However in case of scope fullahead for deferred planning, the
    command line options are ignored and always nocleanup is set.
    
11) New Pegasus Job Classad
    
    Pegasus now publishes a job runtime classad with the jobs. The
    class ad key name is pegasus_job_runtime. The value passed to it
    is picked up from the Pegasus Profile runtime. If the Pegaus
    Profile is not associated, then the globus maxwalltime profile key
    is used. If both are not set, then a value of zero is published.

    This job classad can be used for users in case of glidein, to
    ensure that the jobs complete before the nodes expire. 

    For the coral glidein service the sub expression to job
    requirement swould look something like this 

    (CorralTimeLeft > MY.pegasus_job_runtime)

12) [workflow].job.map file

    Pegasus now creates a [workflow].job.map file that links jobs in
    the DAG with the jobs in the DAX. The contents of the file are in
    netlogger format. 

    The [workflow] is replaced by the name of the workflow i.e. same
    prefix as the .dag file 
    
    In the file there are two types of events.
    a) pegasus.job 
    b) pegasus.job.map

    pegasus.job - This event is for all the jobs in the DAG. The
    following information is associated with this event. 

    - job.id the id of the job in the DAG
    - job.class an integer designating the type of the job
    - job.xform the logical transformation which the job refers to.
    - task.count the number of tasks associated with the job. This is
       equal to the number of pegasus.job.task events created for that
     job.
 
    pegasus.job.map - This event allows us to associate a job in the
    DAG with the jobs in the DAX. The following information is
    associated with this event. 

    -task.id the id of the job in the DAG
    -task.class an integer designating the type of the job
    -task.xform the logical transformation which the job refers to.


13) Source Directory for Worker Package Staging

    Users now can specify the  property
    pegasus.transfer.setup.source.base.url to specify the URL to the
    source directory containing the pegasus worker packages. If it is
    not specified, then the worker packages are pulled from the http
    server at pegasus.isi.edu during staging of executables. 


BUGS FIXED
----------
1)  Critical Bug Fix to rc-client

    SCEC reported a bug with the rc-client while doing bulk inserts
    into RLS. The bug was related to how logging is initialized
    internally in the client. 

    Details of the bug fix can be found at
    http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=38

2)  Bug Fix to tailstatd for parsing jobnames with . in them

    There was a bug where tailstatd incorrectly generated events in
    the jobstate.log while parsing condor logs. This was due to an
    errorneous regex expression for determining the event
    POST|PRE SCRIPT STARTED. 

    The earlier expression did not allow for . in jobnames. This is
    especially prevalent in LIGO workflows where the DAX labels have
    . in them. 

    An example of the problem line in DAGMan log
    1/24 10:11:21 Running POST script of Node
    inspiral_hipe_eobinj_cat2_veto.EOBINJ_CAT_2_VETO.daxlalapps_sire_ID000731...

    Earlier the job id was parsed as inspiral_hipe_eobinj_cat2_veto
    instead of
    inspiral_hipe_eobinj_cat2_veto.EOBINJ_CAT_2_VETO.daxlalapps_sire_ID000731 

3) Pegasus Builds on FC10

   Earlier the Pegasus builds were failed on FC10 as the invoke c tool
   did not build correctly. This is now fixed. 

   Details at
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=41

4) tailstatd killing jobs by detecting starvation

   tailstatd removes a job after four hours when the job has been
   waiting in the queue WITHOUT being marked as EXECUTE in the condor
   log. To override tailstatd has an option of setting starvation time
   to 0 via command line or via pegasus.max.idletime property.  The if
   condition in the perl script was not accepting 0 as a value when
   trying to override the default 4 hour starvation time. This fix
   allows the value to be set to 0 (turn of starvation checks) or any
   other value via the property pegasus.max.idletime. 

   This was tracked in pegasus jira as bug 40
   http://pegasus.isi.edu/jira/browse/PM-40

Documentation
--------------

1) User Guides
   The release has new user guides about the following
   - Pegasus Job Clustering
   - Pegasus Profiles
   - Pegasus Replica Selection

   The guides are checked in $PEGASUS_HOME/doc/guides

   They can be found online at
   http://pegasus.isi.edu/mapper/doc.php
   
2) Property Document was updated with the new properties introduced.

===============================
Release Notes for PEGASUS 2.2.0
===============================
 
NEW FEATURES
--------------
1) Naming scheme changed for auxillary jobs

   Pegasus during the refinement of the abstract workflow to the
   executable workflows adds auxillary jobs to do data stagein/stageout, 
   create work directories for workflow etc. The prefixes/suffixes added
   for these jobs has been changed.

   Type of Job			    |   Old Prefix	 | New Prefix
   -------------------------------------------------------------------
   Data Stage In Job		    |  rc_tx_		 | stage_in_
   Data Stage Out Job 	 	    |  new_rc_tx_	 | stage_out_
   Data Stage In Job between sites  |  inter_tx_	 | stage_inter_
   Data Registration Job	    |  new_rc_register_	 | register_
   Cleanup Job	     		    |  cln_	         | clean_up_   
   Transfer job to transfer the	    |  setup_tx_ 	 | stage_worker_
   worker package  	    	    |  			 |

   Additionally, the suffixes for the create dir jobs are now replaced 
   by prefixes

   Type of Job			    |   Old Suffix	 | New Prefix
   -------------------------------------------------------------------
   Directory creation job	    |  _cdir		 | create_dir_
   Synch Job in HourGlass mode	    |  pegasus_concat	 | pegasus_concat_
   

2) Staging of worker package to remote sites

   Pegasus now supports staging of worker package as part of the workflow.

   This feature is tracked through pegasus bugzilla .

   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=35

   The worker package is staged automatically to the remote site, by 
   adding a setup transfer job to the workflow. 

   The setup transfer job by default uses GUC to stage the data.
   However, this can be configured by setting the property 
   pegasus.transfer.setup.impl property. If you also, have 
   pegasus.transfer.*.impl set in your properties file, then you need 
   explicilty set pegasus.transfer.setup.impl to GUC


   The code discovers the worker package by looking up pegasus::worker
   in the transformation catalog. 
   Note: that the basename of the url's should not be changed. Pegasus
   parses the basename to determine the version of the worker package.
   Pegasus automatically determines the location of the worker package 
   to deploy on the remote site. Currently default mappings are as
   follows 
   INTEL32 => x86 
   AMD64 => x86_64 or x86 if not available 
   INTEL64 =>x86

   OS LINUX = rhel3

   There is an untar job added to the workflow after the setup job that
   un tars the worker package on the remote site. It defaults to /bin/tar .
   However can be overriden by specifying the entry tar in the
   transformation catalog for a particular site.


3) New Site Catalog Schema
   
   This release of Pegasus has support for site catalog schema version 3.

   HTML visualization of schema:
   http://pegasus.isi.edu/mapper/docs/schemas/sc-3.0/sc-3.0.html

   Schema itself:
   http://pegasus.isi.edu/schema/sc-3.0.xsd

   A sample xml file :
   http://pegasus.isi.edu/schema/sc-3.0-sample.xml

   To use a site catalog in the new format set
   pegasus.catalog.site  XML3   

   Changes to sc-client
   sc-client command line tool was updated to convert an existing
   site catalog from old format to the new format.

   Sample usage
   sc-client -i ldg-old-sites.xml -I XML -o ldg-new-sites.xml -O XML3
   sc-client --help gives detailed help

4) pegasus-get-sites
   
   pegasus-get-sites was recoded in JAVA and now generates the site 
   catalog confromant to schema sc-3.0. 

   Sample Usage to query VORS to generate a site catalog for OSG.
   pegasus-get-sites --source VORS --grid osg -s ./sites-new.xml

   The value passed to the source option is case sensitive. 
   
   Additionally, the VORS module of pegasus-get-sites determines the
   value of GLOBUS_LOCATION variable dependant on whether the
   auxillary jobmanager is of type fork or not. 

   If it is of type fork then picks up the value of GLOBUS_LOCATION
   variable published in VORS for that site. else it picks up the
   value from OSG_GRID variable published in VORS for that
   site. i.e. GLOBUS_LOCATION is set to $OSG_GRID/globus 

5) Overhaul of logging
  
   The Pegasus logging interfaces have been reworked. 
   Now users can specify the logger they want to use, by specifying the
   property pegasus.log.manager .

   Currently, two logging implementations are supported.
   Default - Pegasus homegrown logger that logs to stdout and stderr 
   	     directly.
   Log4j  -  Uses log4j to log the messages.
 
   The Log4j properties can be specified at runtime by specifying the
   property pegasus.log.manager.log4j.conf

   The format of the log message themselves can be specified at runtime
   by specifying the property pegasus.log.manager.formatter

   Right now two formatting modes are supported

   a) Simple - This formats the messages in a simple format. The 
      messages are logged as is with minimal formatting. Below are 
      sample log messages in this format while ranking a dax according 
      to performance.

      event.pegasus.ranking dax.id se18-gda.dax - STARTED
      event.pegasus.parsing.dax dax.id se18-gda-nested.dax - STARTED
      event.pegasus.parsing.dax dax.id se18-gda-nested.dax - FINISHED
      job.id jobGDA
      job.id jobGDA query.name getpredicted performace time 10.00
      event.pegasus.ranking dax.id se18-gda.dax - FINISHED

   b) Netlogger - This formats the messages in the Netlogger format , 
      that is based on key value pairs. The netlogger format is useful 
      for loading the logs into a database to do some meaningful analysis.

      Below are sample log messages in this format while ranking a dax 
      according to performance.

      ts=2008-09-06T12:26:20.100502Z event=event.pegasus.ranking.start \
      msgid=6bc49c1f-112e-4cdb-af54-3e0afb5d593c \
      eventId=event.pegasus.ranking_8d7c0a3c-9271-4c9c-a0f2-1fb57c6394d5 \
      dax.id=se18-gda.dax prog=Pegasus

      ts=2008-09-06T12:26:20.100750Z event=event.pegasus.parsing.dax.start \
      msgid=fed3ebdf-68e6-4711-8224-a16bb1ad2969 \			   
      eventId=event.pegasus.parsing.dax_887134a8-39cb-40f1-b11c-b49def0c5232\
      dax.id=se18-gda-nested.dax prog=Pegasus
 
      ts=2008-09-06T12:26:20.100894Z event=event.pegasus.parsing.dax.end \
      msgid=a81e92ba-27df-451f-bb2b-b60d232ed1ad \
      eventId=event.pegasus.parsing.dax_887134a8-39cb-40f1-b11c-b49def0c5232

      ts=2008-09-06T12:26:20.100395Z event=event.pegasus.ranking \
      msgid=4dcecb68-74fe-4fd5-aa9e-ea1cee88727d \
      eventId=event.pegasus.ranking_8d7c0a3c-9271-4c9c-a0f2-1fb57c6394d5 \
      job.id="jobGDA"

      ts=2008-09-06T12:26:20.100395Z event=event.pegasus.ranking \
      msgid=4dcecb68-74fe-4fd5-aa9e-ea1cee88727d \
      eventId=event.pegasus.ranking_8d7c0a3c-9271-4c9c-a0f2-1fb57c6394d5 \
      job.id="jobGDA" query.name="getpredicted performace" time="10.00"

      ts=2008-09-06T12:26:20.102003Z event=event.pegasus.ranking.end \
      msgid=31f50f39-efe2-47fc-9f4c-07121280cd64 \
      eventId=event.pegasus.ranking_8d7c0a3c-9271-4c9c-a0f2-1fb57c6394d5

6) New Transfer Refiner 

   Pegasus has a new transfer refiner named Cluster.

   In this refinement strategy, clusters of stage-in and stageout jobs are
   created per level of the workflow. It builds upon the Bundle refiner. 
   
   The differences between the Bundle and Cluster refiner are as follows.
   - stagein is also clustered/bundled per level. In Bundle it was
     for the whole workflow.
   - keys that control the clustering ( old name bundling are )
     cluster.stagein and cluster.stageout instead of bundle.stagein and
     bundle.stageout

   This refinement strategy also adds dependencies between the stagein 
   transfer jobs on different levels of the workflow to ensure that stagein
   for the top level happens first and so on.

   An image of the workflow with this refinement strategy can be found
   at 
   http://vtcpc.isi.edu/pegasus/index.php/ChangeLog#Added_a_Cluster_Transfer_Refiner

7) New Transfer Implementation for GUC from globus 4.x
   
   Pegasus has a new transfer implementation that allows it to use GUC
   from globus 4.x series to transfer multiple files in one job.
   
   In order to use this transfer implementation
      - the property pegasus.transfer.*.impl must be set to value GUC. 
  
   There should be an entry in the transformation catalog with the
   fully qualified  name as globus::guc for all the sites where
   workflow is run, or on the local site in case of third party
   transfers.
 
   Pegasus can automatically construct the path to the globus-url-copy
   client, if the environment variable GLOBUS_LOCATION is specified in
   the site catalog for the site. 

   The arguments with which the client is invoked can be specified
         - by specifying the property pegasus.transfer.arguments
         - associating the Pegasus profile key transfer.arguments
 

8) Recursive DAX'es
   
   There is prototypical support for recursive dax'es. Recursive
   DAX'es give you the ability to specify a job in the DAX that points
   to another DAX that has to be executed. 

   There is a sample recursive dax at
   $PEGASUS_HOME/examples/recursive.dax

   The dax refers to pegasus jobs in  turn plan and execute a dax

   To get this dax planned by pegasus you will need to have additional
   entries for dagman and pegasus in your transformation catalog.  

   For e.g. 

   local   condor::dagman  /opt/condor/7.1.0/bin/condor_dagman     INSTALLED       INTEL32::LINUX  NULL
   local  pegasus::pegasus-plan:2.0       /lfs1/software/install/pegasus/default  INSTALLED       INTEL32::LINUX  NULL

   The recursive dax needs to be planned for site local, since the
   pegasus itself runs on local site. The jobs in the dax specify -s
   option where you want each of your workflows to run.

   Recursive DAX do not need to contain only pegasus jobs. They can
   contain application/normal jobs that one usually specifies in a
   DAX. Pegasus determines that a particular job is planning and
   execute job by looking for a pegasus profile key named type with
   value recursive e.g.

   <job id="ID0000003" namespace="pegasus" name="pegasus-plan" version="2.0">
    <profile namespace="pegasus" key="type">recursive</profile>
    <argument>-Dpegasus.user.properties=/lfs1/work/conf/properties
    --dax /lfs1/work/dax3  -s tacc -o local --nocleanup  --force
    --rescue 1 --cluster horizontal -vvvvv --dir ./dag_3 
    </argument>
   </job>

09) Rescue option to pegasus-plan for deferred planning

    A rescue option to pegasus-plan has been added. The rescue option
    takes in an integer value, that determines the number of times
    rescue dags are submitted before re-planning is triggered in case
    of failures in deferred planning. For this to work, Condor 7.1.0
    or higher is required as it relies on the recently implemented
    auto rescue feature in Condor DAGMan. 

    Even though re-planning is triggered, Condor DAGMan still ends up
    submitting the rescue dag as it auto detects. The fix to it is to
    remove the  rescue dag files in case of re-planning. This is still
    to be implemented 

10) -j|--job-prefix option to pegasus-plan

    pegasus-plan can now be passed the -j|--job-prefix option to
    designate the prefix that needs to be used for constructing the
    job submit file.


11) Executing workflows on Amazon EC2
   
   Pegasus now has support of running workflows on EC2 with the
   storage of files on S3.  This feature is still in testing phase and
   has not been tested fully.

   To execute workflows on EC2/S3, Pegasus needs to be configured to
   use S3 specific implementations of it's internal API's

   a) First level Staging API - The S3 implementation stages in from the
      local site ( submit node ) to a bucket on S3. Similarly the data is
      staged back from the bucket to the local site ( submit node )
      . All the first level transfers happen between the submit node
      and the cloud. This means that input data can *only* be present
      on the submit node when running on the cloud, and the output
      data can be staged back only to the submit node.

   b) Second Level Staging API - The S3 implementation retrieves input
      data from the bucket to the worker node tmp directory and puts
      created data back in the bucket. 

   c) Directory creation API - The S3 implementation creates a bucket
      on S3 for the workflow instead of a directory. 

   d) Cleanup API - To cleanup files from the workflow specific bucket
      on S3 during workflow execution.
   
   The above implementations rely on s3cmd command line client to
   interface with S3 filesystem. There should be an entry in the
   transformation catalog with the fully qualified name as
   amazon::s3cmd for the site corresponding to the cloud and the local
   site. 

   To configure Pegasus to use these implementations set the following
   properties 

   pegasus.transfer.*.impl                      S3
   pegasus.transfer.sls.*.impl                S3
   pegasus.dir.create.impl                      S3
   pegasus.file.cleanup.impl                  S3

   pegasus.execute.*.filesystem.local   true


12) Support for OSU Datacutter jobs

   Pegasus has new gridstart mode called DCLauncher. This allows us to
   launch the Data Cutter jobs using the wrapper that OSU group wrote. 

   Pegasus now supports the condor parallel universe.
   To launch a job using DCLauncher, the following pegasus profile
   keys need to be associated with the job 
   gridstart          to DCLauncher
   gridstart.path     the path to the DCLauncher script

13) New Pegasus Profiles Keys
    a) create.dir - this profile key triggers kicstart to create and
       change directories before launching a job. 

    b) gridstart.path - this profile key specifies the path to the
       gridstart used to launch a particular job

    c) runtime - this profile key is useful when using Heft based site
       selection. It allows users to associate expected runtimes of
       jobs with the job description in DAX. 

14) Kickstart captures machine information 
    Kickstart now logs machine information in the invocation record
    that it creates for each job invocation. The Kickstart JAVA parser
    can parse both records in old and new format. 

    A snippet of machine information captured is show below
    <machine page-size="4096" provider="LINUX">
     <stamp>2008-09-23T13:58:05.211-07:00</stamp>
     <uname system="linux" nodename="viz-login" release="2.6.11.7"
       machine="i686">#2 SMP Thu Apr 28 18:41:14 PDT  2005</uname> 
     <ram total="2125209600" free="591347712" shared="0" buffer="419291136"/> 
     <swap total="2006884352" free="2006876160"/>
     <boot idle="943207.170">2008-09-12T12:03:49.772-07:00</boot>
     <cpu count="4" speed="2400" vendor="GenuineIntel">Intel(R)
       Xeon(TM) CPU 2.40GHz</cpu> 
     <load min1="0.07" min5="0.04" min15="0.00"/>
     <proc total="110" running="1" sleeping="109" vmsize="614912000"
      rss="206729216"/> 
     <task total="133" running="1" sleeping="132"/>
  </machine>

15) Kickstart works in cygwin environment
    Kickstart now compiles on cygwin. Kickstart could not find
    SYS_NMLN variable in Cygwin to determine the uname datastructure's
    size. Added a fix in the Makefile to add CFLAGS -DSYS_NMLN=20 when
    the OS is Cygwin/Windows 

    The kickstart records generated on cygwin are slightly different
    from the ones generated unix platforms. The kickstart parser was
    modified to handle that.
    
    The differences are as follows - 
    a) On cygwin inode value is double. The inode value is parsed as
    double , but cast to long to prevent errors. 

    b) On cygwin the uid and gid values are long. They are passed as
    long, but cast to int to prevent errors. 

16) Changes to dirmanager
    The dirmanager executable can now remove and create multiple
    directories. This is achieved by specifying a whitespace separated
    list of directories to the --dir option. 


17) Added color-file option to showjob

    There is now a  --color-file option to show-job in
    $PEGASUS_HOME/contrib/showlog to pass a file that has the mappings
    from transformation name to colors. 

    The format of each line is as follows
    transformation-name color

    This can be used to assign different colors to compute jobs in a
    workflow. The default color assigned is gray if none is
    specified. 

18) jobstate-summary tool

    There is a new tool at $PEGASUS_HOME/bin/jobstate-summary.

    It attempts to give a summary for the workflow. Should help in
    jobstate-summ debugging failed job information. It will shows all
    the information associated with a failed job. It gets the list of
    failed job from the jobstate.log file. After that it parses latest
    kickstart file for each failed job and show the exit code and all
    the other information. 

    Usage: jobstate-summary --i <input directory> [--v(erbose)]
           [--V(ersion)] [--h(elp)] 

    Input directory is the place where all the log files including jobstate.log file reside.
    v option is for verbose debugging.
    V option gives the pegasus version.
    h option prints the help message.
    A sample run is like jobstate-summary -i /dags/pegasus/diamond/run0013 -v


19) Support for DAGMan node categories


    Pegasus now supports DAGMan node categories. DAGMan now allows to
    specify CATEGORIES for jobs, and then specify tuning parameters (
    like maxjobs ) per category. This functionality is exposed in
    Pegasus as follows 

    The user can associate a dagman profile key category with the
    jobs. The key attribute for the profile is category and value is
    the category to which the job belongs to. For example you can set
    the dagman category in the DAX for a job as follows 

    <job id="ID000001" namespace="vahi" name="preprocess" version="1.0" level="3" dv-namespace="vahi" dv-name="top" dv-version="1.0">
       <profile namespace="dagman" key="CATEGORY">short-running</profile>
       <argument>-a top -T 6  -i <filename file="david.f.a"/>  -o
         <filename file="vahi.f.b1"/> 
         <filename file="vahi.f.b2"/>
       </argument>
       <uses file="david.f.a" link="input" register="false"
       transfer="true" type="data"/> 
       <uses file="vahi.f.b1" link="output" register="true"
       transfer="true" /> 
       <uses file="vahi.f.b2" link="output" register="true"
       transfer="true" /> 
    </job>

    The property pegasus.dagman.[category].maxjobs can be used to
    control the value. 

    For the above example, the user can set the property as follows 

    pegasus.dagman.short-running.maxjobs 2

    In the DAG file generated you will see the category associated
    with jobs. For the above example, it will look as follows 

    MAXJOBS short-running 2
    CATEGORY preprocess_ID000001 short-running
    JOB preprocess_ID000001 preprocess_ID000001.sub
    RETRY preprocess_ID000001 2

20) Handling of pass through LFN

    If a job in a DAX, specifies the same LFN as an input and an
    output, it is a pass through LFN. Internally, the LFN is tagged
    only as an input for the job. The reason for this, being that we
    need to make sure that the replica catalog is queried for the
    location of the LFN. If this is not handled specially, then LFN is
    tagged internally as inout ( meaning it is generated during
    workflow execution ). LFN's with type inout are not queried for in
    the Replica Catalog in the force mode of operation 


21) Tripping seqexec on first job failures

    By default seqexec does not stop execution even if one of the
    clustered jobs it is executing fails. This is because seqexec
    tries to get as much work done as possible. If for some reason,
    you want to make seqexec stop on first job failure, set the
    following property in the properties file 

    pegasus.clusterer.job.aggregator.seqexec.firstjobfail true


22) New properties to choose the cleanup implementation

    Two new properties were added to select the strategy and
    implementation for file cleanup.

    pegasus.file.cleanup.strategy
    pegasus.file.cleanup.implementation

    Currently there is only one cleanup strategy ( InPlace ) that can
    be used and is loaded by default. 

    The cleanup implementations that can be used are 
    	- Cleanup ( default)
	- RM
	- S3 

    Detailed documentation can be found at
    $PEGASUS_HOME/etc/sample.properties. 

23) New properties to choose the create dir implementation

    The property pegasus.dir.create was deprecated.

    It has been replaced by
    pegasus.dir.create.strategy   

    Additionally, a user can specify a property to choose the
    implementation used to create the directory on the remote sites. 

    pegasus.dir.create.impl   

    The create directory implementation that can be used are 
    - DefaultImplementation   uses $PEGASUS_HOME/bin/dirmanager
                              executable to create a directory on the
    			      remote site.  
    - S3   		      usese s3cmd to create a bucket on amazon S3.


BUGS FIXED
----------
1) Makefile for kickstart to build on Cygwin

   Kickstart could not find SYS_NMLN variable in Cygwin to determine
   the uname datastructure's size. Added a fix in the Makefile to add
   CFLAGS -DSYS_NMLN=20 when the OS is Cygwin/Windows 


2) Bug fix to getsystem release tools

   Some systems have started using / in their system version name
   which causes failures in Pegasus build process. Fixed the getsystem
   release script which converts / into _ 

3) Bug fix in file cleanup module when stageout is enabled.
   
   There was a bug in how the dependencies are added between the
   stageout jobs and the file cleanup jobs. In certain cases, cleanup
   could occur before the output was staged out. This is fixed now.

   This bug was tracked through bugzilla
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=37

4) Bug fix to deferred planning

   Deferred planning used to fail if pegasus-plan was not given -o
   option .

   This is fixed now and was tracked through bugzilla
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=34

5) Bug fix to caching on entries from Transformation Catalog

   In certain cases, caching of entries did not work for the INSTALLED
   case. 

   This is fixed now and was tracked through bugzilla
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=33

   

===============================
Release Notes for PEGASUS 2.1.0
===============================
 
NEW FEATURES
--------------

1) Support for Second Level Staging

   Normally, Pegasus transfers the data to and from a directory on the
   shared filesystem on the head node of a compute site. The directory
   needs to be visible to both the head node and the worker nodes for
   the compute jobs to execute correctly.

   In the case, where the worker nodes cannot see the filesystem of
   the head node there needs to be a Second Level Staging (SLS)
   process that transfers the data from the head node to a directory
   on the worker node tmp. To achieve this, Pegasus uses the pre-job
   and post-job feature of kickstart to pull the input data from the
   head  node and push back the output data of a job to the head
   node. 

   Even though we do SLS, Pegasus still relies on the existence of a
   shared file system due to the following two reasons

   a) for the transfer executable to pick up the proxy, that we
   transfer from the submit host to the head node.

   b) to access sls input and output files that contain the file
   transfer urls to manage the transfer of data to worker node and
   back to headnode. 


   Additionally, if you are running your workflows on a Condor pool,
   one can bypass the use of kickstart to do the SLS. Please contact
   pegasus@isi.edu for more details of this scenario. In this case,
   the workflows generated by Pegasus have been shown to run in total
   non shared filesystem environment.

   To use this feature, user needs to set 

   pegasus.execute.*.filesystem.local  true

   The above change was tracked via bugzilla
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=21


2) New DAX schema
   The new has release moved to the new DAX schema version 2.1. Schema
   is  available online http://pegasus.isi.edu/schema/dax-2.1.xsd

   The main change in it is that the dontTransfer and dontRegister
   flags have been replaced by transfer and register flags. Changes
   were made both to the Java DAX Generator and Pegasus to conform to
   the new schema.  

   Additionally, the DAX parser in Pegasus looks at the schema version to
   determine whether to pick up dontTransfer and dontRegister flags (
   to support backward compatibility with the older daxes).

   Also with the filename type added a type attribute. It defaults to
   data. Additionally user can have the values
   executable|pattern. Users can use type=executable to specify any 
   dependant executables that their jobs required. All executable
   files are tracked in the transformation catalog.

   The above change was tracked via bugzilla
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=6 


3) Workflow and Planner Metrics Logging
   Workflow and Planning metrics are now logged for each workflow that
   is planned by Pegasus. By default, they are logged to
   $PEGASUS_HOME/var/pegasus.log  

   To turn metrics logging off, set pegasus.log.metrics to false

   To change the file to which the metrics are logged set
   pegasus.log.metrics.file  path/to/log/file

   Here is a snippet from the log file that shows what is logged
   {
   user = vahi
   vogroup = pegasus-ligo
   submitdir.base = /nfs/asd2/vahi/jbproject/Pegasus/dags
   submitdir.relative = /vahi/pegasus-ligo/blackdiamond/run0064
   planning.start = 2007-09-24T18:14:23-07:00
   planning.end = 2007-09-24T18:14:29-07:00
   properties
                 =/nfs/asd2/vahi/jbproject/Pegasus/dags/vahi/pegasus-ligo/blackdiamond/run0064/pegasus.6766.properties
   dax = /nfs/asd2/vahi/jbproject/Pegasus/blackdiamond_dax.xml
   dax-label = blackdiamond
   compute-jobs.count = 3
   si-jobs.count = 1
   so-jobs.count = 3
   inter-jobs.count = 0
   reg-jobs.count = 3
   cleanup-jobs.count = 2
   total-jobs.count = 14
   }


4) Support for querying multiple replica catalogs

   Pegasus now allows the users to query multiple replica catalogs at
   the same time to discover the locations of input data sets.

   For this a new Replica Catalog implmentation was developed. 
  
   The users need to do the following to use it.

   Set the replica catalog to MRC in the properties file.

   pegasus.catalog.replica MRC
 
   Each associated replica catalog can be configured via properties as	
   follows. The user associates a variable name referred to as [value]
   for each of the catalogs, where [value] is any legal identifier
   (concretely [A-Za-z][_A-Za-z0-9]*) 

   For each associated replica catalogs the user needs to specify the following properties.

   pegasus.catalog.replica.mrc.[value]      to specify the type of  replica catalog
   pegasus.catalog.replica.mrc.[value].key  to specify a property name key for a
                                            particular catalog
 

   For example, if a user wants to query two lrc's at the same time he/she can specify as follows

    pegasus.catalog.replica.mrc.lrc1 LRC
    pegasus.catalog.replica.mrc.lrc2.url rls://sukhna

    pegasus.catalog.replica.mrc.lrc2 LRC
    pegasus.catalog.replica.mrc.lrc2.url rls://smarty

   In the above example, lrc1, lrc2 are any valid identifier names and
   url is the property key that needed to be specified.  


5) Local Replica Selector
 
   Pegasus has a new local replica selector that only prefers replicas
   from the local host and that start with a file: URL scheme.  It is
   useful, when users want to stagin files to a remote site from your
   submit host using the Condor file transfer mechanism. 

   In order to use this, set the replica selector to Local in the
   properties.

        - pegasus.selector.replica  Local


6) Heft Based Site Selector

   Added a new site selector that is based on the HEFT processor
   scheduling algorithm. 

   The implementation assumes default data communication costs when jobs
   are  not scheduled on to the same site. Later on this may be made more
   configurable. 

   The runtime for the jobs is specified in the transformation catalog
   by associating the pegasus profile key runtime with the entries. 

   The number of processors in a site is picked up from the attribute
   idle-nodes associated with the vanilla jobmanager of the site in
   the site catalog.  


   To use this site selector, users need to set the following property 

   pegasus.selector.site Heft
 

7) Using multiple grid ftp servers for stageout
   If a user specifies multiple grid ftp servers for the output site
   in the site catalog, the stageout jobs will be distributed over all
   of them.

   More info can be found at
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=3 


8) Scalable Directory structure on the stageout site
   Users can now distribute their output files in a directory
   structure on the output site. On setting the Boolean property
   pegasus.dir.storage.deep to true, the relative submit directory
   structure is replicated on the output site. Additionally, within
   this directory the files are distributed into sub directories with
   each subdirectory having 256 files.

   The subdirectories are named in decimal format.
   

9) Specifying the jobmanager universe for the compute jobs in the DAX 
   Users can know specify the jobmanager type for the compute jobs in the
   DAX. This is achieved by specifying the jobmanager.universe profile
   key in the hints namespace. 

   Valid values for this are transfer|vanilla.

   This is useful for users who are running on a grid site, with the
   worker nodes behind a firewall and want a subset of their jobs to
   run on the head node.

   More info can be found at
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=5
 

10) Stork Support for doing transfers
   
   The internal transfer interfaces of Pegasus were updated to use the
   latest version of Stork for managing data transfers. 
   
   To use Stork implementations set the following
   pegasus.transfer.refiner = SDefault
   pegasus.transfer.*.impl=Stork

11) nogrid option for pegasus-run
  
 pegasus-run has now a --nogrid option. This bypasses the checks for
   proxy existence that are done before submitting the workflow for
   execution. It disables all globus checks like check for environment
   variables GLOBUS_lOCATION and LD_LIBRARY_PATH.

   This is useful for running workflows in native Condor environments. 

12) Submitting workflows directly using pegauss-plan

   A new option --submit|-S option was added to pegasus-plan. This
   allows users to submit workflows directly, after they have been
   planned. 

13) Specifying relative submit directory
   Since pegasus 2.0 , pegasus-plan creates a directory structure in
   the base submit directory. The base submit directory is specified
   by --dir option to pegasus-plan. If a user, want to specify a
   relative submit directory, he can use the --relative-dir option to
   pegasus-plan. 

   The above change was tracked via bugzilla
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=14 


BUGS FIXED
----------
1) Specifying Relative Path to the DAX

   An incorrect path to the dax was generated internally when a user
   specified a relative path to the dax to pegasus-plan 

   This is fixed now, and was tracked via bugzilla
   http://vtcpc.isi.edu/bugzilla/show_bug.cgi?id=13


2) RLS java api bug fix 4114 (globus bugzilla number )

   globus_rls_client.jar updated with the bug fix 4114. Also added the
   jar for java 1.4 in lib/java1.4  

3) Passing of DAGMan parameters via properties in case of deferred
   planning 

   In case of deferred planning, the properties that control DAGMan
   execution were not being passed as options to DAGMan. This is now
   fixed. 

   The following properties are being handled correctly now.

   pegasus.dagman.maxjobs
   pegasus.dagman.maxpre
   pegasus.dagman.maxidle
   pegasus.dagman.maxpost


===============================
Release Notes for PEGASUS 2.0.1
================================

There is new documentation in the form of a quick start guide and glossary
in the docs directory. More documentation will be coming soon and will be available
in the release as well as on the pegasus website under documentation.

NEW FEATURES
------------
1) Pegasus now can store provenance data into PASOA. The actions taken
   by the various refiners are logged into the store. It is still an 
   experimental feature. To turn it on, set the property 

   pegasus.catalog.provenance.refinement  pasoa

   The PASOA store needs to run on localhost on port 8080
   https://localhost:8080/preserv-1.0

2) You can also use Pegasus to store execution provenance in PASOA.
   To use set the properties
   pegasus.exitcode.impl=pasoa
   pegasus.exitcode.path.pasoa=${pegasus.home}/bin/pasoa-client
   pegasus.exitcode.arguments=<dax file> <dag file>

BUGS FIXED
----------
1) sitecatalog-converter 
   patch to fix pegasus profile conversion

2) pegasus-submit-dag
   added --maxidle option to allow setting number of idle jobs on
   the remote site. 

3) VORS.pm 
   Fixed a small typo in there, that lead to perl compilation
   errors. 

4) pegasus-get-sites
   Removed local tc entries and added environments 
   for PEGASUS_HOME, GLOBUS_LOCATION and LD_LIBRARY_PATH to local
   site.

5) mpiexec 
   The execution of clustered jobs via mpiexec was broken in 2.0
   release. That is now fixed.

6) exitcode/exitpost
   Fixed a bug in exitcode that caused a call to the DB PTC even though
   the property pegasus.catalog.provenance was not set.

KNOWN BUGS
-----------

===============================
Release Notes for PEGASUS 2.0.0
===============================

NEW FEATURES
--------------

pegasus-plan.
	This is the main client for invoking pegasus. The earlier gencdag command is now called pegasus-plan

pegasus-run
	This is the client that submits the planned workflow to Condor and starts a monitoring tailstatd daemon

pegasus-status
	This client lets you monitor a particular workflow. Its a wrapper around condor-q

pegasus-remove.
	This client lets you remove a running workflow from the condor queue.
        A rescue dag will be generated which can be submitted by just running pegasus-run on the dag directory.