Skip to content

Support conditional file processing based on previous task output #103

Description

@Nina-mvH

This would be relevant in situations like the following:

  • I have a directory of input files (lets say text inputs)
  • I want to run a task which will include a T/F outputs (ie. a chat task outputting JSON with a T/F field0
  • Given the JSON outputs from the previous task, I want to run a second task which takes the original text files as input, but only if they were determined True from the previous outputs. In other words, I want this task to run on a subset of the original input files with the previous task determining the subset.

It would be nice to be able to specify this somehow in the config

task1
     input: .txt
     output: .json
     ...
task2:
     # No depends_on field
     input: .txt #points to same directory as task1
     subset: 
          - depends_on: task1
          - condition: [field] is True

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    discussionAn open ended issue that requires more conversationfeatureImplement new functionalities

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions