First of all, I would like to express my gratitude for making such a complete SFT dataset available for open source. For the domain scenario, this is very meaningful in terms of data ratio. My question is: If I only want to select a subset with the same composition in a geometric ratio as my general dataset, is there any way to do that currently? Because all the dataset files are placed in the "json" folder, it seems impossible to selectively extract them by category, task type, etc.
首先感谢能够开源如此完整的SFT数据集，对于领域场景，这个对于数据配比十分有意义。我想问的是，如果只想按照相同的组成等比选取一个子集作为我的通用数据集，当前有什么方式嘛？因为看数据集文件全部放在了json文件夹下，似乎实现不了按照类别、任务类型等比节选
First of all, I would like to express my gratitude for making such a complete SFT dataset available for open source. For the domain scenario, this is very meaningful in terms of data ratio. My question is: If I only want to select a subset with the same composition in a geometric ratio as my general dataset, is there any way to do that currently? Because all the dataset files are placed in the "json" folder, it seems impossible to selectively extract them by category, task type, etc.
首先感谢能够开源如此完整的SFT数据集，对于领域场景，这个对于数据配比十分有意义。我想问的是，如果只想按照相同的组成等比选取一个子集作为我的通用数据集，当前有什么方式嘛？因为看数据集文件全部放在了json文件夹下，似乎实现不了按照类别、任务类型等比节选