Skip to content

Refactor s3fs (and the aotbotocore) out as a dependency #43

Description

@stijnvanhoey

s3fs is a convenient package to interact with s3 (feels like coding against normal file system), but since s3f3 relies on the async aiobotocore package some issues arised:

  • due to very strict dependencies in both boto3 and aiobotocore, the pip solver was not able to solve this unless defining boto3 as an additional install from s3fs (see daeb285 for fix)
  • aiobotocore requires raw_headers which are not according to specs in the moto-library used in the testing (see aiobotocore AttributeError - missing raw_headers getmoto/moto#3259 for description and 802b9f5 for the fix)
  • aiobotocore/s3fs keep in an recursive loop copy-pasting the files in the pytest-context leasing to an ever increasing file size when running the tests. Has been solved by relying on boto3 instead for the download, see 28a5e42

Hence, these issues have been handled in https://github.com/enram/vptstools/tree/SVH-country-filter, but I'm not sure if the convenience of s3fs is worthwhile having it as a dependency. It might be that excluding s3fs would make things easier to maintain.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions