Add support for deleting project files and access points from S3

[Chrystinne]

Part of #2348

This PR adds support for deleting project files from AWS S3 through the console. It includes deletion of the zip file and, for private projects, the associated access points.

A "Delete files" button has been added to the AWS cloud management section of the manage published project page, visible only when sent_files is True.

For public projects, the function removes all files and the zip file from S3 and deletes the AWS record from the database. For private projects, it also deletes all associated access points from S3 and removes the AWS, AWSAccessPoint, and AWSAccessPointUser records via cascade.

Note: Despite the branch name, this PR covers both public and private projects.

Note: the "Delete project files" button has been renamed to "Delete files" in #2628 for consistency with the other buttons.

[tompollard]

Looks good to me, thanks!

[bemoody]

What does this do?

bucket.objects.filter(Prefix=prefix).delete()

Ultimately, we may have millions of individual files/objects to delete. Does this line perform a single API request to S3 that deletes every file at once? Or as I suspect, does this require making millions of requests to delete the files one at a time?

Even if you can delete multiple files with one request, it might have limits (maximum of 1000 files?) or it might take a very long time to complete. So I strongly suspect that this needs to be implemented as a background task.

Order of operations

Access points should be deleted from S3 before starting to delete files. From a security standpoint: access must be disabled immediately, not hours later. From a data integrity standpoint: people should not see that their aws s3 sync command succeeded if they received an incomplete copy.

Before starting to delete files, the AWS object should also have sent_files set to false.

I think we probably don't want to delete the AWS object until after the files are deleted, but I'm not sure about that.

[Chrystinne]

What does this do?

bucket.objects.filter(Prefix=prefix).delete()

Ultimately, we may have millions of individual files/objects to delete. Does this line perform a single API request to S3 that deletes every file at once? Or as I suspect, does this require making millions of requests to delete the files one at a time?

Even if you can delete multiple files with one request, it might have limits (maximum of 1000 files?) or it might take a very long time to complete. So I strongly suspect that this needs to be implemented as a background task.

Order of operations

Access points should be deleted from S3 before starting to delete files. From a security standpoint: access must be disabled immediately, not hours later. From a data integrity standpoint: people should not see that their aws s3 sync command succeeded if they received an incomplete copy.

Before starting to delete files, the AWS object should also have sent_files set to false.

I think we probably don't want to delete the AWS object until after the files are deleted, but I'm not sure about that.

Thanks for your comments, @bemoody. These are all good points.

Most importantly here: this method does not delete the files/objects one at a time. The operation deletes 1,000 objects in a single batch (and uses only two API requests per batch, one for listing key objects and another for deleting them).
(boto3 documentation: https://docs.aws.amazon.com/boto3/latest/reference/services/s3/bucket/objects.html)

So when we call bucket.objects.filter(Prefix=prefix).delete(), what boto3 does is the following (pagination):

• List the first 1,000 objects under the prefix (1 listing request);
• Delete those 1,000 objects in a single batch (1 deletion request);
• List the next 1,000 objects;
• Delete those 1,000 objects;
• And so on until all objects/files are deleted.

So, for example, for a project with 50,000 files, this would result in 100 requests total.

Although this can be unpredictable, this operation should be executed very rarely when authors spot some problems in the dataset. It took us quite some time of managing projects on AWS before we first needed to use such a function.

I am unsure about the amount of time it would take for very large datasets with millions of files as I have only tested the function with small projects, but I agree that we might need to implement this as a background task considering cases where we have millions of files.

Regarding the order of operations, that makes sense also when thinking about large datasets. I will update the code.

[Chrystinne]

@bemoody I've made the updates. Could you please have another look?

[bemoody]

Some small things:

messages.success(request, 'The project files have been deleted from S3.')

Please change this to say "The project files are being deleted from S3".

def delete_project_files_from_s3(project):
    # Import here to avoid circular import
    from console.views import delete_project_files_task
    """
    Immediately revoke access by deleting access points,
    set project.aws.sent_files = False, then schedule
    file deletion as a background task.
    """

Please put the doc string before the "import" line. Otherwise it isn't recognized as a doc string.

[Chrystinne]

Chrystinne

Good catch, @bemoody! thanks!