transfermanager.UploadDirectory does not close files that it opens

[mogzol]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

I have noticed that the S3 transfermanager is not closing files that it opens when calling UploadDirectory. This has caused problems in my app which calls UploadDirectory and then immediately tries to remove the uploaded directory afterwards with os.RemoveAll. Because there are still open file handles, this results in a "directory not empty" error (note this will only happen for certain filesystems, like in an NFS volume where there will be a bunch of .nfs... files for the open file handles).

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

transfermanager.UploadDirectory should close all file handles it opens before it returns.

Current Behavior

transfermanager.UploadDirectory leaves files open after returning.

Reproduction Steps

Here is a simple app that will reproduce the issue. Create a directory with some files to upload, then run the app while specifying the dir, bucket, and keyPrefix flags, and it will print out the open file handles at the end of its run:

Reproduction code

package main

import (
	"context"
	"flag"
	"fmt"
	"os"
	"sync"

	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
	dir := flag.String("dir", "", "directory to upload")
	bucket := flag.String("bucket", "", "bucket to upload to")
	keyPrefix := flag.String("keyPrefix", "", "keyPrefix to upload to")
	flag.Parse()

	if err := run(dir, bucket, keyPrefix); err != nil {
		fmt.Fprintf(os.Stderr, "ERROR: %v\n", err)
		os.Exit(1)
	}
}

func run(dir, bucket, keyPrefix *string) error {
	ctx := context.Background()
	captured := &fileCapture{}

	awsCfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		return fmt.Errorf("config.LoadDefaultConfig: %w", err)
	}

	s3client := s3.NewFromConfig(awsCfg)
	client := transfermanager.New(s3client)

	output, err := client.UploadDirectory(ctx, &transfermanager.UploadDirectoryInput{
		Source:    dir,
		Bucket:    bucket,
		KeyPrefix: keyPrefix,
		Callback:  captured,
	})
	if err != nil {
		return fmt.Errorf("UploadDirectory: %w", err)
	}

	openFiles := captured.openFiles()

	fmt.Printf("objects uploaded: %d\n", output.ObjectsUploaded)
	fmt.Printf("file handles captured: %d\n", captured.len())
	fmt.Printf("file handles still open after UploadDirectory returned: %d\n", len(openFiles))
	for _, file := range openFiles {
		fmt.Printf("  OPEN: %s (fd=%d)\n", file.Name(), file.Fd())
	}

	return nil
}

// fileCapture receives the original *os.File bodies opened by UploadDirectory.
type fileCapture struct {
	mu    sync.Mutex
	files []*os.File
}

func (capture *fileCapture) UpdateRequest(input *transfermanager.UploadObjectInput) {
	file, ok := input.Body.(*os.File)
	if !ok {
		return
	}

	capture.mu.Lock()
	defer capture.mu.Unlock()
	capture.files = append(capture.files, file)
}

func (capture *fileCapture) len() int {
	capture.mu.Lock()
	defer capture.mu.Unlock()
	return len(capture.files)
}

func (capture *fileCapture) openFiles() []*os.File {
	capture.mu.Lock()
	defer capture.mu.Unlock()

	var open []*os.File
	for _, file := range capture.files {
		if _, err := file.Stat(); err == nil { // Stat should fail if the file is closed
			open = append(open, file)
		}
	}
	return open
}

Example run:

> ls assets
bar.txt  baz.txt  foo.txt
> go build -o repro
> ./repro -dir assets -bucket my.bucket -keyPrefix example
objects uploaded: 3
file handles captured: 3
file handles still open after UploadDirectory returned: 3
  OPEN: assets/bar.txt (fd=5)
  OPEN: assets/foo.txt (fd=7)
  OPEN: assets/baz.txt (fd=6)

Possible Solution

Close any opened file handles before UploadDirectory returns

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

module uploaddirectoryrepro

go 1.24

require (
	github.com/aws/aws-sdk-go-v2/config v1.32.36
	github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager v0.3.12
	github.com/aws/aws-sdk-go-v2/service/s3 v1.107.1
)

require (
	github.com/aws/aws-sdk-go-v2 v1.43.5 // indirect
	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.17 // indirect
	github.com/aws/aws-sdk-go-v2/credentials v1.19.35 // indirect
	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.36 // indirect
	github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.36 // indirect
	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.36 // indirect
	github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.37 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.16 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.29 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.36 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.37 // indirect
	github.com/aws/aws-sdk-go-v2/service/signin v1.5.5 // indirect
	github.com/aws/aws-sdk-go-v2/service/sso v1.33.5 // indirect
	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.5 // indirect
	github.com/aws/aws-sdk-go-v2/service/sts v1.45.5 // indirect
	github.com/aws/smithy-go v1.27.7 // indirect
)

Compiler and Version used

go version go1.26.4 darwin/arm64

Operating System and version

macOS 26.6.1

[wty-Bryant]

that's indeed a impl miss, will fix soon