feature/s3/transfermanager: UploadObject buffers seekable bodies (*os.File) instead of streaming — peak memory regression vs feature/s3/manager

[LiuXinjie06]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

feature/s3/transfermanager's UploadObject buffers every part into an in-memory pool, even when the Body is a seekable file (io.ReaderAt + io.Seeker, e.g. *os.File). For multipart uploads the pool is pre-allocated to (Concurrency+1) * PartSize per call, so peak memory grows with PartSize × Concurrency × (number of concurrent UploadObject calls), independent of whether the source is already on disk.

This is a regression versus feature/s3/manager. The v1 Uploader detects a seekable body and streams each part directly from the source via io.NewSectionReader, never buffering whole parts. Migrating an existing file-upload workload from manager.Uploader to transfermanager.UploadObject therefore makes peak RSS jump from tens of MB to hundreds of MB / GB-scale for the same files and the same PartSize/Concurrency settings.

The AWS SDK for Go v2 Developer Guide still documents the seekable-body fast path as the recommended way to keep upload memory low ("For body values that implement the ReadSeekerAt type, the Uploader doesn't buffer the body contents in memory before sending it to Amazon S3"). transfermanager does not honor that behavior.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

When Body is seekable (implements io.ReaderAt + io.Seeker) and its size is known, UploadObject should stream each part from the source (as manager.Uploader does with its readerAtSeeker path) and not allocate the (Concurrency+1) * PartSize buffer pool. Peak memory for a file upload should stay at tens of MB regardless of PartSize/Concurrency, matching v1.

Current Behavior

Parts are always read into pooled in-memory buffers:

• upload() creates newDefaultSlicePool(PartSizeBytes, Concurrency+1) for every multipart upload.
• pool.go's newDefaultSlicePool allocates all buffers eagerly (for range capacity { make([]byte, sliceSize) }), so (Concurrency+1) * PartSize is resident before any byte is read.
• nextReader() then fills each part via partPool.Get() + readFillBuf(u.in.Body, part).
• The only place the body's seekability is used is initSize(), and it is used only to compute the object size (types.SeekerLen), never to stream.

Measured on real S3 (PartSize=16MiB, Concurrency=24, file-level concurrency 4, comparable medium-file batches — memory is the stable, reproducible signal):

Path	Peak RSS (VmHWM)
transfermanager.UploadObject (*os.File body)	~2.0–2.9 GB
same-shape batch via manager.Uploader (*os.File)	~46 MB

A single UploadObject of one large *os.File already holds (24+1) × 16MiB ≈ 400 MB; it scales linearly with the number of concurrent uploads. (A controlled same-harness A/B of the buffered path vs. a streaming prototype is in Possible Solution below.)

Reproduction Steps

Minimal program (uploads a 1 GiB temp file and prints peak RSS). Set BUCKET, and REGION via env:

package main

import (
	"context"
	"fmt"
	"os"
	"strings"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

func peakRSS() string {
	b, _ := os.ReadFile("/proc/self/status")
	for _, line := range strings.Split(string(b), "\n") {
		if strings.HasPrefix(line, "VmHWM:") {
			return strings.TrimSpace(strings.TrimPrefix(line, "VmHWM:"))
		}
	}
	return "?"
}

func main() {
	// 1 GiB file on disk (seekable => *os.File implements io.ReaderAt + io.Seeker)
	f, _ := os.CreateTemp("", "tm-repro-*")
	defer os.Remove(f.Name())
	if err := f.Truncate(1 << 30); err != nil {
		panic(err)
	}
	f.Close()

	cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion(os.Getenv("REGION")))
	if err != nil {
		panic(err)
	}
	c := transfermanager.New(s3.NewFromConfig(cfg), func(o *transfermanager.Options) {
		o.PartSizeBytes = 16 << 20 // 16 MiB
		o.Concurrency = 24
	})

	body, _ := os.Open(f.Name())
	defer body.Close()
	_, err = c.UploadObject(context.TODO(), &transfermanager.UploadObjectInput{
		Bucket: aws.String(os.Getenv("BUCKET")),
		Key:    aws.String("tm-repro-1g"),
		Body:   body,
	})
	if err != nil {
		panic(err)
	}
	fmt.Println("peak RSS (VmHWM):", peakRSS()) // ~400 MB; with manager.Uploader it is ~tens of MB
}

Run the same upload through feature/s3/manager's Uploader (same PartSize/Concurrency, same *os.File body) to see peak RSS stay at tens of MB.

Possible Solution

Restore the v1 manager.Uploader behavior: when Body implements io.ReaderAt + io.Seeker (and size is known via initSize), serve each part as io.NewSectionReader(body, offset, partLen) instead of copying into a pooled buffer, and skip creating partPool entirely on that path. io.SectionReader is independently readable/seekable, so it works with the concurrent multipart workers and payload signing without buffering.

We prototyped exactly this against v0.2.11 (add a readerAtSeeker branch in nextReader, guard the newDefaultSlicePool allocation in upload() behind !streamable). Result on real S3: peak RSS dropped from ~2.0–2.9 GB to ~35 MB at PartSize=16MiB/Concurrency=24/file-level concurrency 4, throughput unchanged, and uploads verified byte-for-byte (sha256) for both the single-PUT path (object < MultipartUploadThreshold) and the multipart path. Happy to open a PR if the approach looks right.

(The non-seekable io.Reader case still legitimately requires buffering — this only asks to restore the seekable/file fast path that v1 had.)

Additional Information/Context

Relevant source in feature/s3/transfermanager@v0.2.11:

• api_op_UploadObject.go — upload() creates the pool; nextReader()/readFillBuf buffer each part; initSize() uses io.Seeker only for sizing.
• pool.go — newDefaultSlicePool eagerly allocates capacity × sliceSize.

Compare feature/s3/manager@v1.22.x/upload.go — nextReader() case readerAtSeeker: uses io.NewSectionReader (no per-part buffer).

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2 v1.42.0
github.com/aws/aws-sdk-go-v2/config v1.32.25
github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager v0.2.11
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.28
github.com/aws/aws-sdk-go-v2/service/s3 v1.104.0

Compiler and Version used

go1.26.4 linux/amd64

Operating System and version

Linux, x86_64 (Ubuntu)

[goutamadwant]

@wty-Bryant I opened #3468 with a fix for this.

PR adds regression coverage for seekable multipart uploads and updates UploadObject to use io.NewSectionReader for bodies that implement io.ReaderAt and io.Seeker, while keeping the existing buffered path for non-seekable readers. Let me know if there are any suggestions. thanks!