As you may be aware, Go's HTTP client may retain the request Body reference after Do returns. The correct signal that Body can be reused is communicated via the Close method; that's why Go's http.Request.Body happens to be ReadCloser to start with.
When doing high throughput data processing it is imperative to reuse the large buffers used for data transfers. However, it seems, that with a present implementation, Body reference may still be in use by Go's HTTP client even after AWS SDK method returns. This makes it impossible to treat the return of the SDK method as buffer reuse boundary.
By itself it would not be an insurmountable problem - we could implement some nice Close methods on our buffer objects. However, these won't be called because smithy's Request object wraps away whatever custom Close possibly supplied by an user.
Moreover, whomever implemented the default Close method for smithy's HTTP Request objects was clearly aware that it may be called asynchronously from another goroutine and added a mutex in there. Unfortunately, the mutex on its own is not sufficient - in the absence of any API changes, smithy must guarantee, that AWS SDK methods cannot return before last Close was not called on the last rewound Body copy.
To summarize, smithy must ensure either that:
- User supplied Body reference is not accessible by any goroutine by the time enclosing SDK method returns (some sort of waitable object, channel or waitgroup inside the smithy implementation)
- Or, keep track of user supplied
Close method on the Body and call that when last internal request Close is called.
As you may be aware, Go's HTTP client may retain the request Body reference after
Doreturns. The correct signal that Body can be reused is communicated via theClosemethod; that's why Go'shttp.Request.Bodyhappens to beReadCloserto start with.When doing high throughput data processing it is imperative to reuse the large buffers used for data transfers. However, it seems, that with a present implementation, Body reference may still be in use by Go's HTTP client even after AWS SDK method returns. This makes it impossible to treat the return of the SDK method as buffer reuse boundary.
By itself it would not be an insurmountable problem - we could implement some nice
Closemethods on our buffer objects. However, these won't be called because smithy'sRequestobject wraps away whatever customClosepossibly supplied by an user.Moreover, whomever implemented the default
Closemethod for smithy's HTTPRequestobjects was clearly aware that it may be called asynchronously from another goroutine and added a mutex in there. Unfortunately, the mutex on its own is not sufficient - in the absence of any API changes, smithy must guarantee, that AWS SDK methods cannot return before lastClosewas not called on the last rewound Body copy.To summarize, smithy must ensure either that:
Closemethod on the Body and call that when last internal requestCloseis called.