diff --git a/AGENTS.md b/AGENTS.md index 3c39118..7ae53c8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,11 +34,11 @@ GoBatch is a Go library for batch data processing. It provides infrastructure fo - Include practical examples in documentation ## File Structure -- `/batch`: Core batch processing functionality, includes the main Batch type, configuration, errors, and helper functions -- `/processor`: Different processors for data manipulation (Transform, Filter, Channel, Error, Nil) -- `/source`: Data sources for batch processing (Channel, Error, Nil) -- `/doc.go`: Package-level documentation -- `/example_test.go`: Top-level usage examples +- `/batch`: Core batch processing package. Includes `batch.go` (the main `Batch` type and pipeline), `config.go` (Config interface, ConstantConfig, DynamicConfig, BufferConfig), `errors.go` (SourceError, ProcessorError), `constants.go`, `helpers.go` (IgnoreErrors, CollectErrors, RunBatchAndWait, ExecuteBatches), and `doc.go`. Accompanied by unit tests (`*_test.go`) and several runnable `example_*_test.go` files. +- `/processor`: Processor implementations — `transform.go`, `filter.go`, `channel.go`, `error.go`, `nil.go` — plus `doc.go` and tests. +- `/source`: Source implementations — `channel.go`, `error.go`, `nil.go` — plus `doc.go` and tests. +- `/doc.go`: Root package-level documentation. +- `/example_test.go`: Top-level usage examples. ## Key Concepts - **Batch**: Main type that orchestrates the batch processing pipeline @@ -57,7 +57,7 @@ GoBatch is a Go library for batch data processing. It provides infrastructure fo - **Filter**: Filters items based on a predicate function - **Channel**: Writes item data to an output channel - **Error**: Simulates processor errors (for testing) -- **Nil**: Passes items through unchanged (for benchmarking) +- **Nil**: Sleeps for a configurable `Duration` without modifying item data (can mark items cancelled via `MarkCancelled`); useful for simulating slow processing and testing timing behavior ### Sources - **Channel**: Uses Go channels as data sources @@ -75,4 +75,4 @@ GoBatch is a Go library for batch data processing. It provides infrastructure fo - Processors should respect context cancellation - Items with errors are tracked individually through the Error field - Batch processing continues despite individual item errors -- Use `errors.As` to check error types (SourceError, ProcessorError) +- Use `errors.As` to check error types (SourceError, ProcessorError). The engine wraps errors as pointers (`&SourceError{...}`, `&ProcessorError{...}`), so `errors.As` must target the pointer type — e.g. `var se *batch.SourceError; errors.As(err, &se)` (equivalently `errors.As(err, new(*batch.SourceError))`). A value-target such as `var se batch.SourceError; errors.As(err, &se)` will not match. diff --git a/CLAUDE.md b/CLAUDE.md index ddc25b4..b8fc557 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -34,11 +34,11 @@ GoBatch is a Go library for batch data processing. It provides infrastructure fo - Include practical examples in documentation ## File Structure -- `/batch`: Core batch processing functionality, includes the main Batch type, configuration, errors, and helper functions -- `/processor`: Different processors for data manipulation (Transform, Filter, Channel, Error, Nil) -- `/source`: Data sources for batch processing (Channel, Error, Nil) -- `/doc.go`: Package-level documentation -- `/example_test.go`: Top-level usage examples +- `/batch`: Core batch processing package. Includes `batch.go` (the main `Batch` type and pipeline), `config.go` (Config interface, ConstantConfig, DynamicConfig, BufferConfig), `errors.go` (SourceError, ProcessorError), `constants.go`, `helpers.go` (IgnoreErrors, CollectErrors, RunBatchAndWait, ExecuteBatches), and `doc.go`. Accompanied by unit tests (`*_test.go`) and several runnable `example_*_test.go` files. +- `/processor`: Processor implementations — `transform.go`, `filter.go`, `channel.go`, `error.go`, `nil.go` — plus `doc.go` and tests. +- `/source`: Source implementations — `channel.go`, `error.go`, `nil.go` — plus `doc.go` and tests. +- `/doc.go`: Root package-level documentation. +- `/example_test.go`: Top-level usage examples. ## Key Concepts - **Batch**: Main type that orchestrates the batch processing pipeline @@ -57,7 +57,7 @@ GoBatch is a Go library for batch data processing. It provides infrastructure fo - **Filter**: Filters items based on a predicate function - **Channel**: Writes item data to an output channel - **Error**: Simulates processor errors (for testing) -- **Nil**: Passes items through unchanged (for benchmarking) +- **Nil**: Sleeps for a configurable `Duration` without modifying item data (can mark items cancelled via `MarkCancelled`); useful for simulating slow processing and testing timing behavior ### Sources - **Channel**: Uses Go channels as data sources @@ -75,4 +75,4 @@ GoBatch is a Go library for batch data processing. It provides infrastructure fo - Processors should respect context cancellation - Items with errors are tracked individually through the Error field - Batch processing continues despite individual item errors -- Use `errors.As` to check error types (SourceError, ProcessorError) \ No newline at end of file +- Use `errors.As` to check error types (SourceError, ProcessorError). The engine wraps errors as pointers (`&SourceError{...}`, `&ProcessorError{...}`), so `errors.As` must target the pointer type — e.g. `var se *batch.SourceError; errors.As(err, &se)` (equivalently `errors.As(err, new(*batch.SourceError))`). A value-target such as `var se batch.SourceError; errors.As(err, &se)` will not match. \ No newline at end of file diff --git a/README.md b/README.md index 56afdbc..34b0fed 100644 --- a/README.md +++ b/README.md @@ -100,7 +100,7 @@ go get github.com/MasterOfBinary/gobatch - **Filter**: Filters items based on a predicate function. - **Transform**: Transforms item data with a custom function. - **Error**: Simulates processor errors for testing. -- **Nil**: Passes items through unchanged for benchmarking. +- **Nil**: Sleeps for a configurable `Duration` without modifying item data; useful for simulating slow processing and testing timing behavior. - **Channel**: Writes item data to an output channel. ### Built-in Sources diff --git a/doc.go b/doc.go index 74d1ff1..e96fb2c 100644 --- a/doc.go +++ b/doc.go @@ -26,9 +26,7 @@ // batch.IgnoreErrors(errs) // <-b.Done() // -// Output: -// -// hello +// The Transform func above prints each item, so this pipeline writes "hello". // // See the README.md for an overview of how these pieces fit together. package gobatch diff --git a/source/doc.go b/source/doc.go index 66bf44a..8dfc19c 100644 --- a/source/doc.go +++ b/source/doc.go @@ -18,13 +18,15 @@ // src := &Channel[int]{Input: input} // out, errs := src.Read(context.Background()) // for item := range out { -// fmt.Println(item) +// fmt.Println(item) // prints 1 then 2 // } // for range errs { // } // -// Output: -// -// 1 -// 2 +// Caution: draining out fully and then errs sequentially, as shown above, is +// only safe because Channel emits no errors. In general, data and errors should +// be consumed concurrently — the Batch engine reads both channels at the same +// time — otherwise a source that emits an error while data is still pending can +// block. Prefer separate goroutines (or a select) when a source may report +// errors. package source