EachMap? Or safe slice and map types?

[earthboundkid]

On the one hand, results, err := flowmatic.Map(workers, items, task) is a very obvious API. OTOH, it's probably slower than serial code in most instances? It seems like it needs batching to be any good. Does anyone really need it?

So maybe the answer is to approach from the other angle and have flowmatic container types than are goroutine safe and tell people to use those with Each? Say flowmatic.MakeSlice[T](cap) and flowmatic.MakeMap[K, V](cap) that return types with a Push/Add method and a Slice/Map method that panics if you Push/Add after ejecting.

Do we need an Each variant function that bails on failure? Is there some way to reduce the API surface by generalizing without just recreating errorgroup?

[earthboundkid]

Writing spotlightpa/sheets-uploader#1 was useful because it had a good case for using a concurrent safe slice. I needed it to upload the list of invalidations to CloudFront afterwards.

It might also be nice to add a buffer pool to Flowmatic, since that's a common need.

[earthboundkid]

So maybe the API is

var s flowsafe.Slice[T]
s.Append(v)
s.Unwrap()

var m flowsafe.Map[K, V]
m.Store(k, v)
m.Unwrap()

buf, put := flowpool.GetBuffer()
defer put()

var p flowpool.Buffer
buf, put := p.Get()
defer put()

It could be at top level, but it's nice to have that API simple, and hopefully someday flowsafe.Map will be obsoleted by golang/go#47657. There's no proposal for a safe slice though. I think flowsafe containers would still have a usecase because they can be unwrapped when the concurrency is over.

The bytes.Buffer pool could be with the containers, but it's not really "safe". It's just pooled. It leaves open the door for having other kinds of pools, although I can't think of many others that would be better than sync.PoolOf, which will probably happen.

[earthboundkid]

I looked into the buffer pool, but after reading through golang/go#23199 I think maybe it's not worth it. I couldn't figure out to make the nicer buf, put := API not allocate for the closure, and probably most of the time you're better off trusting the GC.

[earthboundkid]

See #4.

[earthboundkid]

I have fully spun around the axel on this one.

• I wanted to have safe slices, so I made flowsafe.Slice
• I wanted the slices to be in order, so I made flowmatic.EachN
• EachN meant I didn't need flowsafe.Slice
• I wanted to be able to cancel further work, so I made flowmatic.Map(ctx)
• Map meant I didn't need EachN…

But I think having been on the journey that the destination makes sense: Each doesn't take a context, return a result, or cancel early; Map takes a context, returns a result, cancels early. They are both useful, but for different things.