@j_liljedahl said:
The StreamFormat property change callback is not called on the main thread, iirc. You could do some tests, but I think it actually is called in the exact right spot between two render cycles.. If so, then at least the sampleRate variable of the context can be updated there, and dispatch the notification to main queue. But, even doing potentially blocking calls on the audio thread would probably be ok here, because you can't expect no glitches and dropouts while changing sample rate anyhow? I'm thinking of de AU deinit/reinit dance needed to change their streamFormat.
It's called on a secondary thread, not the main, but not the render thread either.
@Michael said:
Actually now I think about it, it shouldn't be a problem. I'm setting the sample rate specifically, so I guess it's just going to use a converter internally until the main thread gets the hint and switches
That's right. In between hw SR changing and you updating streamFormat on your IO unit, it will use converters. But you can update your IO streamFormat directly in the streamFormat prop callback (syncing client side fmt with outer-side fmt).
iirc It only does format-conversions automatically, not rate conversions. if it does, then maxframes to process would change. I'll do some testing today.
I don't like this kind of wrapping. Exactly what is the issue trying to be solved? Is it clearness of the code, that you can't see in the code how many buffers are popped and pushed?
If so, what about a naming convention where a module is named MyModule1_2 if it pops 1 and push 2, or MyModuleN_2 if it pops a variable number of buffers and pushes 2, etc?
Or, don't wrap module processing in AEModuleProcess but use the module's C function(s) directly? MyMixerMixNPush1(module, context)?
Change from a module-centric to a context-centric approach. Just and exactly like PostScript, or most other state-machine solutions like CoreGraphics or HW Graphics. You then make the context like an object with data hiding. The context is responsible for the integrity of its internal state. It can check modules as mentioned, and it could probably solve that pesky sampleRate switches problem as well. i.e. the context should send the rate switch statement, since the context may be an offline context which doesn't care for the current hw rate...
I don't like this kind of wrapping. Exactly what is the issue trying to be solved? Is it clearness of the code, that you can't see in the code how many buffers are popped and pushed?
If so, what about a naming convention where a module is named MyModule1_2 if it pops 1 and push 2, or MyModuleN_2 if it pops a variable number of buffers and pushes 2, etc?
Or, don't wrap module processing in AEModuleProcess but use the module's C function(s) directly? MyMixerMixNPush1(module, context)?
Change from a module-centric to a context-centric approach. Just and exactly like PostScript, or most other state-machine solutions like CoreGraphics or HW Graphics. You then make the context like an object with data hiding. The context is responsible for the integrity of its internal state. It can check modules as mentioned, and it could probably solve that pesky sampleRate switches problem as well. i.e. the context should send the rate switch statement, since the context may be an offline context which doesn't care for the current hw rate...
ouch, just came up with that last variation :-(
... rethinking...
Right. I agree AEContextRenderModule(context, module) looks good, like CGContext. But I didn't like the ApplyEffect etc.. Because it's the module that decides number of buffers to pop and push, not the context!
I think it would be more like type checking - means you can tell at a glance what's happening rather than relying on documentation. It wouldn't work generally, but for common cases it's probably feasible
@Michael said:
I think it would be more like type checking - means you can tell at a glance what's happening rather than relying on documentation. It wouldn't work generally, but for common cases it's probably feasible
Then I'd prefer AEContextRenderModule1_1() or similar, or ..Pop1Push1, etc. And maybe the type checking could be done compile time? By having module subclasses for the common cases.
Btw, then you also have the number of channels.. So each could be named PopStereo1PushMono1 etc..
@Michael said:
I think it would be more like type checking - means you can tell at a glance what's happening rather than relying on documentation. It wouldn't work generally, but for common cases it's probably feasible
Then I'd prefer AEContextRenderModule1_1() or similar, or ..Pop1Push1, etc. And maybe the type checking could be done compile time? By having module subclasses for the common cases.
Btw, then you also have the number of channels.. So each could be named PopStereo1PushMono1 etc..
I'd like to see something along the lines of:
AEError AEContextRenderModule(context, module)
{
if (context->sampleRate != module->sampleRate)
return kAEParamError;
if (context->currentBuffers < module->requiredBuffers)
return kAEParamError;
}
One thing is for sure; the context contains the defining state parameters and should technically be the controller for those parameters. Just because a module has a renderer with some sampleRate, doesn't mean it should be able to change the context at will.
@Michael could you separate the AEContext definition from the AERenderer file into a new file? Will probably be easier that way to update things in the near future.
typedef struct
{
//! The current sample rate, in Hertz
double sampleRate;
//! The number of channels available in output
UInt32 channelCount;
//! The number of frames to render to the output
UInt32 frameCount;
//! The current audio timestamp
const AudioTimeStamp * _Nonnull timestamp;
//! The buffer stack. Use this as a workspace for generating and processing audio.
AEBufferStack * _Nonnull stack;
//! The output buffer list. You should write to this to produce audio.
const AudioBufferList * _Nonnull output;
}
AEContext, AERenderContext;
It adds the output channelCount as an additional parameter so you don't have to traverse the output audiobufferlist, and the order is slightly different both for alignment purposes as well as order of importance.
Also, we might want to change naming to AEContext for purposes of brevity...
@32BT said:
Sketching last night I came up with this:
typedef struct
{
//! The current sample rate, in Hertz
double sampleRate;
//! The number of channels available in output
UInt32 channelCount;
//! The number of frames to render to the output
UInt32 frameCount;
//! The current audio timestamp
const AudioTimeStamp * _Nonnull timestamp;
//! The buffer stack. Use this as a workspace for generating and processing audio.
AEBufferStack * _Nonnull stack;
//! The output buffer list. You should write to this to produce audio.
const AudioBufferList * _Nonnull output;
}
AEContext, AERenderContext;
It adds the output channelCount as an additional parameter so you don't have to traverse the output audiobufferlist, and the order is slightly different both for alignment purposes as well as order of importance.
Also, we might want to change naming to AEContext for purposes of brevity...
Looks good, but isn't channelCount the same as output->mNumberBuffers? No traversing needed?
AEError AEContextRenderModule(AEContext *context, AEModule *module)
{
double moduleRate = AEModuleGetSampleRate(module);
// module either doesn't care about sampleRate, or it should match
if ((moduleRate != 0.0)&&(moduleRate != context->sampleRate))
{ return kAEParamError; }
// range should probably be available directly in context
// so we don't carry unnecessary obfuscating overhead of
// unused QT timestamp fields
AERange frameRange =
{ context->timestamp->mSampleTime, context->frameCount };
return AEModuleRender(module, frameRange, context->stack);
}
AEError AEContextFlush(AEContext *context)
{
// copy top of stack to context->output ?
return AEContextFlushToBuffer(context, context->output);
}
AEError AEContextFlushToBuffer(AEContext *context, AudioBufferList *output)
{
// copy top of stack to specific output ?
}
@Michael said:
No chance of that - I'm done supporting a variety of formats
QFT! Do you know whether the surround formats ever have interleaved buffers?
Hmm - I don't think it really matters, because that's a factor of the input/output encoding, rather than the working formats. You'd have a 5 channel noninterleaved float format for working, and then you'd write to whatever format/container you like. If you need to work with another library, you can just chuck in a converter for that step. I think!
@32BT said:
Why is a bufferstack moving its buffers between "free" and "used"? Is there ever a time when it is not using contiguous buffers?
There is indeed! You can actually remove buffers at any index in the stack, not just off the top. This allows you to do things like push a buffer, process the previous two stack items, then remove those two (for a sidechain effect, for example). The alternative would require a separate intermediate memory area to copy popped buffers onto.
@32BT said:
Why is a bufferstack moving its buffers between "free" and "used"? Is there ever a time when it is not using contiguous buffers?
There is indeed! You can actually remove buffers at any index in the stack, not just off the top. This allows you to do things like push a buffer, process the previous two stack items, then remove those two (for a sidechain effect, for example). The alternative would require a separate intermediate memory area to copy popped buffers onto.
If the effect can process in place, this could also be done by reading from the two buffers on stack, writing into the second (counting from the top), then popping the top.
Popping items from the middle of the stack is usually not something you see in other stack machines But yeah, having only contiguous buffers would mean having to copy buffers in some cases.
@j_liljedahl said:
If the effect can process in place, this could also be done by reading from the two buffers on stack, writing into the second (counting from the top), then popping the top.
Popping items from the middle of the stack is usually not something you see in other stack machines But yeah, having only contiguous buffers would mean having to copy buffers in some cases.
Yep, that's true, if it can process in place. Might not always be the case, though - one example, although I accept that there are other solutions, is when you're mixing 2 buffers, stereo on the top and mono on the bottom. You want the final buffer to have MAX(channels), so if you use the second-from-top buffer, that won't work. What AEStackBufferMix does is swap the top two so that the target buffer (second from top) has the higher channel count.
@j_liljedahl said:
If the effect can process in place, this could also be done by reading from the two buffers on stack, writing into the second (counting from the top), then popping the top.
Popping items from the middle of the stack is usually not something you see in other stack machines But yeah, having only contiguous buffers would mean having to copy buffers in some cases.
Yep, that's true, if it can process in place. Might not always be the case, though - one example, although I accept that there are other solutions, is when you're mixing 2 buffers, stereo on the top and mono on the bottom. You want the final buffer to have MAX(channels), so if you use the second-from-top buffer, that won't work. What AEStackBufferMix does is swap the top two so that the target buffer (second from top) has the higher channel count.
Yeah, but Swap is a very common operation on stack machines. The fact that each buffer on the stack can have different number of channels makes it all a bit more complex, of course. As you know, I'd prefer to have only single float buffers on the stack So two channels == two buffers. That's how stack machines usually works: an array on the stack is N single vales pushed to the stack.
@j_liljedahl said:
Yeah, but Swap is a very common operation on stack machines. The fact that each buffer on the stack can have different number of channels makes it all a bit more complex, of course. As you know, I'd prefer to have only single float buffers on the stack So two channels == two buffers. That's how stack machines usually works: an array on the stack is N single vales pushed to the stack.
Yup - swap needs the same architecture to support it as remove-from-n though, so it was just as easy, and I found reduces the intellectual burden on the developer at times.
I know you preferred that option I couldn't get behind it, too much burden on the dev. I'm starting to wonder if the stack stuff is already pushing it too far - thinking a graph adapter might be needed.
@j_liljedahl said:
Yeah, but Swap is a very common operation on stack machines. The fact that each buffer on the stack can have different number of channels makes it all a bit more complex, of course. As you know, I'd prefer to have only single float buffers on the stack So two channels == two buffers. That's how stack machines usually works: an array on the stack is N single vales pushed to the stack.
Yup - swap needs the same architecture to support it as remove-from-n though, so it was just as easy, and I found reduces the intellectual burden on the developer at times.
I know you preferred that option I couldn't get behind it, too much burden on the dev. I'm starting to wonder if the stack stuff is already pushing it too far - thinking a graph adapter might be needed.
When I was a kid and learned some PostScript, I felt a wave of bliss and happiness seeing the elegance of the stack machine. I wouldn't want to spare anyone that experience
The nice thing with the stack approach in TAAE2 however is that the most common situations are very straight forward, even easier than before:
I agree that it's elegant and lovely - I'm certainly going to use it this way. But there is definitely a comprehension problem to be solved one way or the other: either by adding a familiar graph-style interface, or by figuring out how to explain it.
I just spoke with another dev about it on the phone, and he was having trouble understanding; he said it cleared things up a lot for him after I mentioned that (a) the stack is a scratch pad, not a mapping to output channels or sequential buffers, and (b) the stack gets reset every render cycle, the contents have nothing to do with the audio output. So finding things like that that make it easier to understand will be important for the introductory guide.
@Michael said:
I agree that it's elegant and lovely - I'm certainly going to use it this way. But there is definitely a comprehension problem to be solved one way or the other: either by adding a familiar graph-style interface, or by figuring out how to explain it.
I just spoke with another dev about it on the phone, and he was having trouble understanding; he said it cleared things up a lot for him after I mentioned that (a) the stack is a scratch pad, not a mapping to output channels or sequential buffers, and (b) the stack gets reset every render cycle, the contents have nothing to do with the audio output. So finding things like that that make it easier to understand will be important for the introductory guide.
I think scratchpad+reset is a good way of describing it. What might also be helpful for comprehension is the answer to the following question::
What problem is the stack trying to solve?
Intuitively it comes down to the renderer which still doesn't sit right with me. Either everything is a renderer (derived object), or there is one central renderer which is also the controlling object. But to initialize every module with some renderer seems incorrect.
Perhaps there is a parallel with a postscript interpreter, for no reason at all, but I haven't figured out how yet (nor how to do RPN in audio...)
@Michael said:
I agree that it's elegant and lovely - I'm certainly going to use it this way. But there is definitely a comprehension problem to be solved one way or the other: either by adding a familiar graph-style interface, or by figuring out how to explain it.
I just spoke with another dev about it on the phone, and he was having trouble understanding; he said it cleared things up a lot for him after I mentioned that (a) the stack is a scratch pad, not a mapping to output channels or sequential buffers, and (b) the stack gets reset every render cycle, the contents have nothing to do with the audio output. So finding things like that that make it easier to understand will be important for the introductory guide.
I think scratchpad+reset is a good way of describing it. What might also be helpful for comprehension is the answer to the following question::
What problem is the stack trying to solve?
For me it gives less complexity and a lot more flexibility, compared to a graph or other stuff.
There's no need to define "sources", "filters" and "destinations", etc, any module can be any combination, the concept is simple: a module pops 0 to N signals from the stack, and pushes 0 to M signals on the stack.
No need to "connect" anything.
Dynamism: alternative signal paths is a simple matter of running or not running modules, which can be done by conditionals in the render callback.
The buffer stack and everything else are optional, if needed you can just have a single chunk of code in your render callback writing to the output buffers.
Intuitively it comes down to the renderer which still doesn't sit right with me. Either everything is a renderer (derived object), or there is one central renderer which is also the controlling object. But to initialize every module with some renderer seems incorrect.
I agree it feels a bit ugly that each module needs to know about its renderer. But as Michael said, the alternative is for each module to be added to the renderer with [renderer addModule: module] or similar, which doesn't feel good either. The reason it's needed is because a module needs to know about sample rate changes, is that the only thing, @Michael ?
Perhaps there is a parallel with a postscript interpreter, for no reason at all, but I haven't figured out how yet (nor how to do RPN in audio...)
Actually it already is RPN: oscA + oscB is expressed as oscA oscB +:
Yep, that's right - it's about sample rate changes (and, I guess, channel counts, maybe?). My original implementation had modules monitoring the context for sample rate changes only, but some modules need main thread handling cos they need to reinit an audio unit, for example. Couldn't find a nice way to do that without an obj-c connection. Open to suggestions though
@Michael said:
Yep, that's right - it's about sample rate changes (and, I guess, channel counts, maybe?). My original implementation had modules monitoring the context for sample rate changes only, but some modules need main thread handling cos they need to reinit an audio unit, for example. Couldn't find a nice way to do that without an obj-c connection. Open to suggestions though
Right. Which module would need to use the output channel count?
So, the context is passed to AEModuleProcess(), and it can read the sample rate from it. So far there's no need to have a reference to the renderer in the module itself. The problem is when a sample rate change needs to trigger code on the main thread, to reinit an audio unit. Specifically, modules that wraps an AudioUnit (including IAA nodes or AUv3 extension units).
One way would be to simply dispatch_async to main queue from the audio thread: one could argue that having a risk of dropped buffers during a sample rate change is really not much of a problem. A user doesn't expect a sample rate change to be glitch free (I wonder if it even can be)?
Otherwise, one would need a realtime to main messaging mechanism to trigger the code on the main thread, like AEMessageQueue. All AU-wrapping modules could take a messageQueue as init parameter, for this use case. Most apps will probably want a message queue in their audio engine anyway?
Another alternative would be to make an explicit "connection" for those modules that need to handle sample rate changes that way, by hooking them into the StreamFormat callback. This is what I do in AUM. Doing a dispatch_async to main queue from that callback is certainly not a problem. So then, you'd do something like:
Hmm. Interesting. Maybe that could be handled by the AEModule superclass too. It could do all the monitoring of the context passed to it and then dispatch/etc to rendererDidChangeSampleRate on main thread. Means modules now have to be concerned with rendering while changing their state, but that's not too hard. It moves inelegance from one place to another, but it might be nicer.
Comments
It's called on a secondary thread, not the main, but not the render thread either.
iirc It only does format-conversions automatically, not rate conversions. if it does, then maxframes to process would change. I'll do some testing today.
Change from a module-centric to a context-centric approach. Just and exactly like PostScript, or most other state-machine solutions like CoreGraphics or HW Graphics. You then make the context like an object with data hiding. The context is responsible for the integrity of its internal state. It can check modules as mentioned, and it could probably solve that pesky sampleRate switches problem as well. i.e. the context should send the rate switch statement, since the context may be an offline context which doesn't care for the current hw rate...
ouch, just came up with that last variation :-(
... rethinking...
Right. I agree AEContextRenderModule(context, module) looks good, like CGContext. But I didn't like the ApplyEffect etc.. Because it's the module that decides number of buffers to pop and push, not the context!
I think it would be more like type checking - means you can tell at a glance what's happening rather than relying on documentation. It wouldn't work generally, but for common cases it's probably feasible
Then I'd prefer AEContextRenderModule1_1() or similar, or ..Pop1Push1, etc. And maybe the type checking could be done compile time? By having module subclasses for the common cases.
Btw, then you also have the number of channels.. So each could be named PopStereo1PushMono1 etc..
I'd like to see something along the lines of:
One thing is for sure; the context contains the defining state parameters and should technically be the controller for those parameters. Just because a module has a renderer with some sampleRate, doesn't mean it should be able to change the context at will.
@Michael could you separate the AEContext definition from the AERenderer file into a new file? Will probably be easier that way to update things in the near future.
Sketching last night I came up with this:
It adds the output channelCount as an additional parameter so you don't have to traverse the output audiobufferlist, and the order is slightly different both for alignment purposes as well as order of importance.
Also, we might want to change naming to AEContext for purposes of brevity...
Looks good, but isn't channelCount the same as output->mNumberBuffers? No traversing needed?
For the current preferred streamformat, yes, but if there's ever a chance of an interleaved format...
No chance of that - I'm done supporting a variety of formats
QFT! Do you know whether the surround formats ever have interleaved buffers?
For the context it's easy: AEContextGetOutputChannelCount(context)
That will conveniently separate access from internals.
Rough outline ?
Why is a bufferstack moving its buffers between "free" and "used"? Is there ever a time when it is not using contiguous buffers?
Hmm - I don't think it really matters, because that's a factor of the input/output encoding, rather than the working formats. You'd have a 5 channel noninterleaved float format for working, and then you'd write to whatever format/container you like. If you need to work with another library, you can just chuck in a converter for that step. I think!
Yeah, neater - I really like the context-as-interface idea. I'll process this soon.
There is indeed! You can actually remove buffers at any index in the stack, not just off the top. This allows you to do things like push a buffer, process the previous two stack items, then remove those two (for a sidechain effect, for example). The alternative would require a separate intermediate memory area to copy popped buffers onto.
If the effect can process in place, this could also be done by reading from the two buffers on stack, writing into the second (counting from the top), then popping the top.
Popping items from the middle of the stack is usually not something you see in other stack machines
But yeah, having only contiguous buffers would mean having to copy buffers in some cases.
Yep, that's true, if it can process in place. Might not always be the case, though - one example, although I accept that there are other solutions, is when you're mixing 2 buffers, stereo on the top and mono on the bottom. You want the final buffer to have MAX(channels), so if you use the second-from-top buffer, that won't work. What AEStackBufferMix does is swap the top two so that the target buffer (second from top) has the higher channel count.
Yeah, but Swap is a very common operation on stack machines. The fact that each buffer on the stack can have different number of channels makes it all a bit more complex, of course. As you know, I'd prefer to have only single float buffers on the stack
So two channels == two buffers. That's how stack machines usually works: an array on the stack is N single vales pushed to the stack.
Yup - swap needs the same architecture to support it as remove-from-n though, so it was just as easy, and I found reduces the intellectual burden on the developer at times.
I know you preferred that option
I couldn't get behind it, too much burden on the dev. I'm starting to wonder if the stack stuff is already pushing it too far - thinking a graph adapter might be needed.
When I was a kid and learned some PostScript, I felt a wave of bliss and happiness seeing the elegance of the stack machine. I wouldn't want to spare anyone that experience
The nice thing with the stack approach in TAAE2 however is that the most common situations are very straight forward, even easier than before:
Then, expanding it to multiple "channels/tracks" is just a small step:
In those simple cases, you don't even really need to think about the stack, just see it as a signal flow from top to bottom.
I agree that it's elegant and lovely - I'm certainly going to use it this way. But there is definitely a comprehension problem to be solved one way or the other: either by adding a familiar graph-style interface, or by figuring out how to explain it.
I just spoke with another dev about it on the phone, and he was having trouble understanding; he said it cleared things up a lot for him after I mentioned that (a) the stack is a scratch pad, not a mapping to output channels or sequential buffers, and (b) the stack gets reset every render cycle, the contents have nothing to do with the audio output. So finding things like that that make it easier to understand will be important for the introductory guide.
I think scratchpad+reset is a good way of describing it. What might also be helpful for comprehension is the answer to the following question::
What problem is the stack trying to solve?
Intuitively it comes down to the renderer which still doesn't sit right with me. Either everything is a renderer (derived object), or there is one central renderer which is also the controlling object. But to initialize every module with some renderer seems incorrect.
Perhaps there is a parallel with a postscript interpreter, for no reason at all, but I haven't figured out how yet (nor how to do RPN in audio...)
For me it gives less complexity and a lot more flexibility, compared to a graph or other stuff.
I agree it feels a bit ugly that each module needs to know about its renderer. But as Michael said, the alternative is for each module to be added to the renderer with
[renderer addModule: module]or similar, which doesn't feel good either. The reason it's needed is because a module needs to know about sample rate changes, is that the only thing, @Michael ?Actually it already is RPN:
oscA + oscBis expressed asoscA oscB +:Yep, that's right - it's about sample rate changes (and, I guess, channel counts, maybe?). My original implementation had modules monitoring the context for sample rate changes only, but some modules need main thread handling cos they need to reinit an audio unit, for example. Couldn't find a nice way to do that without an obj-c connection. Open to suggestions though
true
Right. Which module would need to use the output channel count?
So, the context is passed to AEModuleProcess(), and it can read the sample rate from it. So far there's no need to have a reference to the renderer in the module itself. The problem is when a sample rate change needs to trigger code on the main thread, to reinit an audio unit. Specifically, modules that wraps an AudioUnit (including IAA nodes or AUv3 extension units).
One way would be to simply dispatch_async to main queue from the audio thread: one could argue that having a risk of dropped buffers during a sample rate change is really not much of a problem. A user doesn't expect a sample rate change to be glitch free (I wonder if it even can be)?
Otherwise, one would need a realtime to main messaging mechanism to trigger the code on the main thread, like AEMessageQueue. All AU-wrapping modules could take a messageQueue as init parameter, for this use case. Most apps will probably want a message queue in their audio engine anyway?
Another alternative would be to make an explicit "connection" for those modules that need to handle sample rate changes that way, by hooking them into the StreamFormat callback. This is what I do in AUM. Doing a dispatch_async to main queue from that callback is certainly not a problem. So then, you'd do something like:
or
Hmm. Interesting. Maybe that could be handled by the AEModule superclass too. It could do all the monitoring of the context passed to it and then dispatch/etc to rendererDidChangeSampleRate on main thread. Means modules now have to be concerned with rendering while changing their state, but that's not too hard. It moves inelegance from one place to another, but it might be nicer.