TAAE2: Pesky Questions Thread

32BT

For purposes of clarifying design objectives, I decided to open a thread to allow annoying questions regarding the current structure. The answers should hopefully help with a better understanding of how to use and implement the new version.

The first thing that boggles me currently:
1. why is a module initialized with a renderer? What is the purpose of the renderer?

I mean from a design perspective. The purpose of objects is to encapsulate and control data by datahiding. How are the module and the renderer related? Does a module control the renderer? Why can a renderer be changed after initialization? Is that ever relevant?

Note that a lot of thought seems to have gone into providing relatively threadsafe parameterchanges, but changing the renderer while observing sampleRate/channelCount changes is a potential pitfall: when the renderer is updated, the sampleRate/channelCount needs to be reset potentially, and threadwise it may generate an observerupdate for the new renderer, while the audiothread is still working with the old renderer etc...

In that regard i found that observing those changes from the system trails behind the actual update, i.e. the renderer may already have been forced to process a different sampleRate on the audioThread, while the management logic still needs to be notified. Perhaps there should be different logic in the audio-thread that continuously checks for these basic parameters before letting a module process a buffer.

32BT

Another thing that eludes me is this vital piece of code:

// We're not actually using AEManagedValue's performAtomicBatchUpdate: method, but if we were, // it's very important to include this commit at the start of the main render cycle. AEManagedValueCommitPendingAtomicUpdates();

I am not entirely sure what it is meant to do, but currently understand that it updates all changed parameters in managed values. This clearly is so extremely important, wouldn't it be sensible to implement by default prior to calling the main renderer.block? When would you not want this to happen?

32BT

When I first saw the initial TAAE2 preview example, I thought it was a great idea to introduce an AEContext and implement a context-centric approach. It made me think of a CGContext as a parallel. But the thing that surprised me was that the processing logic didn't reference the context first, instead there where generic AEProcessModule calls.

Wouldn't it make more sense to write AEContext... calls to help understand what actually happens to the stackstructure?

e.g. from the main example:

AEAudioUnitInputModule * input = [AEAudioUnitInputModule new];
AEDelayModule * delay = [AEDelayModule new];
AEAudioFilePlayerModule * player = [[AEAudioFilePlayerModule alloc] initWithPath:backgroundMusic];
AEAudioFileRecorderModule * recorder = [[AEAudioFileRecorderModule alloc] initWithPath:savePath];

self.output = [[AEAudioUnitOutput alloc] initWithBlock:^(const AERenderContext * _Nonnull context) {
    // Pushes a buffer on the stack (containing the input audio)
    AEModuleProcess(input, context);
    
    // Modifies the top buffer in place (apply delay effect)
    AEModuleProcess(delay, context);
    
    // Pushes a second buffer (containing playback audio)
    AEModuleProcess(player, context);
    
    // Mixes the buffers (input + playback = karaoke funtimes)
    AEBufferStackMix(context, 2);
    
    // Processes the top buffer in place (record it to file)
    AEModuleProcess(recorder, context);
    
    // Sends the top 1 buffer(s) to the output starting at output channel 0
    AERendererMixOutput(context, 1, 0);
}];

Note how the comments have no relation whatsoever with the subsequent calls:
// Pushes a buffer on the stack (containing the input audio) AEModuleProcess(input, context);

It would make more sense, and would be a lot clearer to read if it for example said:

AEContextPushBuffer(context, input)

and

AEContextApplyEffect(context, delay)

---

Similarly, in the current example project, the main renderer seems to address the stack field in the AEContext. But the renderer is clearly not the controller for those fields. e.g.:

AEBufferStackMixToBufferList(context->stack, 1, 0, YES, context->output);

32BT

Is there a reason for using blocks wherever and whenever? blocks are useful constructs for parallel processing, or for communicating minor state-changes etc... It seems in a lot of places, a simple function pointer suffices, and is actually used internally, so the blocks are primarily adding additional unnecessary shells, and may even hide the fact that the actually parameter used is managed by the system, not by the object.

The example that comes to mind, from the AEAudioUnitOutputModule:

    AEManagedValue * rendererValue = [AEManagedValue new];
    rendererValue.objectValue = renderer;
    self.rendererValue = rendererValue;

    self.ioUnit = [AEIOAudioUnit new];
    self.ioUnit.outputEnabled = YES;
    self.ioUnit.renderBlock = ^(AudioBufferList * _Nonnull ioData, UInt32 frames, const AudioTimeStamp * _Nonnull timestamp) {
        __unsafe_unretained AERenderer * renderer = (__bridge AERenderer*)AEManagedValueGetValue(rendererValue);
        if ( renderer ) {
            AERendererRun(renderer, ioData, frames, timestamp);
        } else {
            AEAudioBufferListSilence(ioData, AEAudioDescription, 0, frames);
        }
    };

Note that the rendererValue in the block is a copy of the local context parameter, not the object's self.rendererValue.

32BT

Are these questions annoying enough just yet?

32BT

Consistency? poolSize = poolCount?
numberOfSingleChannelBuffers = channelCount?

Normally I would read poolSize as a number of bytes of memory.

and this assignment confuses me:
numberOfSingleChannelBuffers = poolSize * maxChannelsPerBuffer;

Not trying to be nitpicking here, just showing my thoughtprocess when reading the code. Considering this to be a core structure, I'd personally love to see a bit more consistency in naming. I may be too conventional in my thinking though.

AEBufferStack * AEBufferStackNewWithOptions(int poolSize, int maxChannelsPerBuffer, int numberOfSingleChannelBuffers) {
    if ( !poolSize ) poolSize = kDefaultPoolSize;
    if ( !numberOfSingleChannelBuffers ) numberOfSingleChannelBuffers = poolSize * maxChannelsPerBuffer;

    AEBufferStack * stack = (AEBufferStack*)calloc(1, sizeof(AEBufferStack));
    stack->poolSize = poolSize;
    stack->maxChannelsPerBuffer = maxChannelsPerBuffer;
    stack->frameCount = AEBufferStackMaxFramesPerSlice;

Michael

Whew! Okay, here we go

@32BT said:
1. why is a module initialized with a renderer? What is the purpose of the renderer?

This is primarily because of sample rate changes: the renderer can change its sample rate at any time, which means the module needs to get warning when that happens, in an environment where it can do allocations, un-init/re-init, etc (i.e. the main thread). There may be multiple renderers at one time, so it's not feasible to use a singleton. Giving the renderer to the modules (and allowing them to observe the sample rate of their renderer) allowed a simpler implementation than vice versa, because the alternative would have required a bunch of [renderer addModule:] ... [renderer removeModule:] calls to manage the relationship.

A renderer can be changed after initialization because a module can potentially be moved from one rendering environment to another.

Note that a lot of thought seems to have gone into providing relatively threadsafe parameterchanges, but changing the renderer while observing sampleRate/channelCount changes is a potential pitfall: when the renderer is updated, the sampleRate/channelCount needs to be reset potentially, and threadwise it may generate an observerupdate for the new renderer, while the audiothread is still working with the old renderer etc...

Yeah, that's a fair point. You'd want to take steps to avoid that when moving a module between renderers, like using it in an AEManagedValue, or stopping the old renderer first.

In that regard i found that observing those changes from the system trails behind the actual update, i.e. the renderer may already have been forced to process a different sampleRate on the audioThread, while the management logic still needs to be notified. Perhaps there should be different logic in the audio-thread that continuously checks for these basic parameters before letting a module process a buffer.

Could be - any idea how to reproduce that? I've got an iPhone 6S Plus here which I'm using to test the live sample rate changes (it's 48k without headphones plugged, and 44.1k with), and the switches have been seamless so far.

Michael

@32BT said:
Another thing that eludes me is this vital piece of code:

// We're not actually using AEManagedValue's performAtomicBatchUpdate: method, but if we were, // it's very important to include this commit at the start of the main render cycle. AEManagedValueCommitPendingAtomicUpdates();

I am not entirely sure what it is meant to do, but currently understand that it updates all changed parameters in managed values. This clearly is so extremely important, wouldn't it be sensible to implement by default prior to calling the main renderer.block? When would you not want this to happen?

Close - it manages updates to any changes made with the performAtomicBatchUpdate: utility.

I would've done so, but renderers can be nested, and they don't know whether they're the top renderer, or an auxiliary one (like the varispeed's renderer in the sample app). The change needs to be made once per render, at the beginning, so right now only the developer knows that.

Michael

@32BT said:
It would make more sense, and would be a lot clearer to read if it for example said:

AEContextPushBuffer(context, input)

and

AEContextApplyEffect(context, delay)

Oh, that's an interesting idea - I had a conversation about this the other day with Jesse Chappell about this ambiguity problem, but didn't really solve it at the time. This could help a lot. Perhaps an AEModule could have a property set on init which defines what the module does to the stack, then there are a suite of AEContext methods which take the context and a module, verify that the module does what's expected, then calls AEModuleProcess.

One hiccup is that modules can feasibly do almost anything - there could be a module which consumes, say, 6 buffers and pushes 1, for instance (maybe it's a 5.1 to stereo converter where each input channel is in a different buffer). So it won't apply to every use case - not that I see that as a major problem, as long as one can step outside the box.

So, we'd need to cover generators (push 1), effects (push 1 pop 1) and taps (no-op), I guess. Anything that does more than that, dev's on their own and uses AEModuleProcess directly again.

32BT

@Michael said:

• there could be a module which consumes, say, 6 buffers and pushes 1, for instance (maybe it's a 5.1 to stereo converter where each input channel is in a different buffer).

Yes, I'd like that!!! Exactly like PostScript...

Now, if we'd only could introduce RPN, I would feel right back at home...
(my age is showing, i guess)

Michael

@32BT said:
Is there a reason for using blocks wherever and whenever? blocks are useful constructs for parallel processing, or for communicating minor state-changes etc... It seems in a lot of places, a simple function pointer suffices

I think there are only really 2 blocks in use, for render stuff, anyway; in AEIOAudioUnit like you describe, and AERenderer. For AERenderer I felt blocks were simpler, because of variable capture; the equivalent implementation in the sample app would be a bit more verbose were function pointers to be used, because everything used in the render loop would need to be an instance variable. So I feel that's the right choice.

For AEIOAudioUnit the choice was a bit arbitrary - as it's only used in a few places a function pointer would be just as easy to use. But I don't think it has any performance considerations, and using a block just feels simpler (no need to define a C function).

... a simple function pointer suffices and is actually used internally, so the blocks are primarily adding additional unnecessary shells

Not sure they're unnecessary per se - in AERenderer there's no difference (no containing function that just exists to call a block, for instance); in AEIOAudioUnit, the top-level render callback needs to be there so it can do the latency compensation. As far as I know, there's no more overhead in calling a block than a function pointer, so I don't think that's a factor.

...and may even hide the fact that the actually parameter used is managed by the system, not by the object.

I'm not sure I see a problem there, right now...could you elaborate? Maybe you could convince me if I knew more about your concerns

Michael

@32BT said:
Are these questions annoying enough just yet?

Hells no, bring it on

32BT

@Michael said:
A renderer can be changed after initialization because a module can potentially be moved from one rendering environment to another.

Theoretically yes, but are there actual usecases you can think of where this is useful, and for example it can not be done with freshly initialized modules or copies thereof? (init with renderer as a fixed, readonly value).

@Michael said:
Yeah, that's a fair point. You'd want to take steps to avoid that when moving a module between renderers, like using it in an AEManagedValue, or stopping the old renderer first.

Yes, and crucially when setting the new renderer, you want the module to update samplerate/channelcount as well.

@Michael said:

In that regard i found that observing those changes from the system trails behind the actual update, i.e. the renderer may already have been forced to process a different sampleRate on the audioThread, while the management logic still needs to be notified. Perhaps there should be different logic in the audio-thread that continuously checks for these basic parameters before letting a module process a buffer.

Could be - any idea how to reproduce that? I've got an iPhone 6S Plus here which I'm using to test the live sample rate changes (it's 48k without headphones plugged, and 44.1k with), and the switches have been seamless so far.

I tested with Audio MIDI Setup on the desktop which can change hardware rates while software is producing audio, and specifically got new sampleRates before getting the notification. I presume this should happen for any hardware changes, since I don't see how the system can synchronize that between both the audio-thread and the main-thread where you actually receive the notification, is it not?

Michael

@32BT said:
Consistency? poolSize = poolCount?
numberOfSingleChannelBuffers = channelCount?

Normally I would read poolSize as a number of bytes of memory.

Hmm. "Count" usually indicates a plurality of something, and here the something is "pool", so to me poolCount would mean number of pools, which isn't correct. I'm comfortable with poolSize here - it's the size of the pool! I guess it could feasibly be called maxDepth or something.

numberOfSingleChannelBuffers = channelCount?

This is probably more related to my failing to describe the point of that function. It's not so much the channel count, but the number of float arrays to provide for. The buffer stack makes space for a certain number of AudioBufferLists (poolSize of them), and a certain number of float arrays (by default, poolSize × maxChannelsPerBuffer). When you push a stereo buffer, it takes 2 of those float arrays and 1 buffer list. I realised, though, that if you're going to be working with, say, a 64-channel audio interface, then you don't want every single thing you push to have 64 channels (poolSize × 64 float arrays). That'll use a bunch of memory unnecessarily. But I can't guess in advance how many 64-channel buffers you actually want. Thus AEBufferStackNewWithOptions, which lets you choose specifically how many float arrays you need.

So, I guess the naming could be clearer...or maybe the documentation. Not sure. Any ideas?

and this assignment confuses me:
numberOfSingleChannelBuffers = poolSize * maxChannelsPerBuffer;

This provides the default behaviour: if you don't specify how many float arrays you want, it'll make sure that there's enough for every push to have maxChannelsPerBuffer channels. Ergo, size of the pool times max channels for each buffer.

Not trying to be nitpicking here, just showing my thoughtprocess when reading the code. Considering this to be a core structure, I'd personally love to see a bit more consistency in naming. I may be too conventional in my thinking though.

Please, nitpick away! The more the merrier.

Given that brain dump, do you have any suggestions for improving var naming?

Michael

@32BT said:
Now, if we'd only could introduce RPN, I would feel right back at home...
(my age is showing, i guess)

Haha! Ah, that would be glorious.

Michael

@32BT said:

A renderer can be changed after initialization because a module can potentially be moved from one rendering environment to another.

Theoretically yes, but are there actual usecases you can think of where this is useful, and for example it can not be done with freshly initialized modules or copies thereof? (init with renderer as a fixed, readonly value).

..............no Not really. Nope. Guess it could be readonly...

Yeah, that's a fair point. You'd want to take steps to avoid that when moving a module between renderers, like using it in an AEManagedValue, or stopping the old renderer first.

Yes, and crucially when setting the new renderer, you want the module to update samplerate/channelcount as well.

Yeah - I actually just added a commit that does that. But maybe readonly is better. Use a new module.

In that regard i found that observing those changes from the system trails behind the actual update, i.e. the renderer may already have been forced to process a different sampleRate on the audioThread, while the management logic still needs to be notified. Perhaps there should be different logic in the audio-thread that continuously checks for these basic parameters before letting a module process a buffer.

Could be - any idea how to reproduce that? I've got an iPhone 6S Plus here which I'm using to test the live sample rate changes (it's 48k without headphones plugged, and 44.1k with), and the switches have been seamless so far.

I tested with Audio MIDI Setup on the desktop which can change hardware rates while software is producing audio, and specifically got new sampleRates before getting the notification. I presume this should happen for any hardware changes, since I don't see how the system can synchronize that between both the audio-thread and the main-thread where you actually receive the notification, is it not?

You're right, that makes sense. The only way I know to detect it would be to query the output bus' output scope at the start of every render... do you know if it's safe to do that? I'm distrustful of those APIs because Apple never document what's safe for realtime and what is not, and it's all C.

Michael

https://github.com/TheAmazingAudioEngine/TAAE2/commit/f11b64aa0b1d05a4477173c21876141102426a1a

32BT

@Michael said:
Given that brain dump, do you have any suggestions for improving var naming?

In order of personal preference:
1. shorter is better (less overall typing)
2. whatever the environment uses (iOS/OSX)
3. internal consistency

So poolSize would mean less typing, but stackCapacity might be more descriptive?

On the other hand:
maxChannels = maxBuffers * maxChannelsPerBuffer

I might get back to this one once I'm more familiar with the code. Perhaps by then the original naming feels comfortable just fine

Michael

Okay, standing by on that then.

Lemme know if you've any ideas about the AEContext stuff, enumerating the common module activities and such.

Michael

Thank you, by the way - great feedback and it's good to know someone's got my back in the struggle against code mediocrity

32BT

@Michael said:
You're right, that makes sense. The only way I know to detect it would be to query the output bus' output scope at the start of every render... do you know if it's safe to do that? I'm distrustful of those APIs because Apple never document what's safe for realtime and what is not, and it's all C.

It's what I implemented in RMSAudio (see RMSOutput.m) to mitigate the problem, but I'm not sure if it is actually safe. I'm fairly certain that it is safe to fetch that value.

32BT

@Michael said:

Yeah - I actually just added a commit that does that. But maybe readonly is better. Use a new module.

Similarly we might want to check renderer.block as well.

renderer->initWithBlock, readonly property

want a new block? instantiate a new renderer...

Michael

Actually now I think about it, it shouldn't be a problem. I'm setting the sample rate specifically, so I guess it's just going to use a converter internally until the main thread gets the hint and switches

32BT

@Michael said:
Thank you, by the way - great feedback and it's good to know someone's got my back in the struggle against code mediocrity

YW, Please note that my questions might not always be as diplomatic as could be. Please read through the rudeness if necessary, and chalk it up to language and/or cultural differences.

32BT

@32BT said:

@Michael said:
Thank you, by the way - great feedback and it's good to know someone's got my back in the struggle against code mediocrity

YW, Please note that my questions might not always be as diplomatic as could be. Please read through the rudeness if necessary, and chalk it up to language and/or cultural differences.

or lack of cafeïne...

Michael

@32BT said:
Similarly we might want to check renderer.block as well.

renderer->initWithBlock, readonly property

want a new block? instantiate a new renderer...

I'm in two minds about that one - having it RW means one could feasibly create and store two different blocks and switch between them as needed. The implementation is simple, using AEManagedValue.

Michael

@32BT said:
YW, Please note that my questions might not always be as diplomatic as could be. Please read through the rudeness if necessary, and chalk it up to language and/or cultural differences.

Haha, no worries there. I can be absolutely terrible with diplomacy sometimes, so I'd never judge

j_liljedahl

@Michael said:

@32BT said:
It would make more sense, and would be a lot clearer to read if it for example said:

AEContextPushBuffer(context, input)

and

AEContextApplyEffect(context, delay)

Oh, that's an interesting idea - I had a conversation about this the other day with Jesse Chappell about this ambiguity problem, but didn't really solve it at the time. This could help a lot. Perhaps an AEModule could have a property set on init which defines what the module does to the stack, then there are a suite of AEContext methods which take the context and a module, verify that the module does what's expected, then calls AEModuleProcess.

One hiccup is that modules can feasibly do almost anything - there could be a module which consumes, say, 6 buffers and pushes 1, for instance (maybe it's a 5.1 to stereo converter where each input channel is in a different buffer). So it won't apply to every use case - not that I see that as a major problem, as long as one can step outside the box.

So, we'd need to cover generators (push 1), effects (push 1 pop 1) and taps (no-op), I guess. Anything that does more than that, dev's on their own and uses AEModuleProcess directly again.

I don't like this kind of wrapping. Exactly what is the issue trying to be solved? Is it clearness of the code, that you can't see in the code how many buffers are popped and pushed?

If so, what about a naming convention where a module is named MyModule1_2 if it pops 1 and push 2, or MyModuleN_2 if it pops a variable number of buffers and pushes 2, etc?

Or, don't wrap module processing in AEModuleProcess but use the module's C function(s) directly? MyMixerMixNPush1(module, context)?

j_liljedahl

@Michael said:

@32BT said:

A renderer can be changed after initialization because a module can potentially be moved from one rendering environment to another.

Theoretically yes, but are there actual usecases you can think of where this is useful, and for example it can not be done with freshly initialized modules or copies thereof? (init with renderer as a fixed, readonly value).

..............no Not really. Nope. Guess it could be readonly...

Yeah, that's a fair point. You'd want to take steps to avoid that when moving a module between renderers, like using it in an AEManagedValue, or stopping the old renderer first.

Yes, and crucially when setting the new renderer, you want the module to update samplerate/channelcount as well.

Yeah - I actually just added a commit that does that. But maybe readonly is better. Use a new module.

In that regard i found that observing those changes from the system trails behind the actual update, i.e. the renderer may already have been forced to process a different sampleRate on the audioThread, while the management logic still needs to be notified. Perhaps there should be different logic in the audio-thread that continuously checks for these basic parameters before letting a module process a buffer.

Could be - any idea how to reproduce that? I've got an iPhone 6S Plus here which I'm using to test the live sample rate changes (it's 48k without headphones plugged, and 44.1k with), and the switches have been seamless so far.

I tested with Audio MIDI Setup on the desktop which can change hardware rates while software is producing audio, and specifically got new sampleRates before getting the notification. I presume this should happen for any hardware changes, since I don't see how the system can synchronize that between both the audio-thread and the main-thread where you actually receive the notification, is it not?

You're right, that makes sense. The only way I know to detect it would be to query the output bus' output scope at the start of every render... do you know if it's safe to do that? I'm distrustful of those APIs because Apple never document what's safe for realtime and what is not, and it's all C.

The StreamFormat property change callback is not called on the main thread, iirc. You could do some tests, but I think it actually is called in the exact right spot between two render cycles.. If so, then at least the sampleRate variable of the context can be updated there, and dispatch the notification to main queue. But, even doing potentially blocking calls on the audio thread would probably be ok here, because you can't expect no glitches and dropouts while changing sample rate anyhow? I'm thinking of de AU deinit/reinit dance needed to change their streamFormat.

Michael

@j_liljedahl said:
I don't like this kind of wrapping. Exactly what is the issue trying to be solved? Is it clearness of the code, that you can't see in the code how many buffers are popped and pushed?

Yup, that's it - it'd be nice to be able to tell just from looking at the render block what state the stack's likely to be in at each point.

If so, what about a naming convention where a module is named MyModule1_2 if it pops 1 and push 2, or MyModuleN_2 if it pops a variable number of buffers and pushes 2, etc?

Hmm, I don't like that - it's a bit obtuse.

Or, don't wrap module processing in AEModuleProcess but use the module's C function(s) directly? MyMixerMixNPush1(module, context)?

That makes iteration a bit more difficult, because then we need to know the types in order to get the C function.

j_liljedahl

@Michael said:
Actually now I think about it, it shouldn't be a problem. I'm setting the sample rate specifically, so I guess it's just going to use a converter internally until the main thread gets the hint and switches

That's right. In between hw SR changing and you updating streamFormat on your IO unit, it will use converters. But you can update your IO streamFormat directly in the streamFormat prop callback (syncing client side fmt with outer-side fmt).