Metering module for TAAE2

2»

Comments

  • A copy's probably fine - as long as it's just reading once. It would only produce weird results if the main thread read multiple times expecting them to be the same each time.

  • @Michael said:

    @j_liljedahl said:
    Well, they would be valid as long as the object is valid, since they are alloced on init and released in dealloc. But sure, some might store the pointer somewhere and then the object is deallocated. Copying into a user supplied array would be safer, use a bit more cpu and memory, and need the user to make sure he supplies an array with sufficient size.

    Oh, that too - I was just thinking the values could change at any time, including midway through iterating the array on the main thread. Could produce some odd results in certain circumstances.

    Well, it's just for UI display, so some rare misread wouldn't be harmful. But isn't setting a 32 bit float always atomic on iOS? If so, all that could happen is that some channels are read as having their last value?

  • @Michael said:
    A copy's probably fine - as long as it's just reading once. It would only produce weird results if the main thread read multiple times expecting them to be the same each time.

    But if you're worried that they would change while both main and audio thread are iterating the array, copying doesn't help. Maybe I misunderstood the potential issue?

  • Following construct should pretty much solve all of our concerns, so I am going to repeat ad nauseam (since mr Darwin owning these forums wants the fittest solution) :smile:

    // This is called in a backgroundthread, but not the audiothread
    // so we are free to use whatever language constructs we like
    
    - (void) updateWithRingBufferModule:(AERingBufferModule *)ringBuffer
    {
        // Reinitialize engines if necessary
        Float64 sampleRate = ringBuffer.renderer.sampleRate;
        if (mEngineRate != sampleRate)
        {
            mEngineRate = sampleRate;
            mEngineL = RMSEngineInit(sampleRate);
            mEngineR = RMSEngineInit(sampleRate);
        }
    
        // Compute samplerange since last update
        AERange range = [ringBuffer availableRange];
    
        if (mIndex < range.index)
        { mIndex = range.index; }
        range.count -= mIndex-range.index;
    
        // Process samples
        for (uint64_t n=range.count; n!=0; n--)
        {
            RMSEngineAddSample(&mEngineL, [ringBuffer valueAtIndex0:mIndex]);
            RMSEngineAddSample(&mEngineL, [ringBuffer valueAtIndex1:mIndex]);
            mIndex += 1;
        }
    
        // Transfer result to view on main
        rmsresult_t L = RMSEngineFetchResult(&mEngineL);
        rmsresult_t R = RMSEngineFetchResult(&mEngineR);
    
        dispatch_async(dispatch_get_main_queue(),
        ^{
            mLevelsViewL.levels = L;
            mLevelsViewR.levels = R;
        });
    }
    

    Please note the steps in abstract (happening on a (non audio) background thread):
    1. reinitialize engine if necessary
    2. compute samplerange available in ringbuffer
    3. process the samples since last update
    4. transfer the resultlevels to main for actual displaying

    That's clean, succinct, and to-the-point, and not nearly as complex as previously mentioned. Considering what currently happens in the LevelsMeteringModule, I consider this a more elegant (and for darwinists: more flexible) solution.

    /end-of-elevator-pitch

  • Thanks for the great feedback, I am learning lots. Not trying to solve every use-case totally makes sense...I'll see if I can tackle this again tomorrow. The name AEMeteringModule is shorter - should I just revert to that name and leave it at that?

  • edited April 2016

    @32BT said:
    Following construct should pretty much solve all of our concerns, so I am going to repeat ad nauseam (since mr Darwin owning these forums wants the fittest solution) :smile:

    Haha! Now make them fight! Two implementations enter. One implementation leaves.

    Yeah, I think a ring buffer module would be a great addition (except maybe using TPCircularBuffer, I think, cos it makes accessing easier). For the level monitoring, it's probably overkill though, I think, but there are plenty of good use cases for it.

    @j_liljedahl said:
    But if you're worried that they would change while both main and audio thread are iterating the array, copying doesn't help. Maybe I misunderstood the potential issue?

    It is rather obscure - I was just thinking that if your UI rendering code does two passes (maybe it renders a background element for each level, then a foreground element, for instance), then it might get out of sync. Maybe it's too obscure to worry about, because I do like the simplicity of returning the info. I was even wondering about something like:

    typedef struct {
      float average;
      float peak;
    } AEMeteringChannelLevels;
    
    typedef struct {
      int channelCount;
      AEMeteringChannelLevels channels[1];
    } AEMeteringLevels;
    
    ...
    
    - (AEMeteringLevels *)levels;
    
    ...
    
    AEMeteringLevels * levels = _monitor.levels;
    _levelsView.levels = levels;
    

    @leothiessen - Yeah, I think AEMeteringModule is good!

  • edited April 2016

    @Michael said:

    @j_liljedahl said:
    But if you're worried that they would change while both main and audio thread are iterating the array, copying doesn't help. Maybe I misunderstood the potential issue?

    It is rather obscure - I was just thinking that if your UI rendering code does two passes (maybe it renders a background element for each level, then a foreground element, for instance), then it might get out of sync. Maybe it's too obscure to worry about, because I do like the simplicity of returning the info. I was even wondering about something like:

    typedef struct {
      float average;
      float peak;
    } AEMeteringChannelLevels;
    
    typedef struct {
      int channelCount;
      AEMeteringChannelLevels channels[1];
    } AEMeteringLevels;
    
    ...
    
    - (AEMeteringLevels *)levels;
    
    ...
    
    AEMeteringLevels * levels = _monitor.levels;
    _levelsView.levels = levels;
    

    I don't think that code would compile, or run without crashing? The compiler doesn't know the size of a variable-sized struct as the above, so I'd be surprised if it could just copy it like a value.

    The only way to return variable-sized data in C is by reference, as far as I know.

  • It's a pointer =)

  • @Michael said:
    It's a pointer =)

    Ugh.. I should get glasses! :)

    Well then, so you also suggest returning a reference to the array. Good idea putting it in a struct with the channel count.

  • Yeah, I'm in two minds about it. It'd be elegant to return a direct pointer.

  • the interface connections are a bit dodgy, but the ringbuffermetering works:
    https://github.com/32Beat/TAAE2
    (develop branch)

  • @Michael said:

    @32BT said:
    Following construct should pretty much solve all of our concerns, so I am going to repeat ad nauseam (since mr Darwin owning these forums wants the fittest solution) :smile:

    Haha! Now make them fight! Two implementations enter. One implementation leaves.

    lol!

    @32BT said:
    the interface connections are a bit dodgy, but the ringbuffermetering works:
    https://github.com/32Beat/TAAE2
    (develop branch)

    Now that is some pretty sweet work & works great from what I can tell! Now, any chance that code could get cleaned up so a guy like myself can wrap his mind around it? - I'm not sure you'd like what I might do to it, hehe, though I'm inclined to try if you don't feel like it :smile:

  • @leothiessen said:
    Now that is some pretty sweet work & works great from what I can tell! Now, any chance that code could get cleaned up so a guy like myself can wrap his mind around it? - I'm not sure you'd like what I might do to it, hehe, though I'm inclined to try if you don't feel like it :smile:

    Cleaned up??? I only write clean code, can't you tell? :smirk:

    clone the project and experiment at your hearts content. Let me clean this up first, and let me add some commenting, as well as some additional monitoring examples.

    If you like you can find some interesting examples in the RMSAudioApp project, which displays metering, lissajous, and a spectrogram all at the same time, without incurring any timepenalties on the audiothread...

  • Interesting problem:
    I'm attempting to implement submetering for the three different loops in addition to metering for the overall output. For submetering I need to read the top of the stack, but for the overall output I need to read the output buffer. In the module I do not specifically know which to use...

    Obviously, I can set some indexing ivar to tell me which buffer to use, but me thinks this should not be necessary, since the top of the stack should always contain the latest state. In other words: should there be an output buffer? Or should the renderloop always assume that the final output is available at the top of the stack at the end of the renderloop?

  • @32BT said:
    Interesting problem:
    I'm attempting to implement submetering for the three different loops in addition to metering for the overall output. For submetering I need to read the top of the stack, but for the overall output I need to read the output buffer. In the module I do not specifically know which to use...

    Obviously, I can set some indexing ivar to tell me which buffer to use, but me thinks this should not be necessary, since the top of the stack should always contain the latest state. In other words: should there be an output buffer? Or should the renderloop always assume that the final output is available at the top of the stack at the end of the renderloop?

    This would of course be cleaner, but complicates stuff for multichannel outputs. For example, outputting each track to a specified hw output.

    However, this could perhaps be solved in a different way than having an output buffer in the context: Instead pushing a buffer with N channels (for an N output interface), and use some bufferstack operation that mixes onto this. It would be pushed first, so it's at the bottom of the stack, and a mechanism to store a ref to this buffer in a local var. (This could also be used for any other kind of mix bus). Then, at the end of the loop, this buffer should again be the top buffer (all else has been mixed down to it and popped off the stack), and it's regarded as the final result of the rendering, used by its renderer (output unit or subrenderer such as varispeed or oversampler etc)

  • edited April 2016

    @Michael said:
    ...
    I was even wondering about something like:

    typedef struct {
      float average;
      float peak;
    } AEMeteringChannelLevels;
    
    typedef struct {
      int channelCount;
      AEMeteringChannelLevels channels[1];
    } AEMeteringLevels;
    
    ...
    
    - (AEMeteringLevels *)levels;
    
    ...
    
    AEMeteringLevels * levels = _monitor.levels;
    _levelsView.levels = levels;
    

    Is this what you mean @Michael ?

    typedef struct {
        float average;
        float peak;
    } AEMeteringChannelLevels;
    
    typedef struct {
        int channelCount;
        AEMeteringChannelLevels * _Nonnull channels;
    } AEMeteringLevels;
    
    @interface AEMeteringModule : AEModule
    - (instancetype _Nullable)initWithRenderer:(AERenderer * _Nonnull)renderer; // 2 channels by default
    - (instancetype _Nullable)initWithRenderer:(AERenderer * _Nonnull)renderer channelCount:(int)channelCount; // 1+ channels
    @property (nonatomic, readonly) AEMeteringLevels * _Nonnull levels;
    @end
    
  • Yeah that sounds good, @leothiessen =)

  • edited April 2016

    @j_liljedahl said:

    @32BT said:
    Interesting problem:
    I'm attempting to implement submetering for the three different loops in addition to metering for the overall output. For submetering I need to read the top of the stack, but for the overall output I need to read the output buffer. In the module I do not specifically know which to use...

    Obviously, I can set some indexing ivar to tell me which buffer to use, but me thinks this should not be necessary, since the top of the stack should always contain the latest state. In other words: should there be an output buffer? Or should the renderloop always assume that the final output is available at the top of the stack at the end of the renderloop?

    This would of course be cleaner, but complicates stuff for multichannel outputs. For example, outputting each track to a specified hw output.

    However, this could perhaps be solved in a different way than having an output buffer in the context: Instead pushing a buffer with N channels (for an N output interface), and use some bufferstack operation that mixes onto this. It would be pushed first, so it's at the bottom of the stack, and a mechanism to store a ref to this buffer in a local var. (This could also be used for any other kind of mix bus). Then, at the end of the loop, this buffer should again be the top buffer (all else has been mixed down to it and popped off the stack), and it's regarded as the final result of the rendering, used by its renderer (output unit or subrenderer such as varispeed or oversampler etc)

    Hmm... it feels a bit messy, still, having to keep a separate reference and such. I agree that it'd be neater to not have an output buffer list, but just have the output drawn from the stack, either from the buffer at the top, or by mixing all the remaining buffers.

    @j_liljedahl and I already briefly discussed a "latency" parameter that could be associated with each buffer, so that a module at the end could do some alignment on multiple audio streams with different latencies. Perhaps there could also be a "output channel index" parameter that could be assigned to each buffer, and used to determine the output buffers each remaining buffer is mixed into?

  • edited April 2016

    @Michael said:

    @j_liljedahl said:
    However, this could perhaps be solved in a different way than having an output buffer in the context: Instead pushing a buffer with N channels (for an N output interface), and use some bufferstack operation that mixes onto this. It would be pushed first, so it's at the bottom of the stack, and a mechanism to store a ref to this buffer in a local var. (This could also be used for any other kind of mix bus). Then, at the end of the loop, this buffer should again be the top buffer (all else has been mixed down to it and popped off the stack), and it's regarded as the final result of the rendering, used by its renderer (output unit or subrenderer such as varispeed or oversampler etc)

    Hmm... it feels a bit messy, still, having to keep a separate reference and such. I agree that it'd be neater to not have an output buffer list, but just have the output drawn from the stack, either from the buffer at the top, or by mixing all the remaining buffers.

    Well, the point is that you actually don't need to keep a separate reference, unless you want that to explicitly mix to it from various places in the code (without the output buffer being the top buffer). However, such mix to referenced buffer would be very useful also for other cases, like internal mix busses etc.

    The question/issue is the mapping to multi-channel output..

    @j_liljedahl and I already briefly discussed a "latency" parameter that could be associated with each buffer, so that a module at the end could do some alignment on multiple audio streams with different latencies. Perhaps there could also be a "output channel index" parameter that could be assigned to each buffer, and used to determine the output buffers each remaining buffer is mixed into?

    I think there's three alternatives:

    1. context/renderer has an output buffer, and you mix onto it manually
    2. it mixes all remaining buffers on the stack automatically, and uses a new stack-item field "output channel" to decide onto which channel to mix it.
    3. it automatically copies the top buffer only into the output, using all its channels as needed.

    Actually I don't like 2, it's too automatic, and adds an "output channel" field to each stack item that are mostly only used for one of the items. alternative 1 and 3 is actually almost the same, except in 3 the output buffer would not need to be public and accessible for the render code. In 3, you can use the current stack result directly in the simple cases (plain stereo output) but can optionally push your own mix buffer with N channels for more control.

    But I think alternative 1 is just fine, as it already is now. You would only mix onto output ABL in the top render loop. A metering utility could meter ABLs directly instead of getting them off the stack. But 3 is conceptually cleaner. And any transformations needed (mixing, remapping of channels, etc) would be done explicitly in the render code.

  • @j_liljedahl said:
    1. context/renderer has an output buffer, and you mix onto it manually
    2. it mixes all remaining buffers on the stack automatically, and uses a new stack-item field "output channel" to decide onto which channel to mix it.
    3. it automatically copies the top buffer only into the output, using all its channels as needed.

    I was just pondering the idea of pushing and popping contexts...

    So freeflowing (thinking aloud):
    - you push the main context on the contextstack (= current context)
    - the context pushes its associated buffer on the bufferstack
    - modules push temporary buffers on the bufferstack
    - a subcontext pushes its associated buffer on the bufferstack
    - etc

    popping a context may or may not pop the associated buffer, but you'd have to guard for existence: as long as the owning context exist, the buffer also exist and may reside on the stack.

    So you manage subcontext as an array within the containing context on the main-thread using obj-c. Their existence is guaranteed for the lifetime of the containing context, so the buffers are guaranteed to exist.

    This allows you to make rendergraphs, as previously desired by Michael.

  • Proof-of-concept: added oscilloscope view
    https://github.com/32Beat/TAAE2 (develop branch)

  • Hi all, wanted to ping the group to see what the next steps are for supporting the metering module in TAAE2? I integrated the AEMeteringModule from @leothiessen's fork and it works pretty well for my basic needs (monitoring db levels for an AERenderer, 2 channel audio). I did need to remove the free(&levelsStruct); in the dealloc method because it was causing a crash after removing the module from the renderer. I see Michael commented that the dealloc method was not needed earlier in this thread...? Anyways, if there is anything I can do to help move this forward let me know! Seems like the brunt of the work is finished though. Keep me posted!

  • Hey @manderson - neat to see someone use the AEMeteringModule. I have actually just come to the point where I need to start integrating audio metering in my own work. I'm going to take a look at @32BT develop branch and see if I can better understand it this time around :smiley: - preferably I'd like some ready-to-go metering coming out of the module (smoothed, ready for user interface). I can certainly put in another pull request (with free(&levelStruct); removed).

    How do you find the AEMeteringModule to be working out for you? Would you do something differently?

  • I've just submitted a pull request to add the updated version of AEMeteringModule.

  • @leothiessen Other than the issue I was having with the free(&levelStruct) it seems to be working with no issues. I just checked out your PR, thanks for putting in the time and effort on that! I am essentially passing the values from the realtime thread to the main thread and then calling a block to the owning class:

                    // outside renderer block, after it is initialized but before assigning the block
                    __weak typeof(self)weakSelf = self;
                    AEMainThreadEndpoint *meteringEndpoint = [[AEMainThreadEndpoint alloc] initWithHandler:^(const void * _Nullable data, size_t length) {
    
                        AEMeteringLevels *levels = (AEMeteringLevels *)data;
                        float average = AEDSPRatioToDecibels(levels->channels->average);
                        float peak = AEDSPRatioToDecibels(levels->channels->peak);
    
                        if (weakSelf.dbMeter) {
                            weakSelf.dbMeter(average, peak);
                        }
                    }];
    
               // inside renderer block
                __unsafe_unretained AEMeteringModule *metering
                = (__bridge AEMeteringModule *)AEManagedValueGetValue(meteringValue);
    
                if (metering) {
                    AEModuleProcess(metering, context);
                    AEMainThreadEndpointSend(meteringEndpoint, metering.levels, sizeof(AEMeteringLevels));
                }
    
Sign In or Register to comment.