In Need of some Conceptual Clarification

I've been reading over the documentation for TAAE for the past two days and must say that it seems to do just about everything that I need! However, getting it to do that is where I'm in need of a bit of help. I'm developing an app that uses video and audio filters for a user's recording pleasure. I'm using the beautiful GPUImage Library to do the video filtering and it works wonders, and now I want to implement TAAE for audio filtering. GPUImage takes control of the mic while it's recording and also processes the audio and writes it out to the video file. What I'm expecting to do is add the audio processing inside of the GPUImage audio processing code. I'll put that code here for reference:

- (void)processAudioBuffer:(CMSampleBufferRef)audioBuffer;
{
    if (!isRecording)
    {
        return;
    }
    
//    if (_hasAudioTrack && CMTIME_IS_VALID(startTime))
    if (_hasAudioTrack)
    {
        CFRetain(audioBuffer);

        CMTime currentSampleTime = CMSampleBufferGetOutputPresentationTimeStamp(audioBuffer);
        
        if (CMTIME_IS_INVALID(startTime))
        {
            runSynchronouslyOnContextQueue(_movieWriterContext, ^{
                if ((audioInputReadyCallback == NULL) && (assetWriter.status != AVAssetWriterStatusWriting))
                {
                    [assetWriter startWriting];
                }
                [assetWriter startSessionAtSourceTime:currentSampleTime];
                startTime = currentSampleTime;
            });
        }

        if (!assetWriterAudioInput.readyForMoreMediaData && _encodingLiveVideo)
        {
            NSLog(@1: Had to drop an audio frame: %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, currentSampleTime)));
            if (_shouldInvalidateAudioSampleWhenDone)
            {
                CMSampleBufferInvalidate(audioBuffer);
            }
            CFRelease(audioBuffer);
            return;
        }

        previousAudioTime = currentSampleTime;
        
        //if the consumer wants to do something with the audio samples before writing, let him.
        if (self.audioProcessingCallback) {
            //need to introspect into the opaque CMBlockBuffer structure to find its raw sample buffers.
            CMBlockBufferRef buffer = CMSampleBufferGetDataBuffer(audioBuffer);
            CMItemCount numSamplesInBuffer = CMSampleBufferGetNumSamples(audioBuffer);
            AudioBufferList audioBufferList;
            
            CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(audioBuffer,
                                                                    NULL,
                                                                    &audioBufferList,
                                                                    sizeof(audioBufferList),
                                                                    NULL,
                                                                    NULL,
                                                                    kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
                                                                    &buffer
                                                                    );
            //passing a live pointer to the audio buffers, try to process them in-place or we might have syncing issues.
            for (int bufferCount=0; bufferCount < audioBufferList.mNumberBuffers; bufferCount++) {
                SInt16 *samples = (SInt16 *)audioBufferList.mBuffers[bufferCount].mData;
                self.audioProcessingCallback(&samples, numSamplesInBuffer);
            }
        }
        
//        NSLog(@Recorded audio sample time: %lld, %d, %lld, currentSampleTime.value, currentSampleTime.timescale, currentSampleTime.epoch);
        void(^write)() = ^() {
            while( ! assetWriterAudioInput.readyForMoreMediaData && ! _encodingLiveVideo && ! audioEncodingIsFinished ) {
                NSDate *maxDate = [NSDate dateWithTimeIntervalSinceNow:0.5];
                //NSLog(@audio waiting...);
                [[NSRunLoop currentRunLoop] runUntilDate:maxDate];
            }
            if (!assetWriterAudioInput.readyForMoreMediaData)
            {
                NSLog(@2: Had to drop an audio frame %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, currentSampleTime)));
            }
            else if(assetWriter.status == AVAssetWriterStatusWriting)
            {
                if (![assetWriterAudioInput appendSampleBuffer:audioBuffer])
                    NSLog(@Problem appending audio buffer at time: %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, currentSampleTime)));
            }
            else
            {
                //NSLog(@Wrote an audio frame %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, currentSampleTime)));
            }

            if (_shouldInvalidateAudioSampleWhenDone)
            {
                CMSampleBufferInvalidate(audioBuffer);
            }
            CFRelease(audioBuffer);
        };
//        runAsynchronouslyOnContextQueue(_movieWriterContext, write);
        if( _encodingLiveVideo )

        {
            runAsynchronouslyOnContextQueue(_movieWriterContext, write);
        }
        else
        {
            write();
        }
    }
}

From what I've understood from the TAAE documentation, I must create a channel with an audio receiver that captures the audio from the above code, then use a filter on that channel, and then (this is where I'm very confused) get that filtered audio back to GPUImage for writing to the video file. I'm guessing that I'm missing something here, if someone could point it out I'd be very appreciative!

Comments

  • Hi @tnev,

    Hmm, TAAE isn't really an audio filter, which seems like what you need here; it's an audio engine, designed for live audio. Theoretically you could manually drive TAAE from a callback like that, but it would need a bit of development to make it work that way.

  • edited October 2015

    Ah, okay, thanks @Michael. If you have time to make any recommendations for something that might fit my needs better I would be thankful, if not I very much appreciate your feedback! Im sure I'll have future projects that will need your excellent library.

    Just to clarify, since I realized my first post didn't explain what I needed very well and the use of the word "filter" is probably misguiding in the context I choose, I want to take the audio from the microphone and add pitch shifting or distortion effects of some sort (supposed to make the user sound like a demon). I saw a post from someone that was able to use TAAE with GPUImage but they did all the audio recording in TAAE and then wrote it to the GPUImage video file. I'm programming for iOS.

  • I'm afraid I don't actually know! I use TAAE myself, so I don't have a great deal of experience with alternatives.

    That technique should work, doing the recording from within TAAE and mixing it in with the video from that point.

Sign In or Register to comment.