Crash in AEMessageQueue on ios 9

I've recently updated to ios 9.0, and am now getting a crash inside AEMessageQueue.

The crash occurs when calling AEAudioController::addOutputReceiver:forChannelGroup.
Inside AEMessageQueue::performAsynchronousMessageExchangeWithblock there's a call to TPCircularBufferHead which returns NULL in the crashable instance. This returned pointer is then dereferenced and the crash occurs. Looking into TPCircularBufferHead, the vm_remap is always unsuccessful, and after a few retries, NULL is returned. This wasn't an issue in previous version of ios.

Any thoughts on what might be going on?

Thanks!

Comments

  • Can you reproduce this in the sample app, Mike? I'd like to see a little project where I can reproduce this (because I can't on my end!)

  • Yes unfortunately (or fortunately depending on your perspective!) I can't get the sample app to reproduce this issue, so it appears specific to my project.

    I am linking TAAE library into a non-ARC project, perhaps that's causing issues?
    Any thoughts on why vm_remap might be failing?

  • edited October 2015

    I just updated and am getting a similar crash whenever TAAE stops (as a result of updating inputEnabled, etc).

    Rolling back to 1.4.8 for now :O

  • Oh, @Hernan is having that problem too! Are you getting that in the sample app, @warpling? Are you using ARC? This is a question for you too, @Mike: how have you added TAAE to your project? It should be as a sub-project, the same as the sample app (which builds using ARC).

  • I am using ARC and have TAAE added via the pod.
    The above is not the sample app, but I'll try that next!

  • Haven't been able to get the sample app to crash tonight :/

  • Ahh that's a pain. I'm still thinking it could be something writing to some memory it has no business touching and trashing the AEMessageQueue instance. That's going to be hard to track down if it's that...

  • I've had a little luck with my problem.

    I'm actually using the Marmalade SDK for cross platform development. I've switched my ios extension to use ARC, but the problem still persisted. I'm linking TAAE as a library that I've built seperately using xcode project provided. I also tried compiling the files directly into my extension, but that had no effect, understanably. I then noticed that the problem was localised the 64 bit devices running ios 9.0+. I has been previously running 32bit binaries on the 64bit devices, and when I switched to the 64bit binaries, the crash has disappeared. So now I deploy FAT builds containing both 32 and 64 bit version, and everything is groovy.

    I'm not convinced that the problem is fixed however, as the issue seems to be a race condition within vm_remap, and perhaps my new binary layout has simply muted the problem for now. If it resurfaces I'll let you know.

    If I can provide any more info, please ask.

  • Interesting @mike. I tried creating another demo app to follow the same steps my app uses, but alas no crashes. I don't have my 5s on me at the moment, but I'll try to see tonight if it's crashing the same way.

    (Not sure if this helps at all, but I've seen crash occur in stopPolling via AEAudioController'sreinitializeWithChanges as well as stop.)

  • Same crash on an iPhone 5s and 5th gen Touch (32bit A5)

  • edited October 2015

    Hmm, I've just tried the TAAE sample app running on an iPad 3 WiFi and an iPod Touch 5th Gen, and a 32-bit build on an iPad Air 2, and toggled the sample rate switch really fast (which causes the engine to stop, teardown, setup and start again), and I'm not seeing any crashes. Any tips on how to reproduce?

  • edited October 2015

    For what it's worth It happens consistently for me when I call [self.audioController setInputEnabled:YES error:nil] on an already running TAAE that has inputEnabled set to NO. Just spent another 30 trying to make an example that failed and couldn't…

    My TAAE setup is pretty dead simple:

        AudioStreamBasicDescription streamFormat = [AEAudioController nonInterleaved16BitStereoAudioDescription];
        self.audioController = [[AEAudioController alloc] initWithAudioDescription:streamFormat inputEnabled:NO useVoiceProcessing:NO outputEnabled:YES];
        [self.audioController setAudioSessionCategory:AVAudioSessionCategoryAmbient];
        [self.audioController start:&error];
    
  • Update! It appears it only happens (at least in my case) when input is enabled! I can make TAAE crash during the very first initialization if input is enabled (the other params don't seem to matter). That said, I can't get the sample app to crash with the same, so I'm probably down the wrong path entirely. :[

  • If you startup with input disabled then enable input, does it work correctly?

  • Johnny, that's my default use case that usually fails. As I mentioned, it also fails if I start it with input enabled. :/

    I usually start with input disable so I can respect the user's silence switch unless I really need input, in which case TAAE will switch to a category that plays output regardless.

  • edited October 2015

    Ah-hah! Found the problem, enormous thanks to @Hernan, who lent me his source code. If you're using an old version of TPCircularBuffer somewhere in your code, you need to update it, because the latest version includes an extra member in the TPCircularBuffer struct. If you have an older version, there's a danger that it will be used to define the structure, which means that when the new code writes to that new member (a boolean), it'll actually be writing to the byte following the too-short struct. In the case of AEMessageQueue, that member is an NSThread*, which gets set to 0x1, which causes a crash when it's referenced.

    So: delete or update your old TPCircularBuffer instances. I'm trying to find a way to make this situation refuse to compile, for the future (suggestions welcome there; I'm struggling with it so far!)

    Update: I've made a change which will either refuse to compile, or throw an assert on initialization if you have a conflicting TPCircularBuffer version.

  • Michael! You're a genius! I would never have figured that out O_O
    Pod unlocked and updating!

  • Hehe - I was quite chuffed with that discovery. Do let me know how it goes

  • My problem is back, as I suspected it just went into hibernation for a short time. :(

    Thanks for the suggested fix Michael. Unfortunately I don't think it was relevant to my situation as we're always just used the single version of TPCircularBuffer. I updated to 1.52 to confirm and everything does compile fine.

    I've done a little more investigating and it appears that in TPCircularBufferInit, the vm_remap returns KERN_INVALID_ADDRESS, and fails to create the circular buffer. I've searched around online as to why this might be happening, without any luck. This only happens on occasion, but when it does happen, TPCircularBufferInit will never succeed. I modified the AEMessageQueue to continually try, but it always fails.

    Suggestions appreciated!

  • Ah, this might be a zero length allocation. I've just pushed a change that adds an assert, would you mind seeing if/where that's happening?

  • No assert is fired. Checking the length and it seems valid at 8192. I've tried halving the required buffer length each time the init fails, and it continues to fail all the way to an allocation of 1.

  • Although increase the buffer size did seem to do the trick!

    I've now increased AEAudioController::kMessageBufferLength to 32k, and the problem has disappeared again. I can also run 32 bit builds on 64 bit devices as well, so it seems a little more stable. Still worried it could come back at any time.

    Is it safe to alter kMessageBufferLength to this value?

  • Ouch...Yeah, I'm a bit worried about that as well. Doesn't seem like a stable solution, just plugging in arbitrary numbers until the error goes away. But I don't have anything better to suggest; looks like something has gone awry in your app's memory space, but I've not a clue what or why! If that fix works for you, I guess go with it until we learn more =)

  • Thanks Michael. I'll let you know if it starts playing up again!

Sign In or Register to comment.