Performance across different iOS devices

edited June 2013

Hey Michael & the rest of y'all who know way more than I do about all this,

I'm working on a project that triggers audio files on different devices over a local network. TAAE has been the best of the five audio frameworks I've trialed so far, but testing on an iPhone 4S and a 4th-gen iPod touch, I'm seeing an odd situation.

The files begin playing perfectly in sync, but then they drift perceptibly out of sync — as if one device is playing the file slightly faster than the other. After about a minute, the drift is noticeable, and it gets worse from there.

This occurs consistently whether I use AudioUnitFilePlayer (streaming from disk) or AudioFilePlayer (loading into memory), and with compressed (AAC) and uncompressed (AIF) test files.

I can post code samples, but there's nothing complicated happening in the code, and the audio begins perfectly in sync. I was wondering if anyone could suggest ideas or experiments to see what's going on here? Or if this is something inherent to audio on iOS? I assumed that a five-minute audio file would take pretty close to exactly five minutes to play on any device, and that's not the case here.

Even vague thoughts would be much appreciated as it might help me understand what's going on.

Thanks very much!

Comments

  • This isn't a problem with the audio engine or with iOS. Audio is played back at a rate that is controlled by the hardware clock on the device. Each device's clock has tiny variations. There is the nominal sample rate (say 44,100), which is a mathematical ideal, and there is the actual sample rate, which is subject to the limitations of the real world. So the actual sample rate for Device1 may be 44,100.001 and 43,999.999 for Device2. Over time the same audio file triggered at the same time on these two devices will drift relative to one another.

    The only way you could get around this would be to implement drift correction, but that would be quite complex with the networked project you're working on. You'd need to have one device be the master device, and broadcast timestamps from it. All other devices would need to compare this master device timestamp with local timestamps and adjust for drift by processing the audio with a rate correcting algorithm.

  • That's very interesting, thank you!

    Are the differences generally across hardware versions or even between the same hardware? i.e. would you expect two iPhone 4Ss to have different actual sample rates?

    Do you know of any way to return the actual sample rate of a given device?

    An earlier version of this concept using Pure Data via the now defunct RjDj didn't exhibit perceptible drift over a four-minute run time, which is what led me to ask about TAAE-specific things...

  • Current results: iPad 2 and iPhone 4S stay perfectly in sync, but iPod touch 4th gen gains around 200-250ms over a five-minute file. Hmm...

  • edited June 2013

    The other option you have here is to manually adjust playback position at regular intervals, based on a common timestamp. So, if you start at time t, any time later on (say, t'), you can calculate what your playback position should be (t' - t). Convert that to frames ((t' - t) × 44100.0) and you've got your ideal playback position, regardless of the real hardware sample rate.

  • Every device is going to have tiny variations, even two 4Ss. There is no direct Core Audio API to get the actual sample rate, but this isn't going to be a constant anyway. Even on a single device, the rate may fluctuate over time. You can derive the actual rate at any instantaneous point in time however using APIs provided. See the AudioTimeStamp structure. mRateScalar is the ratio of the actual time to the nominal time.

    So yeah, as Michael says, you could use timestamps to do this. Figure out from mRateScalar whether to run ahead or skip back during each callback in order to keep audio flowing at the ideal 44.1 rate on each device. You'd probably want to do some smoothing or else you'll get pops and clicks.

    One thing that might throw a wrench in the works is that each device has slight variations in what "time" is too. This is corrected by the OS every once in a while by checking in with the network time. I don't know how this will affect your audio application. Probably wont' be a big deal.

  • Good luck. Let us know how it goes!

  • Actually, giving this some more thought, there might be just as much variation on what each device thinks "time" is as what it thinks the sample rate is. I don't see any reason why one would be better than the other. And syncing with network time probably doesn't happen very often (and not at all if you're off the network of course). You might need to go with the original solution I mentioned and broadcast timestamps from a master device. Even then this is going to be a pretty coarse-grained correction. It'll keep you in sync generally, but you might hear phasing.

  • ayedee, Michael — thank you so much for the quick, detailed, and creative responses! That's pretty amazing. I'm back to work on it tomorrow and I'll post my results.

    iPhone 4 was almost perfectly in sync over a 5min track as well, which is encouraging.

    I am going to try out Michael's idea, as even an occasional 'shove' back into sync would be good for older devices. One question — is there a TAAE call I should be using for the time (related to the scheduling abilities) or should I use a system time call?

  • jakemoves - If your application would allow it, using the actual sample rate when calculating audio would be the simplest solution. This would effectively be equivalent to specifying 44100 as the sample rate and then compensating for the difference between that and the actual sample rate. I used this method in my app and had no distinguishable drifts after 1 hour run when comparing iPad 2, iPhone 4 and iPod Touch 4th-gen.

    I had also tried drift compensation according to the system time. It worked well too, but turned out to be unnecessary since using actual sample rate worked equally well.

  • Cool! Good to know it's doable.

    Do you mind if I ask how you implemented the compensation & actual sample rate monitoring?

  • Actual sample rate monitoring:

    TAAE requests the specified sample rate according to the AudioStreamBasicDescription you provide. It does so by calling:

    AudioSessionSetProperty(kAudioSessionProperty_PreferredHardwareSampleRate, ...)

    Then you confirm/get actual sample rate granted by hardware:

    AudioSessionGetProperty(kAudioSessionProperty_CurrentHardwareSampleRate, ...)

    TAAE performs this in order to warn you if they are different, but it's not exposed in the API. You have to do this yourself as follows:

        // Fetch sample rate, in case we didn't get quite what we requested
        Float64 achievedSampleRate;
        UInt32 size = sizeof(achievedSampleRate);
        result = AudioSessionGetProperty(kAudioSessionProperty_CurrentHardwareSampleRate, &size, &achievedSampleRate);
        checkResult(result, "AudioSessionGetProperty(kAudioSessionProperty_CurrentHardwareSampleRate)");
        if ( achievedSampleRate != sampleRate ) {
            NSLog(@Hardware sample rate is %f, achievedSampleRate);
        }
    

    Then use this actual sample rate value throughout your app. This means no hard-coded 44100s anywhere.

    By the way, getting the current hardware sample rate must be done after AudioSession is initialised. But this should not be a problem with TAAE AudioSession since it is initialised in

    initWithAudioDescription

    method, which should be called quite early.

    You may want to check out the documentation:
    http://developer.apple.com/library/ios/#documentation/Audio/Conceptual/AudioSessionProgrammingGuide/UsingAudioSessionProperties/UsingAudioSessionProperties.html

    Drift compensation:

    My method was along the lines of what Michael suggested a few posts back. I had compared timestamp with time generated by mach_absolute_time() in order not to be affected by system time adjustments. I suggest you follow this route only if the above method doesn't work for you.

    Let me know if I couldn't be clear enough.

    Cheers.

  • Thank you so much for the detailed note. I'm never actually hardcoding a sample rate (i.e. 44100), just loading and playing files using AudioUnitFilePlayer. So where would I use the actual sample rate?

    (Sorry if I am asking stupid questions, thank you for helping relieve my ignorance!)

  • Jake,

    I'm trying to tackle a similar problem, but I can't seem to get the files to start playing at the same time. I am accounting for the measured latency between devices and clock differences, the problem seems to be in the latency between calling start: on the audio controller and sound actually coming out of the speaker. Here is how I measure the latency:

    • get current date when calling start: on the audio controller
    • on the first timing receiver callback invocation, compare the new current date with the start date

    It can vary anywhere between .07s and .33s depending on how recently audio was playing before calling start. I've tried accounting for the output latency by factoring in the audio controller's outputLatency value, but this is calculated once and cached, so it won't help with the changing values I'm seeing.

    I have also implemented periodic timestamp syncing which helps correct the bad start, but I can't figure out how to make them start in sync like you've described. Any help is much appreciated. Thanks!

Sign In or Register to comment.