Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optional sample rate conversion to audio::TargetFile and audio::BufferRecorderNode #1869

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Conversation

yvrhdn
Copy link

@yvrhdn yvrhdn commented Jul 9, 2017

As I was discussing with @richardeakin on the forum, I've added the abillty to configure the sample rate of TargetFile.
The sample rate conversion is implemented in audio::TargetFile and exposed in audio::BufferRecorderNode.

I propose the following API changes:

  • TargetFile::create has an additional optional parameter targetSampleRate.
    I placed it before extension since this parameter is (for some reason) not used in the implementation of create.
  • this extra parameter is propagated to TargetFileCoreAudio.
  • I've added a new function TargetFile::getSampleRateNative similar as in Source.
  • BufferRecorderNode::writeToFile has an additional optional parameter destSampleRate which is passed to TargetFile::create.

I've not implemented these changes on MSW since I work on macOS, but instead added an assert firing when sample rate conversion is required.

I feel like my implementation of TargetFile::write is bit hacky, because dsp::Converter 1) processes data in fixed chunk sizes and 2) can't handle frame offsets in buffers. So I first have to copy data into a new buffer (removing the frame offset), convert it using a second buffer before writing it to file.
So could that could be improved imo.

@richardeakin
Copy link
Collaborator

Really excited to see these changes. Just wanted to let you know that I'm onsite for a project and pretty busy for the next week and a half, but looking forward to checking these out when I have a moment.

@richardeakin richardeakin self-requested a review August 8, 2017 18:52
Copy link
Collaborator

@richardeakin richardeakin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay on this, that week and a half project turned into a month.. but overall this looks great.

So as to the part about not implemented on MSW - I believe this is actually the only place where you explicitly need a ci::audio::dsp::Converter, as MSW Media Foundation doesn't do resampling (in our simple usage, anyway). On the other hand, Core Audio's ExtAudioFile does do more advanced stuff like resampling. With SourceFile, I distinguished between implementations within built in samplerate conversion and those without with the virtual method supportsConversion() (

virtual bool supportsConversion() { return false; }
). I imagine you can use ExtAudioFile's built in stuff for conversion on anything Cocoa, and we can use the explicit converter for all other platforms (which is really only MSW atm).

Going to test on my end since I'm on windows..

size_t getSampleRate() const { return mSampleRate; }
//! Returns the true sample rate of the target file. \note Actual input samplerate may differ. \see getSampleRate()
size_t getSampleRateTarget() const { return mSampleRateTarget; }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it's worth naming this getSampleRateNative() to be consistent with SourceFile? I can't say I like one name over the other, though it is slightly ambiguous that the class is called TargetFile and the method has the noun 'target' in in.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me both seem reasonable. I personally didn't find getSampleRateTarget() ambiguous since I thought of it as "get the sample rate of the target [file]". But I can definitely see the consistency argument with SourceFile.

{
if ( ! mSampleRateTarget ) {
mSampleRateTarget = mSampleRate;
} else if ( mSampleRateTarget != mSampleRate) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style nit: else statements should begin on a new line (apologies, this wasn't in the CONTRIBUTING.md, but I'm going to add it as it is something we've all agreed upon AFAIK).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just fixed this with 89ad51f.
I've also removed spaces between if/while and their braces. This isn't mentioned in the CONTRIBUTING.md either but is done very consistently in the code.

auto numDestFrames = size_t( numSourceFrames * (float)getSampleRateTarget() / (float)getSampleRate() );

// copy buffer into temporary buffer to remove frame offset (needed for mConverter->convert)
mConverterSourceBuffer.copyOffset( *buffer, numSourceFrames, 0, currFrame );
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concerning this and your comment in the PR description about this being a bit hacky, I hope that in the future this problem can be alleviated once audio::BufferViews are implemented (see #1701). I think for now a TODO comment linking back to that would suffice. The copy is quite cheap compared to the actual encoding, but obviously eventually we want to do as little extra copying as possible.

@yvrhdn
Copy link
Author

yvrhdn commented Aug 14, 2017

I can see how the supportsConversion() works, I can add this easily. But I'm not familiar with the functionality ExtAudioFile provides and currently don't have to time to figure that out.
Is it okay if I add the supportsConversion() function and leave the implementation within TargetFileCoreAudio to someone else for now?

I'll also try to adapt the getSampleRateTarget to getSampleRateNative the next couple of days.

@yvrhdn
Copy link
Author

yvrhdn commented Aug 16, 2017

This last commit adds supportsConversion() but does not implement it. So what's left for a complete implementation is:

  • implement sample rate conversion in MSW
  • adapt TargetFileCoreAudio to use resampling capabilities of ExtAudioFile

@yvrhdn
Copy link
Author

yvrhdn commented Aug 21, 2017

@richardeakin I don't have experience with resolving conflicts on GitHub. What's the process here? Can I do this since I don't have write access?

Besides a merge conflict in Target.cpp, there also have to be some changes in the constructor of TargetFileOggVorbis. I can make a diff if that's useful.

@richardeakin
Copy link
Collaborator

You should probably locally merge master back into your koenaad:target_sample_rate branch, resolve conflicts, then push it back to your repo - it'll update this PR. You can also rebase and git push --force to your repo / branch if you're comfortable with that, either way works.

I started looking at how to use this on Windows but was only able to put a couple hours towards it. I'm hoping to revisit that soon. Basically my current thought that Windows and Linux will need to use a dsp::Converter, while OS X / iOS can use the conversion feature built into ExtAudioFile, the Core Audio layer doing all the work on those platforms.

@yvrhdn
Copy link
Author

yvrhdn commented Aug 26, 2017

Okay, makes sense. Unfortunately, my macbook broke down this week (which has my development environment) so I will only be able to continue working on this over at least a week I think.

@richardeakin richardeakin mentioned this pull request Aug 27, 2017
@richardeakin
Copy link
Collaborator

No problem, sorry to hear about your macbook. I may continue with trying to get it working on the windows side, but I can replay my mods on top of your branch once it is fixed up.

@yvrhdn
Copy link
Author

yvrhdn commented Sep 4, 2017

Hi @richardeakin, this should now be ready to be merged.

Copy link
Collaborator

@richardeakin richardeakin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking really good. I've synced with this branch and updated the windows side of things in https://github.com/richardeakin/Cinder/tree/target_sample_rate_msw, all seems to be working when testing with the SampleTest (note if you want to try that, open it via AudioTest.msw folder. Do you want to merge that into your branch so it appears on this PR? Alternatively I can merge those changes in separately but it'd be nice and tidy to get the changes all in one shot.

I think there are a couple small things left on the OS X / iOS side of things that we should do so that it uses its internal converter, one of the things I commented on inline. The other is that the file ASBD should use mSampleRateNative.


// Implement to write \a numFrames frames of \a buffer to file. The writing begins at \a frameOffset.
virtual void performWrite( const Buffer *buffer, size_t numFrames, size_t frameOffset ) = 0;
//! Implementations should override and return true if they can provide samplerate conversion. If false (default), a Converter will be used if needed.
virtual bool supportsConversion() { return false; }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think TargetFileCoreAudio still needs to override this and return true.

@yvrhdn
Copy link
Author

yvrhdn commented Oct 5, 2017

Hi, sorry for the delay. I haven't had much time to look at this and won't have much in the future either (started a new job, my first actually).

About the OS X / iOS side: I don't have any experience with these services so I won't be able to help much with this. I'd propose to continue with this (non-optimal) implementation for now and opening an issue as reference for future contributors.

If you would like me to merge something, let me know and I'll help out as much as I can. But I won't be able to provide much development for now.

@richardeakin
Copy link
Collaborator

All good, thanks for your what you've already contributed! I'll try to get to the OS X side the next time I'm on my macbook, which isn't often these days, but I'd like to try to get those changes in there before putting this down and moving on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants