Add optional sample rate conversion to audio::TargetFile and audio::BufferRecorderNode #1869

yvrhdn · 2017-07-09T23:14:45Z

As I was discussing with @richardeakin on the forum, I've added the abillty to configure the sample rate of TargetFile.
The sample rate conversion is implemented in audio::TargetFile and exposed in audio::BufferRecorderNode.

I propose the following API changes:

TargetFile::create has an additional optional parameter targetSampleRate.
I placed it before extension since this parameter is (for some reason) not used in the implementation of create.
this extra parameter is propagated to TargetFileCoreAudio.
I've added a new function TargetFile::getSampleRateNative similar as in Source.
BufferRecorderNode::writeToFile has an additional optional parameter destSampleRate which is passed to TargetFile::create.

I've not implemented these changes on MSW since I work on macOS, but instead added an assert firing when sample rate conversion is required.

I feel like my implementation of TargetFile::write is bit hacky, because dsp::Converter 1) processes data in fixed chunk sizes and 2) can't handle frame offsets in buffers. So I first have to copy data into a new buffer (removing the frame offset), convert it using a second buffer before writing it to file.
So could that could be improved imo.

…a only for now)

richardeakin · 2017-07-20T19:54:28Z

Really excited to see these changes. Just wanted to let you know that I'm onsite for a project and pretty busy for the next week and a half, but looking forward to checking these out when I have a moment.

richardeakin

Sorry for the delay on this, that week and a half project turned into a month.. but overall this looks great.

So as to the part about not implemented on MSW - I believe this is actually the only place where you explicitly need a ci::audio::dsp::Converter, as MSW Media Foundation doesn't do resampling (in our simple usage, anyway). On the other hand, Core Audio's ExtAudioFile does do more advanced stuff like resampling. With SourceFile, I distinguished between implementations within built in samplerate conversion and those without with the virtual method supportsConversion() (

Cinder/include/cinder/audio/Source.h

Line 70 in 649732c

virtual bool supportsConversion() { return false; }

). I imagine you can use ExtAudioFile's built in stuff for conversion on anything Cocoa, and we can use the explicit converter for all other platforms (which is really only MSW atm).

Going to test on my end since I'm on windows..

richardeakin · 2017-08-08T19:24:02Z

include/cinder/audio/Target.h

 	size_t getSampleRate() const	{ return mSampleRate; }
+	//! Returns the true sample rate of the target file. \note Actual input samplerate may differ. \see getSampleRate()
+	size_t getSampleRateTarget() const	{ return mSampleRateTarget; }


Do you think it's worth naming this getSampleRateNative() to be consistent with SourceFile? I can't say I like one name over the other, though it is slightly ambiguous that the class is called TargetFile and the method has the noun 'target' in in.

For me both seem reasonable. I personally didn't find getSampleRateTarget() ambiguous since I thought of it as "get the sample rate of the target [file]". But I can definitely see the consistency argument with SourceFile.

richardeakin · 2017-08-08T19:31:15Z

src/cinder/audio/Target.cpp

+{
+	if ( ! mSampleRateTarget ) {
+		mSampleRateTarget = mSampleRate;
+	} else if ( mSampleRateTarget != mSampleRate) {


Style nit: else statements should begin on a new line (apologies, this wasn't in the CONTRIBUTING.md, but I'm going to add it as it is something we've all agreed upon AFAIK).

I've just fixed this with 89ad51f.
I've also removed spaces between if/while and their braces. This isn't mentioned in the CONTRIBUTING.md either but is done very consistently in the code.

richardeakin · 2017-08-08T19:39:26Z

src/cinder/audio/Target.cpp

+			auto numDestFrames = size_t( numSourceFrames * (float)getSampleRateTarget() / (float)getSampleRate() );
+
+			// copy buffer into temporary buffer to remove frame offset (needed for mConverter->convert)
+			mConverterSourceBuffer.copyOffset( *buffer, numSourceFrames, 0, currFrame );


Concerning this and your comment in the PR description about this being a bit hacky, I hope that in the future this problem can be alleviated once audio::BufferViews are implemented (see #1701). I think for now a TODO comment linking back to that would suffice. The copy is quite cheap compared to the actual encoding, but obviously eventually we want to do as little extra copying as possible.

yvrhdn · 2017-08-14T14:06:38Z

I can see how the supportsConversion() works, I can add this easily. But I'm not familiar with the functionality ExtAudioFile provides and currently don't have to time to figure that out.
Is it okay if I add the supportsConversion() function and leave the implementation within TargetFileCoreAudio to someone else for now?

I'll also try to adapt the getSampleRateTarget to getSampleRateNative the next couple of days.

yvrhdn · 2017-08-16T22:44:58Z

This last commit adds supportsConversion() but does not implement it. So what's left for a complete implementation is:

implement sample rate conversion in MSW
adapt TargetFileCoreAudio to use resampling capabilities of ExtAudioFile

yvrhdn · 2017-08-21T21:44:12Z

@richardeakin I don't have experience with resolving conflicts on GitHub. What's the process here? Can I do this since I don't have write access?

Besides a merge conflict in Target.cpp, there also have to be some changes in the constructor of TargetFileOggVorbis. I can make a diff if that's useful.

richardeakin · 2017-08-25T18:17:42Z

You should probably locally merge master back into your koenaad:target_sample_rate branch, resolve conflicts, then push it back to your repo - it'll update this PR. You can also rebase and git push --force to your repo / branch if you're comfortable with that, either way works.

I started looking at how to use this on Windows but was only able to put a couple hours towards it. I'm hoping to revisit that soon. Basically my current thought that Windows and Linux will need to use a dsp::Converter, while OS X / iOS can use the conversion feature built into ExtAudioFile, the Core Audio layer doing all the work on those platforms.

yvrhdn · 2017-08-26T23:04:46Z

Okay, makes sense. Unfortunately, my macbook broke down this week (which has my development environment) so I will only be able to continue working on this over at least a week I think.

richardeakin · 2017-08-27T18:00:58Z

No problem, sorry to hear about your macbook. I may continue with trying to get it working on the windows side, but I can replay my mods on top of your branch once it is fixed up.

yvrhdn · 2017-09-04T11:50:36Z

Hi @richardeakin, this should now be ready to be merged.

richardeakin

Looking really good. I've synced with this branch and updated the windows side of things in https://github.com/richardeakin/Cinder/tree/target_sample_rate_msw, all seems to be working when testing with the SampleTest (note if you want to try that, open it via AudioTest.msw folder. Do you want to merge that into your branch so it appears on this PR? Alternatively I can merge those changes in separately but it'd be nice and tidy to get the changes all in one shot.

I think there are a couple small things left on the OS X / iOS side of things that we should do so that it uses its internal converter, one of the things I commented on inline. The other is that the file ASBD should use mSampleRateNative.

richardeakin · 2017-09-08T18:41:36Z

include/cinder/audio/Target.h


 	// Implement to write \a numFrames frames of \a buffer to file. The writing begins at \a frameOffset.
 	virtual void performWrite( const Buffer *buffer, size_t numFrames, size_t frameOffset ) = 0;
+	//! Implementations should override and return true if they can provide samplerate conversion.  If false (default), a Converter will be used if needed.
+	virtual bool supportsConversion()	{ return false; }


I think TargetFileCoreAudio still needs to override this and return true.

yvrhdn · 2017-10-05T21:09:26Z

Hi, sorry for the delay. I haven't had much time to look at this and won't have much in the future either (started a new job, my first actually).

About the OS X / iOS side: I don't have any experience with these services so I won't be able to help much with this. I'd propose to continue with this (non-optimal) implementation for now and opening an issue as reference for future contributors.

If you would like me to merge something, let me know and I'll help out as much as I can. But I won't be able to provide much development for now.

richardeakin · 2017-10-06T01:26:20Z

All good, thanks for your what you've already contributed! I'll try to get to the OS X side the next time I'm on my macbook, which isn't often these days, but I'd like to try to get those changes in there before putting this down and moving on.

Add sample rate conversion to TargetFile and BufferRecorderNode (Coco…

7d8a439

…a only for now)

richardeakin added audio feature labels Jul 10, 2017

yvrhdn mentioned this pull request Jul 10, 2017

Added implementation of TargetFileOggVorbis #1870

Merged

Make default destructor TargetFile more explicit

af4b0e6

richardeakin self-requested a review August 8, 2017 18:52

richardeakin reviewed Aug 8, 2017

View reviewed changes

richardeakin mentioned this pull request Aug 8, 2017

Added guideline about placing else statements on a new line. #1889

Merged

Adjust to style guide

89ad51f

Koenraad Verheyden added 2 commits August 17, 2017 00:15

Target: change getSampleRateTarget() to getSampleRateNative()

8202cda

Target: add supportsConversion()

1d1bba6

Target: forgot to change sampleRateNative on MSW

a9ea7ed

richardeakin mentioned this pull request Aug 27, 2017

Unused parameters #1629

Closed

Merge remote-tracking branch 'base/master' into target_sample_rate

8f0036e

richardeakin reviewed Sep 8, 2017

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add optional sample rate conversion to audio::TargetFile and audio::BufferRecorderNode #1869

Add optional sample rate conversion to audio::TargetFile and audio::BufferRecorderNode #1869

yvrhdn commented Jul 9, 2017

richardeakin commented Jul 20, 2017

richardeakin left a comment

richardeakin Aug 8, 2017

yvrhdn Aug 8, 2017

richardeakin Aug 8, 2017

yvrhdn Aug 8, 2017

richardeakin Aug 8, 2017

yvrhdn commented Aug 14, 2017

yvrhdn commented Aug 16, 2017

yvrhdn commented Aug 21, 2017

richardeakin commented Aug 25, 2017

yvrhdn commented Aug 26, 2017

richardeakin commented Aug 27, 2017

yvrhdn commented Sep 4, 2017

richardeakin left a comment

richardeakin Sep 8, 2017

yvrhdn commented Oct 5, 2017

richardeakin commented Oct 6, 2017

Add optional sample rate conversion to audio::TargetFile and audio::BufferRecorderNode #1869

Are you sure you want to change the base?

Add optional sample rate conversion to audio::TargetFile and audio::BufferRecorderNode #1869

Conversation

yvrhdn commented Jul 9, 2017

richardeakin commented Jul 20, 2017

richardeakin left a comment

Choose a reason for hiding this comment

richardeakin Aug 8, 2017

Choose a reason for hiding this comment

yvrhdn Aug 8, 2017

Choose a reason for hiding this comment

richardeakin Aug 8, 2017

Choose a reason for hiding this comment

yvrhdn Aug 8, 2017

Choose a reason for hiding this comment

richardeakin Aug 8, 2017

Choose a reason for hiding this comment

yvrhdn commented Aug 14, 2017

yvrhdn commented Aug 16, 2017

yvrhdn commented Aug 21, 2017

richardeakin commented Aug 25, 2017

yvrhdn commented Aug 26, 2017

richardeakin commented Aug 27, 2017

yvrhdn commented Sep 4, 2017

richardeakin left a comment

Choose a reason for hiding this comment

richardeakin Sep 8, 2017

Choose a reason for hiding this comment

yvrhdn commented Oct 5, 2017

richardeakin commented Oct 6, 2017