My big news from the last two months is that I’ve released a new Pluralsight course – Creating and Selling a Digital Product. I’ve also managed to progress a bit towards the next release of NAudio. Here’s links to the new posts I’ve done in the last few months over at my new blog.
NAudio
Miscellaneous
In this post I am going to explain how the NAudio MediaFoundationEncoder class can be used to convert WAV files into other formats such as WMA, AAC and MP3. And to do so, I'll walk you through a real-world example of some code I created recently that uses it.
The application is my new Skype Voice Changer utility, and I wanted to allow users to save their Skype conversations in a variety of different formats. The WavFormat that Skype uses is 16kHz 16 bit mono PCM, and I capture the audio directly in this format, before converting it to the target format when the call finishes.
The input to the MediaFoundationEncoder doesn't actually have to be a WAV file. It can be any IWaveProvider, so there is no need to create a temporary WAV file before the encoding takes place.
The first step is to make sure Media Foundation is initialised. This requires a call to MediaFoundation.Startup(). You only need to do this once in your application, but it doesn’t matter if you call it more than once. Note that Media Foundation is only supported on Windows Vista and above. This means that if you need to support Windows XP, you will not be able to use Media Foundation.
Since I planned to make use of whatever encoders are available on the user’s machine, I don't need to ship any codecs with my application. However, not all codecs are present on all versions of Windows. The Windows Media Audio (and the Windows Media Voice) codec, are unsurprisingly present on all the desktop editions of Windows from Vista and above. Windows 7 introduced an AAC encoder, and it was only with Windows 8 that we finally got an MP3 encoder (although MP3 decoding has been present in Windows for a long time). There are rumours that a FLAC encoder will be present in Windows 10.
For server versions of Windows, the story is a bit more complicated. Basically, you may find you have to install the "Desktop Experience" before you have any codecs available.
But the best way to find out whether the codec you want is available is simply to ask the Media Foundation APIs whether there are any encoders that can target your desired format for the given input format.
MediaFoundationEncoder includes a useful helper function called SelectMediaType which can help you do this. You pass in the MediaSubtype (basically a GUID indicating whether you want AAC, WMA, MP3 etc), the input PCM format, and a desired bitrate. NAudio will return the "MediaType" that most closely matches your bitrate. This is because many of these codecs offer you a choice of bitrates so you can choose your own trade-off between file size and audio quality. For the lowest bitrate available, just pass in 0. For the highest bitrate, pass in a suitably large number.
So for example, if I wanted to see if I can encode to WMA, I would pass in an audio subtype of WMAudioV8 (this selects the right encoder), a WaveFormat that matches my input format (this is important as it includes what sample rate my input audio is at - encoders don't always support all sample rates), and my desired bitrate. I passed in 16kbps to get a nice compact file size.
var mediaType = MediaFoundationEncoder.SelectMediaType(
AudioSubtypes.MFAudioFormat_WMAudioV8,
new WaveFormat(16000, 1),
16000);
if (mediaType != null) // we can encode…
What about MP3 and AAC? Well you might think that the code would be the same. Just pass in MFAudioFormat_MP3 or MFAudioFormat_AAC as the first parameter. The trouble is, if we do this, we get no media type returned, even on Windows 8 which has both an MP3 encoder and an AAC encoder. Why is this? Well it's because the MP3 and AAC encoders supplied with Windows don't support 16kHz as an input sample rate. So we will need to upsample to 44.1kHz before passing it into the encoder. So now let's ask Windows if there is an MP3 encoder available that can encode mono 44.1kHz audio, and just request the lowest bitrate available:
mediaType = MediaFoundationEncoder.SelectMediaType(
AudioSubtypes.MFAudioFormat_MP3,
new WaveFormat(44100,1),
0);
Now (on Windows 8 at least) we do get back a media type, and it has a bitrate of 48kbps. The same applies to AAC - we need to upsample to 44.1kHz first, and the AAC encoder provided with Windows 7 and above has a minimum bitrate of 96kbps.
So, assuming that we've successfully got a MediaType, how do we go about the encoding? Well thankfully, that's the easy bit. So for example, if we had selected a WMA media type, we could encode to a file like this:
using (var enc = new MediaFoundationEncoder(mediaType))
{
enc.Encode("output.wma"), myWaveProvider)
}
In fact, to make things even simpler, MediaFoundationEncoder includes some helper methods for encoding to WMA, AAC and MP3 in a single line. You specify the wave provider, the output filename and the desired bitrate:
MediaFoundationEncoder.EncodeToMp3(myWaveProvider,
"output.mp3", 48000);
But of course the bit I haven't explained is how to set up the input stream to the encoder. This will need to be PCM (or IEEE float), and as we mentioned, it should be at a sample rate that the encoder supports. Here's an example of encoding a WAV file to MP3, but remember that the input WAV file will need to be 44.1kHz or 48kHz for this to work.
using (var reader = new WaveFileReader("input.wav"))
{
MediaFoundationEncoder.EncodeToMp3(reader,
"output.mp3", 48000);
}
But that was a trivial example. In a real world example, such as my Skype Voice Changer application, we have a more complicated setup. First, we open the inbound and outbound recording files with WaveFileReader. Then we mix them together using a MixingSampleProvider. Then, since I limit unregistered users to 30 seconds of recording, we optionally need to truncate the length of the file (I do this with the OffsetSampleProvider, and using the Take property). Then, if they selected MP3 or AAC we need to resample up to 44.1kHz. Since we’re already working with Media Foundation, we’ll use the MediaFoundationResampler for this. And finally, I go back down to 16 bit before encoding using a SampleToWaveProvider16 (although this is not strictly necessary for most Media Foundation encoders).
// open the separate recordings
var incoming = new WaveFileReader("incoming.wav");
var outgoing = new WaveFileReader("outgoing.wav");
// create a mixer (for 16kHz mono)
var mixer = new MixingSampleProvider(
WaveFormat.CreateIeeeFloatWaveFormat(16000,1));
// add the inputs - they will automatically be turned into ISampleProviders
mixer.AddMixerInput(incoming);
mixer.AddMixerInput(outgoing);
// optionally truncate to 30 second for unlicensed users
var truncated = truncateAudio ?
new OffsetSampleProvider(mixer)
{ Take = TimeSpan.FromSeconds(30) } :
(ISampleProvider) mixer;
// go back down to 16 bit PCM
var converted16Bit = new SampleToWaveProvider16(truncated);
// now for MP3, we need to upsample to 44.1kHz. Use MediaFoundationResampler
using (var resampled = new MediaFoundationResampler(
converted16Bit, new WaveFormat(44100, 1)))
{
var desiredBitRate = 0; // ask for lowest available bitrate
MediaFoundationEncoder.EncodeToMp3(resampled,
"mixed.mp3", desiredBitRate);
}
Hopefully that gives you a feel for the power of chaining together IWaveProvider’s and ISampleProvider’s in NAudio to construct complex and interesting signal chains. You should now be able to encode your audio with any Media Foundation encoder present on the user’s system.
One question you may have is "can I encode to a stream"? Unfortunately, this is a little tricky to do, since NAudio takes advantage of various "sink writers" that Media Foundation provides, which know how to correctly create various audio container file formats such as WMA, MP3 and AAC. It means that the MediaFoundationEncoder class for simplicity only offers encoding to file. To encode to a stream, you'd need to work at a lower level with Media Foundation transforms directly, which is quite a complicated and involved process. Hopefully this is something we can add support for in a future NAudio.
On a recent episode of .NET Rocks (LINK), Carl Franklin mentioned that he had used NAudio to create an application to mix together audio loops, as part of his “Music to Code By” Kickstarter. He had four loops, for drums, bass, and guitar, and the application allows the volumes to be adjusted individually. He made a code sample of his application available for download here.
This is quite simple to set up with NAudio. To perform the looping part, Carl made use of a LoopStream (using a technique I describe here). The key to looping is simply in the Read method, to read from your source, and if you reach the end (your source returns 0 or fewer samples than requested), reposition to the start and keep reading. This means you have a WaveStream that will never end.
Here's the code for a LoopStream that a WaveFileReader can be passed into:
/// <summary>
/// Stream for looping playback
/// </summary>
public class LoopStream : WaveStream
{
WaveStream sourceStream;
/// <summary>
/// Creates a new Loop stream
/// </summary>
/// <param name="sourceStream">The stream to read from. Note: the Read method of this stream should return 0 when it reaches the end
/// or else we will not loop to the start again.</param>
public LoopStream(WaveStream sourceStream)
{
this.sourceStream = sourceStream;
this.EnableLooping = true;
}
/// <summary>
/// Use this to turn looping on or off
/// </summary>
public bool EnableLooping { get; set; }
/// <summary>
/// Return source stream's wave format
/// </summary>
public override WaveFormat WaveFormat
{
get { return sourceStream.WaveFormat; }
}
/// <summary>
/// LoopStream simply returns
/// </summary>
public override long Length
{
get { return sourceStream.Length; }
}
/// <summary>
/// LoopStream simply passes on positioning to source stream
/// </summary>
public override long Position
{
get { return sourceStream.Position; }
set { sourceStream.Position = value; }
}
public override int Read(byte[] buffer, int offset, int count)
{
int totalBytesRead = 0;
while (totalBytesRead < count)
{
int bytesRead = sourceStream.Read(buffer, offset + totalBytesRead, count - totalBytesRead);
if (bytesRead == 0)
{
if (sourceStream.Position == 0 || !EnableLooping)
{
// something wrong with the source stream
break;
}
// loop
sourceStream.Position = 0;
}
totalBytesRead += bytesRead;
}
return totalBytesRead;
}
}
For mixing, the approach Carl took was simply to create four instances of DirectSoundOut and start them playing together. To allow adjusting the volumes of each channel he passed each LoopStream into a WaveChannel32, which converts to 32 bit floating point, and has a Volume property (1.0 is full volume). To ensure that the four parts remained in sync, when you deselect a part, it doesn't actually stop it playing - instead it sets its volume to 0.
This approach to synchronization works surprisingly well, but it is not actually guaranteed to keep the four parts synchronized. Over time, they could drift. So a better approach is to use a single output device, and feed each of the four WaveChannels into a mixer. Here’s an example block diagram showing a modified signal chain with two inputs feeding into a single mixer:
---------- ---------- -----------
| Wave | | Loop | | Wave |
| File |-->| Stream |-->| Channel |---
| Reader | | | | 32 | | ------------
---------- ---------- ----------- --->| Mixing |
| Wave |
---------- ---------- ----------- --->| Provider |
| Wave | | Loop | | Wave | | | 32 |
| File |-->| Stream |-->| Channel |--- ------------
| Reader | | | | 32 |
---------- ---------- -----------
NAudio has a number of options available for mixing. The best is probably MixingSampleProvider, but for Carl's project, it was easier to use MixingWaveProvider32, since he's not making use of the ISampleProvider interface. This allows you to mix together any WaveProviders that are already in 32 bit floating point format.
MixingWaveProvider32 requires that you specify its inputs up front. So here, we could connect each of our inputs, and then start playing. With this simple change, Carl's mixing application is now guaranteed to not go out of sync. This is the recommended way to mix multiple sounds with NAudio.
Here's the code that sets up the mixer (Carl has a class called WavePlayer encapsulating the WaveFileReader, LoopStream and WaveChannel32, allowing you to access the WaveChannel32 with the Channel property):
foreach (string file in files)
{
Clips.Add(new WavePlayer(file));
}
var mixer = new MixingWaveProvider32(Clips.Select(c => c.Channel));
audioOutput.Init(mixer);
You can download my modified version of Carl's application here.
The only caveat is that mixers require all their inputs to be in the same format. For this application, this isn’t a problem, but if you want to mix together sounds of arbitrary formats, you'd need to convert them all to a common format. This is something I cover in my NAudio Pluralsight course if you’re interested in finding out more about how to do this.
There are two extensions to the standard WAV file format which you may sometimes want to make use of. The first is the RF64 extension, which overcomes the inherent limitation that WAV files cannot be larger than 4GB. The second is the Broadcast Wave Format (BWF) which builds on the existing WAV format and specifies various extra chunks containing metadata.
In this post, I’ll explain how you can make a class to create Broadcast Wave Files using NAudio that supports large file sizes using the RF64 extension, and includes the “bext” chunk from the BWF specification.
First of all, a WAV file starts with the byte sequence ‘RIFF’ and then has a four byte size value, which is the number of bytes following in the entire file. However, for large files, instead of ‘RIFF’, ‘RF64’ is used, and the following four byte integer for the RIFF size is then ignored (it should be set to -1).
Then we have the ‘WAVE’ identifier (another 4 bytes), and following that in a normal WAV file we would usually expect the format chunk (with the ‘fmt ‘ 4 byte identifier). But to support RF64, we add a “JUNK” chunk. This is of size 28 bytes, and initially is all set to zeroes. If the overall size of the entire WAV file grows to over 4GB, then we will turn this “JUNK” chunk into a ‘ds64’ chunk. If the overall file size does not grow beyond 4GB, then the junk chunk is simply left in place, and media players will just ignore it.
A ds64 chunk consists of three 8 byte integers and a four byte integer. These are the RIFF size, which is the size of the entire file minus 8 bytes, the data size, which is the number of bytes of sample data in the ‘data’ chunk and the ‘sampleCount’ which is the number of samples. The sample count is optional really, as it corresponds to the sample count found in the ‘fact’ chunk of a standard WAV file. This chunk is usually only present for non-PCM audio formats as it is trivial to calculate the sample count for PCM from the byte count. Finally, a ds64 chunk can have a table containing the sizes of any other huge chunks, but usually this would be unused since it is likely only the ‘data’ chunk that will grow larger than 4GB. So the final four bytes of a typical ds64 chunk are 0s, indicating no table entries.
Following the ds64 chunk, we have the bext chunk from the BWF specification. This has space for a textual description of the file as well as timestamps, and newer versions of the bext chunk also allow you to specify various bits of loudness information. The algorithms for calculating this are hard to track down, so I tend to use version 1 of bext and ignore them.
Then we have the standard ‘fmt ‘ chunk, which works just the same way it does in a standard WAV file, containing a WAVEFORMATEX structure with information about sample rate, bit depth, encoding and number of channels. RF64 files are almost always either PCM or IEEE floating point samples, since it is only with uncompressed audio that you typically end up creating files larger than 4GB.
Finally, we have the ‘data’ chunk, containing the actual audio data. Again this is used in exactly the same way as it is in a regular WAV file, except that the chunk data length only needs to be filled in the file is less than 4GB. If it is a RF64 file, the four byte length for the data chunk is ignored (set it to –1), and the size from the ds64 chunk is used instead.
Here’s a simple implementation of a BWF writer class, that creates BWF files with a simple bext chunk and turns them into RF64 files if necessary. I plan to clean this code up a little and import it into NAudio in the future (either as its own class or upgrade WaveFileWriter to support RF64 and BWF – let me know your preference in the comments).
NAudio provides a number of different mechanisms for resampling audio, but the resampler classes all assume you are doing “output driven” resampling. This is where you know how much resampled audio you want, and you “pull” the necessary amount of input audio through. This is fine for playing back audio, or for resampling existing audio files, but there are cases when you want to do “input driven” resampling. For example, if you are receiving blocks of audio from the network or soundcard, and want to resample just that block of audio before sending it on somewhere else, then input driven is what you want.
All of NAudio’s resamplers (ACM, Media Foundation and WDL) can be used in input driven mode, but in this post I’ll focus on using AcmStream, since it is the most widely available, and doesn’t require floating point samples.
We start by opening the ACM resampler, specifying the desired input and output formats. Here, we open an ACM stream that can convert mono 16 bit 44.1kHz to mono 16 bit 8kHz. (Note that the ACM resampler won’t change bit depths or channel counts at the same time – so the WaveFormats should differ by sample rate only).
var resampleStream = new AcmStream(new WaveFormat(44100, 16, 1), new WaveFormat(8000, 16, 1));
Note that you shouldn’t recreate the AcmStream for each buffer you receive. Codecs like to maintain internal state, so you’ll get the best results if you open it once, and then keep passing audio through it.
Now with the ACM stream, it already has a source and destination buffer allocated that it will use for all conversions. So we need to copy audio into the source buffer, Convert it, then read it out of the destination buffer. The source buffer is a limited size, so if you have more input data than can fit into the source buffer, you’ll need to perform this conversion in multiple stages. Here we’ll get some input data as a byte array (e.g. received from the network or a soundcard), and copy it into the source buffer:
byte[] source = GetInputData(); Buffer.BlockCopy(source, 0, resampleStream.SourceBuffer, 0, source.Length);
To convert the audio in the source buffer, call the Convert function. This will return the number of bytes available in the destination buffer for you to read out. It will also tell you how many source bytes were used. This should be all of them. If they haven’t been used, it probably means you are trying to convert too many at once. Pass the audio through in smaller block sizes, or copy the “leftovers” out and put them through again next time (this is how NAudio’s WaveFormatConversionStream works internally). Here’s the code to convert what you’ve copied into the source buffer:
int sourceBytesConverted = 0;
var convertedBytes = resampleStream.Convert(source.Length, out sourceBytesConverted);
if (sourceBytesConverted != source.Length)
{
Console.WriteLine("We didn't convert everything {0} bytes in, {1} bytes converted");
}
Now we can copy our converted audio out of the ACM stream’s destination buffer:
var converted = new byte[convertedBytes]; Buffer.BlockCopy(resampleStream.DestBuffer, 0, converted, 0, convertedBytes);
And that’s all you have to do (apart from remember to Dispose your AcmStream when you’re done with it). It is a little more convoluted than output driven resampling, but not too difficult once you know what you’re doing.
I’ll try to do some future posts showing how to do input driven resampling with the Media Foundation resampler, and the WDL resampler.
Every now and then you’ll find you need to resample audio with NAudio. For example, to mix files together of different sample rates, you need to get them all to a common sample rate first. Or if you’re playing audio through an API like WASAPI, which doesn’t resample for you, you need to do this yourself (actually WasapiOut in NAudio does include a resampling step on your behalf if needed).
There are also some gotchas you need to be aware of when resampling. In particular there is the danger of “aliasing”. I explain what this is in my Pluralsight “Digital Audio Fundamentals” course. The main takeaway is that if you lower the sample rate, you really ought to use a low pass filter first, to get rid of high frequencies that cannot correctly.
Probably the most powerful resampler available with NAudio is the MediaFoundationResampler. This is not available for XP users, but desktop versions of Windows from Vista onwards include it. If you are using a Windows Server, you’ll need to make sure the desktop experience is installed. It has a customisable quality level (60 is the highest quality, down to 1 which is linear interpolation). I’ve found it’s fast enough to run on top quality. It also is quite flexible and is often able to change to a different channel count or bit depth at the same time.
Here;s a code sample that resamples an MP3 file (usually 44.1kHz) down to 16kHz. The MediaFoundationResampler takes an IWaveProvider as input, and a desired output format:
int outRate = 16000;
var inFile = @"test.mp3";
var outFile = @"test resampled MF.wav";
using (var reader = new Mp3FileReader(inFile))
{
var outFormat = new WaveFormat(outRate, reader.WaveFormat.Channels);
using (var resampler = new MediaFoundationResampler(reader, outFormat))
{
// resampler.ResamplerQuality = 60;
WaveFileWriter.CreateWaveFile(outFile, resampler);
}
}
The second option is even newer, and was added as part of NAudio 1.7.1. It’s based on the Cockos WDL resampler for which we were kindly granted permission to use as part of NAudio. It works with floating point samples, so you’ll need an ISampleProvider to pass in. Here we use AudioFileReader to get to floating point and then make a resampled 16 bit WAV file:
int outRate = 16000;
var inFile = @"test.mp3";
var outFile = @"test resampled WDL.wav";
using (var reader = new AudioFileReader(inFile))
{
var resampler = new WdlResamplingSampleProvider(reader, outRate);
WaveFileWriter.CreateWaveFile16(outFile, resampler);
}
The big advantage that the WDL resampler brings to the table is that it is fully managed. This means it can be used within Windows Store apps (as I’m still finding it very difficult to work out how to create the MediaFoundationResampler in a way that passes WACK).
You can also use WaveFormatConversionStream which is an ACM based Resampler, which has been in NAudio since the beginning and works back to Windows XP. It resamples 16 bit only and you can’t change the channel count at the same time. It predates IWaveProvider so you need to pass in a WaveStream based. Here’s it being used to resample an MP3 file:
int outRate = 16000;
var inFile = @"test.mp3";
var outFile = @"test resampled ACM.wav";
using (var reader = new Mp3FileReader(inFile))
{
var outFormat = new WaveFormat(outRate, reader.WaveFormat.Channels);
using (var resampler = new WaveFormatConversionStream(outFormat, reader))
{
WaveFileWriter.CreateWaveFile(outFile, resampler);
}
}
Of course the fact that NAudio lets you have raw access to the samples means you are able to write your own resampling algorithm, which could be Linear Interpolation or something more complex. I’d recommend against doing this unless you really understand audio DSP. If you want to see some spectograms showing what happens when you write your own naive resampling algorithm, have a look at this article I wrote on CodeProject. Basically, you’re likely to end up with significant aliasing if you don’t also write a low pass filter. Given NAudio now has the WDL resampler, that should probably be used for all cases where you need a fully managed resampler.
If you’re a Pluralsight subscriber, you can watch me doing some resampling in Module 4 of my “Audio Programming with NAudio” course
One question that commonly gets asked with NAudio is, “Why do I get a NoDriver calling acmFormatSuggest error when playing MP3s?” The answer is that you don’t have the ACM MP3 decoder installed. If you’re running a regular desktop version of windows from XP all the way up to 8.1, then the Fraunhofer MP3 ACM decoder is installed as standard and everything should just work.
But if you’re running a server version of Windows (or maybe one of the “N” versions of Windows sold in Europe), then you might be missing the codecs you need. There are various workarounds to this:
If you are running Windows Server, make sure you install the “Desktop Experience” component of Windows. This will install all the standard codecs, for audio playback (including Media Foundation support as well). To install on Azure, this script may be helpful.
The NAudio Mp3FileReader class allows you to inject an alternative MP3 frame decompressor, so you don’t need to use ACM if you don’t want to. To use the DMO MP3 frame decompressor for example, use the following code:
new Mp3FileReader(stream,wave=> new DmoMp3FrameDecompressor(wave));
With NAudio 1.7, the MediaFoundationReader class was introduced, which can not only play MP3 files, but lots of other formats too. In many ways, this should now be the preferred way of playing MP3s with NAudio. You will still need the Desktop Experience installed if you’re on Windows Server though.
Another option is to switch to fully managed MP3 decoder such as NLayer. This has the benefit of working on all platforms and requiring no codecs.
Finally, you may be here not because you’re playing MP3, but because you tried to play a WAV file containing some codec that your system cannot find an ACM driver for. This also results in the “NoDriver calling acmFormatSuggest error”. What you need to do is search the web for an ACM codec that can decode the format you are using. Unfortunately this is not always a simple process, but unless you have a codec installed that matches the compression type of the file you’re trying to play, you won’t be able to play it.
One of the sessions I was most excited about at BUILD this year was Jason Olson and Pete Brown’s session on Sequencers, Synthesizers, and Software, Oh My! Building Great Music Creation Apps for Windows Store.
I had been hoping before the conference that we might see MIDI support for WinRT...
@Pete_Brown looking forward to seeing the synth one, particularly if any MIDI support is coming for Windows Store apps
— Mark Heath (@mark_heath) March 31, 2014 ...and I wasn't disappointed. The talk covered a brand new MIDI API for Windows RT (requires 8.1 Update 1), as well as giving some guidance on low latency WASAPI usage.
Let's look first at the MIDI support…
Basically, it allows you to access MIDI in and out devices (MIDI over USB), receiving and sending messages. This is great news, and enables the creation of several types of application not previously possible in Windows Store apps (I’d wanted to make a software synthesizer for quite some time, so that’s first on my list now).
It’s distributed as a NuGet package allowing access from C#. This is huge in my opinion. One of the most painful things about doing audio work in C# has been endlessly creating wrappers for COM based APIs such as WASAPI and Media Foundation. It's really hard to get right in C# and you can run into all kinds of nasty threading issues. I had hoped when WinRT originally came out that it would wrap WASAPI and MediaFoundation in a way that eliminated the need for all the interop in NAudio, but sadly that was not to be the case. But this represents a great step in the right direction, and I hope it becomes standard.
What can't it do:
It doesn't read or write MIDI files, but that's not a particularly big deal to me. NAudio and MIDI.NET both can do that.
It doesn't provide a MIDI sequencer. You'd need to schedule out your MIDI out events at exactly the right time to play back a MIDI file from your app.
It doesn't offer up a software synthesizer as a MIDI out. That would be a great addition, especially if you could also load the software synthesizer up with SoundFonts.
There is no RTP MIDI support but they suggested it might be a feature under consideration. This would be useful for wireless MIDI, and could also form the basis for sending MIDI streams between applications which could also be useful.
And now onto WASAPI
There was some really useful information in the session on how to set up your audio processing to work at a pro audio thread priority, and how to use a “raw mode” to avoid global software based effects (e.g. a graphic equaliser) introducing additional latency. Getting good information on WASAPI is hard to come by, so it was good to see recommendations as well on using event driven mode and when to use exclusive mode.
What I would love to see with WASAPI is exactly the same approach as has been taken with MIDI. Give us a NuGet package that presents both the Render and Capture devices in a way that can be used from C#. Let me easily open an output device, specify if I need pro audio thread priority, raw mode, exclusive mode, and then give me a callback to fill the output buffer on demand with new samples. Ideally it should incorporate resampling too, as it is a huge pain to resample yourself if you are playing a 44.1kHz file while your soundcard is operating at 48kHz.
Likewise with capture. Let me easily open any capture device (or loopback capture), and get a callback whenever the next audio buffer is received containing the raw sample data. This might be possible already with the MediaCapture class but I'm unclear how you get at the raw samples (somehow create a custom sink?). Again, I’d like to be able to specify my capture sample rate and have the API resample on the fly for me, as often with voice recordings, 16kHz is quite acceptable. WASAPI is much harder to use than the legacy waveIn APIs in this regard.
Now Jason Olson did provide a link to a cool sample application on GitHub, showing the use of these APIs in C++. However, there seems to be a general assumption that if you're using WASAPI you are going to be working in C++. This is probably true for pro audio music applications. Despite having invested a huge amount of my time over the last 12 years in creating a C# audio library, I still recommend to people needing the lowest possible latency that they should consider C++.
But I also know that there is a huge amount of audio related development that does not require super low latency, particularly when dealing with voice recordings such as telephony, radio or VOIP. Recording and replaying voice is something that many businesses want to create software for, and a C# API greatly simplifies the development process. Both my experience from supporting NAudio, and having worked a decade in the telecoms industry tells me that there is a huge market for applications that deal with speech in various formats.
Finally, I'd also like to see Microsoft give some thought to creating an audio buss that allows WinRT apps to transmit real-time audio between themselves, allowing you to link together virtual synthesizers with effects. This could form the basis of solving the audio plugin problem for DAW-like applications in the Windows store.
Anyway, they repeatedly asked for feedback during the talk, so that’s the direction I’d like to see WinRT audio APIs going in. Seamless support for using audio capture and render devices from C# with transparent resampling on playback and record.
Quite often I get questions from people who would like to play audio that they are receiving over the network, or recording from the microphone, but also want to save that audio to WAV at the same time. This is actually quite easy to achieve, so long as you think in terms of a “signal chain” (which is something I talk about a lot in my audio courses on Pluralsight).
Basically, the usual strategy I recommend for playing audio that you receive over the network or from the microphone is to put it into a BufferedWaveProvider. You fill it with (PCM) audio as it becomes available, and then in its Read method, it returns the audio, or silence if the buffer is empty.
Normally, you’d pass the BufferedWaveProvider directly to the IWavePlayer device (such as WaveOut, but to implement save to WAV, we’ll first wrap it in a new signal chain component that we’ll create for this purpose. We’ll call it “SavingWaveProvider” and it will implement IWaveProvider. In it’s Read method, it will read from it’s source wave provider (the BufferedWaveProvider) in our case, and write to a WAV file before we pass it on.
We’ll dispose the WaveFileWriter if we read 0 bytes from the source wave provider, which should normally indicate we have reached the end of playback. But we also make the whole class Disposable, since BufferedWaveProvider is set up to always return the number of bytes we asked for in Read, so it will never reach the end itself.
Here’s the code for SavingWaveProvider:
class SavingWaveProvider : IWaveProvider, IDisposable
{
private readonly IWaveProvider sourceWaveProvider;
private readonly WaveFileWriter writer;
private bool isWriterDisposed;
public SavingWaveProvider(IWaveProvider sourceWaveProvider, string wavFilePath)
{
this.sourceWaveProvider = sourceWaveProvider;
writer = new WaveFileWriter(wavFilePath, sourceWaveProvider.WaveFormat);
}
public int Read(byte[] buffer, int offset, int count)
{
var read = sourceWaveProvider.Read(buffer, offset, count);
if (count > 0 && !isWriterDisposed)
{
writer.Write(buffer, offset, read);
}
if (count == 0)
{
Dispose(); // auto-dispose in case users forget
}
return read;
}
public WaveFormat WaveFormat { get { return sourceWaveProvider.WaveFormat; } }
public void Dispose()
{
if (!isWriterDisposed)
{
isWriterDisposed = true;
writer.Dispose();
}
}
}
And here’s how you use it to both play and save audio at the same time (note this is very simplified WPF app with two buttons and no checks that you don’t press Start twice in a row etc). The key is that we pass the BufferedWaveProvider into the constructor of the SavingWaveProvider, and then pass that to waveOut.Init. Then all we need to do is make sure we dispose SavingWaveProvider so that the WAV file header gets written correctly:
public partial class MainWindow : Window
{
private WaveIn recorder;
private BufferedWaveProvider bufferedWaveProvider;
private SavingWaveProvider savingWaveProvider;
private WaveOut player;
public MainWindow()
{
InitializeComponent();
}
private void OnStartRecordingClick(object sender, RoutedEventArgs e)
{
// set up the recorder
recorder = new WaveIn();
recorder.DataAvailable += RecorderOnDataAvailable;
// set up our signal chain
bufferedWaveProvider = new BufferedWaveProvider(recorder.WaveFormat);
savingWaveProvider = new SavingWaveProvider(bufferedWaveProvider, "temp.wav");
// set up playback
player = new WaveOut();
player.Init(savingWaveProvider);
// begin playback & record
player.Play();
recorder.StartRecording();
}
private void RecorderOnDataAvailable(object sender, WaveInEventArgs waveInEventArgs)
{
bufferedWaveProvider.AddSamples(waveInEventArgs.Buffer,0, waveInEventArgs.BytesRecorded);
}
private void OnStopRecordingClick(object sender, RoutedEventArgs e)
{
// stop recording
recorder.StopRecording();
// stop playback
player.Stop();
// finalise the WAV file
savingWaveProvider.Dispose();
}
}
This technique isn’t only for saving audio that you record or receive over the network. It’s also a great way to get a copy of the audio you just played, which is very handy when you want to troubleshoot audio issues and want to get a copy of the exact audio that was sent to the soundcard. You can even insert many of these at different places in your signal chain, to hear what the audio sounded like earlier in the signal chain.
In NAudio 1.7 I introduced Media Foundation support into NAudio. The main classes are MediaFoundationReader, MediaFoundationEncoder, MediaFoundationResampler and MediaFoundationTransform. Those four classes actually cover the bulk of what most people would want to do with Media Foundation, but they required a huge amount of interop, and there are still a few areas yet to complete which are proving rather tricky.
But I noticed a while ago that SharpDX has actually already got the vast majority of Media Foundation interop made available in its SharpDX.MediaFoundation assembly. It got me thinking, what if NAudio simply relied on the work done in SharpDX rather than creating its own interop wrappers? There are several advantages and disadvantages to taking this approach.
Completeness
A large part of SharpDX is auto-generated, which means that it contains pretty much every interface, every API call, every enumeration, and every Guid you could ever need. By contrast, the NAudio wrappers are done by hand, so there are a number of bits missing. In particular SharpDX has done the hard work of implementing the reading and encoding from standard .NET streams which is something I've really wanted to add to NAudio but it has proved very complicated to implement. SharpDX also includes interop wrappers that allow access to the "presentation attributes" which would be a nice enhancement.
Cross-Threading
SharpDX uses a fancy post-compile trick which allows COM objects created on one thread to be used on another. NAudio needs something like this, as the current workaround I use is a bit of a hack (basically recreate the MF source reader on the new thread). However, I will confess to not fully understanding how the SharpDX technique works, which is why I haven’t yet borrowed the technique for NAudio.
Closing
SharpDX has a really nice approach to creating wrappers for COM objects, which makes them disposable objects. By contrast I simply have been trying to call Marshal.ReleaseComObject in all the right places in NAudio. I try to hide this as much as possible from users of NAudio, but there are places where it does leak out leaving the potential for a memory leak.
Collaboration
I always find it slightly depressing that so many open source projects decide to compete rather than collaborate with each other. If we spent less time re-inventing the wheel and more time building upon what others have already created, I’m sure we could make some amazing software.
So I could just decide to let SharpDX do what it does well (wrap COM-based Windows APIs), and then NAudio could focus on providing helpful ways to construct your audio graph (which for me is the interesting bit). What's more, if NAudio used SharpDX, it could result in enhancements being submitted to SharpDX (I've already made a few contributions as part of my experimentation).
So there’s a lot in favour of using SharpDX, but there are some disadvantages that need to be weighed up.
Dependencies
We'd need to make NAudio depend on two additional DLLs – SharpDX.dll and SharpDX.MediaFoundation.dll. As well as increasing the overall size of the dependencies, this could also cause some namespace confusion as NAudio and SharpDX both have classes with the same name (WaveFormat is a prime example). There would also be potential for versioning conflicts if NAudio was introduced into a project that was using a different version of SharpDX.
Control
By depending on an external library like SharpDX, we'd lose a measure control over all the interop. Currently NAudio's interop is hand-crafted to be exactly how I want it. But with SharpDX, I'd need to submit pull-requests for all the tweaks I want. I'd also be dependent on the release schedule of SharpDX - a new version of NAudio would need to depend on an official release of SharpDX, not a special build. Having said that, Alexandre Mutel has been quick to accept my pull requests so far and there isn’t a pressing need for any more to be made.
Nearly There?
It may be that I'm not that far off at all. If I get read and write from a stream working, and can solve the threading issue, then the main motivation for building on SharpDX would disappear. But I can't really tell how close I am to getting it all working just how I want it.
The good news is that I can trial all of this without needing to change NAudio at all. I've made a new library that depends on NAudio and SharpDX, and offers alternative implementations of NAudio's four Media Foundation classes. I've called them SharpMediaFoundationReader, SharpMediaFoundationEncoder, SharpMediaFoundationResampler and SharpMediaFoundationTransform. It’s called NAudio.SharpMediaFoundation and it’s on GitHub.
Once a version of SharpDX is released containing my customisations, I can release this as its own NuGet package, then that would allow people who want/need the features SharpDX can offer to take advantage of it without any disruption to the existing NAudio library. So maybe I can have the best of both worlds.
Do let me know if you have any feedback on this approach to MediaFoundation. I’ll announce on my blog when an official release of NAudio.SharpMediaFoundation is available.
Every so often I get a request for help from someone wanting to create a simple one-liner function that can play an audio file with NAudio. Often they will create something like this:
// warning: this won't work
public void PlaySound(string fileName)
{
using (var output = new WaveOut())
using (var player = new AudioFilePlayer(fileName))
{
output.Init(player);
output.Play();
}
}
Unfortunately this won’t actually work, since the Play method doesn’t block until playback is finished – it simply begins playback. So you end up disposing the playback device almost instantaneously after beginning playback. A slightly improved option simply waits for playback to stop, creating a blocking call. It uses WaveOutEvent, as the standard WaveOut only works on GUI threads.
// better, but still not ideal
public void PlaySound(string fileName)
{
using (var output = new WaveOutEvent())
using (var player = new AudioFilePlayer(fileName))
{
output.Init(player);
output.Play();
while (output.PlaybackState == PlaybackState.Playing)
{
Thread.Sleep(500);
}
}
}
This approach is better, but is still not ideal, since it now blocks until audio playback is complete. It is also not particularly suitable for scenarios in which you are playing lots of short sounds, such as sound effects in a computer game. The problem is, you don’t really want to be continually opening and closing the soundcard, or having multiple instances of an output device active at once. So in this post, I explain the approach I would typically take for an application that needs to regularly play sounds in a “Fire and Forget” manner.
Use a Single Output Device
First, I’d recommend just opening the output soundcard once. Choose the output model you want (e.g. WasapiOut, WaveOutEvent), and play all the sounds through it.
Use a MixingSampleProvider
This means that to play multiple sounds simultaneously, you’ll need a mixer. I always recommend mixing with 32 bit IEEE floating point samples. and in NAudio the best way to do this is through using the MixingSampleProvider class.
Play Continuously
Obviously there are times when your application won’t be playing any sounds, so you could start and stop the output device whenever playback is idle. But it tends to be more straightforward to simply leave the soundcard running playing silence, and then just add inputs to the mixer. If you set the ReadFully property on MixingSampleProvider to true, it’s Read method will return buffers full of silence even when there are no mixer inputs. This means that the output device will keep playing continuously.
Use a Single WaveFormat
The one down-side of this approach is that you can’t mix together audio that doesn’t share the same WaveFormat. The bit depth won’t be a problem, since we are automatically converting everything to IEEE floating point. But if you are working with a stereo mixer, any mono inputs need to be made stereo before playing them. More annoying is the issue of sample rate conversion. If the files you need to play contain a mixture of sample rates, you’ll need to convert them all to a common value. 44.1kHz would be a typical choice, since this is likely to be the sample rate your soundcard is operating at.
Dispose Readers
The MixingSampleProvider has a nice feature where it will automatically remove an input whose Read method returns 0. However, it won’t attempt to Dispose that input for you, leaving you with a resource leak. The easiest way round this is to create a derived ISampleProvider class that encapsulates the AudioFileReader, and auto-disposes it when it reaches the end.
Cache Sounds
In a computer game scenario, you’ll likely be playing the same sounds again and again. You don’t really want to keep reading them from disk (and decoding them if they compressed). So it would be best to load the whole thing into memory, allowing us to replay many copies of it directly from the byte array of PCM data, using a RawSourceWaveStream. This approach has the advantage of allowing you to dispose the AudioFileReader immediately after caching its contents.
Source Code
That’s enough waffling, let’s have a look at some code that implements the features mentioned above. Let’s start with what I’ve called “AudioPlaybackEngine”. This is responsible for playing our sounds. You can either call PlaySound with a path to a file, for use with longer pieces of audio, or passing in a “CachedSound” for use with sound effects you want to play many times. I’ve included automatic conversion from mono to stereo, but no resampling is included here, so if you pass in a file of the wrong sample rate it won’t play:
class AudioPlaybackEngine : IDisposable
{
private readonly IWavePlayer outputDevice;
private readonly MixingSampleProvider mixer;
public AudioPlaybackEngine(int sampleRate = 44100, int channelCount = 2)
{
outputDevice = new WaveOutEvent();
mixer = new MixingSampleProvider(WaveFormat.CreateIeeeFloatWaveFormat(sampleRate, channelCount));
mixer.ReadFully = true;
outputDevice.Init(mixer);
outputDevice.Play();
}
public void PlaySound(string fileName)
{
var input = new AudioFileReader(fileName);
AddMixerInput(new AutoDisposeFileReader(input));
}
private ISampleProvider ConvertToRightChannelCount(ISampleProvider input)
{
if (input.WaveFormat.Channels == mixer.WaveFormat.Channels)
{
return input;
}
if (input.WaveFormat.Channels == 1 && mixer.WaveFormat.Channels == 2)
{
return new MonoToStereoSampleProvider(input);
}
throw new NotImplementedException("Not yet implemented this channel count conversion");
}
public void PlaySound(CachedSound sound)
{
AddMixerInput(new CachedSoundSampleProvider(sound));
}
private void AddMixerInput(ISampleProvider input)
{
mixer.AddMixerInput(ConvertToRightChannelCount(input));
}
public void Dispose()
{
outputDevice.Dispose();
}
public static readonly AudioPlaybackEngine Instance = new AudioPlaybackEngine(44100, 2);
}
The CachedSound class is responsible for reading an audio file into memory. Sample rate conversion would be best done in here as part of the caching process, so it minimises the performance hit of resampling during playback.
class CachedSound
{
public float[] AudioData { get; private set; }
public WaveFormat WaveFormat { get; private set; }
public CachedSound(string audioFileName)
{
using (var audioFileReader = new AudioFileReader(audioFileName))
{
// TODO: could add resampling in here if required
WaveFormat = audioFileReader.WaveFormat;
var wholeFile = new List<float>((int)(audioFileReader.Length / 4));
var readBuffer= new float[audioFileReader.WaveFormat.SampleRate * audioFileReader.WaveFormat.Channels];
int samplesRead;
while((samplesRead = audioFileReader.Read(readBuffer,0,readBuffer.Length)) > 0)
{
wholeFile.AddRange(readBuffer.Take(samplesRead));
}
AudioData = wholeFile.ToArray();
}
}
}
There’s also a simple helper class to turn a CachedSound into an ISampleProvider that can be easily added to the mixer:
class CachedSoundSampleProvider : ISampleProvider
{
private readonly CachedSound cachedSound;
private long position;
public CachedSoundSampleProvider(CachedSound cachedSound)
{
this.cachedSound = cachedSound;
}
public int Read(float[] buffer, int offset, int count)
{
var availableSamples = cachedSound.AudioData.Length - position;
var samplesToCopy = Math.Min(availableSamples, count);
Array.Copy(cachedSound.AudioData, position, buffer, offset, samplesToCopy);
position += samplesToCopy;
return (int)samplesToCopy;
}
public WaveFormat WaveFormat { get { return cachedSound.WaveFormat; } }
}
And here’s the auto disposing helper for when you are playing from an AudioFileReader directly:
class AutoDisposeFileReader : ISampleProvider
{
private readonly AudioFileReader reader;
private bool isDisposed;
public AutoDisposeFileReader(AudioFileReader reader)
{
this.reader = reader;
this.WaveFormat = reader.WaveFormat;
}
public int Read(float[] buffer, int offset, int count)
{
if (isDisposed)
return 0;
int read = reader.Read(buffer, offset, count);
if (read == 0)
{
reader.Dispose();
isDisposed = true;
}
return read;
}
public WaveFormat WaveFormat { get; private set; }
}
With all this set up, now we can have our goal of using a very simple fire and forget syntax for playback:
// on startup:
var zap = new CachedSound("zap.wav");
var boom = new CachedSound("boom.wav");
// later in the app...
AudioPlaybackEngine.Instance.PlaySound(zap);
AudioPlaybackEngine.Instance.PlaySound(boom);
AudioPlaybackEngine.Instance.PlaySound("crash.wav");
// on shutdown
AudioPlaybackEngine.Instance.Dispose();
Further Enhancements
This is far from complete. Obviously I’ve not added in the Resampler stage here, and it would be nice to add a master volume level for the audio playback engine, as well as allowing you to set individual sound volume and panning positions. You could even have a maximum limit of concurrent sounds. But none of those enhancements are too hard to add.
I’ll try to get something like this added into the NAudio WPF Demo application, maybe with a few of these enhancements thrown in. For now, you can get at the code from this gist.
It’s been just over a year since the last release, so an updated release of NAudio is long overdue and there’s a lot of great new features added over the last year which I’m really excited to make official. In particular, this release adds Media Foundation support, allowing you to play from a much wider variety of audio file types, including extracting audio from video files. There’s also been significant progress on Windows 8 store app support although it still isn’t quite ready for official release.
As usual you can get it via Nuget, or download it from Codeplex. Last year NAudio 1.6 was downloaded 29616 times from Codeplex and 7089 times on nuget.
Another thing I’ve wanted to do for ages is to create some really good training materials for NAudio. I began this with the release of my Digital Audio Fundamentals course on Pluralsight, which is all about giving you the background understanding you need in order to work effectively with NAudio. And I’m on the verge of releasing the follow-on course which is around 6 hours of in-depth training on how to use NAudio. Keep an eye on my Pluralsight Author page and you should see it in a couple of weeks. I’ll also announce it here on my blog. You do need to be a Pluralsight subscriber to watch these videos, but it’s only $29 for a month’s subscription, which also gives you access to their entire library, making it a great deal.
I’ve still got lots of plans for NAudio. Here’s a few of the things near the top of my list for NAudio 1.8:
Hope you find NAudio useful. Don’t forget to get in touch and tell me what you’ve built with it.
I get several emails every day asking me for help with NAudio, and often it is clear that the real issue is that many developers only have a rather vague grasp of the concepts of digital audio. So I wanted to create a really good training course that would introduce and clearly explain all the things I think are really important to developers working with digital audio in general, and NAudio in particular.
So I’m really pleased to announce the availability of my Digital Audio Fundamentals course on Pluralsight. The course begins by explaining the essential topic of sampling. In particular you need to understand sample rates and bit depths if you are to be successful with NAudio. I then talk about various file formats and lots of different codecs, before looking at several types of effects. The final module may in fact be the most important for NAudio developers as I explain the concept of “signal chains” or “pipelines”. To get things done with NAudio you need to have a clear grasp of the signal chain that you are working with. NAudio tries to make it easy for you to construct a signal chain that does exactly what you need out of many simple components.
So I’d really encourage anyone who wants to use NAudio and is able to view this course to do so. It’s just under 3 hours, but it will be a very worthwhile investment of your time. I’m hoping that maybe later this year there will be a follow-up course focusing specifically on using NAudio.
I know some of you may be disappointed that this is not a free course, but I think you’ll find the Pluralsight pricing to be very reasonable. You could always sign up for just one month to watch this course, and get the added benefit of tapping into their vast library of excellent courses on a wide variety of programming topics.
One of the challenges that frequently arises when writing audio code in C# is that you get a byte array containing raw audio that would be better presented as a short (Int16) array, or a float (Single) array. (There are other formats too – some audio is 32 bit int, some is 64 bit floating point, and then there is the ever-annoying 24 bit audio). In C/C++ the solution is simple, cast the address of the byte array to a short * or a float * and access each sample directly.
Unfortunately, in .NET casting byte arrays into another type is not allowed:
byte[] buffer = new byte[1000]; short[] samples = (short[])buffer; // compile error!
This means that, for example, in NAudio, when the WaveIn class returns a byte[] in its DataAvailable event, you usually need to convert it manually into 16 bit samples (assuming you are recording 16 bit audio). There are several ways of doing this. I’ll run through five approaches, and finish up with some performance measurements.
BitConverter.ToInt16
Perhaps the simplest conceptually is to use the System.BitConverter class. This allows you to convert a pair of bytes at any position in a byte array into an Int16. To do this you call BitConverter.ToInt16. Here’s how you read through each sample in a 16 buffer:
byte[] buffer = ...;
for(int n = 0; n < buffer.Length; n+=2)
{
short sample = BitConverter.ToInt16(buffer, n);
}
For byte arrays containing IEEE float audio, the principle is similar, except you call BitConverter.ToSingle. 24 bit audio can be dealt with by copying three bytes into a temporary four byte array and using ToInt32.
BitConverter also includes a GetBytes method to do the reverse conversion, but you must then manually copy the return of that method into your buffer.
Bit Manipulation
Those who are more comfortable with bit manipulation may prefer to use bit shift and or to convert each pair of bytes into a sample:
byte[] buffer = ...;
for (int n = 0; n < buffer.Length; n+=2)
{
short sample = (short)(buffer[n] | buffer[n+1] << 8);
}
This technique can be extended for 32 bit integers, and is very useful for 24 bit, where none of the other techniques work very well. However, for IEEE float, it is a bit more tricky, and one of the other techniques should be preferred.
For the reverse conversion, you need to write more bit manipulation code.
Buffer.BlockCopy
Another option is to copy the whole buffer into an array of the correct type. Buffer.BlockCopy can be used for this purpose:
byte[] buffer = ...; short[] samples = new short[buffer.Length]; Buffer.BlockCopy(buffer,0,samples,0,buffer.Length);
Now the samples array contains the samples in easy to access form. If you are using this technique, try to reuse the samples buffer to avoid making extra work for the garbage collector.
For the reverse conversion, you can do another Buffer.BlockCopy.
WaveBuffer
One cool trick NAudio has up its sleeve (thanks to Alexandre Mutel) is the “WaveBuffer” class. This uses the StructLayout=LayoutKind.Explicit attribute to effectively create a union of a byte[], a short[], an int[] and a float[]. This allows you to “trick” C# into letting you access a byte array as though it was a short array. You can read more about how this works here. If you’re worried about its stability, NAudio has been successfully using it with no issues for many years. (The only gotcha is that you probably shouldn’t pass it into anything that uses reflection, as underneath, .NET knows that it is still a byte[], even if it has been passed as a float[]. So for example don’t use it with Array.Copy or Array.Clear). WaveBuffer can allocate its own backing memory, or bind to an existing byte array, as shown here:
byte[] buffer = ...; var waveBuffer = new WaveBuffer(buffer); // now you can access the samples using waveBuffer.ShortBuffer, e.g.: var sample = waveBuffer.ShortBuffer[sampleIndex];
This technique works just fine with IEEE float, accessed through the FloatBuffer property. It doesn’t help with 24 bit audio though.
One big advantage is that no reverse conversion is needed. Just write into the ShortBuffer, and the modified samples are already in the byte[].
Unsafe Code
Finally, there is a way in C# that you can work with pointers as though you were using C++. This requires that you set your project assembly to allow “unsafe” code. "Unsafe” means that you could corrupt memory if you are not careful, but so long as you stay in bounds, there is nothing unsafe at all about this technique. Unsafe code must be in an unsafe context – so you can use an unsafe block, or mark your method as unsafe.
byte[] buffer = ...;
unsafe
{
fixed (byte* pBuffer = buffer)
{
short* pSample = (short*)buffer;
// now we can access samples via pSample e.g.:
var sample = pSample[sampleIndex];
}
}
This technique can easily be used for IEEE float as well. It also can be used with 24 bit if you use int pointers and then bit manipulation to blank out the fourth byte.
As with WaveBuffer, there is no need for reverse conversion. You can use the pointer to write sample values directly into the memory for your byte array.
Performance
So which of these methods performs the best? I had my suspicions, but as always, the best way to optimize code is to measure it. I set up a simple test application which went through a four minute MP3 file, converting it to WAV and finding the peak sample values over periods of a few hundred milliseconds at a time. This is the type of code you would use for waveform drawing. I measured how long each one took to go through a whole file (I excluded the time taken to read and decode MP3). I was careful to write code that avoided creating work for the garbage collector.
Each technique was quite consistent in its timings:
| Debug Build | Release Build | |
| BitConverter | 263,265,264 | 166,167,167 |
| Bit Manipulation | 254,243,250 | 104,104,103 |
| Buffer.BlockCopy | 205,206,204 | 104,103,103 |
| WaveBuffer | 239.264.263 | 97,97,97 |
| Unsafe | 173.172.162 | 98,98,98 |
As can be seen, BitConverter is the slowest approach, and should probably be avoided. Buffer.BlockCopy was the biggest surprise for me - the additional copy was so quick that it paid for iteself very quickly. WaveBuffer was surprisingly slow in debug build – but very good in Release build. It is especially impressive given that it doesn’t need to pin its buffers like the unsafe code does, so it may well be the quickest possible technique in the long-run as it doesn’t hinder the garbage collector from compacting memory. As expected the unsafe code gave very fast performance. The other takeaway is that you really should be using Release build if you are doing audio processing.
Anyone know an even faster way? Let me know in the comments.
I was recently interviewed by Carl Franklin and Richard Campbell for their .NET Rocks podcast and the episode was published yesterday. You can have a listen here. I was invited onto the show after helping Carl out with an interesting ASIO related problem. I essentially built a mixer for his 48 in and 48 out MOTU soundcard. It is by far the most data that anyone has ever tried to push through NAudio (to my knowledge) and it did struggle a bit – he had to reduce the channel count to avoid corruption, but it was still impressive what was achieved at a low latency. However, I’m hoping to do some performance optimisations, and it would be very interesting to see if we can get 48 in and 48 (at 44.1kHz working smoothly in a managed environment). I’ll hopefully blog about it once I’ve got something working.
Last month I wrote an article on CodeProject explaining how you can use NAudio to encode or decode audio in any format for which you have a codec. I’m linking to it here, as it’s one of the topics I get asked about regularly. If you want to know how to decode or encode MP3, WMA, AAC, GSM, a-law or any other formats, then head over there. The article includes a lot of background information that will help you understand why some conversions can’t be done in a single step. I wrote it to give me something to point people to when they have questions about various encoding or decoding tasks with NAudio.
I’ve been working on adding Media Foundation support to NAudio 1.7 over the past few weeks. There are two reasons for this. The first is that Windows Store apps can use Media Foundation but cannot use ACM or DMO, which were the two codec APIs that NAudio did support. This was the impetus I needed to finally get round to wrapping Media Foundation API, having been put off by the thought of wrapping yet another large COM-based API.
The second is simply that Media Foundation is the future for audio codecs in Windows, and includes some codec support that ACM doesn’t offer, such as AAC encode and decode. Windows 8 even comes with an MP3 encoder.
There was a lot of interop code required to get Media Foundation working, and with the help of a few people (most notably ManU from the Codeplex forums). I have now done enough to enable the three main uses of Media Foundation – encoding, decoding and resampling.
NAudio 1.7 will have the following three main classes to support Media Foundation:
I’m also working on adding Windows Store support for . The main difference is the way you read and write files in Windows Store apps. Currently I’ve got a derived MediaFoundationReaderRT class in the demo, which allows you to open files from an IRandomAccessStream. I’ll probably do a similar thing for the encoder class as well.
I think the code can still be optimised a bit, particularly in the way that Media Buffers are created during resampling, but I am actually very close to completion, and I think this is going to be a fantastic feature for the next version of NAudio. If you want to try it out, you can build the latest NAudio from code yourself, or lookout for preview builds of NAudio 1.7 on Nuget. The NAudio WPF Demo app includes demonstrations of using all three of the main NAudio Media Foundation classes, plus how to enumerate the Media Foundation codecs.
One of my goals for NAudio 1.7 is to have a version available for Windows Store apps. Obviously there are a lot of classes in NAudio that simply won’t work with Windows Store apps, but I have been pleasantly surprised to discover that the bulk of the WASAPI and Media Foundation APIs are allowed. ACM, and all the rest of the old MME functions (waveIn.., waveOut…) are obviously not available. I’m not entirely sure what the status of DirectX Media Objects is (DMOs), but I suspect they are not available.
The first step was simply to create a Windows Store class library and see how much of the existing code I could move across. Here’s some notes on classes that I couldn’t move across
So as you can see, there is plenty of work still to be done. There are a few additional tasks once I’ve got everything I wanted moved across.
I’ve decided it’s time to release NAudio 1.6, as there are a whole load of fixes, features and improvements that have been added since I released NAudio 1.5 which I want to make available to a wider audience (if you’ve been downloading the preview releases on NuGet then you’re already more or less up to date). This marks something of a milestone for the project as it was around this time in 2002 that I first started working on NAudio, using a very early build of SharpDevelop and compiling against .NET 1.0. Some of the code I wrote back then is still in there (the oldest file is MixerInterop.cs, created on 9th December 2002).
NAudio 1.6 can be downloaded from NuGet or CodePlex.
A big thanks to everyone who has contributed bug fixes, features, bug reports, and even a few donations this year. To date NAudio 1.5 has been downloaded 34,213 times from CodePlex and 3,539 times on NuGet. I’ll be continuing to upload pre-release versions on NuGet, so check for the latest builds here.
I announced last release that I would finally be moving from .NET 2.0 to 3.5, and was persuaded to delay the move. However, this time I will be upgrading the project. The main reason is to enable extension methods (I know there are hacky ways to do this in .NET 2.0). With extension methods I can make the new ISampleProvider interface much easier to use, and it will become a more prominent part of NAudio. I have some nice ideas for a fluent interface for NAudio, allowing you to construct your audio pipeline much more elegantly.
I also have plans to move my development environment over to Windows 8 in the very near future, and a WinRT version of NAudio is on my priority list. I have already implemented fully managed MP3 decoding for Windows RT, and hope to release that as an open source project soon.
There are lots of other features on my todo list for NAudio. One of the big drivers behind the ISampleProvider interface is my desire to make audio effects easier to implement, so I’m hoping to get a collection of audio effects in the next version. I’ve also got a managed resampler which is almost working, but wasn’t quite ready to go in to NAudio 1.6.
Anyway, hope you find NAudio useful. Do let me know what cool things you have made with it, and I’ll link to you on the NAudio home page.