Thoughts on Windows RT Audio APIs and BUILD 2014

One of the sessions I was most excited about at BUILD this year was Jason Olson and Pete Brown’s session on Sequencers, Synthesizers, and Software, Oh My! Building Great Music Creation Apps for Windows Store.

I had been hoping before the conference that we might see MIDI support for WinRT...

@Pete_Brown looking forward to seeing the synth one, particularly if any MIDI support is coming for Windows Store apps

— Mark Heath (@mark_heath) March 31, 2014

...and I wasn't disappointed. The talk covered a brand new MIDI API for Windows RT (requires 8.1 Update 1), as well as giving some guidance on low latency WASAPI usage.

Let's look first at the MIDI support…

Basically, it allows you to access MIDI in and out devices (MIDI over USB), receiving and sending messages. This is great news, and enables the creation of several types of application not previously possible in Windows Store apps (I’d wanted to make a software synthesizer for quite some time, so that’s first on my list now).

It’s distributed as a NuGet package allowing access from C#. This is huge in my opinion. One of the most painful things about doing audio work in C# has been endlessly creating wrappers for COM based APIs such as WASAPI and Media Foundation. It's really hard to get right in C# and you can run into all kinds of nasty threading issues. I had hoped when WinRT originally came out that it would wrap WASAPI and MediaFoundation in a way that eliminated the need for all the interop in NAudio, but sadly that was not to be the case. But this represents a great step in the right direction, and I hope it becomes standard.

What can't it do:

It doesn't read or write MIDI files, but that's not a particularly big deal to me. NAudio and MIDI.NET both can do that.

It doesn't provide a MIDI sequencer. You'd need to schedule out your MIDI out events at exactly the right time to play back a MIDI file from your app.

It doesn't offer up a software synthesizer as a MIDI out. That would be a great addition, especially if you could also load the software synthesizer up with SoundFonts.

There is no RTP MIDI support but they suggested it might be a feature under consideration. This would be useful for wireless MIDI, and could also form the basis for sending MIDI streams between applications which could also be useful.

And now onto WASAPI

There was some really useful information in the session on how to set up your audio processing to work at a pro audio thread priority, and how to use a “raw mode” to avoid global software based effects (e.g. a graphic equaliser) introducing additional latency. Getting good information on WASAPI is hard to come by, so it was good to see recommendations as well on using event driven mode and when to use exclusive mode.

What I would love to see with WASAPI is exactly the same approach as has been taken with MIDI. Give us a NuGet package that presents both the Render and Capture devices in a way that can be used from C#. Let me easily open an output device, specify if I need pro audio thread priority, raw mode, exclusive mode, and then give me a callback to fill the output buffer on demand with new samples. Ideally it should incorporate resampling too, as it is a huge pain to resample yourself if you are playing a 44.1kHz file while your soundcard is operating at 48kHz.

Likewise with capture. Let me easily open any capture device (or loopback capture), and get a callback whenever the next audio buffer is received containing the raw sample data. This might be possible already with the MediaCapture class but I'm unclear how you get at the raw samples (somehow create a custom sink?). Again, I’d like to be able to specify my capture sample rate and have the API resample on the fly for me, as often with voice recordings, 16kHz is quite acceptable. WASAPI is much harder to use than the legacy waveIn APIs in this regard.

Now Jason Olson did provide a link to a cool sample application on GitHub, showing the use of these APIs in C++. However, there seems to be a general assumption that if you're using WASAPI you are going to be working in C++. This is probably true for pro audio music applications. Despite having invested a huge amount of my time over the last 12 years in creating a C# audio library, I still recommend to people needing the lowest possible latency that they should consider C++.

But I also know that there is a huge amount of audio related development that does not require super low latency, particularly when dealing with voice recordings such as telephony, radio or VOIP. Recording and replaying voice is something that many businesses want to create software for, and a C# API greatly simplifies the development process. Both my experience from supporting NAudio, and having worked a decade in the telecoms industry tells me that there is a huge market for applications that deal with speech in various formats.

Finally, I'd also like to see Microsoft give some thought to creating an audio buss that allows WinRT apps to transmit real-time audio between themselves, allowing you to link together virtual synthesizers with effects. This could form the basis of solving the audio plugin problem for DAW-like applications in the Windows store.

Anyway, they repeatedly asked for feedback during the talk, so that’s the direction I’d like to see WinRT audio APIs going in. Seamless support for using audio capture and render devices from C# with transparent resampling on playback and record.

Enabling NAudio for Windows 8 Store Apps–First Steps

One of my goals for NAudio 1.7 is to have a version available for Windows Store apps. Obviously there are a lot of classes in NAudio that simply won’t work with Windows Store apps, but I have been pleasantly surprised to discover that the bulk of the WASAPI and Media Foundation APIs are allowed. ACM, and all the rest of the old MME functions (waveIn.., waveOut…) are obviously not available. I’m not entirely sure what the status of DirectX Media Objects is (DMOs), but I suspect they are not available.

The first step was simply to create a Windows Store class library and see how much of the existing code I could move across. Here’s some notes on classes that I couldn’t move across

• WaveFormatCustomMarshaller - not supported because there is no support for System.Runtime.InteropServices.ICustomMarshaller. This is a bit of a shame, but not a huge loss.
• ProgressLog and FileAssociations in the Utils folder probably should have been kicked out of the NAudio DLL a long time ago. I’ll mark them as Obsolete
• Some of the DMO interfaces were marked with System.Security.SuppressUnmanagedCodeSecurity. I can’t remember why I needed to do this. It may be irrelevant if Windows Store apps can’t use DMO. I’ve simply allowed the code to compile by hiding this attribute with #if !NETFX_CORE
• One really annoying thing is that the Guid constructor has subtly changed, meaning that you can’t pass in unsigned int and shorts. It means that I had to put unchecked casts to short or int on lots of them
• One apparent oversight is that COMException no longer has an error code property. I guess it might be available in the exception data dictionary. It was only needed for DMO so again it may not matter
• The ApplicationException class has gone away, so I’ve replaced all instances of it with more appropriate exception types (usually InvalidDataException or ArgumentException)
• The fact that there is no more GetSafeHandle on wait handles means that I will need to rework the WASAPI code to use CreateEventEx.
• I’ve not bothered to bring across the Cakewalk drum map or sfz support. Both can probably be obsoleted from NAudio.
• The AcmMp3FrameDecompressor is not supported, and I suspect that Media Foundation will become the main way to decode MP3s (with the other option being fully managed decoders for which I have a working prototype – watch this space)
• Encoding.ASCIIEncoding is no longer present. Quite a bit of my code uses it, and I’ve switched to UTF8 for now even though it it is not strictly correct. I’ll probably have to make my own byte encoding utility for legacy file formats. Also Encoding.GetString has lost the overload that takes one parameter.
• I had some very old code still using ArrayList removing it had some knock-on effects throughout the SoundFont classes (which I suspect very few people actually use).
• WaveFileChunkReader will have to wait until RiffChunk gets rewritten to not depend on mmioToFourCC
• Everything in the GUI namespace is WindowsForms and won’t come across
• The Midi namespace I have left out for now. The classes for the events should move across, and the file reader writer will need reworking for Windows 8 file APIs. I don’t think windows store apps have any support for actual MIDI devices unfortunately.
• The old Mixer API is not supported at all in Win 8. The WASAPI APIs will give some control over stream volumes.
• ASIO – I’m assuming ASIO is not supported at all in Windows Store apps
• The Compression folder has all the ACM stuff. None of this is supported in Windows Store apps.
• The MmeInterop folder also doesn’t contain anything that is supported in Windows Store apps.
• SampleProviders - all came across successfully. These are going to be a very important part of NAudio moving forwards
• MediaFoundation (a new namespace), has come across successfully, and should allow converting MP3, AAC, and WMA to WAV in Windows Store apps. It will also be very useful for regular Windows apps on Vista and above. Expect more features to be added in this area in the near future..
• WaveInputs – not much of this folder could be ported
• WasapiCapture - needs rework to not use Thread or WaitHandle. Also I think the way you specify what device to use has changed in Windows Store apps
• WasapiLoopbackCapture – I don’t know if Windows Store apps are going to support loopback capture, but I will try to see what is possible
• I may revisit the IWaveIn interface, which I have never really been happy with, and come up with an IRecorder interface in the future,to make it easier to get at the samples as they are recorded (rather than just getting a byte array)
• WaveOutputs:
• WasapiOut – should work in Windows Store, but because it uses Thread and EventWaitHandle it needs some reworking
• AsioOut, WaveOut, WaveOutEvent, DirectSoundOut  - not supported. For Windows Store apps, it will either be WasapiOut or possibly a new output device depending on what I find in the Windows RT API reference.
• AiffFileWriter, CueWaveFileWriter , WaveFileWriter – all the classes that can write audio files need to be reworked as you can’t use FileStreams in Windows Store. I need to find a good approach to this that doesn’t require the Windows Store and regular .NET code to completely diverge. Suggestions welcome.
• WaveProviders – mostly came across with a few exceptions:
• MixingWaveProvider32 - used unsafe code, MixingSampleProvider should be preferred anyway
• WaveRecorder – relies on WaveFileWriter which needs rework
• WaveStream – lots of classes in this folder will need reworking for
• WaveFileReader, AiffFileReader, AudioFileReader, CueWaveFileReader  all need to support Windows Store file APIs
• Mp3FileReader – may be less important now we have MediaFoundationReader, but it still can be useful to have a frame by frame decode, so I’ll see if I can make a new IMp3FrameDecompressor that works in Windows Store apps.
• RiffChunk – to be reworked
• WaveInBuffer, WaveOutBuffer are no longer applicable (and should really be moved into the MmeInterop folder)
• Wave32To16Stream – contains unsafe code, should be obsoleted anyway
• WaveMixerStream32 – contains unsafe code, also should be obsoleted

So as you can see, there is plenty of work still to be done. There are a few additional tasks once I’ve got everything I wanted moved across.

• I want to investigate all the new Media APIs (e.g transcoding) and see if NAudio can offer any value-add to using these APIs
• Make a Windows Store demo app to show off and test what can be done. Would also like to test on a Surface device if possible (not sure if I’ll run into endian issues on ARM devices – anyone know?).
• Update the nuget package to contain a Windows Store binary

Upgrading family PCs to Windows 8

I have five children, and with only three PCs in the house, it can be a challenge to get access to one. Probably the time to buy another has come. However, managing accounts on three PCs for seven people is hardly fun, and a fourth would just add to the workload. So the idea of upgrading to Windows 8 with its accounts that can sync their settings across PCs appeals to me. I decided this week to take advantage of the £25 upgrade offer and set to work on upgrading.

Upgrade Assistant

Normally when I upgrade to a new version of Windows, I will do a fresh install. But the thought of having to recreate seven user accounts and set them all up with their preferences was not appealing, so I opted for an upgrade, keeping all apps and settings. There is also an upgrade that only keeps settings, but I presume that would mean that programs like Office, iTunes, CrashPlan, DropBox etc would get lost along the way.

The upgrade assistant examines your system and warns you about issues you might face. This is a nice touch and it warned me that I needed to uninstall Microsoft Security Essentials and that there might be a problem with VS2010 SP1. It also told me I needed to deauthorize my computer on iTunes, which it didn’t explain, but this turns out to be a fairly simple task (my wife uses iTunes but I don’t so I’m not too familiar with the interface). It warns that Windows 8 doesn’t come with DVD playback as standard, although there does seem to be a rather convoluted way to add Windows Media Center for free if you take up the upgrade offer. The one thing the upgrade assistant forgot to tell me was to uninstall Windows Live Family Safety first, which I should have done as I want to use with Windows 8 family safety settings.

It was a little irritating that the upgrade assistant insists on downloading Windows on every machine you upgrade. It is also worth noting that it won’t give you the option to go from x86 to x64, although for my two 32 bit Windows 7 machines I had I have no real need to give them more than 4GB RAM, so it is no big deal.

All told, the installation took around 2 hours. It backs up the old installation to a folder called Windows.old, which can be deleted if you are sure the transition went correctly and none of your data is lost.

Windows 8 Usability

There has been a lot of fuss in the media about the loss of the start menu and that the touch-centric design of the Start screen and Metro apps would be confusing for new users, so I was interested to see how my wife and children got on with it. The answer is, just fine. They picked it up incredibly quickly. Within an hour or so they knew how to get to the start screen, how to sign in and out, how to change their account picture and colour scheme (most important to them!), how to organize live tiles, and how to install stuff from the store. Probably they don’t know how to use the charms or search for stuff, but they haven’t really needed to do that so far.

I also have to say that I have been won over to the new start screen despite my scepticism. It is a much better version of the start menu, with the ability to organize and pin stuff, better searching, and the live tiles are great for things like calendar and email notifications (as on my Windows Phone).

There were however a few examples of poor usability I encountered. I’m not sure why the search doesn’t automatically show you results in other categories if your selected category has no results. If downloading apps and updates stalls, you get very little feedback as to what is going on. I’d like to see bytes downloaded and remaining to help me troubleshoot. I occasionally got stuck in certain screens, like the store updates screen not letting me go back to the store front page, or the user settings screen not letting me out until I converted an account from Microsoft to local. Hopefully Microsoft are planning to fix a lot of these types of annoyances in updates soon.

Family PC

For my family PCs, there are two things I want. First, to be able to easily control what my children can access, and second, to sync as many settings between the PCs as possible to avoid having to manually configure accounts on each machine.

The first is well handled by Windows 8 family safety, which is essentially an improved Windows Live family safety. Its web filtering lets you choose by category and then add entries to a blacklist or whitelist. You can also control Windows Store apps by rating, but there doesn’t seem to be a blacklist or whitelist for apps, which is a great shame, as some apps I wanted to allow my younger children access to (e.g. OneNote), had a 12+ rating.

Family safety also has settings to control how many hours a child can be on the PC, and select times of day when they can use the computer. This is great, as we can stop them using the PC on weekdays from 8:00-8:30 when they should be getting ready for school. We can also limit them to 3 hours a day on the computer, and since the family safety website lets you link up local accounts, this should work even if they switch between computers during the day.

To get the benefit of syncing settings between PCs, you have to upgrade from a local account to a Microsoft account. I did this for my wife and eldest child who both have hotmail accounts. But it is a little less clear what Microsoft expect me to do for younger children. It would be nice if I could somehow enable their accounts for syncing but manage them via my Microsoft account.

The syncing is a little confusing. It wasn’t clear to me what exactly would be synced. For example, I was expecting my calendar settings to sync, but they didn’t seem to. There is no way to tell if settings sync has completed or not, so maybe I just needed to wait a bit longer. It also appears that installing a Windows store app on one PC does not automatically install it on the others, so it would at be nice if the Windows Store had a “my apps” area showing me apps that I had installed on at least one of my computers.

The next step is to take my development laptop through the same procedure, and then I can get to grips with seeing how much of NAudio will work with Windows RT.

Windows 8–First Impressions

I thought I’d post a few first impressions having actually used a Windows 8 machine for a day after setting my laptop up to boot Win 8 from VHD.

I like the fact that you can search just by typing in the start screen, although it seems a little unintuitive, and sometimes it appears to have found nothing, when actually it has discovered some matching applications and you need to click again to actually see the matches.

The new task manager is very cool. I’m especially pleased that you can bring a processes child windows to the front with it, which is something I used to need SysInternals Process Explorer for. It helps you to recover applications that appear to have hung because they are showing a message box or save as dialog that is not visible.

I’ve only briefly played with the default applications. I like the fact that the Sports one lets you specify your favourite team. It still needs a bit of tweaking as there are confusing ‘results’ shown for future fixtures, but its a good start. The Weather application using my location to find where I am is also a nice touch. I also like that it comes with a SkyDrive app by default as well.

My biggest annoyance so far is that when you use Alt-tab or Windows-tab to cycle through open applications, it includes the Metro apps as well. This is a pain because you can’t close Metro apps. If you’ve clicked through a bunch of Metro apps, then they will forever clutter up your alt-tab experience.

The Windows Store is rather sparsely populated, and once again it was not obvious to me how to search. Hopefully it will have good search capabilities as I doubt any app I create will ever be one of the top apps in any category.

The lack of a start menu in desktop mode still feels weird to me, and I still would like the metro interface to be available as a floating window in desktop mode, but I’ll give myself a bit longer to see if I can learn to live with this new way of working.

Installing and Booting Windows 8 RTM off a VHD (Virtual Hard Disk)

I discovered this week that simply having Visual Studio 2012 is not enough to write WinRT applications (Metro/Windows Store apps); you also need to have Windows 8 as your Operating System. Although the RTM Windows 8 is available now from MSDN, I’m not quite ready yet to switch my main laptop over to it, so a virtual machine is the way forward. Unfortunately though, Windows Virtual PC refuses to install Windows 8 and while Oracle VirtualBox can run it, performance isn’t exactly brilliant.

Fortunately, about a year ago, Scott Hanselman wrote a brilliant blog post describing how to use the boot to VHD functionality in Windows 7 to install Windows 8 as an alternative boot option, only running off a VHD you create. I was too nervous to try this out when he originally posted it, but I finally plucked up the courage and did my first Win 8 booting from VHD install today.

The instructions Scott gives are for the Windows 8 developer preview, so I thought I’d briefly review the steps, noting a couple of very small differences for the RTM and a few other things I noticed along the way:

Step 1 – Create a bootable USB stick
The instructions in Scott Hanselman’s post are very easy to follow if you have the Win 8 RTM ISO. Just download the utility and away you go. The space requirements are smaller – you should be able to get away with a 4GB USB stick, although I used a 16GB one.

Step 2 – Make a VHD
This is nice and easy using the “Disk Management” tool in Windows (or follow Scott’s instructions to use the DISKPART command-line). I went for an 80GB dynamically expanding VHD.

Step 3 – Write stuff down
Remember exactly where you put that VHD, and you might also want to have a Windows 8 product key handy. You’ll probably need a bunch of other passwords too, most notably your Windows Live one, and your wireless network.

Step 4 – Boot from the USB stick
Scott says F12 is the key you need to press to select a boot device, and it worked for me on my DELL XPS laptop.

Step 5 – Select new install
The Windows 8 RTM setup offers an upgrade option that the developer preview didn’t, so you need to say that you are doing a fresh install.

Step 6 – Attach the VHD within setup
Scott’s instructions are very good here. The keyboard shortcut you need to remember to get to a command prompt is Shift-F10. The other thing I noticed was that my Win 7 C drive was actually called the E drive, so if you have multiple disk partitions, find out which one your VHD is in before using DISKPART.

Step 7 – Select the VHD partition to install to
After you have refreshed the list of available partitions to install to, your VHD ought to be obvious (e.g. mine was the 80GB one), but take great care that you choose the right one. Scott’s instructions say that you will get a warning message before you proceed, but I didn’t, leaving me worried that I might have chosen the wrong one!

Step 8 – Let it reboot and finish setting up Windows 8
The process creates a very nice looking new boot menu when your computer starts up, which Scott shows screenshots of. This gives me the choice between Windows 8 and 7. Windows 8 is set the default, but you get to change the defaults if you like, so I set mine back to default to Win 7. Update: After a few reboots, my computer has switched back to using the boring Windows 7 text based boot menu. I'm not quite sure why this happened, but possibly was after I had a audio driver crash in Windows 7.

Step 9 – Accessing data between Win 7 and Win 8
One nice thing is that when you boot from the VHD, your Windows 7 drive containing the VHD gets mounted as a drive within Windows 8, making it easy to access any data (although you do need to let it change the folder permissions to look into your User folder). And vice-versa, on Win 7 you can easily use the Disk Management tool to mount the Windows 8 VHD and copy files on or off it as you wish.

Step 10 – Cleaning up
This is the step I haven’t done yet, but probably at some point I will decide I don’t want to use this VHD anymore. Obviously you wouldn’t want to just delete the VHD as you’d end up with an entry in your boot menu that won’t work anymore. Hopefully there is a way to take the Win 8 VHD out of the boot menu. I’ll update this blog post with a link if I find instructions somewhere.
From beginning to end, the process of setting up my laptop to boot to Win 8 from a VHD took me just over an hour, and I would recommend it to anyone wanting to get started with Win 8 development.

The Perils of Removing Features

In the last couple of weeks, Microsoft have released preview versions of two new software products. First up is Windows 8, the highly anticipated successor to Windows 7, sporting the brand new “Metro” interface, which promises to deliver a Windows UI suitable for tablet PCs. The other new product is Visual Studio 11, which debuts a number of key new features, including the next version of .NET and the ability to create Metro applications for Windows 8. All very exciting, and lots of us rushed to try out the preview versions as soon as possible.

Given that both products essentially are upgrades to existing and successful products, one would expect the reaction to these new products to be very positive. Both are loaded with cool new features and usability improvements, and yet the majority of the online chatter has been to complain.

What is the problem? As well as giving us new stuff, both products have taken something away. Windows 8 has taken away the Start menu, and VS 11 has taken away the colours from the icons. These two omissions, though relatively minor in the context of the overall changes, have managed to dominate the reaction to the new products.

One senses a little bit of frustration on the part of Microsoft (e.g. these two posts from Scott Hanselman), and you can understand why. All that hard work on new features has seemingly gone unnoticed, and all people can do is moan about two decisions that were intended as a bold step to improve user experience.

So what is the moral of the story? Users hate having features taken away from them. It doesn’t matter if you tell them that the feature removed was redundant, or poorly implemented. They feel robbed and short-changed. They feel like they are not understood. And they resent having to change the way of working they have been comfortable with.

Microsoft have a few options available. First, they stick to their guns, and say “you’ll thank us later, once you understand”. That may of course be the case, but is it really worth upsetting your customers over this? Second, they back down, and give us back our start menu and our coloured icons. A victory for the customer perhaps, but that is a recipe for never making any progress. The third option is to make it configurable. Let us choose if we want the old-fashioned start menu, or the colourful icons. If the new UX is truly an improvement, over time, people will be won over, and the obsolete features can be removed in the next version.

In summary, if you want to remove a feature from your application, even if you consider it to be an unnecessary one, consider first making it optional (whether that be on or off by default). The customer isn’t always right about what the best UX is, but they are right about what they like, and if they don’t like it, they won’t buy it. It will be very interesting to see which direction Microsoft take when the next preview versions come out. If they get it right, I’m sure they’ll get more bloggers talking about how cool these new products are, instead of moaning about what was taken away from them.

So … is Silverlight dead then?

Microsoft’s Windows 8 announcement has generated a surprising amount of controversy given that what was shown (a fancy new start menu that can be navigated using touch alone) was exactly what everyone was expecting. The only thing that could conceivably be described as a surprise was that Microsoft’s offering for the slate will be the “full” Windows OS, rather than the Windows Phone 7 OS adapted for a larger screen.

Apparently you will be able to make stuff for this new touch interface with HTML5 and JavaScript. Quite how that makes Silverlight dead I am not sure. I certainly don’t see any reason to panic. Given that it is full Windows, you will be able to write apps for it in VB6, Delphi or QBasic if you really want to. If you can’t write Windows 8 apps in Silverlight I’ll eat my hat. And if there is, as expected, a Windows 8 app store, you can be sure you’ll be able to sell Silverlight apps on it (in fact, if they base it on the Windows Phone app store, then Silverlight might even be the preferred/required dev platform).

I suppose it is possible that you won’t be able to make “tiles” using Silverlight (though they haven’t announced that you can’t), but I can’t get too worked up about that. I don’t want my applications consuming loads of memory and CPU cycles before I load them. Tiles should only be doing minimal processing.

Which brings us to the question everyone is asking. Is Silverlight dead?

When people call a technology “dead”, what they often really mean is “stable” and “mature”. I still develop WinForms apps on a daily basis. The API is effectively complete. It hasn’t had any meaningful new features for years. But it is a mature framework. It does what it does well enough, and if Microsoft were still actively trying to shoehorn new features into it that would be a bad thing. After a while, what is needed is not new features, but a new development paradigm.

WPF was, at first, that new paradigm, making it easy to do what was a nightmare in WinForms. And it followed a similar path to maturity. It’s got all the main capabilities you need to build impressive looking LOB apps. Microsoft got some things right with WPF, but like WinForms, it too has its limitations. I am not concerned that there isn’t a vast number of new features being announced for it every six months.

And guess what, Silverlight is on the same trajectory. Early releases picked up new capabilities at rapid pace. The fact it was chosen as the Windows Phone 7 dev platform has given it a new lease of life. But already it is reaching the stage where new versions are more incremental than revolutionary. That’s a good thing – the technology is approaching maturity.

To summarise: no need to panic. Keep building apps with the most appropriate technology that is available now. Those apps will continue to work just fine for years to come. Don’t worry that new technologies come along every few years that seem to “kill” older ones. I for one am glad that I am not currently developing using Visual Basic 16 or MFC version 23. And in 5 years time, I fully expect that there will be something even better than Silverlight to write Windows apps in. If there isn’t, it will be Microsoft, not Silverlight that is dead.