Saturday 4 July 2009

Audio WaveForm Drawing Using WPF

A while ago I blogged about how to display audio waveforms in WinForms. Porting this code to WPF is not as simple as it may first appear. The rendering models are quite different. In Windows Forms, you can draw points or lines using GDI functions. You are responsible for drawing them all every time the window is invalidated and its Paint method is called. However, in WPF, you create objects to put on a Canvas, and WPF manages the rendering and invalidation.

My first attempt was to stay as close to the WinForms model as I could. I have a sample aggregator that looks for the highest sample value over a short period of time. It then uses that to calculate the height of the line it should draw. Every time we calculate a new one, we add a line to our Canvas at the next X position, wrapping round if necessary, and deleting any old lines.

wpf-waveform-1

As can be seen, this gives a reasonable waveform display. I made use of a LinearGradientBrush to try to improve the visual effect a little (although this requires we cheat and keep the waveform symmetrical). There is a big performance problem however - it is very inefficient to keep throwing new lines onto a Canvas and removing old ones. The solution was to start re-using lines once we had wrapped past the right-hand edge of the display.

private Line CreateLine(float value)
{
    Line line;
    if (renderPosition >= lines.Count)
    {
        line = new Line();
        lines.Add(line);
        mainCanvas.Children.Add(line);
    }
    else
    {
        line = lines[renderPosition];
    }
    line.Stroke = this.Foreground;
    line.X1 = renderPosition;
    line.X2 = renderPosition;
    line.Y1 = yTranslate + -value * yScale;
    line.Y2 = yTranslate + value * yScale;
    renderPosition++;
    line.Visibility = Visibility.Visible;
    return line;
}

This solves our performance issue, but I still wasn’t too happy with the visual effect – it is too obviously composed of vertical lines. I tried a second approach. This added two instances of PolyLine to the canvas. Now, we would add a point to each line when a new maximum sample was created. Again the same trick of re-using points when we had scrolled off the right-hand edge was used for performance reasons.

wpf-waveform-2

As nice as this approach is, there is an obvious problem that we are not able to render the bit in between the top and bottom lines. This requires a Polygon. However, we can’t just add new points to the end of the Polygon’s Points collection. We need all of the top line points first, followed by all of the bottom line points in reverse order if we are to create a shape.

The trick is that when we get a new sample maximum and minimum in, we have to insert those values into the middle of the existing Points collection, or calculate the position in the points array. Notice that I create a new Point object every time to make sure that the Polygon is invalidated correctly.

private int Points
{
    get { return waveForm.Points.Count / 2; }
}

public void AddValue(float maxValue, float minValue)
{
    int visiblePixels = (int)(ActualWidth / xScale);
    if (visiblePixels > 0)
    {
        CreatePoint(maxValue, minValue);

        if (renderPosition > visiblePixels)
        {
            renderPosition = 0;
        }
        int erasePosition = (renderPosition + blankZone) % visiblePixels;
        if (erasePosition < Points)
        {
            double yPos = SampleToYPosition(0);
            waveForm.Points[erasePosition] = new Point(erasePosition * xScale, yPos);
            waveForm.Points[BottomPointIndex(erasePosition)] = new Point(erasePosition * xScale, yPos);
        }
    }
}

private int BottomPointIndex(int position)
{
    return waveForm.Points.Count - position - 1;
}

private double SampleToYPosition(float value)
{
    return yTranslate + value * yScale;
}

private void CreatePoint(float topValue, float bottomValue)
{
    double topYPos = SampleToYPosition(topValue);
    double bottomYPos = SampleToYPosition(bottomValue);
    double xPos = renderPosition * xScale;
    if (renderPosition >= Points)
    {
        int insertPos = Points;
        waveForm.Points.Insert(insertPos, new Point(xPos, topYPos));
        waveForm.Points.Insert(insertPos + 1, new Point(xPos, bottomYPos));
    }
    else
    {
        waveForm.Points[renderPosition] = new Point(xPos, topYPos);
        waveForm.Points[BottomPointIndex(renderPosition)] = new Point(xPos, bottomYPos);
    }
    renderPosition++;
}

This means that our minimum and maximum lines join together to create a shape, and we can fill in the middle bit.

wpf-waveform-3

Now we are a lot closer to the visual effect I am looking for, but it is still looking a bit spiky. To smooth the edges, I decided to only add one point every two pixels instead of every one:

wpf-waveform-4

This tidies up the edges considerably. You can take this a step further and have a point every third pixel, but this highlights another problem – that our polygons have sharp corners as they draw straight lines between each point. The next step would be to try out using Bezier curves, although I am not sure what the performance implications of that would be. Maybe that is a subject for a future post.

The code for these waveforms will be made available in NAudio in the near future.

16 comments:

Roland Tomczak said...

Hi ! I'm not sur that using the high-level geometry figures is the best approach in this particular case. I think that you should have define your custom Visual (Say, WaveformVisual), and override its Render method. This is the "lightweight" approach for drawing in WPF, and often recommanded for dynamic curves & so on ( eg : financial graphs,... ).

Unknown said...

hi Roland. thanks, that is useful information. Anything that improves performance will be useful. Will need to check that it works for Silverlight too though.

Choons said...

Wow- thanks so much for sharing your efforts! You make it all so clear. Consider me a new loyal viewer

Anonymous said...

When will you release this Code?

I'm looking for a Control to draw the Waveform of 8,16 and 24bit mono/stereo wave files.

I don't know where to start in WPF.

Unknown said...

hi Anonymous, have a look in the WPFDemo of NAudio.

Visualisation of entire WAV files is something that I am still working on. You can see a solution that works for short files at my new VoiceRecorder project on CodePlex (voicerecorder.codeplex.com)

Anonymous said...

wow, thank you very much!
this is so much of a help for me :D

Helmut Obertanner said...

Hello Mark,
Nice Post!
In JingleJim we use another approach for displaying waveforms.
We get the numsamples of the canvas. Then we can calculate the numSamples required for onbe pixel. For the num samples we calculate the avg-value. We split the view into let's say 10 Blocks and we convert the audiodata to a 8 bit waveform in a memory bitmap using a backgroundthread. Then we blitting the bitmap into the display - not drawing directly.
Much faster then drawing into canvas. Currently we are using Winforms, but should work the same for WPF.
NAudio is great!
Greets - Helmut

Unknown said...

hi Helmut,
yes, in some ways WaveForm drawing is easier in WinForms than WPF. I am wondering whether to try using a writeable bitmap at some point

Helmut Obertanner said...

Hi Mark,

yes - that will be our next aproach - currently we are converting our app to WPF.
I want to try different approaches and come back to you with my solution. Would you be interessted in having an WaveView/Selection Control in the NAudio-Package?

Greets - Helmut

Unknown said...

Hi Helmut,

that would be awesome. I have a selection control I made as part of the voicerecorder project on CodePlex, but it is far from complete

Mark

des said...

hey mark,
can you please provide the codes for displaying the waveform from recording(like the one you did in Voice Recorder) in WinForms?

Thanks a lot.

Unknown said...

@des - download the NAudio source code for an example of WaveForm drawing in WinForms

des said...

thanks for replying mark. i referred to Naudio, but there's some limitation to it. Like i cannot redraw the waveform(meaning to clear it and draw again), it cant scroll left and right if it gets too long, and it cant work with recording, it only does with playing. Any help? Thanks a lot.

Unknown said...

@des, I'm afraid the NAudio winforms waveform drawing doesn't have those capabilities - you'd have to write that yourself

Anonymous said...

Please tell me how to do this:
The way we achieve this is simply by starting recording and displaying the level of audio detected to the user with a volume meter. The waveIn APIs do not write anything to disk, so no audio is actually being ‘recorded' at this point, we are simply examining the input level and then throwing the captured audio samples away.
I just want to know where's the function for this. Thank you.

free said...

Thank you so much for posting this Just months and months for looking for some sort of clue. I had been toiling at a combined Tcl and Maxima application calling on some C scripts to make a fortran file. If any of that interests you have a look at Theo Verelst's work at wiki.tcl.tk, fascinating stuff I still hope to get it all to work. I have had success with a skinned transparent very simple wav player in VB and used sharpdevelop. I can't WAIT to get all into exploring this. Again thanx so much. freemenemoryerror@gmail.com