Native Extensions for Silverlight (NESL)?

Silverlight applications are written as managed code which runs on a CLR which sandboxes the code.

In Silverlight 1 and 2 this was easy to understand because all applications were running in a browser and it’s common to expect code running in a browser to be security sandboxed such that (for example) a Silverlight application could not read a file from anywhere on the hard-drive unless the user had first consented to a file dialog requesting access to that specific file.

Equally, the Silverlight application could only raise that file dialog if the user had caused it to be raised by some direct interaction with the UI.

It’s fairly simple once you get used to it although an early area for confusion was for developers coming from the “full” .NET Framework and CLR who expected the sandbox to be configurable via policy and so on whereas it isn’t with Silverlight.

With Silverlight 3, things initially seemed a little more confusing because applications could now run as “Out Of Browser” (OOB) which opened the question as to whether that meant that they were running with less sandboxing but, in reality, they weren’t and the situation was unchanged.

With Silverlight 4 things did change in that an out-of-browser application could mark itself as wanting to be “trusted” at the point where it is installed which meant that the user then sees one of 2 dialogs;

depending upon whether the application has been signed or not (verified or unverified in the terminology) and, in the latter case the application is free to display a nice logo and also to take part in Silverlight’s out-of-browser automatic update capability so going verified is highly recommended.

Regardless, if the user installed the “trusted” application then it now ran in a relaxed sandbox where it got more permissions as below taken from one of my own PowerPoint slides on the topic;

This is true cross-platform – i.e. for both Windows applications and OS X applications. However, there is still a sandbox in place and, for instance, an application still can’t just go and read the contents of c:\temp\ even if it is running in a trusted environment.

However, there’s one extra piece of the puzzle which is that on Windows only the sandbox has a hole cut into it and that hole is COM shaped;

and, because there’s no trust model for COM components this effectively means that a Silverlight application running out-of-browser and trusted on Windows has full access to the machine and can go and read the contents of c:\temp\ if it can find a scriptable COM component to do its dirty work for it

That’s great if;

1. You were happy sacrificing the cross-platform requirement.
2. Your users were happy to installed a “trusted” application.
3. You could find a COM component already installed on the machine that did what you wanted that exposed its capabilities via IDispatch.

So, for example, it’d be a pretty safe bet if you wanted some kind of Office 2010 interoperability with Outlook, Excel or similar.

However, there’s a heck of a lot of Windows functionality that isn’t exposed via well-known-COM-components and so you’re left pondering what the best route is to take.

The “full” .NET framework/CLR already solved this problem in that, from day 1, it offered interoperability with both existing COM components (via more than just IDispatch) alongside interoperability with existing Windows DLLs via Platform Invocation (or PInvoke).

With Silverlight 5 (see keynote) it sounds like there will be a couple of new developments in this area;

1. Silverlight 5 looks to also be getting PInvoke capabilities so that you’ll be able to call into DLLs.
2. Silverlight 5 looks to be be getting the ability to run a trusted application inside the browser subject to some criteria being met about the application’s provenance.

That’s great but, in the meantime, there’s an “interesting” development over at the MSDN Code Gallery called the Native Extensions for Silverlight (NESL) which has taken an approach of making specific APIs easier to call from Silverlight. The approach is;

1. Write COM components exposing the functionality of the APIs that you want via IDispatch.
2. Write Silverlight library wrapper classes that make calling those COM components easy.

and the particular APIs covered at the time of writing look to be;

1. Sensors
2. Speech
3. Portable Devices
4. Taskbar Integration
5. Webcam encoding and window capture
6. Low Level Windowing APIs (i.e. WM_* style messages)

and ( from the discussions forum ) it looks like this has been tested/built for Windows 7 only at this point ( makes sense for some of those APIs are they are Windows 7 specific ).

If a Silverlight application is going to take a dependency on a library like this then it means that the COM components have to be pre-installed ( which I think requires admin rights ) and that’s done one of two ways by this library;

1. Have the user pre-run some .MSI installer.
2. Have the application use provided library classes which, effectively, provide a programmatic way to run the installer.

That’s quite a big deviation from how a standard Silverlight application operates plus there’s then also the consideration around how you would then version this kind of dependency as it changes over time.

That does lead me to something of a “sidebar”…

Sidebar – Should We Be Doing This Stuff?

My view on the COM interop functionality of Silverlight 4 has always been;

use sparingly

I’d say that if you’re writing an application with Silverlight and you find that there’s some small, well-contained bits of functionality that you can add by (e.g.) integrating with Outlook or offering an “Export to Excel” piece of functionality then that’s fine but if you’re writing reams of interop code then I suspect you’re using the wrong technology and you probably need to start thinking about this guy;

That’s what I’d recommend. You get all the goodness of .NET 4.0 plus you have all the flexibility of COM interop and PInvoke and you can go with whatever installation experience works best for the application.

So, generally, that would be the direction I’d take rather than getting into the distribution of COM wrappers that are likely to cause headaches down the line.

Trying Out the NESL

With that said, I thought I’d try out the NESL anyway and see how it works out and so I downloaded the bits in source form rather than runtime form. Downloading that way gave me;

so the runtime is also provided and I opened up the VS solution file, unloaded the projects within the samples folder and then built the rest of it;

and hit a post build error which I figured out was because the project has been set up to copy its outputs into a folder;

SilverlightLibraries\Binaries\Debug ( or release )

and the Debug part doesn’t exist so the post-build steps fail until you create that folder.

With everything built, I thought I’d try out the installation experience and so I made a new Silverlight project, set it up to be an elevated out-of-browser application and then added a reference to the NESL Microsoft.Silverlight.Windows assembly.

Next step is to add the MSI of the installer for NESL to my project so I added it in and set its build action to be Content (note – this is just one way of getting the MSI installer run );

bundling an MSI file into a Silverlight XAP felt a little weird so after a stiff drink I carried on and put a button onto a form and wrote a tiny bit of code behind it;

      if (!Microsoft.Silverlight.Windows.Installer.CheckNESLInstalled(1, 0)){Microsoft.Silverlight.Windows.Installer.InstallNESL(new Uri("NESLSetup.msi", UriKind.Relative),false, // is the installation remote?true); // do I want to show progress? You BET!MessageBox.Show("NESL Installed!");}

and then I ran it and clicked the button and saw;

and then;

oh my. Time for another stiff drink before hitting the Yes button which leads to;

and then my own MessageBox (note that the installation process was synchronous);

Ok, I’m installed and I note that clicking on my button again has no effect as I would hope for now that the installation is done.

Now time to try out an API or two. I figured Speech would be a good place to start so it’s time for a TextBox and a Button;

and a reference to the NESL library for speech;

and then I played around with a few bits of code. This one was to figure out what voices are available;

      SpeechSynthesizer synth = new SpeechSynthesizer();foreach (var item in synth.GetVoices()){string speech = string.Format("I have a voice called {0} which is {1} years old and is {2}",item.Name,item.Age.ToString(),item.Gender.ToString());synth.Speak(speech);}    

on my system this says;

“I have a voice called Microsoft Anna which is Adult years old and is female”

and getting default speech is crazily easy;

  SpeechSynthesizer synth = new SpeechSynthesizer();synth.SpeakAsync(txtSpeech.Text);

I figured I’d have a play with TaskBar integration and so I went ahead and brought in that library and made myself a very quick “media player” in Silverlight and added some bits of code to set up the task bar;

firstly setting up icon overlays for when the video is playing or paused;

    void ChangeTaskBarIconOverlay(){byte[] iconBits = this.GetImageBitsForCurrentPlayState();TaskbarButton.Current.SetOverlayIcon(iconBits, "Play State", ButtonImageDataType.PNG);}

where the GetImageBitsForCurrentPlayState function just returns a byte[] of a 16x16 PNG resource based on whether the video is currently playing or paused.

I can also update the progress indicator as the video plays ( this function is called on a timer tick );

 void OnTick(object sender, EventArgs e){if (this.playState == PlayState.Idle){// NB: This line of code fails...maybe a bug?// TaskbarButton.Current.SetProgressState(TaskbarItemProgressState.None);// Doing the dynamic dispatch myself...TaskbarButton.Current.COMObject.SetProgressState((int)TaskbarItemProgressState.None);}else{TaskbarButton.Current.COMObject.SetProgressState((int)TaskbarItemProgressState.Normal);TaskbarButton.Current.SetProgressValue((ulong)this.mediaElement.Position.TotalSeconds,(ulong)this.mediaElement.NaturalDuration.TimeSpan.TotalSeconds);}}

and that gives me an overlay and progress indication on the task bar;

and I can also create Play/Pause/Stop buttons on the taskbar – as an example for the Stop button;

   void CreateTaskbarButtons(){ThumbbarButton button = TaskbarButton.Current.CreateThumbbarButton(1);button.Tooltip = "Stop";button.ImageDataType = ButtonImageDataType.PNG;button.Image = GetImageBitsForResource("SilverlightApplication11;component/StopHS.png");TaskbarButton.Current.ShowThumbbarButtons();}

which gives me;

but I was unsure of how to handle the button click event as it doesn’t seem to be surfaced from the wrapper and the source code doesn’t seem to include the source code for the COM objects that the libraries are wrapping so that led me to a bit of a dead end .

However, the docs revealed that in order to handle the button click you have to use the interop libraries to pick up a WM_COMMAND message. I left this as an exercise

The other API which felt like it would be of interest was the video encoding API which offers the chance to output H.264 video from video/audio captured from a Silverlight webcam and microphone.

This is “non trivial” ( practically impossible ) in Silverlight 4 today and so this API offers some possibility for people wanting to do that kind of capture.

There’s a pretty complete sample of this in the docs for the NESL API and I won’t experiment with it for this post because writing Video/Audio sinks for the webcam/mic is quite a lot of code ( there are some samples along those lines on this blog site from when Silverlight 4 first came out ) and so this post would become huge but this might be one area where you look to the NESL for help.

Wrapping Up

It’s interesting to see the NESL libraries and experiment with them a little and it’s clever stuff but I suspect that it’s not for most Silverlight applications because of its need for (an admin driven) pre-installation and because it doesn’t quite fit into the model for updating a Silverlight application. I’m also not sure how it works in a 32/64-bit environment.

Generally, I’d say if you need this kind of functionality then you’re perhaps better looking to WPF but I’m sure there will be certain cases where Silverlight developers will pick this library up as it gives them “just that one additional capability” that their app needs so it’s an interesting development for those edge cases until Silverlight 5 comes along and offers PInvoke capabilities.