# Wednesday, August 23, 2006

We've had an idea for a new project for some time, and we've finally decided to put out a code seed for it to see if the community wants to get involved.

The idea is this - Create a native coredll.dll library for the desktop that exactly matches the one exposed by Windows CE (same funtions at the same ordinals).  In theory, this would allow you to run Compact Framework applications against the full framework including code that P/Invokes.  The long-term goal is to implement every funtion (there are about 1800 of them, we've seeded the project with 50), but the milestone I'm shooting for is to get this library to a point that the SDF will run on the desktop.

Why would you want this library you ask?  The answer is fairly simple - to help in debugging and unit testing.  I don't envision you shipping a product that runs on CE and XP, but I do see great value in being able to run your CF assemblies through NUnit or Team Suite unit tests, which today cannot be done on a device.  This project is an enabler for that.

THe project is located at CodePlex as the OpenNETCF Advanced Debugging Toolkit.  Look for more pieces to the toolkit as time progresses.

Wednesday, August 23, 2006 1:00:11 PM (Central Daylight Time, UTC-05:00)  #     | 
# Tuesday, August 22, 2006

The Bitmap class in the Compact Framework is a confusing thing, largely because it has abstracted what the OS is doing underneath a little too far.  For example, look at the following code:

Bitmap bmp1 = new Bitmap(fileStream);
Bitmap bmp2 = new Bitmap(200, 200);

Let's assume that fileStream is a valid stream to a resource bitmap file that is 100x100 in size.  So is there any difference between bmp1 and bmp2, other than the fact bmp1 presumably has some color data in it?  The answer is yes - there's a very big difference, and that difference can have a huge impact on application performace as well as cause exceptions.

So let's look at this a little deeper with some examples.  Here's the first:

int iterations = 0;

while (true)
{
  try
  {
    iterations++;
    Bitmap b = new Bitmap(GetImageStream());
    if (iterations % 100 == 0)
    {
      Debug.WriteLine(string.Format("{0} objects", iterations));
    }
  }
  catch
  {
    Debug.WriteLine(string.Format("Failed after {0} objects", iterations));
    Debugger.Break();
  }
}

If I run this code (GetImageStream() just pulls an image from an embedded resource), the app will run forever, occasionally spitting out the debug port how many hundreds of objects it's created.  If you run RPM on it you'll see memory getting allocated, the GC firing occasionally and resources being freed up.  All is well in the world of managed code and everything is working as expected.  Hooray.

Now let's change that ever so slightly to this:

int iterations = 0;

while (true)
{
  try
  {
    iterations++;
    Bitmap b = new Bitmap(200, 200); // <--- CHANGED HERE
    if (iterations % 100 == 0)
    {
      Debug.WriteLine(string.Format("{0} objects", iterations));
    }
  }
  catch
  {
    Debug.WriteLine(string.Format("Failed after {0} objects", iterations));
    Debugger.Break();
  }
}

Note the single change where the bitmap is created.  Try running this and after a few iterations - the exact number depends on available device memory - it will OOM (throw an out of memory exception).  On the device in front of me it was about 40.

So the first thing to do is theorize why this would happen.  Seems like the Bitmap's resources aren't getting freed after it goes out of scope at the end of the while block.  An explicit call to Dispose() may solve it if that's the case, so let's try another test.

int iterations = 0;

Bitmap b = null;
while (true)
{
  if (b != null)  // explicit disposal
    b.Dispose();

  try
  {
    iterations++;
    b = new Bitmap(200, 200);
    if (iterations % 100 == 0)
    {
      Debug.WriteLine(string.Format("{0} objects", iterations));
    }
  }
  catch
  {
    Debug.WriteLine(string.Format("Failed after {0} objects", iterations));
    Debugger.Break();
  }
}

Sure enough, when we run this, it behaves like the first.  Strange that the Bitmap behaves differently depending on which constructor we use - this is contrary to common sense, right? 

So let's think a little more.  A Bitmap has a large area of unmanaged resources and some managed resources.  It seems that when we create a bitmap using the size ctor, the finalizer doesn't get run when an OOM happens.  Let's test again and see if that really is what's going on. We'll remove the explicit Dispose call and wait for the finalizers and try again when we OOM.

int iterations = 0;

while (true)
{
  try
  {
    iterations++;
    Bitmap b = new Bitmap(200, 200);
    if (iterations % 100 == 0)
    {
      Debug.WriteLine(string.Format("{0} objects", iterations));
    }
  }
  catch (OutOfMemoryException)
  {
    Debug.WriteLine("Waiting for finalizers to run..."));
    GC.WaitForPendingFinalizers();
    Bitmap b = new Bitmap(GetImageStream());
  }
}

When we run this one, again all is well in managed code land, though we see it waiting for finalizers to run a lot, and that catch is an expensive one for perf (as all exceptions are). At this point I think "well that surely has to be a bug" but I often like a second opinion, so I went right to the source and asked the CF team about the behavior.  The response from them is actually quite informative.  Their response in in italics below.

I think you are probably seeing is several interactions that can be quite confusing.

  1. Creating a bitmap using the stream constructor will construct a DIB (Device Independent Bitmap).
  2. Creating a bitmap using the width/height constructor will construct a DDB (Device Dependent Bitmap).
  3. DIB's are allocated out of the virtual address space of the application.
  4. DDB's are allocated by the driver. This typically means that they are allocated in the virtual address space of gwes.exe. Alternatively, the driver could allocate these in dedicated video ram.
  5. Creating a bitmap with the stream constructor will generate a fair amount of garbage as it copies data from one buffer to the other.

When we perform a GC because of an OOM in the stream constructor case, we will almost certainly have some amount of garbage that we can free back to the OS immediately. This will also trigger the finalizer to run on another thread as soon as possible. That should help the next call to bitmap creation.

When we perform a GC because of an OOM in the width/height constructor case, it is fairly likely that the OOM is caused because of virtual memory exhaustion in gwes.exe. Thus freeing memory in our process will not help the memory condition in gwes.exe. We need the bitmap finalizer to run before this would actually free memory in a way that would help this scenario. While the finalizer thread would certainly have been triggered to start, it most likely will not get a chance to free bitmaps before we OOM while trying to allocate a bitmap immediately after triggering a GC on the initial thread.

In short, we have 2 different types of Bitmap in our runtime with varying performance and allocation characteristics. DDBs are generally faster to manipulate and draw to the screen than DIBs, but they are constructed in an external memory space that can cause allocation confusion and cause the performance of calls to LockBits or Save to be slow. If a DIB is desired and you wish to construct it based on width and height, we provide a function that constructs a Bitmap with a width, height, and pixelformat specified. This function will construct a DIB instead of a DDB.

I personally still consider this a bug in the implementation - the CF should catch these occasions and handle it for us rather than OOMing all the way back to the app to wait for the Finalizers and retry - that's an implementation that should be done below us. 

Still the answer sheds light on the fact that how we create a Bitmap should be highly dependent on how we intend to use that Bitmap.

Tuesday, August 22, 2006 1:24:50 PM (Central Daylight Time, UTC-05:00)  #     | 
# Friday, June 30, 2006

Check this out!  Software stored on 12-inch vinyl.  That's insane.

Friday, June 30, 2006 12:29:24 PM (Central Daylight Time, UTC-05:00)  #     | 

It occurred to me today that I never posted my presentation on Memory Management in the CF from MEDC 2006, so here it is.

Friday, June 30, 2006 10:42:23 AM (Central Daylight Time, UTC-05:00)  #     | 
# Thursday, June 29, 2006

Alex Feinman found this interesting reference to us.  Evidently University of California at Berkeley "officially" recommends us in their CS160 curriculum.  How cool is that?  Have you seen the SDF on campus anywhere?  Let us know.

Thursday, June 29, 2006 10:45:31 PM (Central Daylight Time, UTC-05:00)  #     | 

Since most people like to try before they buy, we've released Evaluation Versions of our Calendar Controls.  Download the evaluation binaries and a sample project using them here.

Thursday, June 29, 2006 11:37:14 AM (Central Daylight Time, UTC-05:00)  #     | 
# Tuesday, June 27, 2006

First let me go on record as saying that I think using a 'z' to terminate words is utterly moronic, like using the number 2 instead of the word 'to'.

Anyway, on 2 my skillz....

Last week I posted a fantastic kludge for turning on the Bluetooth radio on an Axim X30.  Well, like any kludge, as soon as it shipped it broke.  Turns out the notification icon isn't always in the rightmost position - it can move.  That screwed with my intricate algorithm of moving in and up 10 pixels from the lower left corner of the screen.

So what's a developer to do?  First, let's take a quick detour into how those icons work. 

They "tray" icons are actually called Notification icons and they are displayed by calling the Shell_NotifyIcon API.  When the icon is created you provide a window handle for it to notify when it's clicked.  The icon itself doesn't have any other abilities.  This is a critical piece of info for this hack.

Since I know it's posting messages to another Window when it's clicked, I simply needed to figure out exactly what it's doing.  Time to break out Remote Spy++ in eVC (are you in the group that never really knew what the hell that tool was used for? This is a classic case).

Loaded up Spy++ and I see a Window conspicuously named "Bluetooth Console" - that's promising.  I put a watch on it and sure enough, when I tap the icon, messages get posted to that window (off is in the blue box, on in the red).  Now all I need to do is post the same messages.

So first, I need the handle for that Window.  Time for the FindWindow P/Invoke:

IntPtr btWindow = FindWindow("WCE_BTTRAY", "Bluetooth Console");

Next, replicate the messages the tap generates:


SendMessage(btWindow, WM_USER + 1, 0x1267, 0x201);
SendMessage(btWindow, WM_USER + 1, 0x1267, 0x202);
SendMessage(btWindow, WM_USER + 1, 0x1267, 0x200);

That causes the Bluetooth Console to create and show the popup menu. Now it needs a message to tell it to wait for a menu tap:

SendMessage(btWindow, WM_ENTERMENULOOP, 0x01, 0x00);

Now "generate" the tap:

SendMessage(btWindow, WM_COMMAND, BluetoothRadioState ? CMD_BT_OFF : CMD_BT_ON, 0x00);

And tell it to quit listening for menu taps:

SendMessage(btWindow, WM_EXITMENULOOP, 0x01, 0x00);

It does something else that I can't tell what the effect is, but since it's doing it, I will too:

// not sure what this does, but physically clicking does it, so replicate it here
SendMessage(btWindow, WM_USER + ((BluetoothRadioState) ? (uint)0xC00D : 0xC00C), 0x01, 0x00);

And finally get the Menu window and hide it:

IntPtr btmenu = FindWindow("MNU", "");

SendMessage(btmenu, WM_DESTROY, 0x00, 0x00);
SendMessage(btmenu, WM_CANCELMODE, 0x00, 0x00);

While it's still ugly, it's a bit cleaner than the original, and much smaller.  This is our new class in its entirety:

using System;
using Microsoft.Win32;
using System.Runtime.InteropServices;
using System.Windows.Forms;

namespace OpenNETCF.Devices
{
  public static class AximX30
  {
    public static bool BluetoothRadioState
    {
      get
      {
        RegistryKey key = Registry.LocalMachine.OpenSubKey(@"SOFTWARE\WIDCOMM\BtConfig\General");
        bool currentState = (((int)key.GetValue("StackMode")) == 1);
        key.Close();
        return currentState;
      }
      set
      {
        // see if any action is needed
        if (BluetoothRadioState == value)
        {
          return;
        }

        IntPtr btWindow = FindWindow("WCE_BTTRAY", "Bluetooth Console");

        // pop up the menu
        SendMessage(btWindow, WM_USER + 1, 0x1267, 0x201);
        SendMessage(btWindow, WM_USER + 1, 0x1267, 0x202);
        SendMessage(btWindow, WM_USER + 1, 0x1267, 0x200);

        // give it time to create the menu
        System.Threading.Thread.Sleep(100);

        // find the menu that popped up
        IntPtr btmenu = FindWindow("MNU", "");

        // start the window listening for menu messages
        SendMessage(btWindow, WM_ENTERMENULOOP, 0x01, 0x00);
        // send it the on or off message
        SendMessage(btWindow, WM_COMMAND, BluetoothRadioState ? CMD_BT_OFF : CMD_BT_ON, 0x00);
        // tell it it's done listening
        SendMessage(btWindow, WM_EXITMENULOOP, 0x01, 0x00);

        // not sure what this does, but physically clicking does it, so replicate it here
        SendMessage(btWindow, WM_USER + ((BluetoothRadioState) ? (uint)0xC00D : 0xC00C), 0x01, 0x00);

        // now hide the menu
        if (btmenu != IntPtr.Zero)
        {
          SendMessage(btmenu, WM_DESTROY, 0x00, 0x00);
          SendMessage(btmenu, WM_CANCELMODE, 0x00, 0x00);
        }

        return;
      }
    }

    [DllImport("coredll.dll")]
    private static extern IntPtr FindWindow(string lpClassName, string lpWindowName);

    [DllImport("coredll.dll")]
    private static extern int SendMessage(IntPtr hWnd, uint msg, int wParam, int lParam);

    private const uint WM_DESTROY = 0x02;
    private const uint WM_CANCELMODE = 0x1F;
    private const uint WM_USER = 0x400;
    private const uint WM_ENTERMENULOOP = 0x0211;
    private const uint WM_EXITMENULOOP = 0x0212;
    private const uint WM_COMMAND = 0x0111;

    private const int CMD_BT_OFF = 0x1001;
    private const int CMD_BT_ON = 0x1002;

  }
}

Tuesday, June 27, 2006 2:58:19 PM (Central Daylight Time, UTC-05:00)  #     |