A friend of mine is working on a project that he's been having some major stability problems with. All of these issues seem to be related with interop to a Sybase database.
After working on and off with him for the past few months, it seems that the root of the problems lie in the fact that a previous developer had written some functions in a "shim" native DLL that calls intoo Sybase, so instead of P/Invoking the Sybase stuff, the developer created a wrapper DLL that he P/Invoked.
After looking at some of the wrapper two things were obvious. First, the Sybase functions were undecorated C functions, so the shim was unnecessary in the first place, and second, the developer's skill with C are terrible and the shim he created was likely the cause of the problems. So I recommended throwing out the shim, as it was garbage and it was completely unnecessary.
That lead to yet more questions - most notably "how do I marshal X or Y" type things along with "how the hell does one figure this stuff out" and "is this stuff documented anywhere?" Well other than experience I know of little for figuring it out, and once you get beyond the capabilities of the CF's marshaler I also know of know real documentation. So I decided I'd put together a fairly simple example of complex marshaling (yes, simple complex).
So first let's look at a typical scenario. The Sybase stuff often takes in pointers to big, ugly structures. These structures contain pointers. Sometimes they are pointers to ASCII strings. Well instead of wrapping one of these big nasty things, let's create a small representative example that you as a reader can expand upon.
The Native Side
Let's create a native DLL that exports one funtion that takes a fairly easy-to-grasp structure. The function will simply present a MessageBox with the info. So here's a structure:
typedef struct MESSAGE_INFO
{
char *message;
WORD length;
DWORD number;
} MESSAGE_INFO;
Note that 'message' is a char *, indicating ASCII. I'd never write a CE function like this, but this is an example of how to call an existing function, not a best-practices lesson. Note also that 'length' is only a WORD (2-bytes) making 'number' not DWORD aligned, so there is actually an unused 2 bytes in this thing as well. Again, I'm trying to show some ugliness that you will see in the wild, not cover how it should be done.
So let's look at the native function just as a reference. I'll not explain the whole thing as it's simple and the comments should answer most questions:
__declspec(dllexport)
void ShowMessage(MESSAGE_INFO *info)
{
TCHAR message[512];
// create a buffer for the Unicode string
wchar_t *wideString = (wchar_t *)LocalAlloc(LPTR, (info->length + 1) * sizeof(wchar_t));
// convert the ASCII message to Unicode
mbstowcs(wideString, info->message, info->length);
// show a message box with the data
_stprintf(message, _T("Message: %s\r\nNumber: %i"), wideString, info->number);
MessageBox(NULL, message, NULL, 0);
// free the Unicode buffer
LocalFree(wideString);
}
The Managed Side
Ok, so the native side is pretty straightforward. It's not ideal from a managed developer's perspective, but let's face it - native developers rarely think of making managed developer's lives easy when they write code.
We need to figure out how to create a MESSAGE_INFO "struct" and pass it in in a format that the API can use it. First, we can safely rule out letting the CF marshaler handle this. It would have no clue what to do with this. We can also rule out creating this as a managed struct. The interned char * is not blittable, so we'll have to hand-marshal this. The pointer itself also means we'll have to manage some memory manually too, and that also means "Danger Will Robinson!" as we have the potential for a memory leak. Don't fear that fact - just be aware of it.
So first, we need to understand what the native DLL is expecting. It wants a pointer to a struct - so simply put a 4-byte address. At that address will be 12 bytes of data representing a MESSAGE_INFO struct. The first 4 bytes of that is yet another address. At that address is ASCII character data. That's it. Now lets convert that English to C#.
First we'll create a MESSAGE_INFO class (we choose a class instead of a struct becasue we'll be doing some work internally on it). In the class we'll define a byte array that holds the 12 bytes that represent a native MESSAGE_INFO as well as some constants for member offsets and lengths for readability:
private const int MESSAGE_PTR_OFFSET = 0;
private const int MESSAGE_PTR_LENGTH = 4;
private const int LENGTH_OFFSET = MESSAGE_PTR_OFFSET + MESSAGE_PTR_LENGTH;
private const int LENGTH_LENGTH = 2;
// DWORD compiler alignment forces an unused 2-byte block here
private const int RESERVED_OFFSET = LENGTH_OFFSET + LENGTH_LENGTH;
private const int RESERVED_LENGTH = 2;
private const int NUMBER_OFFSET = RESERVED_OFFSET + RESERVED_LENGTH;
private const int NUMBER_LENGTH = 4;
public static int StructLength = NUMBER_OFFSET + NUMBER_LENGTH;
// in-memory representation of a native MESSAGE_INFO struct
private byte[] m_data = new byte[StructLength];
So when we call the API, we'll just pass it a byte array (a byte array is already a reference type and will get passes as a pointer) instead of a MESSAGE_INFO struct. Remember, the native side just wants a 4-byte number. All of the decorations we use of sturct names and all of that is simply for developer convenience. As long as we know that our m_data byte array is arranged to match a native MESSAGE_INFO, it will all work.
So our P/Invoke declaration looks like this:
[DllImport("NativeLibrary.dll", SetLastError = true)]
internal static extern void ShowMessage(byte[] messageInfo);
Simple, right? To call the native API, we simply call ShowMessage with our m_data bytes. Of course that's a bit ugly and non-managed, so we'll provide an operator to get the m_data without having to expose it explicitly:
// provide an operator to turn a MESSAGE_INFO into a byte array for marshaling
public static implicit operator byte[](MESSAGE_INFO mi)
{
return mi.m_data;
}
Now we can just call ShowMessage with a managed MESSAGE_INFO instance and we're done. So far it's all been pretty simple. The difficult part is actually populating the m_data bytes with the right info.
First, let's start with an easy one - the numeric data. The 'number' member is a publicly exported value, while 'length' is meant to be used internally as the length of the string data. So let's expose a Number property for the managed MESSAGE_INFO class, with set and get accessors. Instead of using a private member variable to hold the value, we'll hold it in the m_data array:
public int Number
{
set
{
// copy the "value" bytes to our in-memory representation
Buffer.BlockCopy(BitConverter.GetBytes(value), 0, m_data, NUMBER_OFFSET, NUMBER_LENGTH);
}
get
{
// get the number from our in-memory representation
return BitConverter.ToInt32(m_data, NUMBER_OFFSET);
}
}
Ok, so now the fun one. The string. Since we're passing just a pointer, we need to allocate memory for that pointer to "point to" plus we need to make sure that the GC isn't allowed to move that memory around in the event of a compaction. There are a few ways to do this, but the "recommended" mechanism is a GCHandle. A fixed unsafe pointer would also work, but it assumes you'll only use C# and that you know a bit more about pointers than you might.
So we'll add a GCHandle private member to the class, plus a byte array for our actual message (the memory that the GCHandle will point to):
// GCHandle for our message string
private GCHandle m_messageHandle;
// byte array for the actual message data
private byte[] m_messageData;
Now we need to expose a Message string property. In the set accessor we want to take the incoming string and put it's ASCII representation into the m_messageData member, get a GCHandle that points to it as m_messageHandle and then stuff that into our m_data. We also want to make sure that we don't leak if someone sets the message twice. The get accessor is a lot simpler - it just converts the m_messageData bytes back to a string. So the Message property looks like this:
public string Message
{
set
{
// see if we already have allocated data
if (m_messageHandle.IsAllocated)
{
m_messageHandle.Free();
}
// null check for safety
if (value == null)
{
m_messageData = null;
return;
}
// get the ASCII representation of the passed-in value (plus a null terminator)
m_messageData = Encoding.ASCII.GetBytes(value + '\0');
// pin it
m_messageHandle = GCHandle.Alloc(m_messageData, GCHandleType.Pinned);
// store the address of the pinned object into our 'struct'
IntPtr dataPointer = m_messageHandle.AddrOfPinnedObject();
byte[] pointerBytes = BitConverter.GetBytes(dataPointer.ToInt32());
Buffer.BlockCopy(pointerBytes, 0, m_data, MESSAGE_PTR_OFFSET, MESSAGE_PTR_LENGTH);
// store the length
short length = (short)m_messageData.Length;
Buffer.BlockCopy(BitConverter.GetBytes(length), 0, m_data, LENGTH_OFFSET, LENGTH_LENGTH);
}
get
{
// only return a string is something is allocated
if (m_messageHandle.IsAllocated)
{
// create a string, stripping the null terminator
return Encoding.ASCII.GetString(m_messageData, 0, m_messageData.Length).TrimEnd(new char[] {'\0'});
}
else
{
return null;
}
}
}
Lastly we need to make sure our class itself doesn't leak when it's destroyed. My solution is simple - a Finalizer:
~MESSAGE_INFO()
{
// free the GCHandle if necessary
if (m_messageHandle.IsAllocated)
{
m_messageHandle.Free();
}
m_messageData = null;
}
Yes, implementing Dispose might be a better way to go, but this is less code, and again we're looking at marshaling, not implementing best practices (and I see nothing wrong with a Finalizer for this anyway).
That's it. You're done. When run, you'll get something like this:
Download the full solution source here (note I included several SDKs and built and tested with one you probably don't have, so you might need to hack up the project file if it complains).