What are you trying to launch? I'm asking because launching notepad or mspaint, for example, worked just fine for me. I tried the following:
var a = ShellExecuteA(Game.Hwnd, "open", "c:\windows\notepad.exe", "", "", 8 );
and:
var a = ShellExecuteA(Game.Hwnd, "open", "c:\windows\system32\mspaint.exe", "", "", 8 );
and both opened the respective application in the background while the game remained on screen.
Did I miss something?
Btw, I used 8 (SW_SHOWNA) instead of 0 (SW_HIDE) as the nShowCmd parameter but I don't think that's what's causing your problem.