I have been working on a small project that ports Python’s ctypes functionality to Mozilla’s XPCOM system. For those unfamiliar with ctypes, its a Python module that allows developers to declare and call exported methods from binary/shared libraries. Developers can then wrap these binary calls in pure Python.
The JavaScript port, called js-ctypes, will give developers this same kind of functionality in Mozilla’s privileged JavaScript. The idea for the library came after helping many extension developers through the process of building binary (C++) XPCOM components. The story usually went something like this:
- Developer wants functionality not built into the Mozilla platform, but easily supported by the native OS.
- Developer wants to use a 3rd party library in an extension or xul app.
- Developer has methods in native code for performance reasons.
The usual answer to these problems is creating a binary (C++) component to act as a wrapper so the native features can be exposed to JavaScript via XPCOM. However, the process of building binary XPCOM components is significantly harder than JavaScript components. The macros and memory management rules of binary components are harder than JavaScript, but its really the compiler and linker flags, IDL, makefiles and SDK versions that make the process painful. The build process must be repeated for each platform too. For many cases, its just too much work.
The goal of js-ctypes is to allow developers to declare methods in binary libraries and then expose those methods as callable JavaScript functions. Developers can then create pure JavaScript library wrappers around binary libraries – without making binary XPCOM wrappers.
Here is a really simple example of using Window’s native MessageBox call from JavaScript:
const nsINativeType = Components.interfaces.nsINativeType;
function msgbox() {
var library = Components.classes["@mozilla.org/js-ctypes;1"]
.createInstance(Components.interfaces.nsINativeLibrary);
library.open("user32.dll"); // Load the shared Windows DLL
var messageBox = library.declare(
"MessageBoxW", // name of the exported method
nsINativeType.INT32, // return type
nsINativeType.INT32, // parent hwnd (its a Window's thing)
nsINativeType.WSTRING, // Unicode message string
nsINativeType.WSTRING, // Unicode title string
nsINativeType.INT32 // bitflag for buttons to show
);
var ret = messageBox(0, "This is the message", "Msg Title 1", 3);
alert(ret);
// You can reuse the method
var ret = messageBox(0, "This is another message", "Msg Title 2", 3);
alert(ret);
}
The js-ctypes code uses the same libffi library as the Python ctypes module. It is currently known to work on Windows and compiles on Linux, but I haven’t got around to testing it. We need to start the Mac implementation. We also need to add support for struct and more primitive types.
The code is in SVN here and we have a Bugzilla component too (see bugs, file bug).
Note: This functionality cannot be accessed from JavaScript used in web content. Only JavaScript that runs with chrome privileges (extensions and Firefox UI for example) can use js-ctypes.
Thank you for ushering in a new era of fear.
ActiveX for OpenSource systems! Yesssssss
Would it be possible to test through a Firefox trunk debug build? If so, any pointers on where to put it in the tree. In extensions/ or modules/ perhaps? (I’ve managed to build Fx with PyXPCOM even, but it’d be a stretch claiming I know my way around the tree.)
Wouldn’t mind doing some testing, perhaps writing some unit tests or things of that nature. Using Linux btw.
This sounds great Mark –
for me, binary components have been a stumbling block in trying to have better OS integration with my extensions
Keep us updated
Anonymous Cow – I added the note about privileged JavaScript for you, so take your medication and relax.
Fredrik – Eventually you’ll be able to test using Firefox (debug or release), but js-ctypes is _not_ part of the Firefox source tree. You’ll need to pull it from the Mozilla SVN tree and then build it as part of a Firefox (or other project) build. I’ll get a wiki page started with build and contribution instructions.
Yeah, I pulled the sources from SVN already (and I have a semi-current trunk tree from before). I figured I could plop the js-ctypes dir somewhere in the tree and build away. Just not sure where in the tree to put it, and if I need to tweak a Makefile to make the build system pick up. (The Wiki page on C++ extensions seems to fit reasonably well, but not perfectly.)
If I was less tired (1.30AM CET) and my computer wasn’t dog slow (a full build takes 6hrs with ccache) I’d be trial-and-erroring already.
That’s totally rad.
Any excuse to be able to stay in JavaScript sounds great to me. Perhaps this will get me to restart some of my moz/gnome integration projects – the elusive seg-faults tend to discourage me.
Sounds like you’re making it much easier to use a whole variety of libraries out there that aren’t already exposed to JS through XPCOM. Is that correct? If so sounds like you are really building another bridge between the world of web developers who should have some JS ability, and ‘desktop’ functionality. In other words, blurring the line between the two again 🙂
As a web developer with JS abilities who is very interested in extension development and XULrunner application development, I thank you!
As another example, could js-ctypes be employed to facilitate access to libraries like ffmpeg
Sounds like you’re making it much easier to use a whole variety of libraries out there that aren’t already exposed to JS through XPCOM. Is that correct? If so sounds like you are really building another bridge between the world of web developers who should have some JS ability, and ‘desktop’ functionality. In other words, blurring the line between the two again 🙂
As a web developer with JS abilities who is very interested in extension development and XULrunner application development, I thank you!
Just to clarify, could js-ctypes be employed to facilitate access to libraries like ffmpeg?
Your Idea sounds very cool.
A slightly OFF-Topic question. Is it possible to build under Linux cross platform XPIs?
Its really Painful to boot Linux to build and test the Linux XPCOM and the same for Windows.
And for OS X which I doesn’t own, i can’t build any XPCOMs.
I ask this here because the libaries you load are not XPCOMs but still platform dependent (The good things is in some cases you have no dependency to th moz SDK, if I understand correct).
Who needs typesafety anyway?
So this will add to the list of things that are accessible from JS and JS only, right? Note that one of the most common issues people have with binary code in Firefox is how to access the tabbrowser, which is another example of JS-only API…
Biesi – I am hoping we can do some kind of type checking. Python’s ctypes module does have a sort of type checking feature.
Boris – Yes, this is really to be used by JS to make accessing binary code easier. If you are already in C++ to begin with, you shouldn’t need this feature.
Mark – just wanted to salute you for yet another extremely important project.
I can testify first hand that this exact functionality has been very important ingredient in many projects I’ve done in the past – for components written in Tcl/Tk, Perl, C# etc.
I believe that introducing this into Mozilla/JavaScript will really take the whole platform to a new level by removing the binary XPCOM barriers and allowing true rapid development and reuse of existing binary components. I actually started working on the exact same thing a while ago, but had to stop because it looked like quite a lot of work and I just couldn’t allocate the time to see it through.
One suggestion – I think that it’s very important to ship this with predefined definitions of all existing win32 functions (and maybe system library equivalents on other platforms). This will allow developers to quickly use any system service without having to create the needed definitions, which can sometimes result in mistakes and bugs.
Anyway, great job – I’m looking forward to seeing this as an integral part of the Mozilla platform!
Mark, if I’m in C++ in an extension and want to access Firefox code or extension code in another extension, and the API is only exposed to JS (because this wrapper is being used to access the underlying C++), then I get screwed: I have no idea what those libraries are, or whatnot, and more importantly can’t depend on it. Again, this is exactly the situation we’re in with tabbrowser: one _can_ use JSAPI to get to it from C++, but it’s a world of hurt.
This is hot, hot, hot!
Just to clarify, I think this is a great idea. I’m just worried that we need to either do something similar the other direction, come up with some other way to improve information flow back from JS into C++, or abandon the idea of C++ as a platform development engine and stick to only using it for the core layout code.
It seems this last is the direction we’re moving toward….
Boris, if you’re native code you can just link directly against the library you’re trying to access (or use the PR_LoadLibrary dynamic equivalents). How would this API be useful from C++ when you can do the binary call directly?
And yes, I believe we are moving away from a C++ platform to a JS platform.
This will radically simplify XUL development for a lot of programmers. Very impressive!
I’m really looking forward to seeing better GNOME integration for Firefox when this becomes available. Hopefully, somebody will use this to make a nice JS library on top of libdbus!
As ctypes author I’m amazed how ctypes is going to rule the world 😉