The most standard way of circumventing the Same Origin Policy is to use Json with Padding, or JSONP.  The technique in a nutshell is that the server returns a JavaScript function call.  The client thas prepared for this by creating this function in the DOM of the current page in the browser.  The client tells the server "The name of the function is xyzzy" by sending in a query parameter, usually named "callback." This post shows some ways around this issue.

public interface MyService extends RestService {
    @GET
    @Path("/object")
    public void getSomeObject(MethodCallback<SomeObject> callback);
}

To convert this to use JSONP you just add an annotation;

public interface FlickrService extends RestService {

    @GET
    @Path("/object")
    @JSONP(callbackParam="calback")
    public void photoFeed(MethodCallback<SomeObject> callback);    
}

I have shown that you can name the callbackParam, though if you don't "callback" is the default.  You can also specify a parameter failureCallbackParam, which sets up a second client-side method to receive error objects (for example if you wanted to have your service return a detailed exception object).

Unfortuantely, none of this seems to actually work:

• Looking at RestyGWT 1.2, they create the JSONP annotation but don't see to actually use it anywhere
• The object returned is, therefore, treated like any other RestyGWT response - RestyGWT tries to interpret it as pure json, and fails
• Another pecularity is that it makes an OPTIONS call to your server.  OPTIONS has to return some headers, including Acess-Control-Allow-Method, Access-Control-Allow-Headers, Access-Control-Allow-Origin.   This is somewhat painful to do with Jersey.  You have to either set up a second resource method with @OPTIONS as the http method; or, per someone on the RestyGWT forums, create a Servlet (or possibly a jersey-server) filter that always or selectively sends back those headers.  Yuck.  I was told, perhasp correctly (perhaps not), that checking OPTIONS is a feature of the browser.

One alternative to all of this is to forget RestyGWT entirely and use GWT's JsonpRequestBuilder class directly.  This is relatively straight-forward, and behaves, from the server side, exactly how you'd expect: it calls the resource and adds a query parameter "callback" (or something different, if you specify).  The javadoc linked above has a good example of how to make this work.

This, too, has a down side, though: the objects received by your code will be some derivative of JavaScriptObject that you define as a GWT Overlay Type.  This is, in my opinion. rather messy boilerplate.  Jason tells me it gives you the worst of both worlds (Java and JavaScript) by forcing you to deal with both.  This complication is the main reason RestyGWT is so useful - it takes care of all the json to java object marshaling for you.

An Alternative

Fortunately, there's a way to take advantage of RestyGWT's marshalling capabilities and avoid the incomplete or buggy parts.  The trick is to use JsonpRequestBuilder to make the call to the backend, but use RestyGWT to parse the JSONP result.  This works great.  You begin by explicitly creating a codec for your object:

public interface MyCodec extends JsonEncoderDecode<MyObject> { }

// ...

MyCodec myCodec = GWT.create(MyCodec.class);

JsonpRequestBuilder jsonp = new JsonpRequestBuilder();
jsonp.setCallbackParam("callback"); //if you wish to change it
jsonp.setTimeout(15000);
jsonp.requestObject(url, new AsyncCallback<JavaScriptObject>() {

   public void onFailure(Throwable throwable) {
       GWT.log("Error: " + throwable);
       updateTimer.schedule(20000);
   }

   public void onSuccess(JavaScriptObject jso) {
       GWT.log("Wow, it worked!");

       JSONObject o = new JSONObject(jso);
       MyObject myObject = myCodec.decode(o);

       //Do something wth your MyObject here
   });
}

This works exactly the way you would expect, both on the server and the client side (all the server see is a GET, no more OPTIONS and no special headers required).

The Server Side

Jersey makes generation of JSONP responses fairly easy.  The best way to demonstrate is by example:

@GET
@Path("/object")
@Produces({"application/x-javascript")
public Response getLocationsByCustomerJsonP(@QueryParam("callback") String callback) throws Exception {

    /*
     * Note that no @DefaultParam is required.  If the paramter isn't specified, then it will
     * be null, and JSONWithPadding's constructor will default to "callback"
     */


    MyObject returnObject = getMyObject(); //get or create the object you want to return

    JSONWithPadding j = new JSONWithPadding(returnObject, callback);
    return Response.ok().entity(j).build();
}

Thanks goes out to Héber Costa Ferreira for finding this problem in GNE/HawkNL and providing suggestions, code, and testing effort to fix this issue.

This change has been commited into GNE SVN as of revision 680. If you are not using the SVN version of GNE, you should, it is much more stable than 0.70 and contains many critical fixes.

1. The protocol specification, all documentation, and comments say little endian and I don't want to have to change them all
2. It was little endian properly in NO_NET mode
3. I guess client and server side will typically use same version of GNE, because they would be compiled together most of the time, so they both would change endianess at the same time.
4. I still agree with my original thinking years ago that virtually all machines running GNE will likely be little endian. This is even more true today with the usage of x86 for Macs instead of PowerPC, and typical ARM is little endian. Therefore we can save effort by not swapping on these machines.

The counter arguments:

1. Big endian is easier to read in a debugger or when dumping raw packets
2. Big endian is "network byte order" and is typically assumed unless otherwise stated in Internet-based binary protocols.

I fully understand if someone prefers to use big endian over the wire. If you don't agree with this change, simply call nlEnable(NL_BIG_ENDIAN_DATA) immediately after initGNE. This workaround is safe for GNE versions both before and after this change. I will officially support bug requests if GNE doesn't work when you do this, of course at this time it is obvious that it works since that is the mode of operation until now.

Since the build version of the GNE connection packet is 16 bits in length, you can be safe in knowing that big-endian GNE will not connect to little-endian GNE due to a "protocol version mismatch", even though they have the same version. In a future version of the protocol I may add a special part of the packet to detect byte order specifically to output a better error message.

Note that the endian setting affects all Buffer operations (except strings and raw writing), which includes all GNE protocol code.

Groovy is a dynamic language for the Java Virtual machine inspired by Python, Ruby, and Smalltalk. After Java, it is the first approved JVM language (via JSR 241). Groovy can be used as a scripting language in a shell environment, a way for applications to provide scripting and extension capabilities, domain specific languages and configuration files, or as the base language for whole applications and frameworks such as Grails (like Ruby on Rails) and Griffon. Due to supporting most Java syntax and ability to integrate with any existing JVM code, Groovy is particularly useful for existing Java developers wanting to leverage dynamic languages while retaining use of the full Java ecosystem.

This talk will cover the following areas:

• Groovy syntax compared with Java
• How dynamic calls are implemented
• Closures
• Runtime Metaprogramming: how Groovy extends existing Java libraries
• Compile-time Metaprogramming (AST transformations)
• Creation of Domain Specific Langauges (DSLs)
• Code examples where Groovy's features make the strongest impact 

• Maintains the entire cross-compilation toolchain for you
• Enables reproducable open-source. When you bitbake systemd-image, it builds everything, from gcc and eglibc and upwards and you end up with a full filesystem and ipks. You really do have the source and you really can prove that you've built it all.
• Maintains a clean and and up-to-date ipk repository (making sure to clean old and obsolete packages)
• Handles building of dependencies -- if you say you need libreadline, just declare it as depends and it will make sure readline is built before your package builds. This is possible via the BBFILES path which scans and understands all recipes it can compile and forms a dependency tree. This tree can also be exported to graphviz
• Enforces a consistent build pattern for all of the OE projects, which gives you a lot of things for "free" like building ipk files, cleaning and fetching sources, etc.

The first page you will likely find on your searching is a very minimal and out-of-date (6 years old) bitbake manual. It explains a little bit about the syntax but not a lot about how to do recipes or how it's used within OpenEmbedded. There is some information also on the OpenEmbedded wiki but also nearly non-existant. After a LOT of searching, I've come across some more useful resources:

A "hello world' C program tutorial. This is useful but it doesn't work verbatim. However bitbake/Angstrom is now, the LICENSE and LIC_FILES_CHKSUM are required. This sounds like a great QA check for them, you have to say what the license is of the software and point to an actual file with that license and confirm it with a checksum. To just do my "hello" example I created an empty LICENSE file.

LICENSE = "TEST LICENSE"
LIC_FILES_CHKSUM = "file://LICENSE;md5=d41d8cd98f00b204e9800998ecf8427e"

SRC_URI = "file://hello.c\
           file://LICENSE"

I calculated the checksum simply with the "md5sums" command line tool. Now my package built and I got an ipk in the build/tmp-angstrom_2010_x-eglibc/deploy/ipk/armv7a directory. You can copy that ipk to the BeagleBone and install it "opkg install", then run hello.

This is as far as I've gone in practice so far, I want to start building real things soon, but this shows how easy it is. Just recently I finally found some real documentation that will let me take the next step. This documentation is a little old but still seems relevant to how things appear now. I found an OpenEmbedded manual linked from the beagleboard Google Group. There are three useful parts:

• All of chapter 8 is the most useful thing I've found yet.
• Chapter 3 contains something similar to the "hello world" tutorial I linked above but with more details (although harder to follow)
• The tasks section of chapter 7 describes the states bitbake goes through while building software. If you are are familar with the Java build tool Maven, this sounds a lot like the Maven lifecycle. Note you can also run bitbake <PACKAGE> -c listtasks to see what tasks can be run on a certain project.

One of the first things you will notice is that bitbake recipes are more declarative than imperative. If you're a Java developer familar with Ant and Maven, bitbake seems more like Maven, although the tasks are defined via imperative functions similar to make or Ant. Because of this, recipes seem very magical, for example they just set SRC_URI and then not bitbake not only knows how to obtain the source, but if patch files are included in SRC_URI then bitbake knows they should be patches applied before building, automagically. This is why documentation is very important for such a system so I hope to find a better resource for this. However, a lot of the magic variables have a default definition which you can find in the bitbake .conf files. There are 3 places you should look at when trying to figure things out:

• setup-scripts/conf/
• setup-scripts/sources/bitbake/conf/
• setup-scripts/sources/openembedded-core/meta/conf/

For example, from the openembedded conf file you can see the default definition of FILES_${PN}, which shows you the standard places that bitbake searches for files to go into your main package. That's how the hello world example automagically figured out that /usr/bin/hello should be packaged up.

And finally, of course the other good place to look is by example and the multitude of .bb files in the Angstrom layers. Once some of the magical variables are deceiphered, they should become more understandable.

I hope this helps you get started on your way, as it has for me. I'm still learning this so if I'm missing things, please leave a comment.

Normally in Apache I would turn on MultiViews and also rely on default files like index.htm/php. While IIS 6 supports default files, it doesn't support MultiViews as far as I can tell. My searching didn't find much workarounds. I did find Ionic's Isapi Rewrite Filter though, which bascially is an open-source mod_rewrite for IIS. Read on for details on my configuration.

• For a resource or downloadable content (like jpg, css, zip, etc), leave the extension.
• If requesting a directory (URL ending with /), try first for index.php, then index.html, then index.htm.
• If requesting a non-directory location that doesn't exist (like /whatever), look for whatever.php, whatever.html, and whatever.htm in that order, and use the first one found. I do rely on IIS's behavior that if the user goes to /somedir and somedir is a directory, it redirects them to /somedir/, then my rules will look for index properly.

I allowed direct links with .php on the end to work, and I didn't redirect those to the "php-free" URL, but this is something I think is easily possible. Here is my configuration:

RewriteEngine ON
#uncomment this to enable the iirf status page (see documentation)
#StatusInquiry ON /iirfStatus RemoteOk

IterationLimit 5

# don't rewrite any request that ends with one of these extensions,
# even if they don't exist.
RewriteRule (.+\.)(jpg|png|jpeg|gif|ttf|sql|txt|zip|css)$   -   [L]

# If there is an exact match, then don't do any rewriting
RewriteCond %{REQUEST_FILENAME} -f
RewriteRule ^.*$ - [L]

#If this is a URL ending with a /, try index.php, then index.html
#(I don't know how to get it to search as if we had done /index):
RewriteCond %{REQUEST_FILENAME}index.php -f
RewriteRule ^(.*)/$  $1/index.php [L]
RewriteCond %{REQUEST_FILENAME}index.html -f
RewriteRule ^(.*)/$  $1/index.html [L]
RewriteCond %{REQUEST_FILENAME}index.htm -f
RewriteRule ^(.*)/$  $1/index.htm [L]

# Try to see if this is a PHP page:
RewriteCond %{REQUEST_FILENAME}.php -f
RewriteRule ^/(.*)$  /$1.php [L]

# Try to see if this is an HTML page:
RewriteCond %{REQUEST_FILENAME}.html -f
RewriteRule ^/(.*)$  /$1.html [L]
RewriteCond %{REQUEST_FILENAME}.htm -f
RewriteRule ^/(.*)$  /$1.htm [L]

Read on for my current progress...

However, my experience with JEPP was horrible and non-working at all, at least on Windows x86-32. The initial warning sign was the lack of updates for awhile and the combination of the files being named with each permutation of java version and python version; it sounded like the PHP/Apache madness of matching 32/64 bit, Apache/PHP version, and C runtime version. It's like the native precompiled binary wheel of fortune where you spin it and just hope you get a set of compatible libraries. Here were the list of problems I ran into before I gave up:

• JEPP has a very minimal website and virtually no documentation, although it does look simple to use.
• JEPP's latest version supports only up to Python 2.6. I couldn't find an old pythonxy verison for Python 2.6, besides even if I did I'd never see an update again, so I couldn't use pythonxy.
• I tried the Python 2.6.4 installer from the JEPP site. The installer appeared to install all files but at the end the installer said that it failed to complete; however, everything seemed to be installed and I could run Python so I went with it. JEPP would actually run with this version and I sucessfully ran some trivial Python with the included console.py. However, I couldn't get numpy 1.6.1 to work even in the native Python; it complains the DLL was not found, so I didn't even try with JEPP.
• OK, so I uninstalled everything and tried the latest 2.6 ActiveState ActivePython (2.6.7.20). I installed Numpy 1.6.1 and it works. Then I install JEPP and try to use it in the same was as before, now the JVM segfaults calling some unicode string handling method while trying to run the script, so I couldn't run anything at all in JEPP.
• At this point I did a lot of searching the web and found no solace. In fact, I even found a stackoverflow post only a few months old saying that numpy and JEPP aren't compatible due to the sub-interpreters used by JEPP and the GIL.

After that experience, I didn't try to go any farther, even if I got it to work I would be stuck forever on an old Python and there would be many more C extensions to worry about.

In the end I am thinking it is not currently possible to run typical scientific Python code in the same process as JVM langauges. So the fallback would be either porting code, or running JVM and Python in two separate processes and using whatever IPC mechanism is the easiest (pipes, TCP, XML, etc.) with a very loosely bound interface. However, this makes it much more challenging (and much higher overhead) to build a single, integrated product and eliminates any ability for rapidly leveraging library code in JVM or Python for prototyping. I just heard about execnet, that might make the IPC easier, though.

I've had the privilege of starting to play around with Angstrom (fork of OpenEmbedded) on the BeagleBone. I needed to build some bleeding-edge software with bitbake on an Ubuntu 10.10 but I ran into an interesting problem. When building the distribution, it builds all of the tools from the ground up, including automake. However, automake would crash when building some sort of support for emacs with a jove crash. I couldn't understand how they were related.

In the end, I found that since jove was installed but Emacs was not, /usr/bin/emacs actually pointed to starting jove. Automake was trying to use some kind of compiler for emacs (I guess it was making some emacs macros or syntax thing?). It must have thought emacs was properly installed and called Jove with a set of parameters that caused it to crash. In the end, the solution was to install Emacs. The build system itself wasn't in error, it was my machine, so it can't be reasonably "fixed" in the code so I put this post in hopes of people finding it if they search for the same problem.

The two attack vectors I looked into is XSS (Cross-site scripting) and CSRF (cross-site request forgery, aka XSRF).

Cross-Site Scripting

This exploit happens when someone can inject script code into your site. The primary way an application is vulnerable to this is when you provide unescaped code to a client that would be interpreted as HTML. An obvious way would be a forum post that allowed users to post a <script>. This script could steal information, the most important being the session cookie information and sending that to the attacker's site who can then "log in" as you.

OK, everyone knows to protect forum posts, of course. A good programmer protects any user-developed content on your site. But keep thinking -- one place you might not realize: The error handling pages. Everyone loves a good "404" page with a message "Oh, the URL ABC you tried to visit doesn't exist." Typically some web admin set that up once and no one thought about it again. Well, what if someone gets your user to visit (encoded of course) http://mysite.example.com/?<script>....? Now they have just injected a script into your site through your 404 page. What about a login page? "The user ABC doesn't exist". What about the user <script>bad_code();</script>? Does he exist?

There is plenty of more information on the Internet, and I'm not trying to outdo Wikipedia here, but here are some tips:

• Don't echo back ANYTHING to the user without escaping potential HTML code on the server side. In the case of the 404 and login examples above, consider not even echoing anything back at all.
• If you have client-side JavaScript reading from your server, escape there as well for another layer of precaution when it is possible.
• Consider a cookie value that wouldn't work if requested from a different machine (IP address) or browser. This isn't bulletproof if the attacker has control of the network to spoof this information, but it makes it a lot harder.

Another resource is this OpenAjax site page, which also covers CSRF.

Cross-Site Request Forgery

Learning about this really blew my mind, because I understood XSS but I didn't realize CSRF. CSRF occurs when an attacker causes an operation on your site. Even though the "request" came from the attacker's site, the browser will send the cookies (or HTTP authentication credentials) for your site to your site. This makes sense. Of course, the attacker can't see your cookies or see what you are doing on your site, but it might not matter.

In order to exploit a CSRF vulnerability, your application needs to have a request that performs an action beneficial to the user that is the same for all users. The typical example is the bank one, where visiting http://mybank.example.com/transfer?to=attacker@evil.com&amount=1234 will result in a money transfer. The attacker doesn't need the user's credentials nor does he even need to observe the request or response. That URL can be placed anywhere on the Internet easily, because most sites will let you publish content with images linked to any URL. Put that URL into the "img src" and the user will see a broken image but not know what happened. The link also needs to be put on a site that your visitors would visit as well, so of course large targets like Facebook and Google are far more likely targets.

OK, so HTTP "GET" requests with side-effects are bad style anyway. You can fix the img problem but not allowing this, but it doesn't fix the problem. If the evil (or compromised) site contains a hidden HTML form and javascript, it can perform a POST as well.

Unfortunately, to me it appears the browser cookie model and HTTP authentication models are broken in browsers, because they provide authentication information without prompting you. This makes sense otherwise every single page you click on would require the browser to ask you to confirm first. There's no way for a user to stop this except to always "log out" of your site and not view any other sites in that browser while logged in.

The solution is to foil the attacker by making it so that each user has a unique request that can only be generated by that user that does not involve solely the browser's cookie or HTTP authentication system. My understanding through reading is that any of the following ways to add user-specific data to the request would work:

• If using AJAX:
  • Take the value of the user's auth/session cookie; put it into the XML/JSON/form data. Since the evil site can't read your cookies, they can't form that request
  • Put the cookie's value into an HTTP header. Headers can be forged by attackers but they still can't see the cookie.
  • You could use a token from the server other than the auth cookie, but has to be specific to that user. But I'm not sure it provides any more protection.
• If not using scripting, generate an HTML form with a hidden field with a random value, saved to that user or session. Verify that the value is the same on submit.

You can also read more at Wikipedia.

Continue reading for the details.

I measured the width and height of the LCD monitor at 16 x 12 inches. Both monitors are 4:3, which means a 1.33 aspect ratio. The height of the viewable area on the CRT is 10.8 inches. From here the math was simple:

CRT height / LCD height = height ratio
10.8 / 12 = 0.9

Ratio * LCD resolution height = equivalent CRT resolution height
0.9 * 1200 = 1080

Since the CRT is 4:3, we can calculate the width:
1080 * 1.333 = 1440

This results in the CRT having an ideal resolution of 1440 x 1080.

Based on the spec sheet, which listed refresh rates at a list of standard supported resolutions, I guessed that I should be able to support 85hz at this setting.

Now all I had to do is figure out is how to set this resolution in Windows. Fortunately, I found a way in the NVIDIA control panel. I don't know if there is a way to do it in Windows in a video card-independent manner. Other vendors may have similar options in their control panels.

In the NVIDIA control panel, go to Display -> Change Resolution. Select the CRT and pick customize.

Select the resolution you want and the refresh rate. You might have to play with the timings; I set mine to GTF and that seemed to work and get me the refresh rate I wanted.

In my case, the results came out quite nicely. Now when I move windows/images between the monitors I don't get a disconcerting zooming effect. Hopefully another perfectionist out there can benefit from this.

Summer is great for food because you can always find fresh ingredients at the local farmers' market or your own garden. Sometimes you bought (or harvested) a lot of great vegtables and don't know what to do with them. With no meal planned on a lazy Saturday afternoon, we were getting very hungry and needed something fast. I made a fresh pasta sauce from the tons of fresh vegtables in the house.

The proper way, given enough time, would be to reverse engineer (or find documentation for) the MT database format and understand it well enough to construct all of the rows to describe the assets as if I had uploaded them. But I already had a published and tested XMLRPC interface, this is a one off approach, and I was running out of steam working on this, so I used the XMLRPC way to reupload all of the files. But, my custom approach had a few drawbacks:

• I had to reupload all of the files that were already on my server, overwriting them
• Since I was overwriting the files, the modify/create dates on the files were all reset to the time of the import; I couldn't find a reasonable way within my motivation to fix the file timestamps (thus affecting Apache's return) or via MT
  • The table in MT looked like it was easy to fix, but who cares, if the HTTP server still returns the wrong date because the file's date on disk is wrong?
• I had to modify the DeniedAssetFileExtensions in mt-config.cgi to allow the upload of certain files (like exe for my installers) (this was new upon upgrade to 5.11 from 5.04)

For the code, I used the newMediaObject implementation of MoveableTypeImpl from part 2 to actually upload the files. The files, their content and metadata was found simply by walking the filesystem. I wrote some custom code in the SiteScanner object (not shown below) to filter out certain paths that I didn't want to manage as assets in my site. From a high-level view, it's a simple as that, so the rest of the post will be the code. The sitePath is the root of the website, which is also used to determine the relative path needed for the MT interface (i.e. C:\Projects\MyWebsite\tutorials\file.txt to /tutorials/file.txt).

    private static final String FILE_SEP = System.getProperty( "file.separator" );

    private static void importAssets( MovableTypeImpl mt, File sitePath, SiteScanner scanner )
            throws IOException, XmlRpcException {

        long totalLength = 0;
        long totalFiles = 0;
        for ( File file : scanner ) {
            totalLength += file.length();
            ++totalFiles;
        }
        System.out.println( "About to upload " + totalLength + " bytes in " + totalFiles + " files" );

        for ( File file : scanner ) {
            String filePath = convertToSiteRelativePath( sitePath, file );

            //TODO: can I preserve the date?
            System.out.println( "Uploading " + filePath + "( " + file.length() + " bytes)" );
            String dest = mt.newMediaObject( filePath, getFileData( file ) );
            System.out.println( "  Uploaded to: " + dest );
        }
    }

    private static String convertToSiteRelativePath( File sitePath, File file ) {
        String filePath = file.getAbsolutePath();
        if ( file.getAbsolutePath().startsWith( sitePath.getAbsolutePath() ) ) {
            filePath = filePath.substring( sitePath.getAbsolutePath().length(), filePath.length() );
            if ( filePath.startsWith( FILE_SEP ) )
                filePath = filePath.substring( FILE_SEP.length(), filePath.length() );
            filePath = filePath.replace( FILE_SEP, "/" );
        }
        return filePath;
    }

    private static byte[] getFileData( File file ) throws IOException {
        int total = 0;
        int length = (int) file.length();
        byte[] ret = new byte[length];
        FileInputStream reader = new FileInputStream( file );
        while ( total < length ) {
            int read = reader.read( ret );
            if ( read < 0 ) throw new IOException( "Cannot read all data from " + file );
            total += read;
        }

        return ret;
    }

Upload Interface

The key to uploading pages is through the wp.newPage method, which MT supports and is best defined in the WordPress API documentation. The following is the relevant parts for my project, as well as some MT-specific information I gleaned from the soruce code, because I never could find a good, proper page on MT's site documenting this.

wp.newPage( blogId, userId, pass, item )

item is a struct:

• title = HTML title
• description = HTML content
• permaLink = the actual, absolute URL where you want it published
• dateCreated = date of the file imported

It appears that if dateCreated is in the future that there is code to set the post to be published in the future, but I did not test this.

Other fields I didn't care about that I could set (For others who might care):

• mt_convert_breaks
• mt_allow_comments
• mt_excerpt
• mt_text_more
• mt_keywords
• mt_tags (based on the source, uses author's tag delimiter setting to separate)
• mt_tb_ping_urls

The fields that can't be "empty" (such as mt_allow_comments) are set to their defaults if not explicitly specified in the XML-RPC call. There also appears to be some kind of generic mechanism to support mt_* fields, but I'm not sure what that can set, perhaps custom fields?

The permaLink was the tricky part, because I thought that Movable Type folders mapped to categories (especically since they are reported back as such when you get post information). But the category field is read only for blog posts, not pages. By looking at the Perl source code I determined that it only looks at permaLink, and parses the fragments out of it to determine the folder and "basename" (or "Filename" in the web interface). Before I had tried setting property "mt_basename" and this works great if you don't care about folders and want everything in the root.

I didn't try this, but looking at the source code it appears the permaLink parsing only takes affect if entry type is set to "page", which appears only to happen if you call wp.newPage as opposed to metaWeblog.newPost. I could be wrong, though.

I set MT to have "htm" instead of "html" as the end file name to match my existing site. I used htm on the permaLinks too. As a test of curiousity I tried "html" and got the error message "Requested permalink ... is not available for this page".

So in the client, I have to set both the URL prefix ("http://gillius.org/" as well as the suffix ".htm") in the client.

XMLRPCServer bug

The XML-RPC calls filter all of the basename settings for adds or edits, whether through mt_basename, wp_slug, or permaLink, through the _apply_basename function. The basename is checked whether or not it is unique. The problem is that it doesn't check if it is unique within a folder. The web interface seems to handle this. For example folderA/the_page.htm and folderB/the_page.htm are considered "the same" to the XML-RPC interface but unique in the web interface. When the XML-RPC interface encounters an attempt to edit or add the second "the_page.htm", it will rename it based on the content as if you didn't give it a basename at all.

I tried hard to find a way around this, but in the end I had no alternative but to edit the source code for my import.

This even has a problem where if you "wp.getPage" then pass back that exact, unmodified page to "wp.editPage", it actually changes the permaLink and basename, whereas normally it wouldn't. In my opinion, that's a bug if you can't do a "round trip".

I'm still not sure whether this is a "bug" or a feature, but since the web interface lets me do it, I don't see any reason not to:

    if (defined $basename) {
        # Ensure this basename is unique.
        my $entry_class = ref $entry;
        my $basename_uses = $entry_class->exist({
            blog_id  => $entry->blog_id,
            basename => $basename,
            ($entry->id ? ( id => { op => '!=', value => $entry->id } ) : ()),
        });
        if ($basename_uses) {
            $basename = MT::Util::make_unique_basename($entry);
        }

        $entry->basename($basename);
    }

I commented the lines except for the if statement itself and the setter for basename. In MT 5.04, these were lines 180 to 188 in lib/MT/XMLRPCServer.pm

I'm not totally sure the solution to this because I know virtually nothing about Perl. I would guess that one could just add folders to the associative array in the exist call to "narrow" the search, but I'm not sure. Since I was importing my existing, valid, site from files on disk, I knew that there wouldn't be any actual basename collisions, so for the import I commented it out. I decided to uncomment it after the import, to leave the code in a "pristine" state.

This is one of those moments that I'm really glad for open source.

 Writing the Code

I may release the importer source code at some point, in some form. I thought of making a "polished" program but I ended up having to be pretty specific to my website and it took enough effort that I wasn't motivated to make a version that is bullet-proof, so my code would cause a lot more questions than answers. When it failed or if I found a special case or file it had trouble handling I had to coax it along. There's no GUI for it which would be a necessity to review what it was going to import (based on your scanning rules) as well as review if it parsed the content properly.

That said, there are a lot of things I can say to help others interested in writing their own. I wrote it in Java but python or groovy would work well.

In python you can use xmlrpclib:

import xmlrpclib
proxy = xmlrpclib.ServerProxy("http://mt.example.com/cgi-bin/mt/mt-xmlrpc.cgi")
proxy.mt.newPage( blogId, username, password, [ "title" : title, "dateCreated" : dateCreated, "permaLink" : permaLink ] )

In groovy you can use groovy-xmlrpc:

import groovy.net.xmlrpc.*
def server = new XMLRPCServerProxy("http://mt.example.com/cgi-bin/mt/mt-xmlrpc.cgi")
server.mt.supportedMethods().join('\n')

I ended up ultimately going with Java because I was more comfortable with it at the time compared to python. Since I wrote that code, I've been seriously learning Groovy and I would have used it had I known at the time.

In Java I used the org.apache.xmlrpc:xmlrpc-client:3.1.3 library, which was easily obtained from Maven. I made interfaces MetaWeblog, MovableType, and WordPress to encapsulate the methods that I cared about, and implemented them in a MovableTypeImpl, which is shown below:

public class MovableTypeImpl implements WordPress, MovableType, MetaWeblog {
    private final XmlRpcClient client;
    private final String username;
    private final String password;
    private int blogId;

    public MovableTypeImpl( URL url, String username, String password, int blogId ) {
        XmlRpcClientConfigImpl config = new XmlRpcClientConfigImpl();
        config.setServerURL( url );
        client = new XmlRpcClient();
        client.setConfig( config );

        this.username = username;
        this.password = password;
        this.blogId   = 1;
    }

    public String[] supportedMethods() throws XmlRpcException {
        Object[] result = (Object[]) execute( "mt.supportedMethods" );
        return Arrays.copyOf( result, result.length, String[].class );
    }

    public String newPage( Date pageDate, String title,
                           String content, String link ) throws XmlRpcException {
        Map<String, Object> item = new HashMap<String, Object>();
        if ( title != null )
            item.put( "title", title );
        if ( pageDate != null )
            item.put( "dateCreated", pageDate );
        if ( link != null )
            item.put( "permaLink", link );

        item.put( "description", content );

        return execute( "wp.newPage", blogId, username, password, item ).toString();
    }

    public String newMediaObject( String fileName, byte[] fileData ) throws XmlRpcException {
        Map<String, Object> file = new HashMap<String, Object>();
        file.put( "name", fileName );
        file.put( "bits", fileData );

        return execute( "metaWeblog.newMediaObject", blogId, username, password, file ).toString();
    }

    private Object execute( String method, Object... params )
            throws XmlRpcException {
        return client.execute( method, params );
    }
}

For the final conclusion, view the third part covering asset upload.

In this first part, I will cover conversion of the blog posts, conversion of the style, and parsing the original static HTML pages into a format that can be uploaded into MT. Read on for the details.

Ultimately, I chose MT because it allowed for "static publishing" -- meaning it would generate static HTML pages and put them on the site, but still allow for hooks for comments/trackbacks/etc. such that I can update the site easily. The performance on my host for PHP and other dynamic content is not quite fast, and I've always enjoyed my site's very low overhead and fast loading, even on mobile devices. I'm not trying to be the next media mogul here.

Another requirement that I wanted was a remotely accessible interface (web service). MT supports XMLRPC, similarly to other solutions as well -- there are even "standard" interfaces like MetaWeblog API and WordPress has an API which MT also supports some of those calls as well. The biggest problem I had is that I had a horrible time trying to find actual documentation from MT. Ultimately to figure out the details of some items I ended up diving into the source code. Even though I'm not a Perl wizard, I was able to glean enough to see how some of the calls worked. That came in handy to workaround/fix some of the quirks/bugs/limitations I encountered when trying to import the site. This is one situation where it's really handy to work with an open-source project.

I noticed two main versions of MT, MT 4 and MT 5. I chose MT 5 to be the "latest" at first but then noticed that at least at this time the vast majority are still on MT 4. I'm not completely sure why, but my understanding is that MT 5 supports the concept of a "website" in addition to a "blog", which was important for my site -- it's mostly a website and secondly a blog, although I wanted to move towards the latter over time.

I ended up writing a custom piece of software in Java to convert my site (except for the blog part, which was easy enough to do in pure SQL).

Converting the Blog

The blog was by far the easiest part to convert. The original blog was a simple PHP script that I wrote against a MySQL database. To move this to Movable Type, I just performed a SQL SELECT to put the data into the Movable Type import/export format as documented here. The only fields in my news items were title, date and content (as string of HTML), which made the export/import straightforward. I was able to do the export with a single statement:

SELECT CONCAT('TITLE: ', `title`, '\nDATE: ', DATE_FORMAT( `newsDate`, '%m/%d/%Y %H:%i:%s' ), '\n-----\nBODY:\n', `content`, '\n-----\n--------' )
FROM `News`
ORDER BY `newsDate` ASC

The output then, looks like the following:

--------
TITLE: Super IsoBomb 3D Announced
DATE: 09/15/2003 00:00:00
-----
BODY:
A continuation to the Super IsoBomb game...

The News table is specific to my site, but if you had a custom system or even another database-based blog, the SQL may be similar. You just need a view that consists of the above items, but more importantly the content needs to be an HTML fragment and not in some markup language.

Converting the Style

My original static HTML site was originally based on Dreamweaver-style templates, since I worked with Dreamweaver in the late 90s. I used only the basic functionality for templates, which basically is given a template page you could view on its own with markers to indicate where content should be placed:

<html><!-- #BeginTemplate "/Templates/main.dwt" --><!-- DW6 -->
<head>
<!-- #BeginEditable "doctitle" -->
<title>Contact Gillius</title>
<!-- #EndEditable -->
...
</head>
<body>
... template content ...
<div class="contentArea"><!-- #BeginEditable "content" -->
<!-- #EndEditable --> </div>
</body>
<!-- #EndTemplate --></html>

Since everything generated was just plain HTML, after I lost access to Dreamweaver, I could still edit it by hand (using gVim), and upload by SCP. You could also see how easy it would be to replace the template content with regular expressions -- just select everything between the unique tags and replace. Even still, it was a pain and you can see it by the very limited updates to the site in the past years. This was my motivation to move to a content management system.

Since my pages were structured this way, I could copy and paste the shared portions of the template into MT's template design system, and bring over my CSS styles. I had to tweak some of MT's built-in widget HTML to add/change styles to make it easier to fit into my existing CSS. I thought about a site redesign, but decided to keep things easy for myself and tackle only one problem at a time and try to convert the site verbatim before I redesign when I have the time and motivation.

Exporting the Pages

Ultimately, for the export process, I would need to get the pages into a similar format in memory to export via a series of XMLRPC calls. Pages are basically like blog posts, you need the following:

1. Location of the page (aka the "permalink")
2. Date the page was last updated
3. Title of the page
4. Content of the page as an HTML fragment

Since I have the web site mirrored in my disk and in version control, it was easy to get the first two items just by traversing the filesystem and looking at the file's name and date. Getting the title is possible since it's in the HTML head, and getting the content was easy because it's simply everything between the tags. Therefore, I was able to use the following Java patterns:

    public static finalPattern titlePattern = Pattern.compile(
            "(?s)(?i)<html.*?>.*<head>.*<title>(.*)</title>" );

    public static final List<Pattern> contentPatterns = asList(
            Pattern.compile( "(?s)<!-- #BeginEditable \"content\" -->(.*)<!-- #EndEditable -->" ),
            Pattern.compile( "(?s)(?i)<body.*?>(.*)</body>" )
    );

The second pattern in the contentPatterns list is in case the first one failed to select any content. Not all of my pages on the site fit into the template, and for those pages I wanted to just simply pick out all of the HTML content in the body. There were a few pages I excluded from the export, such as the GNE tutorials, that don't use my site template at all since they are meant to be distributed standalone. As with any export process you also find a few "exceptions" to the rule that I had to just smooth over by hand. Now that I had my pages broken up into the 4 pieces, I was ready to upload them with XMLRPC.

The second part covers XMLRPC and uploading the static HTML content into Movable Type.

The third part covers the asset (images, zip files, etc) conversion.

I also learned that Expires doesn't mean that the cache goes away -- it just means the browser will "revalidate" the page -- that is contact the origin webserver with a If-Modified-Since, which only returns content if it's changed. Also, apparently IE's default settings work more like how I thought browsers always work, which is to revalidate pages in the cache at least once for each time you start the browser. Firefox won't do this even through a restart.

So what is the solution? Instead of setting Expires, you can set Cache-Control's max-age field to have an offset, which has the same effect as setting Expires to some time in the future from when the browser/proxy read the file. You can also set no-cache, which at first I thought meant don't store it at all on the disk, which is not true. All it means is that (at least for Firefox 5 according to Firebug) the page is revalidated against the cache each time. I could be wrong for IE, maybe max-age=0 is better. Please comment if you have some better suggestions.

Another post on HttpWatch suggests setting no-cache for HTML/dynamic pages, and "expire forever" for all other resources (images, JS, CSS, etc). What if you need to change one of these resources? The suggestion is to change the name. This would be annoying, except that specifying a query parameter is sufficient. I checked and I saw this in practice. If you have site.js, and you update it, change your HTML to link to site.js?2. Your HTTP server won't care, but the proxy/browser will load it as a new file. The content of the query string is irrelevant except to make it unique.

I needed a fast fix, and I'm scared I can follow that pattern since I'm still tweaking my site heavily, so for now I just set all resources to expire in 2 hours and HTML pages to no-cache. That doesn't mean resources are totally redownloaded every 2 hours, it just means an "If-Modified-Since" query is made rather than no query at all; the server will respond "Not Modified" if there has been no change; just a quick ping. I'm not concerned since I get 1000s of hits a month and not millions right now. The following .htaccess is what I've got for now:

#Force clients to revalidate cache for "resources" every 2 hours
#This is because I'm still commonly changing resources. If I start to
#add "version query parameters" to js files and such, I can bump this
#up for those types
<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">
  Header merge Cache-Control max-age=7200
</FilesMatch>

#Except for HTML files, we want to revalidate each time...
#This handles comments/new posts/page edits as well as forums
<FilesMatch "\.(html|htm|xml|txt|xsl)$">
  Header merge Cache-Control no-cache
</FilesMatch>

Moving to a proper system now means that you can search the site, leave comments and trackbacks, and navigate by categories/tags/dates. It also means you will see more content as it is easier for me to post now.

In the short-term, my next posts will detail some of my experiences converting my existing site to Movable Type.