My hobbyist coding updates and releases as the mysterious "Mr. Tines"

Wednesday 23 April 2008

Integrating .Net with Erlang, part 5

I have a first drop that has all the C# input code written, and a start of some unit tests for interoperability with the J# build of jinterface 1.4. Nowhere near enough tests are written yet, and a couple of those that have been (encoding a 64-bit integer in J# and unpacking in C#) don't yet work. But at least it's a start.

Still to do are the full gamut of integer types, and the complex ones -- lists, tuples, ports and such, then replicate for writing by C# and reading in J#.

One fix to the tests was for my code -- I needed to take the surprising little-endian wire format for an extended integer, and reverse it to fit into the BigInteger class I got from Codeplex; the second was a fix to that class:

381c381,384
<    
---
> 
>             // BUGFIX -- use an accumulator rather than |=ing into
>             // m_digits[m_digits.DataUsed], which will be zero
>             DType accumulator = 0;
384c387,389
<     m_digits[m_digits.DataUsed] |= (DType)(array[offset + leftOver - i] << ((i - 1) * 8));
---
>                 DType digit = array[offset + leftOver - i];
>                 digit = (digit << ((i - 1) * 8));
>                 accumulator |= digit;
385a391
>             m_digits[m_digits.DataUsed] = accumulator;

Code drop for the curious at the usual site. The project assumes you have NCover and NUnit around your system to run the tests and take coverage stats.

Update: More tests, and a few more fixes, refactorings and extensions added 24-Apr. Doubles now get encoded in the new style and integers that fall into the Int64 range are handled so the jinterface code can handle them.

Update: 26-Apr -- Added an export to little-endian byte array for BigInteger and added unit tests for short and long BigInteger representations.

Update: 27-Apr -- A first pass of Java type to C# and back for all the Java wire types, and a start of interface breaking changes to shut FxCop up a bit. Still needs a lot of like-for-like based unit tests, and detailed code cleaning.

Update: 28-Apr -- Unit tests about complete for all the wire types. Now the fun really begins.

Update: 4-May -- Unit tests extended to the input and output classes, including adding compression via a modified version of the ComponentAce port JZlib. All code now clean after FxCop scrubbing. Still needs all the node related classes being compared with the jinterface 1.4 versions and tested.

Tuesday 22 April 2008

Integrating .Net with Erlang, part 4

So where are we going from here with the Otp.Net code? First off, if we bring the codebase into .Net 2.0 from its '04-'06 vintage, and make all the data classes in the Otp.Erlang namespace immutable -- returning ReadOnlyConnection(of T)rather than arrays as needed -- we can get rid of the ICloneable support. That lets us make Otp.Erlang.Object into an interface containing just the encode method(dropping the decode method, or making it an extension method as per C# 3.0).

Then we can use some subset of the fields used in equality tests to do GetHashCode properly, and replace arrays with growable collections.

Next, fix the range check on Double's floatValue method, and make it return a nullable value:

    public float? floatValue()
    {
        if (d > Single.MaxValue || d < Single.MinValue)
            return null;

        return (float) d;
    }

and rename Long as Integer, kill both their derived classes and make Integer carry a BigInteger to which we can delegate conversions returning nullables.

That will mean some fixing up of input and output, but we need that anyway to handle new-style float values encoded as 8-byte big-endian blobs (and the change of tense means that this is where we move past what I've done so far).

To come after that will be making GenericQueue delegate to Queue(of IObject), and then a general scrub done in comparison with jinterface 1.4, before it will be sorta-releasable.

At the moment it builds, but will not work, so no interim archive is yet publicly available.

Saturday 19 April 2008

Integrating .Net with Erlang, part 3

A kind anonymous has pointed out that someone has already done the job of porting an earlier version (1.2.1.1) of the jinterface code, even though it has not made its way into the standard distro. Via this blog post, one eventually gets to the sourceforge repository details for it.

This actually does work with IronPython (after the trivial edits to load the appropriate assembly and the slightly different class names) and without the J# runtime being involved.

I spotted the comment after spending a little while porting the 1.4 code. Actually the underlying Java code is quite horrid -- to start with there are lots of methods with the same name as fields; and a lot of identically named fields shadowing ones elsewhere in the inheritance hierarchy; and methods and fields are used interchangeably. Then there are blocks of cut-and-paste and tweak code; plenty of swallowing of exceptions and blurred responsibilities (where one object performs a series of manipulations on a member of another object). And there there is this (where d is a double):

  public float floatValue()
    throws OtpErlangRangeException {
    float f = (float)d;

    if (f != d) {
      throw new OtpErlangRangeException("Value too large for float: " + d);
    }

    return f;
  }

which doesn't look too tested to me -- what's this (f != d), in case you couldn't spot it. This is carried over into otp.net, and, I think, shows how little floating point is used.

I fixed up the name clashes and some of the blurred responsibility in the Java, then ran the conversion over; and by taking out the BigInteger stuff (not there in otp.net's Long class -- haven't checked the rest of these), compression, most of the floating point, got what was left to build, though it currently fails in the challenge/response handshake (where otp.net has replaced the roll-your-own MD5 with calls into System.Cryptography).

Where could we go from here?

Two main things need doing. The otp.net code needs to be brought up to the 1.4 standard and make better use of .Net 2.0 e.g. for tracing; and it needs to be refactored to clean up inherited code smells.

Ideally, some of the changes should be back-ported into jinterface, too.

To date, this includes fallback to Java's file based location for cookie and a replacement for some of the non-ported tracing code (logging the remote endpoint).

Friday 18 April 2008

Integrating .Net with Erlang, part 2

So, we have pure J#. Where do we go from here?

C#

The body of the driver class carries over unchanged but for the Exception type. The driver project has to itself reference vjslib, as well as the jinterface assembly but is otherwise a standard C# class-with-main, and "just works".

using System;
using com.ericsson.otp.erlang;

public class ClientNode
{
 
 [STAThread]
 public static void  Main(System.String[] _args)
 {
  
  OtpSelf cNode = new OtpSelf("clientnode", "cookie");
  OtpPeer sNode = new OtpPeer("servernode@chloe.ravnaandtines.com");
  OtpConnection connection = cNode.connect(sNode);
  
  OtpErlangObject[] args = new OtpErlangObject[]{new OtpErlangLong(1), new OtpErlangLong(2)};
  connection.sendRPC("mathserver", "add", args);
  OtpErlangLong sum = (OtpErlangLong) connection.receiveRPC();
  if (sum.intValue() != 3)
  {
   throw new System.SystemException("Assertion failed, returned = " + sum.intValue());
  }
  System.Console.Out.WriteLine("OK!");
 }
}

How do we divest ourselves of the dependency on the J# runtime?

The Java Language Conversion Assistant generates C# code that doesn't compile, and would require a moderate amount of patching up for differences between superficially similar API classes. It is by no means a one evening exercise.

The OtpErlang.jar similarly fails to be converted by the jbImp tool, but IKVM converts it nicely; so at the penalty of having 27Mb worth of IKVM.OpenJDK.ClassLibrary.dll and IKVM.Runtime.dll instead of 3.7Mb of J#, driving the ~100kb dll, you can build the C# program and it passes this simple proof of concept.

IronPython and the DLR

Porting the code is easy enough

import clr
clr.AddReference("jinterface.dll")
from System import Array

from com.ericsson.otp.erlang import *

cNode = OtpSelf("clientnode", "cookie")
sNode = OtpPeer("servernode@chloe.ravnaandtines.com")
connection = cNode.connect(sNode);

args = Array[OtpErlangObject]([OtpErlangLong(1), OtpErlangLong(2)])
connection.sendRPC("mathserver", "add", args)
sum = connection.receiveRPC()
if (sum.intValue() != 3):
  print "Assertion failed, returned = " + sum.intValue()
else:
  print "OK!"

A direct conversion, the only slightly tricky bit being coercing the list to an Array.

This works fine when calling into the J# built library, as expected. IKVM needs the latest version -- before 0.36.0.11 (see comments), though, is a different story.

>\IronPython-2.0B1\ipy.exe Program.py
Traceback (most recent call last):
  File Snippets, line unknown, in Initialize
  File OtpErlang, line unknown, in connect
  File OtpErlang, line unknown, in .ctor
  File OtpErlang, line unknown, in .ctor
  File IKVM.OpenJDK.ClassLibrary, line unknown, in .ctor
  File IKVM.OpenJDK.ClassLibrary, line unknown, in init
  File IKVM.OpenJDK.ClassLibrary, line unknown, in getContext
  File IKVM.OpenJDK.ClassLibrary, line unknown, in getStackAccessControlContext
  File IKVM.Runtime, line unknown, in getStackAccessControlContext
  File IKVM.Runtime, line unknown, in GetProtectionDomainFromType
  File IKVM.Runtime, line unknown, in getProtectionDomain0
  File IKVM.Runtime, line unknown, in GetProtectionDomain
SystemError: The invoked member is not supported in a dynamic assembly.

Trace taken with 0.36.0.5, from last December. Fortunately, that wasn't some fundamental architectural thing being run into.

Integrating .Net with Erlang, part 1

So, you can integrate Java with Erlang, using the jinterface code supplied as part of the standard Erlang distribution. It's Java, so can we do this with .Net?

Let's start with J#. We set up the erlang source

-module(mathserver).
-export([start/0, add/2]).

start() ->
   Pid = spawn(fun() -> loop() end),
   register(mathserver, Pid).

loop() ->
   receive
      {From, {add, First, Second}} ->
        From ! {mathserver, First + Second},
        loop()
   end.

add(First, Second) ->
   mathserver ! {self(), {add, First, Second}},
   receive
      {mathserver, Reply} -> Reply
   end.

and test it with

>erl -name servernode@chloe.ravnaandtines.com -setcookie cookie
Eshell V5.6  (abort with ^G)
(servernode@chloe.ravnaandtines.com)1> node().
'servernode@chloe.ravnaandtines.com'
(servernode@chloe.ravnaandtines.com)2> pwd().
jinterface          mathserver.erl
ok
(servernode@chloe.ravnaandtines.com)3> c(mathserver).
l{ok,mathserver}
(servernode@chloe.ravnaandtines.com)4> s().
jinterface          mathserver.beam     mathserver.erl
ok
(servernode@chloe.ravnaandtines.com)5> mathserver:start().
true
(servernode@chloe.ravnaandtines.com)6> mathserver:add(1,2).
3
(servernode@chloe.ravnaandtines.com)7>

Now fire up Visual Studio, and create a J# class library project. Add existing items to copy the jinterface code into the project, and build it.

Cannot find type 'java.lang.ref.WeakReference'

Well, that's not too bad. Give it a stub, and build again:

package java.lang.ref;
import System.*;

public class WeakReference extends System.WeakReference
{
    public WeakReference()
    {
    }
}

gives us

error VJS1227: Cannot find constructor 'java.lang.ref.WeakReference(com.ericsson.otp.erlang.OtpMbox)'
error VJS1223: Cannot find method 'get()' in 'java.lang.ref.WeakReference'

So give it what it needs:

package java.lang.ref;
import System.*;

public class WeakReference extends System.WeakReference
{
    public WeakReference(Object o)
    {
        super(o);
    }
    public Object get()
    {
        try
        {
            return this.get_Target();
        }
        catch (InvalidOperationException ioe)
        {
            return null;
        }
    }
}

And it builds. Now put in the driver program:

import com.ericsson.otp.erlang.*;

public class ClientNode
{

    public static void main(String[] _args) throws Exception
    {

        OtpSelf cNode = new OtpSelf("clientnode", "cookie");
        OtpPeer sNode = new OtpPeer("servernode"+"@"+"chloe.ravnaandtines.com");
        OtpConnection connection = cNode.connect(sNode);

        OtpErlangObject[] args = new OtpErlangObject[]{
            new OtpErlangLong(1), new OtpErlangLong(2)};
        connection.sendRPC("mathserver", "add", args);
        OtpErlangLong sum = (OtpErlangLong)connection.receiveRPC();
        if (sum.intValue() != 3)
        {
                throw new IllegalStateException("Assertion failed, returned = " + sum.intValue());
        }
        System.out.println("OK!");

    }

}

Now, run that and get:

OK!

So, we have first light!

Get the goodies from here.

Tuesday 8 April 2008

Blast from the past

I had need to play with .ico files for the first time in ages the other day, and pulled out my old Java IcoImage utility to start with -- only to realise that it didn't handle modern 32-bit depth icons.

So I fixed it.

Latest versions here.