Erlang and Python

Following the suggestion that I try TwOtp, I've tried the "kicking the tyres" with that library.

Adapting the sample code to work against my own local node seemed easy enough:

from twisted.internet import reactor
from erlang import OneShotPortMapperFactory, buildNodeName

def gotConnection(inst):
    return inst.factory.callRemote(inst, "file", "get_cwd"
        ).addCallback(gotResult)

def gotResult(resp):
    print "Got response", resp
    reactor.stop()

def eb(error):
    print "Got error", error
    reactor.stop()

epmd = OneShotPortMapperFactory('client@chloe.ravnaandtines.com', "cookie")
epmd.connectToNode("servernode@chloe.ravnaandtines.com").addCallback(gotConnection).addErrback(eb)
reactor.run()

which involves rather more baggage than using IronPython against my tidying of the C# port of jinterface, for just a "hello world!", given that the jinterface-derived "hello world!" actually does more work (passing arguments, testing results):

import clr
clr.AddReference("OtpErlang.dll")
from System import Array

from Tinesware.Otp import *
import Tinesware.Otp.Erlang as Erlang

cNode = Self("clientnode@chloe.ravnaandtines.com", "cookie")
sNode = Peer("servernode@chloe.ravnaandtines.com")
connection = cNode.Connect(sNode)

args = Array[Erlang.IObject]([Erlang.Integer(1), Erlang.Integer(2)])

connection.SendRpc("mathserver", "add", args)
sum = connection.ReceiveRpc()

if  sum.Value.ToInt32() != 3:
  print "Assertion failed, returned = " + sum.Value
else:
  print "OK!"

And, like py_interface, I find that this doesn't "just work" either:

C:\Documents and Settings\Steve\My Documents\code\babel\erlang>python twotphw.py

Got error [Failure instance: Traceback (failure with no frames): <class 'erlang.epmd.NodeNotFound'>: 1]

which will require a similar amount of rolling sleeves up and getting hands dirty to resolve.

Erlang with Python and Ruby

Having played a bit with Erlang for real, in the sense of something non-trivial, even if not a complete product, I find I'm in the mood for more. But while it comes with some UI widgets, it's something more for headless use, and some client(s) in other languages doing the display.

To date, I can easily do Swing or WinForms as the UI toolkit -- but what about wxPython or FXRuby?

There are a number of bridges that I've found, but the closest to jinterface seem to be:

• py_interface for Python (about 2 years old)
• Un-named code drop on Google for Ruby (about 1 year old)

Neither of them have "just worked" for me in the way that the Java and C# versions have done against the simple mathserver example. Some of it might be down to my not figuring how the differences in the APIs are meant to work.

For py_interface, I had to comment out the line

[__import__(item) for item in __all__]

from init.py to get imports to work at all; and even then, the simple

from otp import *

def Handler(Result):
    print Result

cNode = erl_node.ErlNode("clientnode@chloe.ravnaandtines.com", erl_opts.ErlNodeOpts(cookie="cookie"))
print cNode
print cNode.Publish()
mBox = cNode.CreateMBox()
print mBox
mBox.SendRPC("servernode@chloe.ravnaandtines.com", 
    "mathserver", "add", [1, 2], Handler)
raw_input("wait")

just sits there; while the Ruby code

require 'erlang_node'
 
s_node = Erlang::Node.new("servernode@chloe.ravnaandtines.com")
c_node = Erlang::LocalNode.new("clientnode@chloe.ravnaandtines.com", "cookie")
connection = Erlang::Connection.new(s_node, c_node)
connection.sendRPC("mathserver", "add", [1,2])
sum = connection.receiveRPC()
puts "Fail" unless sum == 3

throws a different exception every time I run it, from some place or other inside the connection initializer.

So they both need deeper inspection than this simple kicking of the tyres, maybe modifying for R12 nodes, maybe just reverse engineering the kinks out of the driver programs.

Integrating .Net with Erlang, part 7

Another update, after work during the holiday weekend.

By using a lot of spare cycles, I found a test case for the division code not previously exercised:

9659964175213286916418497243082984930780677040655778513759952405592597918851617939992802011783376153721017703858175 / 3248063261698017931917474008545947342556579729240422363527 = 2974068974926080862795323621370295193198999568425344221053

Also made the astoundingly obvious discovery that I needed the real Erlang port mapper dæmon running, so put a unit test in which spawns an Erlang process to contact. It works fine in NUnit-gui, but fails every time in the build time coverage plus test cycle, so is labelled as [Explicit]. I need to to write at least one test that handles the call in the other direction, but this is getting close to as far as can easily be done in unit test form.

And it's getting close to my usual summer break from coding in the evenings, so this is probably close to being "it" for a while.

Integrating .Net with Erlang, part 6

Time for a major drop of the code.

It's been tidied up to remove most of the deprecation and to start unifying the comments as .Net XML. The BigInteger class has had a number of bugs found and fixed by unit testing (initialization, big left shifts, hex-string), and been made into a struct to reflect its value nature.

Pretty much all (say, 80%) of the stuff that can be tested statically in isolation has been, and the code has been revised with dependency injection and interfaces to do that (including a bug-fixed, back ported (to C# 2.0) version of NullObject.For<T>). Further testing will require some more extensive test set-up, such as spinning up a Java peer in a separate thread and talking TCP with it.

Code coverage is currently just over 60% for the Otp library, just over 70% for the ZLib code, for the static testing; beyond upgrading the wire types and the constants, the upgrade to 1.4 spec from 1.2 is not really begun.

Integrating .Net with Erlang, part 5

I have a first drop that has all the C# input code written, and a start of some unit tests for interoperability with the J# build of jinterface 1.4. Nowhere near enough tests are written yet, and a couple of those that have been (encoding a 64-bit integer in J# and unpacking in C#) don't yet work. But at least it's a start.

Still to do are the full gamut of integer types, and the complex ones -- lists, tuples, ports and such, then replicate for writing by C# and reading in J#.

One fix to the tests was for my code -- I needed to take the surprising little-endian wire format for an extended integer, and reverse it to fit into the BigInteger class I got from Codeplex; the second was a fix to that class:

381c381,384
<    
---
> 
>             // BUGFIX -- use an accumulator rather than |=ing into
>             // m_digits[m_digits.DataUsed], which will be zero
>             DType accumulator = 0;
384c387,389
<     m_digits[m_digits.DataUsed] |= (DType)(array[offset + leftOver - i] << ((i - 1) * 8));
---
>                 DType digit = array[offset + leftOver - i];
>                 digit = (digit << ((i - 1) * 8));
>                 accumulator |= digit;
385a391
>             m_digits[m_digits.DataUsed] = accumulator;

Code drop for the curious at the usual site. The project assumes you have NCover and NUnit around your system to run the tests and take coverage stats.

Update: More tests, and a few more fixes, refactorings and extensions added 24-Apr. Doubles now get encoded in the new style and integers that fall into the Int64 range are handled so the jinterface code can handle them.

Update: 26-Apr -- Added an export to little-endian byte array for BigInteger and added unit tests for short and long BigInteger representations.

Update: 27-Apr -- A first pass of Java type to C# and back for all the Java wire types, and a start of interface breaking changes to shut FxCop up a bit. Still needs a lot of like-for-like based unit tests, and detailed code cleaning.

Update: 28-Apr -- Unit tests about complete for all the wire types. Now the fun really begins.

Update: 4-May -- Unit tests extended to the input and output classes, including adding compression via a modified version of the ComponentAce port JZlib. All code now clean after FxCop scrubbing. Still needs all the node related classes being compared with the jinterface 1.4 versions and tested.

Integrating .Net with Erlang, part 4

So where are we going from here with the Otp.Net code? First off, if we bring the codebase into .Net 2.0 from its '04-'06 vintage, and make all the data classes in the Otp.Erlang namespace immutable -- returning ReadOnlyConnection(of T)rather than arrays as needed -- we can get rid of the ICloneable support. That lets us make Otp.Erlang.Object into an interface containing just the encode method(dropping the decode method, or making it an extension method as per C# 3.0).

Then we can use some subset of the fields used in equality tests to do GetHashCode properly, and replace arrays with growable collections.

Next, fix the range check on Double's floatValue method, and make it return a nullable value:

    public float? floatValue()
    {
        if (d > Single.MaxValue || d < Single.MinValue)
            return null;

        return (float) d;
    }

and rename Long as Integer, kill both their derived classes and make Integer carry a BigInteger to which we can delegate conversions returning nullables.

That will mean some fixing up of input and output, but we need that anyway to handle new-style float values encoded as 8-byte big-endian blobs (and the change of tense means that this is where we move past what I've done so far).

To come after that will be making GenericQueue delegate to Queue(of IObject), and then a general scrub done in comparison with jinterface 1.4, before it will be sorta-releasable.

At the moment it builds, but will not work, so no interim archive is yet publicly available.

Integrating .Net with Erlang, part 3

A kind anonymous has pointed out that someone has already done the job of porting an earlier version (1.2.1.1) of the jinterface code, even though it has not made its way into the standard distro. Via this blog post, one eventually gets to the sourceforge repository details for it.

This actually does work with IronPython (after the trivial edits to load the appropriate assembly and the slightly different class names) and without the J# runtime being involved.

I spotted the comment after spending a little while porting the 1.4 code. Actually the underlying Java code is quite horrid -- to start with there are lots of methods with the same name as fields; and a lot of identically named fields shadowing ones elsewhere in the inheritance hierarchy; and methods and fields are used interchangeably. Then there are blocks of cut-and-paste and tweak code; plenty of swallowing of exceptions and blurred responsibilities (where one object performs a series of manipulations on a member of another object). And there there is this (where d is a double):

  public float floatValue()
    throws OtpErlangRangeException {
    float f = (float)d;

    if (f != d) {
      throw new OtpErlangRangeException("Value too large for float: " + d);
    }

    return f;
  }

which doesn't look too tested to me -- what's this (f != d), in case you couldn't spot it. This is carried over into otp.net, and, I think, shows how little floating point is used.

I fixed up the name clashes and some of the blurred responsibility in the Java, then ran the conversion over; and by taking out the BigInteger stuff (not there in otp.net's Long class -- haven't checked the rest of these), compression, most of the floating point, got what was left to build, though it currently fails in the challenge/response handshake (where otp.net has replaced the roll-your-own MD5 with calls into System.Cryptography).

Where could we go from here?

Two main things need doing. The otp.net code needs to be brought up to the 1.4 standard and make better use of .Net 2.0 e.g. for tracing; and it needs to be refactored to clean up inherited code smells.

Ideally, some of the changes should be back-ported into jinterface, too.

To date, this includes fallback to Java's file based location for cookie and a replacement for some of the non-ported tracing code (logging the remote endpoint).