My hobbyist coding updates and releases as the mysterious "Mr. Tines"

Friday 28 December 2007

Phase 2 — “The best code is code someone else has already written and debugged.”

So goes the old saying.

And looking at the almost decade old Angerona project code, I see there's a lot of code that is written, was lightly debugged, probably works, and just needs refactoring. After all, it was the lack of a ZLib that stopped that effort dead, not anything with the existing code.

So, the next phase will be to put together an infrastructure library (jar/dll) from the useful bits, which should be enough to get to the point of being able to parse a key-ring or similar binary file to OpenPGP format.

At that point, some weeks hence, it will make sense to put a first bit of GUI on top; and start filling in the gaps in a less clunky language than Java where possible.

Tuesday 25 December 2007

angerona.algorithms 1.0.2915.29952

Here. and a back-up here if Demon's DNS flakes out.

A complete replacement for the old crypt.zip, with source and pre built .jar and .dll, unit tests, and a .Net build of junit 3.8.2 -- includes:

  • JZlib (inflate/deflate)
  • Blowfish, CAST5, DES (including s3' and key-dependent variants), IDEA, Safer, Square, TEA, ThreeWay, TripleDES (encryption)
  • BlockCypherHash, Haval, MD5, RIPEM160, SHA0, SHA1 (message digests); and
  • Bignum (mutable multiple precision integers).

Needs the J# redistributables for .Net use,

Happy holidays! and 100th post!

Thursday 20 December 2007

J# and Ruby.Net — string and baling wire

A proof of concept

//Factory.java
public class Factory
{
    private Factory(){}

    public static Object Build(String classname) 
        throws InstantiationException, IllegalAccessException, 
        ClassNotFoundException
    {
        Class target = Class.forName(classname);
        return target.newInstance();
    }
}
//Utility.java
package Tinesware.Adapter;

public class Utility
{
    private Utility() { }

    public static void exit(int status)
    {
        java.lang.System.exit(status);
    }
}
//WindowAdapter.java
package Tinesware.Adapter;

public class WindowAdapter implements java11.awt.event.WindowListener  
{
    private final Ruby.Object payload;

    public WindowAdapter(Ruby.Object arg)
    {
        payload = arg;
    }

    public void windowActivated(java11.awt.event.WindowEvent e)
    {
    }

    public void windowClosed(java11.awt.event.WindowEvent e)
    {
        System.out.println("window Closed");
        Object[] args = {e};
        Ruby.Runtime.Eval.Call(payload, "windowClosed", args);
    }

    public void windowClosing(java11.awt.event.WindowEvent e)
    {
        System.out.println("window Closing");
        Object[] args = { e };
        Ruby.Runtime.Eval.Call(payload, "windowClosing", args);
    }

    public void windowDeactivated(java11.awt.event.WindowEvent e)
    {
    }

    public void windowDeiconified(java11.awt.event.WindowEvent e)
    {
    }

    public void windowIconified(java11.awt.event.WindowEvent e)
    {
    }

    public void windowOpened(java11.awt.event.WindowEvent e)
    {
    }
}
##Program.rb
require 'Tinesware.Adapter.dll'
require 'gwt221.dll'

window = Tinesware::Adapter::Factory.Build("dtai.gwt.GadgetFrame");
window.setSize (800, 600)
window.setTitle 'Proof of Concept'
window.show

class WindowHandler
  def initialize(w)
    @window = w
   end

  def windowClosing(e)
    @window.hide
  end
  def windowClosed(e)
    Tinesware::Adapter::Utility.exit 0
  end
end

adapter = Tinesware::Adapter::WindowAdapter.new(WindowHandler.new(window))
window.addWindowListener adapter

It only works grace of Ruby.Runtime.Eval.Call() being public, but that's all I need; that the event argument gets through is a bonus.

It is a mild hack, it does involve repetitive boilerplate coding, but you only have to do it for a few interface classes. And it will need some smoothing over on the Ruby side to make it look less awkward with code that's supposed to interoperate with Java as JRuby.

Extra bonus -- if the Call()'d method doesn't exist, it does nothing silently, so you only have to implement the parts of an interface you actually want.

Monday 17 December 2007

J# and Ruby.Net — foiled again

Carrying on from last post…

// C#
        public static void typeTest(object arg)
        {
            System.Type wtf = arg.GetType();
            Console.WriteLine(wtf.ToString());
            Console.WriteLine("Base = "+wtf.BaseType.ToString());

            System.Type[] iface = wtf.GetInterfaces();
            Console.WriteLine("# interfaces = " + iface.Length);
            foreach (System.Type t in iface)
                Console.WriteLine(":: "+t.ToString())
        }
##Ruby caller
q = Observer.new
JsharpToRuby::ClassDictionary.typeTest q

yields disappointing results

Observer
Base = Ruby.Object
# interfaces = 0

Oh dear! it doesn't seem to work the naïve way for proper interfaces

// C#
namespace JsharpToRuby
{
    public interface ITest
    {
        Boolean isPresent();
    }
##Ruby 
class EyeTest 
  include JsharpToRuby::ITest
  def isPresent
    1
  end
end
q = EyeTest.new
puts q.class
q.class.ancestors.each { |x| puts x }
JsharpToRuby::ClassDictionary.typeTest q

yields

EyeTest
EyeTest
ITest
Object
Kernel
EyeTest
Base = Ruby.Object
# interfaces = 0

This is of course the bit missed out in the interop tutorial, alas; I think for the good and simple reason that it is not yet implemented (from a quick inspection of the code).

*le sigh*

Maybe not all is lost. It would mean more hand-rolled adapter classes to wrap the objects and delegate by reflection.

But later, definitely, later.

Sunday 16 December 2007

J# and Ruby.Net — further steps

So what do we need Java classes for in a JRuby/Ruby.Net environment?

Things that we have to new because that defines system services in an unambiguous fashion — I'm thinking encodings for I/O and GUI components here, as well as libraries already written that we don't want to port just yet (my algorithms library); and interfaces/abstract classes we want to make concrete (needing to be an ActionListener or similar).

Concrete things are easy; abstractions are trickier, but not impossible.

//C# code to generate a Ruby class type
        public static Ruby.Class getThing()
        {
            object o = new java.awt.Panel();
            return Ruby.Class.CLASS_OF(o);
        }
##Ruby code to use it
thing = JsharpToRuby::ClassDictionary.getThing

puts thing.to_s

puts thing.ancestors

panel = thing.new
puts panel.to_s

which generates

Panel
Container
Component
Serializable
MenuContainer
ImageObserver
Object
Object
Kernel
java.awt.Panel[panel1,0,0,0x0,invalid,layout=java.awt.FlowLayout]

which goes … java.lang.Object, Ruby Object, …

So now try

io = nil
thing.ancestors.each { |x| io = x if x.to_s.eql? "ImageObserver" }
MyImageObserver = io

class Observer 
  include MyImageObserver
  def imageUpdate(img, infoflags, x, y, width, height) 
    nil
   end
end

q = Observer.new
puts q.class.ancestors

to get

Observer
ImageObserver
Object
Kernel

which means the task is not immediately insuperable; but may need some hand-rolling of concrete classes for some of the interesting interfaces, rather than being something that can be totally read-only…

J# and Ruby.Net — first steps

This seems to work

//C# assembly referencing the Ruby.NET.Runtime assembly
namespace Inspect
{
    public class Access
    {
        public static Ruby.Class getRubyPerson()
        {
            ClassLibrary2.nested.Person p = new ClassLibrary2.nested.Person("", 0);
            return Ruby.Class.CLASS_OF(p);
        }
    }
}
## Ruby 
x = Inspect::Access.getRubyPerson()
fred = x.new(name, age)
fred.print

It's a pity that I need an object and cannot just send a System.Type into Interop.CLRClass.Load(type, null, false);, which this ends up doing, but that method is internal to the Runtime DLL.

It should be possible, though, to have a require_jsharp('assembly.dll') that does the tedious running around back and forth through the Ruby/C# boundary in an anutmated fashion, and creates appropriate proxies for lowercase package-names in a tree-like object.

Thursday 13 December 2007

J# and Ruby.Net — not all is rosy in the garden

Ruby expects modules to be constants (capitalised). Just like .Net namespace naming conventions dictate.

Java, OTOH, uses lower-case package names which map directly to .Net namespaces, which cause Ruby.Net to bork when trying to instantiate classes. In fact, this is a general issue — it affects any .Net module names which are not capitalised, like this C# code

namespace ClassLibrary2.nested
{
    public class Person
    {
        private String name;
        private int age;

        public Person(String name, int age)
        {
            this.name = name;
            this.age = age;
        }

        public void print()
        {
            Console.WriteLine("Hello " + name + " " + age);
        }
    }
}

which won't work when invoked by

require 'ClassLibrary2.dll'

name = 'Fred'
age = 42
fred = ClassLibrary2::nested::Person.new(name, age)
fred.print

But if you change nested to Nested, all is well.

*le sigh* — that means an adapter layer between Java-style names and Ruby/.Net ones. At least that should be doable in Java/J#, even if it makes things more painful than I had hoped.

Sunday 9 December 2007

What's wrong with this code?

Spent today getting about 2/3 coverage on the Bignums class, including rewriting the modularInverse function to complete in a finite time, based on the recursive method given in Wikipedia -- I can always redo it as iterative if there are problems, but for now I have achieved RSA.

So everything was running fine in the JVM; go to .Net, and … infinite loop trying to divide 40! by 20! (in a bit taken from the Ruby bignum unit tests. Eventually, I tracked it down to this:

    private static void s_m_mult_sub(
        int[] result, int ro,
        int[] multi, int mo,
        int scalar, int length)
    {
        long borrow = 0;
        int index = 0;

        /* Is basically practical => proceed */
        while(index < length)
        {
            long product = multUnitxUnit(scalar, multi[mo+index]) + borrow;
            borrow = product >>> 32;
            product &= MASK;
            if(product > (result[ro+index]&MASK))
                ++borrow;
            result[ro+index++] -= (int)product;
        }
        result[ro+index] -= (int)(borrow&MASK);
    }

which Java -- and the original 'C' from which this was taken -- runs "as you'd expect" but J# handles as if it were

result[ro+index++] = result[ro+index++] - (int)product;
and increments index twice, as well as splattering other mayhem around the structure.

Next up, getting JZlib tidied and turning its tests into JUnit form, and adding the PGP needed twiddles; then a first release to replace my old crypt.zip.

The original 'C' code:

static void s_m_mult_sub(unit result[], unit multi[], unit scalar, unit length)
{
    register unit high, low1, low2;
    unit borrow = 0;
    unit index = 0;

    /* Is basically practical => proceed */
    while(index < length)
    {
        multUnitxUnit(high, low1, scalar, multi[(size_t)index]);
        low2 = low1 + borrow;
        if(low2 < low1) high++;
        if(low2 > result[(size_t)index])
            borrow = high + 1;
        else
            borrow = high;
        result[(size_t)(index++)] -= low2;
    }
    assert(result[(size_t)index] >= borrow);
    result[(size_t)index] -= borrow;
}

Another step

Packagecom.ravnaandtines.crypt.mdacom.ravnaandtines.crypt.cea
# Classes831
Line Coverage95% 1051/110499% 1641/1659
Branch Coverage81% 170/21095% 399/422
Complexity2.3692.146

That includes real test vectors for TEA (and fixing the implementation at last, so it's incompatible with what I used to have for Java, but is real for the first time -- problems with operator precedence and signed vs unsigned shift), as well as things like Safer not implementing the common interface and a few problems with non-zero array offsets and with triple encryption.

Just need to get bignums and ZLib sorted out with tests, and it'll be time for a proper drop.

Tuesday 4 December 2007

Moore at work

The CAST5 algorithm defines a maintenance test:

B.2. Full Maintenance Test

    A maintenance test for CAST-128 has been defined to verify the
    correctness of implementations.  It is defined in pseudo-code as
    follows, where a and b are 128-bit vectors, aL and aR are the
    leftmost and rightmost halves of a, bL and bR are the leftmost and
    rightmost halves of b, and encrypt(d,k) is the encryption in ECB mode
    of block d under key k.

    Initial a = 01 23 45 67 12 34 56 78 23 45 67 89 34 56 78 9A (hex)
    Initial b = 01 23 45 67 12 34 56 78 23 45 67 89 34 56 78 9A (hex)
 
    do 1,000,000 times
    {
        aL = encrypt(aL,b)
        aR = encrypt(aR,b)
        bL = encrypt(bL,a)
        bR = encrypt(bR,a)
    }

 Verify a == EE A9 D0 A2 49 FD 3B A6 B3 43 6F B8 9D 6D CA 92 (hex)
 Verify b == B2 C9 5E B0 0C 31 AD 71 80 AC 05 B8 E8 3D 69 6E (hex)

In my code I had a comment "takes about an hour to run", dating from about the end of 1998. Trying today, I got

  • In shiny new NetBeans 6.0, run from a JUnit 3.8.2 unit test with cobertura coverage instrumentation: 885s
  • With instrumentation disabled: 6s
  • In VS 2005 as a post-build run of NCover 1.5.8 running the ported test as a debug build: 34s
  • In VS 2005 as a post-build run of NCover 1.5.8 running the ported test as a release build: 17s
  • In VS 2005 as a command-line run of the test as a release build (100 lines of output piped to a file, rather than the console): 6s

So, you pay through the nose (~150 times slower) for your branch coverage instrumentation, as opposed to line coverage (~6 times). And both JVM and CLR are seriously faster, even allowing for the maybe 20x faster processor, than what I was running a decade ago.

Oh, yeah, today's unit test…

It passes.

Sunday 2 December 2007

Making a hash of it

Wrote one (meta-)test today. Find every class in com.ravnaandtines.crypt.mda, instantiate through the default constructor, and check that putting in a short sequence of bytes one at a time, as an array or as an array-slice all gave the same results.

On the up-side, this ups coverage:

Packagecom.ravnaandtines.crypt.mda
# Classes8
Line Coverage95% 1051/1104
Branch Coverage81% 170/210
Complexity2.369

and found one bug, in BlockCypherHash, where partial blocks were being lost (I wasn't updating the count of pending bytes when I should).

On the down side, while the test driver was easy to write for J#

MD5 prototype = new MD5(); // an arbitrary class in the right assembly
System.Reflection.Assembly algorithms = prototype.GetType().get_Assembly();
System.Type[] types = algorithms.GetTypes();
System.Type mda = algorithms.GetType("com.ravnaandtines.crypt.mda.MDA");

for (int k = 0; k < types.length; ++k )
{
 final System.Type underTest = types[k];
 if (!mda.IsAssignableFrom(underTest))
 {
  continue;
 }
…

but for Java, it requires some magic to do similar blind enumeration of classes. The following fudge (based off this one, with the fix noted below) gets the all loaded classes in a named package (stuck with Java 1.1 for convenience when cross compiling)

    public static java.util.Vector find(final String pckgname) {
        // Translate the package name into an absolute path
        String name = pckgname;
        if (!name.startsWith("/")) {
            name = "/" + name;
        }
        name = name.replace('.', '/');
        // Get a File object for the package
        final java.net.URL url = // get an instance of the current class to hook from
                new CryptTestUtils().getClass().getResource(name);

        // NOTE -- this url will put %20 for spaces in your path 
        // It needs to be cleared up before creating a File object
        final String dirpath = java.net.URLDecoder.decode(// 1.2 class easily stubbed for J#
                url.getFile()); 

        final java.io.File directory = new java.io.File(dirpath);
        final java.util.Vector contents = new java.util.Vector();
        if (directory.exists()) {
            // Get the list of the files contained in the package
            final String[] files = directory.list();
            for (int i = 0; i < files.length; i++) {

                // we are only interested in .class files
                if (files[i].endsWith(".class")) {
                    // removes the .class extension
                    final String classname = 
                            files[i].substring(0, files[i].length() - 6);
                    try {
                        // Try to create an instance of the object
                        final Object obj = 
                                Class.forName(pckgname + "." + classname).newInstance();
                        contents.add(obj.getClass()); // save off the class
                    } catch (ClassNotFoundException cnfex) { 
                    } catch (InstantiationException iex) { 
                    // We try to instantiate an interface
                    // or an object that does not have a 
                    // default constructor
                    } catch (IllegalAccessException iaex) { 
                    // The class is not public
                    }
                }
            }
        }
            return contents;
    }

and then we do

final java.util.Vector classes = 
                com.ravnaandtines.crypt.CryptTestUtils.find(
                "com.ravnaandtines.crypt.mda"
                );       

for (final java.util.Enumeration iter = classes.elements(); iter.hasMoreElements();)
{
      final Class underTest = (Class) iter.nextElement();

      final Object test = underTest.newInstance();
      if(!(test instanceof com.ravnaandtines.crypt.mda.MDA ))
      {
          continue;
      }
…

So, that's all the hash algorithms done. Next is the similarly mindless conversion of the crypto algorithm test vectors (and a similar meta-test). Then turning the JZlib system tests into a set of unit tests, and porting something like the Ruby Bignum unit tests. Maybe by the turn of the year, I shall actually have a platform based on existing work that can be taken forward.

Saturday 1 December 2007

Tinkering away

The standard test vectors for the various algorithms do a good job of shaking down the code for an implementation.

Packagecom.ravnaandtines.crypt.mda
# Classes8
Line Coverage93% 1024/1103
Branch Coverage76% 159/210
Complexity2.369

Coverage would be higher if there weren't a still few CTC-related entry-points, or code-paths like compensating for a broken Haval implementation in a PGP extension (EPB)…

Friday 30 November 2007

Life in the old dog yet…

Contradicting my earlier assertion that “it's only when you want GUI that you have to get off the fence.” — I've built the Java 1.0-based Gadget Windowing Toolkit against JVM for Java 5 and (with just a little added C#) CLR.

It might not be “teh shiny”, and won't give you mouse wheel events, but it's the fastest way to a common GUI toolkit for both VMs.

Weaning it off the Java 1.0 event model that it uses inside a 1.1 emulator, and using Container instead of its own Gadget class (written because Container and Component had protected constructors at Java 1.0) would make it a lot leaner, and more robust, but it's not a simple matter of mechanically replacing the classes. There is a lot of code dedicated to managing focus and graphics contexts — including managing the masks and offsets — that would have to be carefully moved across.

And of course, there's only the one huge demo/test, rather than unit tests.

Tuesday 27 November 2007

Smelling the coffee…

… before work goes crazy in the 4 weeks left to holidays — a first drop of angerona.redcoffee.zip; what may (ha! ha!) become a hybrid Java/Ruby CTClib equivalent for both JVM and CLR.

This contains code for various symmetric crypto algorithms, strong hashes, erasable big numbers and Zlib compression (based on JZlib 1.0.7); it contains some initial JUnit tests aimed at JUnit 3.8.2 in NetBeans 6.0 (currently using RC1). It also contains a project to build JUnit 3.8.2 (almost) as a J# project generating a command line executable (.exe also included). The "almost" is because the assertEquals() methods for Double and boolean have been taken from 3.8.1 to permit them to compile under J#.

The NetBeans project has been amended to use cobertura 1.9 to perform coverage analysis as the unit tests are run -- assumed to be in C:\cobertura-1.9, adjust build.xml to fit your location; or use it as a prototype for other projects. The VS 2005 solution builds the JUnit ~3.8.2 executable, the angerona.algorithms.dll library, and the unit tests as three projects; and as a post-build step for the last runs the tests under NCover (assumed to be installed to directory C:\Program Files\NCover) -- I'm using 1.5.4, having had some problems with 1.5.8 run over managed C++ code not contained in gc classes.

Also included is an FxCop project; I shall be using that run manually, and PMD as a live plug-in while the code is being groomed; they will get incorporated into the build process later when the noise level has been considerably reduced.

Note that the two build environments are set up to use the same source file structures, though J# has to be hand-held to point it at the files that NetBeans just picks up automagically.

The only files in any sort of stable state at the moment, the proofs of concept, are SHA0.java and SHA1.java, which have unit tests based on the FIPS PUB 180-1 test vectors. These tests all pass and give 100% coverage (including branch coverage) to the base SHA class and the two wrappers. The first phase will be getting the state of no PMD warnings, no FxCop warnings (except where naming conventions are involved, and there, Java conventions set out by PMD will win if there is a conflict), unit tests with ideally 100% coverage; this will overlap with doing some refactoring.

Then with a stable foundation, the plan will be to gradually migrate the rest of CTClib, primarily to Ruby, but probably with a few Java interface types to make the external interface more easily usable by other JVM or CLR projects.

Archive is 257,055 bytes; MD5 7335e89a 171560f3 003f225e d97a3b3c & SHA-1 4d421c00 d4b3b45d be0244e3 649b19bd 04923a94.

Monday 26 November 2007

Shaking some dust off…

After a long distraction by paying work, I'm starting to turn attention back to projects. The IronPython FTP client still needs wiring up (and bug fixing), but a few developments in recent months -- JRuby, and now Ruby.Net 0.9, have led me to turn my attention once again to taking my old PGP-emulator library, currently languishing as 'C' and a number of abortive ports, out of the 1970s.

I have several key bits -- crypto algorithm implementations I rolled myself, and, now the JZlib port of the key ZLib component (for reasons best known to PRZ, the version in PGP 2.x uses an 8k window, and no standard APIs give you control over that so as to be able to emit compatible compression) -- in Java. But now I could build the rest of the system in Ruby and build it to the CLR as well as the JVM.

Then it's only when you want GUI that you have to get off the fence.

Of course, uploading the library will mean I shall want to have completed my improved FTP client, so win-win,… or something

Meanwhile a long term-goal for self-development stuff at work will involve some F#…

Friday 28 September 2007

One small step…

A first part of the refactoring of the file system representation for the local files in the FTP client, removing all use of module os (to simply deployment by taking out all dependencies on FePy's extended library code), and encapsulating all references to the file data in a LocalFile class.

TODO list includes:

  • saving settings
  • hooking a remote file system class to the FTP client class
  • linking the FTP client and any low-level error feedback to the UI
  • Ensuring that file transfers but not directory listings get caught by the progress bar
  • Handling reconnects intelligently, including recording current directory state on the remote site

Drop here again.

Sunday 16 September 2007

Coding formula

Normal service has resumed -- i.e. hobby coding is light again, as I'm deep into the dev cycle on the current project at work, the weather has improved since the height of the summer so I'm out at weekends, except I've been being distracted by my teeth, and have just been moping.

Tuesday 28 August 2007

10 Minute refactor

After a long weekend in the garden...

module Com_ravnaandtines
  module Zlib
    class Adler32
      # largest prime smaller than 65536
      @@ADLER_BASE = 65521
    
      def initialize
        @value = 1
      end
      
      def update(buffer)
        if not buffer
          return @value
        end
        ## build up the checksum
        low = @value & 0xffff
        high = (@value >> 16) & 0xffff
        buffer.each do |x| 
          low += (x.to_i & 0xff)
          high += low
        end

        ## collapse into modular parts
        low %= @@ADLER_BASE
        high %= @@ADLER_BASE
        @value = (high << 16) | low      
      end
      
      def reset
        @value = 1
      end
      
    end
  end
end

and tweak the tests thus:

  def basic_test_engine(seq, expected)
    ## string to array
    ## otherwise expect an each method to yield integers
    if seq.respond_to? :unpack
      seq = seq.unpack("C*")
    end
    adler = Com_ravnaandtines::Zlib::Adler32.new()
    a = adler.update(seq)
    assert_equal(expected, a)
  end

  def test_boundary
    basic_test_engine(nil, 1)  
  end

That feels better. No special cases, no leakage of state. Looking at JZlib, most of the work will be in doing the encapsulation properly.

Next up, though, when time and energy combine, more on the IronPython FTP client.

Friday 24 August 2007

Selection Factors

I had started porting CTClib to managed C++ to do an Iron<something> piecemeal conversion. But at the moment IronPython doesn't have a good deployment story, IronRuby isn't all there; and I still much prefer Java's UI model to WinForms or WPF… So what about JRuby -- familiar UI style and run from jar -- then?

The stumbling block as always is PhilZ's choice of deflate with a 2^13 bit window (as opposed to Zlib's fixed 2^15 bit window size for Ruby or java.util.zip) for compression. This was where I paused my first Java port -- JZlib which could do the job has appeared since I last looked, around the turn of the century. Even so, for sake of portability I've decided to bite the bullet, and do a minimal zlib/deflate implementation in Ruby for the purpose, to do something meaningful with the language, using JZlib as a guide. Inflate can, of course, have the larger window (as in current CTClib-C builds), and use the built-in version, be it the C version from native Ruby or JRuby's java.util.zip wrapper.

So, the easy bit first -- Adler 32 checksum…

module Com_ravnaandtines
  module Zlib
    # largest prime smaller than 65536
    ADLER_BASE = 65521
    # Adler32 checksum : takes a seed (usually 1), and a byte sequence, 
    # returns 32-bit integer
    def adler32(adler, buffer)
      if not buffer
        return 1
      end
      
      ## string to array
      ## otherwise expect an each method to yield integers
      if buffer.respond_to? :unpack
        buffer = buffer.unpack("C*")
      end
      
      ## build up the checksum
      low = adler & 0xffff
      high = (adler >> 16) & 0xffff
      buffer.each do |x| 
        low += (x.to_i & 0xff)
        high += low
      end

      ## collapse into modular parts
      low %= ADLER_BASE
      high %= ADLER_BASE
      (high << 16) | low
    end
  end
end

Test vectors for the unit tests had to be scavenged from the internet:

require 'test/unit'
require 'tinesware_zlib'
include Com_ravnaandtines::Zlib

class AdlerTest < Test::Unit::TestCase

  def basic_test_engine(seq, expected)
    a = Com_ravnaandtines::Zlib.adler32(1, seq)
    assert_equal(expected, a)
  end

  def test_boundary
    a = Com_ravnaandtines::Zlib.adler32(0, nil)
    assert_equal(1, a)    
  end

  def test_simple_0
    basic_test_engine("Mark Adler", 0x13070394)
  end
  
  def test_simple_1
    basic_test_engine("\x00\x01\x02\x03", 0x000e0007)
  end
 
  def test_simple_2
    basic_test_engine("\x00\x01\x02\x03\x04\x05\x06\x07",  0x005c001d)
  end

  def test_simple_3
    basic_test_engine("\x00\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f", 0x02b80079)
  end

  def test_simple_4
    basic_test_engine("\x41\x41\x41\x41", 0x028e0105)
  end

  def test_simple_5
    basic_test_engine("\x42\x42\x42\x42\x42\x42\x42\x42", 0x09500211)
  end

  def test_simple_6
    basic_test_engine("\x43\x43\x43\x43\x43\x43\x43\x43\x43\x43\x43\x43\x43\x43\x43\x43", 0x23a80431)
  end

  class Vector #arrays filled with value = (byte) index
    def initialize(size)
      @size = size
    end
    def each
      index = 0
      while index < @size
        yield index & 0xff
        index += 1
      end
    end
  end

  def test_total
    index = 0
    results = [  486795068,
                1525910894,
                3543032800,
                2483946130,
                4150712693,
                3878123687,
                3650897945,
                1682829244,
                1842395054,
                 460416992,
                3287492690,
                 479453429,
                3960773095,
                2008242969,
                4130540683,
                1021367854,
                4065361952,
                2081116754,
                4033606837,
                1162071911 ]

    
    while index < 20
      size = 5*index + 1
      xx = Vector.new(1000*size)
      basic_test_engine(xx, results[index])
      index += 1
    end
  end

end

That was one evening. Now, how long will the rest of it take?

Sunday 19 August 2007

A wet weekend

The FTP client code is pretty much there, though I still need to write code to drain headers for the HTTP proxy case. And then the rest is pretty much just plumbing it all together -- of which there will be a fair volume.

Today's screen capture is how the UI will look during a transfer -- a float-up panel with a live throbber in front of the disabled directory views.

Saturday 18 August 2007

After a break

Last week I was at Recombination so nothing done then.

Now I've started to wire up the logon dialog. This included a bit of shuffling of the main UI, and preparing to split the DirectoryPanel into UI (repeated) and Directory Model classes (local and FTP); and starting the process of writing an FTP client against the .Net socket APIs.

Writing the logon dialog exposed one Mono quirk. The UseSystemPasswordChar property on the text-box doesn't seem to do anything. I resorted to self.pwdTB.PasswordChar = u'\u2022' as a portable solution.

Also, you can use the Data Protection API for more safely storing passwords:

import clr
clr.AddReference("System.Security")
from System.Security.Cryptography import *
import System.Text
import System

…

  def getSecureValue(self, section, key, default):
    raw = self.getValue(section, key, None)
    if raw== None:
      return default
    try:
      array = System.Convert.FromBase64String(raw)
      chars = ProtectedData.Unprotect(array,
        None, DataProtectionScope.CurrentUser)
      result = System.Text.Encoding.UTF8.GetString(chars)
      for i in range(chars.Length):
        chars[i] = 0
      return result
    except System.Exception, ex:
      print ex.ToString()
      return default

  def setSecureValue(self, section, key, value):
    bytes = System.Text.Encoding.UTF8.GetBytes(value)
    try:
      safed = ProtectedData.Protect(bytes,
        None, DataProtectionScope.CurrentUser)
      for i in range(bytes.Length):
        bytes[i] = 0
      string = System.Convert.ToBase64String(safed)
      for i in range(safed.Length):
        safed[i] = 0
      self.setValue(section, key, string)
    except System.Exception, ex:
      print ex.ToString()

which stores the password as an encrypted Base64 blob -- and works with Mono on Win32 as well. The weak point is the password kept in memory as a string (immutable) -- if you can use it as char array, you can wipe that when done.

Tuesday 7 August 2007

Meets minimum

After a bit more fiddling with P/Invoke code, I have hooked into the Win32 Recycle or Delete APIs; though at the moment multiple file delete is done one file at a time. But the directory pane is roughly meets-minimum standard.

Next I can get to the object of the exercise, making an FTP connection and getting the remote directory information back to its own version of DirectoryPane. This will be an interesting combination of BackgroundWorker and asynchronous I/O, with pauses built in so as to throttle the uplink to what my router is prepared to handle.

And then I get to tweak it -- like switching to WPF if available -- and port it -- Mono on Debian working around the P/Invoke stuff.

Sunday 5 August 2007

Directory pane almost there

Some of the problems I was cursing Mono for turn out to be IronPython 1.x-isms -- mainly the insisting on having Images rather than Icons. Mono has other peculiarities, like in how it actually lays out widgets (not that .Net is much better) at runtime, so for the moment I'm forcing the issue by trapping any size change to the directory pane so that it fits inside the viewport.

Apart from wiring up the delete context menu, the directory pane is done.

Saturday 4 August 2007

More Mono

A lot of work done for the benefit of Mono, redoing how I wrap files to use os.path rather than .Net directly; and rewriting the P/Invoke layer from scratch (and from pinvoke.net) --including finding that Bitmap.FromHicon() is the simplest way of getting the transparent pixels roughly honoured as well as feeding that system what it wants.

I said that the DirectoryPane would be the worst of it, and it's doing its best to prove me right.

Monday 30 July 2007

Not quite so pretty

After remarkably little work, it runs under Mono 1.2.4 and IPCE-r6. The main changes are that as Mono doesn't support System.Management (and the extra work wasn't useful for the icon related info), I fell back to Directory.GetLogicalDrives(), and had to do a bit of type tweaking (explicitly converting to Bitmap to draw the icons).

Everything else is non-visible : the lower left sector has been factored out into a separate class and modules have been rationalised, ahead of making a next step in functionality.

First drop for the curious and for back-up distribution. Now updated.

Sunday 29 July 2007

You move 16 tons...

The gaps in the Win32 API support in .Net are random and frustrating. You can get all the drive letters (and the "DriveType" or "MediaType" properties which together give the offset into Shell32.dll for the drive icon), but short of unpicking the PE format by hand, Google gives no way to get the icons without using P/Invoke.

So with a combination of existing recipes for wrapping SHGetFileInfo in C#, and feeding it through Fuzzyman's dynamic compiler utility, I have some approximately script-like behaviour for icon extraction -- as can be seen from the left-hand dropdown in the screenshot.

I still need to add some extra hooks to use SHGetFileInfo to check if a file is executable or not, so I can get custom icons, and then refactor out a proper utility. And there's still more work on the generic directory pane widget before it'll be something stand-alone (sorted file list, hooking up events); though I hope that should not require any more exploting of exotic APIs to achieve.

Saturday 28 July 2007

Sneak Peek

I've starting recoding my FTP client in IronPython, mainly as a self-teaching exercise (but hoping it might work in Mono/IPCE too). So far, this is the shell of the main UI, with a ribbon-style bar at the top.

The biggest piece of work is likely to be the Directory Pane component that Fox just gave to me on a plate.

Friday 29 June 2007

PassiveFTP 1.10

Now available on the home server too.

Wednesday 27 June 2007

PassiveFTP 1.10

As promised…

PassiveFTP 1.10 — build 1.10.2734.38706 — squeezes in an up-one-level button into the remote site list, for FTP sites that manage to conceal their ".." directories. Or for people who'd prefer it anyway.

From the GeoCities Windows page at the moment. I need to do a physical rather than a remote logon to my home server before I can upload to that, so in a day or two.

Tuesday 26 June 2007

PassiveFTP -- new revision

As the revised Demon homepages site doesn't provide ".." as a directory in the default ls response, I shall have to add an explicit "go up one level" button to the server side.

Expect version 1.10 soonish.

Sunday 24 June 2007

Current directions

CTClib/Angerona is taking a strange turn.

I want to use IronPython for .Net and Mono UI; but for the core library keep it as pure Python as feasible. But to get there, I'm porting via C++/CLI.

I have most of it building (I have not ported I/O, or deflate as yet), and I'm separating out the test code into a more formal set of unit tests.

The plan goes get it into "managed 'C'" with test coverage (side release 1); refactor to provide CLR interfaces at suitable places (side release 2); then pull code through the language barrier, one module at a time, to achieve a pyCTC library (library release 1), followed by a WinForms GUI on top. Or maybe Jython for the GUI.

No promises, as usual.

Friday 27 April 2007

Drophash in Python dialects

Iron Python 1.1

Now for IronPython 1.1 — annoyingly the WebBrowser control doesn't accept normal DnD, nor does it let you paint it to a bitmap to blit on another control, so this one uses a ListView.

import clr
clr.AddReferenceByPartialName("System.Windows.Forms")
clr.AddReferenceByPartialName("System.Drawing")

from System.Windows.Forms import *
from System.Drawing import *

from md5 import *
from sha import *

pane = None
hashes = [md5, sha]
nbsp = list(" ")


def getHash(file, hashOp, delimiter) :

    try :
        reader = open(file, 'rb')
    except :
        return ""
   
    try :
        digest = hashOp()
        while True :
            chunk = reader.read(1024)
            if not chunk :
                break
            digest.update(chunk)
    finally:
        reader.close()

    raw = digest.hexdigest()
    work = []
    i = 0
    while i < len(raw) :
        work.append(raw[i])
        i += 1
        if 0 == (i%8) :
            work = work + delimiter

    return ''.join(work)

def dragOver(sender, evt):
  evt.Effect = DragDropEffects.Link
 
def dragDrop(sender, evt):
  global pane
  data = evt.Data
  data = data.GetData(DataFormats.FileDrop)
  num = pane.Items.Count
 
  for file in data:
    pane.Items.Add(file)
    for hashOp in hashes :
      anHash = getHash(file, hashOp, nbsp)
      pane.Items[num].SubItems.Add(anHash)
    num += 1
      
    


def decorate(frame):
    global pane
    frame.Text = "IronPython - Drophash"
    ## optional icon
    ## ico = Image.FromFile("drophash.ico")
    ## frame.Icon = Icon.FromHandle(ico.GetHicon());
    pane = ListView()
    pane.Dock = DockStyle.Fill;
    frame.Controls.Add(pane)
    pane.AllowDrop = True
    pane.DragOver += dragOver
    pane.DragEnter += dragOver
    pane.DragDrop += dragDrop
    pane.View = View.Details
    pane.Columns.Add("File name", 100, HorizontalAlignment.Left)
    pane.Columns.Add("MD5", 200, HorizontalAlignment.Left)
    pane.Columns.Add("SHA-1", 200, HorizontalAlignment.Left)


if __name__ == "__main__":
    frame = Form()
    decorate(frame)
    Application.Run(frame)

Jython 2.2

Jython next; but rather than coding the messy .ico to image transformation again, use a PNG icon (save as drophash.png) instead:—

import sys
import os.path
sys.packageManager.makeJavaPackage("javax.swing", "JWindow", None)
sys.packageManager.makeJavaPackage("java.awt", "Window", None)
sys.packageManager.makeJavaPackage("java.awt.dnd", "DropTargetListener", None)
sys.packageManager.makeJavaPackage("java.awt.datatransfer", "DataFlavor", None)
import javax.swing
import java.lang
import java.awt.dnd
import java.awt.datatransfer

from md5 import *
from sha import *

pane = None
hashes = [md5, sha]
nbsp = list(" ")
target = None

def getHash(file, hashOp, delimiter) :

  try :
    reader = open(file, 'rb')
  except :
    return ""
  
  try :
    digest = hashOp()

  while True :
    chunk = reader.read(1024)
    if not chunk :
      break

      digest.update(chunk)
  finally:
    reader.close()
        
  raw = digest.hexdigest()
  work = []
  i = 0
  while i < len(raw) :
    work.append(raw[i])
    i += 1
    if 0 == (i%8) :
      work = work + delimiter

  return ''.join(work)
  
def OnDropFiles(filenames) :
  global pane, nbsp, hashes
  pane.text += "<table border=1><tr><th>File</th><th>MD5</th><th>SHA-1</th></tr>"
  for file in filenames :
    pane.text += "<tr><td>%s</td>" % file

    for hashOp in hashes :
      anHash = getHash(str(file), hashOp, nbsp)
      pane.text += "<td><pre>%s</pre></td>" % anHash

  pane.text +="</tr>"
  pane.text +="</table>"

  

class dropHandler(java.awt.dnd.DropTargetAdapter):
  def __init__(self):
    java.awt.dnd.DropTargetAdapter.__init__(self)

  def drop(self, evt):
    if not evt.transferable.isDataFlavorSupported(java.awt.datatransfer.DataFlavor.javaFileListFlavor):
      return
    evt.acceptDrop(-1)
    names = evt.transferable.getTransferData(java.awt.datatransfer.DataFlavor.javaFileListFlavor)
    OnDropFiles(names)

def decorate(frame):
  global pane
  frame.setDefaultCloseOperation(javax.swing.WindowConstants.EXIT_ON_CLOSE)
  frame.title = "Jython - Drophash"
  frame.layout = java.awt.BorderLayout()

  pane = javax.swing.JLabel()
  pane.border = javax.swing.BorderFactory.createEmptyBorder(5,5,5,5)
  frame.contentPane.add(javax.swing.JScrollPane(pane), java.awt.BorderLayout.CENTER)
  pane.text = "<html>"
  icon = java.awt.Toolkit.getDefaultToolkit().createImage("drophash.png")
  frame.setIconImage(icon)
  target = java.awt.dnd.DropTarget(pane, dropHandler())
  
  
  width = 800
  height = 600
  frame.setSize(width, height);
  screenDim = java.awt.Toolkit.getDefaultToolkit().getScreenSize();
  frame.setLocation(
        (screenDim.width - width) / 2,
        (screenDim.height - height) / 2
        )
  



if __name__ == "__main__":
  frame = javax.swing.JFrame()
  decorate(frame)
  frame.visible = True

Made with Cascading Style Sheets

Saturday 3 March 2007

Drophash in Python

First little Python project, a little utility to take the MD5 and SHA hashes of a bunch of files dragged and dropped onto it. Same function as but better presentation that a Win32 version (on the Windows page) I wrote back in late '00 (and which I have still been making regular use of for checking MD5s of file downloads).

WxPython UI

This one is neater than the Win32 version since I can use the nifty HTML window widget and place the results for each new batch of files in a table, with the files and hashes cross-indexed. Simply cut, paste and save the text of the section below as drophash.py.

#!/usr/bin/env python
# -*- coding: ISO-8859-1 -*-

#MD5 test suite:
#MD5 ("") = d41d8cd98f00b204e9800998ecf8427e
#MD5 ("a") = 0cc175b9c0f1b6a831c399e269772661
#MD5 ("abc") = 900150983cd24fb0d6963f7d28e17f72
#MD5 ("message digest") = f96b697d7cb7938d525a2f31aaf161d0
#MD5 ("abcdefghijklmnopqrstuvwxyz") = c3fcd3d76192e4007dfb496cca67e13b
#MD5 ("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789") = d174ab98d277d9f5a5611c2c9f419d9f
#MD5 ("12345678901234567890123456789012345678901234567890123456789012345678901234567890") = 57edf4a22be3c955ac49da2e2107b67a

#SHA test suite:
#SHA ("abc") = A9993E36 4706816A BA3E2571 7850C26C 9CD0D89D
#SHA ("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq") = 84983E44 1C3BD26E BAAE4AA1 F95129E5 E54670F1
#SHA (A million repetitions of "a") = 34AA973C D4C4DAA4 F61EEB2B DBAD2731 6534016F


import wx
import wx.html as html
from md5 import *
from sha import *

def getHash(file, hashOp, delimiter) :

    try :
        reader = open(file, 'rb')
    except :
        return ""
    
    try :
        digest = hashOp()
        while True :
            chunk = reader.read(1024)
            if not chunk :
                break
            digest.update(chunk)
    finally:
        reader.close()
        
    raw = digest.hexdigest()
    work = []
    i = 0
    while i < len(raw) :
        work.append(raw[i])
        i += 1
        if 0 == (i%8) :
            work = work + delimiter

    return ''.join(work)

class FileDropTarget(wx.FileDropTarget):
    def __init__(self, window):
        wx.FileDropTarget.__init__(self)
        self.window = window

    hashes = (md5, sha)
    nbsp = list(" ")

    def OnDropFiles(self, x, y, filenames) :
        self.window.AppendToPage("<table border=1><tr><th>File</th><th>MD5</th><th>SHA-1</th></tr>")
        for file in filenames :
            self.window.AppendToPage("<tr><td>%s</td>" % file)

            for hashOp in self.hashes :
                anHash = getHash(file, hashOp, self.nbsp)
                self.window.AppendToPage("<td><pre>%s</pre></td>" % anHash)

            self.window.AppendToPage("</tr>")
        self.window.AppendToPage("</table>")

def decorate(frame):
    frame.SetTitle("wxPython - Drophash")
    ## optionally, an icon
    ## _icon = wx.EmptyIcon()
    ## _icon.CopyFromBitmap(wx.Bitmap("drophash.ico", wx.BITMAP_TYPE_ANY))
    ## frame.SetIcon(_icon)
    frame.style = wx.DEFAULT_FRAME_STYLE
    pane = html.HtmlWindow(frame, -1)
    dt = FileDropTarget(pane)
    pane.SetDropTarget(dt)


if __name__ == "__main__":
    theApp = wx.App(False)
    frame  = wx.Frame(None, -1)
    decorate(frame)
    theApp.SetTopWindow(frame)
    frame.Show()
    theApp.MainLoop()

Python + Tk UI

As an alternative, a pure Tk version. Not so fancy; you have to select the files you want (or put them on the command line). Just extract this section as tkhash.py and double click…

#!/usr/bin/env python
# -*- coding: ISO-8859-1 -*-

#MD5 test suite:
#MD5 ("") = d41d8cd98f00b204e9800998ecf8427e
#MD5 ("a") = 0cc175b9c0f1b6a831c399e269772661
#MD5 ("abc") = 900150983cd24fb0d6963f7d28e17f72
#MD5 ("message digest") = f96b697d7cb7938d525a2f31aaf161d0
#MD5 ("abcdefghijklmnopqrstuvwxyz") = c3fcd3d76192e4007dfb496cca67e13b
#MD5 ("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789") = d174ab98d277d9f5a5611c2c9f419d9f
#MD5 ("12345678901234567890123456789012345678901234567890123456789012345678901234567890") = 57edf4a22be3c955ac49da2e2107b67a

#SHA test suite:
#SHA ("abc") = A9993E36 4706816A BA3E2571 7850C26C 9CD0D89D
#SHA ("abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq") = 84983E44 1C3BD26E BAAE4AA1 F95129E5 E54670F1
#SHA (A million repetitions of "a") = 34AA973C D4C4DAA4 F61EEB2B DBAD2731 6534016F

import Tkinter
import ScrolledText
import tkFileDialog
import sys
import tkFont

from md5 import *
from sha import *

hashes = {"MD5" : md5, "SHA" : sha}
nbsp = list(" ")

def getHash(file, hashOp, delimiter) :

    try :
        reader = open(file, 'rb')
    except :
        return ""
    
    try :
        digest = hashOp()
        while True :
            chunk = reader.read(1024)
            if not chunk :
                break
            digest.update(chunk)
    finally:
        reader.close()
        
    raw = digest.hexdigest()
    work = []
    i = 0
    while i < len(raw) :
        work.append(raw[i])
        i += 1
        if 0 == (i%8) :
            work = work + delimiter

    return ''.join(work)


def doGetHashes() :
  filenames = tkFileDialog.askopenfilenames()
  updateCanvas(filenames)
  
def updateCanvas(filenames):
  textarea.config(state=Tkinter.NORMAL)
  for file in filenames :
    textarea.insert(Tkinter.END, ("%s\n" % file))

    for hashOp in hashes :
      anHash = getHash(file, hashes[hashOp], nbsp)
      textarea.insert(Tkinter.END, ("%s " % hashOp) )

      textarea.insert(Tkinter.END, ("%s\n" % anHash), 'mono' )

    textarea.insert(Tkinter.END, "-------------------------\n")
  textarea.tag_config('mono', font='courier')
  textarea.config(state=Tkinter.DISABLED)

if __name__ == "__main__":
  root = Tkinter.Tk()
  root.title("tkFileHasher")

  textarea = ScrolledText.ScrolledText()
  textarea.pack()
  textarea.config(state=Tkinter.DISABLED)
  Tkinter.Button(text="Select files to hash...", command=doGetHashes).pack()
  updateCanvas(sys.argv[1:])
  Tkinter.mainloop()

Wednesday 14 February 2007

Fun with JavaScript

“E”-ddress encoder script 1.0 — a utility based on the the old Hiveware “Enkoder”, only this time done entirely in JavaScript. Because you can.

Also the same thing for Acrete script 1.0

This sort of thing doesn't need heavyweight stuff like a Java applet; though Acrete currently uses the <canvas> element which isn't supported in IE (though I'm investigating alternatives for that). And, besides, I might as well do something useful, rather than just playing, with dynamic languages like JavaScript, Ruby and Python. Java projects (at least, those not for work) have probably reached the end of the road as a result.