F# vs. Haskell vs. Go: or, upgrade Mono!

Inspired by some recent discussion on Stack Overflow (and here), starting with Jon Harrop's code, I have performed some file reading speed tests to compare Haskell, F#, and Go programs running on Ubuntu 10.10.

These tests were, of course, not scientifically rigorous.  I haven't done hardly anything to optimize the code; and the thing being tested, reading from disks and doing a computation on one byte at a time, is not particularly useful for every application. Although my sample size was small enough that Zed Shaw might kill me, these tests do, however, let me tell you several things...

First, you really should use a newer version of Mono with F#.  Version 2.8.2 is an order of magnitude faster than 2.6.7 (the default on Ubuntu 10.10). A comparison of a typical sample on Mono 2.6.7 vs. my average on Mono 2.8.2 (this version) showed an 11x speed boost to the F# program using BufferedStream!



Second, although Jon Harrop reports that with .NET, F# has performance better than Haskell, with the newer Mono I find that F# is not as fast as Haskell (GHC 6.12.1). Google's language, Go, however, happens to be pretty impressive for such a new language! The following shows the average transfer rate in gigabits per second, based on the real time measured for the file sizes shown.

Transfer rates for the lazy Haskell, F#, and Go programs:


Finally, the greedy file reading functions provided by Haskell and F# perform impressively for moderately large files, but cannot be used for big files.  Haskell's greedy tries valiantly but falls apart on the larger files (I need to file a bug report...), while F# simply stops with a message noting that it doesn't like reading files over 2 GB. Nevertheless, note how fast the greedy Haskell and F# programs are on the smaller files!

Transfer rates shown in the first chart, plus those of the eager Haskell and F# programs:


The Haskell bug may be something resolved with the new IO manager in 7.0, which I have not tried to build yet. Nevertheless, what I observed was that the lazy version holds steady, but for every test at ~2.7 GB the eager version failed, and for tests at ~5.4 GB the eager version seemed to explode. Even a web browser with a text page I was reading became unusable while it ran. The following graphs show time spent per gigabyte, and compare real, user, and system time as reported by the Bash `time` command.










(Both the Haskell and F# routines are from Jon Harrop. My source code and rough little test script are here and shown below: errors are mine alone.)

F#

The "on-demand" F# routine recursively reads and XORs each byte from a buffered stream of the file. The "eager" routine simply reads the whole memory and folds and XOR through the bytes.

On-demand F# reading from a BufferedStream:

open System.IO

let filename = System.Environment.GetCommandLineArgs().[1]

let turtle f : int =
    use stream = new FileStream(f, FileMode.Open)
    use stream' = new BufferedStream(stream)
    
    let rec loop n : int =
        let b = stream'.ReadByte()
        if b >= 0 then loop (n ^^^ b) else n
    loop 0 

printfn "F# XOR of bytes (on demand): %d" <| turtle filename

Eager F# calling ReadAllBytes:

open System.IO

let filename = System.Environment.GetCommandLineArgs().[1]

let hare f : byte = 
    f
    |> File.ReadAllBytes
    |> Array.fold (^^^) 0uy

printfn "F# XOR of bytes (eager): %d" <| hare filename

Haskell
Both Haskell routines use `readFile` from ByteString -- one lazy, one not -- on the first argument to the program, and then fold XOR through the bytes.

Haskell, using Data.ByteString, not lazy:

import System
import Bits
import Data.List
import Data.ByteString as B

main =
    Prelude.putStr "Haskell XOR of bytes (eager): "
    >> getArgs
    >>= B.readFile . Data.List.head
    >>= print . B.foldl xor 0

Lazy Haskell, almost identical:

import System
import Bits
import Data.List
import Data.ByteString.Lazy as B

main =
    Prelude.putStr "Haskell XOR of bytes (on demand): "
    >> getArgs
    >>= B.readFile . Data.List.head
    >>= print . B.foldl xor 0

Go
The Go routine keeps a running total of the XOR of the bytes as it reads them from a buffered reader. The bufio library uses a default buffer 4096 bytes long.

Go, with bufio {Curly braces represent!}:

package main

import (
   "fmt"
   "os"
   "bufio"
)

var me string = "xor"

func perr (err os.Error) {
   fmt.Println(me, ":", err)
}

func main() {
   if len(os.Args) < 2 {
      fmt.Println("Usage: xor FILENAME")
      return
   }
   filename := os.Args[1]

   var result byte = 0

   var content byte
   var err os.Error

   file, err := os.Open(filename, os.O_RDONLY, 0666)
   if err != nil {
      perr(err)
      return
   }

   br := bufio.NewReader(file)
   //br := bufio.NewReader(os.Stdin)

   for {
      content, err = br.ReadByte()
      if err != nil {
         perr(err)
         break
      }
      result ^= content
   }

   fmt.Printf("Go XOR of \"%s\": %d\n", filename, result)
}

As a bonus, I can even give you a couple numbers for the compilers used.  F#'s numbers rely on the --resident flag, with the compiler already idle and ready.  Both Haskell and F# were optimized code (-O2 and --optimize), but Go's 6g provides no such optimization flag.



Though other languages or better computation tests are tempting, if I spend more time on this it will be memory profiling. Memory usage, at least, is not limited by the hard disk...

Tell me what you think!