Matrixes and a Ruby Method of Least Squares Implementation

I’ve been working on a little statistics side project for a while now. I’ve built a simple rails app which tracks the value of several pension funds, based on data it pulls from different websites.

Anyway, the app stores daily values for fund value. Here is a simple example of the data it’s storing:

1
2
3
4
5
6
# [n, value (pence)]
[0, 120]
[1, 123]
[2, 128]
[3, 125]
# ... and so on

I thought it would be pretty cool to generate some statistics for performance over the previous week. To do this, we can look at standard statistics such as standard deviation, sample variance, and advanced statistics like trends.

Variance and Standard Deviation

Variance is handy because it tells you how how disparate your data is from the standard deviation. It creates a statistical measure so you can work out if a pension fund is fairly stable, or tends to fluctuate wildly.

A higher variance means the data is more variable.

Variance method Ruthlessly stolen from Stack Overflow
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# See  http://stackoverflow.com/a/7749613
def values
  [23, 24, 25, 26]
end

def sum
  values.inject(0){ |a,i| a+i }
end

def sample_variance
  m = sum / values.length.to_f
  sum = values.inject(0){ |a,i| a + (i-m)**2 }
  sum / ( values.length - 1 ).to_f
end

Calculating standard deviation is just a case of square rooting the sample variance:

Standard Deviation
1
2
3
def standard_deviation
  Math.sqrt(sample_variance)
end

Trends (Least Squares method)

Wouldn’t it be handy to know whether the fund is generally trending upwards, or downwards? For this, we need some more advanced measures. One popular method is the Method of Least Squares.

Broadly, this formula tries to produce a trend line by calculating the best line which has the least gap between the trend line and the points. I won’t delve in the details for the formula here, rather present my approach to accomplishing this in Ruby.

I’m not a computer scientist; I wasn’t familiar with the concept of a Matrix. They can be used to solve a simultaneous equation. I followed the (very simple!) math on this page, which yields these two equations:

This can be represented as:

To get a and b, you can provide the first matrix and the desired solution matrix:

Matrix methods View full source on GitHub
1
2
3
4
require 'matrix'
input = Matrix[ [n, sig_x], [sig_x, sig_x_squared] ]
result = input.lup.solve [sig_y, sig_xy]
# => Vector([a, b])

And here is how we derive these values, based on the input data above:

Least Squares methods View full source on GitHub
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
def n
  values.length
end

def sig_y
  values.inject(0){ |a,i| a+i }
end

def sig_x
  (1..n).inject(:+)
end

def sig_xy
  # I borrowed inject_with_index from here: 
  # http://archived.rpheath.com/posts/341-ruby-inject-with-index
  values.inject_with_index(0) do |count, y, x|
      count += ( x + 1 ) * y
  end
end

def sig_x_squared
  (1..n).map{ |n| n**2 }.inject :+
end

Working example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
class Fund
  include Statistics

  def values
      [5, 6, 8, 7]
  end
end

f = Fund.new

f.values
# => [5, 6, 8, 7]
f.sig_x
# i.e. 1 + 2 + 3 + 4
# => 10
f.sig_y
# => 26
f.sig_xy
# i.e. 1*5 + 2*6 + 3*8 + 4*7
# => 69
f.sig_x_squared
# i.e. 1^2 + 2^2 + 3^2 + 4^2
# => 30
f.least_squares
# => [4.5, 0.8]
# i.e. y = 4.5 + 0.8x
# :. 0.8 is our trend

View the source code on GitHub

IE6, Filters and Quirky Behaviour With Input + Iframe Fields

We stumbled across a problem at work recently with IE6, css filters and input + iframe elements.

Here was our layout:

1
2
3
4
<div style="filter:progid:DXImageTransform.Microsoft.Gradient(...)">
  .....
  <iframe src="file.html"></iframe>
</div>

We found that the iframe was causing some screen tearing and really bizarre scroll patterns. If you remove the filter, the iframe works properly (as well as it can in IE6 I guess).

Related to this, we also discovered the following:

1
2
3
<div style="filter:alpha(opacity=x)">
  <input type="text" .. />
</div>

If you put some values in the input text which scrolled beyond the length of the field, there were some weird issues with scrolling left/right using the keyboard.

In both cases, we removed the filter and found noticable improvements across the board. We’ve rethought how to implement this kind of behaviour while not breaking the special objects on top.

Ubuntu on a Samsung NP700Z5A

So I’ve had one of these NP700Z5A [samsung.com] for about two months now. It’s a snappy beast, but I’ve been using linux for so many years now I just can’t get on with Windows 7.

It was tough choice of distro, but in the end I settled on Ubuntu 11.10 Oneiric for compatibility (more users = more patches for different devices). I thought I should document my experiences for others with similar machines.

This machine has two significant problems as far as I can see.

Touchpad (kernel 3.0 and below)

The first problem is the touchpad. It’s made by a third party called Elantech, who provide a lot of the glossy mac-type touchpads used by products such as the Asus UX31. The particular model touchpad was not immediately recognised by the synaptec driver, so the touchpad has no multi-touch or scrolling support out of the box. This has been confirmed to be an issue in the 3.0.0.14 version of the kernel, and there is a bug report on the problem.

There is a partial fix available, if you wish to upgrade to kernel 3.2, which recognises the touchpad ( = scrolling, multi-touch, disable touchpad while typing etc) but unfortunately breaks the right-click button (clicking either-side performs a left-click). To do a right-click, you need to tap the touchpad with two fingers, which is acceptable but not ideal.

(I used this tutorial to upgrade to 3.2).

I’ve upgraded to 3.2, and had to run this command to get the touchpad to detect your finger touches (improve the sensitivity and increase the scroll margin)

1
$ synclient FingerLow=3 FingerHigh=8 RightEdge=2950

(I messed with the new style xorg.conf.d’s at my peril and broke all sorts of things, so now I run this command as a startup script with “Startup Applications”)

Power consumption

The other significant problem is the power consumption. This machine has a dual graphics configuration of ATI + Intel graphics. Additionally, it has 8 cores. The idling power consumption, with no tweaks, was clocking at over 40W(!) on my machine, which translates to a little over two hours of careful usage.

With the addition of powertop and laptop-mode-tools, I managed to bring power usage down to under 30W, but I think there are still more things I can do to improve this further.

I also enabled the intel sandy bridge RC6 power savings support (see article on phoronix.com), by adding

i915.i915_enable_rc6=1

to my grub arguments. This did seem to have a positive effect on power consumption.

Update After some helpful tips from RĂ©gis, I’ve also enabled these kernel parameters:

i915.i915_enable_fbc=1 i915.lvds_downclock=1 pcie_aspm=force

More information about their efficacy can be found at http://www.phoronix.com/scan.php?page=article&item=intel_i915_power. I’ve noticed that my power consumption average has dropped to around 25W - it erratically flicks between 22w and 38w though!

Other issues

There are a couple of other minor problems with the laptop:

  • Keyboard backlight - doesn’t seem to be any way of adjusting this, the buttons on the keyboard don’t impact the brightness.
  • Wifi problems after resume - very occasionally, the wifi is disabled after resuming. I don’t think this is laptop specific as I’ve noticed this elsewhere…I’m using the stock opensource broadcom driver which is included in the kernel, rather than the proprietary one, which seemed to be more stable.

Further reading

Stefano has a great article which goes into greater detail on these issues!

I’ll try to update this page as I discover ways to improve these problems. You can get in touch at nickcampbell18@gmail.com or leave a comment below.