<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.9.5">Jekyll</generator><link href="https://compton.nu/feed.xml" rel="self" type="application/atom+xml" /><link href="https://compton.nu/" rel="alternate" type="text/html" /><updated>2024-02-17T14:54:52+00:00</updated><id>https://compton.nu/feed.xml</id><title type="html">Tom’s Thoughts</title><subtitle>Write an awesome description for your new site here. You can edit this line in _config.yml. It will appear in your document head meta (for Google search results) and in your feed.xml site description.
</subtitle><entry><title type="html">Obfuscated Passwords…</title><link href="https://compton.nu/2017/09/obfuscated-passwords" rel="alternate" type="text/html" title="Obfuscated Passwords…" /><published>2017-09-19T19:08:17+01:00</published><updated>2017-09-19T19:08:17+01:00</updated><id>https://compton.nu/2017/09/obfuscated-passwords</id><content type="html" xml:base="https://compton.nu/2017/09/obfuscated-passwords"><![CDATA[<p>A cron job that I run every day to screen scrape some figures from
a financial services provider recently broke and upon investigation
it transpired that it was no longer able to login to my account. On
it’s own that isn’t particularly unusual but it was when I started
investigating what had changed and needed to be fixed in my script
that things got a bit weird…</p>

<p>The login process involves submitting a form consisting of a username,
a PIN and a password over a secure https connection. The PIN and password
are sent in plain text, but that is fine as the connection is secure.</p>

<p>An initial investigation of what was now happening when the login form
was submitted showed that the PIN and password appeared to have been
obfuscated in some way, so I grabbed what appeared to be the relevant
javascript and ran it through a beautifier, revealing this:</p>

<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">$</span><span class="p">(</span><span class="dl">"</span><span class="s2">#obfuscatedPin</span><span class="dl">"</span><span class="p">).</span><span class="nx">val</span><span class="p">(</span><span class="nx">CryptoJS</span><span class="p">.</span><span class="nx">MD5</span><span class="p">(</span><span class="nx">Value</span><span class="p">(</span><span class="dl">"</span><span class="s2">originalPin</span><span class="dl">"</span><span class="p">)).</span><span class="nx">toString</span><span class="p">(</span><span class="nx">CryptoJS</span><span class="p">.</span><span class="nx">enc</span><span class="p">.</span><span class="nx">Base64</span><span class="p">).</span><span class="nx">toUpperCase</span><span class="p">());</span>
<span class="nx">$</span><span class="p">(</span><span class="dl">"</span><span class="s2">#obfuscatedPwd</span><span class="dl">"</span><span class="p">).</span><span class="nx">val</span><span class="p">(</span><span class="nx">CryptoJS</span><span class="p">.</span><span class="nx">MD5</span><span class="p">(</span><span class="nx">Value</span><span class="p">(</span><span class="dl">"</span><span class="s2">originalPwd</span><span class="dl">"</span><span class="p">).</span><span class="nx">toUpperCase</span><span class="p">()).</span><span class="nx">toString</span><span class="p">(</span><span class="nx">CryptoJS</span><span class="p">.</span><span class="nx">enc</span><span class="p">.</span><span class="nx">Base64</span><span class="p">));</span>
</code></pre></div></div>

<!-- more -->

<p>So it seems they are just being run through MD5 on the client before
being sent to the server which seems a little odd given the connection
is secure. This does however give rise to a whole series of questions:</p>

<ol>
  <li>
    <p>Why hash passwords on the client? They are secure in transit so
the only threat this seems to protect against is somebody at the
server end reading them.</p>
  </li>
  <li>
    <p>Given that this is new code why on earth is it using MD5 which is
widely considered to be at best unsuitable for new applications.</p>
  </li>
  <li>
    <p>Why reduce entropy unnecessarily by uppercasing the password before
hashing it?</p>
  </li>
</ol>

<p>The answer to the those last two questions probably lies in what this
whole scheme implies about how both the PIN and password are being
stored on the server, namely that this code is mirroring how they have
historically been stored, which is to say as unsalted, unstretched MD5
hashes, in violation of pretty much every modern password security guideline.</p>

<p>We pretty much know that must be the case, as the client side hashing
means they are not able recover the original password in order to hash
it more securely, so unless they are rehashing the MD5 hash with a salt
which would be very odd they must just be comparing what the client
sends directly with their database.</p>

<p>So the end result is that what was presumably intended as a security
upgrade (hashing passwords on the client) has wound up revealing just
how bad the backend security is without actually improving anything!</p>

<h2 id="postscript">Postscript</h2>

<p>One odd thing I noticed is that although the code shown above
appears to be base64 encoding the hash what I was seeing on the
wire appeared to be a hex string rather than a base64 string.</p>

<p>A little investigation revealed that although they were using a
standard <a href="https://www.npmjs.com/package/crypto-js">javascript cryptography library</a>
it had apparently been run through a javascript minifier that
had wrongly decided that <code class="language-plaintext highlighter-rouge">CryptoJS.enc.Base64</code> was unused and
removed it meaning that it evaluated to <code class="language-plaintext highlighter-rouge">undefined</code> and
caused <code class="language-plaintext highlighter-rouge">toString</code> to default to hex encoding instead.</p>

<p>It then turned out that the server didn’t actually accept base64
encoding so obviously it was either changed to accept what the
client was actually sending or the attempt to do base64 encoding
was never right in the first place!</p>]]></content><author><name></name></author><category term="fail" /><summary type="html"><![CDATA[A cron job that I run every day to screen scrape some figures from a financial services provider recently broke and upon investigation it transpired that it was no longer able to login to my account. On it’s own that isn’t particularly unusual but it was when I started investigating what had changed and needed to be fixed in my script that things got a bit weird… The login process involves submitting a form consisting of a username, a PIN and a password over a secure https connection. The PIN and password are sent in plain text, but that is fine as the connection is secure. An initial investigation of what was now happening when the login form was submitted showed that the PIN and password appeared to have been obfuscated in some way, so I grabbed what appeared to be the relevant javascript and ran it through a beautifier, revealing this: $("#obfuscatedPin").val(CryptoJS.MD5(Value("originalPin")).toString(CryptoJS.enc.Base64).toUpperCase()); $("#obfuscatedPwd").val(CryptoJS.MD5(Value("originalPwd").toUpperCase()).toString(CryptoJS.enc.Base64));]]></summary></entry><entry><title type="html">British Gas? or Central Recoveries?</title><link href="https://compton.nu/2014/06/british-gas-or-central-recoveries" rel="alternate" type="text/html" title="British Gas? or Central Recoveries?" /><published>2014-06-27T14:26:46+01:00</published><updated>2014-06-27T14:26:46+01:00</updated><id>https://compton.nu/2014/06/british-gas-or-central-recoveries</id><content type="html" xml:base="https://compton.nu/2014/06/british-gas-or-central-recoveries"><![CDATA[<p>In light of the recent stories about first
<a href="http://www.buzzfeed.com/jimwaterson/wonga-created-fake-law-firms-to-frighten-customers-into-repa">Wonga</a>,
and now the
<a href="http://www.buzzfeed.com/jimwaterson/how-the-student-loans-company-pretends-to-be-an-independent">Student Loans Company</a>,
pretending to be other people when collecting debts I thought I would point our
that <a href="http://www.britishgas.co.uk/">British Gas</a> certainly used to do much the
same thing.</p>

<p>Back in 2007 I was in dispute with them, because I had moved supplier but they
had decided to ignore my final meter reading and try and bill me for gas that I
should have been paying my new supplier for and in the course of that dispute I
received this letter from “Central Recoveries” saying that my account had been
“passed to them”:</p>

<figure><a data-fancybox="gallery" href="/assets/images/british-gas.png"><img src="/assets/images/british-gas.png" /></a><figcaption>Central Recoveries Letter</figcaption></figure>

<p>If you look closely at the small print ringed in red you will see that they are
in fact a “Centrica business” and appear to be just another name for British Gas
Trading Limited.</p>

<p>As much as I would have loved to see them in court and ask them to explain why I
should pay them for gas that somebody else had supplied me, my new supplier
decided that they would credit me what British Gas was wrongly demanding so I
unfortunately had to let the whole thing drop.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[In light of the recent stories about first Wonga, and now the Student Loans Company, pretending to be other people when collecting debts I thought I would point our that British Gas certainly used to do much the same thing. Back in 2007 I was in dispute with them, because I had moved supplier but they had decided to ignore my final meter reading and try and bill me for gas that I should have been paying my new supplier for and in the course of that dispute I received this letter from “Central Recoveries” saying that my account had been “passed to them”: Central Recoveries Letter If you look closely at the small print ringed in red you will see that they are in fact a “Centrica business” and appear to be just another name for British Gas Trading Limited. As much as I would have loved to see them in court and ask them to explain why I should pay them for gas that somebody else had supplied me, my new supplier decided that they would credit me what British Gas was wrongly demanding so I unfortunately had to let the whole thing drop.]]></summary></entry><entry><title type="html">Economics, Amazon Style</title><link href="https://compton.nu/2014/02/economics-amazon-style" rel="alternate" type="text/html" title="Economics, Amazon Style" /><published>2014-02-06T14:06:23+00:00</published><updated>2014-02-06T14:06:23+00:00</updated><id>https://compton.nu/2014/02/economics-amazon-style</id><content type="html" xml:base="https://compton.nu/2014/02/economics-amazon-style"><![CDATA[<p>I’m sure there’s some logic somewhere in the offer that Amazon just presented to
me, it’s just not entirely clear to me what sort of logic…</p>

<figure><a data-fancybox="gallery" href="/assets/images/amazon-economics.png"><img src="/assets/images/amazon-economics.png" /></a></figure>

<p>All I need now then is an option to have them hold the CD until I have something
else I want that will put me over the free delivery limit!</p>]]></content><author><name></name></author><summary type="html"><![CDATA[I’m sure there’s some logic somewhere in the offer that Amazon just presented to me, it’s just not entirely clear to me what sort of logic… All I need now then is an option to have them hold the CD until I have something else I want that will put me over the free delivery limit!]]></summary></entry><entry><title type="html">Are Movie Studios now Restricting when Films can be Reserved?</title><link href="https://compton.nu/2012/09/are-movie-studios-now-restricting-when-films-can-be-reserved" rel="alternate" type="text/html" title="Are Movie Studios now Restricting when Films can be Reserved?" /><published>2012-09-05T19:58:03+01:00</published><updated>2012-09-05T19:58:03+01:00</updated><id>https://compton.nu/2012/09/are-movie-studios-now-restricting-when-films-can-be-reserved</id><content type="html" xml:base="https://compton.nu/2012/09/are-movie-studios-now-restricting-when-films-can-be-reserved"><![CDATA[<p>It has been well documented in the US that one thing movie studios have been
demanding in new rental contracts is a time delay between films being released
for sale and being made available for renting, based on the frankly bizarre
theory that if they stop us being able to rent a film for a few months we will
all go and buy a copy instead, thus making them more profit.</p>

<p>It was always my suspicion that such a demand was behind the long impasse that
prevented Universal films being available on
<a href="http://www.lovefilm.com/">LOVEFiLM</a> for the last couple of years, although I
have no actual evidence to support this.</p>

<p>That impasse was recently resolved, and at much the same time some films started
appearing on LOVEFiLM with a statement at the top that reads:</p>

<blockquote>
  <p>The studio have licensed us to make this title available to rent on the
release date below.</p>
</blockquote>

<p>When that message appears the rental release date is normally shown as roughly
two months after the sale release date - in other words just the sort of delay
the studios have been demanding.</p>

<!-- more -->

<p>That doesn’t bother me too much though, because you can normally reserve a film
as soon as it is listed and it will be sent to you once it is released for
rental, but it has been noticeable that at least some films displaying that
message cannot be reserved.</p>

<p>I was interested therefore when I noticed the
<a href="https://twitter.com/LOVEFiLM">@LOVEFiLM</a> twitter account replying to a query
about why such a film couldn’t be reserved as follows:</p>

<div class="jekyll-twitter-plugin"><blockquote class="twitter-tweet" align="center"><p lang="en" dir="ltr"><a href="https://twitter.com/sunshine_sarah6">@sunshine_sarah6</a> The studio have licensed us to rent this title from the 08 Feb 2013. You&#39;ll be able to add this to your list on this date.</p>&mdash; Amazon Video UK (@AmazonVideoUK) <a href="https://twitter.com/AmazonVideoUK/status/242968793243070464">September 4, 2012</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></div>

<p>so I replied, asking why the fact that it wasn’t released yet should affect the
ability to reserve it:</p>

<div class="jekyll-twitter-plugin"><blockquote class="twitter-tweet" align="center"><p lang="en" dir="ltr"><a href="https://twitter.com/lovefilm">@LOVEFiLM</a> <a href="https://twitter.com/sunshine_sarah6">@sunshine_sarah6</a> So what happened to allowing us to reserve before things are released?</p>&mdash; Tom Hughes (@thughes) <a href="https://twitter.com/thughes/status/242995410703114241">September 4, 2012</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></div>

<p>in return I got this rather surprising response:</p>

<div class="jekyll-twitter-plugin"><blockquote class="twitter-tweet" align="center"><p lang="en" dir="ltr"><a href="https://twitter.com/thughes">@thughes</a> This will be down to the license we have with the studios, sorry for any disappointment this may cause.</p>&mdash; Amazon Video UK (@AmazonVideoUK) <a href="https://twitter.com/AmazonVideoUK/status/243322998789111809">September 5, 2012</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></div>

<p>So it seems that the studios now want to control not only when we can rent a
film, but when we should be allowed to add it to our queue of things we would
like to see… Presumably they they think that the more annoying they can make
the process of renting a film, the more likely we are to buy it instead.</p>

<p>I guess what we need now is an external site that runs a pre-reservation list
and tracks what we would like to be able to reserve but can’t, so that it can
notify us when they become available for reservation in the normal way…</p>]]></content><author><name></name></author><category term="fail" /><summary type="html"><![CDATA[It has been well documented in the US that one thing movie studios have been demanding in new rental contracts is a time delay between films being released for sale and being made available for renting, based on the frankly bizarre theory that if they stop us being able to rent a film for a few months we will all go and buy a copy instead, thus making them more profit. It was always my suspicion that such a demand was behind the long impasse that prevented Universal films being available on LOVEFiLM for the last couple of years, although I have no actual evidence to support this. That impasse was recently resolved, and at much the same time some films started appearing on LOVEFiLM with a statement at the top that reads: The studio have licensed us to make this title available to rent on the release date below. When that message appears the rental release date is normally shown as roughly two months after the sale release date - in other words just the sort of delay the studios have been demanding.]]></summary></entry><entry><title type="html">LOVEFiLM and SMS Spam</title><link href="https://compton.nu/2012/01/lovefilm-and-sms-spam" rel="alternate" type="text/html" title="LOVEFiLM and SMS Spam" /><published>2012-01-01T17:03:31+00:00</published><updated>2012-01-01T17:03:31+00:00</updated><id>https://compton.nu/2012/01/lovefilm-and-sms-spam</id><content type="html" xml:base="https://compton.nu/2012/01/lovefilm-and-sms-spam"><![CDATA[<p>One thing that consistently annoys me is when seemingly reputable companies
decide for some reason that the rules on unsolicited marketing communications,
that is to say, in the United Kingdom, the snappily titled
“<a href="http://www.legislation.gov.uk/uksi/2003/2426/made">Privacy and Electronic Communications (EC Directive) Regulations 2003</a>”,
don’t apply to them for some reason.</p>

<p>The latest organisation to suddenly conclude it has the right to send me such
unwanted communications is <a href="http://www.lovefilm.com/">LOVEFiLM</a> which has
recently decided it should send me regular SMS messages full of some banal
nonsense.</p>

<p>The first such message I received was two weeks ago, on 18th December 2011, and
when I then checked my account settings on LOVEFiLM I was surprised to find that
all the various “LOVEFiLM Marketing” preferences where unchecked apart from one
labelled “by SMS” which I am quite sure I would never have checked, and
certainly not while I was refusing much less annoying things like email
marketing.</p>

<!-- more -->

<p>I immediately unchecked the box and then expressed my annoyance on twitter:</p>

<div class="jekyll-twitter-plugin"><blockquote class="twitter-tweet" align="center"><p lang="en" dir="ltr">Wondering if &quot;please SMS spam me&quot; box on <a href="https://twitter.com/lovefilm">@LOVEFiLM</a> preferences is new as mine is set and I wouldn&#39;t have done that voluntarily...</p>&mdash; Tom Hughes (@thughes) <a href="https://twitter.com/thughes/status/148434160636268544">December 18, 2011</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></div>

<p>Needless to say my tweet was not one that the
<a href="https://twitter.com/LOVEFiLM">@LOVEFiLM</a> account chose to respond to…</p>

<p>I assumed however that, having unchecked the box allowing SMS marketing, that
would an end to it. It appears that I couldn’t have been more wrong however as
today, two weeks later, another spam SMS was received from them. So twitter has
one again been deployed:</p>

<div class="jekyll-twitter-plugin"><blockquote class="twitter-tweet" align="center"><p lang="en" dir="ltr">So <a href="https://twitter.com/lovefilm">@LOVEFiLM</a> can you explain why you are still sending me SMS spam two weeks after I turned off the SMS marketing flag on my account?</p>&mdash; Tom Hughes (@thughes) <a href="https://twitter.com/thughes/status/153507024645259265">January 1, 2012</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script></div>

<p>In addition to which I have contacted them directly via their web site to let
them know what I think:</p>

<blockquote>
  <p>Two weeks ago, on Sunday 18th December, I was somewhat surprised to received a
marketing text message from you. I was surprised because I had never knowingly
agreed to receive such messages and I would normally refuse any such
invitation as a matter of routine.</p>

  <p>I was even more surprised when I checked my account details on your website
and found that all the marketing permission boxes were unchecked except the
SMS one. I don’t believe I would ever have chosen those settings, so I wonder
if you added that option and defaulted it to on without requesting my
permission?</p>

  <p>In any case I unchecked that box on Sunday 18th December.</p>

  <p>Perhaps you can therefore explain why, two weeks later, I have just received
another unwanted spam SMS from you in defiance of my clearly stated
preferences, and hence illegally, being in breach of the Privacy and
Electronic Communications (EC Directive) Regulations 2003.</p>

  <p>PLEASE CEASE AND DESIST WITH THIS BEHAVIOUR IMMEDIATELY.</p>

  <p>For the avoidance of all doubt please take this message as notice pursuant to
regulation 22 of the Privacy and Electronic Communications (EC Directive)
Regulations 2003 that you are not to send me unsolicited SMS messages again.</p>

  <p>Any repeat will be reported to the Information Commissioner’s office for
possible enforcement action with no further notice to yourselves.</p>
</blockquote>

<p>Now I will sit back and see if I have managed to get their attention this time
and if they are willing to learn that this sort of behaviour is completely
unacceptable.</p>]]></content><author><name></name></author><category term="fail" /><summary type="html"><![CDATA[One thing that consistently annoys me is when seemingly reputable companies decide for some reason that the rules on unsolicited marketing communications, that is to say, in the United Kingdom, the snappily titled “Privacy and Electronic Communications (EC Directive) Regulations 2003”, don’t apply to them for some reason. The latest organisation to suddenly conclude it has the right to send me such unwanted communications is LOVEFiLM which has recently decided it should send me regular SMS messages full of some banal nonsense. The first such message I received was two weeks ago, on 18th December 2011, and when I then checked my account settings on LOVEFiLM I was surprised to find that all the various “LOVEFiLM Marketing” preferences where unchecked apart from one labelled “by SMS” which I am quite sure I would never have checked, and certainly not while I was refusing much less annoying things like email marketing.]]></summary></entry><entry><title type="html">Yes, I Would Like a Refund</title><link href="https://compton.nu/2011/08/yes-i-would-like-a-refund" rel="alternate" type="text/html" title="Yes, I Would Like a Refund" /><published>2011-08-09T21:55:59+01:00</published><updated>2011-08-09T21:55:59+01:00</updated><id>https://compton.nu/2011/08/yes-i-would-like-a-refund</id><content type="html" xml:base="https://compton.nu/2011/08/yes-i-would-like-a-refund"><![CDATA[<p>So I recently cancelled one of my credit cards - it was an
<a href="http://www.americanexpress.com/">American Express</a> card with an annual fee that
I had only ever intended to hold for one year to get the signup bonuses, and
once the fee for the second year came due I cancelled it.</p>

<p>Today I got the final statement which, because of the refund of the fee for the
second year, showed a credit balance. Rather than enclosing a cheque for the
balance though, or indicating that they had repaid it to the account that I had
been using to pay the card bills, the statement had this curious message:</p>

<figure><a data-fancybox="gallery" href="/assets/images/refund.png"><img src="/assets/images/refund.png" /></a></figure>

<p>What I want to know is, how many people exactly don’t want a refund, and would
instead prefer to hand their credit balance over to American Express?!?</p>

<p>Of course their secure messaging system insists on me selecting a card before I
can send a message, and won’t let me select a cancelled card, so sending them a
message to ask for my refund turned into a bit of a palaver as well…</p>]]></content><author><name></name></author><category term="fail" /><summary type="html"><![CDATA[So I recently cancelled one of my credit cards - it was an American Express card with an annual fee that I had only ever intended to hold for one year to get the signup bonuses, and once the fee for the second year came due I cancelled it. Today I got the final statement which, because of the refund of the fee for the second year, showed a credit balance. Rather than enclosing a cheque for the balance though, or indicating that they had repaid it to the account that I had been using to pay the card bills, the statement had this curious message: What I want to know is, how many people exactly don’t want a refund, and would instead prefer to hand their credit balance over to American Express?!? Of course their secure messaging system insists on me selecting a card before I can send a message, and won’t let me select a cancelled card, so sending them a message to ask for my refund turned into a bit of a palaver as well…]]></summary></entry><entry><title type="html">Important Information</title><link href="https://compton.nu/2011/07/important-information" rel="alternate" type="text/html" title="Important Information" /><published>2011-07-25T17:51:21+01:00</published><updated>2011-07-25T17:51:21+01:00</updated><id>https://compton.nu/2011/07/important-information</id><content type="html" xml:base="https://compton.nu/2011/07/important-information"><![CDATA[<p>Apparently my bank had some important information to share with me. I know this
because they put an extra page in with my statement headed “Important
Information for you” which I reproduce here:</p>

<figure><a data-fancybox="gallery" href="/assets/images/lloyds.png"><img src="/assets/images/lloyds.png" /></a><figcaption>Lloyds TSB had some Important Information to share with me...</figcaption></figure>

<p>I wonder how many million of those they’ve just sent out…</p>]]></content><author><name></name></author><category term="fail" /><summary type="html"><![CDATA[Apparently my bank had some important information to share with me. I know this because they put an extra page in with my statement headed “Important Information for you” which I reproduce here: Lloyds TSB had some Important Information to share with me... I wonder how many million of those they’ve just sent out…]]></summary></entry><entry><title type="html">Rails Warning Fail</title><link href="https://compton.nu/2011/03/rails-warning-fail" rel="alternate" type="text/html" title="Rails Warning Fail" /><published>2011-03-15T19:09:04+00:00</published><updated>2011-03-15T19:09:04+00:00</updated><id>https://compton.nu/2011/03/rails-warning-fail</id><content type="html" xml:base="https://compton.nu/2011/03/rails-warning-fail"><![CDATA[<p>Recent versions of rails 3 have started spitting out a new deprecation warning:</p>

<blockquote>
  <p>String-based interpolation of association conditions is deprecated. Please use
a proc instead. So, for example, has_many :older_friends, :conditions =&gt; ‘age</p>
  <blockquote>
    <p>#{age}’ should be changed to has_many :older_friends, :conditions =&gt; proc {
“age &gt; #{age}” }.</p>
  </blockquote>
</blockquote>

<p>Now call me confused if you like, but isn’t the suggested replacement still
doing just as much string interpolation as the original?</p>]]></content><author><name></name></author><category term="fail" /><category term="rails" /><summary type="html"><![CDATA[Recent versions of rails 3 have started spitting out a new deprecation warning: String-based interpolation of association conditions is deprecated. Please use a proc instead. So, for example, has_many :older_friends, :conditions =&gt; ‘age #{age}’ should be changed to has_many :older_friends, :conditions =&gt; proc { “age &gt; #{age}” }. Now call me confused if you like, but isn’t the suggested replacement still doing just as much string interpolation as the original?]]></summary></entry><entry><title type="html">City Labels in OpenStreetMap</title><link href="https://compton.nu/2010/10/city-labels-in-openstreetmap" rel="alternate" type="text/html" title="City Labels in OpenStreetMap" /><published>2010-10-19T21:45:29+01:00</published><updated>2010-10-19T21:45:29+01:00</updated><id>https://compton.nu/2010/10/city-labels-in-openstreetmap</id><content type="html" xml:base="https://compton.nu/2010/10/city-labels-in-openstreetmap"><![CDATA[<p>In the
<a href="http://www.41latitude.com/post/1349685626/openstreetmap-critique-2">second part of his critique</a>
of <a href="http://www.openstreetmap.org/">OpenStreetMap</a> Justin O’Beirne discusses
various issues surrounding labelling of cities in OpenStreetMap’s cartography,
specifically in our default mapnik rendering of the US.</p>

<p>The issues he highlights can be broadly divided into two categories: problems
with our stylesheets and rendering technology; and problems with our data, and
in particular with our US data.</p>

<p>The issue which I intend to address here is the one he tackles first – that of
label density which is something that stems largely from data quality and, more
importantly, consistency issues. Specifically, although the post talks about
cities, the real question is about what is tagged as a city and what is tagged
as some lesser type of place.</p>

<!-- more -->

<p>By way of explanation I should probably start by explaining that in
OpenStreetMap tagging there are four commonly used used values for the place tag
which designate a populated place. In order, from largest to smallest, those
are: city, town, village and hamlet. The question which then arises is, how do
we decide which of those values to use for a given settlement?</p>

<p>Like so many tags the specific names used come, because of OpenStreetMap’s
origins, from typical British usage. It is therefore generally not a good idea
to interpret the names too literally in other jurisdictions – indeed some tag
values like highway=trunk aren’t even interpreted literally in England!</p>

<p>To the British the question of which places should be cities is fairly clear –
there are a few alternative definitions (places with royal charters vs places
with cathedrals) but those only relate to a few edge cases and in general there
is little debate and only a relatively small number of large and/or important
towns will qualify.</p>

<p>At the other end of the spectrum a hamlet would normally only be used for very
small places that amount to little more than a handful of houses.</p>

<p>In between lies the distinction between villages and towns which is much less
well defined but in my opinion would generally lie around the few thousand mark
in population terms – once you reach 2-3 thousand residents you are probably a
town rather than a village.</p>

<p>Interestingly the
<a href="http://wiki.openstreetmap.org/wiki/Key:place">OpenStreetMap wiki</a> disagrees a
little here and suggests hamlet for populations up to one thousand and village
up to ten thousand. I would argue that both of those values are too high for
normal British usage and certainly larger than I would use when tagging places.</p>

<p>All of which brings us back to the variations in density in the US map…</p>

<p>The first thing to understand about the US is that most populated places there
appear have been initially imported from the
<a href="http://wiki.openstreetmap.org/wiki/GNIS">USGS GNIS</a> data set. I haven’t found
any documentation as to how places were categorised but I suspect it was done
based on population and most likely using the values in the OpenStreetMap wiki
or something close to them.</p>

<p>Justin’s first example starts with the apparent high density of places in
Florida so I took a look at a randomly selected place in his example which
appeared to be fairly small – the town(?) of Frostproof. The
<a href="http://www.openstreetmap.org/browse/node/154051432/history">OpenStreetMap history for Frostproof</a>
reveals that it was originally imported from GNIS as a village (probably because
of it’s population of 2922) but has recently been retagged as a city.</p>

<p>My suspicion is that this is the result of an overly literal interpretation of
the place=city tag – as I understand things many relatively small places in the
US officially style themselves as cities – certainly
<a href="http://en.wikipedia.org/wiki/Frostproof">Wikipedia describes Frostproof</a> in
this way. Nobody in Britain, or indeed probably in Europe as a whole, would
consider somewhere that small to be a city however and tagging it as such
certainly goes against normal OpenStreetMap tagging practice.</p>

<p>In most of the rest of the US no such retagging of small towns as cities appears
to have taken place, making place names there appear much less dense at low zoom
levels. The sort of places which Justin’s article suggests should be appearing
in those areas mostly appear to be in the 25-100 thousand population range and
hence have been tagged as towns during the GNIS import. The solution here, if
more place names are considered cartographically desirable, would either be to
adjust the threshold at which places are tagged as cities instead of town, or to
alter the stylesheets to render towns at lower zoom levels.</p>

<p>The relatively high density around Los Angeles which the article mentions
appears to be the result of a fairly large number of places with populations
just over the 100 thousand mark. Despite their large populations, and the fact
they are likely independent cities legally, I suspect that many of them would be
tagged as suburbs in Britain rather than as cities or towns and hence would be
given lower priority when rendering.</p>

<p>The real lesson to be drawn from all this however is that the US OpenStreetMap
community probably needs to reach a consensus on how to map populated places to
tag values so that a better level of consistency can be achieved with less
variation from area to area across the map.</p>]]></content><author><name></name></author><category term="geo" /><category term="openstreetmap" /><summary type="html"><![CDATA[In the second part of his critique of OpenStreetMap Justin O’Beirne discusses various issues surrounding labelling of cities in OpenStreetMap’s cartography, specifically in our default mapnik rendering of the US. The issues he highlights can be broadly divided into two categories: problems with our stylesheets and rendering technology; and problems with our data, and in particular with our US data. The issue which I intend to address here is the one he tackles first – that of label density which is something that stems largely from data quality and, more importantly, consistency issues. Specifically, although the post talks about cities, the real question is about what is tagged as a city and what is tagged as some lesser type of place.]]></summary></entry><entry><title type="html">The Mystery of John Unwin’s Diary</title><link href="https://compton.nu/2010/09/the-mystery-of-john-unwins-diary" rel="alternate" type="text/html" title="The Mystery of John Unwin’s Diary" /><published>2010-09-29T17:16:05+01:00</published><updated>2010-09-29T17:16:05+01:00</updated><id>https://compton.nu/2010/09/the-mystery-of-john-unwins-diary</id><content type="html" xml:base="https://compton.nu/2010/09/the-mystery-of-john-unwins-diary"><![CDATA[<p>Two weeks ago I travelled to <a href="http://www.saltairevillage.info/">Saltaire</a> with
my uncle to hand over a number of items of historical interest from my
grandfather’s papers to the Saltaire Archive.</p>

<p>The most significant item in the collection was a personal diary for the year
1897 which carried an inscription on the flyleaf of “John Unwin, Fanny Street,
Saltaire”.</p>

<p>In itself the diary is a fascinating piece of social history and that is
certainly the main reason for including it in the archive for the benefit of
future generations. The diary is of interest to my family for a second reason
however, which is the mysterious way in which it came to be in the possession of
my grandfather.</p>

<!-- more -->

<figure><a data-fancybox="gallery" href="/assets/images/diary-cover.jpg"><img src="/assets/images/diary-cover.jpg" /></a><figcaption>The cover of the diary.</figcaption></figure>

<p>The story is that in the 1930s my great-grandfather, David Unwin, was the
general foreman on a building site in Cricklewood when one of the workers on the
site came to him with the diary and asked, given that as foreman my
great-grandfather would simply have been “Mr Unwin” and that the diary belonged
to John Unwn, if the diary was his.</p>

<p>Obviously David Unwin knew that the diary was not his, but equally he knew that
his father and his older siblings had been born in Shipley and that he had many
relatives in the Saltaire area, and he therefore assumed that the diary must
belong to a relative of his.</p>

<p>Despite a number of attempts by members of the family over the last eighty years
to figure out who John Unwin was and how, or if, he was related to our family no
real progress was made until I started on some genealogical investigations a few
years ago in an effort to draw up a family tree.</p>

<p>Looking at the tree I had assembled there was one obvious candidate for a John
Unwin who would have been a suitable age in 1897 and reading the diary confirmed
beyond any doubt that he was in fact the author – among other things the diary
records the departure of his brother Robert on his way to a new life in
Bridgeport, Connecticut; an event that I had already discovered evidence of
during my research.</p>

<figure><a data-fancybox="gallery" href="/assets/images/diary-ru-departure.jpg"><img src="/assets/images/diary-ru-departure.jpg" /></a><figcaption>Diary entry for 1st October recording departure of Robert Unwin (RU) for the USA. Shipping records show that he sailed from Liverpool the next day.</figcaption></figure>

<p>The result of my research was therefore to discover that the author was in fact
the first cousin of my great-grandfather, and my first cousin three times
removed. Quite how the diary came to travel from Shipley in West Yorkshire to
Cricklewood in North London between 1897 and 1930 remains a mystery, especially
given the excellent condition in which it survived despite being found on a
building site!</p>

<p>The diary is now, as I indicated at the start, in the Saltaire Archive, along
with my scans of the diary and my uncle’s transcription, having been handed over
to representatives of the
<a href="http://www.saltairevillage.info/historyclub.html">Saltaire History Club</a> and
members of the Salt family (the great-grandson of Sir Titus Salt, Denys Salt,
and his nephew, Jonathan) during this year’s Saltaire Festival.</p>]]></content><author><name></name></author><category term="genealogy" /><category term="history" /><summary type="html"><![CDATA[Two weeks ago I travelled to Saltaire with my uncle to hand over a number of items of historical interest from my grandfather’s papers to the Saltaire Archive. The most significant item in the collection was a personal diary for the year 1897 which carried an inscription on the flyleaf of “John Unwin, Fanny Street, Saltaire”. In itself the diary is a fascinating piece of social history and that is certainly the main reason for including it in the archive for the benefit of future generations. The diary is of interest to my family for a second reason however, which is the mysterious way in which it came to be in the possession of my grandfather.]]></summary></entry></feed>