David Grant's blog

Just a month ago I got a job at Grow Financial. We're a small fintech startup that provides unsecured loans to prime borrowers. So far we're just in Canada and just offering unsecured loans up to $30,000, but we're growing fast and hopefully we'll be offering even more products in the future. We are really small; I was the third software engineer to join the team (neglecting a very key part of our team, the data scientists), and since I started in December 2015, we've since added 2 more engineers to our team. We have an awesome team of top-notch people, which is the main reason why I accepted an offer at Grow (despite having a few other opportunities available to me). Management's enthusiasm, vision, and success so far blew me away.

We just made a big announcement today, a partnership with . Full article in the Globe and Mail: "First West partners with online lending startup Grow."

Just documenting a few issues I've had in setting up GWT + GAE + Maven + Eclipse.

This is based on the sample "mobilewebapp" that comes with GWT (the version on master as of December 2015).

Eclipse not finding GAE

Eclipse might complain about not being able to find GAE under the ~/.m2/repository folder. To fix this, in the project's build path settings, under the "Order and Export" tab, put the Maven dependencies last.

"Choose war folder" prompt

WARNING: whatever folder you choose here will be deleted!!! When you try to run (as "Web Application (GWT Super Dev Mode)") and it asks you to choose the war folder, choose the target/${artifactId}-${version} folder.

GWT complains about CssResource

You need to add this to your .gwt.xml file:

<set-configuration-property name="CssResource.enableGss" value="true" />
<set-configuration-property name="CssResource.legacy" value="true" />

I was recently trying to connect to a CentOS box from Windows and had the following error in /var/log/secure (after enabling DEBUG3 level logging in /etc/ssh/sshd_config):

Dec 19 18:01:05 hostname sshd[25119]: debug1: trying public key file /root/.ssh/authorized_keys
Dec 19 18:01:05 hostname sshd[25119]: debug1: Could not open authorized keys '/root/.ssh/authorized_keys': Permission denied

It's an SELinux problem, and I don't quite understand it, but here's the one-liner that fixes it:

restorecon -FRvv ~/.ssh

Found on stackoverflow: "SSHD Gives error could not open Authorized Keys, although permissions seem correct"

A recent article titled "Vancouver housing prices tied to China’s economic growth", Barbara Yaffe quoted two "experts" (David Ley and Robin Wiebe) who will tell anyone who will listen that "housing prices here [in Vancouver] are tied inextricably to purchase activity by foreign buyers." You'll find the use of words like "coincide," "relationship," and "exceptional correlation" between 1) periods of faster growth in China and growth of Vancouver's housing market, 2) house sales and housing starts and GDP growth in China, and 3) international immigration to Vancouver and the city's property prices. Nowhere do we see the words "cause" or "leads to."

Correlation is mostly meaningless, especially when you're using it in the way that David Ley does (see Figure 5.8). One commenter says:

I work at the same institution as Professor Ley, but still I must critique the "evidence" presented in the graph. Perhaps there is more elsewhere in the study, but Figure 5.8 does not provide evidence of anything.
A correlation of 0.94 or 94% looks impressive, but those who know statistics will understand that in the analysis of time-series data, one cannot just correlate the level of something (house prices) and the level of something else (immigration). It is very easy to get very high correlations. Don't believe me. Just try computing the correlation of the house prices with your age. I did. Guess what--the correlation is 94.5%. Evidently, my age predicts Vancouver house prices better than net immigration. Obviously, you can put in anyone's age and the result would be the same.

You can see more examples of ridiculous correlations at spurious correlations, like "US spending on science, space, and technology correlates with suicides by hanging, strangulation and suffocation" (0.992) and "Divorce rate in Maine correlates with Per capita consumption of margarine" (0.993).

Repeat after me: correlation does not imply causation.

Also lacking in the Yaffe article was any serious discussion of the more obvious causes of housing price increases, such as interest rates, 0% down mortgages and 40 year amortizations in the mid 2000s, and the effect of CMHC default insurance. These all have a real effect on housing prices. There was a singling out of Vancouver, and no discussion of all the other Canadian cities that have experienced similar relative price increases, and no mention of all the non-foreign buyers owning multiple condos (and renting them for very little profit), thus speculating on price increases on the value of the home.

The effect that interest rates have is huge. The Bank of Canada bank rate was 6% at it's peak of the tech bubble (2000)! then the tech bubble burst and the Bank of Canada started slashing the interest rates to as low as 0.5% (2009); a swing of 5.5%! The bank variable rates are a bit more than that, I'm getting 2.2% which is about the lowest they've ever gone. So let's say 2% and 7.5% for a spread of 5.5%. To make things even crazier, throw in the 40 year mortgage which was available for a little while:

Effect of amortization period and interest rate for a $500,000 home:
$3650 monthly payment (at 7% for 25 years)
$1500 monthly payment (at 2% for 40 years)

That's with no change to down payments. Add in the effect of lower down payments (increased leveraging) due to lower requirements + CMHC default insurance, people owning multiple properties (more leveraging and speculation), and people buying more house than they can afford (if interest rates increased but they assume house value will only go up), and you have a recipe for the bubble we have today.

Finally, Vancouver has experienced the same growth in housing prices that the rest of Canada has.

All figures below are from http://www.housepriceindex.ca (which actually measures same house sale pairs over time, not just "average home prices"). I downloaded the spreadsheet and calculated the total percentage increase in the index from early 1999 (data is spotty before then, some cities go back to 1990, others don't).

Winnipeg 305%
Edmonton 290%
Quebec City 285%
Vancouver 276%
Calgary 272%
Montreal 259%
Toronto 242%
Victoria 231%
Hamilton 229%
Ottawa 225%
Halifax 217%

So is the "foreigner effect" happening everywhere in Canada? Or maybe the increases are due to something other than foreigners? During a bubble people are often quoted as saying "it's different this time" or "it's different here." They always end up being wrong.

I did several things to increase the battery life on my Nexus 4 phone. Here's a list of all the things I did. I am not sure exactly what led to the biggest improvement in battery life because I didn't do extensive testing at each step. Here they are:

Unlock, custom ROM, packages

• Unlock, install ClockworkMod Recovery, install CyanogenMod 11 and Google Apps.
• Install the hells-Core kernel from ClockworkMod Recovery.
• Install the Hellscore Kernel Manager
• Install the Greenify app.
• Install the BetterBatteryStats app.

Configuration

• If you can, delete the Facebook app and use Facebook in your browser instead.
• Disable notifications and auto-updating/auto-syncing in as many apps as you can.
• Look at what is using the most battery under "Setings->Battery" and remove any apps that you don't actually need.
• If you can, disable Google Now and Google Location Reporting (found in "Settings->Location").
• Put "Location->Mode" into "Battery saving" mode
• Under "Settings->Sound" disable "Touch sounds", "Vibrate on touch", and "Screen lock sound"
• Under "Settings->Language & Input->Android Keyboard (AOSP)" disable "Sound on keypress" and "Vibrate on keypress"
• In the Hellscore Kernel Manager, set the CPU "Min Freq" to 94 MHz.
• After recharging your phone and using it for a whole discharge cycle, look at Partial Wakelocks in the BetterBatteryStats app.
• Using the Greenify app, Hibernate as many of the apps that show up in the "App Analyzer" as you can.
• In "Settings->Display & lights->Brightness", set to "Auto" and then click the gear to configure. Enable "adjust to sunrise and sunset". Click "Adjust" and move all the sliders to the left by about 5% or so.
• Get a watch so you don't have to wake-up your phone to check the time.

Now I can get about 48 hours on one charge with minimal usage (checking email once in a while, sending a few messages, checking a few web pages.

Just recently, I was implementing a protocol at work. The protocol was handed to me in the form of a Word document and it appeared to have already cleared some approval stages. The protocol seemed over-engineered to me. It used HTTP POST with XML in the request and response bodies. There was actually only one message type and it seemed like it could have easily been implemented with a simple GET request and no response body (just an HTTP response with a proper response code). The document was 24 pages long when it only required 1. It had about 10 error response message types, about half of which had to do with XML errors. I looked at who reviewed the document and it was as follows:

• Author: Engineering Manager
• Reviewer: VP Engineering
• Reviewer: Technical Director
• Reviewer: Director of Engineering, Client Engineering
• Reviewer: Director of Engineering, Host and Infrastructure Engineering
• Reviewer: Engineering Manager, Quality Engineering

Missing are Principal Engineers and most importantly, Senior Developers, like myself, who will be implementing the protocol. It's not even necessary to consult the people who will be implementing the protocol, but you need to consult with people who understand HTTP, have experience with using web protocol, and an understanding of programming. One of those listed above qualifies in that respect, but that is still just one person. I am sure that if this was shown to three other people who aren't managers, directors, or VPs, they would say this protocol is over-engineered. If even one person thinks it's over-engineered, and over-engineering things is costly. It adds complexity on both sides of the protocol which means more development, more bugs, more testing. There's no excuse for over-engineering things and not excuse not to have technical documents revealed by those who will implement, the hands-on technical people.

For long time I've been wondering why I can't seem to burn DVDs or Blu-Rays properly using k3b in Linux (or any other graphical burning software for that matter). The problem is cdrkit (a fork of the original cdrtools).

You can get the latest version of cdrtools, which actually supports burning to Blu-Ray and also should create DVDs properly, by installing cdrecord from this PPA. You might as well also install mkisofs from there too.

Prior to finding this PPA, I had compiled cdrtools from source, which is pretty easy. That also involved installing smake, also from the cdrtools author.

I have a simple video website for my kids and each kid has a separate login. This is so they can each have their own videos, but also so that some videos can be private (ie. hidden from the outside world, or other logged in users). Typing in a username and password is impossible for my kids to do, as they are almost 5 and 2 years old, and they use this website on Google TV. So, with a magic token-style login, all they need to do is navigate to their bookmark on the Google TV homepage and press OK on the remote control.

(I don't need crazy security--it wouldn't be the end of the world if somehow someone guessed the magic token and saw some private videos, which are basically just home videos uploaded to Youtube. Videos that I really wouldn't want the public to see don't get uploaded to Youtube in the first place.)

I couldn't find how to do this easily, although one person on stackoverflow suggested "logging in the user in the view by calling 'login'". The tricky part was figuring out that I had to set the User object's backend to 'django.contrib.auth.backends.ModelBackend'. It's a bit of a hack, but it works, and it's simple.

models.py:

class MagicToken(models.Model):
    user = models.OneToOneField(User)
    magictoken = models.CharField(max_length=128, unique=True)
 
    def __unicode__(self):
        return unicode(self.user)

views.py:

from django.http import HttpResponse, HttpResponseRedirect, Http404
import django.contrib.auth.login
 
class MagicTokenLogin(View):
    def get(self, request, token):
        try:
            magic_token_obj = MagicToken.objects.get(magictoken=token)
        except MagicToken.DoesNotExist:
            raise Http404
 
        user = magic_token_obj.user
        user.backend = 'django.contrib.auth.backends.ModelBackend'
        django.contrib.auth.login(request, user)
        if request.user.is_authenticated():
            # login successful
            return HttpResponseRedirect(reverse('some-view-for-logged-in-users'))
        else:
            # login failed
            return HttpResponseRedirect(reverse('some-view'))

I recently renamed a folder in my copy.com folder and caused Copy.com to think that I had deleted it. For example, I had the a folder called "Photos 2013." I installed the copy.com agent on a new computer and it began syncing. Midway through the sync, I wanted to change the folder's name to "Photos 2013-2014". This was interpreted as "Delete 'Photos 2013' folder and all sersver-side contents" and create a new folder called Photos 2014 with the photos in it. So let's say there were 100 photos, and 10 were were synced when I did it. The 90 photos that had not yet been synced would be flagged for deletion on the copy.com server. The 10 photos that actually got synced will remain, in the new "Photos 2013-2014" folder.

All I can suggest to avoid this is don't touch the folders while they are being synced.

To wipe a hard drive using zeros:

dd if=/dev/zero | pv -s 250000000000 | dd of=/dev/sdX bs=10M

To wipe a hard drive with random data (more secure):

dd if=/dev/urandom | pv -s 250000000000 | dd of=/dev/sdX bs=10M

For the "pv" progress bar, do "apt-get install pv" or similar. I highly recommend using "pv" so you have some idea of when the dd job is going to finish.

I just set up a site using Django-CMS over at Pomme d'Api Preschool. Overall, I am pretty impressed. I really like the Django admin interface enhancements as well as the site overlay editing feature. One of the best things it he use of django-filer, a file management tool to manage all the uploads to the site. It's awesome and it makes me want to throw out this home page completely and switch over to Django-CMS. Image handling in Drupal has always sucked and still does. The fact that there was no official media/image handling in Drupal 6 sucks, and it's made worse by the fact that there is no real upgrade path to Drupal 7 for media/images, especially if you're like me and you used the "Image" and "image-assist" modules, two of the most popular modules dealing with images.

Django-CMS is pretty powerful and so far it has allowed me to do pretty much anything I have wanted to do.

If you ever get something like this:

Running /usr/bin/svn add -N --targets /tmp/svn_load_dirs_ErUQhjXmWG/targets.00001
/usr/bin/svn_load_dirs: /usr/bin/svn add -N --targets /tmp/svn_load_dirs_ErUQhjXmWG/targets.00001 
failed with this output:
svn: warning: W150002: '/tmp/svn_load_dirs_ErUQhjXmWG/my_import_wc/scripts/drupal.sh' is already under version control
svn: warning: W150002: '/tmp/svn_load_dirs_ErUQhjXmWG/my_import_wc/scripts/run-tests.sh' is already under version control
svn: warning: W150002: '/tmp/svn_load_dirs_ErUQhjXmWG/my_import_wc/scripts/password-hash.sh' is already under version control
svn: E200009: Could not add all targets because some targets are already versioned
svn: E200009: Illegal target for the requested operation

when running svn_load_dirs, try running svn_load_dirs with the -no_auto_exe option. That fixed it for me.

I was working on a personal website for my kids where I can post videos for them and I noticed that for each video link that was displayed on the page there were 2 additional SQL queries. One to get the username of the user that added the video link, and another to get the list of categories for the link.

The model looks like this:

class Link(models.Model):
    ...
    # There was an extra query per link to get the categories for a link
    category = models.ManyToManyField(Category)
    # There was an extra query per link to get the user.username
    user = models.ForeignKey(User)
    ...

The reason that an extra query was required to get the username is because the query to get all the links does not look at the user table at all. It doesn't do any joins. So it has to look up the username by doing something like "select username from users where users.pk == link.user_id". You get the idea. And it has to do this for every link on the page. This is easily fixed by doing

Link.objects.select_related().all()

or something similar. Django will then do a JOIN on the User table so it will now have all the user-related fields in the result set.

I was still getting a query for the category names on each link. To resolve this, I did:

Link.objects.select_related().all().prefetch_related('category')

This causes Django to do a fetch on the Category_Link table (the table that describes the many-to-many relationship) to get all the categories that match the primary keys for the links displayed on the current page. Then a join is done in Python, essentially mapping a list of category objects with each link object.

The number of queries for my home page was reduced from 44 down to 5. Page load time has been decreased by 1-2 orders of magnitude.

So, the naive rule-of-thumb might be to always use select_related() if you have a ForeignKey and you are going to reference fields in the ForeignKey in your template, and to always use prefetch_related if you have a ManyToManyField and you need information in the related table. For now I am going to NOT use these commands unless I first use the django-debug-toolbar and figure out what is going on. Then if it's necessary to use select_related or prefetch_related I will.

I've been maintaining and old Joomla 1.5-based website for my daughter's preschool, Pomme d'Api. It got hacked, I'm not sure how, via FTP or PHP but they basically managed to get 2 .php files on the server that were causing search engine crawlers to get a different version of the site that contained all sorts of links to various pages. It also changed the meta-information such as keywords and description. I've now removed the hack and instructed Google to re-index the site via Google's Webmaster tools, but frustratingly, Google's crawler still hasn't re-indexed the site! It's been over a week since Google last visited it... hopefully providing this link to the website will cause Google to re-index it.

This is intended as a dumping ground for all the tips on using exceptions that I accumulate over the years.

Don't swallow exceptions

Never, ever do this:

try {
  ...
} catch (IOException ioe) {
  // do nothing here
}

Always log a message, or print a stacktrace, or re-throw the exception, or throw a new exception (and wrap the caught exception).

Allow your checked exceptions to wrap other exceptions

class MyException extends Exception {
  public MyException(String message, Throwable cause) {
    super(message, cause);
  }
}

This is almost always useful. You will often find yourself catching an IOException and re-throwing it as your own exception. The main point is DON'T DISCARD THE ORIGINAL EXCEPTION. If you do, you won't see it any stack traces as the "caused by:" exception.

Don't log an exception that you are re-throwing or wrapping

This is a minor detail but it makes for much cleaner logs.

try {
  ...
} catch (IOException ioe) {
  log.error("Got an IOException", ioe); // not necessary
  throw new MyException("Had a problem contacting the server on port " + port, ioe);
}

In the above, assuming the logger is log4j or something similar, it is going to print the stacktrace for the IOException exception. Then whatever catches MyException up the stack will also log the IOException. This makes for message logging. Either log the IOException and do not wrap the IOException, or don't log it and just wrap it.

Create your own top-level exception classes

I've often found it useful to create two top-level exception classes, one that extends Exception and one that extends RuntimeException. For example, ABCCompanyException and ABCCompanyRuntimeException. Sometimes at the top-level of your application (in the main() method or the HTTP request handler, for example) you want to catch everything that hasn't been dealt with yet. It can be useful to handle "ABCCompanyException" and it's sub-classes separately from Java's "Exception" or "ABCCompanyRuntimeException" separately from Java's RuntimeException.

Don't forget to catch run-time exceptions in callback methods

Let's say you're implementing an event handler to be used in a third-party event bus framework, or implementing an "onOpen/onClose/onMessage" type interface for a web socket client. You should wrap the entire body of the method in a try...catch (Throwable t) otherwise a run-time message might be thrown somewhere, bubble up into the third-party library, and be swallowed there. Do this any time you are implementing a method that you don't actually call yourself. The same thing goes for the run() method in the Runnable class.