Vancouver Sun Parking Tickets Website Screen Scraper

[img_assist|nid=367|title=|desc=|link=none|align=right|width=150|height=146]
When the Vancouver Sun came out with their Vancouver parking tickets database I immediately had some burning questions, like, did the meter maids work on holidays? Do the work less in the evening than during the day? I found it difficult to answer these questions using their interface, so I decided to screen scrape all 1.6 million parking tickets in to my own MySQL database. This was a bit challenging as they made it difficult to screen scrape the data but eventually it could be done simply by first getting an AppKey, a hidden value inside the HTML source and then doing queries using that AppKey as a parameter. It took about a week to get all 1.6 million tickets downloaded. By using Django, it was easy to get them in to a database and view the results. Initially I just put all the data in to one table, then later I decided to normalize the data a bit which was interesting as I decided to do that in pure SQL which I hadn't done before. I did the scraping itself using a combination of BeautifulSoup, lxml, and mechanize.

vancouver-parking-tickets project at GitHub

MySQL SQL dump (42 MB)

Here's some data:

Number of tickets by day of the week:

Day of week Count
Monday 238482
Tuesday 277797
Wednesday 294478
Thursday 274529
Friday 243830
Saturday 181235
Sunday 121036

Number of tickets at each address:

Address Count
1050 Robson St. 17899
1150 Robson St. 15674
850 Hornby St. 10501
650 Broadway St. W. 10426
1050 Alberni St. 9729
1750 Broadway St. W. 9121
1050 Hornby St. 9100
650 Hornby St. 8863
1050 Mainland St 8328
1050 Homer St. 8269
1150 Hamilton St. 8094
850 Howe St. 7941
850 Broadway St. W. 7848
2250 4th Ave W. 7634
950 Hornby St. 7462
1950 4th Ave W. 7403
2650 Granville St. 7354
1650 Broadway St. W. 7062
1350 Broadway St. W. 6961
1150 Broadway St. W. 6903
1050 Hamilton St. 6889
950 Broadway St. W. 6821
150 Davie St. 6676
1150 Homer St. 6658
1150 Mainland St 6562
750 Broadway St. W. 6553
950 Homer St. 6504
1150 Davie St. 6461
1150 Alberni St. 6386
1050 Davie St. 6202
Topic: 

Comments

I'm going to put the data up on here. I need to gather it up again. I just did some queries in Django to the console so didn't save it or anything. I had some trouble querying for the day of the week, I had to add a patch to Django to make it work, not sure if I still have that patch or not. Ended up using plain SQL, but I didn't save the queries anwhere.

Seriously: scraping millions of parking tickets and cross referencing dates,hours, locations etc. is brilliant. Imagine an iPhone app for that in each major city: "what are the chances I'll get a ticket here?"!!!

Most of the time I don’t make comments on websites. but I'd like to say that this article really forced me to do so. Really nice post!
coparenting

This is just the information I am finding everywhere. Thanks for your blog. I just subscribe your blog. This is a nice blog..
For More Information

I admire what you have done here. I like the part where you say you are doing this to give back but I would assume by all the comments that this is working for you as well.
moped insurance

I am unable to get many games on xbox so i am wondering for the xbox live codes so that it could help in buying these paid games.

I am happy to find this post very useful for me. as it contains lot of information. I always prefer to read the quality content and this thing I found in you post. Thanks for sharing.
scooter insurance

This is a great post. I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again.
Brookside

This is a good post. This post gives truly quality information. I’m definitely going to look into it. Really very useful tips are provided here. Thank you so much. Keep up the good works.
walmartone

This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post.
Survival gear

I can’t imagine focusing long enough to research; much less write this kind of article. You’ve outdone yourself with this material. This is great content.
How to survive in the woods

Super-Duper site! I am Loving it!! Will come back again. Im taking your feed also. Thanks.
More Info

i am always looking for some free stuffs over the internet. there are also some companies which gives free samples.
oneskin

Nice blog and absolutely outstanding. You can do something much better but i still say this perfect.Keep trying for the best.
Wilderness Survival

Nice post. I was checking constantly this blog and I am impressed! Extremely helpful information specially the last part I care for such info a lot. I was seeking this particular information for a very long time. Thank you and good luck.
AIRPORT LIMO MINNEAPOLIS

If more people that write articles really concerned themselves with writing great content like you. more readers would be interested in their writings. Thank you for caring about your content.
MINNEAPOLIS AIRPORT CAR SERVICE

i am for the first time here. I found this board and I in finding It truly helpful & it helped me out a lot. I hope to present something back and help others such as you helped me.
rodents

Thanks for taking the time to discuss this. I feel strongly about it and love learning more on this topic. If possible. as you gain expertise. would you mind updating your blog with more information? It is extremely helpful for me. Lazma Cream

I have been checking out a few of your stories and i can state pretty good stuff. I will definitely bookmark your blog
xiaomi goods

ugg outlet
coach factory outlet
coach outlet
cheap nfl jerseys
nfl jerseys wholesale
ray ban sunglasses
cheap ray ban sunglasses
ray ban sunglasses
ray ban sunglasses discount
kate spade handbag
kate spade bags
canada goose jackets
canada goose outlet
air max 2018
moncler jackets outlet
moncler outlet
adidas yeezy
adidas yeezy boost
harden vol 1
nike air max
louboutin shoes
christian louboutin outlet
kate spade
kate spade handbags
nike shoes
fitflops sale
fitflops
cheap uggs
ugg boots for women
ugg boots outlet
ugg boots
ugg outlet
ugg outlet store
ugg sale
ugg boots on sale
uggs canada
ugg boots canada
canada goose jackets
canada goose
canada goose outlet
canada goose sale
canada goose outlet
canada goose sale
louis vuitton outlet
louis vuitton factory outlet
ugg ustralia
ugg boots
ugg ustralia
ugg australia
canada goose outlet
canada goose
burberry outlet
burberry sale
canada goose jackets
canada goose outlet
canada goose outlet
canada goose sale
cat boots
coach outlet
coach canada
canada goose outlet
canada goose sale
kate spade
kate spade outlet
longchamp outlet
longchamp handbags
louis vuitton outlet
louis vuitton outlet store
mbt shoes
michael kors outlet
michael kors outlet store
north face outlet
north face outlet online
polo ralph lauren
polo ralph lauren outlet online
red bottom shoes
louboutin shoes
timberland outlet
ugg boots
ugg boots on sale
ugg boots
ugg outlet
ugg slippers
ugg sandals
cheap uggs
cheap uggs
ugg sale
canada goose jackets
ugg boots
uggs outlet
moncler outlet
moncler jackets
ralph lauren outlet online
ralph lauren outlet
polo outlet
polo ralph lauren outlet online
air jordans
air jordan shoes
canada goose clothing
canada goose coats
canada goose jackets
canada goose coats
cheap uggs
coach factory outlet
coach outlet
coach outlet store online
coach outlet
coach outlet
coach factory outlet
coach factorty outlet
coach outlet
columbia sportswear
ray ban sunglasses
cheap ray ban sunglasses
hermes outlet
hermens bags
louis vuitton outlet
louis vuitton outlet store
moncler jackets
moncler coats
moncler outlet
moncler jackets
north face jackets
pandora charms sale clearance
pandora charms sale
ralph lauren outlet
polo ralph lauren outlet online
salvatore ferragamo shoes
salvatore ferragamo
timberland boots
toms outlet
tory burch outlet
ugg outlet
ugg boots
ugg boots
uggs on sale
ugg boots
uggs
ugg outlet
ugg boots
ugg shoes
ugg shoes
ugg outlet
uggs clearance
ugg outlet
ugg outlet online
louis vuitton
louis vuitton outlet online
burberry outlet
cheap jerseys
cheap jerseys from china
michael kors outlet
michael kors outlet online
nike air max
air max 2017
adidas superstar
fred perry
fred perry polo
louis vuitton outlet
louis vuitton outlet online
mbt shoes
michael kors outlet
michael kors outlet store
coach factory outlet
coach outlet online
canada goose outlet
canada goose jackets
burberry outlet
uggs clearance
canada goose outlet
canadian goose
ugg boots
cheap uggs
canada goose outlet
canada goose sale
canada goose jackets
canada goose clothing
ugg outlet
moncler outlet
moncler jackets
coach factorty outlet
coach outlet
canada goose jackets
canada goose uk
adidas shoes
cheap ray bans
cheap ray ban sunglasses
polo outlet
polo ralph lauren outlet online
canada goose outlet
canada goose
michael kors outlet
michael kors outlet clearance
air jordan shoes
jordan shoes
burberry outlet
burberry outlet store
north face jackets
ralph lauren outlet
polo ralph lauren outlet online
ugg australia
cheap jordans
cheap jordans for sale
ugg outlet
ugg uk
ugg
moncler jackets
canada goose jackets
canada goose
oakley sunglasses
ugg boots
ugg boots
ferragamo outlet
ferragamo shoes
burberry outlet
burberry
canada goose outlet
supreme clothing
uggs outlet
ugg outlet
the north face
north face outlet
moncler uk
moncler
canada goose jackets
canada goose
canada goose jackets
canada goose coats
michael kors outlet
michael kors outlet store
nike outlet
nike outlet store
moncler uk
moncler jackets
ugg outlet
ugg outlet online
gucci handbags
cheap ugg boots
ugg boots
supreme new york
supreme clothing
ugg canada
uggs canada
longchamp outlet
longchamp outlet online
north face jackets
north face outlet
north face jackets
ralph lauren outlet
polo ralph lauren outlet
ugg outlet
uggs clearance
canada goose jackets
ugg outlet
uggs outlet
ugg outlet
uggs outlet
canada goose outlet
ugg boots
canada goose sale
uggs on sale
uggs sale
ugg boots
ugg boots on sale
uggs outlet
ugg outlet
canada goose outlet
ugg australia
uggs canada
uggs
ralph lauren outlet
ugg outlet
ugg outlet
ugg outlet
ugg outlet
ugg outlet
ugg boots
canada goose jackets
canada goose jackets
canada goose outlet
canada goose outlet
ralph lauren
ralph lauren outlet
ugg outlet
ugg outlet
ugg outlet
canada goose outlet
canada goose outlet
ugg boots
ugg boots
canada goose outlet
canada goose outlet
canada goose outlet
ralph lauren outlet
2017.11.4yangping

Add new comment