Roc's blog

About web development

Data Visulazation - Import Data

I have been worked on some data visualization projects in last few years for different kind of clients, used some different set of library/framework like backbonejs, raphaeljs, d3js. Start from this post, i’ll try to write down process on how we can do data visualization with modern technologies.

The data source will be http://data.police.uk , which include different crimes records in UK.

Install mongodb

Mongodb has many features like document, schemaless, query, etc, the reason i choose it is it’s very easy to load the data into it using mongoimport.

brew install mongodb

Start mongodb

mongod

Create database

use ukcrimes

Download data from http://data.police.uk/data/

Import data into database

mongoimport --db ukpolice --collection crimes --type csv --headerline --file file_path

Check imported data

mongo
use ukcrimes
db.crimes.find()  

You’ll see data like:

{ "_id" : ObjectId("544a7172d0e2b18dc73d060c"), "Crime ID" : "ca8de697ca1a5b0b1f76a3c0f3f652bf2e9e20a31fbf5af40df5de7fa55f63a4", "Month" : "2014-07", "Reported by" : "Avon and Somerset Constabulary", "Falls within" : "Avon and Somerset Constabulary", "Longitude" : "", "Latitude" : "", "Location" : "No location", "LSOA code" : "", "LSOA name" : "", "Outcome type" : "Offender given penalty notice" }
{ "_id" : ObjectId("544a7172d0e2b18dc73d060d"), "Crime ID" : "c50470f2a1af2a5fffaa3dd69218777200c1a9289b87dd36849325ed2585472e", "Month" : "2014-07", "Reported by" : "Avon and Somerset Constabulary", "Falls within" : "Avon and Somerset Constabulary", "Longitude" : "", "Latitude" : "", "Location" : "No location", "LSOA code" : "", "LSOA name" : "", "Outcome type" : "Offender given a drugs possession warning" }
{ "_id" : ObjectId("544a7172d0e2b18dc73d060e"), "Crime ID" : "b7458f39eb58018af53c85fd481aa2028e247baa30ceb1f095437359e78b63f6", "Month" : "2014-07", "Reported by" : "Avon and Somerset Constabulary", "Falls within" : "Avon and Somerset Constabulary", "Longitude" : "", "Latitude" : "", "Location" : "No location", "LSOA code" : "", "LSOA name" : "", "Outcome type" : "Suspect charged" }
{ "_id" : ObjectId("544a7172d0e2b18dc73d060f"), "Crime ID" : "3093f07675cd62ce39cc8eeb80261edb15d00cbee2bf2ece8d295ac6a0efab4e", "Month" : "2014-07", "Reported by" : "Avon and Somerset Constabulary", "Falls within" : "Avon and Somerset Constabulary", "Longitude" : "", "Latitude" : "", "Location" : "No location", "LSOA code" : "", "LSOA name" : "", "Outcome type" : "Suspect charged" }

To understand the data columns, you can check http://data.police.uk/about/

Query crimes stats

db.crimes.group({key: { 'Crime type': 1}, reduce: function(curr, result){ result.total += 1}, initial: { total: 0 } })

result:

[
{
    "Crime type" : "Anti-social behaviour",
    "total" : 5593
},
{
    "Crime type" : "Burglary",
    "total" : 888
},
{
    "Crime type" : "Criminal damage and arson",
    "total" : 1215
},
{
    "Crime type" : "Other theft",
    "total" : 1343
},
{
    "Crime type" : "Violence and sexual offences",
    "total" : 2123
},
{
    "Crime type" : "Drugs",
    "total" : 292
},
{
    "Crime type" : "Public order",
    "total" : 577
},
{
    "Crime type" : "Shoplifting",
    "total" : 1019
},
{
    "Crime type" : "Vehicle crime",
    "total" : 689
},
{
    "Crime type" : "Possession of weapons",
    "total" : 22
},
{
    "Crime type" : "Bicycle theft",
    "total" : 271
},
{
    "Crime type" : "Other crime",
    "total" : 88
},
{
    "Crime type" : "Theft from the person",
    "total" : 172
},
{
    "Crime type" : "Robbery",
    "total" : 91
}
]

Issues of Upgrading to Yosemite Related to Homebrew

Fix brew command

In Yosemite the ruby command path changed, yosemite use ruby 2.0 as default also create a new path to link to ruby 2.0:

/System/Library/Frameworks/Ruby.framework/Versions/Current/usr/bin/ruby

You need to change ruby path in the first line of /usr/local/Library/brew.rb

Update brew

After fix the brew command, it’s better to update brew itself to avoid other issues

brew update

Fix postgresql

After upgrade, you’ll get following error when try to run: postgres -D /usr/local/var/postgresql

FATAL:  could not open directory "pg_tblspc": No such file or directory

the fix is easy, just create those directories, those directories are gone after upgrade.

mkdir /usr/local/var/postgres/pg_tblspc
mkdir /usr/local/var/postgres/pg_twophase
mkdir /usr/local/var/postgres/pg_stat_tmp

install xcode, run ‘xcode select —install’ to install command tools

references

http://stackoverflow.com/questions/25970132/pg-tblspc-missing-after-installation-of-os-x-yosemite-beta/26001639#26001639 https://jimlindley.com/blog/yosemite-upgrade-homebrew-tips/

Client Form Validation in Angularjs

angularjs has built-in helper to validate form in client side, here is how you can use it.

Your html:

1
2
3
4
5
6
7
8
9
10

<form name="myForm" ng-submit="submitForm(myForm)">
  <div>
    <input type="email" ng-model="user.email" name="uEmail" required />
    <div ng-show="myForm.uEmail.$dirty && myForm.uEmail.$invalid">Invalid:
      <span ng-show="form.uEmail.$error.required">Tell us your email.</span>
      <span ng-show="form.uEmail.$error.email">This is not a valid email.</span>
    </div>
  </div>
</form>

Notice that the name in form and input fields which is used as referenced when pass form object and check whether the field is valid.

$dirty use to check whether the field value changed and $invalid to check whether the field in valid, these two methods are useful to show errors message for the fields.

Your controller:

1
2
3
4
5
6
7
8
9
angular.module('app').controller('SampleController', [

  '$scope', ($scope)->

    $scope.submitForm = (form)->
      if form.$valid
        console.log 'submit the form to server'

])

In controller, we can use the form object to check whether the form is valid before submit data to server.

There’s a lot of ways that you can valid your input fields.

  • required
  • ng-minlength
  • ng-max-length
  • ng-pattern (use regular expression for validation)
  • email ( if you specify email as type )
  • number ( if you specify number as type )
  • url ( if you specify url as type )
  • custom validation

Reference url:

http://www.ng-newsletter.com/posts/validations.html http://scotch.io/tutorials/javascript/angularjs-form-validation

Log Login Activity in Devise

In devise, there’s option Trackable to record sign in count and ip, but in some case there’s need to save custom logs in database. Turns out you can use Devise hook for that.

Just need to put this under config/initializer/devise.rb.

1
2
3
4
5
6
7
Warden::Manager.after_set_user except: :fetch do |record, warden, options|
  if record.respond_to?(:update_tracked_fields!) && warden.authenticated?(options[:scope])
    if record.is_a?(User)
      LoginLog.create(user: record, organization: record.organization, ip: warden.request.remote_ip)
    end
  end
end

Implement Pagination in Backbone

Before today I’ve written multiple times for a pagination in front end, thought it’s good time to write down in blog for future implementation.

1. Include pagination in your api for backbone collection

1
2
3
4
5
6
7
8
9
10
  def index
    customers = current_organization.customers.page(params[:page]).per(25)
    render json: {
      total_count: customers.total_count,
      total_pages: customers.total_pages,
      current_page: customers.current_page,
      per_page: 25,
      models: customers
    }
  end

2. Implemment paginated backbone collection to handle the api response.

1
2
3
4
5
6
7
8
9
10
11
  class App.Collections.PaginatedCollection extends Backbone.Collection

    parse: (resp)->
      @page = resp.current_page
      @perPage = resp.per_page
      @totalCount = resp.total_count
      @totalPages = resp.total_pages
      resp.models

    url: ->
      @baseUrl + "?" + $.param({page: @page, perPage: @perPage})

3. Implement paginated view

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
  class App.Views.Shared.PaginationView extends Backbone.View

    template: JST["backbone/templates/shared/pagination"]

    events:
      "click .page-item": "changePage"

    initialize: (opts={})->
      @collection  = opts.collection
      @currentPage = @collection.page
      @perPage     = @collection.perPage
      @totalCount  = @collection.totalCount
      @totalPages  = @collection.totalPages
      @pageSize    = @collection.length
      @calPageRange()
      @calPages()
      @calBackwardPages()
      @calForwardPages()
      @pageChanged = opts.pageChanged

    changePage: (e)->
      pageNumber = $(e.target).attr('data-page')
      if @pageChanged?
        @pageChanged(pageNumber)

    calPageRange: ->
      @pageStart = (@currentPage - 1) * @perPage + 1
      @pageEnd = ((@currentPage - 1) * @perPage) + @pageSize

    calPages: ->
      min = _.max([1, @currentPage - 2])
      max = _.min([@currentPage + 2, @totalPages])
      @pages = [min..max]

    calBackwardPages: ->
      if _.indexOf(@pages, 1) != -1
        @firstPage = null
      else
        @firstPage = 1
      prevPage = _.max([1, @pages[0] - 1])
      if (_.indexOf(@pages, prevPage) != -1) || prevPage == @firstPage
        @prevPage = null
      else
        @prevPage = prevPage

    calForwardPages: ->
      if _.indexOf(@pages, @totalPages) != -1
        @lastPage = null
      else
        @lastPage = @totalPages
      nextPage = _.min([@totalPages, _.last(@pages) + 1])
      if (_.indexOf(@pages, nextPage) != -1) || nextPage == @lastPage
        @nextPage = null
      else
        @nextPage = nextPage

    render: ->
      if @totalCount == 0
        @$el.html('')
        return
      @$el.html(@template({
        totalCount: @totalCount,
        pageStart: @pageStart,
        pageEnd: @pageEnd,
        pages: @pages,
        currentPage: @currentPage,
        firstPage: @firstPage,
        prevPage: @prevPage,
        lastPage: @lastPage,
        nextPage: @nextPage
      }))
      @

4. Implement pagination template.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
  <div class="col-sm-4">
      <small class="text-muted inline m-t-sm m-b-sm"> <%= pageStart %>-<%= pageEnd %> <%= totalCount %> </small>
  </div>
  <div class="col-sm-8 text-right text-center-xs">
      <ul class="pagination pagination-sm m-t-none m-b-none">
          <% if(firstPage != null) { %>
            <li><a class="page-item" data-page="<%= firstPage %>"><i class="fa fa-angle-double-left"></i></a></li>
          <% } %>
          <% if(prevPage != null) { %>
            <li><a class="page-item" data-page="<%= prevPage %>"><i class="fa fa-angle-left"></i></a></li>
          <% } %>
          <% for(var i=0; i < pages.length; i ++) { %>
            <% if(pages[i] == currentPage) { %>
              <li class="active"><a class="page-item" data-page="<%= pages[i] %>"><%= pages[i] %><span class="sr-only">(current)</span></a></li>
            <% } else { %>
              <li><a class="page-item" data-page="<%= pages[i] %>"><%= pages[i] %></a></li>
            <% } %>
          <% } %>
          <% if(nextPage != null) { %>
            <li><a class="page-item" data-page="<%= nextPage %>"><i class="fa fa-angle-right"></i></a></li>
          <% } %>
          <% if(lastPage != null) { %>
            <li><a class="page-item" data-page="<%= lastPage %>"><i class="fa fa-angle-double-right"></i></a></li>
          <% } %>
      </ul>
  </div>

Organize Angularjs Files Ftw

In a recent project I just started to use angularjs to develop complicated front UI, this project is not a SPA, we only use angularjs in places that ui elements is so complicated that is easier to do with front end. Since new on angularjs, i searched on the internet for best practices that organize large codebase and module naming, found couple ways.

1. Put everything in one file for simplest case.

app.js.coffee everything in one file
1
2
3
4
5
6
7
8

  angularjs
    .module('myApp', [])
    .controller('MyCtrl', ['$scope', ($scope)->
      $scope.hello = "world"
    ])
    .constant('Contacts', ['Foo', 'Bar'])
    ...

2. Put different kind of stuff in different files

app.js.coffee define app namespace
1
2
3
4

  angularjs.
    .module('myApp', [])

controllers.js.coffee define controllers
1
2
3
4
5
6
7
8

  angularjs.module('myApp')
    .controller('OneCtrl', ['$scope', ($scope)->
      $scope.hello = 'world'
    ])
    .controller('TwoCtrl', ['$scope', ($scope)->
      $scope.foo = 'bar'
    ])
constants.js.coffee define constants
1
2
3

  angularjs.module('myApp')
    .constant('Contacts', ['Foo', 'Bar'])

3. Create directories to save different kind of files

constants.js.coffee define constants
1
2
3
4
5
6
7
8
  - app.js.coffee
  controllers/
    OneCtrl.js.coffee
    TwoCtrl.js.coffee
  constants/
    Contants.js.coffee
    Contacts.js.coffee

4. Create features directories to store everything for each features

constants.js.coffee define constants
1
2
3
4
5
6
7
  - app.js.coffee
  Users
    - UserCtrl.js.coffee
    - UserModel.js.coffee
  Contacts
    - ContactsCtrl.js.coffee
    - ContactModel.js.coffee

5. How do organize code for multiple angularjs apps

constants.js.coffee define constants
1
2
3
4
5
6
7
8
9
10
11
12
13
14
  - app.js.coffee
  - modules
    AppOne.js.coffee
    AppTwo.js.coffee
  - controllers
    AppOne/
      OneCtrl.js.coffee
    AppTwo/
      TwoCtrl.js.coffee
  - models
    AppOne/
      UserModel.js.coffee
    AppTwo/
      ContactModel.js.coffee

In app.js.coffee, I define top level module to include basic dependencies share between apps.

app.js.coffee
1
  angularjs.module('app', ['dep1', 'dep2'])

In AppOne.js.coffe, it define a new module and load app as dependencies and the shared dependencies define in app will be available in AppOne.

AppOne.js.coffee
1
  angularjs.module('app.appOne', ['app'])

In html only need to specify AppOne in place that app one is need.

1
2
3
4
  <div ng-app='app.appOne'>
    <div ng-controller="OneCtrl">
    </div>
  </div>

Conculsion

I didn’t choose solution 4 because it seems not good to put different stuff together even though it’s easier to find files for changes. So I follow solution 3 and extend it for multiple apps.