Bitcoin or blockchain is crazy hot on internet today, and I’m curious what’s the best way for a programmer to get start with it. After read some books, I found that you can setup a test bitcoin testnet and play with it, so in this post I will share how it works.

Get a ubuntu server

First of all, you need to get a server or in your local environment. I’m running with latest ubuntu on a VPS server.

Install docker-ce in ubuntu

I’m using docker-ce here. To install docker-ce in ubuntu, you can follow the steps:

remove the old docker if exists

1

$ sudo apt-get remove docker docker-engine docker.io

Update apt repos.

1

$ sudo apt-get update

Install dependencies packages

1
2
3
4

$ sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
$ sudo apt-get update

Install docker-ce

1

$ sudo apt-get install docker-ce

Install bitcoin-testnet-box

Get the docker image.

1

docker pull freewil/bitcoin-testnet-box

Run bitcoin-testnet-box

1

docker run -t -i -p 19001:19001 -p 19011:19011 freewil/bitcoin-testnet-box

Then you will see something similar with:

1

tester@84074da951a2:~/bitcoin-testnet-box$

Check the info

1

make getinfo

Generate blocks

Must generate 200 blocks to get balance

1

make generate BLOCKS=200

Install ruby

I’m using ruby to call the rpc-json server, so ruby have to be installed in server first.

1

apt-get install ruby

Run ruby script to bitcoin rpc-json

I got the code sample from bitcoin.it wiki, but replace the server with the username/password/post from the configuration in bitcoin-testnet-box. Save following code outside the docker instance as test.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

require 'net/http'
require 'uri'
require 'json'

class BitcoinRPC
  def initialize(service_url)
    @uri = URI.parse(service_url)
  end

  def method_missing(name, *args)
    post_body = { 'method' => name, 'params' => args, 'id' => 'jsonrpc' }.to_json
    resp = JSON.parse( http_post_request(post_body) )
    raise JSONRPCError, resp['error'] if resp['error']
    resp['result']
  end

  def http_post_request(post_body)
    http    = Net::HTTP.new(@uri.host, @uri.port)
    request = Net::HTTP::Post.new(@uri.request_uri)
    request.basic_auth @uri.user, @uri.password
    request.content_type = 'application/json'
    request.body = post_body
    http.request(request).body
  end

  class JSONRPCError < RuntimeError; end
end

if $0 == __FILE__
  h = BitcoinRPC.new('http://admin1:123@127.0.0.1:19001')
  p h.getbalance
  p h.getinfo
  p h.getnewaddress
  p h.dumpprivkey( h.getnewaddress )
  # also see: https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_Calls_list
end

Run the code:

1

ruby test.rb

You will see something like:

1
2
3
4

5539.9999616
{"version"=>130200, "protocolversion"=>70015, "walletversion"=>130000, "balance"=>5539.9999616, "blocks"=>211, "timeoffset"=>0, "connections"=>1, "proxy"=>"","difficulty"=>4.656542373906925e-10, "testnet"=>false, "keypoololdest"=>1520155323, "keypoolsize"=>100, "paytxfee"=>0.0, "relayfee"=>1.0e-05, "errors"=>""}
"mve5QiJZmVQTGqeYNwi3xf78vrVp7eArJL"
"cRtYox2ALixpCuhL7rdx7LzCKMKaA6gCbogVkb3YgsxBAdTYZwK6"

References

• https://github.com/freewil/bitcoin-testnet-box
• https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce-1
• https://en.bitcoin.it/wiki/API_reference_(JSON-RPC))

When deal with data related project, if often need to import data from csv or excel, you can write loop to import with active record model one by one or by transaction, you can also use database command to do the bulk import, in mongodb there’s mongoimport, for postgresql you can use copy.

Here I want to import crimes data (http://data.police.uk) into postgresql, I wrote follow sql command to import:

sql = "copy crimes (crime_id, month, reported_by, falls_within, lon, lat, location, lsoa_code, lsoa_name, crime_type, last_outcome_category, context) from '/Users/rociiu/sandbox/ukpolice/uk-crimes/xx.csv' DELIMITERS ',' CSV HEADER;"

ActiveRecord::Base.connection.execute(sql) # run with active record connection

You can find more document with the link above, here I list column names following the table name to match the columns in csv, also specify the csv delimiters and tell the command that there’s header in csv.

The full rake task for importing is here: https://github.com/rociiu/uk-crimes/blob/master/lib/tasks/import.rake.

1. install dependencies

Download and install from nodejs, it include NPM for node package manage

Install yeoman, grunt, bower.

npm install -g yo grunt-cli bower

2. install ember generator

Run yo command, you will see options for execute yo commands:

yo

Use your up/down keys to select different menu/command to execute, select install generator.

❯ Install a generator

It will prompt:

? Search npm for generators:

Enter ember to search emberjs related generator. after a while it will return search results for ‘ember’:

? Here's what I found. Official generator → ෴
  Install one? (Use arrow keys)
❯ ember ෴
  ember-plugin
  rjs-ember
  ember-plus
  base

Select ember marked as ‘෴’ and press enter to install it.

3. use yo command to create the app

First you need to create a project folder, you can create with command:

mkdir myapp
yo

After run ‘yo’, it will give options to run generator, select Ember (use up/down arrow keys):

  Run a generator
  Angular
 ❯Ember
  Mocha
  Karma

Press enter, after a while it will ask whether to include Bootstrap, feel free to select Y or n. After that you’ll see something like this:

 create .gitignore
 create .gitattributes
 create .bowerrc
 create bower.json
 create package.json
 create .jshintrc
 create .editorconfig
 create Gruntfile.js
 create app/templates/application.hbs
 create app/templates/index.hbs
 create app/index.html
 create app/styles/style.scss
 create app/scripts/app.js
 create app/scripts/store.js
 create app/scripts/router.js
 create app/scripts/routes/application_route.js

  I'm all done. Running bower install & npm install for you to install the required dependencies. If this fails, try running the command yourself.
  ....

It will install all js/libraries dependencies with: bower install & npm install. Finally you will see ascii art like:

     _-----_
    |       |    .----------------------.
    |--(o)--|    |     Bye from us!     |
   `---------´   |      Chat soon.      |
    ( _´U`_ )    |                      |
    /___A___\    |      Yeoman team     |
     |  ~  |     |   http://yeoman.io   |
   __'.___.'__   '----------------------'
 ´   `  |° ´ Y `

Yeah!

4. run grunt to build the app

Run grant command to build the project.

grunt

5. run grunt serve to preview and edit the app

grunt serve

6. deploy the app

after edit and create some awesome features, you definitely want to deploy the app, right? Here I’ll show you how to deploy the app in divshot.com static web hosting for developers.

First you need to register a account in divshot.com, after register you need to install npm package for running divshot command

npm install -g divshot-cli

Then use divshot command to login your account:

divshot login

After that you can go to your project directory, run ‘divshot init’ command:

divshot init
name: (myapp) rociiu-ember-demo
root directory: (current) dist
clean urls: (y/n) y
error page: (error.html) n
Would you like to create a Divshot.io app from this app?: (y/n) y
Creating app ...

After the steps, it will create a configuration file(divshot.json) inside the root directory. Then deploy the app with ‘divshot push’:

divshot push

Creating build ...  App does not yet exist. Creating app rociiu-ember-demo ... ✔
Hashing Directory Contents ... ✔

Syncing 15 files: [==================================================] 100%


Finalizing build ... ✔
Releasing build to development ... ✔

Application deployed to development
You can view your app at: http://development.rociiu-ember-demo.divshot.io

Your app should be live now.

References:

• http://nodejs.org
• https://www.npmjs.com
• http://yeoman.io
• http://gruntjs.com
• http://emberjs.com
• http://bower.io

Refinement is introduced in ruby 2.0, no longer a experiment feature in ruby 2.1, but still not very stable. The idea of refinement is to limit the monkeying patch scope.

For example, suppose we define a refinement:

1
2
3
4
5
6
7

module ACoolFeature
  refine(String) do
    def is_it_cool?
      puts "YES"
    end
  end
end

Here we define a refinement to String class, to use the refinement we need to call using inside a class.

1
2
3
4
5
6
7

class Person
  using(ACoolFeature)

  def hello
    "test".is_it_cool?
  end
end

After call using in Person class, the method is_it_cool? is available for string inside this scope.

1
2
3

Person.new.hello
>     Person.new.hello
YES

That’s it.

what is d3js

(http://d3js.org)[d3js] is very popular in visualization recently, it has unique way to render chart compare to other libraries. d3 refer to data-driven document.

setup html

<html>
  <head>
    <title>D3</title>
    <script src="http://d3js.org/d3.v3.min.js" charset="utf-8"></script>
  </head>
  <body>
    <h2>D3js example</h2>
    <svg width=500 height=300>
    </svg>
    <script>
    </script>
  </body>
</html>

use data in previous post

var crimesData = [
{
    "Crime type" : "Anti-social behaviour",
    "total" : 5593
},
{
    "Crime type" : "Burglary",
    "total" : 888
},
{
    "Crime type" : "Criminal damage and arson",
    "total" : 1215
},
{
    "Crime type" : "Other theft",
    "total" : 1343
},
{
    "Crime type" : "Violence and sexual offences",
    "total" : 2123
},
{
    "Crime type" : "Drugs",
    "total" : 292
},
{
    "Crime type" : "Public order",
    "total" : 577
},
{
    "Crime type" : "Shoplifting",
    "total" : 1019
},
{
    "Crime type" : "Vehicle crime",
    "total" : 689
},
{
    "Crime type" : "Possession of weapons",
    "total" : 22
},
{
    "Crime type" : "Bicycle theft",
    "total" : 271
},
{
    "Crime type" : "Other crime",
    "total" : 88
},
{
    "Crime type" : "Theft from the person",
    "total" : 172
},
{
    "Crime type" : "Robbery",
    "total" : 91
}
]

write code to render bar chart

  var extent = d3.extent(crimesData, function(d){return d.total})
  var widthScale = d3.scale.linear().domain(extent).range([0, 500])
  var colorScale = d3.scale.category10()

  d3.select("svg").selectAll("rect")
    .data(crimesData)
  .enter()
    .append("rect")
    .attr("x", function(d, i){ return 0 })
    .attr("y", function(d, i){ return i * 30})
    .attr("width", function(d, i){ return widthScale(d.total)})
    .attr("height", function(d, i){ return 25})
    .attr("fill", function(d, i){ return colorScale(i)})

get min/max value with d3.exten

var extent = d3.extent(crimesData, function(d){return d.total})

add scale to map domain value to width in chart

var widthScale = d3.scale.linear().domain(extent).range([0, 500])

add color scale to fill different bar with colors

var colorScale = d3.scale.category10()

select element and bind data

d3.select("svg").selectAll("rect")
  .data(crimesData)
.enter()
  .append("rect")
  .attr("x", function(d, i){ return 0 })
  .attr("y", function(d, i){ return i * 30})
  .attr("width", function(d, i){ return widthScale(d.total)})
  .attr("height", function(d, i){ return 25})
  .attr("fill", function(d, i){ return colorScale(i)})

Here I first select the svg in the html, then select ‘rect’ in this svg and bind crimes data with those rect, enter function is used to start render the element that with data binding, first I append a new rect in the svg, assign different attributes with scales defined before, notice that the function as attr second parameter as two parameter d and i, d represent each item in the crimes data and i of cource is the index of the item in crimes data.

Final result

http://codepen.io/anon/pen/IwlDA

Noted that we still missing a lot of stuff in this chart like x/y axis, labels though, d3js offer functions/generators for that.

If you understand how svg works, you’ll know that d3js just help us to render svg elements with helper functions.

I have been worked on some data visualization projects in last few years for different kind of clients, used some different set of library/framework like backbonejs, raphaeljs, d3js. Start from this post, i’ll try to write down process on how we can do data visualization with modern technologies.

The data source will be http://data.police.uk , which include different crimes records in UK.

Install mongodb

Mongodb has many features like document, schemaless, query, etc, the reason i choose it is it’s very easy to load the data into it using mongoimport.

brew install mongodb

Start mongodb

mongod

Create database

use ukcrimes

Download data from http://data.police.uk/data/

Import data into database

mongoimport --db ukpolice --collection crimes --type csv --headerline --file file_path

Check imported data

mongo
use ukcrimes
db.crimes.find()  

You’ll see data like:

{ "_id" : ObjectId("544a7172d0e2b18dc73d060c"), "Crime ID" : "ca8de697ca1a5b0b1f76a3c0f3f652bf2e9e20a31fbf5af40df5de7fa55f63a4", "Month" : "2014-07", "Reported by" : "Avon and Somerset Constabulary", "Falls within" : "Avon and Somerset Constabulary", "Longitude" : "", "Latitude" : "", "Location" : "No location", "LSOA code" : "", "LSOA name" : "", "Outcome type" : "Offender given penalty notice" }
{ "_id" : ObjectId("544a7172d0e2b18dc73d060d"), "Crime ID" : "c50470f2a1af2a5fffaa3dd69218777200c1a9289b87dd36849325ed2585472e", "Month" : "2014-07", "Reported by" : "Avon and Somerset Constabulary", "Falls within" : "Avon and Somerset Constabulary", "Longitude" : "", "Latitude" : "", "Location" : "No location", "LSOA code" : "", "LSOA name" : "", "Outcome type" : "Offender given a drugs possession warning" }
{ "_id" : ObjectId("544a7172d0e2b18dc73d060e"), "Crime ID" : "b7458f39eb58018af53c85fd481aa2028e247baa30ceb1f095437359e78b63f6", "Month" : "2014-07", "Reported by" : "Avon and Somerset Constabulary", "Falls within" : "Avon and Somerset Constabulary", "Longitude" : "", "Latitude" : "", "Location" : "No location", "LSOA code" : "", "LSOA name" : "", "Outcome type" : "Suspect charged" }
{ "_id" : ObjectId("544a7172d0e2b18dc73d060f"), "Crime ID" : "3093f07675cd62ce39cc8eeb80261edb15d00cbee2bf2ece8d295ac6a0efab4e", "Month" : "2014-07", "Reported by" : "Avon and Somerset Constabulary", "Falls within" : "Avon and Somerset Constabulary", "Longitude" : "", "Latitude" : "", "Location" : "No location", "LSOA code" : "", "LSOA name" : "", "Outcome type" : "Suspect charged" }

To understand the data columns, you can check http://data.police.uk/about/

Query crimes stats

db.crimes.group({key: { 'Crime type': 1}, reduce: function(curr, result){ result.total += 1}, initial: { total: 0 } })

result:

[
{
    "Crime type" : "Anti-social behaviour",
    "total" : 5593
},
{
    "Crime type" : "Burglary",
    "total" : 888
},
{
    "Crime type" : "Criminal damage and arson",
    "total" : 1215
},
{
    "Crime type" : "Other theft",
    "total" : 1343
},
{
    "Crime type" : "Violence and sexual offences",
    "total" : 2123
},
{
    "Crime type" : "Drugs",
    "total" : 292
},
{
    "Crime type" : "Public order",
    "total" : 577
},
{
    "Crime type" : "Shoplifting",
    "total" : 1019
},
{
    "Crime type" : "Vehicle crime",
    "total" : 689
},
{
    "Crime type" : "Possession of weapons",
    "total" : 22
},
{
    "Crime type" : "Bicycle theft",
    "total" : 271
},
{
    "Crime type" : "Other crime",
    "total" : 88
},
{
    "Crime type" : "Theft from the person",
    "total" : 172
},
{
    "Crime type" : "Robbery",
    "total" : 91
}
]

Fix brew command

In Yosemite the ruby command path changed, yosemite use ruby 2.0 as default also create a new path to link to ruby 2.0:

/System/Library/Frameworks/Ruby.framework/Versions/Current/usr/bin/ruby

You need to change ruby path in the first line of /usr/local/Library/brew.rb

Update brew

After fix the brew command, it’s better to update brew itself to avoid other issues

brew update

Fix postgresql

After upgrade, you’ll get following error when try to run: postgres -D /usr/local/var/postgresql

FATAL:  could not open directory "pg_tblspc": No such file or directory

the fix is easy, just create those directories, those directories are gone after upgrade.

mkdir /usr/local/var/postgres/pg_tblspc
mkdir /usr/local/var/postgres/pg_twophase
mkdir /usr/local/var/postgres/pg_stat_tmp

install xcode, run ‘xcode select —install’ to install command tools

references

http://stackoverflow.com/questions/25970132/pg-tblspc-missing-after-installation-of-os-x-yosemite-beta/26001639#26001639 https://jimlindley.com/blog/yosemite-upgrade-homebrew-tips/

angularjs has built-in helper to validate form in client side, here is how you can use it.

Your html:

1
2
3
4
5
6
7
8
9
10

<form name="myForm" ng-submit="submitForm(myForm)">
  <div>
    <input type="email" ng-model="user.email" name="uEmail" required />
    <div ng-show="myForm.uEmail.$dirty && myForm.uEmail.$invalid">Invalid:
      <span ng-show="form.uEmail.$error.required">Tell us your email.</span>
      <span ng-show="form.uEmail.$error.email">This is not a valid email.</span>
    </div>
  </div>
</form>

Notice that the name in form and input fields which is used as referenced when pass form object and check whether the field is valid.

$dirty use to check whether the field value changed and $invalid to check whether the field in valid, these two methods are useful to show errors message for the fields.

Your controller:

1
2
3
4
5
6
7
8
9

angular.module('app').controller('SampleController', [

  '$scope', ($scope)->

    $scope.submitForm = (form)->
      if form.$valid
        console.log 'submit the form to server'

])

In controller, we can use the form object to check whether the form is valid before submit data to server.

There’s a lot of ways that you can valid your input fields.

• required
• ng-minlength
• ng-max-length
• ng-pattern (use regular expression for validation)
• email ( if you specify email as type )
• number ( if you specify number as type )
• url ( if you specify url as type )
• custom validation

Reference url:

http://www.ng-newsletter.com/posts/validations.html http://scotch.io/tutorials/javascript/angularjs-form-validation

In devise, there’s option Trackable to record sign in count and ip, but in some case there’s need to save custom logs in database. Turns out you can use Devise hook for that.

Just need to put this under config/initializer/devise.rb.

1
2
3
4
5
6
7

Warden::Manager.after_set_user except: :fetch do |record, warden, options|
  if record.respond_to?(:update_tracked_fields!) && warden.authenticated?(options[:scope])
    if record.is_a?(User)
      LoginLog.create(user: record, organization: record.organization, ip: warden.request.remote_ip)
    end
  end
end

Before today I’ve written multiple times for a pagination in front end, thought it’s good time to write down in blog for future implementation.

1. Include pagination in your api for backbone collection

1
2
3
4
5
6
7
8
9
10

  def index
    customers = current_organization.customers.page(params[:page]).per(25)
    render json: {
      total_count: customers.total_count,
      total_pages: customers.total_pages,
      current_page: customers.current_page,
      per_page: 25,
      models: customers
    }
  end

2. Implemment paginated backbone collection to handle the api response.

1
2
3
4
5
6
7
8
9
10
11

  class App.Collections.PaginatedCollection extends Backbone.Collection

    parse: (resp)->
      @page = resp.current_page
      @perPage = resp.per_page
      @totalCount = resp.total_count
      @totalPages = resp.total_pages
      resp.models

    url: ->
      @baseUrl + "?" + $.param({page: @page, perPage: @perPage})

3. Implement paginated view

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

  class App.Views.Shared.PaginationView extends Backbone.View

    template: JST["backbone/templates/shared/pagination"]

    events:
      "click .page-item": "changePage"

    initialize: (opts={})->
      @collection  = opts.collection
      @currentPage = @collection.page
      @perPage     = @collection.perPage
      @totalCount  = @collection.totalCount
      @totalPages  = @collection.totalPages
      @pageSize    = @collection.length
      @calPageRange()
      @calPages()
      @calBackwardPages()
      @calForwardPages()
      @pageChanged = opts.pageChanged

    changePage: (e)->
      pageNumber = $(e.target).attr('data-page')
      if @pageChanged?
        @pageChanged(pageNumber)

    calPageRange: ->
      @pageStart = (@currentPage - 1) * @perPage + 1
      @pageEnd = ((@currentPage - 1) * @perPage) + @pageSize

    calPages: ->
      min = _.max([1, @currentPage - 2])
      max = _.min([@currentPage + 2, @totalPages])
      @pages = [min..max]

    calBackwardPages: ->
      if _.indexOf(@pages, 1) != -1
        @firstPage = null
      else
        @firstPage = 1
      prevPage = _.max([1, @pages[0] - 1])
      if (_.indexOf(@pages, prevPage) != -1) || prevPage == @firstPage
        @prevPage = null
      else
        @prevPage = prevPage

    calForwardPages: ->
      if _.indexOf(@pages, @totalPages) != -1
        @lastPage = null
      else
        @lastPage = @totalPages
      nextPage = _.min([@totalPages, _.last(@pages) + 1])
      if (_.indexOf(@pages, nextPage) != -1) || nextPage == @lastPage
        @nextPage = null
      else
        @nextPage = nextPage

    render: ->
      if @totalCount == 0
        @$el.html('')
        return
      @$el.html(@template({
        totalCount: @totalCount,
        pageStart: @pageStart,
        pageEnd: @pageEnd,
        pages: @pages,
        currentPage: @currentPage,
        firstPage: @firstPage,
        prevPage: @prevPage,
        lastPage: @lastPage,
        nextPage: @nextPage
      }))
      @

4. Implement pagination template.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

  <div class="col-sm-4">
      <small class="text-muted inline m-t-sm m-b-sm">第 <%= pageStart %>-<%= pageEnd %>条 共<%= totalCount %> 条</small>
  </div>
  <div class="col-sm-8 text-right text-center-xs">
      <ul class="pagination pagination-sm m-t-none m-b-none">
          <% if(firstPage != null) { %>
            <li><a class="page-item" data-page="<%= firstPage %>"><i class="fa fa-angle-double-left"></i></a></li>
          <% } %>
          <% if(prevPage != null) { %>
            <li><a class="page-item" data-page="<%= prevPage %>"><i class="fa fa-angle-left"></i></a></li>
          <% } %>
          <% for(var i=0; i < pages.length; i ++) { %>
            <% if(pages[i] == currentPage) { %>
              <li class="active"><a class="page-item" data-page="<%= pages[i] %>"><%= pages[i] %><span class="sr-only">(current)</span></a></li>
            <% } else { %>
              <li><a class="page-item" data-page="<%= pages[i] %>"><%= pages[i] %></a></li>
            <% } %>
          <% } %>
          <% if(nextPage != null) { %>
            <li><a class="page-item" data-page="<%= nextPage %>"><i class="fa fa-angle-right"></i></a></li>
          <% } %>
          <% if(lastPage != null) { %>
            <li><a class="page-item" data-page="<%= lastPage %>"><i class="fa fa-angle-double-right"></i></a></li>
          <% } %>
      </ul>
  </div>