Jupyter: JUlia PYThon and R

it's "ggplot2", not "ggplot", but it is ggplot()

 

Did you know that @projectJupyter's Jupyter Notebook (and JupyterLab) name came from combining 3 programming languages: JUlia, PYThon and R.

Readers of my blog do not need an introduction to Python. But what about the other 2?  

Today we will talk about R. Actually, R and Python, on the Raspberry Pi.

R Origin

R traces its origins to the S statistical programming language, developed in the 1970s at Bell Labs by John M. Chambers. He is also the author of books such as Computational Methods for Data Analysis (1977) and Graphical Methods for Data Analysis (1983). R is an open source implementation of that statistical language. It is compatible with S but also has enhancements over the original.

 

A quick getting started guide is available here: https://support.rstudio.com/hc/en-us/sections/200271437-Getting-Started

 

Installing Python

As a recap, in case you don't have Python 3 and a few basic modules, the installation goes as follow (open a terminal window first):

pi@raspberrypi: $ sudo apt install python3 python3-dev build-essential

pi@raspberrypi: $ sudo pip3 install jedi pandas numpy

Installing R

Installing R is equally easy:

 

pi@raspberrypi: $ sudo apt install r-recommended

 

We also need to install a few development packages:

pi@raspberrypi: $ sudo apt install libffi-dev libcurl4-openssl-dev libxml2-dev

This will allow us to install many packages in R. Now that R is installed, we can start it:

pi@raspberrypi: $ R

Installing packages

Once inside R, we can install packages using install.packages('name') where name is the name of the package. For example, to install ggplot2 (to install tidyverse, simply replace ggplot2 with tidyverse):

> install.packages('ggplot2')

To load it:

> library(ggplot2)

And we can now use it. We will use the mpg dataset and plot displacement vs highway miles per gallon and set the color to:

>ggplot(mpg, aes(displ, hwy, colour=class))+

 geom_point()

Combining R and Python

We can go at this 2 ways, from Python call R, or from R call Python. Here, from R we will call Python.

First, we need to install reticulate (the package that interfaces with Python):

> install.packages('reticulate')

And load it:

> library(reticulate)

We can verify which python binary that reticulate is using:

> py_config()

 Then we can use it to execute some python code. For example, to import the os module and use os.listdir(), from R we do ($ works a bit in a similar fashion to Python's .):

> os <- import("os")

> os$listdir(".")

Or even enter a Python REPL:

> repl_python()
>>> import pandas as pd

>>>

Type exit to leave the Python REPL.

One more trick: Radian

we will now exit R (quit()) and install radian, a command line REPL for R that is fully aware of the reticulate and Python integration:

pi@raspberrypi: $ sudo pip3 install radian

pi@raspberrypi: $ radian

This is just like the R REPL, only better. And you can switch to python very quickly by typing ~:

r$> ~

As soon as the ~ is typed, radian enters the python mode by itself:

r$> reticulate::repl_python()

>>> 

Hitting backspace at the beginning of the line switches back to the R REPL:

r$> 

I'll cover more functionality in a future post.

Francois Dion

@f_dion

Raspberry Pi 3 Canakit

It's been a long time...

I don't even remember when was the last time I talked about Raspberry Pi hardware on my blog. I do remember the first time, however, some 5 years ago. Meanwhile, the price of Raspberry Pis have both gone down (Zero) and up (Raspberry Pi 3) due to economies of scale, Moore's law and removal / addition of components from the various versions.

Raspberry Pi 3

So, I realize that I've covered everything from the original Raspberry Pi model B with 256MB of ram, all the way to the Raspberry Pi 2 and Zero, but nothing on the Raspberry Pi 3. I typically don't buy kits, but I picked one up to see how that would work for people who had never used a Raspberry Pi before. The reason there is that I suggested to my fellow data scientists (and those interested by data science, and whoever else who reads my posts) to also be technologists, to get better acquainted with technology and hardware. And to get one (or four - I'll follow up on that) Raspberry Pi 3. I wanted to make sure it would be smooth sailing.

The Canakit

I ordered a Canakit with a Raspberry Pi board and two heatsinks, a case, a 2.5A power supply and a guide. It is convenient to get a single ready to go package quickly, particularly with free 2nd day shipping (you know who).

I say ready to go, but not quite. Of course you'll need a monitor, keyboard and mouse if you don't enable ssh (distro dependent ways, some have ssh enabled by default). But you'll also need an SD card.

No SD card?

The kit I ordered basically got you the case, heatsinks and power supply for $15 above the price of the Raspberry Pi 3. But no microSD card. The step above kit includes a microSD card with the OS already pre-installed, but when I looked at it, the price difference was too much for just that and a plain HDMI cable.

Oh, but it's so hard to prepare an SD card! Don't panic. Use Etcher. Later, when you get your command line skills up, you'll be able to do it with dd.

If you get stuck, leave a comment or contact me on twitter

Francois Dion
@f_dion

Something For Your Mind, Polymath Podcast episode 3

"Improving your communications and Entrepreneurship"

In this episode, , we will cover these two different topics. Specifically, we will talk about kinematic displays, improving your communications, coworking spaces, entrepreneurship, innovation and more. This episode will wrap up with a learn more section, on what is an entrepreneur.

"these are people that can't help but generate ideas all the time..."

A few selected links are on the episode web page, and includes a few items related to colors (some for all platforms, others are Linux only)

Something for your mind is available on

art·chiv.es

/'ärt,kīv/

at artchiv.es/s4ym/

Francois Dion
@f_dion

And thus ends 2015...

Yet it is also just the beginning

This is not going a long review of the year. Perhaps in January I'll do that. But I did want to point out that it was a good year for python. Earlier this month I looked at the TIOBE ratings for Python, R and Scala, the main languages I use on a regular basis (and in decreasing order of use by me - I might do java, C++ or javascript on occasion, but not on a regular basis anymore):

Dec. 2015 TIOBE index: https://t.co/NRQ13Pi00O My current primary languages #Python at #4, #Rstats at #18 and #Scala at #28.

— Francois Dion (@f_dion) December 9, 2015

And that was a peak for Python at #4. Back in 2007, you might remember, TIOBE had named Python "Language of the year". And if we do a quick check on google trends of a good indicator of the popularity worldwide ("learn python"), we see that this is when it started to pick up some steam. For some fun, I'm comparing to "learn java" (ranked #1 on latest TIOBE rating):

Hey wait, what is going on in November / December 2015? :P

Let's zoom in and take a closer look at 2015:

It appears it might be overtaking Java there... It is quite early to really see if this is just a fluke, only the next few months will reveal this.

That credit card sized computer thingy

What is also worth mentioning is that the level of interest in learning Python and the Raspberry Pi seem to follow a similar path, but that will be for a follow up post. See you next year!

Francois Dion
@f_dion

The Star Wars star ships

How fast are they?

Added full size version, just click on the above

I posted the above on linkedin earlier this month. I hinted at the code in the header picture, but no code. Ok, so let's get into some code here.

WARNING: Once you discover this API, you are guaranteed to wastespend a lot of time with it. It's not too late to turn around, you've been warned!

REST API

The main thing I want to point out tonight, is how easy it is to interact with web services in python. In this case, the Star Wars API (swapi). First, import the usual suspects for visualization and analytics, and add some json handling:

In [1]:

%matplotlib inline

import matplotlib.patches as mpatches

import matplotlib.pyplot as plt

import numpy as np

import pandas as pd

import seaborn as sns

In [2]:

import requests

import json

from pandas.io.json import json_normalize

Ok, what next? Let's first check the swapi api:

In [4]:

r = requests.get('http://swapi.co/api/')

urls = r.json()

urls

Out[4]:

{'films': 'http://swapi.co/api/films/',
 'people': 'http://swapi.co/api/people/',
 'planets': 'http://swapi.co/api/planets/',
 'species': 'http://swapi.co/api/species/',
 'starships': 'http://swapi.co/api/starships/',
 'vehicles': 'http://swapi.co/api/vehicles/'}

Each service will spit back out only part of the data, so we need to follow the link to the next page. Let's write up a function, something quick as a helper. It is not the most efficient, but it is the most readable way of doing this, and for a blog that's important.

In [5]:

def get_swapi(url):

    r = requests.get(url)

    data = r.json()

    df = json_normalize(data['results'])

    while len(df.index) < data['count']:

        r = requests.get(data['next'])

        data = r.json()

        df = pd.concat([df,json_normalize(data['results'])])

    return df

Chewie, we're home

And now to use it:

In [6]:

df = get_swapi(urls['starships'])

Finally, let's clean up the data to correct the 1000km into 1000, remove the unknowns etc, then we can display the full table.

In [7]:

df['max_atmosphering_speed'][df['max_atmosphering_speed']=='1000km'] = 1000

df = df[~(df['hyperdrive_rating']=='unknown')]

df = df[~(df['max_atmosphering_speed'].isin(['unknown','n/a']))]

In [8]:

df['max_atmosphering_speed'] = df['max_atmosphering_speed'].astype(int)

In [9]:

df['hyperdrive_rating'] = df['hyperdrive_rating'].astype(float)

df.sort_values(by='hyperdrive_rating', inplace=True)

df

Out[9]:

	MGLT	cargo_capacity	consumables	cost_in_credits	created	crew	edited	films	hyperdrive_rating	length	manufacturer	max_atmosphering_speed	model	name	passengers	pilots	starship_class	url
2	75	100000	2 months	100000	2014-12-10T16:59:45.094000Z	4	2014-12-22T17:35:44.464156Z	[http://swapi.co/api/films/7/, http://swapi.co...	0.5	34.37	Corellian Engineering Corporation	1050	YT-1300 light freighter	Millennium Falcon	6	[http://swapi.co/api/people/13/, http://swapi....	Light freighter	http://swapi.co/api/starships/10/
0	unknown	unknown	unknown	unknown	2014-12-20T19:55:15.396000Z	3	2014-12-22T17:35:45.258859Z	[http://swapi.co/api/films/6/]	0.5	29.2	Theed Palace Space Vessel Engineering Corps/Nu...	1050	J-type star skiff	Naboo star skiff	3	[http://swapi.co/api/people/10/, http://swapi....	yacht	http://swapi.co/api/starships/64/
7	unknown	unknown	1 year	2000000	2014-12-20T11:05:51.237000Z	5	2014-12-22T17:35:45.124386Z	[http://swapi.co/api/films/5/]	0.7	39	Theed Palace Space Vessel Engineering Corps, N...	2000	J-type diplomatic barge	J-type diplomatic barge	10	[]	Diplomatic barge	http://swapi.co/api/starships/43/
0	unknown	unknown	unknown	unknown	2014-12-20T17:46:46.847000Z	4	2014-12-22T17:35:45.158969Z	[http://swapi.co/api/films/5/]	0.9	47.9	Theed Palace Space Vessel Engineering Corps	8000	H-type Nubian yacht	H-type Nubian yacht	unknown	[http://swapi.co/api/people/35/]	yacht	http://swapi.co/api/starships/49/
2	100	110	5 days	155000	2014-12-20T20:03:48.603000Z	3	2014-12-22T17:35:45.287214Z	[http://swapi.co/api/films/6/]	1.0	14.5	Incom Corporation, Subpro Corporation	1000	Aggressive Reconnaissance-170 starfighte	arc-170	0	[]	starfighter	http://swapi.co/api/starships/66/
1	unknown	60	2 days	320000	2014-12-20T19:56:57.468000Z	1	2014-12-22T17:35:45.272349Z	[http://swapi.co/api/films/6/]	1.0	5.47	Kuat Systems Engineering	1500	Eta-2 Actis-class light interceptor	Jedi Interceptor	0	[http://swapi.co/api/people/10/, http://swapi....	starfighter	http://swapi.co/api/starships/65/
9	unknown	20000000	2 years	59000000	2014-12-20T19:52:56.232000Z	7400	2014-12-22T17:35:45.224540Z	[http://swapi.co/api/films/6/]	1.0	1137	Kuat Drive Yards, Allanteen Six shipyards	975	Senator-class Star Destroyer	Republic attack cruiser	2000	[]	star destroyer	http://swapi.co/api/starships/63/
3	unknown	50000	56 days	1000000	2014-12-20T19:48:40.409000Z	5	2014-12-22T17:35:45.208584Z	[http://swapi.co/api/films/6/]	1.0	18.5	Cygnus Spaceworks	2000	Theta-class T-2c shuttle	Theta-class T-2c shuttle	16	[]	transport	http://swapi.co/api/starships/61/
9	unknown	60	7 days	180000	2014-12-20T17:35:23.906000Z	1	2014-12-22T17:35:45.147746Z	[http://swapi.co/api/films/5/, http://swapi.co...	1.0	8	Kuat Systems Engineering	1150	Delta-7 Aethersprite-class interceptor	Jedi starfighter	0	[http://swapi.co/api/people/10/, http://swapi....	Starfighter	http://swapi.co/api/starships/48/
5	unknown	60	15 hours	102500	2014-12-20T20:43:04.349000Z	1	2014-12-22T17:35:45.396711Z	[http://swapi.co/api/films/6/]	1.0	7.9	Kuat Systems Engineering	1050	Alpha-3 Nimbus-class V-wing starfighter	V-wing	0	[]	starfighter	http://swapi.co/api/starships/75/
4	unknown	65	7 days	200000	2014-12-19T17:39:17.582000Z	1	2014-12-22T17:35:45.079452Z	[http://swapi.co/api/films/5/, http://swapi.co...	1.0	11	Theed Palace Space Vessel Engineering Corps	1100	N-1 starfighter	Naboo fighter	0	[http://swapi.co/api/people/11/, http://swapi....	Starfighter	http://swapi.co/api/starships/39/
0	70	180000	1 month	240000	2014-12-10T15:48:00.586000Z	5	2014-12-22T17:35:44.431407Z	[http://swapi.co/api/films/1/]	1.0	38	Sienar Fleet Systems, Cyngus Spaceworks	1000	Sentinel-class landing craft	Sentinel-class landing craft	75	[]	landing craft	http://swapi.co/api/starships/5/
3	80	110	1 week	134999	2014-12-12T11:00:39.817000Z	2	2014-12-22T17:35:44.479706Z	[http://swapi.co/api/films/3/, http://swapi.co...	1.0	14	Koensayr Manufacturing	1000	BTL Y-wing	Y-wing	0	[]	assault starfighter	http://swapi.co/api/starships/11/
1	120	40	1 week	175000	2014-12-18T11:16:34.542000Z	1	2014-12-22T17:35:44.978754Z	[http://swapi.co/api/films/3/]	1.0	9.6	Alliance Underground Engineering, Incom Corpor...	1300	RZ-1 A-wing Interceptor	A-wing	0	[http://swapi.co/api/people/29/]	Starfighter	http://swapi.co/api/starships/28/
4	100	110	1 week	149999	2014-12-12T11:19:05.340000Z	1	2014-12-22T17:35:44.491233Z	[http://swapi.co/api/films/3/, http://swapi.co...	1.0	12.5	Incom Corporation	1050	T-65 X-wing	X-wing	0	[http://swapi.co/api/people/1/, http://swapi.c...	Starfighter	http://swapi.co/api/starships/12/
5	105	150	5 days	unknown	2014-12-12T11:21:32.991000Z	1	2014-12-22T17:35:44.549047Z	[http://swapi.co/api/films/1/]	1.0	9.2	Sienar Fleet Systems	1200	Twin Ion Engine Advanced x1	TIE Advanced x1	0	[http://swapi.co/api/people/4/]	Starfighter	http://swapi.co/api/starships/13/
8	50	80000	2 months	240000	2014-12-15T13:04:47.235000Z	6	2014-12-22T17:35:44.795405Z	[http://swapi.co/api/films/3/, http://swapi.co...	1.0	20	Sienar Fleet Systems	850	Lambda-class T-4a shuttle	Imperial shuttle	20	[http://swapi.co/api/people/1/, http://swapi.c...	Armed government transport	http://swapi.co/api/starships/22/
6	unknown	2500000	30 days	55000000	2014-12-20T09:39:56.116000Z	1	2014-12-22T17:35:45.105522Z	[http://swapi.co/api/films/4/]	1.5	26.5	Republic Sienar Systems	1180	Star Courier	Scimitar	6	[http://swapi.co/api/people/44/]	Space Transport	http://swapi.co/api/starships/41/
2	unknown	50000000	4 years	125000000	2014-12-20T19:40:21.902000Z	600	2014-12-22T17:35:45.195165Z	[http://swapi.co/api/films/6/]	1.5	1088	Rendili StarDrive, Free Dac Volunteers Enginee...	1050	Providence-class carrier/destroyer	Trade Federation cruiser	48247	[http://swapi.co/api/people/10/, http://swapi....	capital ship	http://swapi.co/api/starships/59/
8	unknown	240	7 days	35700	2014-12-20T18:37:56.969000Z	3	2014-12-22T17:35:45.183075Z	[http://swapi.co/api/films/5/]	1.5	15.2	Huppla Pasa Tisc Shipwrights Collective	1600	Punworcca 116-class interstellar sloop	Solar Sailer	11	[]	yacht	http://swapi.co/api/starships/58/
5	unknown	unknown	unknown	unknown	2014-12-19T17:45:03.506000Z	8	2014-12-22T17:35:45.091925Z	[http://swapi.co/api/films/4/]	1.8	76	Theed Palace Space Vessel Engineering Corps, N...	920	J-type 327 Nubian royal starship	Naboo Royal Starship	unknown	[http://swapi.co/api/people/39/]	yacht	http://swapi.co/api/starships/40/
6	60	3000000	1 year	3500000	2014-12-10T14:20:33.369000Z	165	2014-12-22T17:35:45.408368Z	[http://swapi.co/api/films/6/, http://swapi.co...	2.0	150	Corellian Engineering Corporation	950	CR90 corvette	CR90 corvette	600	[]	corvette	http://swapi.co/api/starships/2/
1	60	36000000	2 years	150000000	2014-12-10T15:08:19.848000Z	47060	2014-12-22T17:35:44.410941Z	[http://swapi.co/api/films/3/, http://swapi.co...	2.0	1,600	Kuat Drive Yards	975	Imperial I-class Star Destroyer	Star Destroyer	0	[]	Star Destroyer	http://swapi.co/api/starships/3/
9	40	6000000	2 years	8500000	2014-12-15T13:06:30.813000Z	854	2014-12-22T17:35:44.848329Z	[http://swapi.co/api/films/3/, http://swapi.co...	2.0	300	Kuat Drive Yards	800	EF76 Nebulon-B escort frigate	EF76 Nebulon-B escort frigate	75	[]	Escort ship	http://swapi.co/api/starships/23/
2	91	45	1 week	220000	2014-12-18T11:18:04.763000Z	1	2014-12-22T17:35:45.011193Z	[http://swapi.co/api/films/3/]	2.0	16.9	Slayn & Korpil	950	A/SF-01 B-wing starfighter	B-wing	0	[]	Assault Starfighter	http://swapi.co/api/starships/29/
3	unknown	unknown	unknown	unknown	2014-12-19T17:01:31.488000Z	9	2014-12-22T17:35:45.027308Z	[http://swapi.co/api/films/4/]	2.0	115	Corellian Engineering Corporation	900	Consular-class cruiser	Republic Cruiser	16	[]	Space cruiser	http://swapi.co/api/starships/31/
7	70	70000	1 month	unknown	2014-12-15T13:00:56.332000Z	1	2014-12-22T17:35:44.716273Z	[http://swapi.co/api/films/5/, http://swapi.co...	3.0	21.5	Kuat Systems Engineering	1000	Firespray-31-class patrol and attack	Slave 1	6	[http://swapi.co/api/people/22/]	Patrol craft	http://swapi.co/api/starships/21/
5	20	19000000	6 months	unknown	2014-12-15T12:34:52.264000Z	6	2014-12-22T17:35:44.680838Z	[http://swapi.co/api/films/3/, http://swapi.co...	4.0	90	Gallofree Yards, Inc.	650	GR-75 medium transport	Rebel transport	90	[]	Medium transport	http://swapi.co/api/starships/17/
4	unknown	140	7 days	168000	2014-12-20T20:38:05.031000Z	1	2014-12-22T17:35:45.381900Z	[http://swapi.co/api/films/6/]	6.0	6.71	Feethan Ottraw Scalable Assemblies	1100	Belbullab-22 starfighter	Belbullab-22 starfighter	0	[http://swapi.co/api/people/10/, http://swapi....	starfighter	http://swapi.co/api/starships/74/

That'll be it for tonight, enjoy playing around with swapi, but as the intro mentions, be warned (and it is the disclaimer I've used each time I've introduced the api into a data science class), you can spend a lot of time with this...

Francois Dion
@f_dion

The 10 colors of Pi

Picking up patterns

Or the lack thereof. The brain is pretty good at spotting patterns and anomalies. But we have to help it with something that can be easily abstracted. Numbers are not good for that.

Of reds and greens and blues

Colors are good helpers. Shades, hues. Unfortunately, many people are affected by colorblindness. It is said that in some segments of the population, up to 8% of men and 0.4% of women experience congenital color deficiency, with the most common being red-green color blindness.

In [1]:

%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle
import seaborn as sns
sns.set_context("talk")

Seaborn palettes

According to Seaborn's documentation (http://web.stanford.edu/~mwaskom/software/seaborn/tutorial/color_palettes.html), there is a ready made color palette called colorblind. There is not a lot of details on this, so I thought I'd experiment with this and ask for feedback. Unfortunately, only 6 colors are available until it starts recycling itself. That is clearly not good if we want to see patterns in numbers that range from 0 to 9 on each digit.
Another interesting choice is cubehelix. It works in color and grayscale. You can learn more about it here: http://www.mrao.cam.ac.uk/~dag/CUBEHELIX/
First thing first, let's set it as the default palette, and display it.

In [2]:

sns.set_palette(sns.color_palette("cubehelix", 10))
sns.palplot(sns.color_palette())

An infinite source of entertainment

At the very least, an infinite source of digits: pi
So we will be plotting a large grid, with each square representing a digit of pi and filled in the color corresponding to the color in the Seaborn palette. Let's grab a pi digit generator. There is one here that's been around since the days of python 2.5:
https://www.daniweb.com/programming/software-development/code/249177/pi-generator-update

In [3]:

def pi_generate():
    """
    generator to approximate pi
    returns a single digit of pi each time iterated
    """
    q, r, t, k, m, x = 1, 0, 1, 1, 3, 3
    while True:
        if 4 * q + r - t < m * t:
            yield m
            q, r, t, k, m, x = 10*q, 10*(r-m*t), t, k, (10*(3*q+r))//t - 10*m, x
        else:
            q, r, t, k, m, x = q*k, (2*q+r)*x, t*x, k+1, (q*(7*k+2)+r*x)//(t*x), x+2

The 10 colors of Pi

We are now ready to do our actual visualization of pi.
For legal size paper, 55x97 is a good size (plus the ratio is the square root of pi). For poster, I use 154 x 204 = 31416 digits :)
It'll run for a while, even at the reduced size, probably a good time to go and get your favorite drink...

In [4]:

width = 55  # 154
height = 97  # 204 
digit = pi_generate()

fig = plt.figure(figsize=((width + 2) / 3., (height + 2) / 3.))
ax = fig.add_axes((0.05, 0.05, 0.9, 0.9),
                  aspect='equal', frameon=False,
                  xlim=(-0.05, width + 0.05),
                  ylim=(-0.05, height + 0.05))

for axis in (ax.xaxis, ax.yaxis):
    axis.set_major_formatter(plt.NullFormatter())
    axis.set_major_locator(plt.NullLocator())
    
for j in range(height-1,-1,-1):
    for i in range(width):
        pi_digit = next(digit)
        ax.add_patch(Rectangle((i, j),
                               width=1,
                               height=1,
                               ec=sns.color_palette()[pi_digit],
                               fc=sns.color_palette()[pi_digit],
                              )
                    )
        ax.text(i + 0.5, j + 0.5,
                pi_digit, color='k',
                fontsize=10,
                ha='center', va='center')
ax.text(0,-1,"'THE 10 COLORS OF PI' by Francois Dion", fontsize=15)

Out[4]:

<matplotlib.text.Text at 0x10966f1d0>

The original jupyter notebook can be downloaded here:
https://github.com/fdion/infographics_research/blob/master/The%2010%20colors%20of%20Pi.ipynb

Francois Dion
@f_dion