Reading Data Fast via Feather

A few interesting packages that are newish to me.

The first of which is called feather. It stems from the Apache Arrow project and makes it super fast to read datafiles in R.

The process which this uses seems rather intutive as to why it works better.

# devtools::install_github("dgrtwo/fuzzyjoin")
# devtools::install_github("hadley/tibble")
# devtools::install_github("wesm/feather/R")
# devtools::install_github("hadley/readr")

library(feather)
library(readr)
library(data.table)
library(tibble)
library(dplyr)

This data can be found here.

system.time(x <- read.csv('2008.csv'))
#   user  system elapsed
# 96.721   2.850  99.914

write_feather(x, '2008.feather')

rm(x);gc();
system.time(x <- read_feather('2008.feather'))
#   user  system elapsed
# 0.765   0.396   1.162

rm(x);gc();
system.time(x <- read_csv('2008.csv'))
#  user  system elapsed 
# 15.345   1.413  18.642

rm(x);gc();
system.time(x <- fread('2008.csv'))
#  user  system elapsed 
# 6.923   0.374   7.302

Another useful package is tibble. Normally I start all of my code with by turning strings to factors as FALSE, becuase if I don’t 15 minutes later I have to figure out why something that should be easy is not working. I modify this option then re-reun everything I am good to go, except that my train of thought was derailed pretty hard.

I have also never been a fan of the odd process of creating a throw away data.frame.

throw_away <- data.frame(a = c(1, 2, 3), b = c('a', 'b', 'c'))
str(throw_away)

## 'data.frame':    3 obs. of  2 variables:
##  $ a: num  1 2 3
##  $ b: Factor w/ 3 levels "a","b","c": 1 2 3

rbind(throw_away, c(1, 'd'))

## Warning in `[<-.factor`(`*tmp*`, ri, value = structure(c(1L, 2L, 3L,
## NA), .Label = c("a", : invalid factor level, NA generated

##   a    b
## 1 1    a
## 2 2    b
## 3 3    c
## 4 1 <NA>

That was in no way what you would expect to happen.

The tibble method to construct a dataset is very similar to SAS or Matlab, which is clean. And most of all it does not make anything into a factor for me.

a <- tibble::frame_data(
  ~x, ~y,  ~z,
  "a", 2,  3.6,
  "b", 1,  8.5,
  "c", 1,  8.5,
  "D", 1,  8.5)