stringi 1.1.1 released

stringi is among the top 10 most downloaded R packages, providing various string processing facilities. A new release comes with a few bugfixes and new features.

* [BUGFIX] #214: allow a regex pattern like `.*`  to match an empty string.

* [BUGFIX] #210: `stri_replace_all_fixed(c("1", "NULL"), "NULL", NA)`
now results in `c("1", NA)`.

* [NEW FEATURE] #199: `stri_sub<-` now allows for ignoring `NA` locations
(a new `omit_na` argument added).

* [NEW FEATURE] #207: `stri_sub<-` now allows for substring insertions
(via `length=0`).

* [NEW FUNCTION] #124: `stri_subset<-` functions added.

* [NEW FEATURE] #216: `stri_detect`, `stri_subset`, `stri_subset<-` gained
a `negate` argument.

* [NEW FUNCTION] #175: `stri_join_list` concatenates all strings
in a list of character vectors. Useful with, e.g., `stri_extract_all_regex`,
`stri_extract_all_words` etc.
Tagged with: , ,
Posted in Blog/News, Blog/R

stringi 1.0-1

First of all, we’re happy to announce that since the 1.0.0 release, the stringr package for R is now powered by stringi. For more details, read more here.

Also please note that the stringi package version 1.0-1 is now on CRAN. Changelog:

* [GENERAL] #88: C++ API is now available for use in, e.g., Rcpp packages, see for an example.

* [BUGFIX] #183: Floating point exception raised in `stri_sub()` and `stri_sub<-()` when `to` or `length` was a zero-length numeric vector.

* [BUGFIX] #180: `stri_c()` warned incorrectly (recycling rule) when using more than 2 elements.
Tagged with: , ,
Posted in Blog/News, Blog/R

Speeding up R packages’ installation process

There is a time for some things, and a time for all things; a time for great things, and a time for small things — Miguel de Cervantes

Building R packages from sources may take a long time, especially if they contain a lot of C/C++/Fortran code. Long compile time might be especially frustrating if you are a package developer and you need to recompile your project very often.

Here is how long it takes to compile the stringi package on my laptop (if the ICU library is also compiled from sources):

$ time R CMD INSTALL ~/R/stringi --preclean --configure-args='--disable-pkg-config'
# real    2m6.989s

On many R installations, the build process is set up so that only one C/C++ source file is compiled at a time:

CPU and RAM Usage - before

Yet, there is a simple solution for that — we may ask GNU make to allow more than one job to be submitted at once. In order to do so, we edit the /lib64/R/etc/Renviron file (where /lib64/R/etc/ is the result to a call to the R.home() function in R) and set:

MAKE='make -j 8' # submit 8 jobs at once

instead of previously used settings.

This significantly decreases the time needed to compile stringi :

$ time R CMD INSTALL ~/R/stringi --preclean --configure-args='--disable-pkg-config'
# real    0m38.831s

CPU and RAM Usage - after

Thanks to that, we may now spend the time saved to enjoy more whomever or whatever we love. :)

Note that MAKE is an environmental variable and can also be changed from within the current R session (Sys.setenv) or while we start R from the Linux/Unix terminel (MAKE="..." R – thanks to Nick Kennedy for noticing that).

Tagged with: , , ,
Posted in Blog/R, Blog/R-bloggers

Pull the (character) strings with stringi 0.5-2

A reliable string processing toolkit is a must-have for any data scientist.

A new release of the stringi package is available on CRAN (please wait a few days for Windows and OS X binary builds). As for now, about 850 CRAN packages depend (either directly or recursively) on stringi. And quite recently, the package got listed among the top downloaded R extensions.

# install.packages("stringi") or update.packages()
## [1] "stringi_0.5.2; en_US.UTF-8; ICU4C 55.1; Unicode 7.0"
apkg <- available.packages(contriburl="")
length(tools::dependsOnPkgs('stringi', installed=apkg, recursive=TRUE))
## [1] 845

Refer to the INSTALL file for more details if you compile stringi from sources (Linux users mostly).

Here’s a list of changes in version 0.5-2. There are many major (like date&time processing) and minor new features, enhancements, as well as bugfixes. In the current release we also focused on bringing stringr package’s users even better string processing experience, as since the 1.0.0 release it is now powered by stringi.

  • [BACKWARD INCOMPATIBILITY] The second argument to stri_pad_*() has been renamed width.

  • [GENERAL] #69: stringi is now bundled with ICU4C 55.1.

  • [NEW FUNCTIONS] #137: date-time formatting/parsing (note that this is draft API and it may change in future stringi releases; any comments are welcome):
    • stri_timezone_list() – lists all known time zone identifiers
    sample(stri_timezone_list(), 10)
    ##  [1] "Etc/GMT+12"                  "Antarctica/Macquarie"       
    ##  [3] "Atlantic/Faroe"              "Antarctica/Troll"           
    ##  [5] "America/Fort_Wayne"          "PLT"                        
    ##  [7] "America/Goose_Bay"           "America/Argentina/Catamarca"
    ##  [9] "Africa/Juba"                 "Africa/Bissau"
    • stri_timezone_set(), stri_timezone_get() – manage current default time zone
    • stri_timezone_info() – basic information on a given time zone
    ## List of 6
    ##  $ ID              : chr "Europe/Warsaw"
    ##  $ Name            : chr "Central European Standard Time"
    ##  $ Name.Daylight   : chr "Central European Summer Time"
    ##  $ Name.Windows    : chr "Central European Standard Time"
    ##  $ RawOffset       : num 1
    ##  $ UsesDaylightTime: logi TRUE
    stri_timezone_info('Europe/Warsaw', locale='de_DE')$Name
    ## [1] "Mitteleuropäische Normalzeit"
    • stri_datetime_symbols() – localizable date-time formatting data
    ## $Month
    ##  [1] "January"   "February"  "March"     "April"     "May"      
    ##  [6] "June"      "July"      "August"    "September" "October"  
    ## [11] "November"  "December" 
    ## $Weekday
    ## [1] "Sunday"    "Monday"    "Tuesday"   "Wednesday" "Thursday"  "Friday"   
    ## [7] "Saturday" 
    ## $Quarter
    ## [1] "1st quarter" "2nd quarter" "3rd quarter" "4th quarter"
    ## $AmPm
    ## [1] "AM" "PM"
    ## $Era
    ## [1] "Before Christ" "Anno Domini"
    ##  [1] "มกราคม"  "กุมภาพันธ์"    "มีนาคม"    "เมษายน"  "พฤษภาคม" "มิถุนายน"    "กรกฎาคม"
    ##  [8] "สิงหาคม"   "กันยายน"   "ตุลาคม"    "พฤศจิกายน" "ธันวาคม"
    ##  [1] "תשרי"   "חשון"   "כסלו"   "טבת"    "שבט"    "אדר א׳" "אדר"   
    ##  [8] "ניסן"   "אייר"   "סיון"   "תמוז"   "אב"     "אלול"   "אדר ב׳"
    • stri_datetime_now() – return current date-time
    • stri_datetime_fstr() – convert a strptime-like format string to an ICU date/time format string
    • stri_datetime_format() – convert date/time to string
        stri_datetime_format(stri_datetime_now(), "datetime_relative_medium")
    ## [1] "today, 6:21:45 PM"
    • stri_datetime_parse() – convert string to date/time object
    stri_datetime_parse(c("2015-02-28", "2015-02-29"), "yyyy-MM-dd")
    ## [1] "2015-02-28 18:21:45 CET" NA
    stri_datetime_parse(c("2015-02-28", "2015-02-29"), stri_datetime_fstr("%Y-%m-%d"))
    ## [1] "2015-02-28 18:21:45 CET" NA
    stri_datetime_parse(c("2015-02-28", "2015-02-29"), "yyyy-MM-dd", lenient=TRUE)
    ## [1] "2015-02-28 18:21:45 CET" "2015-03-01 18:21:45 CET"
    stri_datetime_parse("19 lipca 2015", "date_long", locale="pl_PL")
    ## [1] "2015-07-19 18:21:45 CEST"
    • stri_datetime_create() – construct date-time objects from numeric representations
    stri_datetime_create(2015, 12, 31, 23, 59, 59.999)
    ## [1] "2015-12-31 23:59:59 CET"
    stri_datetime_create(5775, 8, 1, locale="@calendar=hebrew") # 1 Nisan 5775 -> 2015-03-21
    ## [1] "2015-03-21 12:00:00 CET"
    stri_datetime_create(2015, 02, 29)
    ## [1] NA
    stri_datetime_create(2015, 02, 29, lenient=TRUE)
    ## [1] "2015-03-01 12:00:00 CET"
    • stri_datetime_fields() – get values for date-time fields
    ##   Year Month Day Hour Minute Second Millisecond WeekOfYear WeekOfMonth
    ## 1 2015     6  23   18     21     45          52         26           4
    ##   DayOfYear DayOfWeek Hour12 AmPm Era
    ## 1       174         3      6    2   2
       stri_datetime_fields(stri_datetime_now(), locale="@calendar=hebrew")
    ##   Year Month Day Hour Minute Second Millisecond WeekOfYear WeekOfMonth
    ## 1 5775    11   6   18     21     45          56         40           2
    ##   DayOfYear DayOfWeek Hour12 AmPm Era
    ## 1       272         3      6    2   1
      stri_datetime_fields(stri_datetime_now(), locale="@calendar=hebrew")$Month
    ## [1] "Tamuz"
    • stri_datetime_add() – add specific number of date-time units to a date-time object
    x <- stri_datetime_create(2015, 12, 31, 23, 59, 59.999)
    stri_datetime_add(x, units="months") <- 2
    ## [1] "2016-02-29 23:59:59 CET"
    stri_datetime_add(x, -2, units="months")
    ## [1] "2015-12-29 23:59:59 CET"
  • [NEW FUNCTIONS] stri_extract_*_boundaries() extract text between text boundaries.

  • [NEW FUNCTION] #46: stri_trans_char() is a stringi-flavoured chartr() equivalent.

stri_trans_char("id.123", ".", "_")
## [1] "id_123"
stri_trans_char("babaab", "ab", "01")
## [1] "101001"
  • [NEW FUNCTION] #8: stri_width() approximates the width of a string in a more Unicodish fashion than nchar(..., "width")
## [1] 1 1 1 1 1
nchar(stri_trans_nfkd("\u0105"), "width") # provides incorrect information
## [1] 0
stri_width(stri_trans_nfkd("\u0105")) # A and ogonek (width = 1)
## [1] 1
stri_width( # Full-width equivalents of ASCII characters:
   stri_enc_fromutf32(as.list(c(0x3000, 0xFF01:0xFF5E)))
##  [1] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
## [36] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
## [71] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
  • [NEW FEATURE] #149: stri_pad() and stri_wrap() now by default bases on code point widths instead of the number of code points. Moreover, the default behavior of stri_wrap() is now such that it does not get rid of non-breaking, zero width, etc. spaces
x <- stri_flatten(c(
   stri_dup(LETTERS, 2),
), collapse=' ')
# Note that your web browser may have problems with properly aligning
# this (try it in RStudio)
cat(stri_wrap(x, 11), sep='\n')
## YY ZZ A B
## C D E F
## G H I J
## K L M N
## O P Q R
## S T U V
## W X Y Z
  • [NEW FEATURE] #133: stri_wrap() silently allows for width <= 0 (for compatibility with strwrap()).

  • [NEW FEATURE] #139: stri_wrap() gained a new argument: whitespace_only.

  • [GENERAL] #144: Performance improvements in handling ASCII strings (these affect stri_sub(), stri_locate() and other string index-based operations)

  • [GENERAL] #143: Searching for short fixed patterns (stri_*_fixed()) now relies on the current libC’s implementation of strchr() and strstr(). This is very fast e.g. on glibc utilizing the SSE2/3/4 instruction set.

x <- stri_rand_strings(100, 10000, "[actg]")
   stri_detect_fixed(x, "acgtgaa"),
   grepl("actggact", x),
   grepl("actggact", x, perl=TRUE),
   grepl("actggact", x, fixed=TRUE)
## Unit: microseconds
##                                expr       min        lq       mean
##     stri_detect_fixed(x, "acgtgaa")   349.153   354.181   381.2391
##                grepl("actggact", x) 14017.923 14181.416 14457.3996
##   grepl("actggact", x, perl = TRUE)  8280.282  8367.426  8516.0124
##  grepl("actggact", x, fixed = TRUE)  3599.200  3637.373  3726.6020
##      median         uq       max neval  cld
##    362.7515   391.0655   681.267   100 a   
##  14292.2815 14594.4970 15736.535   100    d
##   8463.4490  8570.0080  9564.503   100   c 
##   3686.6690  3753.4060  4402.397   100  b
  • [GENERAL] #141: a local copy of icudt*.zip may be used on package install; see the INSTALL file for more information.

  • [GENERAL] #165: the ./configure option --disable-icu-bundle forces the use of system ICU when building the package.

  • [BUGFIX] locale specifiers are now normalized in a more intelligent way: e.g. @calendar=gregorian expands to DEFAULT_LOCALE@calendar=gregorian.

  • [BUGFIX] #134: stri_extract_all_words() did not accept simplify=NA.

  • [BUGFIX] #132: incorrect behavior in stri_locate_regex() for matches of zero lengths.

  • [BUGFIX] stringr/#73: stri_wrap() returned CHARSXP instead of STRSXP on empty string input with simplify=FALSE argument.

  • [BUGFIX] #164: libicu-dev usage used to fail on Ubuntu.

  • [BUGFIX] #135: C++11 is now used by default (see the INSTALL file, however) to build stringi from sources. This is because ICU4C uses the long long type which is not part of the C++98 standard.

  • [BUGFIX] #154: Dates and other objects with a custom class attribute were not coerced to the character type correctly.

  • [BUGFIX] #168: Build now fails if icudt is not available.

  • [BUGFIX] Force ICU u_init() call on stringi dynlib load.

  • [BUGFIX] #157: many overfull hboxes in the package PDF manual has been corrected.

Enjoy! Any comments and suggestions are welcome.

Tagged with:
Posted in Blog/News, Blog/R, Blog/R-bloggers



Being a teacher can be a very gratifying job. If you teach programming, which is your favorite hobby too, nothing can be better than that. Only thing can spoil your dream: cheating students. As we all know, one can learn programming only by writing code him/herself. Copying source code of another student completely makes no sense, as student does not learn, and what is more, he/she gets points for something he/she didn’t make.

When there are only few homeworks to check, it is easy to do it manually. But what if there is a large number of submissions? Then we need some application to automate the process. There are some known tools for “standard” programming languages, such as MOSS or JPLAG for e.g. C, C++, C#, Java, Scheme or Javascript.

But what if we want to automate the process of checking similarity of R source code? Till now there were no such a tool available. But things have changed.


SimilaR is a service designed to detect similar source code patterns in the R language code snippets. To create an account, you got to possess an e-mail address in edu domain and prove us somehow that you’re a tutor (show us your webpage etc.). Once the account is activated you just upload your students’ submissions and wait a moment for the results.

Let see a working example. Assume that one student submitted the following file:

almostPi <- function(n)
# this is a function which approximate a Pi constant

x <- runif(n,-1,1);
y <- runif(n,-1,1);

# I arrange triples in a matrix
l[[length(a)+1]] <- (a^2+b^2==c^2 & a*b*c!=0 & a*c>0)

and the other one sent:

almostPi<-function (n=10000) {
# Checking if n is a numeric vetor of length 1,
# and if it is a natural number



So we log into SimilaR, choose Antiplagiarism system -> New submission and we get a picture like:

In the area marked with a green rectangle we provide a name for a new submission. We can identify a group of files with this name. In the blue rectangle we choose what is the smallest group of functions (functions in one group are not compared): group of files, one file, or we compare every function with each other. Since every student in our example provide her homework in separate file, we choose a second option.

After we click Submit, we obtain:

In this view we can make sure that system understands uploaded files as we expect. If something is wrong, e.g. the source code has syntax errors, we will be notified at this step. Please note that there are no comments in source codes and a style of indentation is homogeneous. If everything is OK, we click Confirm button.

After that we see a list of our submissions. We can see a progress of our submission which is dynamically updated. When it is ready, it goes to a top of the list and we can see it.

Let us see the results. There are 4 pairs, as there were 2 functions in each file. The pairs are ordered from most similar to the least. In the beginning, we see only first 10 pairs, and we can assess every pair, if we believe it is similar or not. After evaluating some pairs (see green rectangle), we can see more of them. This solution is needed, as the system is based on some statistical learning algorithms and we need as many learning data as we can obtain so that it will become even more useful in the future.


We hope that SimilaR will be a useful tool, and that it will make evaluating the similarity of students’ homeworks faster and more accurate as well as a teacher’s job more convenient. With this tool, R tutors can focus on what is the most important thing in the teaching process: teaching, not searching for a plagiarism and dishonest students. Prior using the system, make sure you agree with the Terms and Conditions


  1. Bartoszuk M., Gagolewski M., A fuzzy R code similarity detection algorithm, In: Laurent A. et al. (Eds.), Information Processing and Management of Uncertainty in Knowledge-Based Systems, Part III (CCIS 444), Springer-Verlag, Heidelberg, 2014, pp. 21-30.
  2. Bartoszuk M., Gagolewski M., Detecting similarity of R functions via a fusion of multiple heuristic methods, 2015. (submitted paper)
Posted in Blog/R, Blog/R-bloggers

Using Hadoop Streaming API to perform a word count job in R and C++

by Marek Gagolewski, Maciej Bartoszuk, Anna Cena, and Jan Lasek (Rexamine).


In a recent blog post we explained how we managed to set up a working Hadoop environment on a few CentOS7 machines. To test the installation, let’s play with a simple example.

Hadoop Streaming API allows to run Map/Reduce jobs with any programs as the mapper and/or the reducer.

Files are processed line-by-line. Mappers get appropriate chunks of the input file. Each line is assume to store information on key-value pairs. By default, the following form is used:

key1 \t val1 \n
key2 \t val2 \n

If there is no TAB character, then the value is assumed to be NULL.

In fact this is a hadoop version of a program that rearranges lines in the input file so that duplicated lines appear one after another – the output is always sorted by key.

This is because:

hadoop jar /opt/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar \
   -input /input/test.txt \
   -output /output
   -mapper /bin/cat
   -reducer /bin/cat
hdfs dfs -cat /output/part-00000

This is roughly equivalent to:

cat input | mapper | sort | reducer > output

More specifically, in our case that was:

cat input | cat | sort | cat > output

A sample Map/Reduce job

Let’s run a simple Map/Reduce job written in R and C++ (just for fun – we assume that all the nodes run the same operating system and they use the same CPU architecture).

  1. As we are in the CentOS 7 environment, we will need a newer version of R on all the nodes.
$ su
# yum install readline-devel
# cd
# wget
# tar -zxf R-3.1.2.tar.gz
# cd R-3.1.2
# /configure --with-x=no --with-recommended-packages=no
# make
# make install
# R
R> install.packages('stringi')
R> q()
  1. Edit yarn-site.xml (on all nodes):

Without that, Hadoop may complain about too huge virtual memory memory consumption by R.

  1. Create script wc_mapper.R:
#!/usr/bin/env Rscript

stdin <- file('stdin', open='r')

while(length(x <- readLines(con=stdin, n=1024L))>0) {
   x <- unlist(stri_extract_all_words(x))
   xt <- table(x)
   words <- names(xt)
   counts <- as.integer(xt)
   cat(stri_paste(words, counts, sep='\t'), sep='\n')
  1. Create a source file wc_reducer.cpp:
#include <iostream>
#include <string>
#include <cstdlib>

using namespace std;

int main()
  string line;
  string last_word = "";
  int last_count = 0;

    size_t found = line.find_first_of("\t");
    if(found != string::npos)
      string key = line.substr(0,found);
      string value = line.substr(found);
      int valuei = atoi(value.c_str());
      //cerr << "key=" << key << " value=" << value <<endl;
      if(key != last_word)
              if(last_word != "") cout << last_word << "\t" << last_count << endl;

              last_word = key;
              last_count = valuei;
              last_count += valuei;
  if(last_word != "") cout << last_word << "\t" << last_count << endl;

  return 0;

Now it’s time to compile the above C++ source file:

$ g++ -O3 wc_reducer.cpp -o wc_reducer
  1. Let’s submit a map/reduce job via the Hadoop Streaming API
$ chmod 755 wc_mapper.R
$ hadoop jar /opt/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar \
   -input /input/test.txt \
   -output /output
   -mapper wc_mapper.R
   -reducer wc_reducer
   -file wc_mapper.R
   -file wc_reducer

By the way, Fedora 20 RPM Hadoop distribution provides Hadoop Streaming API jar file under /usr/share/hadoop/mapreduce/hadoop-streaming.jar.


In this tutorial we showed how to submit a simple Map/Reduce job via the Hadoop Streaming API. Interestingly, we used an R script as the mapper and a C++ program as the reducer. In an upcoming blog post we’ll explain how to run a job using the rmr2 package.

Tagged with: , , , , ,
Posted in Blog/Hadoop, Blog/R, Blog/R-bloggers

Installing Hadoop 2.6.0 on CentOS 7

by Marek Gagolewski, Maciej Bartoszuk, Anna Cena, and Jan Lasek (Rexamine).

Configuring a working Hadoop 2.6.0 environment on CentOS 7 is a bit of a struggle. Here are the steps we made to set everything up so that we have a working hadoop cluster. Of course, there many tutorials on this topic over the internet. None of the solutions presented there worked in our case. Thus, there is a high possibility that also this step-by-step guide will make you very frustrated. Anyway, resolving errors generated by Hadoop should make you understand this environment much better. No pain no gain.

Basic CentOS setup

Let’s assume that we have a fresh CentOS install. On each node:

  1. Edit /etc/hosts
# nano /etc/hosts

Add the following lines (change IP addresses accordingly): hmaster hslave1 hslave2 hslave3
  1. Create user hadoop
# useradd hadoop
# passwd hadoop
  1. Set up key-based (passwordless) login:
# su hadoop
$ ssh-keygen -t rsa
$ ssh-copy-id -i ~/.ssh/ hadoop@hmaster
$ ssh-copy-id -i ~/.ssh/ hadoop@hslave1
$ ssh-copy-id -i ~/.ssh/ hadoop@hslave2
$ ssh-copy-id -i ~/.ssh/ hadoop@hslave3
$ chmod 0600 ~/.ssh/authorized_keys

This will be useful when we’d like to start all necessary hadoop services on all the slave nodes.

Installing Oracle Java SDK

  1. Download latest Oracle JDK and save it in the /opt directory.

  2. On hmaster, unpack Java:

# cd /opt
# tar -zxf jdk-8u31-linux-x64.tar.gz
# mv jdk1.8.0_31 jdk

Now propagete /opt/jdk to all the slaves

# scp -r jdk hslave1:/opt
# scp -r jdk hslave2:/opt
# scp -r jdk hslave3:/opt
  1. On each node, let’s use the alternatives tool to set up Oracle Java as the default Java framework.
# alternatives --install /usr/bin/java java /opt/jdk/bin/java 2
# alternatives --config java # select appropriate program (/opt/jdk/bin/java)
# alternatives --install /usr/bin/jar jar /opt/jdk/bin/jar 2
# alternatives --install /usr/bin/javac javac /opt/jdk/bin/javac 2
# alternatives --set jar /opt/jdk/bin/jar
# alternatives --set javac /opt/jdk/bin/javac 

Check if everything is OK by executing java -version.

  1. Set up environmental variables:
# nano /etc/bashrc

Add the following:

export JAVA_HOME=/opt/jdk
export JRE_HOME=/opt/jdk/jre
export PATH=$PATH:/opt/jdk/bin:/opt/jdk/jre/bin

And also possibly:

alias ll='ls -l --color'
alias cp='cp -i'
alias mv='mv -i'
alias rm='rm -i'

Check if everyting is OK:

# source /etc/bashrc
# echo $JAVA_HOME

Installing and configuring hadoop 2.6.0

On master:

# cd /opt
# wget
# tar -zxf hadoop-2.6.0.tar.gz
# rm hadoop-2.6.0.tar.gz
# mv hadoop-2.6.0 hadoop

Propagate /opt/hadoop to slave nodes:

# scp -r hadoop hslave1:/opt
# scp -r hadoop hslave2:/opt
# scp -r hadoop hslave3:/opt

Add the following lines to /home/hadoop/.bashrc on all the nodes (you may play with scp for that too):

export HADOOP_PREFIX=/opt/hadoop

Edit /opt/hadoop/etc/hadoop/core-site.xml – set up NameNode URI on every node:


Create HDFS DataNode data dirs on every node and change ownership of /opt/hadoop:

# chown hadoop /opt/hadoop/ -R
# chgrp hadoop /opt/hadoop/ -R
# mkdir /home/hadoop/datanode
# chown hadoop /home/hadoop/datanode/
# chgrp hadoop /home/hadoop/datanode/    

Edit /opt/hadoop/etc/hadoop/hdfs-site.xml – set up DataNodes:


Create HDFS NameNode data dirs on master:

# mkdir /home/hadoop/namenode
# chown hadoop /home/hadoop/namenode/
# chgrp hadoop /home/hadoop/namenode/    

Edit /opt/hadoop/etc/hadoop/hdfs-site.xml on master. Add further properties:


Edit /opt/hadoop/etc/hadoop/mapred-site.xml on master.

   <value>yarn</value> <!-- and not local (!) -->

Edit /opt/hadoop/etc/hadoop/yarn-site.xml – setup ResourceManager and NodeManagers:

        <value>hmaster</value> <!-- or hslave1, hslave2, hslave3 -->

Edit /opt/hadoop/etc/hadoop/slaves on master (so that master may start all necessary services on slaves automagically):


Now the important step: disable firewall and IPv6 (Hadoop does not support IPv6 – problems with listening on all the interfaces via

# systemctl stop firewalld

Add the following lines to /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

Format NameNode:

# su hadoop
$ hdfs namenode -format

Start HDFS (as user hadoop):


Check out with jps if DataNode are running on slaves and if DataNode, NameNode, and SecondaryNameNode are running on master. Also try accessing http://hmaster:50070/

Start YARN on master:


Now NodeManagers should be alive (jps) on all nodes and a ResourceManager on master too.

We see that the master node consists of a ResourceManager, NodeManager (YARN), NameNode and DataNode (HDFS). A slave node acts as both a NodeManager and a DataNode.

Testing hadoop 2.6.0

You may want to check out if you are able to copy a local file to HDFS and run the standalone Hadoop Hello World (i.e. wordcount) Job.

$ hdfs dfsadmin -safemode leave # ??????
$ hdfs dfs -mkdir /input
$ hdfs dfs -copyFromLocal test.txt /input
$ hdfs dfs -cat /input/test.txt | head
$ hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /input/test.txt /output1

If anything went wrong, check out /opt/hadoop/log/*.log. Good luck :)

Tagged with: , , ,
Posted in Blog/Hadoop

stringi 0.4-1 released – fast, portable, consistent character string processing

A new release of the stringi package is available on CRAN (please wait a few days for Windows and OS X binary builds).

# install.packages("stringi") or update.packages()

Here’s a list of changes in version 0.4-1. In the current release, we particularly focused on making the package’s interface more consistent with that of the well-known stringr package. For a general overview of stringi’s facilities and base R string processing issues, see e.g. here.

  • (IMPORTANT CHANGE) n_max argument in stri_split_*() has been renamed n.

  • (IMPORTANT CHANGE) simplify=FALSE in stri_extract_all_*() and stri_split_*() now calls stri_list2matrix() with fill="". fill=NA_character_ may be obtained by using simplify=NA.

  • (IMPORTANT CHANGE, NEW FUNCTIONS) #120: stri_extract_words has been renamed stri_extract_all_words and stri_locate_boundariesstri_locate_all_boundaries as well as stri_locate_wordsstri_locate_all_words. New functions are now available: stri_locate_first_boundaries, stri_locate_last_boundaries, stri_locate_first_words, stri_locate_last_words, stri_extract_first_words, stri_extract_last_words.

# uses ICU's locale-dependent word break iterator
stri_extract_all_words("stringi: THE string processing package for R")
## [[1]]
## [1] "stringi"    "THE"        "string"     "processing" "package"   
## [6] "for"        "R"
  • (IMPORTANT CHANGE) #111: opts_regex, opts_collator, opts_fixed, and opts_brkiter can now be supplied individually via .... In other words, you may now simply call e.g.
stri_detect_regex(c("stringi", "STRINGI"), "stringi", case_insensitive=TRUE)
## [1] TRUE TRUE

instead of:

stri_detect_regex(c("stringi", "STRINGI"), "stringi", opts_regex=stri_opts_regex(case_insensitive=TRUE))
## [1] TRUE TRUE
  • (NEW FEATURE) #110: Fixed pattern search engine’s settings can now be supplied via opts_fixed argument in stri_*_fixed(), see stri_opts_fixed(). A simple (not suitable for natural language processing) yet very fast case_insensitive pattern matching can be performed now. stri_extract_*_fixed is again available.

  • (NEW FEATURE) #23: stri_extract_all_fixed, stri_count, and stri_locate_all_fixed may now also look for overlapping pattern matches, see ?stri_opts_fixed.

stri_extract_all_fixed("abaBAaba", "ABA", case_insensitive=TRUE, overlap=TRUE)
## [[1]]
## [1] "aba" "aBA" "aba"
  • (NEW FEATURE) #129: stri_match_*_regex gained a cg_missing argument.

  • (NEW FEATURE) #117: stri_extract_all_*(), stri_locate_all_*(), stri_match_all_*() gained a new argument: omit_no_match. Setting it to TRUE makes these functions compatible with their stringr equivalents.

  • (NEW FEATURE) #118: stri_wrap() gained indent, exdent, initial, and prefix arguments. Moreover Knuth’s dynamic word wrapping algorithm now assumes that the cost of printing the last line is zero, see #128.

cat(stri_wrap(stri_rand_lipsum(1), 40, 2.0), sep="\n")
## Lorem ipsum dolor sit amet, et et diam
## vitae est ut. At tristique, tincidunt
## taciti, ac egestas vestibulum magna.
## Volutpat nisl non sed ultricies nisl
## nibh magna. Nullam rhoncus ut phasellus
## sed. Congue enim libero congue massa
## eget. Ligula, quis est amet velit.
## Accumsan amet nunc ad. Porttitor,
## sed vestibulum diam vestibulum quis
## sed gravida ultrices. Per urna enim.
## Scelerisque interdum sed vestibulum
## rhoncus quis imperdiet pharetra. Sapien
## iaculis, lacinia ac cras ante, sed
## vitae inceptos dis tristique dignissim.
## Venenatis volutpat lectus sodales,
## hac feugiat molestie mollis. A, urna
## pellentesque ante himenaeos ante at
## potenti in.
  • (NEW FEATURE) #122: stri_subset() gained an omit_na argument.
stri_subset_fixed(c("abc", NA, "def"), "a")
## [1] "abc" NA
stri_subset_fixed(c("abc", NA, "def"), "a", omit_na=TRUE)
## [1] "abc"
  • (NEW FEATURE) stri_list2matrix() gained an n_min argument.

  • (NEW FEATURE) #126: stri_split() now is also able to act just like stringr::str_split_fixed().

stri_split_regex(c("bab", "babab"), "a", n = 3, simplify=TRUE)
##      [,1] [,2] [,3]
## [1,] "b"  "b"  ""  
## [2,] "b"  "b"  "b"
  • (NEW FEATURE) #119: stri_split_boundaries() now have n, tokens_only, and simplify arguments. Additionally, stri_extract_all_words() is now equipped with simplify arg.

  • (NEW FEATURE) #116: stri_paste() gained a new argument: ignore_null. Setting it to TRUE makes this function more compatible with paste().

for (test in c(TRUE, FALSE))
   print(stri_paste("a", if (test) 1:9, ignore_null=TRUE))
## [1] "a1" "a2" "a3" "a4" "a5" "a6" "a7" "a8" "a9"
## [1] "a"

Enjoy! Any comments and suggestions are welcome.

Tagged with: , , ,
Posted in Blog/News, Blog/R, Blog/R-bloggers

Faster, easier, and more reliable character string processing with stringi 0.3-1

A new release of the stringi package is available on CRAN (please wait a few days for Windows and OS X binary builds).

# install.packages("stringi") or update.packages()

stringi is an R package providing (but definitely not limiting to) equivalents of nearly all the character string processing functions known from base R. While developing the package we had high performance and portability of its facilities in our minds.

We implemented each string processing function from scratch. The internationalization and globalization support, as well as many string processing facilities (like regex searching) is guaranteed by the well-known IBM’s ICU4C library.

Here is a very general list of the most important features available in the current version of stringi:

  • string searching:
    • with ICU (Java-like) regular expressions,
    • ICU USearch-based locale-aware string searching (quite slow, but working properly e.g. for non-Unicode normalized strings),
    • very fast, locale-independent byte-wise pattern matching;
  • joining and duplicating strings;
  • extracting and replacing substrings;
  • string trimming, padding, and text wrapping (e.g. with Knuth’s dynamic word wrap algorithm);
  • text transliteration;
  • text collation (comparing, sorting);
  • text boundary analysis (e.g. for extracting individual words);
  • random string generation;
  • Unicode normalization;
  • character encoding conversion and detection;

and many more.

Here’s a list of changes in version 0.3-1:

  • (IMPORTANT CHANGE) #87: %>% overlapped with the pipe operator from the magrittr package; now each operator like %>% has been renamed %s>%.

  • (IMPORTANT CHANGE) #108: Now the BreakIterator (for text boundary analysis) may be better controlled via stri_opts_brkiter() (see options type and locale which aim to replace now-removed boundary and locale parameters to stri_locate_boundaries, stri_split_boundaries, stri_trans_totitle, stri_extract_words, stri_locate_words).

    For example:

test <- "The\u00a0above-mentioned    features are very useful. Warm thanks to their developers. 123 456 789"
stri_split_boundaries(test, stri_opts_brkiter(type="word", skip_word_none=TRUE, skip_word_number=TRUE)) # cf. stri_extract_words
## [[1]]
##  [1] "The"        "above"      "mentioned"  "features"   "are"       
##  [6] "very"       "useful"     "Warm"       "thanks"     "to"        
## [11] "their"      "developers"
stri_split_boundaries(test, stri_opts_brkiter(type="sentence")) # extract sentences
## [[1]]
## [1] "The above-mentioned    features are very useful. "
## [2] "Warm thanks to their developers. "                
## [3] "123 456 789"
stri_split_boundaries(test, stri_opts_brkiter(type="character")) # extract characters
## [[1]]
##  [1] "T" "h" "e" " " "a" "b" "o" "v" "e" "-" "m" "e" "n" "t" "i" "o" "n"
## [18] "e" "d" " " " " " " " " "f" "e" "a" "t" "u" "r" "e" "s" " " "a" "r"
## [35] "e" " " "v" "e" "r" "y" " " "u" "s" "e" "f" "u" "l" "." " " "W" "a"
## [52] "r" "m" " " "t" "h" "a" "n" "k" "s" " " "t" "o" " " "t" "h" "e" "i"
## [69] "r" " " "d" "e" "v" "e" "l" "o" "p" "e" "r" "s" "." " " "1" "2" "3"
## [86] " " "4" "5" "6" " " "7" "8" "9"

By the way, the last call also works correctly for strings not in the Unicode Normalization Form C:

stri_split_boundaries(stri_trans_nfkd("zażółć gęślą jaźń"), stri_opts_brkiter(type="character"))
## [[1]]
##  [1] "z" "a" "ż"  "ó"  "ł" "ć"  " " "g" "ę"  "ś"  "l" "ą"  " " "j" "a" "ź"  "ń"
  • (NEW FUNCTIONS) #109: stri_count_boundaries and stri_count_words count the number of text boundaries in a string.
stri_count_words("Have a nice day!")
## [1] 4
  • (NEW FUNCTIONS) #41: stri_startswith_* and stri_endswith_* determine whether a string starts or ends with a given pattern.
stri_startswith_fixed(c("a1o", "a2g", "b3a", "a4e", "c5a"), "a")
  • (NEW FEATURE) #102: stri_replace_all_* gained a vectorize_all parameter, which defaults to TRUE for backward compatibility.
stri_replace_all_fixed("The quick brown fox jumped over the lazy dog.",
     c("quick", "brown", "fox"), c("slow",  "black", "bear"), vectorize_all=FALSE)
## [1] "The slow black bear jumped over the lazy dog."
# Compare the results:
stri_replace_all_fixed("The quicker brown fox jumped over the lazy dog.",
     c("quick", "brown", "fox"), c("slow",  "black", "bear"), vectorize_all=FALSE)
## [1] "The slower black bear jumped over the lazy dog."
stri_replace_all_regex("The quicker brown fox jumped over the lazy dog.",
     "\\b"%s+%c("quick", "brown", "fox")%s+%"\\b", c("slow",  "black", "bear"), vectorize_all=FALSE)
## [1] "The quicker black bear jumped over the lazy dog."
  • (NEW FUNCTIONS) #91: stri_subset_*, a convenient and more efficient substitute for str[stri_detect_*(str, ...)], added.
stri_subset_regex(c("", "", "no email here"),
## [1] "" ""
  • (NEW FEATURE) #100: stri_split_fixed, stri_split_charclass, stri_split_regex, stri_split_coll gained a tokens_only parameter, which defaults to FALSE for backward compatibility.
stri_split_fixed(c("ab_c", "d_ef_g", "h", ""), "_", n_max=1, tokens_only=TRUE, omit_empty=TRUE)
## [[1]]
## [1] "ab"
## [[2]]
## [1] "d"
## [[3]]
## [1] "h"
## [[4]]
## character(0)
stri_split_fixed(c("ab_c", "d_ef_g", "h", ""), "_", n_max=2, tokens_only=TRUE, omit_empty=TRUE)
## [[1]]
## [1] "ab" "c" 
## [[2]]
## [1] "d"  "ef"
## [[3]]
## [1] "h"
## [[4]]
## character(0)
stri_split_fixed(c("ab_c", "d_ef_g", "h", ""), "_", n_max=3, tokens_only=TRUE, omit_empty=TRUE)
## [[1]]
## [1] "ab" "c" 
## [[2]]
## [1] "d"  "ef" "g" 
## [[3]]
## [1] "h"
## [[4]]
## character(0)
  • (NEW FUNCTION) #105: stri_list2matrix converts lists of atomic vectors to character matrices, useful in connection with stri_split and stri_extract.
stri_list2matrix(stri_split_fixed(c("ab_c", "d_ef_g", "h", ""), "_", n_max=3, tokens_only=TRUE, omit_empty=TRUE))
##      [,1] [,2] [,3] [,4]
## [1,] "ab" "d"  "h"  NA  
## [2,] "c"  "ef" NA   NA  
## [3,] NA   "g"  NA   NA
  • (NEW FEATURE) #107: stri_split_* now allow setting an omit_empty=NA argument.
stri_split_fixed("a_b_c__d", "_", omit_empty=FALSE)
## [[1]]
## [1] "a" "b" "c" ""  "d"
stri_split_fixed("a_b_c__d", "_", omit_empty=TRUE)
## [[1]]
## [1] "a" "b" "c" "d"
stri_split_fixed("a_b_c__d", "_", omit_empty=NA)
## [[1]]
## [1] "a" "b" "c" NA  "d"
  • (NEW FEATURE) #106: stri_split and stri_extract_all gained a simplify argument (if TRUE, then stri_list2matrix(..., byrow=TRUE) is called on the resulting list.
stri_split_fixed(c("ab,c", "d,ef,g", ",h", ""), ",", omit_empty=TRUE, simplify=TRUE)
##      [,1] [,2] [,3]
## [1,] "ab" "c"  NA  
## [2,] "d"  "ef" "g" 
## [3,] "h"  NA   NA  
## [4,] NA   NA   NA
stri_split_fixed(c("ab,c", "d,ef,g", ",h", ""), ",", omit_empty=FALSE, simplify=TRUE)
##      [,1] [,2] [,3]
## [1,] "ab" "c"  NA  
## [2,] "d"  "ef" "g" 
## [3,] ""   "h"  NA  
## [4,] ""   NA   NA
stri_split_fixed(c("ab,c", "d,ef,g", ",h", ""), ",", omit_empty=NA, simplify=TRUE)
##      [,1] [,2] [,3]
## [1,] "ab" "c"  NA  
## [2,] "d"  "ef" "g" 
## [3,] NA   "h"  NA  
## [4,] NA   NA   NA
  • (NEW FUNCTION) #77: stri_rand_lipsum generates (pseudo)random dummy lorem ipsum text.
   stri_wrap(stri_rand_lipsum(3), 80, simplify=FALSE),
   stri_flatten, collapse="\n"), sep="\n\n")
## Lorem ipsum dolor sit amet, eu turpis pellentesque est, lectus, vestibulum.
## Iaculis et nam ad eu morbi, ultrices enim pellentesque est fusce. Etiam
## ipsum varius, maecenas dapibus. Netus molestie non adipiscing netus,
## aptent sed malesuada, placerat suscipit. A, sed eu luctus imperdiet odio
## tempor. In velit ut vel feugiat felis eros risus. Sed sapien, facilisis
## ullamcorper, senectus efficitur sit id sociis sed purus. Ipsum, a, blandit
## faucibus. In vivamus, duis et sed sollicitudin maximus. Sodales magnis
## ac senectus facilisis, dolor faucibus a. Cursus in cum, cubilia egestas
## ut platea turpis. Maximus sit vel cursus nec in vel, eu, lacinia in ut.
## Libero maximus potenti penatibus amet nisl non ut. Commodo nullam rhoncus,
## bibendum quisque sem aliquam sed, quam enim et, sed. Lacinia netus inceptos
## sapien nostra tincidunt facilisis montes nascetur non pharetra convallis
## id. Netus diam nulla montes nec tincidunt facilisis eros porttitor nisl urna
## cubilia. Aliquet egestas mus nisl, nisi vehicula, ac mauris rutrum, felis
## aenean tristique magna. Ante maecenas phasellus id class. Finibus iaculis purus
## volutpat posuere phasellus magna class blandit augue morbi torquent. Taciti
## ullamcorper venenatis at nulla eget auctor ante neque metus sed metus. Dolor,
## platea sit sed pellentesque ipsum. Dapibus sed nisi vestibulum ex integer.
## Duis iaculis sapien habitasse, facilisi habitasse leo nam. Egestas,
## libero tempor purus in. Aliquam himenaeos conubia egestas cum vestibulum
## nec. Sociosqu mauris cum mus non lobortis eu et dapibus vel integer.
## Blandit quis inceptos cursus vel pellentesque lectus amet egestas.
## Pharetra ac eros nisi. Finibus nec, ac congue in molestie sed.
## Tincidunt faucibus a interdum facilisis, sed nulla, tortor, felis,
## sociis. Sem porttitor himenaeos pharetra nec eu torquent elementum.
  • (NEW FEATURE) #98: stri_trans_totitle gained a opts_brkiter parameter; it indicates which ICU BreakIterator should be used when performing case mapping.
stri_trans_totitle("GOOD-OLD cOOkiE mOnSTeR IS watCHinG You. Here HE comes!",
    stri_opts_brkiter(type="word")) # default boundary
## [1] "Good-Old Cookie Monster Is Watching You. Here He Comes!"
stri_trans_totitle("GOOD-OLD cOOkiE mOnSTeR IS watCHinG You. Here HE comes!",
## [1] "Good-old cookie monster is watching you. Here he comes!"
  • (NEW FEATURE) stri_wrap gained a new parameter: normalize.

  • (BUGFIX) #86: stri_*_fixed, stri_*_coll, and stri_*_regex could give incorrect results if one of search strings were of length 0.

  • (BUGFIX) #99: stri_replace_all did not use the replacement arg.

  • (BUGFIX) #94: R CMD check should no longer fail if icudt download failed.

  • (BUGFIX) #112: Some of the objects were not PROTECTed from being garbage collected, which might have caused spontaneous SEGFAULTS.

  • (BUGFIX) Some collator’s options were not passed correctly to ICU services.

  • (BUGFIX) Memory leaks causes as detected by valgrind --tool=memcheck --leak-check=full have been removed.

  • (DOCUMENTATION) Significant extensions/clean ups in the stringi manual.

    Check out yourself. In particular, take a glimpse at stringi-search-regex, stringi-search-charclass and, more generally, at stringi-search.

Enjoy! Any comments and suggestions are welcome.

Tagged with: , , , , , ,
Posted in Blog/R, Blog/R-bloggers

R now will keep children away from drugs

Do you find this plot fancy? If yes, you can find the code at the end of this article BUT if you spend a little time to read it thoroughly, you can learn how to create better ones.

We would like to encourage you and your children (or children you teach) to use our new R package – TurtleGraphics!

TurtleGraphics package offers R-users functionality of the “turtle graphics” from Logo educational programming language. The main idea standing behind it is to inspire the children to learn programming and show that working with computer can be entertaining and creative.

It is very elementary, clear and requires basic algorithm thinking skills, that even children are able to form them. You can learn it in just five short steps.

  • turtle_init() – To start the program call the turtle_init() function. It creates a plot region (called “Terrarium”) and places the Turtle in the middle pointing north.

  • turtle_forward() and turtle_backward() – Argument to these functions is the distance you desire the Turtle to move. For example, to move the Turtle forward for a distance of 10 units use the turtle_forward() function. To move the Turtle backwards you can use the turtle_backward() function.

  • turtle_turn()turtle_right() and turtle_left(). They change the Turtle's direction by a given angle.

  • turtle_up() and turtle_down() – To disable the path from being drawn you can simply use the turtle_up() function. Let's consider a simple example. We use the turtle_up() function. Now, when you move forward the path is not visible. If you want the path to be drawn again you should call the turtle_down() function.

  • turtle_show() and turtle_hide() – Similarly, you may show or hide the Turtle image, using the turtle_show() and turtle_hide() functions respectively. If you call a lot of functions it is strongly recommended to hide the Turtle first as it speeds up the process.

These were just the basics of the package. Below we show you the true potential of it:

  turtle_star <- function(intensity=1){
  y <- sample(1:657, 360*intensity, replace=TRUE)
  for (i in 1:(360*intensity)){
  x <- sample(1:100,1)

One may wonder what turtle_do() function is doing here. It is an advanced way to use the package. The turtle_do() function is designed to call more complicated plot expressions, because it automatically hides the Turtle before starting the operations that results in a faster proceed of plotting.

  drawTriangle<- function(points){
  getMid<- function(p1,p2) c((p1[1]+p2[1])/2, c(p1[2]+p2[2])/2)
  sierpinski <- function(points, degree){
  if (degree  > 0){
  p1 <- matrix(c(points[1,], getMid(points[1,], points[2,]),
  getMid(points[1,], points[3,])), nrow=3, byrow=TRUE)

  sierpinski(p1, degree-1)
  p2 <- matrix(c(points[2,], getMid(points[1,], points[2,]),
  getMid(points[2,], points[3,])), nrow=3, byrow=TRUE)

  sierpinski(p2, degree-1)
  p3 <- matrix(c(points[3,], getMid(points[3,], points[2,]),
  getMid(points[1,], points[3,])), nrow=3, byrow=TRUE)
  sierpinski(p3, degree-1)
  turtle_init(520, 500, "clip")
  p <- matrix(c(10, 10, 510, 10, 250, 448), nrow=3, byrow=TRUE)
  turtle_do(sierpinski(p, 6))
  turtle_setpos(250, 448)

We kindly invite you to use TurtleGraphics! Enjoy!
A full tutorial of this package is available here.

Marcin Kosinski,
Natalia Potocka,

Tagged with: , , ,
Posted in Blog/R, Blog/R-bloggers