Agenda

• Announcements
• Final Project Chat
• Regular Expression (Regex) Basics
• Regex Examples
• Activity: Street or Road?

Announcements

• Book Swap Update
  • Bookstore & SBS both seem fine with the rental swap
• Final Project (Due 12/9 at NOON)
  • Optional Checkpoints
    • Data Preview (11/13)
    • Analysis Strategy Conversation
    • Peer review of complete draft (11/25)
  • DataCamp Alternative
    • Credit awarded for completion of assigned courses listed on DataCamp.com
    • Earn up to 85% credit on Final Project by completing assigned courses
    • “Introduction to R” doesn’t count for this
  • What if I do both??
  • Current Progress as far as I can tell: “Meh… I have plenty of time…”

Here’s How Much Time is Left…

# Date the script was last refreshed
today()

## [1] "2016-11-09"

# Data Set Review
mdy("11-13-2016") - ymd(today())

## Time difference of 4 days

# Sumbit Draft for Peer Review
mdy("11-25-2016") - ymd(today())

## Time difference of 16 days

# Total Remaining
mdy("12-09-2016") - ymd(today())

## Time difference of 30 days

Final Projects - Class Feedback

The offer…

• Optional class feedback
• 1-2 per week
  • first come first served
  • 5 minutes to talk about project (progress, problems, etc)
  • 5 minutes feedback from class & instructors

Any Takers?

Chapter 16: Key Ideas

• Regular expressions allow us to match meaningful patterns in character strings
• Some popular uses:
  • detect whether a pattern is contained in a string (use filter() & grepl())
  • replace the elements of that pattern with something else (use mutate() & gsub())
  • extract a component that matches the patterns (use tidyr::extract())

Some Exploits in the Land of Regex

• Medtronic, Inc - quality monitoring for medical technology
  • Match key word or phrase in offline complaint data (uncommon)
  • Subset of complaint data and evaluate rate of some outcome over time
• PSU Men’s Volleyball
  • Teams now have access to complete data for play in every match
  • Using Regex to help parse the data to gain competitive advantage for PSU
  • (Sort of like Moneyball for Volleyball…)
• Scraping HTML data
  • We scraped the Mile Run World Records from Wikipedia
  • we could use regex to drop the footnotes in the Date column

Rejecting Footnotes with Regex

page <- "https://en.wikipedia.org/wiki/Mile_run_world_record_progression"
XPATH <- '//*[@id="mw-content-text"]/table'

table_list <- page %>%
  read_html() %>%
  html_nodes(xpath = XPATH) %>%
  html_table(fill = TRUE)

IAAFmen <- table_list[[4]] 
head(IAAFmen, 3)

##     Time Auto         Athlete    Nationality              Date
## 1 4:14.4      John Paul Jones  United States    31 May 1913[5]
## 2 4:12.6         Norman Taber  United States   16 July 1915[5]
## 3 4:10.4          Paavo Nurmi        Finland 23 August 1923[5]
##            Venue
## 1 Allston, Mass.
## 2 Allston, Mass.
## 3      Stockholm

Now we can use mutate() & gsub() to help us clean up the footnotes from Date:

IAAFmen %>%
  mutate(Date = gsub("\\[.\\]$", "", Date)) %>%
  head(3)

##     Time Auto         Athlete    Nationality           Date          Venue
## 1 4:14.4      John Paul Jones  United States    31 May 1913 Allston, Mass.
## 2 4:12.6         Norman Taber  United States   16 July 1915 Allston, Mass.
## 3 4:10.4          Paavo Nurmi        Finland 23 August 1923      Stockholm

How to Survive in the Land of Regex

• Step 1: Memorize the following special characters and their use: \d, \w, \S, [0-9], [^0-9], [[:lower:]], [[:alnum:]], \W, , ?, ., $, %, |, \<, ^, \, {3}, *, +, \s, \B, \>, \x

How to Survive in the Land of Regex

• NO!!! DO NOT Memorize the special characters!
• Use the RStudio Cheat Sheet: https://www.rstudio.com/resources/cheatsheets/
• Use Google
• Just like everythinkg else in (R) Programming:
  • Don’t start from scratch
  • Find working code doing something kind of similar
  • Make many iterations of small changes checking that each change still works
  • Keep going until the original code evolves into the thing you want!

Homework

• Choose final project data & submit on Canvas for approval (not graded; due 11/13 at 11:59pm)
• DC Ch 16 Exercises: 16.1, 16.2, 16.3
  
• DC chapter 17 reading quiz on Canvas
• Activity: Street or Road?

Activity: Street or Road?

Grading

The assignment is worth a total of 10 points.

• [2 points] Turn in HTML with embedded .Rmd file (e.g. “DataComputing simple” template)
• [2 points] Work through the “Solved Example” section showing progress with each step
• Back to the Streets:
  • [2 points] Your Turn #1: explain each line of code in English (either narrative or commented code)
  • [1 point] Your Turn #2: expand to match several more patterns (at least 12 total)
  • [1 point] Your Turn #2: provide a table in descending order of popularity for the street name identifiers you found
  • [2 points] Your Turn #2: use ggplot to construct a bar chart in descending order of popularity for the street name identifiers you found.

Assignment Remarks:

Two data sets are provided. One includes 15,000 street addresses of registered voters in Wake County, North Carolina. The other includes over 900,000 street addresses of Medicare Service Providers. You can use either data set (or both!) for the activity.

Note: There’s nothing to do in the “For the professional…” section at the very end except to be impressed.