```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE,warning=F)
options(scipen=999)
library(tidyverse)    #Essential Functions
#install.packages("rvest")
library(rvest)        #Read Tables From Webpages
#install.packages("devtools")
library(devtools)
#install_github("cran/noncensus")
library(noncensus)    #Contains Zip Codes for US Cities
```

# Introduction

In this tutorial, we will learn how to scrape the web for information. We will look at data spread across multiple websites that is worthy to be joined and analyzed. The R package `rvest` was developed by Hadley Wickham to provide a user-friendly web scraping experience. Combining `rvest` with a CSS selector tool, such as [SelectorGadget](https://selectorgadget.com/), or built-in browser developer tools, makes this entire process more efficient and less painful. This tutorial will be guided and completed in stages during class. 


# Part 1: Violent Crimes in US Cities

In the early stages of [Wikipedia](https://www.wikipedia.org/), instructors considered it unreliable due to its *openly editable* format. In academia, Wikipedia is not found in research citations, but is often a first stop in the investigation on almost any topic in any field. A 
search may not give you all the information you want, but you will often be provided with helpful links to the source of the information. The search for [Wikipedia](https://en.wikipedia.org/wiki/Wikipedia) on Wikipedia provides you with a historical overview including positives and negatives.

Many Wikipedia pages contain information organized in tables. Consider the [US Crime Rates](https://en.wikipedia.org/wiki/List_of_United_States_cities_by_crime_rate#Crime_rates_per_100.2C000_people_.282012.29) table that provides the crime rates per 100,000 people for major metropolitan areas in the United States. Statistics are given for violent crimes, property crimes and arson. We desire to convert the HTML table into an R data frame object so we can utilize this information in future analysis.

## Chunk 1: Initial Web Scraping Attempt

Using `rvest` functions such as `read_html()` and `html_table()`, we create a data frame called `VIOLENT` that contains all information read from the desired Wikipedia page.

```{r}
URL.VIOLENT="https://en.wikipedia.org/wiki/List_of_United_States_cities_by_crime_rate#Crime_rates_per_100.2C000_people_.282012.29"
VIOLENT = URL.VIOLENT %>%
  read_html() %>%
  html_table(fill=T) 
VIOLENT = VIOLENT[[1]]
str(VIOLENT)
```

```{r}
head(VIOLENT,5)
```

## Chunk 2: Initial Selection of Violent Crimes and Cleaning

You probably noticed an issue with the first row. Why do you think this occurred? Carefully, examine the webpage table to identify the cause. Also, we only want the violent crime statistics for each city and state. What columns do we need to keep?

```{r}
VIOLENT2=VIOLENT[-c(1:2),1:8]
colnames(VIOLENT2)=c("State","City","Population","Total","Murder","Rape","Robbery","Assault")
str(VIOLENT2)
```

```{r}
head(VIOLENT2, 5)
```

## Chunk 3: Fixing Variable Types

The subheadings in the Wikipedia table create problems when parsed to data frame. The presence of this problem causes known numeric variables to be treated as categorical variables. We need to use functions inside and outside `tidyverse` to fix this issue.

```{r}
VIOLENT3=VIOLENT2 %>%
            mutate_at(3:8,as.numeric)
str(VIOLENT3)
```

## Chunk 4: Identifying Problematic Cases in `City`

Using `str_detect()` we can identify cases in `City` where unwanted characters {",","0" to "9"} make an appearance.

```{r}
VIOLENT3[str_detect(VIOLENT3$City,"[,(0-9)]"),]$City
```

```{r}
VIOLENT3[str_detect(VIOLENT3$State,"[,(0-9)]"),]$State
```

## Chunk 5: Cleaning Problematic Cities

The unwanted characters {",","0" to "9"} are special, but not our friends. In Wikipedia, they were used for indicating footnote information. We need to efficiently fix these occurrences by banishing them to the abyss. We see this in both cities and states. Also, there are other scenarios where cities have unusual names that may be problematic when involving additional data. 

```{r}
VIOLENT4 = VIOLENT3 %>%
              mutate(City=str_replace_all(City,"[,(0-9)]","")) %>%
              mutate(State=str_replace_all(State,"[,(0-9)]",""))

VIOLENT5 = VIOLENT4 %>% 
              mutate(City=ifelse(City=="Charlotte-Mecklenburg","Charlotte",City),
                     City=ifelse(City=="Savannah-Chatham Metropolitan","Savannah",City),
                     City=ifelse(City=="Las Vegas Metropolitan Police Department","Las Vegas",City))

VIOLENT6 = VIOLENT5 %>%
              mutate(City=str_replace(City,"Metropolitan",""))

write_csv(VIOLENT6,"FINAL_VIOLENT.CSV")
```


# Part 2: Geographical Locations of US Cities

The [`noncensus`](https://cran.r-project.org/web/packages/noncensus/noncensus.pdf) package in R contains preloaded datasets useful for population studies. Imagine if we wanted to plot the crime information from `VIOLENT6` on a US map. What information would we need?

## Chunk 1: Read in ZIP Data

The data in `zip_codes` provided in the `noncensus` package gives us important geographical information for all US cities.

```{r}
data(zip_codes)
ZIP=zip_codes
str(ZIP)
```


## Chunk 2: Summarize for Joining

Many US metropolitan areas cover multiple zip codes. Using `group_by()` and `summarize()`, we can obtain the average position for each combination of city and state.

```{r}
ZIP2 = ZIP %>%
        group_by(city,state) %>%
        summarize(lat=mean(latitude),lon=mean(longitude),.groups='keep') %>%
        ungroup()
write_csv(ZIP2,"FINAL_ZIP.CSV")
```

## Chunk 3: Assessing Future Join Issues

Run the code below. What makes merging `ZIP2` with `VIOLENT6` problematic? 

```{r}
head(VIOLENT6[,1:4],3)
head(ZIP2,3)
```

# Part 3: Linking State Names to State Abbreviations

Using your favorite search engine, you can look up the government established abbreviations for each state in the US of A. An HTML table found at https://state.1keydata.com/state-abbreviations.php makes this connection for us. However, the way this data is presented is not in the *tidy* spirit. 

## Chunk 1: Web Scraping Table of State Abbreviations

```{r}
URL.STATE_ABBREV = "https://state.1keydata.com/state-abbreviations.php"
STATE_ABBREV = URL.STATE_ABBREV %>%
                read_html() %>%
                html_table(fill=T) %>%
                .[[3]] %>%
                .[-1,]

head(STATE_ABBREV)
```

## Chunk 2: Transforming the Table

```{r}
STATE_ABBREV_TOP = STATE_ABBREV[,1:2]
names(STATE_ABBREV_TOP)=c("State","state")
STATE_ABBREV_BOT = STATE_ABBREV[,3:4]
names(STATE_ABBREV_BOT)=c("State","state")
STATE_ABBREV2=rbind(STATE_ABBREV_TOP,STATE_ABBREV_BOT) %>% arrange(State)
head(STATE_ABBREV2)
write_csv(STATE_ABBREV2,"FINAL_STATE_ABBREV.CSV")
```

# Reading
The following lists contain a lot more information that should be observed and cherished. 

* Blog Tutorials for Web Scraping in R Using `library(rvest)`.
    + **[Blog 1](http://blog.corynissen.com/2015/01/using-rvest-to-scrape-html-table.html)**
    + **[Blog 2](https://blog.rstudio.com/2014/11/24/rvest-easy-web-scraping-with-r/)**
    + **[Blog 3](http://categitau.com/using-rvest-to-scrape-data-from-wikipedia/)**
    + **[Blog 4](https://stat4701.github.io/edav/2015/04/02/rvest_tutorial/)**
    + **[Blog 5](https://www.analyticsvidhya.com/blog/2017/03/beginners-guide-on-web-scraping-in-r-using-rvest-with-hands-on-knowledge/)**
    
* Video Tutorials for Web Scraping in R Using `library(rvest)`
    +  **[Video 1](https://www.youtube.com/watch?v=4IYfYx4yoAI)**
    +  **[Video 2](https://www.youtube.com/watch?v=9ATgpE0yTxA)**
    +  **[Video 3](https://www.youtube.com/watch?v=gSbuwYdNYLM)**

* Information on Open Source SelectorGadget
    + **[Blog](https://cran.r-project.org/web/packages/rvest/vignettes/selectorgadget.html)**
    + **[Video](https://www.youtube.com/watch?v=A5LIHrpZBnY)**