```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE,warning=F)
options(scipen=999)
library(tidyverse)    #Essential Functions
library(rvest)        #Read Tables From Webpages
```

# Introduction

This tutorial is the second part in our web scraping adventure. The R package `rvest` was developed by Hadley Wickham to provide a user-friendly web scraping experience. Combining `rvest` with a CSS selector tool, such as [SelectorGadget](https://selectorgadget.com/), or built-in browser developer tools, makes this entire process more efficient and less painful. This tutorial like the last will be guided and completed in stages during class. 

In the last tutorial, we created 3 CSV files with primary and foreign keys. These files should be in the same folder as this RMarkdown file. The code below loads previously created CSV files into the global environment
```{r, message=F}
VIOLENT=read_csv("FINAL_VIOLENT.CSV")
ZIP=read_csv("FINAL_ZIP.CSV")
ABBREV=read_csv("FINAL_STATE_ABBREV.CSV")
```

The data frame `VIOLENT` links city and state names to violent crime statistics, the data frame `ZIP` links city and state abbreviations to latitude and longitude, and the data frame `ABBREV` links state names to state abbreviations. All 3 of these tables are required to join geographical locations to the violent crime statistics. 

# Part 1: Connection to Population Change and Density

Recall the Wikipedia page listing the [crime rates](https://en.wikipedia.org/wiki/List_of_United_States_cities_by_crime_rate) of US cities. This information comes from an FBI report from 2017. Suppose we wanted to investigate the effect of crime in 2017 on the population data in 2019. An HTML table, provided online, reports the [2019 population estimate](https://en.wikipedia.org/wiki/List_of_United_States_cities_by_population) of 317 US cities.

- Why is this information useful when related to violent crimes?
- Which columns do you think are useful?
- What would you expect to discover?

## Chunk 1: Web Scraping and Variable Selection

We start by web scraping the population statistics table into R using functions from `rvest`. We only select the variables of interest `2019 estimate`, `2016 land area` and `2016 population density` and the keys `City` and `State` required to link this information to violent crimes. 

```{r}
URL.US_CENSUS = "https://en.wikipedia.org/wiki/List_of_United_States_cities_by_population"
US_CENSUS = URL.US_CENSUS %>%
              read_html() %>%
              html_nodes(xpath = '//*[@id="mw-content-text"]/div[1]/table[5]') %>%
              html_table(header=T,fill=T) %>%
              .[[1]] %>%
              select(c(2,3,4,8,10))
str(US_CENSUS)
```

## Chunk 2: Population, Land Area and Density Variables to Numeric

The column names are messy and the last three columns should be numeric. The following code starts by giving the columns more succinct names and then continues to provide the necessary mutations required to convert character vectors to numeric vectors. 

```{r}
US_CENSUS2 = US_CENSUS %>%
      rename(Name=City, State=names(US_CENSUS)[2],Pop_2019=names(US_CENSUS)[3],Land=names(US_CENSUS)[4],Density_2016=names(US_CENSUS)[5]) %>%
      mutate(Pop_2019=as.numeric(str_replace_all(Pop_2019,",","")))%>%
      mutate(Land=as.numeric(str_replace_all(Land,",|km2",""))) %>%
      mutate(Density_2016=as.numeric(str_replace_all(Density_2016,",|/km2","")))

US_CENSUS$`2019estimate`[1] # Example
str_replace_all(US_CENSUS$`2019estimate`[1],",","") # Test

str(US_CENSUS2)
```

## Chunk 3: Create Change and Density (2019) Variables and Clean the Name Column
Now we have population information in 2019 and 2016. We can create a variable `Change` to represent the population change from 2016 to 2019 and a variable `Density` to represent population density in 2019. Note that the `Name` column contains unwanted footnote characters. The following code creates `Change` and `Density` variables, clean the `Name` column and keep variables we are interested in.
```{r}
US_CENSUS3 = US_CENSUS2 %>%
      mutate(Pop_2016=Density_2016*Land, Change=round((Pop_2019-Pop_2016)/Pop_2016*100,2)) %>%
      mutate(Density=round(Pop_2019/Land)) %>%
      mutate(Name=str_replace_all(Name,"\\[.\\]|\\[..\\]","")) %>%
      select(1,2,7,8)

str(US_CENSUS3)
write_csv(US_CENSUS3,"FINAL_CENSUS.CSV")
```


# Part 2: Inclusion of Expert Opinion

An article at https://www.securitysales.com/fire-intrusion/2018-safest-most-dangerous-states-us/ summarizes a study performed by [WalletHub](https://wallethub.com/edu/safest-states-to-live-in/4566/) which compared all 50 states using 48 key safety indicators. The article provides a link to the source along with lists providing the 10 most safe and 10 most dangerous states. 

## Chunk 1: Web Scraping Top 10 Most Safe and Least Safe States

Using [SelectorGadget](https://selectorgadget.com/), we point-and-click the text information we want on the webpage. This is the list of state names which comprise the most safe and most dangerous states. After deselecting the information we don't want, we can identify the CSS selector that links to the HTML element we want to scrape. Specifically for this case, the CSS selector is *#articleContentWrapper li*. The `html_nodes()` function in `rvest` locates in the HTML code where this CSS selector is being used to present the information visually on the webpage. Since this information is text, we use `html_text()` to read the list of states from the subsetted HTML code into a vector names `SAFE_VS_DANGEROUS`. 

```{r}
URL.SAFE_VS_DANGEROUS = "https://www.securitysales.com/fire-intrusion/2018-safest-most-dangerous-states-us/"
SAFE_VS_DANGEROUS = URL.SAFE_VS_DANGEROUS %>%
                      read_html() %>%
                      html_nodes(css="#articleContentWrapper li") %>%
                      html_text() 


print(SAFE_VS_DANGEROUS)
```


## Chunk 2: Create a Data Frame Useful for Joining

Upon inspection of `SAFE_VS_DANGEROUS`, we notice that this vector of length *20* contains the top 10 most safe states and most dangerous states in the order they appear on the website. We create  new data frame combining these states with their classification, according to the website, that indicates whether a state is considered safe or dangerous. By linking this information to the violent crimes dataset, we can determine analytically the extent to which violent crimes were considered in this expert opinion. This may lead to either disagreement or agreement with the expert opinion. **The most important thing here is that we don't behave like the miserable bloggers who wave their flag of ignorance with anecdotal evidence.**

```{r}
CLASS=c(rep("S", 10),rep("D", 10)) #S=SAFE and D=DANGEROUS
print(CLASS)
SAFE_VS_DANGEROUS2=as.tibble(cbind(SAFE_VS_DANGEROUS,CLASS))
names(SAFE_VS_DANGEROUS2)=c("STATE","CLASS")
write_csv(SAFE_VS_DANGEROUS2,"FINAL_SAFE_VS_DANGEROUS.CSV")
head(SAFE_VS_DANGEROUS2)
tail(SAFE_VS_DANGEROUS2)
```

# Reading
Remember the following lists linking to additional information on this topic.

* [SelectorGadget](https://selectorgadget.com/) is a point-and-click CSS selector useful for identifying the [CSS](https://en.wikipedia.org/wiki/Cascading_Style_Sheets)  or [XPath](https://en.wikipedia.org/wiki/XPath) to the specific portion of the webpage you want to scrape.

* Blog Tutorials for Web Scraping in R Using `library(rvest)`.
    + **[Blog 1](http://blog.corynissen.com/2015/01/using-rvest-to-scrape-html-table.html)**
    + **[Blog 2](https://blog.rstudio.com/2014/11/24/rvest-easy-web-scraping-with-r/)**
    + **[Blog 3](http://categitau.com/using-rvest-to-scrape-data-from-wikipedia/)**
    + **[Blog 4](https://stat4701.github.io/edav/2015/04/02/rvest_tutorial/)**
    + **[Blog 5](https://www.analyticsvidhya.com/blog/2017/03/beginners-guide-on-web-scraping-in-r-using-rvest-with-hands-on-knowledge/)**
    
* Video Tutorials for Web Scraping in R Using `library(rvest)`
    +  **[Video 1](https://www.youtube.com/watch?v=4IYfYx4yoAI)**
    +  **[Video 2](https://www.youtube.com/watch?v=9ATgpE0yTxA)**
    +  **[Video 3](https://www.youtube.com/watch?v=gSbuwYdNYLM)**

* Information on Open Source SelectorGadget
    + **[Blog](https://cran.r-project.org/web/packages/rvest/vignettes/selectorgadget.html)**
    + **[Video](https://www.youtube.com/watch?v=A5LIHrpZBnY)**