```{r , include=FALSE}
# This is a good place to put libraries required for using the ggplot function
knitr::opts_chunk$set(echo = TRUE)
options(scipen=999)
library(dplyr) #Loads the dplyr package for data manipulation
library(nycflights13) #Loads the nycflights13 package for flights data
```

# Introduction

The main purpose of this tutorial is to practice 4 key functions from **[dplyr](https://dplyr.tidyverse.org/reference/index.html)**. The functions and their purposes are listed as follows:

- `filter()` *Selects Observations Based on Values*

- `arrange()` *Sorts Observations Based on Criteria*

- `select()` or `rename()` *Selects, Deselects, Renames, and Reorders Variables*

- `mutate()` or `transmute()` *Creates New Variables Which Were Originally Nonexistant*

We will practice our skills using the dataset `flights` by loading the R package `nycflights13`. 

# Part 1: The Uniqueness of Flight Numbers

Using `filter()`, start by creating a new tibble called `f1a` that only contains records from flight number 807.

```{r}
f1a<-filter(flights, flight==807)
f1a
```

Now, apply `select()` to create a new dataset `f1b` based on `f1a` only containing variables "flight", "carrier", "origin", and "dest".
```{r}
f1b<-select(f1a, flight, carrier, origin, dest)
f1b
```

Rename the variable "dest" to "destination" in `f1c`.
```{r}
f1c=rename(f1b,destination=dest)
f1c
```

Because `f1c` only contains records for flight number 807, sorting by flight number is irrelevant. Create a new dataset `f1d` that is identical in content to `f1b` but is sorted by "carrier" first, "origin" second, and "destination"" last. Use the function `head()`to display the first 5 rows of the data in `f1d`.
```{r}
f1d<-arrange(f1c,carrier, origin, destination)
head(f1d,5)
```

At first glance, it seems that 807 uniquely represents the flight from __EWR___ to __ATL__ by carrier ___Delta___. To confirm this, create a new dataset `f1e` that is `f1d` sorted by the three variables previously mentioned, all in descending order. The function `desc()` is required here. Follow this by displaying the first 18 rows of the data in `f1d`.

```{r}
f1e<-arrange(f1d,desc(carrier),desc(origin),desc(destination))
head(f1e,18)
```

May we conclude that flight numbers can be mapped to a unique carrier? __No____ (*Yes/No*)

Why or why not?

If we know the flight number, carrier, and origin, can we know for sure what the destination will be? __No___ (*Yes/No*)

Why or why not?

# Part 2: Time to Get on the Pipe

The pipe `%>%` is used in a similar fashion to `+` in the `ggplot2()` package. 

One simple example: Compute the logarithm of $x$, return suitably lagged and iterated differences, compute the exponential function and round the result to 1 decimal place.

```{r}
x <- c(0.109, 0.359, 0.63, 0.996, 0.515, 0.142, 0.017, 0.829, 0.907)
```

Method 1:
```{r}
round(exp(diff(log(x))), 1)
```

Method 2:
```{r}
x %>% log() %>%
    diff() %>%
    exp() %>%
    round(1)
```

It's time to take a magical journey through the tidyverse on the pipe. Start by using `transmute()` to create a new variable "dep_hr" based on "dep_time" which represents departure time in hours since midnight. Recall the code for this is `dep_hr=dep_time%/%100+(dep_time%%100)/60`. In the same step, do this also for "sched_dep_time","arr_time", and "sched_arr_time" naming the new variables "sched_dep_hr", "arr_hr", and "sched_arr_hr", respectively. Save all these new variables to a new tibble called `f2a`. Use the function `names()` to ensure that `f2a` only contains the new variables and the function `head()` to view the top 5 rows.

```{r}
f2a=transmute(flights,
          dep_hr=dep_time%/%100+(dep_time%%100)/60,
          sched_dep_hr=sched_dep_time%/%100+(sched_dep_time%%100)/60,
          arr_hr=arr_time%/%100+(arr_time%%100)/60,
          sched_arr_hr=sched_arr_time%/%100+(sched_arr_time%%100)/60)
names(f2a)
head(f2a,5)
```

Now we can create true delay variables, measured in hours, for both departure and arrival. Using `mutate()`, create a new variable "dep_delay_hr" which equals the difference between "dep_hr" and "sched_dep_hr". Analogously, perform the same operation for arrival. Call the new dataset `f2b`.
```{r}
f2b=mutate(f2a,
           dep_delay_hr=dep_hr-sched_dep_hr,
           arr_delay_hr=arr_hr-sched_arr_hr)
f2b
```


Use the function `percent_rank()` to create a new variable "percent_dep_delay_hr" which represents the percentiles of the variable you previously created. Notice that you can develop variables based on recently transformed variables in the same iteration of `mutate()`, but be careful little buddy because order matters. Name the new dataset `f2c`.
```{r}
f2c=mutate(f2b,
           percent_dep_delay_hr=percent_rank(dep_delay_hr))
f2c
```


Use `filter()` to select the observations where `percent_dep_delay_hr<0.1` or `percent_dep_delay_hr>0.9` in a new dataset `f2d`. The tibble `f2d` will contain the bottom 10% and top 10% of flights based off "dep_delay_hr".

```{r}
f2d<-filter(f2c,percent_dep_delay_hr<0.1|percent_dep_delay_hr>0.9)
f2d
```

Finally, sort the data using `arrange()` from largest to smallest based on the variable "percent_dep_delay_hr". Name the sorted tibble f2e. Use `head()` on `f2e` to show the top 5 flights based on the constructed variable "dep_delay_hr".
```{r}
f2e<-arrange(f2d,desc(percent_dep_delay_hr))
head(f2e,5)
```

Getting the original data from `flights` to `f2e` required multiple steps. If we know what we want to do with the raw data from `flights`, we can use the pipe `%>%` to obtain the same result without intermittently introducing new tibbles into our global environment. The R code below turns your pipe dreams into reality. 
```{r}
f2e.pipedream = 
  flights %>%
  transmute(dep_hr=dep_time%/%100+(dep_time%%100)/60,
            sched_dep_hr=sched_dep_time%/%100+(sched_dep_time%%100)/60,
            arr_hr=arr_time%/%100+(arr_time%%100)/60,
            sched_arr_hr=sched_arr_time%/%100+(sched_arr_time%%100)/60) %>%
  
  mutate(dep_delay_hr=dep_hr-sched_dep_hr,
        arr_delay_hr=arr_hr-sched_arr_hr) %>%
  
  mutate(percent_dep_delay_hr=percent_rank(dep_delay_hr)) %>%
  
  filter(percent_dep_delay_hr<0.1|percent_dep_delay_hr>0.9) %>%
  
  arrange(desc(percent_dep_delay_hr))

head(f2e.pipedream,5)
```

The tibble `f2e.pipedream` is identical to `f2e` in the number of observations (65,737), the number of variables (7), and the order of observations. We can check to see if the tibble `f2e` is identical to `f2e.pipedream` using `identical()`. Can you feel the sensation? It's piping hot up in here.

```{r}
identical(f2e,f2e.pipedream)
```

# Part 3: Measuring Accuracy

How would you measure the accuracy of individual flights? We can say that a flight is accurate if it leaves on time and arrives on time. Suppose we want to create an accuracy measure that captures this information where larger values indicate more inaccurate flights. Try to think of a creative way to measure accuracy by using `mutate()` to construct a new variable named "accuracy". Call the new tibble `f.accuracy`. As a data scientist, the metric you want is not always in the raw data. This is an example of problem that requires a level of innovation for which a job will grant you some cash money.

```{r}
f.accuracy<-mutate(flights,
                dep_hr=dep_time%/%100+(dep_time%%100)/60,
                sched_dep_hr=sched_dep_time%/%100+(sched_dep_time%%100)/60,
                arr_hr=arr_time%/%100+(arr_time%%100)/60,
                sched_arr_hr=sched_arr_time%/%100+(sched_arr_time%%100)/60,
                dep_delay_hr=dep_hr-sched_dep_hr,
                arr_delay_hr=arr_hr-sched_arr_hr,
                accuracy=abs(dep_delay)+abs(arr_delay))
head(f.accuracy,5)
```

Preparing for comparisons of airline carriers on accuracy, we use the `select()` function to create a new tibble named `f.accuracy2` which only contains the variables "carrier" and "accuracy".
```{r}
f.accuracy2=select(f.accuracy,carrier,accuracy)
```

Next, we can evaluate carriers based on their average accuracy across all flights based on our new metric. Furthermore, the standard deviation of this accuracy metric can help us measure the consistency of these airline carriers in performance. The `summarize()` function combined with `group_by()` allows for quick aggregation on the carrier level.

```{r}
carrier.summary<- f.accuracy2 %>%
                  group_by(carrier) %>%
                  summarize(
                    mean.accuracy=mean(accuracy,na.rm=T),
                    sd.accuracy=sd(accuracy,na.rm=T)
                  ) %>%
                  arrange(mean.accuracy)
carrier.summary
```

# Conclusion

Following this tutorial, we will explore the depth of `summarize()`, the most important function in **dplyr**. For right now, just know that not all pipes are created free and equal; however, this pipe `%>%` is straight-up magical. In the early stages, use the aforementioned pipe with caution, but once you develop a tolerance, you are prescribed to use it liberally. 


