Clustering Analysis of Universities: Schools in Alabama, California, and Florida According to SAT Averages and Admission Rates

Thank you for checking out my post on Clustering earlier last week. I wanted to expanded on the method with an exercise on clustering universities in the states of Alabama, California, and Florida. I hope you have fun with this one. I know I did!

Let’s begin by loading packages, setting our directory and loading our data set. We will be using the “colleges” data set created by the US Department of Education, and cleaned by Fred Nwanganga and Mike Chapple.

As always, if you wish to follow along, you can download the data set on the Google Drive by clicking on the "Data Sets" tab on the homepage. And like all of my other posts, the code that you can replicate is highlighted in BLUE.

library(tidyverse)

## -- Attaching packages --------------------------------------- tidyverse 1.3.1 --

## v ggplot2 3.3.5     v purrr   0.3.4
## v tibble  3.1.4     v dplyr   1.0.7
## v tidyr   1.1.3     v stringr 1.4.0
## v readr   2.0.1     v forcats 0.5.1

## Warning: package 'stringr' was built under R version 4.1.2

## -- Conflicts ------------------------------------------ tidyverse_conflicts() --
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()

setwd("C:/Users/firstnamelastinitial/OneDrive/Documents/R")

college <- read_csv("college.csv", col_types = "nccfffffnnnnnnnnn")

Let’s take the time to do some fundamental exploratory data analysis:

dim(college)

## [1] 1270   17

str(college)

## spec_tbl_df [1,270 x 17] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
##  $ id                : num [1:1270] 102669 101648 100830 101879 100858 ...
##  $ name              : chr [1:1270] "Alaska Pacific University" "Marion Military Institute" "Auburn University at Montgomery" "University of North Alabama" ...
##  $ city              : chr [1:1270] "Anchorage" "Marion" "Montgomery" "Florence" ...
##  $ state             : Factor w/ 51 levels "AK","AL","AR",..: 1 2 2 2 2 2 2 2 2 2 ...
##  $ region            : Factor w/ 4 levels "West","South",..: 1 2 2 2 2 2 2 2 2 2 ...
##  $ highest_degree    : Factor w/ 4 levels "Graduate","Associate",..: 1 2 1 1 1 1 1 1 1 1 ...
##  $ control           : Factor w/ 2 levels "Private","Public": 1 2 2 2 2 2 2 1 2 2 ...
##  $ gender            : Factor w/ 3 levels "CoEd","Women",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ admission_rate    : num [1:1270] 0.421 0.614 0.802 0.679 0.835 ...
##  $ sat_avg           : num [1:1270] 1054 1055 1009 1029 1215 ...
##  $ undergrads        : num [1:1270] 275 433 4304 5485 20514 ...
##  $ tuition           : num [1:1270] 19610 8778 9080 7412 10200 ...
##  $ faculty_salary_avg: num [1:1270] 5804 5916 7255 7424 9487 ...
##  $ loan_default_rate : num [1:1270] 0.077 0.136 0.106 0.111 0.045 0.062 0.096 0.007 0.103 0.063 ...
##  $ median_debt       : num [1:1270] 23250 11500 21335 21500 21831 ...
##  $ lon               : num [1:1270] -149.9 -87.3 -86.3 -87.7 -85.5 ...
##  $ lat               : num [1:1270] 61.2 32.6 32.4 34.8 32.6 ...
##  - attr(*, "spec")=
##   .. cols(
##   ..   id = col_number(),
##   ..   name = col_character(),
##   ..   city = col_character(),
##   ..   state = col_factor(levels = NULL, ordered = FALSE, include_na = FALSE),
##   ..   region = col_factor(levels = NULL, ordered = FALSE, include_na = FALSE),
##   ..   highest_degree = col_factor(levels = NULL, ordered = FALSE, include_na = FALSE),
##   ..   control = col_factor(levels = NULL, ordered = FALSE, include_na = FALSE),
##   ..   gender = col_factor(levels = NULL, ordered = FALSE, include_na = FALSE),
##   ..   admission_rate = col_number(),
##   ..   sat_avg = col_number(),
##   ..   undergrads = col_number(),
##   ..   tuition = col_number(),
##   ..   faculty_salary_avg = col_number(),
##   ..   loan_default_rate = col_number(),
##   ..   median_debt = col_number(),
##   ..   lon = col_number(),
##   ..   lat = col_number()
##   .. )
##  - attr(*, "problems")=<externalptr>

colnames(college)

##  [1] "id"                 "name"               "city"             
##  [4] "state"              "region"             "highest_degree"   
##  [7] "control"            "gender"             "admission_rate"   
## [10] "sat_avg"            "undergrads"         "tuition"          
## [13] "faculty_salary_avg" "loan_default_rate"  "median_debt"      
## [16] "lon"                "lat"

There are a total of 1270 rows, and 17 columns. We can confirm the data types with this function. There are 17 variables.

This time, we will do our clustering analysis on the state of Alabama. So let’s create a new data set with ONLY schools from this state.

alabama_schools <- college %>%
  filter(state == "AL") %>%
  column_to_rownames(var = "name")

After we pass the state code “MD” in the filter() function, we use the column_to_rowname() function which allows us to see the name of the school for our observations. Let’s view the new data set:

View(alabama_schools)

We now have 24 observations, or schools, and 16 variables in our new data set with only schools from Maryland. Now, let’s use the select() function to specify which features we are interested in. 

For this case, we are interested in admission_rate and sat_avg. Sorry for making that decision for you!! 

alabama_schools %>%
  select(admission_rate, sat_avg) %>%
  summary()

##  admission_rate      sat_avg   
##  Min.   :0.4414   Min.   : 811 
##  1st Qu.:0.5309   1st Qu.: 969 
##  Median :0.5927   Median :1035 
##  Mean   :0.6523   Mean   :1033 
##  3rd Qu.:0.8064   3rd Qu.:1109 
##  Max.   :1.0000   Max.   :1219

Before moving on, I would like to observe that we are using the pipe (%>%) operator which allows us to compose cleaner code. Let’s move on:

As expected, we can notice that the range of values between both variables have vastly different ranges. As a reminder, we MUST normalize our variables BEFORE building our model.

alabama_schools_scaled <- alabama_schools %>%
  select(admission_rate, sat_avg) %>%
  scale()

We create a new object, using our alabama_schools data set, that utilizes the scale() function to “normalize” our data to z-scores. This creates an optimal scenario to enable us to use a clustering model.

Let’s see the new values for each variable by piping the summary() function into our new data object.

alabama_schools_scaled %>%
  summary()

##  admission_rate       sat_avg       
##  Min.   :-1.3758   Min.   :-1.92418 
##  1st Qu.:-0.7922   1st Qu.:-0.55343 
##  Median :-0.3884   Median : 0.01916 
##  Mean   : 0.0000   Mean   : 0.00000 
##  3rd Qu.: 1.0051   3rd Qu.: 0.66332 
##  Max.   : 2.2685   Max.   : 1.61547

Our variables have been standardized to z-scores and our data is officially normalized. Remember, by normalizing our data we don’t need to worry about extreme differences between our variables. Let’s load the stats package so we can run the algorithm.

Let’s set the seed so we can have reproducible results.

library(stats)

set.seed(1234)

Notice that we have set our seed to randomize the data. Simply remember the number that you use in the argument so that you can reproduce the results every time you run this algorithm.

Reminder:

The kmeans() functions takes three arguments. Our scaled data set, which recall is alabama_schools_scaled, is the first argument. Centers will determine how many cluster centers we will have. This is essentially our k-value. We are going to set our centers (k-value) to 3, because we want three clusters. The nstart will be set to 25, and this will be the total number of configuration attempts. Let’s run it!

k_3 <- kmeans(alabama_schools_scaled, centers = 3, nstart = 25)

Great! Let’s start to explore this cluster. Let’s see how many observations are in each of the three clusters.

k_3$size

## [1]  8  6 10

This output tells us that one cluster has 8 observations, another has 6 observations, and the third cluster has the remaining 10 observations.

We can also get a sense of the values of the three cluster centers by using the centers attribute.

k_3$centers

##   admission_rate    sat_avg
## 1     -0.6824578  0.5743977
## 2     -0.8282133 -1.2431427
## 3      1.0428942  0.2863675

Let’s break this down per cluster:

The first cluster center has a value of roughly -.68 for admission_rate and .57 for sat_avg.

The second cluster center has a value of roughly -.82 for admission_rate and -1.24 for sat_avg.

The third cluster center has a value of roughly 1.04 for admission_rate and approximately .28 for sat_avg.

THESE VALUES ARE NORMALIZED VALUES OF THE ORIGINAL DATASET!!!

With this in mind, we have lost some ability to interpret this part of the data. This is fine as our objective is to group these schools as accurately as possible.

What good is clustering if we can’t have a visualization of our model. Let’s load another package called factoextra.

library(factoextra)

## Warning: package 'factoextra' was built under R version 4.1.2

## Welcome! Want to learn more? See two factoextra-related books at https://goo.gl/ve3WBa

Next, will pass three arguments into a very powerful function called the fviz_cluster() function. The first argument will be an object where our model is stored, k_3. The second component in the function will be the data set, which is the scaled data set, alabama_schools_scaled. The last element of the function will contain a specification that will help organize our labels.

fviz_cluster(k_3, data = alabama_schools_scaled, repel = TRUE)

## Warning: ggrepel: 1 unlabeled data points (too many overlaps). Consider
## increasing max.overlaps

There is a lot to unpack here, but for brevity, I will describe the clusters as simply as possible.

Cluster 1 is labeled in red and as you can notice it contains schools that have the highest SAT average and lowest admission rates. I would argue that the 8 schools in this cluster are the most elite schools in Alabama as observed by the two metrics that we have used.

Cluster 2 is labeled in green. Universities in this cluster have the lowest SAT average, and are low in admission rates.

Cluster 3 is labeled in blue. Schools in this cluster have high admission rate and have high sat averages; this cluster is similar to cluster 1 with regards to high SAT scores.

I suppose all school clusters will have similar arrangements on two dimensional space if they are measured by same features, sat_avg and admission_rate. 

Let’s see the how the three clusters looks for the state of California, and Florida. Just to test out my hypothesis.

california_schools <- college %>%
  filter(state == "CA") %>%
  column_to_rownames(var = "name")


california_schools %>%
  select(admission_rate, sat_avg) %>%
  summary()

##  admission_rate      sat_avg     
##  Min.   :0.0509   Min.   : 871.0 
##  1st Qu.:0.4051   1st Qu.: 993.5 
##  Median :0.5800   Median :1086.0 
##  Mean   :0.5477   Mean   :1113.7 
##  3rd Qu.:0.7425   3rd Qu.:1222.0 
##  Max.   :0.8750   Max.   :1545.0

california_schools_scaled <- california_schools %>%
  select(admission_rate, sat_avg) %>%
  scale()


set.seed(1234)


k_3 <- kmeans(california_schools_scaled, centers = 3, nstart = 25)


fviz_cluster(k_3, data = california_schools_scaled, repel = TRUE)

## Warning: ggrepel: 58 unlabeled data points (too many overlaps). Consider
## increasing max.overlaps

Now for Florida:

florida_schools <- college %>%
  filter(state == "FL") %>%
  column_to_rownames(var = "name")


florida_schools %>%
  select(admission_rate, sat_avg) %>%
  summary()

##  admission_rate      sat_avg   
##  Min.   :0.2286   Min.   : 803 
##  1st Qu.:0.4481   1st Qu.: 948 
##  Median :0.5188   Median :1065 
##  Mean   :0.5436   Mean   :1057 
##  3rd Qu.:0.6157   3rd Qu.:1153 
##  Max.   :1.0000   Max.   :1330

florida_schools_scaled <- florida_schools %>%
  select(admission_rate, sat_avg) %>%
  scale()


set.seed(1234)


k_3 <- kmeans(florida_schools_scaled, centers = 3, nstart = 25)


fviz_cluster(k_3, data = florida_schools_scaled, repel = TRUE)

## Warning: ggrepel: 19 unlabeled data points (too many overlaps). Consider
## increasing max.overlaps

An observation that I would like to make is that the clusters are labeled different from graphic to graphic. For example, Alabama’s elite schools are labeled red, California’s elite schools are labeled color green, and Florida’s top schools are labeled blue.

Yet, regardless of how the cluster is labeled by color, we can deduce that elite level schools are positioned in the upper left portion of the quadrant.

THANK YOU!

Thank you so much reading this different take on Clustering. This machine learning method is used widely in a Marketing context, namely when managers are looking to categorize, or "segment," groups of costumers.

I am proud of all of you! I know that the steps that you are taking towards a safer, healthier life, are tedious, and at times cumbersome. 

But you know what? 

Your dedication to ridding yourselves of a toxic profession will pay off. You will live happier lives, filled with new found love from family and friends, and even acquaintances who are drawn to your hard work and professional ethic. 

Thank you for allowing me to be part of your journey to becoming data analysts, and data entrepreneurs. 

Read More

Unsupervised Machine Learning For Porn Stars: COVID-19 Cluster Analysis for Miami-Dade County, Critical Status

INTRODUCTION

Hello Porn Stars and all who are employed in the Sex Work Industry. Thank you as always for coming to this page. You have come such a long way in your learning and you should be proud of yourselves. Great work!

I posted, recently, about clustering analysis, which is an UNSUPERVISED MACHINE LEARNING method. 

I wanted this post to give you an idea of how I might complete a report to a supervisor or manager. This includes how I would structure the report, and how I would deliver the results of the report. 

I will use an analytical framework called MAP (or, Modified Analytical Process) to determine the allocation of COVID-19 preventative measures across the state of Florida.

 

The process will look like this:

 

1.         Plan- “Define Goals”

2.         Collect- “Lock, Load and Clean”

3.         Explore- “Learn Your Data”

4.         Exploit- “Use Statistical Approaches for Problem Solving”

5.         Report- “Persuade Management”

 

So, let’s begin the analysis:

 

*** PLAN: “DEFINE GOALS”

I would like to do a cluster analysis of the state of Florida involving all the counties in the state. Using the most current data from the New York Times, it is my goal to determine which counties are at the most critical stages during recent pandemic news, and thus, I wish to offer guidance on which counties should be prioritized for state-wide COVID-19 prevention measures. 

This goal might change depending on what I discover.

(One important caveat is that this data was collected just months after the COVID-19 outbreak. Since then, we have had access to vaccines and other worthy medical solutions for the better part of a year.)

I will use the K-Means algorithm for my Clustering analysis, and I intend to separate counties into 3 groups called: Lowest Priority (or Not Critical), Moderate Priority (Somewhat Critical), and High Priority (Extremely Critical).

 

*** COLLECT: “LOCK, LOAD, and CLEAN”

 

I will load tidyverse into my environment, set my working directory, and store my data into an object called “covid.” Please observe the following bit of code to execute, highlighted in blue:

 

library(tidyverse)

## -- Attaching packages --------------------------------------- tidyverse 1.3.1 --

## v ggplot2 3.3.5     v purrr   0.3.4
## v tibble  3.1.4     v dplyr   1.0.7
## v tidyr   1.1.3     v stringr 1.4.0
## v readr   2.0.1     v forcats 0.5.1

## Warning: package 'stringr' was built under R version 4.1.2

## -- Conflicts ------------------------------------------ tidyverse_conflicts() --
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()

setwd("C:/Users/firstnamelastinitial/Documents/R")

covid <- read.csv("us-counties-recent.csv", header = TRUE)

 

*** EXPLORE: “LEARN YOUR DATA”

 

I will perform some preliminary exploratory techniques to get a superficial sense of the data.

 

dim(covid)

## [1] 97576     6

str(covid)

## 'data.frame':    97576 obs. of  6 variables:
##  $ date  : chr  "2022-01-27" "2022-01-27" "2022-01-27" "2022-01-27" ...
##  $ county: chr  "Autauga" "Baldwin" "Barbour" "Bibb" ...
##  $ state : chr  "Alabama" "Alabama" "Alabama" "Alabama" ...
##  $ fips  : int  1001 1003 1005 1007 1009 1011 1013 1015 1017 1019 ...
##  $ cases : int  13251 50313 5054 5795 13427 2149 4519 28519 7882 4529 ...
##  $ deaths: int  163 608 83 95 204 46 106 544 147 67 ...

colnames(covid)

## [1] "date"   "county" "state"  "fips"   "cases"  "deaths"

 

The data set that I have stored into a data frame called “covid” is 97,576 observations (rows), multiplied by 6 variables (columns). These variables are divided into character types (date, county, and state), and integer types (fips, cases, and deaths).

Understand that character types are not considered numerical. Conversely, integer types are considered as numbers. 

Next, I am going to create an object that focuses on the state of Florida, and its counties, specifically. I will refer to this object as “florida_counties.”

 

florida_counties <- covid %>%
  filter(state == "Florida")

 

In order to further clean the data, I will remove two columns that are irrelevant for this analysis, date and fips.

 

florida_counties <- florida_counties %>%
  select(-date, -fips)

 

Finally, I will remove all duplicated information, keeping only unique rows names, which happen to be the individual counties and their respective statistics.

 

florida_counties <- florida_counties[!duplicated(florida_counties$county), ]

 

From prior exercises in clustering analysis, I know that I need to scale the data accordingly. Not all counties are equal in population.

For instance, some counties like Miami-Dade have enormous populations. Other counties similar to Baker county are much smaller. 

So, I would like to create 3-scaled clusters that take both cases and deaths into a scaled form. The code expressing these changes might look like this:

 

florida_counties_scaled <- florida_counties %>%
  select(cases, deaths) %>%
  scale()

florida_counties_scaled %>%
  summary()

##      cases              deaths       
##  Min.   :-0.48881   Min.   :-0.62228 
##  1st Qu.:-0.45243   1st Qu.:-0.56647 
##  Median :-0.35627   Median :-0.29518 
##  Mean   : 0.00000   Mean   : 0.00000 
##  3rd Qu.:-0.02328   3rd Qu.: 0.09143 
##  Max.   : 6.35542   Max.   : 5.79201

 

Now let’s load the stats library, set our seed, and perform the kmeans() function.

 

library(stats)
set.seed(1234)

 

*** EXPLOIT: “USE STATISTICAL APPROACHES FOR PROBLEM SOLVING”

 

Let’s build our model by creating a new object called “k_3” as I will be using 3 clusters in this model. And the model will run itself 30 times before determining a final optimal model.

 

k_3 <- kmeans(florida_counties_scaled, centers = 3, nstart = 30)

 

Let’s explore our model:

 

k_3$size

## [1]  1 59  8

 

This output tells us that in the 3 clusters that I have specified: there is 1 observation in cluster 1, 59 observations in cluster 2, and 8 observations in cluster 3. Keep in mind that when I use the word observation I am generally referring to the counties. Meaning, each unique country is a single observation.

 

k_3$centers

##        cases     deaths
## 1  6.3554163  5.7920092
## 2 -0.2862105 -0.3002287
## 3  1.3163750  1.4901853

 

The “centers” attribute tells us the location of our 3 centroids. They are the centers of each cluster and they were determined randomly by the algorithm. 

We can begin to visualize our model with the factoextra package. This would be ideal as clusters are meant to be visualized.

 

library(factoextra)

## Warning: package 'factoextra' was built under R version 4.1.2

## Welcome! Want to learn more? See two factoextra-related books at https://goo.gl/ve3WBa

 

We can ignore the warning!

 

fviz_cluster(k_3, data = florida_counties_scaled, repel = TRUE)

## Warning: ggrepel: 58 unlabeled data points (too many overlaps). Consider
## increasing max.overlaps

Let’s see if we can put the county names on a ggplot, along with some labels:

 

k_3$cluster <- as.factor(k_3$cluster)

ggplot(florida_counties, aes(cases,deaths, color = k_3$cluster)) +
  geom_point() +
  geom_text(aes(label=county)) +
  ggtitle("Clusters of Florida Counties based on number of covid cases and de  aths") +
  labs(colour = "Clusters")+
  xlab("Cases per 10,000")

*** REPORT: “PERSUADE MANAGEMENT”

 

I would like to conclude my research at this point. I have enough information to determine which clusters of counties in the state of Florida are at the most risk of current COVID-19 circumstances.

 

As you can see from the plot above there are 3 clusters of counties, and I will label them accordingly:

 

 

RED CLUSTER: Extremely Critical, has high number of cases per 10,000 citizens, and high number of deaths relative to other clusters. 

 

BLUE CLUSTER: Somewhat Critical, has moderate number of cases per 10,000 citizens, and moderate number of deaths compared to red cluster. 

 

GREEN CLUSTER: Not Critical, has a lower number of cases per 10,000 citizens, and low number of deaths compared to both red and blue clusters. 

 

There is only one county that is in the RED CLUSTER: Miami-Dade County.

 

There are 8 counties in the BLUE CLUSTER: Including Broward, Palm Beach, Orange, among others.

 

And the remaining 59 counties in the GREEN CLUSTER have a status of “Not Critical.”

 

**From the information that I have gathered, it is clear that Miami-Dade County is in need of the most resources, whether in the form of ICU assistance, PPE, testing kits, and vaccine allocation. 

This conclusion is not, by any means, meant to be a political statement, as it mainly deals with numbers, cold, unemotional numbers. If this county is not given thoughtful consideration, COVID-19 cases and deaths will reduce at a much slower rate, causing an unnecessary amount to turmoil in the region.**

 

MANAGERIAL IMPLICATIONS

I wish that this contents of this report could guide leaders in the Miami-Dade county to be more proactive about pursuing aid from the state and federal government. If there is ONE TAKEWAY from this analysis, it is that Miami-Dade’s status with regards to COVID-19 is EXTREMELY CRITICAL. 

Excess resources should be divided equally into the next critical cluster, counties that are labeled in blue. And finally, mild attention is needed for counties listed in the green cluster. 

 

Thank you so much for reading my analysis. I hope that we, the citizens of the world, can use data in a way that can impact change within our communities. We can begin to have a conversation about the future of data, its usefulness, and effectiveness when it is interpreted ethically. I hope this project can be of use to leaders in the Miami-Dade medical community.

THANK YOU 

Thank you so much for reading this post. I hope it gives you a simple and concise breakdown for how to conduct a basic report. Depending on the topic, and guidelines given to you by your manager, the report can be much lengthier. 

I am so proud of you for taking these crucial and necessary steps for changing the trajectory of your lives by becoming efficient at data analysis, and machine learning topics. Your hard work will pay off in multiples!

Read More

Covid Analysis for 2020: A Basic Clustering Method to Inform Leaders... Porn Stars Can Inform Leaders Too!

 *** Introduction:

 

Hello ladies, thank you so much for coming to this page, and for continuing to be so supportive. Today's topic is about an UNSUPERVISED MACHINE LEARNING method called CLUSTERING. 

This method detects patterns in the data that any normal person could not see on their own. Well, maybe Wonder Woman can, but she's not normal!

This method is useful for Marketing departments in organizations as they can use Clustering to see if there are any segments, or groups, inside their data that they can target in upcoming sales campaigns. 

Clustering is also helpful for other use cases, like in the Healthcare Industry, for instance. Today, we are going to go over COVID-19 data from two years ago, and group states in categories of Mild, Moderate Priority and High Priority in terms of vaccine supplies.

ALERT: I don't mean for this post to be political at all. Covid has been a touchy subject for everyone on the Planet, and especially for the United States. The subject of covid, and it's dangers, has been politicized by both political parties, conservatives and progressives. 

The reason why I chose Covid for this Clustering analysis is because the data is easy to obtain. Further, you will be able to see how each state, or observation, is grouped due to number of cases and deaths, in a low to high manner. 

I end this post with a sample letter to then President Donald J. Trump, urging him to allocate Covid resources effectively as not all states are suffering equally from this deadly disease. ANYONE, especially Sex Workers, can and should have the inspiration to speak up when they see a problem! In fact, it is our civic duty to do just that! Having a voice should not be for only the privileged folk!

Again, this post is not mean to be political. If you love former President Trump. It is irrelevant! If you hate former President Trump. It is irrelevant! What is important is that this post will give you another tool for tackling Machine Learning and Data Analytics.

With that, I hope you enjoy this one. I had so much fun putting it together. I hope it helps all of you on your journey!

 

*** Idea in Brief:

 

I will be analyzing two cleaned data sets that are related to COVID deaths and cases, and population of each US state. I will take the joined-data set on a journey through the Modified Analytics Process (MAP), which is a process that was introduced to me by one of my professors at Kellstadt Graduate School of business. The 5 stages will be used sequentially, and at times, out of order. It can be summarized as follows:

 

1.         Plan- “Define Goals”

2.         Collect- “Lock, Load and Clean”

3.         Explore- “Learn Your Data”

4.         Exploit- “Use Statistical Approaches for Problem Solving”

5.         Report- “Persuade Management”

 

So then, I will begin this project with the PLAN phase of MAP.

 

*** PLAN: “Define Goals”

 

This part of the analysis is qualitative and might be iterated upon as the process goes on. As I learn more about the data set, its constraints, and how those constraints are related to my goals, I might need to modify my objective(s).

 

Initially, I would like to see if this data is compatible with techniques that were learned in the class, specifically: clustering. This is my main objective. Along with trying to fit this algorithm over the data, I will be focusing on using as many of the functions as I can. Visualizations and other EDA tools (Exploratory Data Analysis) will be considered as a vital component of the overarching analysis.

 

This is the extent to the PLAN phase that I wish to contribute currently. Allow me to move on to the next phase.

 

*** COLLECT: “Lock, Load, and Clean”

 

I will now load the necessary libraries into our environment. Additionally, I will set my directory, and create an object to store the data sets of interest; I will call the objects, “covid” and “population.”

If you wish to follow along, I will post the data sets on the Google Drive that you can access on the home page menu. Click on "Data Sets" and you will be brought to the page. L

Lastly, the code that you need to type is highlighted in blue. Please remember to press CTRL and ENTER after typing the coding highlighted in blue. 

library(tidyverse)

## -- Attaching packages --------------------------------------- tidyverse 1.3.1 --

## v ggplot2 3.3.5     v purrr   0.3.4
## v tibble  3.1.4     v dplyr   1.0.7
## v tidyr   1.1.3     v stringr 1.4.0
## v readr   2.0.1     v forcats 0.5.1

## Warning: package 'stringr' was built under R version 4.1.2

## -- Conflicts ------------------------------------------ tidyverse_conflicts() --
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()

setwd("C:/Users/firstnamelastinitial/OneDrive/Documents/R")

covid <- read.csv("covid.csv", header = TRUE)
population <- read.csv("population.csv", header = TRUE)

 

In the Plan Phase, I mentioned that I would be doing a “join” on the covid data and the population data. Normally, I feel that one would do this near the end of the analysis when they have discovered their data set more thoroughly. However, I already have an idea of what the data will look like graphically due to prior exposure to these sets, so I am going to create an object that will combine certain features of both sets right now. I will call my object “scale,” and it will be referenced later in my analysis.

 

I want to treat the “scale” data frame differently that the data frames labeled “covid” and “population.” My intention is to create more columns that can better represent the information between the 50 states that is now present in the original data.

 

scale <- covid %>% inner_join(population, by = “state”)

 

Since my data has been cleaned prior to loading, I can move on to the next phase of MAP where I will learn more about my data to determine if I can run any, if not all, of the statistical models listed in the plan phase.

 

*** EXPLORE: “Learn Your Data”

 

I will use some basic exploration functions to do a preliminary analysis of the data set so as to best determine which direction I take. I will see if it is more convenient to explore my two initial data sets simultaneously for the sake of brevity in this report.

 

dim(covid)

## [1] 55  4

str(covid)

## 'data.frame':    55 obs. of  4 variables:
##  $ ï..date: chr  "10/20/20" "10/20/20" "10/20/20" "10/20/20" ...
##  $ state  : chr  "Alabama" "Alaska" "Arizona" "Arkansas" ...
##  $ cases  : int  174528 12349 232939 100441 887093 87966 64455 23325 16445 760381 ...
##  $ deaths : int  2805 63 5837 1728 17067 2212 4559 668 642 16104 ...

colnames(covid)

## [1] "ï..date" "state"   "cases"   "deaths"

head(covid)

##    ï..date      state  cases deaths
## 1 10/20/20    Alabama 174528   2805
## 2 10/20/20     Alaska  12349     63
## 3 10/20/20    Arizona 232939   5837
## 4 10/20/20   Arkansas 100441   1728
## 5 10/20/20 California 887093  17067
## 6 10/20/20   Colorado  87966   2212

 

We can see that the covid data frame is 55 observations (rows) multiplied by 4 variables (columns). The four columns are comprised of date, and state, which are both character types (non-numeric), and cases and deaths, which are both integer types (numeric). 

Consequently, these columns are also the names of the variables: date, state, cases, and deaths. The head() function shows us the first 6 observations which are represented by the states of Alabama, Alaska, Arizona, Arkansas, California, and Colorado. Let’s do the same exploration of the population data frame.

 

dim(population)

## [1] 52  2

str(population)

## 'data.frame':    52 obs. of  2 variables:
##  $ state     : chr  "Alabama" "Alaska" "Arizona" "Arkansas" ...
##  $ population: int  4903185 731545 7278717 3017804 39512223 5758736 3565287 973764 705749 21477737 ...

colnames(population)

## [1] "state"      "population"

head(population)

##        state population
## 1    Alabama    4903185
## 2     Alaska     731545
## 3    Arizona    7278717
## 4   Arkansas    3017804
## 5 California   39512223
## 6   Colorado    5758736

 

The population data frame is composed of 52 observations (rows) and 2 variables (columns). This data frame is much smaller. It only has two columns, state and population, which are of character and integer data types respectively. The head() function tells us the population for Alabama, Alaska, Arizona, Arkansas, California, and Colorado.

 

Now that I have some idea of what the data means, I am going to circle back to my objectives in the plan phase. 

First, I would like to see if I can run a clustering algorithm on the data frame entitled “covid.” Then, as I already have an idea of the output, I will use the data frame “scale” to get a more accurate representation of how certain variables relate to the states.

 

I will do a K-means cluster approach, and I will set k equal to 4 (to represent the number of Clusters I wish to see) when I get the opportunity. I am going to set my seed so that I can reproduce my models with 100% equivalency when needed.

 

set.seed(1234)

 

*** EPLOIT: “Use Statistical Approaches for Problem Solving”

 

Now that my seed is set, I will build the model using the kmeans() function. I will use the “covid” data frame and I will index that argument with a vector with all rows contained in the two columns, cases and deaths. The K-means value will be set to 4 clusters, and this model will be iterated upon 20 times to find the best centroids for the clusters.

 

km_covid <- kmeans(covid[, c("cases","deaths")], 4, nstart = 20)

 

I can now interpret the output of the model that has been stored in the object called “km_covid.”

 

km_covid

## K-means clustering with 4 clusters of sizes 17, 3, 32, 3
##
## Cluster means:
##       cases     deaths
## 1 181126.29  5309.8235
## 2 391839.00 16684.6667
## 3  47952.19   932.8438
## 4 842297.67 16938.3333
##
## Clustering vector:
##  [1] 1 3 1 3 4 3 3 3 3 4 2 3 3 3 2 1 3 3 3 1 3 1 1 1 1 3 1 3 3 3 3 1 3 2 1 3 3 1
## [39] 3 3 1 3 3 1 3 1 4 3 3 3 1 3 3 1 3
##
## Within cluster sum of squares by cluster:
## [1] 19901981416 15607035773 45933578173 10096166187
##  (between_SS / total_SS =  95.5 %)
##
## Available components:
##
## [1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss"
## [6] "betweenss"    "size"         "iter"         "ifault"

 

Cluster means for cases and deaths are listed for 4 total clusters.

 

km_covid$cluster

##  [1] 1 3 1 3 4 3 3 3 3 4 2 3 3 3 2 1 3 3 3 1 3 1 1 1 1 3 1 3 3 3 3 1 3 2 1 3 3 1
## [39] 3 3 1 3 3 1 3 1 4 3 3 3 1 3 3 1 3

 

This output gives me each of the observations that have been assigned to each cluster. For instance, I can refer to our first observation in the “covid” data frame, Alabama, and it will be grouped in cluster 1. Our second observation in the “covid” data frame is Alaska, and it will be assigned to cluster 3. One can continue this method of referring the observation to the data frame to the cluster number for all the states.

 

table(km_covid$cluster)

##
##  1  2  3  4
## 17  3 32  3

 

This table shows us the observations or localities assigned to each cluster, and this assignment is due to similarities between the observations with regards to cases and deaths. Cluster 1 will have 17 localities. Cluster 2 will have 3 localities. Cluster 3 will have 32 localities, and Cluster 4 will have 3 localities.

 

km_covid$centers

##       cases     deaths
## 1 181126.29  5309.8235
## 2 391839.00 16684.6667
## 3  47952.19   932.8438
## 4 842297.67 16938.3333

 

This output will show us the values for the centroid observations within our cluster model. I will now visualize the data, but first, I need to use the as.factor() function to make our model easier to graph.

 

km_covid$cluster <- as.factor(km_covid$cluster)

 

I will use ggplot to graph our cluster, and give it a title, as well.

 

ggplot(covid, aes(cases,deaths, color = km_covid$cluster)) +
  geom_point() +
  geom_text(aes(label=state)) +
  ggtitle("Clusters of US State based on number of covid cases and deaths") +
  labs(colour = "Clusters")

This is the graphical output of our model, and I feel that something is off. I think that New York should have its own cluster, ideally, since it is not at all similar with other states in that cluster, Illinois and Georgia. Since our centroids are random in my 4-cluster model, there is no way that I can control where I these centroids exist. 

I am going to try to update our model with 5 clusters to see if I can have a more sensible cluster model. Further, I will have 30 iterations instead of 20. If this fails, I will choose a k-means of 3. My fingers are crossed!

 

km_covid2 <- kmeans(covid[, c("cases","deaths")], 5, nstart = 30)
km_covid2$cluster <- as.factor(km_covid2$cluster)

 

Let’s see the graphical representation of the new model called km_cluster2.

 

ggplot(covid, aes(cases,deaths, color = km_covid2$cluster)) +

  geom_point() +
  geom_text(aes(label=state)) +
  ggtitle("Clusters of US State based on number of covid cases and deaths") +
  labs(colour = "Clusters")

In trying to pick a cluster number that would assign New York in its own cluster, I made the cluster map more complex. This is not ideal! I will now try to assign a k-means of 3 in our model to see if I can rectify my error. Below is what my newer model will look like:

 

km_covid3 <- kmeans(covid[, c("cases","deaths")], 3, nstart = 30)
km_covid3$cluster <- as.factor(km_covid3$cluster)

 

Let’s visualize the newer model:

 

ggplot(covid, aes(cases,deaths, color = km_covid3$cluster)) +
  geom_point() +
  geom_text(aes(label=state)) +
  ggtitle("Clusters of US State based on number of covid cases and deaths") +
  labs(colour = "Clusters")

 

I suppose this is the best I can do for this data frame. Ideally, I would prefer to have New York in its own cluster as I don’t think it is similar to other observations in its cluster which is Cluster 3, in this case. It is clearly an outlier! But there is an issue that I haven’t address yet. Here is the issue:

If we look at the states in the green cluster (cluster 2), we notice, based on prior knowledge, that these states have enormous populations. Florida, Texas, and California are three of the most populated states in the country, so how can we compare all observations appropriately knowing that this model is somewhat biased?

 

I can address this issue by scaling the data. I have already created a data frame, “scale,” that begins to address the conflict. What if I could create new variables of interest that would scale the data? 

I might create two columns that address cases as divided by 100k population, and deaths as divided by 100k population. This would make more sense, and we might even begin to see different clusters that are based on proportion rather than raw deaths and cases. So, let’s begin!

 

final<- scale %>%
  mutate(cases100k=(cases/population)*100000, deaths100k=(deaths/population)*100000)

 

What does the top of this data frame look like?

 

head(final)

##    ï..date      state  cases deaths population cases100k deaths100k
## 1 10/20/20    Alabama 174528   2805    4903185  3559.482   57.20771
## 2 10/20/20     Alaska  12349     63     731545  1688.071    8.61191
## 3 10/20/20    Arizona 232939   5837    7278717  3200.276   80.19270
## 4 10/20/20   Arkansas 100441   1728    3017804  3328.281   57.26018
## 5 10/20/20 California 887093  17067   39512223  2245.110   43.19423
## 6 10/20/20   Colorado  87966   2212    5758736  1527.523   38.41121

 

Now I can see two more columns added to the “scale” data frame called “cases100k” and “deaths100k.” Let me set the seed once more.

 

set.seed(1234)

 

I can create my final model called “km_final,” and I will set k-means equal to 3, and I will have 30 iterations of the model before settling on our final model. I will index our “final” data frame to include all rows and two columns “cases100k,” and “deaths100k.” Let me run it:

km_final <- kmeans(final[, c("cases100k","deaths100k")], 3, nstart = 30)

 

I will use the as.factor() function to make the necessary adjustments before I visualize the new model.

 

km_final$cluster <- as.factor(km_final$cluster)

 

… and the visualization, with labels changed, will look like such:

 

ggfinal <- ggplot(final, aes(cases100k,deaths100k, color = km_final$cluster)) +
  geom_point() +
  geom_text(aes(label=state)) +
  ggtitle("Clusters of US State based on number of covid cases and deaths per 100k") +
  labs(colour = "Clusters")
ggfinal

You know what?! Fine! I will submit! I surrender! Setting the model to 3 clusters looks good, although I still feel that it is not perfect. I will leave this the way it is. Let me interpret this final model in my own words.

 

The first observation that I would like to take notice of is that this new model has shifted some of the states in terms of proportionality rather than total deaths and cases alone. In our first model, Florida, Texas, and California, were positioned all the wayyyyyyyyy on the right side of the plot. But this is only because those three states are the largest by population. In our new scaled model, we have a more accurate representation of covid infection rates by cases and death PER 100,000 citizens.

 

This is a more meaning metric as state governments can have a better idea of how to implement safety guidelines to reduce covid cases and death cases PER its population. The severity of Covid-19 is widespread, BUT is DIFFERENT from state to state. In our new model, Florida is still at a critical level, but so is North Dakota, which is a much smaller state with regards to its population.

 

Notice how Pennsylvania, a heavily populated state, has a different status than Florida, another heavily populated state. These two states give a "bigger-picture-overview" that Covid safety measures should be SPECIFIC to the state’s population. Meaning, Pennsylvania might, and should, have a different strategy of case and death reduction initiatives than Florida.

 

*** REPORT: “Persuade Management”

 

In conclusion, the final cluster model will do a better job of informing the local, state, and federal governments of where, why, and how they should divert resources from one state to another. 

One state might require more ICU assistance, PPE aid, and financial resources to combat cases and deaths, and this is due to the proportion of the population to covid cases and deaths.

 

(Let’s say hypothetically, I wanted to persuade then President Trump to act on covid relief measures, and detail the situation in plain English. I might write to him the following letter)

 

Greetings Mr. President Trump,

I am writing to briefly, and humbly, to act on the Covid situation that is greatly impacting our nation. As you know, there is a shortage of resources that we have, as covid is a new disease. Our supply chains, as you have mentioned in a previous press conference, are stretched thin. Here is a graphical representation of the states that are in the following three clusters. These clusters can be noted as:

 

Cluster 1 (RED): Less Critical 

Cluster 2 (GREEN): Moderately Critical 

Cluster 3 (BLUE): Highly Critical

 

ggfinal <- ggplot(final, aes(cases100k,deaths100k, color = km_final$cluster)) +
  geom_point() +
  geom_text(aes(label=state)) +
  ggtitle("Clusters of US State based on number of covid cases and deaths per 100k") +
  labs(colour = "Clusters")
ggfinal

States that are in the BLUE cluster should be given the highest priority access to covid relief, as these are the states that have the most cases and deaths for every 100,000 citizens, on average. Access to such care is vital for these states. 

Conversely, states that are in the RED cluster should have the lowest priority in your allocation of covid resources. 

And finally, once you have appropriately allocated the highest priority to the BLUE states, you can allocate a moderate amount of resources to the states in the GREEN cluster, as they are your second highest priority.

 

 

What might this allocation ratio look like? I will leave this to your discretion. However, a basic ratio might be 60:30:10, in which:

 

The Highly Critical cluster (BLUE) might receive 60% of covid relief. 

The Moderately Critical cluster (GREEN) might receive 30% of covid relief. 

The Less Critical cluster (RED) might receive 10% of covid relief.

 

Again, I will leave this to the discretion of you and your team, as you are the group that will be held accountable for this decision.

 

Thank you so much for your attention, and for hearing out a concerned citizen. You and your team have reacted as well as possible considering we have been unprepared to handle this black swan event.

I hope you, your family, your staff, and the rest of the world, can remain healthy during this pandemic.

 

Best regards,

Concerned Citizen

 

*** CONCLUSION: “Be Persuasive In Your Report”

 

The goal of the report is NOT to be complicated. The visualizations and the analysis should be clear enough for a layperson to understand. If the implications are clear, what are the most IMMEDIATE actions that a leader should take with your report? 

Try to tell a story with the data if you can, so that leaders can be emotionally tied to the report. This will enable such leaders to be more involved in the decision-making process AND more effective.

THANK YOU!!

This post was more complicated than my previous ones on Regression analysis. Clustering resides in a different branch of Machine Learning called Unsupervised Machine Learning, so that is why the code is different and the interpretation of the output is different as well. The algorithm detects patterns that you can't on its own... it just needs more data to do this!

I am so proud of you for reaching this far into the blog. I hope that your journey is going well, and that you are having fun learning about this subject. It is always so much fun for me to write these posts, so I have y'all to thank for that. Thank you!

Read More