The Basics of Data Visualization: A Guide on GGplot2

I would like this post to be a demonstration on Data Visualization with the ggplot package. As you will discover, the package is adept at creating plots that are attractive and appealing. 

There won't be too much explanation on how to interpret these plots. I list a worthy resource for that at the conclusion of this post. Please look into it if this post inspires you.

I would like this post to go into the basics of ggplot2, which is a package that is meant to specialize in the visualization of data. I am using a book called “ggplot2: Elegant Graphics for Data Analysis,” by Hadley Wickham, as well as lecture material I have learned in class. 

I will use one of my favorite data sets called bikes, and the mpg data set later on, but I will not go into great details for this post. I want this post to show you how you can plot two numerical variables on the x and y-axis, and add appropriate labels to make this plot more interpretable.

As always, if you wish to follow along, please type the text that is highlighted in BLUE in your R Studio environment.

library(tidyverse)

## -- Attaching packages --------------------------------------- tidyverse 1.3.1 --

## v ggplot2 3.3.5     v purrr   0.3.4
## v tibble  3.1.4     v dplyr   1.0.7
## v tidyr   1.1.3     v stringr 1.4.0
## v readr   2.0.1     v forcats 0.5.1

## Warning: package 'stringr' was built under R version 4.1.2

## -- Conflicts ------------------------------------------ tidyverse_conflicts() --
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()

setwd("C:/Users/firstnamelastinitial/OneDrive/Documents/Fund of Bus Anal")
bikes <- read.csv("bikes.csv", header = TRUE)
str(bikes)

## 'data.frame':    731 obs. of  10 variables:
##  $ date       : chr  "2011-01-01" "2011-01-02" "2011-01-03" "2011-01-04" ...
##  $ season     : int  1 1 1 1 1 1 1 1 1 1 ...
##  $ holiday    : int  0 0 0 0 0 0 0 0 0 0 ...
##  $ weekday    : int  6 0 1 2 3 4 5 6 0 1 ...
##  $ weather    : int  2 2 1 1 1 1 2 2 1 1 ...
##  $ temperature: num  46.7 48.4 34.2 34.5 36.8 ...
##  $ realfeel   : num  46.4 45.2 25.7 28.4 30.4 ...
##  $ humidity   : num  0.806 0.696 0.437 0.59 0.437 ...
##  $ windspeed  : num  6.68 10.35 10.34 6.67 7.78 ...
##  $ rentals    : int  985 801 1349 1562 1600 1606 1510 959 822 1321 ...

So, I have loaded tidyverse, which contains ggplot2, and my data set. Now I would like to use ggplot to visualize variables within the bikes data frame. 

But first, allow me to summarize the basic format, or grammar, of the ggplot() function.

1.         You need a data frame.

2.         the second argument for this function requires a mapping aesthetic, or aes, which will have the variables you wish to see in graphical form.

3.         After you have satisfied these two arguments for the ggplot() function, you will need include a geom function, which follows a + sign.

Let me demonstrate what this may look like.

ggplot(bikes, aes(x = temperature, y = rentals))+
  geom_point()

Here is what that plot looks like. Throughout this post, I will use the same plot, but I will make it look more presentable, by making use of various functions and phrases. 

I hope that each iteration of this plot will give you an idea, and basic understanding of how ggplot is used, and its effectiveness in presenting your findings to your managers at your place of work.

Let’s change the color of the plot to blue.

ggplot(bikes, aes(x = temperature, y = rentals))+
  geom_point(colour = "blue")

I would like to note that color and shape work well for categorical variables (non-numerical), and for continuous variables (numerical), size might be a better argument.

Now, it might be difficult to determine what type of relationship exists between the variables in our visualization. To me, it looks like there is a moderate (?) positive LINEAR (?) relationship between temperature and rentals. 

So, let’s test my hypothesis by superimposing a regression line over the data. We can do this with the geom_smooth() function, and inserting “lm” into “method” for our argument.

ggplot(bikes, aes(x = temperature, y = rentals))+
  geom_point()+
  geom_smooth(method = "lm")

## `geom_smooth()` using formula 'y ~ x'

My assumption that the variables in question have a linear relationship appears to have only some merit. Although, at this point, I don’t know how strong the correlation is. And now that I think about it, a linear model might NOT fit this data appropriately, as this line does not fit the outer extremities of our data too well. That is the subject of another post. Let’s continue.

In order to keep this post shorter, I want to touch quickly on how you can simply add labels to your graph. Please examine the following code, and note it’s output:

ggplot(bikes, aes(x = temperature, y = rentals))+
  geom_point(colour = "red")+
  geom_smooth(method = "lm")+
  labs(title = "Relationship Between Rentals and Temperature") +
  ylab("Rentals") +
  xlab("Temperature")

## `geom_smooth()` using formula 'y ~ x'

In our final input, we specified that the color of our observations will be red in the geom_point() function. We kept our Regression Line to give management an idea how a linear model might or might not be appropriate for our data set. This was specified in our geom_smooth() function. 

Lastly, we created a title for our graph called, “Relationship Between Rentals and Temperature.” We labeled our y-axis “Rentals” and this is our dependent variable. We labeled our x-axis “Temperature” and this will be our independent variable.

Let me do a comparison of both graphs so we can see how far we have come in our interpretation of the two variables in question.

ggplot(bikes, aes(x = temperature, y = rentals))+
  geom_point()

ggplot(bikes, aes(x = temperature, y = rentals))+
  geom_point(colour = "red")+
  geom_smooth(method = "lm")+
  labs(title = "Relationship Between Rentals and Temperature") +
  ylab("Rentals") +
  xlab("Temperature")

## `geom_smooth()` using formula 'y ~ x'

There are other ways that you can graph variables in ggplot, this goes without saying. Here are some other plots you can experiment with. To demonstrate these variations, I will use the mpg data set.

ggplot(mpg, aes(drv, hwy))+
  geom_jitter()

ggplot(mpg, aes(drv, hwy))+
  geom_boxplot()

ggplot(mpg, aes(drv, hwy))+

  geom_violin()

You can find more information about the advantages and disadvantages of these plots in the book by Hadly Wickham, “ggplot2: Elegant Graphics for Data Analysis,” and I will not go into these elements for this post. I just wish to show you the potential of this powerful and useful package. Here is another plot that you can make:

ggplot(mpg, aes(displ, colour = drv))+
  geom_freqpoly(binwidth = 0.5)

You can explore this further in your own time.

To begin, pick two variables from your data set, try numerical values at first. Alter the default graphs color, add some sort of line to fit the data, and lastly, change the labels of the graph. 

THANK YOU!

I hope that this post gives you a quick solution for visualizing two variables on a plot, and also inspires you to go out and buy Hadley Wickham’s book, “ggplot2: Elegant Graphics for Data Analysis.”

Thank you so much for looking at this post. I hope it was helpful to you in your journey of becoming a data analyst, machine learning leader, or data entrepreneur.

Read More

Project Management Template: Modified Analytics Process for Data Analysts

Greetings again, to all Porn Stars and Sex Workers on the Web, and thank you so much for continuing to learn on my page, designed for you, from the bottom of my heart! I am honored to be creating this content for you.

 

At this time in my studies, I have learned many tools to jump-start a project, and the following tool (called MAP 2.0) is the project management framework that I use when starting class assignments, and extracurricular projects. 

Also, as I will explain later, this framework is MY version of an initial four-step framework called the Analytics Process. That version is a capable model, but it is too broad for our purposes. 

MAP 2.0 (Modified Analytics Process) is more specific, yet general enough for you to start your own analytics projects. The idea is that you will have freedom to complete any project, combined with a strict frame for addressing ambiguity within the data. 

I am encouraging all of you to grind it up, consume it, abuse it, re-use it, and even change MAP 2.0 to whatever project you are working on for data analysis. You will find that different data sets have different questions that need to be answer, and therefore, some of the steps in the framework might need to be revisited, or even eliminated from your project management process.

For instance, you might find that the first Phase, called “Plan,” might not be achievable until you have seen the data. Other times, your instructor, client or manager might have a set of questions for you to answer based on the data.

 

The point is that the steps in this framework are not static. They are in essence meant to be addressed both specifically and broadly, AND in a different order if need be. This framework will typically be used in combination with a coding language (Python) or statistical computing software (R programming), so if some of the content is confusing, no need to worry. 

Come back to this page as often as you like to become familiar with the process... you will be using it quite a bit!

 

For now, here is what MAP 2.0 looks like:

 

Steps in Brief

 

1. Plan- “Define Goals”

            Insights

2. Collect- “Lock, Load and Clean”

            Insights

3. Explore- “Learn Your Data”

            Insights

4. Exploit- “Use Statistical Approaches for Problem Solving”

            Insights

5. Report- “Persuade Management”

6. Managerial Implications- “What Actions are Acceptable Given the Data”

7. Gratitude- “Thank Your Audience and Sponsors”

 

Steps in More Detail

 

1. Plan: Define Goals

A. Objective: What is your business question? Or problem that you are trying  to solve?

B. Complimentary questions/considerations: What are some restraints on your resources? Are there any key competitors to consider? Any other questions worth asking, that can collaborate with your key objective?

C. Sources: Be sure to gather appropriate literature and materials to help you plan and code more effectively. Be certain to cite these authors and contributors of these resources.

D. Insights: What are some of your hypotheses at this point? Use this space to determine them and how you might proceed when and if you encounter certain elements of the data.

 

Note: There is no coding in R, at this point. The Plan Phase is for brainstorming and meant to stimulate questions to help you prepare for the project. As you progress in your project, you might need to revisit this stage so that you can refine your questions, goals, and create new hypotheses.

 

2. Collect: Lock, Load, and Clean

A. Gather the appropriate data via web scrape techniques, or via the use of csv files (you will mostly use csv files).

B. Load data into your analytics environment (like R Studio, or Jupyter), and any packages that might be of use to you, like tidyverse.

C. Clean data, transform it, modify it, wrangle with it so that it is suitable for supervised and unsupervised machine learning analysis.

D. Insights: What data sets are important for your analysis? Which variables (columns) might you use, which ones might be redundant? Are there any additional columns that you might add to your data frames which will explain more of the ambiguity of your data later on (an increased R-Squared Score)? 

 

Note: Coding in your analytics environment begins to dominate your time and effort, from this phase onward.

 

3. Explore: Learn Your Data

A. Begin to derive insights from the data using functions in the environment like dim(), str(), head(), tail(), View(), and colnames().

B. Understand the data types of the variables that you will be working with.

i. Nominal: labels which have NO overlap (Gender).

ii. Ordinal: labels that are in a specific order (Likert Scale).

iii. Interval: numeric value that is order dependent as well (Temperature).

iv. Ratio: numeric value that has an absolute zero (Weight and Height).

C. Use functions that will uncover descriptive statistics of the data.

D. Dig deeper into the by use of visualizations. Start to visualize with ggplot2.

E: Insights: Is the data linear? Do you need to run any Log transformations that will make a linear model acceptable for analysis? Is the data informing you that a Clustering model might be more preferable than a linear model? Is Classification the way to go? Do you have any other hypotheses at this point that might make the next stage more expeditious?

 

Note: The Analyze Phase will enable you to derive certain “ground truths” about this data and will provide inspiration for making hypotheses that will be tested in the next phase.

 

4. Exploit: Use Statistical Approaches for Problem Solving

A. Now that you have many hypotheses to drive your learning and insights, you can now build supervised and unsupervised machine learning models to make predictions about the future. You can use techniques such as classification, regression, and clustering.

B. Choose an appropriate model that addresses your business question (or problem you are trying to solve) from the Plan Phase.

C. Once you have chosen a suitable model that fits your data, use optimization techniques like "ensembling" to optimize the predictive power of your model (if necessary).

D. Insights: Which models best describe the data set that you are using? Are these models difficult to interpret, and thus might be difficult to explain to management? Might you lean on one model more than another (linear regression vs. classification)?

 

Note: Be sure that you are NOT over fitting or under fitting your model, and that you partition the data into training and testing sets, using a 75/25 split, or other variations like 80/20 or 70/30.

 

5. Report: Persuade Management

A. Use visual aids to show the management team the results of your findings, ggplot2 and R Markdown would be appropriate.

B. Tell a “story” that COMBINES the business problem with analysis of the data. (Spend time on this as it is a difficult task to accomplish)

C. Remember to be clear, and persuasive, as the ultimate goal is for the management team to ACT and MAKE A DECISION on your analysis.

 

Note: Your report should be simplified, as if you were communicating with Laypersons. What good is doing a multi-month analysis if management does not understand the results, and is not persuaded to impact the business in a meaningful way???

 

“No one cares about how amazing your R-Squared Score is… explain how the model answers your business question. What impact does it have on the business problem?” – A revised Thomas Davenport quote

 

6. Managerial Implications: What Actions are Acceptable Given the Data

A. This section should be concise and EXTREMELY clear, 4 sentences at most. It is meant to alleviate any confusion about WHAT to do next.

B. Actions SHOULD be informed by close-ended questions that require “yay” or “nay” in response, like “The marketing team should NOT spend $45,000 for the next Super Bowl campaign.”

C. A leader should be able to use this section of your project to be CONFIDENT about making an impactful yet ethical decision. There should be NO confusion here.

 

Note: If this project presentation is given in a meeting at office setting, management should NOT be confused when they leave the room. They should have a clear path on which actions to take.

 

7. Gratitude: Thank Your Audience and Sponsors

A. This section is short, although it can be greater in length than Managerial Implications.

B. Thank your AUDIENCE for their attention.

C. Thank your SPONSORS, MENTORS, and THOUGHT LEADERS who helped you on the project.

D. Cite any additional sources so that external parties are given due credit and recognition.

E.  Finally, if you are using this framework for extracurricular projects for self-growth or competitions, be sure to praise and consider donating to open source software like R Studio, if you feel that such sources are helpful in your professional and academic journeys.

 

Note: Be humble, be gracious, and give abundant credit to those who have helped you.

 

Sources:

 

The original 4-step “Analytics Process (AP)” was initially introduced to me by a Professor of the Kellstadt Graduate School of Business at DePaul University. My professor presented this information to me and her students in a class called “Fundamentals of Business Analytics,” in Chicago, IL, 2022. I would like to give her credit for the “Managerial Implications” section of MAP 2.0.

 

I would like to credit the “Insights” portion of each of the first four steps to the Director of the Program of the Kellstadt Graduate School of Business at DePaul University. He is an educator and thought leader on Data Visualization and Data Communication. His insight is always appreciated.

 

Additional insights were provided from other resources by other “Big Data” thought leaders like Thomas Davenport (books, and articles), and Machine Learning Techniques literature (Wiley, HBR, Packt>).

THANK YOU!

I hope this template is useful to you in your journey to becoming data analysts and data entrepreneurs. Continue to show courage in the face of the toxic Sex Work Industry. Your hard work and dedication WILL pay off. Thank you for visiting my blog!

Read More

Database Management Post 2: SQL, From The Beginning

Where better to start than from the beginning? I would like to use this post to achieve the following objective:

 

1.         Create a basic database by using DDL and DML statements.

 

 

SQL is divided into two main families of statements, DDL and DML.

 

 

DDL, or Data Definition Language, encompasses the statements CREATE, DROP, ALTER, RENAME. And DML, or Data Manipulation Language contains the statements SELECT, INSERT, UPDATE, DELATE. 

These statements have different functionality within the entire framework of SQL and will be identified more specifically later.

 

Let’s start!

 

We will begin by loading the “sqldf” package into our environment and turn off any warnings that we might receive.

 

options(warn=-1)
library(sqldf)

## Loading required package: gsubfn

## Loading required package: proto

## Loading required package: RSQLite

 

Great! Now we can create our first relational database. Let’s create a database that tracks income over a period of time. (We won’t be focused on specificity or complexity at first, however). We will call our database, INCOME. Here is how we will create it:

 

db <- dbConnect(SQLite(), dbname="INCOMEdb.sqlite")

 

Notice that we have created an object called “db,” short for database, and that we have given it the title of “INCOME” as expressed in the dbname argument.

 

In that database, we will create our first table called MEMBERS. In that table, we will add four columns. This table is meant to be basic, at first, as we can add columns later in our journey to have more specific information regarding the observations within the table. 

For now, this is what the code looks like if we add the four columns, MNo (as in member number), Name, Earnings, Gender (Male or Female).

 

sqldf("CREATE TABLE MEMBERS
            (MNo INTEGER PRIMARY KEY,
            Name TEXT NOT NULL,
            Earnings INTEGER NOT NULL,
            Gender TEXT NOT NULL
            )",dbname="INCOMEdb.sqlite")

## data frame with 0 columns and 0 rows

 

Before I display the table in our environment, I would like to describe the attributes and constraints that I have given each column. This is important, as it helps the algorithm determine what types of information is related to that column.

 

Firstly, I used the CREATE TABLE statement, and created a table called MEMBERS. Then I gave the column called “MNo” the attribute of an INTEGER (which is a whole number), and it is a Primary Key, as in it is unique attribute related this table.

 

Next, I gave the “Name” column the attribute of TEXT. Meaning that the information stored in this column will have text, and not a number. I also constrained this column to be NOT NULL. And this means that the values in this column are known and NOT MISSING.

 

The “Earnings” column will also be an integer, as incomes are expressed in numbers. It will have the constraint of NOT NULL. This information will be recorded for every person in that is listed in the MEMBERS table, and as a result, there will be no missing information pertaining to “Earnings.”

Finally, Gender will have the attribute of TEXT (to be recorded as Male or Female), and this information will always be recorded in the MEMBERS table, so we will give the constraint of NOT NULL.

 

This table is very basic at the moment. Later on, when creating a database that is more informative, we might add additional columns like “Job Function” or ”Role.”

 

Let’s see what the table looks like in our environment, so far.

 

sqldf("pragma table_info(MEMBERS)",dbname="INCOMEdb.sqlite")

##   cid     name    type notnull dflt_value pk
## 1   0      MNo INTEGER       0         NA  1
## 2   1     Name    TEXT       1         NA  0
## 3   2 Earnings INTEGER       1         NA  0
## 4   3   Gender    TEXT       1         NA  0

 

This output only tells us the attributes and constraints that we have given the columns of the table. We can’t see what the actual table looks like yet as we have only created the theoretical framework for it. Let’s begin the process of adding values to the table.

 

sqldf(c("INSERT INTO MEMBERS VALUES (1, 'Sally', 900, 'Female')",
      "INSERT INTO MEMBERS VALUES (2, 'Fred', 1200, 'Male')",
      "INSERT INTO MEMBERS VALUES (3, 'Mary', 2000, 'Female')"),
      dbname="INCOMEdb.sqlite")

## data frame with 0 columns and 0 rows

sqldf("SELECT * FROM MEMBERS",
      dbname="INCOMEdb.sqlite")

##   MNo  Name Earnings Gender
## 1   1 Sally      900 Female
## 2   2  Fred     1200   Male
## 3   3  Mary     2000 Female

 

Here, I have used the INSERT INTO statement into the MEMBERS table. I have added three rows for demonstrative purposes. Let’s take the first row: It will have a MNo of 1, Sally will be listed under Name, 900 is Sally’s Earnings to date, and Sally’s Gender is Female. The same logic can be applied to the second and third rows in the code.

 

Then we can see the entire MEMBERS table with the SELECT statement, and as you can see, the values programmed in the previous chuck of code are listed in the table.

 

One last point, I would like to touch on is deleting a table, or the DROP statement. This statement drops, or deletes, the entire table from the database. Are here is what that looks like:

 

 

sqldf("DROP TABLE MEMBERS", dbname = "INCOMEdb.sqlite")

## data frame with 0 columns and 0 rows

 

This will conclude my post on generating a basic database in R Studio. In the following posts we will get a deeper dive into SQL, and see what it might look like with more complicated functions and statements. Thank you for your support.

THANK YOU!

Another slight change of pace for this post. SQL is not typically used in R, but with the appropriate package in R Studio, we can program SQL functions in the environment and learn SQL this way. 

I wanted to post this so that you have a basic understanding of SQL, and I will be posting more SQL as it is a skill that is useful in data analytics. 

Thank you so much for visiting my blog! Enjoy your journey into a more "data-fied" world!

Read More

Linear Regression and Regularization: The LASSO algorithm, A Brief Essay

As you all know by now, I enrolled in DePaul University’s MS in Business Analytics program this year. I am loving the program so much. I am learning new ways of solving business problems, and I am meeting such nice folks in small teams that are assigned for homework and other assignments. My professors are amazing as well!

 

At this point, I have been learning R programming for the past 7 months, since before I matriculated into the MS program. And in that time, I have studied mostly linear regression. It is simple, powerful, and relatively easy to interpret. No math wizardry needed for this!

 

For simple linear regression, there are 4 values of interest:

 

1. There is a y-variable. This is known as the “response” variable (dependent variable), and we as analysts are interested in predicting this value.

 

2. There is the y-intercept.

 

3. There is an x-variable, and this is known as the “predictor” variable (independent variable). This variable IMPACTS the response variable. Conversely, the response variable is IMPACTED by the predictor variable. In simple linear regression, there is only one predictor variable, but in multivariate regression there can be two or more.

 

4. There is a very subtle e-value, which is the approximation of all estimation errors. “E” represents the sum of ALL the differences between the actual values for y and the predicted values for y. The goal regression is to find the best values of the b coefficient and the x coefficient such that “E” is minimized.

 

To regression!

 

Running the lm() function brings me such joy, it’s a strange sort of joy. I mean, what’s not to love? You pass two variables into the function, along with the name of the data set, and BOOM, R Studio will spit out a formula with very specific numbers that are attached to the four points listed above. 

If you don’t like your R Squared score—that pesky thing just never seems to get to 1—then you can try a different predictor variable to see if your model improves.

 

This strategy seems to work well, adding and subtracting a predictor until you find a decent model worth replicating. But what if you had dozens if not hundreds of variables in your data set?  

Further, what if you could add more than one predictor variable to your model that will be able to explain more of the data that you are trying to account for? On top of that, what if you had a way of determining which variable is statistically insignificant with regards to all the other variables in your data?

 

This is where Regularization and Feature Selection comes into play.

 

To make use of Regularization (the maximization of more significant variables) and Feature Selection (the minimization of less significant variables), you can use a specific method known as LASSO regression.

In short, the LASSO “penalizes more variables in the regression,” confirmed by one of my professors at DePaul University Kellstadt Graduate School of Business. This penalization reduces the chances of our model “overfitting” the data. This is ideal as an overfitting of the data will lead to a model predicting perfectly with data that it has seen, and poorly with data that it hasn’t seen.

 

So, by eliminating statistically insignificant variables, and keeping the regression equation to 2 or 3 predictors, you can reduce your chances of overfitting, AND increase the predictive power of your model. This is all thanks to our good friend LASSO! Thank you, my good friend LASSO!

 

I am going to post a Word document with all the code to run a very basic LASSO regression. This code was sourced from a book called, “Introduction to Machine Learning with R: Rigorous Mathematical Modeling” by Scott V. Burger.

 

However, I will not be using this algorithm with the data set he uses. Instead, I will be using the bikes.csv data set that is provided in another book called “Practical Machine Learning in R,” by Notre Dame Professors Fred Nwanganga and Mike Chapple.

 

The data sets that these authors used with their demonstration of their algorithms were meticulously sourced by them, and as such, I strongly encourage you to purchase their works. I promise you will learn so much, and these books are just so rewarding to learn from.

 

Thank you so much for reading, and I hope you enjoy the LASSO process.

 

Sources for your Interest:

 

Lecture and Class Materials presented at DePaul University, Kellstadt Graduate School of Business, in a class entitled “Fundamentals of Business Analytics,” 2022, Chicago, Illinois.

 

“Practical Machine Learning in R,” by Fred Nwanganga and Mike Chapple, Professors at Notre Dame Mendoza College of Business, 2020.

 

“Introduction to Machine Learning with R: Rigorous Mathematical Modeling,” by Scott V. Burger, 2018.

 

Read More