Mean, Median and Mode¶
Mean - The average value, Median - The mid point value, Mode - The most common value
speed = [99,86,87,88,111,86,103,87,94,78,77,85,86]
import numpy
speed = [99,86,87,88,111,86,103,87,94,78,77,85,86]
x = numpy.mean(speed)
x
x = numpy.median(speed)
x
from scipy import stats
speed = [99,86,87,88,111,86,103,87,94,78,77,85,86]
x = stats.mode(speed)
x
Standard Deviation¶
Standard deviation is a number that describes how spread out the values are. A low standard deviation means that most of the numbers are close to the mean (average) value. A high standard deviation means that the values are spread out over a wider range.
speed = [86,87,88,86,87,85,86]
x = numpy.std(speed)
x
speed = [32,111,138,28,59,77,97]
x = numpy.std(speed)
x
Variance¶
Variance is another number that indicates how spread out the values are. In fact, if you take the square root of the variance, you get the standard deviation! Or the other way around, if you multiply the standard deviation by itself, you get the variance!
to calculate the variance: 1. find the mean [77.4], 2. for each value find the difference from the mean [32-77.4], 3. for each difference find the square [-45.4*-45.4]. The variance is the average number of these squared differences
(-45.4)**2 # find the square of a number using exponent operator **
import math
math.pow((-45.4),2) # find the square of a number using the math library
(-45.4)*(-45.4) # find the square of a number by multiplying it by itself
speed = [32,111,138,28,59,77,97]
x = numpy.var(speed)
x # calculate the variance of speed
print(math.sqrt(1432.25)) # calculate the standard deviation of speed, using the square root of variance
Percentiles¶
Percentiles are used in statistics to give you a number that describes the value that a given percent of the values are lower than.
ages = [5,31,43,48,50,41,7,11,15,39,80,82,32,2,8,6,25,36,27,61,31]
x = numpy.percentile(ages, 75)
x # print the age that 75% of people are younger than
Data Distribution & Histogram¶
x = numpy.random.uniform(0.0, 5.0, 250)
x # create a data set of 250 floating numbers between 0 and 5 for testing
import numpy
import matplotlib.pyplot as plt
x = numpy.random.uniform(0.0, 5.0, 250)
plt.hist(x, 5) # create a histogram to visualise the numbers in the data set
plt.show()
Normal Data Distribution¶
In probability theory this kind of data distribution is known as the normal data distribution, or the Gaussian data distribution, after the mathematician Carl Friedrich Gauss who came up with the formula of this data distribution.
x = numpy.random.normal(5.0, 1.0, 100000)
plt.hist(x, 100)
plt.show()
Histogram Explained
We use the array from the numpy.random.normal() method, with 100000 values, to draw a histogram with 100 bars. We specify that the mean value is 5.0, and the standard deviation is 1.0. Meaning that the values should be concentrated around 5.0, and rarely further away than 1.0 from the mean. And as you can see from the histogram, most values are between 4.0 and 6.0, with a top at approximately 5.0.
Scatter Plot¶
A scatter plot is diagram where each value in the data set is represented by a dot.
The Matplotlib module has a method for drawing scatter plots, it needs two arrays of the same length, one for the values of the x-axis, and one for the values of the y-axis:
import matplotlib.pyplot as plt
x = [5,7,8,7,2,17,2,9,4,11,12,9,6]
y = [99,86,87,88,111,86,103,87,94,78,77,85,86]
plt.scatter(x, y)
plt.show()
Scatter Plot Explained
The x-axis represents ages, and the y-axis represents speeds. What we can read from the diagram is that the two fastest cars were both 2 years old, and the slowest car was 12 years old.
# random numbers for testing
x = numpy.random.normal(5.0, 1.0, 1000) # The mean set to 5.0 with a standard deviation of 1.0.
y = numpy.random.normal(10.0, 2.0, 1000) # The mean set to 10.0 with a standard deviation of 2.0:
plt.scatter(x, y)
plt.show()
Scatter Plot Explained
We can see that the dots are concentrated around the value 5 on the x-axis, and 10 on the y-axis. We can also see that the spread is wider on the y-axis than on the x-axis.
Linear Regression¶
Regression
The term regression is used when you try to find the relationship between variables. In Machine Learning, and in statistical modeling, that relationship is used to predict the outcome of future events.
Linear Regression
Linear regression uses the relationship between the data-points to draw a straight line through all them. This line can be used to predict future values.
import matplotlib.pyplot as plt
from scipy import stats
# Create two arrays with age(x) and speed(y) of cars
x = [5,7,8,7,2,17,2,9,4,11,12,9,6]
y = [99,86,87,88,111,86,103,87,94,78,77,85,86]
# Execute a method that returns some important key values of Linear Regression:
slope, intercept, r, p, std_err = stats.linregress(x, y)
# Create a function that uses the slope and intercept values to return a new value.
# This new value represents where on the y-axis the corresponding x value will be placed:
def myfunc(x):
return slope * x + intercept
# Run each value of the x array through the function. This will result in a new array with new values for the y-axis:
mymodel = list(map(myfunc, x))
plt.scatter(x, y) # Draw the scatter plot
plt.plot(x, mymodel) # Draw the line of linear regression:
plt.show()
r # how well does my data fit linear regression, r-squared
R-Squared
It is important to know how well the relationship between the values of the x-axis and the values of the y-axis is, if there are no relationship the linear regression can not be used to predict anything.
The relationship is measured with a value called the r-squared. The r-squared value ranges from 0 to 1, where 0 means no relationship, and 1 means 100% related. Python and the Scipy module will computed this value for you, all you have to do is feed it with the x and y values:
speed = myfunc(10) # predict the speed of a car thats 10 years old
speed
Create an example where linear regression is a bad fit
x = [89,43,36,36,95,10,66,34,38,20,26,29,48,64,6,5,36,66,72,40]
y = [21,46,3,35,67,95,53,72,58,10,26,34,90,33,38,20,56,2,47,15]
slope, intercept, r, p, std_err = stats.linregress(x, y)
def myfunc(x):
return slope * x + intercept
mymodel = list(map(myfunc, x))
plt.scatter(x, y)
plt.plot(x, mymodel)
plt.show()
r # a very low r-squared number
Polynomial Regression¶
If your data points clearly will not fit a linear regression (a straight line through all data points), it might be ideal for polynomial regression.
Polynomial regression, like linear regression, uses the relationship between the variables x and y to find the best way to draw a line through the data points.
18 cars registered as they were passing a tollbooth. We have registered the car's speed, and the time of day (hour) the passing occurred. The x-axis represents the hours of the day and the y-axis represents the speed:
x = [1,2,3,5,6,7,8,9,10,12,13,14,15,16,18,19,21,22] # time of day
y = [100,90,80,60,60,55,60,65,70,70,75,76,78,79,90,99,99,100] # speed of cars
mymodel = numpy.poly1d(numpy.polyfit(x, y, 3)) # NumPy method that lets us make a polynomial model
myline = numpy.linspace(1, 22, 100) # Specify how the line will display, we start at position 1, and end at position 22:
plt.scatter(x, y)
plt.plot(myline, mymodel(myline)) # Draw the line of polynomial regression:
plt.show()
from sklearn.metrics import r2_score # Import the relevant library to perform the r-squared calculation
r2_score(y, mymodel(x)) # Print the r-squared number to see if the relationship is good for polynomial regression
speed = mymodel(17) # predict the speed of a car passing the tollbooth at 5pm (17:00)
speed
Create an example where polynomial regression is a bad fit
x = [89,43,36,36,95,10,66,34,38,20,26,29,48,64,6,5,36,66,72,40]
y = [21,46,3,35,67,95,53,72,58,10,26,34,90,33,38,20,56,2,47,15]
mymodel = numpy.poly1d(numpy.polyfit(x, y, 3))
myline = numpy.linspace(2, 95, 100)
plt.scatter(x, y)
plt.plot(myline, mymodel(myline))
plt.show()
r # A very low r-squared number, bad fit
Multiple Regression¶
Multiple regression is like linear regression, but with more than one independent value, meaning that we try to predict a value based on two or more variables.
import pandas
from sklearn import linear_model
df = pandas.read_csv("cars.csv")
X = df[['Weight', 'Volume']] # make a list of the independent values
y = df['CO2'] # dependent values
# From the sklearn module we will use the LinearRegression() method to create a linear regression object.
regr = linear_model.LinearRegression()
# This object has a method called fit() that takes the independent and dependent values as parameters and
# fills the regression object with data that describes the relationship:
regr.fit(X, y)
# predict the CO2 emission of a car where the weight is 2300g, and the volume is 1300ccm:
predictedCO2 = regr.predict([[2300, 1300]])
predictedCO2
We have predicted that a car with 1.3 liter engine, and a weight of 2.3 kg, will release approximately 107 grams of CO2 for every kilometer it drives.
Coefficient
The coefficient is a factor that describes the relationship with an unknown variable.
Example: if x is a variable, then 2x is x two times. x is the unknown variable, and the number 2 is the coefficient. In this case, we can ask for the coefficient value of weight against CO2, and for volume against CO2. The answer(s) we get tells us what would happen if we increase, or decrease, one of the independent values.
regr.coef_ # Print the coefficient values of the regression object weight=0.00755095 and volume=0.00780526
These values tells us that if the weight increases by 1g, the CO2 emission increases by 0.00755095g.
And if the engine size (Volume) increases by 1 ccm, the CO2 emission increases by 0.00780526 g.
predictedCO2 = regr.predict([[3300, 1300]])
predictedCO2
We have predicted that a car with 1.3 liter engine, and a weight of 3.3 kg, will release approximately 115 grams of CO2 for every kilometer it drives.
Which shows that the coefficient of 0.00755095 is correct:
107.2087328 + (1000 * 0.00755095) = 114.75968
107.2087328 + (1000 * 0.00755095)
Scale¶
When your data has different values, and even different measurement units, it can be difficult to compare them. What is kilograms compared to meters? Or altitude compared to time?
The answer to this problem is scaling. We can scale data into new values that are easier to compare.
There are different methods for scaling data, in this tutorial we will use a method called standardization.
The standardization method uses this formula:
z = (x - u) / s
Where z is the new value, x is the original value, u is the mean and s is the standard deviation.
If you take the weight column from the data set above, the first value is 790, and the scaled value will be:
(790 - 1292.23) / 238.74 = -2.1 If you take the volume column from the data set above, the first value is 1.0, and the scaled value will be:
(1.0 - 1.61) / 0.38 = -1.59
Now you can compare -2.1 with -1.59 instead of comparing 790 with 1.0.
You do not have to do this manually, the Python sklearn module has a method called StandardScaler() which returns a Scaler object with methods for transforming data sets.
import pandas
from sklearn import linear_model
from sklearn.preprocessing import StandardScaler
scale = StandardScaler()
df = pandas.read_csv("cars2.csv")
X = df[['Weight', 'Volume']]
scaledX = scale.fit_transform(X)
scaledX
import pandas
from sklearn import linear_model
from sklearn.preprocessing import StandardScaler
scale = StandardScaler()
df = pandas.read_csv("cars2.csv")
X = df[['Weight', 'Volume']]
y = df['CO2']
scaledX = scale.fit_transform(X)
regr = linear_model.LinearRegression()
regr.fit(scaledX, y)
scaled = scale.transform([[2300, 1.3]])
predictedCO2 = regr.predict([scaled[0]])
print(predictedCO2)
Train/Test¶
Train/Test is a method to measure the accuracy of your model.
It is called Train/Test because you split the the data set into two sets: a training set and a testing set.
80% for training, and 20% for testing.
import numpy
import matplotlib.pyplot as plt
numpy.random.seed(2)
x = numpy.random.normal(3, 1, 100)
y = numpy.random.normal(150, 40, 100) / x
plt.scatter(x, y)
plt.show()