Using the Pandas Python Library¶
In this tutorial, you’ll learn how to:
- Calculate metrics about your data
- Perform basic queries and aggregations
- Discover and handle incorrect data, inconsistencies, and missing values
- Visualize your data with plots
You’ll also learn about the differences between the main data structures that Pandas and Python use. To follow along, you can get all of the example code in this tutorial at the link below:
nba dataset¶
In [1]:
import requests
download_url = "https://raw.githubusercontent.com/fivethirtyeight/data/master/nba-elo/nbaallelo.csv"
target_csv_path = "nba_all_elo.csv"
response = requests.get(download_url)
response.raise_for_status() # Check that the request was successful
with open(target_csv_path, "wb") as f:
f.write(response.content)
print("Download ready.")
In [3]:
import pandas as pd
nba = pd.read_csv("nba_all_elo.csv")
type(ds)
Out[3]:
In [4]:
len(nba)
Out[4]:
In [5]:
nba.shape
Out[5]:
In [6]:
nba.head()
Out[6]:
In [7]:
pd.set_option("display.precision", 2)
In [8]:
nba.tail()
Out[8]:
In [9]:
nba.info()
Getting to know the data¶
In [10]:
nba.describe()
Out[10]:
In [11]:
import numpy as np
nba.describe(include=np.object)
Out[11]:
In [12]:
nba["team_id"].value_counts()
Out[12]:
In [13]:
nba["fran_id"].value_counts()
Out[13]:
In [14]:
nba.loc[nba["fran_id"] == "Lakers", "team_id"].value_counts()
Out[14]:
In [17]:
nba.loc[nba["team_id"] == "MNL", "date_game"].agg(("min", "max"))
Out[17]:
In [19]:
nba.loc[nba["team_id"] == "MNL", "date_game"].max()
Out[19]:
In [20]:
nba.loc[nba["team_id"] == "MNL", "date_game"].min()
Out[20]:
Getting to know Pandas Data Structures¶
In [24]:
revenues = pd.Series([5555, 7000, 1980])
revenues
Out[24]:
In [25]:
revenues.values
Out[25]:
In [26]:
revenues.index
Out[26]:
In [27]:
type(revenues.values)
Out[27]:
In [28]:
city_revenues = pd.Series([4200, 8000, 6500], index=["Amsterdam", "London", "New York"])
In [29]:
city_revenues
Out[29]:
In [31]:
city_employee_count = pd.Series({"Amsterdam": 5, "London": 8})
city_employee_count
Out[31]:
In [32]:
city_employee_count.keys()
Out[32]:
In [33]:
"Tokyo" in city_employee_count
Out[33]:
In [34]:
"London" in city_employee_count
Out[34]:
In [35]:
city_data = pd.DataFrame({"revenue": city_revenues, "employee_count": city_employee_count})
In [36]:
city_data
Out[36]:
In [37]:
city_data.index
Out[37]:
In [39]:
city_data.values
Out[39]:
In [40]:
city_data.axes
Out[40]:
In [41]:
city_data.axes[0]
Out[41]:
In [42]:
city_data.axes[1]
Out[42]:
In [43]:
city_data.keys()
Out[43]:
In [44]:
"revenue" in city_data
Out[44]:
In [45]:
"Tokyo" in city_data
Out[45]:
In [46]:
nba.index
Out[46]:
In [47]:
nba.axes
Out[47]:
In [50]:
"pts" in nba.keys()
Out[50]:
Accessing series elements¶
In [51]:
city_revenues["London"]
Out[51]:
In [52]:
city_revenues[1]
Out[52]:
In [53]:
city_revenues[-1]
Out[53]:
In [54]:
city_revenues[1:]
Out[54]:
In [56]:
city_revenues["New York"]
Out[56]:
In [58]:
colors = pd.Series(["red", "purple", "blue", "green", "yellow"], index=[1, 2, 3, 5, 8])
In [60]:
colors
Out[60]:
In [61]:
colors.loc[1]
Out[61]:
In [62]:
colors.iloc[1]
Out[62]:
In [63]:
colors.iloc[1:3]
Out[63]:
In [65]:
colors.loc[3:8]
Out[65]:
In [66]:
colors.iloc[-2]
Out[66]:
In [67]:
city_data["revenue"]
Out[67]:
In [68]:
type(city_data["revenue"])
Out[68]:
In [69]:
city_data.revenue
Out[69]:
In [70]:
city_data
Out[70]:
In [71]:
toys = pd.DataFrame([
{"name": "ball", "shape": "sphere"},
{"name": "rubik's cube", "shape": "cube"}
])
In [72]:
toys["shape"]
Out[72]:
In [73]:
city_data.loc["Amsterdam"]
Out[73]:
In [74]:
city_data.iloc[1]
Out[74]:
In [75]:
nba.iloc[-2]
Out[75]:
In [78]:
nba[
(nba["_iscopy"] == 0) &
(nba["pts"] > 100) &
(nba["opp_pts"] > 100) &
(nba["team_id"] == "BLB")
]
Out[78]:
In [79]:
nba[
(nba["_iscopy"] == 0) &
(nba["team_id"].str.startswith("LA")) &
(nba["year_id"] == 1992) &
(nba["notes"].notnull())
]
Out[79]:
Grouping and aggregating the data¶
In [80]:
city_revenues.sum()
Out[80]:
In [81]:
city_revenues
Out[81]:
In [83]:
city_revenues.max()
Out[83]:
In [84]:
points = nba["pts"]
In [85]:
type(points)
Out[85]:
In [86]:
points.sum()
Out[86]:
In [87]:
nba.groupby("fran_id", sort=False)["pts"].sum()
Out[87]:
In [89]:
maxpoints = nba.groupby("fran_id", sort=False)["pts"].max()
In [90]:
maxpoints.max()
Out[90]:
In [91]:
nba[
(nba["fran_id"] == "Spurs") &
(nba["year_id"] > 2010)
].groupby(["year_id", "game_result"])["game_id"].count()
Out[91]:
In [93]:
nba[
(nba["fran_id"] == "Warriors") &
(nba["year_id"] == 2015)
].groupby(["is_playoffs", "game_result"])["game_id"].count()
Out[93]:
Manipulating Columns¶
In [94]:
df = nba.copy()
In [95]:
df.shape
Out[95]:
In [96]:
df["difference"] = df.pts - df.opp_pts
df.shape
Out[96]:
In [97]:
df["difference"].max()
Out[97]:
In [98]:
renamed_df = df.rename(
columns={"game_result": "result", "game_location": "location"}
)
In [99]:
renamed_df.info()
In [100]:
df.shape
Out[100]:
In [101]:
elo_columns = ["elo_i", "elo_n", "opp_elo_i", "opp_elo_n"]
In [103]:
df.drop(elo_columns, inplace=True, axis=1)
In [104]:
df.shape
Out[104]:
Specifing data types¶
In [106]:
df.info()
In [107]:
df["date_game"] = pd.to_datetime(df["date_game"])
In [108]:
df["game_location"].nunique()
Out[108]:
In [109]:
df["game_location"].value_counts()
Out[109]:
In [111]:
df["game_location"] = pd.Categorical(df["game_location"])
In [112]:
df["game_location"].dtype
Out[112]:
In [113]:
df.info()
In [114]:
df["game_result"].nunique()
Out[114]:
In [115]:
df["game_result"].value_counts()
Out[115]:
In [116]:
df["game_result"] = pd.Categorical(df["game_result"])
In [117]:
df.info()
Cleaning data¶
In [118]:
rows_without_missing_data = nba.dropna()
In [119]:
rows_without_missing_data.shape
Out[119]:
In [120]:
data_without_missing_columns = nba.dropna(axis=1)
In [121]:
data_without_missing_columns.shape
Out[121]:
In [122]:
data_with_default_notes = nba.copy()
In [123]:
data_with_default_notes["notes"].fillna(value="no notes at all", inplace=True)
In [124]:
data_with_default_notes["notes"].describe()
Out[124]:
In [125]:
nba[nba["pts"] == 0]
Out[125]:
In [126]:
nba[(nba["pts"] > nba["opp_pts"]) & (nba["game_result"] != 'W')].empty
Out[126]:
In [127]:
nba[(nba["pts"] < nba["opp_pts"]) & (nba["game_result"] != 'L')].empty
Out[127]:
Combining multiple datasets¶
In [129]:
further_city_data = pd.DataFrame(
{"revenue": [7000, 3400], "employee_count":[2, 2]},
index=["New York", "Barcelona"]
)
In [130]:
all_city_data = pd.concat([city_data, further_city_data], sort=False)
In [131]:
all_city_data
Out[131]:
Querying the dataset¶
In [132]:
current_decade = nba[nba["year_id"] > 2010]
In [133]:
current_decade.shape
Out[133]:
In [134]:
games_with_notes = nba[nba["notes"].notnull()]
In [135]:
games_with_notes.shape
Out[135]:
In [137]:
ers = nba[nba["fran_id"].str.endswith("ers")]
In [138]:
ers.shape
Out[138]:
Visualising the dataframe¶
In [140]:
%matplotlib inline
nba[nba["fran_id"] == "Knicks"].groupby("year_id")["pts"].sum().plot()
Out[140]:
In [142]:
nba["fran_id"].value_counts().head(10).plot(kind="bar")
Out[142]:
In [143]:
nba[
(nba["fran_id"] =="Heat") &
(nba["year_id"] == 2013)
]["game_result"].value_counts().plot(kind="pie")
Out[143]:
In [ ]: