import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
If you have already used Python, you know about its standard data structures (list, dicts etc). If you use Python for science, you also probably know Numpy arrays which underlying almost all other specialized scientific packages.
None of these structures offers a simple way to handle database style data, nor to easily do standard database operations. This is why Pandas exists: it offers a complete ecosystem of structures and functions dedicated to handle large tables with inhomogeneous contents.
In this first chapter, we are going to learn about the two main structures of Pandsa: Series and Dataframes.
Series are a the Pandas version of 1-D Numpy arrays. To understand their specificities, let's create one. Usually Pandas structures (Series and Dataframes) are creates from other simpler structures like Numpy arrays or dictionaries:
(MZ: Pandas builds on numpy; Series are equivalent of a list)
numpy_array = np.array([4,8,38,1,6])
# MZ:
numpy_array**2
The function pd.Series() allows us to convert objects into Series:
pd_series = pd.Series(numpy_array)
pd_series
# on the left => the indices; can be anything, e.g. names of towns
The underlying structure can be recovered with the .values attribute:
pd_series.values
# MZ: output is numpy array
Otherwise, indexing works as for regular arrays:
pd_series[1]
On top of accessing values in a series by regular indexing, one can create custom indices for each element in the series:
pd_series2 = pd.Series(numpy_array, index=['a', 'b', 'c', 'd','e'])
# MZ: force the index to be what we give
# MZ: to retrieve the indexes (keys)
pd_series2.keys()
pd_series2
Now a given element can be accessed either by using its regular index:
pd_series2[1]
or its chosen index:
pd_series2['b']
A more direct way to create specific indexes is to transform as dictionary into a Series:
# MZ: use dict to build the Series
composer_birth = {'Mahler': 1860, 'Beethoven': 1770, 'Puccini': 1858, 'Shostakovich': 1906}
pd_composer_birth = pd.Series(composer_birth)
pd_composer_birth
pd_composer_birth['Puccini']
In most cases, one has to deal with more than just one variable, e.g. one has the birth year and the death year of a list of composers. Also one might have different types of information, e.g. in addition to numerical variables (year) one might have string variables like the city of birth. The Pandas structure that allow one to deal with such complex data is called a Dataframe, which can somehow be seen as an aggregation of Series with a common index.
To see how to construct such a Dataframe, let's create some more information about composers:
composer_death = pd.Series({'Mahler': 1911, 'Beethoven': 1827, 'Puccini': 1924, 'Shostakovich': 1975})
composer_city_birth = pd.Series({'Mahler': 'Kaliste', 'Beethoven': 'Bonn', 'Puccini': 'Lucques', 'Shostakovich': 'Saint-Petersburg'})
Now we can combine multiple series into a Dataframe by precising a variable name for each series. Note that all our series need to have the same indices (here the composers' name):
# MZ: put Series together into a Dataframe
composers_df = pd.DataFrame({'birth': pd_composer_birth, 'death': composer_death, 'city': composer_city_birth})
composers_df
A more common way of creating a Dataframe is to construct it directly from a dictionary of lists:
dict_of_list = {'birth': [1860, 1770, 1858, 1906], 'death':[1911, 1827, 1924, 1975],
'city':['Kaliste', 'Bonn', 'Lucques', 'Saint-Petersburg']}
pd.DataFrame(dict_of_list)
# MZ: default indexes from 0 to 3
However we now lost the composers name. We can enforce it by providing, as we did before for the Series, a list of indices:
pd.DataFrame(dict_of_list, index=['Mahler', 'Beethoven', 'Puccini', 'Shostakovich'])
# MZ: you can explicitly pass the index
There are multiple ways of accessing values or series of values in a Dataframe. Unlike in Series, a simple bracket gives access to a column and not an index, for example:
composers_df['city']
returns a Series. Alternatively one can also use the attributes synthax and access columns by using:
composers_df.city
# rather recommended to use the brackets
The attributes synthax has some limitations, so in case something does not work as expected, revert to the brackets notation.
When specifiying multiple columns, a DataFrame is returned:
composers_df[['city', 'birth']]
One of the important differences with a regular Numpy array is that here, regular indexing doesn't work:
#composers_df[0,0]
Instead one has to use either the .iloc[] or the .loc[] method. .ìloc[] can be used to recover the regular indexing:
# MZ: recover by positions
composers_df.iloc[0,1]
While .loc[] allows one to recover elements by using the explicit index, on our case the composers name:
# MZ: recover by indices
composers_df.loc['Mahler','death']
Remember that loc and ``ìloc``` use brackets [] and not parenthesis ().
Numpy style indexing works here too
composers_df.iloc[1:3,:]
If you are working with a large table, it might be useful to sometimes have a list of all the columns. This is given by the .keys() attribute:
composers_df.keys()
It is very simple to add a column to a Dataframe. One can e.g. just create a column a give it a default value that we can change later:
# MZ: if pass a single value, add the same value everywhere
composers_df['country'] = 'default'
composers_df
Or one can use an existing list:
country = ['Austria','Germany','Italy','Russia']
composers_df['country2'] = country
composers_df