Need to Know Python Libraries: Nikita Tayal

I consider data like raw vegetables. One consumes the delicious food after washing, chopping, and cooking. Similarly, different organizations store data in different forms which are considered to be in raw form. To make the best use of it one has to first extract and clean it, post which one can derive insights using various visualization techniques. Anyone who wants to grow their business would like to know their customer trends, their performance in various areas, and would like to make use of the data they receive through their products every day. Python libraries have made this task simpler for us, and since python is an open-source library, it is cost-effective to use its libraries.

Data Processing and Modelling

Data Processing is the basic step in finding patterns from the given data. Python has provided us with umpteen libraries to do this task. Few important ones are listed below which data scientists use day in and out.

NumPy provides fast mathematical operations over arrays. Be it addition, subtraction, multiplication, division, finding floor or ceiling in an array, NumPy does it all. It is the fundamental library for scientific computing in Python. It also does operations on Linear Algebra, Fourier Transforms, Random Number Generation, etc. Basic to advanced mathematical operations cannot be completed without this library. Elements in NumPy arrays are accessed by using square brackets and can be initialized by using nested Python Lists.

Types are part of what make NumPy so powerful and flexible. They map directly onto an underlying machine representation, which makes it easy to read and write binary streams of data to disk.

It also has methods to calculate eigenvalues and eigenvectors, which are used in Principal Component Analysis (PCA). Every data scientist would know the importance of vector decomposition and hence, the knowledge of this library becomes the basic need for everyone who is dealing with data analysis.

Reference website: https://numpy.org/

PANDAS

Pandas is primarily known to any newbie in Python for input-output operations. As you know, one needs to first load the dataset to perform any action on it. It is extensively helpful in data wrangling and manipulation. The Pandas library is built on NumPy and provides easy-to-use data structures and data analysis tools for the Python programming language. Read/Write to SQL Query or Database Table, Selecting, Boolean indexing & Setting are few operations for which Pandas is used. Functions like max(), min(), mean(), first(), last() can be quickly applied to the Group By object to obtain summary statistics for each group — an immensely useful function. This functionality is similar to the dplyr and plyr libraries for R.

Sample code:

Reference Website: https://pandas.pydata.org/

Sklearn (Scikit-learn)

Think of advanced mathematical computations, classification, regression, and clustering algorithms, sklearn is there for you. It is a separately-developed and distributed third-party extension to SciPy. Scikit-learn is largely written in Python but some core algorithms are written in Cython to improve performance. This library can also provide algorithms for data mining such as classification, regression, clustering, as well as many algorithms for standard machine learning.

Sample code:

Reference website: https://scikit-learn.org/stable/

SciPy

The scipy.integrate sub-package provides several integration techniques. SciPy is not just a library, but a whole ecosystem of libraries that work together to help you accomplish complicated scientific tasks quickly and reliably. It also has a library for Clustering which is a popular technique to categorize data by associating it into groups. The SciPy library includes an implementation of the k-means clustering algorithm as well as several hierarchical clustering algorithms. A simple help(‘scipy’) will give you a list of all the functions.

Reference Website: https://www.scipy.org/

Theano

Deep learning is a branch of machine learning. While various machine learning algorithms have the fine capacity to learn, deep learning systems can improve their performance with access to more data. Convolutional neural network is one of the deep learning models which can be trained using a large number of images. For tasks such as computer vision, speech recognition, machine translation, and robotics, the performance of deep learning systems far exceeds that of conventional machine learning systems. Software libraries used for deep learning include Tensorflow, PyTorch, and Theano.

Theano is a numerical computation library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently across the CPU or GPU. Theano allows you to write model specifications rather than the model implementations. Hence, Theano sits somewhere between NumPy and the Python symbolic mathematics library SymPy.

Reference website: http://deeplearning.net/software/theano/

Tensor Flow

TensorFlow is a software library or framework, designed by the Google team to implement machine learning and deep learning concepts in the easiest manner. It is one of the top 3 libraries used for neural networks, the other two being Pytorch and Keras. Core tensor flow is written in C++. It can train and run deep neural networks for handwritten digit classification, image recognition, word embedding, and creation of various sequence models. The training of images helps in storing the recognizable patterns within specified folders in a very easy manner. After all, it’s a Google product. It offers a myriad of Math and Machine Learning libraries.

Reference website: https://www.tensorflow.org/

Keras

Keras is a high-level Python library run on top of Theano and TensorFlow framework. Keras contains numerous implementations of commonly used neural-network building blocks such as layers, objectives, activation functions, optimizers, and a host of tools to make working with image and text data easier to simplify the coding necessary for writing deep neural network code. In addition to standard neural networks, Keras has support for convolutional and recurrent neural networks.

Reference website: https://keras.io/

Pytorch

TensorFlow, Keras, and Pytorch are the libraries extensively used in deep learning. While Keras is a high-level API capable of running on top of TensorFlow, Pytorch is a lower-level API focused on experimentation, giving the user more freedom to write custom layers and look under the hood of numerical optimization tasks. Pytorch is more verbose and easier to learn when compared to Keras and hence it is the preferred library for someone who is starting with neural networks.

Reference website: https://pytorch.org/

NLTK

The moment you open a website nowadays, do you notice a chatbot? When you search something on Google, do you see ample suggestions when you start typing a few letters? If you have ever noticed these things, then thank the natural language processing technique. NLP is used to analyze text, allowing machines to understand how humans speak. NLP is commonly used for text mining, machine translation, and automated question answering.

NLTK is a suite of libraries and programs for symbolic and statistical natural language processing for English written in the Python programming language. NLTK supports classification, tokenization, stemming, tagging, parsing, and semantic reasoning functionalities.

Reference Website: https://www.nltk.org/

VISUALIZATION

This is the part I like the most in data analysis. You know when you get to see the colorful output on your screen. Wow! That’s the feeling we all want to see after putting so much effort into washing the vegetables, chopping, and decorating them just to relish the color and the taste and as we know humans understand things better when they see things visually. A combination of Pandas, NumPy, and matplotlib can help in creating a variety of visualization charts. Seaborn is based on matplotlib and is meant more for statistical plotting while matplotlib is used for basic plots like bar graphs and histograms.

Matplotlib

It is a 2D, 3D plotting library using which you can plot a variety of charts. For basic plotting One can go to PyPlot within matplotlib. Histograms, bar plot, and scatterplots are widely used to derive insightful and meaningful observations. The plot() function and the show() function does this task all the more simple.

Sample code:

Reference website: https://matplotlib.org

Seaborn

Seaborn library has umpteen interactive plots to make the visualization attractive. Matplotlib predates Pandas by more than a decade and thus is not designed for use with Pandas Dataframes. To visualize data from a Pandas Dataframe, you must extract each Series and often concatenate them together in the right format. It would be nicer to have a plotting library that can intelligently use the Data Frame labels in a plot. This is overcome in Seaborn. Here Histograms and KDE can be combined using distplot.

Reference website : https://seaborn.pydata.org/

Plotly

Plotly is also widely used for visualization. Plotly does not natively handle Python Pandas Dataframes. To make Plotly work with these, you’ll need to convert those to dictionaries first or use plugins. What makes Plotly different is that it supports JavaScript, so it will respond to mouse events. For example, you can make annotation boxes pop up when someone moves the cursor over the chart. Pretty cool right?

Sample code:

Reference Website: https://plotly.com/

DATABASE

Data is generally not stored in CSV or Excel files. Huge volumes of data are stored in databases like SQL, Oracle, Big Data platforms like Hadoop, and it is often required to connect to these databases. Python has provided us with multiple libraries to connect to various data sources. Few packages include — Sqlite3, Mysql.connector, and psycopg2, CX_Oracle, and many more. My favorite is MySQL.Connector.

Mysql.connector

Python needs a MySQL driver to access the MySQL database. Very simple and easy to connect. All you need to do is to import the MySQL libraries and make a connection. It allows you to convert the parameter’s value between Python and MySQL data types. It is an API implemented using pure Python.

Read more post from this writer at Great Learning's Medium