Your Title - data-newsletter

Podcast Episode Overview: What Machine Learning Engineers need to Know

2018-04-23T20:41:27-03:00

The original podcast episode can be found here: O'Reilly Media: What Machine LEarning Engineers need to know

You may have noticed job ads regarding a position called Machine Learning Engineer (MLE). You may have wondered what it is; this is a new term so there's a bit of confusion around it.

In this episode, Ben Lorica interviewed two people on this topic. They discussed whether it's actually a thing or just a fad and in which cases it makes sense to hire (or be) a Machine Learning Engineer.

What are MLEs

There's no consensus as of now.

Most people think of them as a hyper-specialized role, someone whose sole attribution is that of putting ML systems in production, therefore bridging the semantic gap between data scientists and data engineers.

The rationale is that they are needed because data scientists aren't suited for writing the systems side of products and data engineers lack the necessary skill to do that as well.

"I see a Machine Learning Engineer as someone who knows enough machine learning to get a product out; but they still need help from a data scientist for fine-tuning.". Ben Lorica

However, some people think of Machine Learning Engineers as a generalist role (much like Full-Stack Engineers in the world of Web Development), i.e. someone who knows enough about the whole data science pipeline to be able to get a working product out in the street.

In other words, someone who knows some data science, some data engineering and some general software engineering to be able to work on all parts of the product.

Domain expertise

While the data scientist is the role that's closer to the actual business problems to be solved, Machine Learning Engineers are expected to be able to better translate business objectives to system engineers and programmers.

Role of uncertainty

It's been said that engineers have, in general, difficulty with dealing with the ambiguity and uncertainty common in many businesses, above all those where data science permeates the decision-making process.

"If you just have pure risk, you buy insurance. But uncertainty is where you're going to find the upside." Paco Nathan

Once again, MLEs are expected to bridge the gap between the precise definitions needed in systems engineering with softer and more abstract notions from the data science world (think likelihoods, confidence intervals and continuous distributions).

Pipeline maturity

The question of whether MLEs are an actual needed role or not will probably depend a lot on what scale you are working at and how mature your data pipeline is.

If you have a mature pipeline (e.g. model monitoring, automation, off-the-shelf feature stores, smooth and well-tested data engineering processes) then having an MLE (in the sense of someone who is just focused on operationalizing models) may make sense because there's actual demand for that specialized role.

"The scramble to get to the point of having a feature store; that's the game. Until you get to that point, your models are very ad hoc." Paco Nathan

On the other hand, if you work at a small startup, it probably is overkill to have someone focused on just productionizing ML systems (unless you think of MLEs as generalist engineers), and it hints at a dysfunctional team.

My 2 cents

I would add that building tools (be it shell scripts, command-line applications or web apps) is also a very important part of data science work in general and MLEs could be just the role to help with that.

Quotes

"I see a Machine Learning Engineer as someone who knows enough machine learning to get a product out; but they still need help from a data scientist for fine-tuning." Ben Lorica

"First make your batch processes solid, then move into streaming data." Jesse Anderson

"The scramble to get to the point of having a feature store; that's the game. Until you get to that point, your models are very ad hoc." Paco Nathan

"If you just have pure risk, you buy insurance. But uncertainty is where you're going to find the upside." Paco Nathan

This short post is part of the Data Newsletter. Click here to sign up.

References

Package a Python Project and Make it Available via pip install: Simple Example

2017-11-15T21:46:00-02:00

Here is an example project built following the instructions on this post: Python Data Science Utils

For more information on what terms like pip, virtualenv, etc., mean see Python Environment Cheatsheet

sample setup.py file

File setup.py is the most important configuration file for your project. Here is an example to get you started:

# -*- coding: utf-8 -*-

import setuptools

with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="myproject",
    version="0.0.1",                        # Update this for every new version
    author="Your name",
    author_email="your@email.com",
    description="long description",
    long_description=long_description,
    long_description_content_type="text/markdown",
    install_requires=[                      # Add project dependencies here
        "pandas>=0.20.0"                    # example: pandas version 0.20 or greater                          
    ],                                             
    url="https://github.com/your/github/project",  
    packages=setuptools.find_packages(),
    classifiers=(                                 # Classifiers help people find your 
        "Programming Language :: Python :: 3",    # projects. See all possible classifiers 
        "License :: OSI Approved :: MIT License", # in https://pypi.org/classifiers/
        "Operating System :: OS Independent",   
    ),
)

Project Structure

project-name/        <--- project name, does not need to match the inner name
│             
├── myproject        <--- module name, this is the name you'll use in "import"                   
│   ├── __init__.py  <--- init file must be here, even if your project is Python 3 only
│   └── module.py    <--- submodules
│ 
├── LICENSE          <--- optional but recommended  
│ 
├── README.md
│ 
└── setup.py         <--- package information. name, author, version, and DEPENDENCIES

Package the project

To package the project, run python setup.py sdist bdist_wheel (preferably under a virtualenv):

(test-venv)$ python setup.py sdist bdist_wheel
running sdist
running egg_info
creating myproject.egg-info
writing requirements to myproject.egg-info/requires.txt
...
...

After running the packaging commands, your project should now look like this:

project-name/
│ 
├── build
│   ├── bdist.linux-x86_64
│   └── lib
│       └── myproject
│           ├── __init__.py
│           └── module.py 
├── dist                                     <--- this is the directory that
│   ├── myproject-0.0.1-py3-none-any.whl          will get uploaded to pypi
│   └── myproject-0.0.1.tar.gz
│ 
├── LICENSE
│ 
├── myproject
│   ├── __init__.py
│   └── module.py
│ 
├── myproject.egg-info
│   ├── dependency_links.txt
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   └── top_level.txt
│ 
├── README.md
│ 
└── setup.py

Upload project to PyPi

Trying to upload the myproject project will fail because someone has already created a project on PyPi with that name!

To upload the project you need a tool called twine, which can be installed with pip:

(test-venv)$ pip install twine
Collecting twine
...
...

At this point, you should register an account on pypi.org so that you can upload packages.

After you created your account, you can upload the packaged files using: twine upload dist/*

The error message is expected because the name already exists. You should pick a unique project name.

(test-venv)$ twine upload dist/*
Uploading distributions to https://upload.pypi.org/legacy/
Enter your username: xxxxx
Enter your password: 
Uploading myproject-0.0.1-py3-none-any.whl
100%|████████████████████████████████████████████████████████| 4.52k/4.52k [00:00<00:00, 5.46kB/s]
HTTPError: 403 Client Error: The user 'xxxxx' isn't allowed to upload to project 'MyProject'. 
See https://pypi.org/help/#project-name for more information. for url: https://upload.pypi.org/legacy/

Congratulations! After uploading the project (as above), anyone can install your project via pip.

Troubleshooting

Can't import module

If you can't import the module you just uploaded, make sure you have an __init__.py file as in the instructions, even if your project is full Python 3!

Module 'pkg_resources' has no attribute 'iter_entry_points'

Force reinstall setuptools:

(venv)$ pip install --upgrade --force-reinstall setuptools

This short post is part of the Data Newsletter. Click here to sign up.

References

The Hitchhiker's Guide to Packaging

Scaling Data Teams

2017-10-09T07:01:26-03:00

Original content: Data Teams with Rya Sciban on SE Daily Podcast

Interesting podcast episode on Software Engineering Daily, where the hosts talk about what specific needs data teams have and what changes when you go from small-scale to large-scale data science.

5 Needs of data teams

Having easy access to data (preferably at a single location throughout the organization)
Having the means to analyze and visualize data (i.e. good tools)
Being able to easily reproduce analyses and research (again, good tools but also processes)
Sharing insights and findings in the form of reports and or presentations (ditto)
Automating recurrent work (maybe built into tools, but often manually scripted)

Data Analysts vs Data Scientists

Data Analysts	Data Scientists
Work with SQL, Excel	Work with scripting tools such as R, Python or SAS
Focus on reporting, using existing data	Focus on training ML models for predictive or exploratory analysis

Relational vs Non-relational Databases

There's a tradeoff between flexibility and consistency.

Non-relational databases allow for more flexibility but there may be a higher risk of inconsistency (because the data may not be normalized as in relational databases)

Row-based vs Column-based data stores

Column-based data stores are more suitable for most analytics workflows because you are very often interested in viewing a single attribute over many records (as opposed to viewing the same record over many attributes)

Row-based stores sometimes allow you do remodel your existing row-based data as columns, for example using the PostgreSQL Columnar Store Extension for Postgres versions starting from 9.3.

Dramatically lower IO usage when using columnar data store (brown
and black) vs regular Postgres data types (blue)
Source: citusdata.com

Excel is still useful at many scales

It's easy to use and intuitive.

Especially for exploratory and ad-hoc (one-off) analysis.

Stakeholder communication

Try to understand the user's actual problem and needs rather than just blindly implementing what you're asked to do.

By talking (really listening) to your stakeholders you can help them understand the data work your team is doing you can suggest other ways of tackling problems.

For example, developing a simpler solution that covers 80% of their needs for 20% of the cost is a massive win.

Metrics

If you build data products, you want to collect metrics so that you know what specific features your users/stakeholders are using, for example:

What features are more often used, by what types of users?
What features are really not worth their screen real estate?
- I.e. maybe you could hide them from the product's UI so as to reduce clutter and improve usability?

This short post is part of the Data Newsletter. Click here to sign up.

References

citusdata/cstore_fdw Postgres Extension Github
- Provides all sorts of optimizations for modelling Postgres data as columns.
- Uses ORC (Optimized Row Columnar) format.

Matplotlib, Pyplot, Pylab etc: What's the difference between these and when to use each?

2017-09-26T17:41:24-03:00

TL;DR: Matplotlib is the toolkit, PyPlot is an interactive way to use Matplotlib and PyLab is the same thing as PyPlot but with some extra shortcuts. Using PyLab is discouraged now.

All example notebooks can be found on this link

Matplolib

It's a Python plotting library, inspired by MATLAB, meaning that the terms used (Axis, Figure, Plots) will be similar to those used in MATLAB.

It can be used both within a Python system (via the object-oriented API) but also in more convenient form in Jupyter and IPython notebooks via the Pyplot interface.

Pyplot

PyPlot is a shell-like interface to Matplotlib, to make it easier to use for people who are used to MATLAB.

Pyplot maintains state across calls.
Useful for use in Jupyter or IPython notebooks.
Your can import it via the matplotlib.pyplot namespace.

import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0,10,9)
y = np.randn(9)

plt.scatter(x,y)
plt.show()

Simplest possible example: scatter function from pyplot namespace

Pylab

Someone had the idea of combining both the PyPlot and NumPy namespaces into a single one (to avoid having to import two namespaces), the result is PyLab.

Using Pylab is now discouraged.

from matplotlib.pylab import *

# you can now use both pyplot and numpy
# functions as if they had been imported

# from numpy namespace
x = linspace(0,10,9)

# from numpy.random namespace
y = randn(9)

# from pyplot namespace
scatter(x,y)
show()

Functions like scatter() from pyplot are available when you import pylab too.

Key plotting concepts

These are important parts of the Matplotlib/Pyplot ecosystem and here are examples and information on what they stand for.

Matplotlib: Figure

Figure is the object that keeps the whole image output.

You can use it to configure things like:

Image size (set_size_inches())
Whether to use tight_layout (set_tight_layout())
- This adjusts the subplot padding; generally makes things look better.

Matplotlib: Axes

See all methods available in the Axes Class API

The Axes object represents the pair of axis that contain a single plot (x-axis and y-axis).

The Axes object has methods to allow you to configure things like:

The plot frame (set_frame_on() or set_frame_off())
X-axis and Y-axis limits (set_xlim() and set_ylim())
X-axis and Y-axis Labels (set_xlabel() and set_ylabel())
The plot title (set_title())

Pyplot: plt.gcf()

GCF stands for Get Current Figure

plt.gcf() allows you to get a reference to the current figure when using pyplot.

Example: change image sizes using fig.set_size_inches()

import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0,10,9)
y = np.randn(9)

# this call creates a figure in the background
plt.scatter(x,y)

# this allows you to retrieve the figure created
# by the call to scatter() above
fig = plt.gcf()
fig.set_size_inches(6,2)

plt.show()

Same image as above, resized.
We used the plt.gcf() function to get a reference to
the current figure and then called the
set_size_inches() method on it.

Pyplot: plt.gca()

GCA stands for Get Current Axes

Same as with plt.gcf(), you can use plt.gca() to get a reference to the current axes, if you need to change the limits on the y-axis, for example.

import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0,10,9)
y = np.randn(9)

# this call creates a figure in the background
plt.scatter(x,y)

# this allows you to retrieve the axis
# in the figure created 
# by the call to scatter() above
axis = plt.gca()
axis.set_ylim(-3,3)

plt.show()

Note that the y-axis now ranges from -3 to 3.
We got a reference to the current axis by
calling plt.gca(). Then we called the set_lim()
method to adjust the limits for that axis.

Pyplot: plt.cla() VS plt.clf()

cla stands for 'clear current axes' and clf stands for 'clear current figure'

These methods are used to clear the current figure (plt.clf()) or the current axes (plt.cla()).

It's common to call these methods just before defining a chart so that previous plots don't interfere with the next ones.

References

5 Tips for moving your Data Science Operation to the next Level

2017-09-25T23:54:58-03:00

Based on: Linear Digressions Podcast: Disciplined Data Science

Here are a couple of principles to keep in mind to if you want to avoid getting stuck in low-level data analytics, in any organization, but particularly larger ones.

These principles were taken from the Linear Digressions podcast, in an episode where suggest we apply these principles so as to add more "method" or "discipline" to the practice of data science.

Discoverability

You need to make your work as a data scientist easily available and findable by other teams, otherwise people may not fully benefit from what you do, or worse, someone may redo work you have already done.

A couple of easy ways to make your work available are Project Wikis, APIs or dashboarding.

Automation

Once you've solved a problem, automate it (e.g. Cron Jobs, Workflows) so that you can move on to other tasks.

Collaboration

Data science is very often a team sport. Knowing how to collaborate with other team members (across roles) is very important.

Empowerment

Use tools (for data science, machine learning, etc) that are widely used by the community to minimize risk and make it easier for other teams to verify your work or help you out.

Deployment

People have expectations on how accurate and reliable your work is.

When you publish or otherwise make your work available as a data product, be careful not to let them down so that they know they can trust you.

My 2c: Data science code is software too

I.e. all known good practices from software engineering still apply (low coupling, high cohesion, testable code, comments and documentation)

My 2c: Focus on business needs

Data science is only as good as the business problems it solves.

Sometimes a simple exploratory analysis or simple model is a cost-effective solution to a simple problem. There's no place for egos here.

This short post is part of the Data Newsletter. Click here to sign up.

References

Civis Analytics Whitepaper: Build a Disciplined Data Science Team

Highlights of the Talk with Dr. Konstan on Recommender Systems

2017-09-23T23:04:59-03:00

Original content: Recommender Systems: (live from Farcon) at Data Skeptic

Here are some topics I've considered interesting after listening to the above Podcast on Data Skeptic.

Serendipity

It is the extent to which users are "positively suprised" when recommended new content.

Maximizing serendipity is an actively pursued objective of Recommender Systems, particularly in recent years.

Serpentining

A bit counter-intuitive - it refers to actively refraining from showing the best (most relevant) recommendations first so as to force users to keep looking at other recommendations and promote more interaction with the system.

Diversity

It refers to promoting a diverse set of recommendations, so as to avoid recommending items that are too similar to one another.

Note that diversity is especially important when your customers are not repeat customers (i.e. they will very rarely come to your website, or even once only), because you may only have a single chance of conversion, so you want to hedge your bets using diverse content.

On the other hand, focusing too much on diversity may affect recurring customers, because they may want to be able to learn how you system behaves. I.e. they expect your system to behave consistently, and may not like too much variation in the recommendations.

Focus on business model

Knowing about the business model your organization wants to pursue is critical to thinking and planning good Recommender Systems.

In other words, you need to consider the precise business objective you want to achieve when designing RSs. This will guide you towards choosing what metrics you want to keep an eye on: For example:

Does your company want to maximize short-term item sales or long-term user retention?
Does your company want to maximize any sale, even if it's not the item that was recommended, or just maximize conversion for shown items?

Temporal effects

People's tastes change over time!

You need to take this into account (introduce adaptive behaviour into your models), otherwise you risk failing to capitalize on new trends and other changes in user behaviour.

This short post is part of the Data Newsletter. Click here to sign up.

References

Data Skeptic Podcast: September 15, 2017

Data Provenance: Quick Summary + Reasons Why

2017-09-07T00:16:15-03:00

Original content: Linear Digressions Podcast: Data Lineage

Data Provenance (also called Data Lineage) is to datasets as version control systems (e.g. Git, SVN) are to source code.

The basic idea is that you keep track of modifications you make to a dataset as you work with it.

Some common ways in which datasets are modified in this way are:

Missing Data Imputation
Data Cleaning, removing noisy or corrupted records
Dimensionality Reduction
etc

Two approaches

The two basic approaches to keeping track of data provenance are:

Track the states: You basically take snapshots of your dataset after each modification.
- This is more robust but takes more space
Track the modifications: You keep track of the operations you performed in each step of the way.
- A bit less robust (modifications may not be reversible) but takes much less space.

4 Reasons why you will want to version control your data

It helps you Keep data quality, because you can easily track down possible bugs and missing data.
It is very useful for Auditing Puposes, in cases you need to prove why you (or your model) made some decision.
It's probably unavoidable for Experiment Reproducibility. You can only reproduce a result having the same data it was obtained from.
It's also useful from a purely Informational standpoint. Having this kind of evolution-based view of your data enables you to look at your process from outside, think about how it can be made better, etc.
- This may also be needed if you are part of a team where each member role only works at a specific part of the pipeline.

This short post is part of the Data Newsletter. Click here to sign up.

References

Pachyderm.io
- It looks like some sort of language-agnostic toolkit that enables you to define data pipelines and workflows and run your code in a distributed way
- Both open-source and enterprise editions
Cookiecutter data science
- It's a command-line data science project generator
- It helps you create a standard directory structure for your data science project.

Lessons from the Netflix Prize: Changing Requirements and Cost-Effectiveness

2017-09-03T22:15:03-03:00

Original Article on Techdirt: Why Netflix Never Implemented the Algorithm that won the Netflix $1 Million Challenge

The Winning Solution (1st Place) was an Ensemble of more than 100 separate models

Out of all 107 component models, the two best individual algorithms were based on

Matrix Factorization and
RBM (Restricted Bolztmann Machines)

These two were the only parts actually used by Netflix.

Too Complex for the Improvements on the other Solutions

The Netflix team made a simplified model using just the two best performing individual models (as above) and used that in actual production systems.

The full model had slightly better performance than the simplified model, but this came at a much higher engineering effort (the full model was much more complicated to build).

Requirements had changed by the time Netflix actually implemented it

The challenge was made with movie rentals in mind but, by the time it was going to be used, streaming had become a much more important part of the Netflix business model.

Also, Netflix had recently gone international and become available on many more devices, thus affecting the way viewers consumed the products.

Apparently, data science is software too.

Recommending movies for rental vs recommending streaming content

Recommending streaming services is fundamentally different from recommending movies for rental.

When you rent something, you want to be more certain that it's good, because of the time/money you'll put into it, but When viewing streaming content, you can afford to experiment a little bit, but you are more likely to give up on something if you don't immediately like it.

Also, there's much more data available (implicit feedback e.g. half-watched movies, movies watched multiple times) in a streaming service context and this information can be leveraged to improve recommendations.

This short post is part of the data newsletter. Click here to sign up.

References

Netflix Blog: Netflix Recommendations: Beyond the 5 Stars (Part 1)
- Numerous insights into the (then) way Netflix recommended stuff: movie rows, recommendation diversity, recommendation awareness
Neftlix Prize Webpage

Winning Solutions Overview: Kaggle Instacart Competition

2017-09-03T21:54:26-03:00

Data Newsletter This short post is part of the data newsletter. Click here to sign up.

The Instacart "Market Basket Analysis" competition was about predicting which products in the next order would be a product that user had already ordered before.

1st Place (F1 0.4091)

User didn't publish his strategy.

2nd Place (F1: 0.4082) (LINK)

Approach: Manual Feature Engineering + XGBoost (Python)

Note: Optimizes F1-score directly (see references)

3rd Place (F1: 0.4081) (LINK)

Approach: Feature Learning with Neural Networks, then classification using Tensorflow and LightGBM(Python)

4th Place (F1: 0.4074) (LINK)

Approach: Manual Feature Engineering + Catboost

Note: Optimizes F1-score directly (see references)

5th Place (F1: 0.4073)

User didn't publish his strategy.

6th Place (F1: 0.4072) (LINK)

Approach: Manual Feature Engineering + CatBoost

Note: Optimizes F1-score directly (see references)

References

CatBoost is a gradient boosting library with easier handling for categorical features.
Queirozf.com: Evaluation Metrics for Classification Problems: Quick Examples + References
Ye et al., 2012: Optimizing F-Measures: A Tale of Two Approaches
- Here's a simple implementation in Python: F1-Expectation-Maximization
Dembczynksi et al., 2011: An Exact Algorithm for F-Measure Maximization
Solution summary instacart
- More solutions and overviews here. Thanks to Kaggle user Lannister.

A Quick Summary of Ensemble Learning Strategies

2017-09-01T07:52:09-03:00

Ensemble learning refers to mixing the outputs of several underlying classifiers in various ways, in order to:

Get more accurate predictions that each model individually.
Help in generalization, thus reducing the risk of overfitting.

Most Kaggle competitions are won by ensemble methods (as of 2018).

The main types of ensemble techniques are:

Simple Voting

Train each underlying model on the whole training data
- The output of the ensemble model is the output of the majority of underlying methods.
- All underlying models have the same voting power.
The equivalent technique for regression is Model Averaging

Weighted Voting

Train each underlying model on the whole training data.
- The output of the ensemble model is the output of the majority of underlying methods (as above)
- But some models have more voting power than others.

Bagging

Split the training data into random subsets (sampling with replacement)
- The output of the ensemble model is the average of the underlying methods.
- All underlying models have the same voting power.
Example: Random Forest is Bagging applied to Decision Trees.
The objective is to increase generalization power.

Stacking

Train each underlying model on the whole training data.
Train another model (e.g. Logistic Regression) to learn how to best combine the outputs of each underlying model.
The objective is to increase model accuracy and generalization power.

Boosting

Train a model on the whole training data.
- Then train a model on the errors (residuals) of the previous model.
- Repeat until convergence.
The objective is to increase the accuracy.
Examples: AdaBoost and XGBoost are variants of boosting.

This short post is part of the data newsletter. Click here to sign up.

References

Toptal: Ensemble Methods in Machine Learning
Learn by marketing: Kaggle Competition Analysis
- Ensembles and XGBoost (a specific type of ensemble) win by a large margin.
MLWave: Kaggle Ensembling Guide
- Large document, with tons of examples and Kaggle competitions to try the methods in.
Cross Validated Answer to: Bagging Boosting and Stacking in Machine Learning
- Good overview of pros/cons of bagging and boosting.

Evaluation Metrics for Classification Problems: Quick Examples + References

2017-08-31T19:58:01-03:00

True Positives (TP): should be TRUE, you predicted TRUE

False Positives (FP): should be FALSE, you predicted TRUE

True Negative (TN): should be FALSE, you predicted FALSE

False Negatives (FN): should be TRUE, you predicted FALSE

All machine learning toolkits provide these model evaluation metrics.

Accuracy

"What percentage of my predictions are correct?"

$$ \text{Accuracy} = \frac{true \ positives + true \ negatives}{true \ positives + false \ positives + true \ negatives + false \ negatives} $$

Good for single label, binary classifcation.
Not good for imbalanced datasets.
- If, in the dataset, 99% of samples are TRUE and you blindly predict TRUE for everything, you'll have 0.99 accuracy, but you haven't actually learned anything.

Precision

"Of the points that I predicted TRUE, how many are actually TRUE?"

$$ \text{Precision} = \frac{true \ positives}{true \ positives + false \ positives} $$

Good for multi-label / multi-class classification and information retrieval
Good for unbalanced datasets

Recall

"Of all the points that are actually TRUE, how many did I correctly predict?"

$$ \text{Recall} = \frac{true \ positives}{true \ positives + false \ negatives} $$

Good for multi-label / multi-class classification and information retrieval
Good for unbalanced datasets

F1

"Can you give me a single metric that balances precision and recall?"

$$ F_1 = 2 \cdot \frac{Precision \cdot Recall}{Precision + Recall} $$

Or, in terms of primitive metrics:

$$ F_1 = \frac{2 \cdot true \ positives}{2 \cdot true \ positives + false \ negatives + false \ positives} $$

Gives equal weight to precision and recall
Good for unbalanced datasets

AUC (Area under ROC Curve)

"Is my model better than just random guessing?"

The ROC curve is obtained plotting your model's true-positive and false-positive rates at different points.

If your model scores less than 0.5 AUC, it's no better than just random guessing.

Source: http://gim.unmc.edu/dxtests/roc3.htm

Rank-based metric
Good for cases when you need to estimate how well your model is at discriminating TRUE from FALSE values.
- In other words, for every pair of instances A and B, when your model says A is more likely than B, how often is A is actually TRUE and B FALSE?

Gini Coefficient

The Gini coefficient is comparable to the AUC score inasmuch as it also is a rank-based metric that can be used to measure how well your model is at discriminating TRUE from FALSE values.

Rank-based metric
It ranges from 0 (chance results) to 1.0, which corresponds to perfect discrimination.

A comparison between AUC Score and Gini Index on the same scoring results.
Note that the Gini index corresponds to the area between the main diagonal and the model curve, while
the AUC corresponds to the full area under the ROC curve.
Adapted from https://bit.ly/2Jb6LtQ

This short post is part of the data newsletter. Click here to sign up.