Note this tutorial uses cleanlab v1. The code examples run, but you may need to install the older version explicitly.
python -m pip install cleanlab==1.0
Another interesting feature from cleanlab is the ability to train a classifier
that is more robust against bad labels. There's a meta model called
that is able to take any scikit-learn compatible pipeline and adapt it in order
to become more robust.
from cleanlab.classification import LearningWithNoisyLabels from sklearn.linear_model import LogisticRegression # Wrap around any classifier that has `sample_weights`. fresh_pipe = make_pipeline( CountVectorizer(), LogisticRegression(class_weight='balanced', max_iter=1000) ) lnl = LearningWithNoisyLabels(clf=fresh_pipe) # Pay attention! It's s=, not y=! lnl.fit(X=X, s=y.values)
LearningWithNoisyLabels model from cleanlab is meant as a
general method to allow machine learning to work on uncertain labels.
As long as the underlying pipeline implements a
method you're good to use this trick. The approach is sometimes referred
to as "confident learning" and the idea is getting traction.
LearningWithNoisyLabels vs. Utility
Although it's useful to have a more robust model via this trick, it would be best
not to fully rely on it. It's better to have higher quality labels, even when you're
using robustness methods. There's another use-case here though. We might be able to
lnl model for another purpose; finding bad labels!
A simple way to use it, is to compare it's output with the output of another machine learning model. When these two models disagree, we might again have a good proxy to fix a label.
Let's first make a new pipeline (one that isn't used in the
new_pipe = make_pipeline( CountVectorizer(), LogisticRegression(class_weight='balanced', max_iter=1000) ) new_pipe.fit(X=X, y=y)
With these two models trained, you can now start comparing.
df.loc[lnl.predict(X) != new_pipe.predict(X)][['text', 'excitement']].sample(5)