When dealing with features with high cardinality, you might want to mark the rare categories as “Other”. Feature-engine’s RareLabelEncoder makes it easy for you to do so.
In the code above, we use RareLabelEncoder to replace categories with the frequency below 0.05 in the column “education” with “Other”.
Encode Rare Labels with Feature-engine Read More »
When using scikit-learn’s train_test_split, if you want to keep the proportion of classes in the sample the same as the proportion of classes in the entire dataset, use stratify=y.
Split Data in a Stratified Fashion in scikit-learn Read More »
Imagine you try to determine whether a job posting is fake or not. You come up with 4 different assumptions about a fake job posting. How do you test which of these features are the most accurate in predicting fraud?
This PDF below shows how you can use Snorkel AI to test different assumptions with ease.
My full article about Snorkel.
Link to Snorkel.
Favorite
Snorkel — Programmatically Build Training Data in Python Read More »
Applying a scikit-learn’s transformer on your DataFrame will return a NumPy array. If you want to return a pandas DataFrame instead, use SklearnTransformerWrapper along with your scikit-learn’s transformer.
This is a method of feature-engine.
Return a DataFrame When Using a scikit-learn’s Transformer Read More »
If you want to use Python to quickly build a URL using information such as scheme, host, port, path, query, fragment, etc, try yarl.
yarl also makes it easy to replace one part with another, such as a query.
yarl: Build a URL Using Python Read More »
When encoding categorical variables, you might want to capture the similarities among these categories such as ‘Master Police Officer’ and ‘Police Officer III’. If so, use dirty-cat.
In the code above, I use dirty-cat’s SimilarityEncoder to encode the titles while capturing their similarities.
The correlation matrix shows how similar two labels are using the encoded values. We can see that the similarity between ‘Master Police Officer’ and ‘Police Officer III’ is 0.86.
Similarity Encoding for Dirty Categories Using dirty_cat Read More »
If you want to remove the correlated variables from a dataframe, use feature_engine.DropCorrelatedFeatures.
In the code above, I drop the variables with a correlation above 0.8.
Feature-engine: Drop Correlated Features Read More »
If you want to get some interesting statistics about a location in one line of code, try Datacommons.Datacommons is a publicly available data from open sources (census.gov, cdc.gov, data.gov, etc.). In the example above, I used Datacommons to get the median income in California over time.
Find other interesting statistics using Datacommons here.
Link to Datacommons.
Favorite
Datacommons: Get Statistics about a Location in One Line of Code Read More »
If you want to get the root mean squared error using sklearn, pass squared=False to sklearn’s mean_squared_error method.
The code above shows an example of this method.
Link to the source code.Favorite
squared=False: Get RMSE from Sklearn’s mean_squared_error method Read More »
To get a DataFrame’s continuous and categorical variables based on their cardinality, use fastai’s cont_cat_split method.
If a column consists of integers, but its cardinality is smaller than the max_card parameter, it is considered as a category variable.
Find an example of this method above.
Link to the source code.
Link to the documentation.Favorite
CodeCut is a platform that offers short and visually appealing code snippets related to data science, data analysis, data engineering, and Python programming.
Copyright © 2025 Code Cut - All rights reserved.