Introduce dlookr

Preface

After you have acquired the data, you should do the following:

  • Diagnose data quality.
    • If there is a problem with data quality,
    • The data must be corrected or re-acquired.
  • Explore data to understand the data and find scenarios for performing the analysis.
  • Derive new variables or perform variable transformations.

The dlookr package makes these steps fast and easy:

  • Performs a data diagnosis or automatically generates a data diagnosis report.
  • Discover data in various ways and automatically generate EDA (exploratory data analysis) reports.
  • Impute missing values and outliers, resolve skewed data, and bin continuous variables into categorical variables. It also generates automated reports to support these tasks.

dlookr increases synergy with dplyr. Particularly in data exploration and data wrangling, it increases the efficiency of the tidyverse package group.

Supported data structures

Data diagnosis supports the following data structures.

  • data frame: data.frame class.
  • data table: tbl_df class.
  • table of DBMS: table of the DBMS through tbl_dbi
    • Use dplyr as the back-end interface for any DBI-compatible database.

List of supported tasks of data analytics

Diagnose Data

Overall Diagnose Data

Tasks Descriptions Functions Support DBI
describe overview of data Inquire about basic information to understand the data in general overview()
summary overview object summarize the described overview of data summary.overview()
plot overview object plot described overview of data plot.overview()
diagnose data quality of variables The scope of data quality diagnosis is information on missing values and unique value information diagnose() x
diagnose data quality of categorical variables frequency, ratio, rank by levels of each variable diagnose_category() x
diagnose data quality of numerical variables descriptive statistics, number of zeros, negative values, and outliers diagnose_numeric() x
diagnose data quality for outliers number of outliers, ratio, mean of outliers, mean with outliers, mean without outliers diagnose_outlier() x
plot outliers information of numerical data box plot and histogram with outliers, without outliers plot_outlier.data.frame() x
plot outliers information of numerical data by target variable box plot and density plot with outliers, without outliers plot_outlier.target_df() x
diagnose combination of categorical variables Check for sparse cases of level combinations of categorical variables diagnose_sparese()

Visualize Missing Values

Tasks Descriptions Functions Support DBI
Pareto chart for missing values visualize the Pareto chart for variables with missing values. plot_na_pareto()
combination chart for missing values visualize the distribution of missing values by combining variables. plot_na_hclust()
plot the combinations of variables that include missing values visualize the combinations of missing values across cases plot_na_intersect()

Reporting

Types Descriptions Functions Support DBI
report the information of data diagnosis into a PDF file report the information for diagnosing the data quality diagnose_report() x
reporting the information of data diagnosis into an HTML file report the information for diagnosing the quality of the data diagnose_report() x
reporting the information of data diagnosis into an HTML file dynamically report the information for diagnosing the quality of the data diagnose_web_report() x
reporting the information of data diagnosis into PDF and HTML files generate a paged report of the information for diagnosing the quality of the data diagnose_paged_report() x

EDA

Univariate EDA

Types Tasks Descriptions Functions Support DBI
categorical summaries frequency tables univar_category()
categorical summaries chi-squared test summary.univar_category()
categorical visualize bar charts plot.univar_category()
categorical visualize bar charts plot_bar_category()
numerical summaries descriptive statistics describe() x
numerical summaries descriptive statistics univar_numeric()
numerical summaries descriptive statistics of standardized variables summary.univar_numeric()
numerical visualize histogram, box plot plot.univar_numeric()
numerical visualize Q-Q plots plot_qq_numeric()
numerical visualize box plot plot_box_numeric()
numerical visualize histogram plot_hist_numeric()

Bivariate EDA

Types Tasks Descriptions Functions Support DBI
categorical summaries frequency tables of cross cases compare_category()
categorical summaries contingency tables, chi-squared test summary.compare_category()
categorical visualize mosaic plot plot.compare_category()
numerical summaries correlation coefficient, linear model summaries compare_numeric()
numerical summaries correlation coefficient, linear model summaries with threshold summary.compare_numeric()
numerical visualize scatter plot with marginal box plot plot.compare_numeric()
numerical Correlate correlation coefficient correlate() x
numerical Correlate summaries with correlation matrix summary.correlate() x
numerical Correlate visualization of a correlation matrix plot.correlate() x
both PPS PPS (Predictive Power Score) pps() x
both PPS summaries with PPS summary.pps() x
both PPS visualization of a PPS matrix plot.pps() x

Normality Test

Types Tasks Descriptions Functions Support DBI
numerical summaries Shapiro-Wilk normality test normality() x
numerical summaries normality diagnosis plot (histogram, Q-Q plots) plot_normality() x

Relationship between target variable and predictors

Target Variable Predictor Descriptions Functions Support DBI
categorical categorical contingency tables relate() x
categorical categorical mosaic plot plot.relate() x
categorical numerical descriptive statistics for each level and the total observations relate() x
categorical numerical density plot plot.relate() x
categorical categorical bar charts plot_bar_category()
numerical categorical ANOVA test relate() x
numerical categorical scatter plot plot.relate() x
numerical numerical simple linear model relate() x
numerical numerical box plot plot.relate() x
categorical numerical Q-Q plots plot_qq_numeric()
categorical numerical box plot plot_box_numeric()
categorical numerical histogram plot_hist_numeric()

Reporting

Types Descriptions Functions Support DBI
reporting the information of EDA into a PDF file reporting the information of EDA eda_report() x
reporting the information of EDA into an HTML file reporting the information of EDA eda_report() x
reporting the information of EDA into an HTML file dynamically reporting the information of EDA eda_web_report() x
reporting the information of EDA into PDF and HTML files paged reporting the information of EDA eda_paged_report() x

Transform Data

Find Variables

Types Descriptions Functions Support DBI
missing values find the variables that contain missing values in an object that inherits from data.frame find_na()
outliers find the numerical variables that contain outliers in an object that inherits from data.frame find_outliers()
skewed variable find the skewed numerical variables in an object that inherits from data.frame find_skewness()

Imputation

Types Descriptions Functions Support DBI
missing values missing values are imputed with some representative values and statistical methods. imputate_na()
outliers outliers are imputed with some representative values and statistical methods. imputate_outlier()
summaries calculate descriptive statistics of the original and imputed values. summary.imputation()
visualize the imputation of a numerical variable is a density plot, and the imputation of a categorical variable is a bar plot. plot.imputation()

Binning

Types Descriptions Functions Support DBI
binning converts a numeric variable to a categorical variable binning()
summaries calculate frequency and relative frequency for each level (bin) summary.bins()
visualize visualize two plots on a single screen. The plot at the top is a histogram representing the frequency of the level. The plot at the bottom is a bar chart representing the frequency of the level. plot.bins()
optimal binning categorizes a numeric characteristic into bins for ulterior usage in scoring modeling binning_by()
summaries summary metrics to evaluate the performance of a binomial classification model summary.optimal_bins()
visualize generates plots to understand the distribution, bad rate, and weight of evidence after running binning_by() plot.optimal_bins()
infogain binning categorizes a numeric characteristic into bins for multi-class variables using recursive information gain ratio maximization binning_rgr()
visualize generates plots for understanding the distribution and the distribution by target variable after running binning_rgr() plot.infogain_bins()
evaluate calculates metrics to evaluate the performance of a binned variable for a binomial classification model performance_bin()
summaries summary metrics to evaluate the performance of a binomial classification model after performance_bin() summary.performance_bin()
visualize generates plots to understand frequency and WoE by bins using performance_bin after running binning_by() plot.performance_bin()
visualize extract bins from “bins” and “optimal_bins” objects extract.bins()

Diagnose Binned Variable

Types Descriptions Functions Support DBI
diagnosis performs a performance diagnosis that calculates metrics to evaluate the performance of a binned variable for a binomial classification model performance_bin()
summaries summary method for “performance_bin”. summary metrics to evaluate the performance of the binomial classification model summary.performance_bin()
visualize visualize frequency and WoE by bins using performance_bin, and more plot.performance_bin()

Transformation

Types Descriptions Functions Support DBI
transformation performs variable transformations for standardization and resolving skewness of numerical variables transform()
summaries compares the distribution of data before and after data transformation summary.transform()
visualize visualize two kinds of plots by attribute of the ‘transform’ class. The transformation of a numerical variable is a density plot plot.transform()

Reporting

Types Descriptions Functions Support DBI
reporting the information of transformation into a PDF file reporting the information of transformation transformation_report()
reporting the information of transformation into an HTML file reporting the information of transformation transformation_report()
reporting the transformation information into an HTML file dynamically reporting the transformation information transformation_web_report()
reporting the information of transformation into PDF and HTML files paged reporting the information of transformation transformation_paged_report()

Miscellaneous

Statistics

Types Descriptions Functions Support DBI
statistics calculate the entropy entropy()
statistics calculate the skewness of the data skewness()
statistics calculate the kurtosis of the data kurtosis()
statistics calculate the Jensen-Shannon divergence between two probability distributions jsd()
statistics calculate the Kullback-Leibler divergence between two probability distributions kld()
statistics calculate the Cramer’s V statistic between two categorical (discrete) variables cramer()
statistics calculate the Theil’s U statistic between two categorical (discrete) variables theil()
statistics finding the percentile of a numerical variable. get_percentile()
statistics transform a numeric vector using several methods like “log”, “sqrt”, “log+1”, “log+a”, “1/x”, “x^2”, “x^3”, “Box-Cox”, “Yeo-Johnson” get_transform()
statistics calculate the Cramer’s V statistic cramer()
statistics calculate the Theil’s U statistic theil()

Programming

Types Descriptions Functions Support DBI
programming extracts variable information having a certain class from an object inheriting from data.frame find_class()
programming gets the class of variables in a data.frame or tbl_df get_class()
programming retrieves the column information of the DBMS table through the tbl_dbi object of dplyr get_column_info()
programming finding the user machine’s OS. get_os()
programming import Google fonts import_google_font()