Package 'dlookr'

Description

dlookr provides data diagnosis, data exploration and transformation of variables during data analysis.

Details

It has three main goals:

• When data is acquired, it is possible to judge whether data is erroneous or to select a variable to be corrected or removed through data diagnosis.

• Understand the distribution of data in the EDA process. It can also understand the relationship between target variables and predictor variables for the prediction model.

• Imputes missing value and outlier to standardization and resolving skewness. And, To convert a continuous variable to a categorical variable, bin the continuous variables.

To learn more about dlookr, start with the vignettes: 'browseVignettes(package = "dlookr")'

Author(s)

Maintainer: Choonghyun Ryu [email protected]

See Also

Useful links:

• https://github.com/choonghyunryu/dlookr/

• https://choonghyunryu.github.io/dlookr/

• Report bugs at https://github.com/choonghyunryu/dlookr/issues

Description

The binning() converts a numeric variable to a categorization variable.

Usage

binning(
  x,
  nbins,
  type = c("quantile", "equal", "pretty", "kmeans", "bclust"),
  ordered = TRUE,
  labels = NULL,
  approxy.lab = TRUE
)

Arguments

Details

This function is useful when used with the mutate/transmute function of the dplyr package.

See vignette("transformation") for an introduction to these concepts.

Value

An object of bins class. Attributes of bins class is as follows.

• class : "bins"

• type : binning type, "quantile", "equal", "pretty", "kmeans", "bclust".

• breaks : breaks for binning. the number of intervals into which x is to be cut.

• levels : levels of binned value.

• raw : raw data, numeric vector corresponding to x argument.

See Also

binning_by, print.bins, summary.bins, plot.bins.

Examples

# Generate data for the example
heartfailure2 <- heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 20), "platelets"] <- NA

# Binning the platelets variable. default type argument is "quantile"
bin <- binning(heartfailure2$platelets)
# Print bins class object
bin

# Using labels argument
bin <- binning(heartfailure2$platelets, nbins = 4,
              labels = c("LQ1", "UQ1", "LQ3", "UQ3"))
bin

# Using another type argument
bin <- binning(heartfailure2$platelets, nbins = 5, type = "equal")
bin
bin <- binning(heartfailure2$platelets, nbins = 5, type = "pretty")
bin
# "kmeans" and "bclust" was implemented by classInt::classIntervals() function.
# So, you must install classInt package.
if (requireNamespace("classInt", quietly = TRUE)) {
  bin <- binning(heartfailure2$platelets, nbins = 5, type = "kmeans")
  bin
  bin <- binning(heartfailure2$platelets, nbins = 5, type = "bclust")
  bin
} else {
  cat("If you want to use this feature, you need to install the 'classInt' package.\n")
}

x <- sample(1:1000, size = 50) * 12345679
bin <- binning(x)
bin
bin <- binning(x, approxy.lab = FALSE)
bin

# extract binned results
extract(bin)

# -------------------------
# Using pipes & dplyr
# -------------------------
library(dplyr)

# Compare binned frequency by death_event
heartfailure2 %>%
  mutate(platelets_bin = binning(heartfailure2$platelets) %>% 
           extract()) %>%
  group_by(death_event, platelets_bin) %>%
  summarise(freq = n(), .groups = "drop") %>%
  arrange(desc(freq)) %>%
  head(10)

Description

The binning_by() finding intervals for numerical variable using optical binning. Optimal binning categorizes a numeric characteristic into bins for ulterior usage in scoring modeling.

Usage

binning_by(.data, y, x, p = 0.05, ordered = TRUE, labels = NULL)

Arguments

Details

This function is useful when used with the mutate/transmute function of the dplyr package. And this function is implemented using smbinning() function of smbinning package.

Value

an object of "optimal_bins" class. Attributes of "optimal_bins" class is as follows.

• class : "optimal_bins".

• type : binning type, "optimal".

• breaks : numeric. the number of intervals into which x is to be cut.

• levels : character. levels of binned value.

• raw : numeric. raw data, x argument value.

• ivtable : data.frame. information value table.

• iv : numeric. information value.

• target : integer. binary response variable.

attributes of "optimal_bins" class

Attributes of the "optimal_bins" class that is as follows.

• class : "optimal_bins".

• levels : character. factor or ordered factor levels

• type : character. binning method

• breaks : numeric. breaks for binning

• raw : numeric. before the binned the raw data

• ivtable : data.frame. information value table

• iv : numeric. information value

• target : integer. binary response variable

See vignette("transformation") for an introduction to these concepts.

See Also

binning, summary.optimal_bins, plot.optimal_bins.

Examples

library(dplyr)

# Generate data for the example
heartfailure2 <- heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "creatinine"] <- NA

# optimal binning using character
bin <- binning_by(heartfailure2, "death_event", "creatinine")

# optimal binning using name
bin <- binning_by(heartfailure2, death_event, creatinine)
bin

Description

The binning_rgr() finding intervals for numerical variable using recursive information gain ratio maximization.

Usage

binning_rgr(.data, y, x, min_perc_bins = 0.1, max_n_bins = 5, ordered = TRUE)

Arguments

Details

This function can be usefully used when developing a model that predicts y.

Value

an object of "infogain_bins" class. Attributes of "infogain_bins" class is as follows.

• class : "infogain_bins".

• type : binning type, "infogain".

• breaks : numeric. the number of intervals into which x is to be cut.

• levels : character. levels of binned value.

• raw : numeric. raw data, x argument value.

• target : integer. binary response variable.

• x_var : character. name of x variable.

• y_var : character. name of y variable.

See Also

binning, binning_by, plot.infogain_bins.

Examples

library(dplyr)

# binning by recursive information gain ratio maximization using character
bin <- binning_rgr(heartfailure, "death_event", "creatinine")

# binning by recursive information gain ratio maximization using name
bin <- binning_rgr(heartfailure, death_event, creatinine)
bin

# summary optimal_bins class
summary(bin)

# visualize all information for optimal_bins class
plot(bin)

# visualize WoE information for optimal_bins class
plot(bin, type = "cross")

# visualize all information without typographic
plot(bin, type = "cross", typographic = FALSE)

# extract binned results
extract(bin) %>% 
  head(20)

Description

A simulated data set containing sales of child car seats at 400 different stores.

Usage

data(Carseats)

Format

A data frame with 400 rows and 11 variables. The variables are as follows:

Sales

Unit sales (in thousands) at each location.

CompPrice

Price charged by competitor at each location.

Income

Community income level (in thousands of dollars).

Advertising

Local advertising budget for company at each location (in thousands of dollars).

Population

Population size in region (in thousands).

Price

Price company charges for car seats at each site.

ShelveLoc

A factor with levels Bad, Good and Medium indicating the quality of the shelving location for the car seats at each site.

Age

Average age of the local population.

Education

Education level at each location.

Urban

A factor with levels No and Yes to indicate whether the store is in an urban or rural location.

US

A factor with levels No and Yes to indicate whether the store is in the US or not.

Source

"Sales of Child Car Seats" in ISLR package <https://CRAN.R-project.org/package=ISLR>, License : GPL-2

Description

The compare_category() compute information to examine the relationship between categorical variables.

Usage

compare_category(.data, ...)

## S3 method for class 'data.frame'
compare_category(.data, ...)

Arguments

Details

It is important to understand the relationship between categorical variables in EDA. compare_category() compares relations by pair combination of all categorical variables. and return compare_category class that based list object.

Value

An object of the class as compare based list. The information to examine the relationship between categorical variables is as follows each components.

• var1 : factor. The level of the first variable to compare. 'var1' is the name of the first variable to be compared.

• var2 : factor. The level of the second variable to compare. 'var2' is the name of the second variable to be compared.

• n : integer. frequency by var1 and var2.

• rate : double. relative frequency.

• first_rate : double. relative frequency in first variable.

• second_rate : double. relative frequency in second variable.

Attributes of return object

Attributes of compare_category class is as follows.

• variables : character. List of variables selected for comparison.

• combination : matrix. It consists of pairs of variables to compare.

See Also

summary.compare_category, print.compare_category, plot.compare_category.

Examples

# Generate data for the example
heartfailure2 <- heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "smoking"] <- NA

library(dplyr)

# Compare the all categorical variables
all_var <- compare_category(heartfailure2)

# Print compare_numeric class objects
all_var

# Compare the categorical variables that case of joint the death_event variable
all_var %>% 
  "["(grep("death_event", names(all_var)))

# Compare the two categorical variables
two_var <- compare_category(heartfailure2, smoking, death_event)

# Print compare_category class objects
two_var

# Filtering the case of smoking included NA 
two_var %>%
  "[["(1) %>% 
  filter(!is.na(smoking))

# Summary the all case : Return a invisible copy of an object.
stat <- summary(all_var)

# Summary by returned objects
stat

# component of table 
stat$table

# component of chi-square test 
stat$chisq

# component of chi-square test 
summary(all_var, "chisq")

# component of chi-square test (first, third case)
summary(all_var, "chisq", pos = c(1, 3))

# component of relative frequency table 
summary(all_var, "relative")

# component of table without missing values 
summary(all_var, "table", na.rm = TRUE)

# component of table include marginal value 
margin <- summary(all_var, "table", marginal = TRUE)
margin

# component of chi-square test 
summary(two_var, method = "chisq")

# verbose is FALSE 
summary(all_var, "chisq", verbose = FALSE)

#' # Using pipes & dplyr -------------------------
# If you want to use dplyr, set verbose to FALSE
summary(all_var, "chisq", verbose = FALSE) %>% 
  filter(p.value < 0.26)

# Extract component from list by index
summary(all_var, "table", na.rm = TRUE, verbose = FALSE) %>% 
  "[["(1)

# Extract component from list by name
summary(all_var, "table", na.rm = TRUE, verbose = FALSE) %>% 
  "[["("smoking vs death_event")

# plot all pair of variables
plot(all_var)

# plot a pair of variables
plot(two_var)

# plot all pair of variables by prompt
plot(all_var, prompt = TRUE)

# plot a pair of variables
plot(two_var, las = 1)

Description

The compare_numeric() compute information to examine the relationship between numerical variables.

Usage

compare_numeric(.data, ...)

## S3 method for class 'data.frame'
compare_numeric(.data, ...)

Arguments

Details

It is important to understand the relationship between numerical variables in EDA. compare_numeric() compares relations by pair combination of all numerical variables. and return compare_numeric class that based list object.

Value

An object of the class as compare based list. The information to examine the relationship between numerical variables is as follows each components. - correlation component : Pearson's correlation coefficient.

• var1 : factor. The level of the first variable to compare. 'var1' is the name of the first variable to be compared.

• var2 : factor. The level of the second variable to compare. 'var2' is the name of the second variable to be compared.

• coef_corr : double. Pearson's correlation coefficient.

- linear component : linear model summaries

• var1 : factor. The level of the first variable to compare. 'var1' is the name of the first variable to be compared.

• var2 : factor.The level of the second variable to compare. 'var2' is the name of the second variable to be compared.

• r.squared : double. The percent of variance explained by the model.

• adj.r.squared : double. r.squared adjusted based on the degrees of freedom.

• sigma : double. The square root of the estimated residual variance.

• statistic : double. F-statistic.

• p.value : double. p-value from the F test, describing whether the full regression is significant.

• df : integer degrees of freedom.

• logLik : double. the log-likelihood of data under the model.

• AIC : double. the Akaike Information Criterion.

• BIC : double. the Bayesian Information Criterion.

• deviance : double. deviance.

• df.residual : integer residual degrees of freedom.

Attributes of return object

Attributes of compare_numeric class is as follows.

• raw : a data.frame or a tbl_df. Data containing variables to be compared. Save it for visualization with plot.compare_numeric().

• variables : character. List of variables selected for comparison.

• combination : matrix. It consists of pairs of variables to compare.

See Also

correlate, summary.compare_numeric, print.compare_numeric, plot.compare_numeric.

Examples

# Generate data for the example
heartfailure2 <- heartfailure[, c("platelets", "creatinine", "sodium")]

library(dplyr)
# Compare the all numerical variables
all_var <- compare_numeric(heartfailure2)

# Print compare_numeric class object
all_var

# Compare the correlation that case of joint the sodium variable
all_var %>% 
  "$"(correlation) %>% 
  filter(var1 == "sodium" | var2 == "sodium") %>% 
  arrange(desc(abs(coef_corr)))
  
# Compare the correlation that case of abs(coef_corr) > 0.1
all_var %>% 
  "$"(correlation) %>% 
  filter(abs(coef_corr) > 0.1)
  
# Compare the linear model that case of joint the sodium variable  
all_var %>% 
  "$"(linear) %>% 
  filter(var1 == "sodium" | var2 == "sodium") %>% 
  arrange(desc(r.squared))
  
# Compare the two numerical variables
two_var <- compare_numeric(heartfailure2, sodium, creatinine)

# Print compare_numeric class objects
two_var
  
# Summary the all case : Return a invisible copy of an object.
stat <- summary(all_var)

# Just correlation
summary(all_var, method = "correlation")

# Just correlation condition by r > 0.1
summary(all_var, method = "correlation", thres_corr = 0.1)

# linear model summaries condition by R^2 > 0.05
summary(all_var, thres_rs = 0.05)

# verbose is FALSE 
summary(all_var, verbose = FALSE)
  
# plot all pair of variables
plot(all_var)

# plot a pair of variables
plot(two_var)

# plot all pair of variables by prompt
plot(all_var, prompt = TRUE)

# plot a pair of variables not focuses on typographic elements
plot(two_var, typographic = FALSE)

Description

The correlate() compute the correlation coefficient for numerical or categorical data.

Usage

correlate(.data, ...)

## S3 method for class 'data.frame'
correlate(
  .data,
  ...,
  method = c("pearson", "kendall", "spearman", "cramer", "theil")
)

## S3 method for class 'grouped_df'
correlate(
  .data,
  ...,
  method = c("pearson", "kendall", "spearman", "cramer", "theil")
)

## S3 method for class 'tbl_dbi'
correlate(
  .data,
  ...,
  method = c("pearson", "kendall", "spearman", "cramer", "theil"),
  in_database = FALSE,
  collect_size = Inf
)

Arguments

Details

This function is useful when used with the group_by() function of the dplyr package. If you want to compute by level of the categorical data you are interested in, rather than the whole observation, you can use grouped_df as the group_by() function. This function is computed stats::cor() function by use = "pairwise.complete.obs" option for numerical variable. And support categorical variable with theil's U correlation coefficient and Cramer's V correlation coefficient.

Value

An object of correlate class.

correlate class

The correlate class inherits the tibble class and has the following variables.:

• var1 : names of numerical variable

• var2 : name of the corresponding numeric variable

• coef_corr : Correlation coefficient

When method = "cramer", data.frame with the following variables is returned.

• var1 : names of numerical variable

• var2 : name of the corresponding numeric variable

• chisq : the value the chi-squared test statistic

• df : the degrees of freedom of the approximate chi-squared distribution of the test statistic

• pval : the p-value for the test

• coef_corr : theil's U correlation coefficient (Uncertainty Coefficient).

See Also

cor, summary.correlate, plot.correlate.

Examples

# Correlation coefficients of all numerical variables
tab_corr <- correlate(heartfailure)
tab_corr

# Select the variable to compute
correlate(heartfailure, "creatinine", "sodium")

# Non-parametric correlation coefficient by kendall method
correlate(heartfailure, creatinine, method = "kendall")

# theil's U correlation coefficient (Uncertainty Coefficient)
tab_corr <- correlate(heartfailure, anaemia, hblood_pressure, method = "theil")
tab_corr
   
# Using dplyr::grouped_dt
library(dplyr)

gdata <- group_by(heartfailure, smoking, death_event)
correlate(gdata)

# Using pipes ---------------------------------
# Correlation coefficients of all numerical variables
heartfailure %>%
  correlate()
  
# Non-parametric correlation coefficient by spearman method
heartfailure %>%
  correlate(creatinine, sodium, method = "spearman")
 
# ---------------------------------------------
# Correlation coefficient
# that eliminates redundant combination of variables
heartfailure %>%
  correlate() %>%
  filter(as.integer(var1) > as.integer(var2))

# Using pipes & dplyr -------------------------
# Compute the correlation coefficient of 'creatinine' variable by 'smoking'
# and 'death_event' variables. And extract only those with absolute
# value of correlation coefficient is greater than 0.2
heartfailure %>%
  group_by(smoking, death_event) %>%
  correlate(creatinine) %>%
  filter(abs(coef_corr) >= 0.2)

# extract only those with 'smoking' variable level is "Yes",
# and compute the correlation coefficient of 'Sales' variable
# by 'hblood_pressure' and 'death_event' variables.
# And the correlation coefficient is negative and smaller than 0.5
heartfailure %>%
  filter(smoking == "Yes") %>%
  group_by(hblood_pressure, death_event) %>%
  correlate(creatinine) %>%
  filter(coef_corr < 0) %>%
  filter(abs(coef_corr) > 0.5)


# If you have the 'DBI' and 'RSQLite' packages installed, perform the code block:
if (FALSE) {
library(dplyr)
# connect DBMS
con_sqlite <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")

# copy heartfailure to the DBMS with a table named TB_HEARTFAILURE
copy_to(con_sqlite, heartfailure, name = "TB_HEARTFAILURE", overwrite = TRUE)

# Using pipes ---------------------------------
# Correlation coefficients of all numerical variables
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  correlate()

# Using pipes & dplyr -------------------------
# Compute the correlation coefficient of creatinine variable by 'hblood_pressure'
# and 'death_event' variables.
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  group_by(hblood_pressure, death_event) %>%
  correlate(creatinine) 

# Disconnect DBMS   
DBI::dbDisconnect(con_sqlite)
}

Description

Computes the Cramer's V statistic and Chisquare p value between two categorical variables in data.frame.

Usage

cramer(dfm, x, y)

Arguments

Value

data.frame. It has the following variables.:

• var1 : character. first variable name.

• var2 : character. second variable name.

• chisq : numeric. Chisquare statistic.

• df : integer. degree of freedom.

• pval : numeric. p value of Chisquare test.

• coef_corr : numeric. Cramer's V statistic.

See Also

theil.

Examples

cramer(mtcars, "gear", "carb")

Description

The describe() compute descriptive statistic of numeric variable for exploratory data analysis.

Usage

describe(.data, ...)

## S3 method for class 'data.frame'
describe(.data, ..., statistics = NULL, quantiles = NULL)

## S3 method for class 'grouped_df'
describe(
  .data,
  ...,
  statistics = NULL,
  quantiles = NULL,
  all.combinations = FALSE
)

Arguments

Details

This function is useful when used with the group_by function of the dplyr package. If you want to calculate the statistic by level of the categorical data you are interested in, rather than the whole statistic, you can use grouped_df as the group_by() function.

From version 0.5.5, the 'variable' column in the "descriptive statistic information" tibble object has been changed to 'described_variables'. This is because there are cases where 'variable' is included in the variable name of the data. There is probably no case where 'described_variables' is included in the variable name of the data.

Value

An object of the same class as .data.

Descriptive statistic information

The information derived from the numerical data describe is as follows.

• n : number of observations excluding missing values

• na : number of missing values

• mean : arithmetic average

• sd : standard deviation

• se_mean : standard error mean. sd/sqrt(n)

• IQR : interquartile range (Q3-Q1)

• skewness : skewness

• kurtosis : kurtosis

• p25 : Q1. 25% percentile

• p50 : median. 50% percentile

• p75 : Q3. 75% percentile

• p01, p05, p10, p20, p30 : 1%, 5%, 20%, 30% percentiles

• p40, p60, p70, p80 : 40%, 60%, 70%, 80% percentiles

• p90, p95, p99, p100 : 90%, 95%, 99%, 100% percentiles

See Also

describe.tbl_dbi, diagnose_numeric.data.frame.

Examples

# Generate data for the example
heartfailure2 <- heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 20), "sodium"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "smoking"] <- NA

# Describe descriptive statistics of numerical variables
describe(heartfailure2)

# Select the variable to describe
describe(heartfailure2, sodium, platelets, statistics = c("mean", "sd", "quantiles"))
describe(heartfailure2, -sodium, -platelets)
describe(heartfailure2, 5, statistics = c("mean", "sd", "quantiles"), quantiles = c(0.01, 0.1))

# Using dplyr::grouped_dt
library(dplyr)

gdata <- group_by(heartfailure2, hblood_pressure, death_event)
describe(gdata, "creatinine")

# Using pipes ---------------------------------
# Positive values select variables
heartfailure2 %>%
  describe(platelets, sodium, creatinine)

# Negative values to drop variables
heartfailure2 %>%
  describe(-platelets, -sodium, -creatinine)

# Using pipes & dplyr -------------------------
# Find the statistic of all numerical variables by 'hblood_pressure' and 'death_event',
# and extract only those with 'hblood_pressure' variable level is "Yes".
heartfailure2 %>%
  group_by(hblood_pressure, death_event) %>%
  describe() %>%
  filter(hblood_pressure == "Yes")
  
# Using all.combinations = TRUE
heartfailure2 %>%
  filter(!hblood_pressure %in% "Yes" | !death_event %in% "Yes") %>% 
  group_by(hblood_pressure, death_event) %>%
  describe(all.combinations = TRUE) 
  
# extract only those with 'smoking' variable level is "Yes",
# and find 'creatinine' statistics by 'hblood_pressure' and 'death_event'
heartfailure2 %>%
  filter(smoking == "Yes") %>%
  group_by(hblood_pressure, death_event) %>%
  describe(creatinine)

Description

The describe() compute descriptive statistic of numerical(INTEGER, NUMBER, etc.) column of the DBMS table through tbl_dbi for exploratory data analysis.

Usage

## S3 method for class 'tbl_dbi'
describe(
  .data,
  ...,
  statistics = NULL,
  quantiles = NULL,
  all.combinations = FALSE,
  in_database = FALSE,
  collect_size = Inf
)

Arguments

Details

This function is useful when used with the group_by function of the dplyr package. If you want to calculate the statistic by level of the categorical data you are interested in, rather than the whole statistic, you can use grouped_df as the group_by() function.

From version 0.5.5, the 'variable' column in the "descriptive statistic information" tibble object has been changed to 'described_variables'. This is because there are cases where 'variable' is included in the variable name of the data. There is probably no case where 'described_variables' is included in the variable name of the data.

Value

An object of the same class as .data.

Descriptive statistic information

The information derived from the numerical data describe is as follows.

• n : number of observations excluding missing values

• na : number of missing values

• mean : arithmetic average

• sd : standard deviation

• se_mean : standard error mean. sd/sqrt(n)

• IQR : interquartile range (Q3-Q1)

• skewness : skewness

• kurtosis : kurtosis

• p25 : Q1. 25% percentile

• p50 : median. 50% percentile

• p75 : Q3. 75% percentile

• p01, p05, p10, p20, p30 : 1%, 5%, 20%, 30% percentiles

• p40, p60, p70, p80 : 40%, 60%, 70%, 80% percentiles

• p90, p95, p99, p100 : 90%, 95%, 99%, 100% percentiles

See Also

describe.data.frame, diagnose_numeric.tbl_dbi.

Examples

# If you have the 'DBI' and 'RSQLite' packages installed, perform the code block:
if (FALSE) {
library(dplyr)

# connect DBMS
con_sqlite <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")

# copy heartfailure to the DBMS with a table named TB_HEARTFAILURE
copy_to(con_sqlite, heartfailure, name = "TB_HEARTFAILURE", overwrite = TRUE)

# Using pipes ---------------------------------
# Positive values select variables
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  describe(platelets, creatinine, sodium)
  
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  describe(platelets, creatinine, sodium, 
    statistics = c("mean", "sd", "quantiles"), quantiles = 0.1)

# Negative values to drop variables, and In-memory mode and collect size is 200
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  describe(-platelets, -creatinine, -sodium, collect_size = 200)

# Using pipes & dplyr -------------------------
# Find the statistic of all numerical variables by 'smoking' and 'death_event',
# and extract only those with 'smoking' variable level is "Yes".
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  group_by(smoking, death_event) %>%
  describe() %>%
  filter(smoking == "Yes")

# Using all.combinations = TRUE
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  filter(!smoking %in% "Yes" | !death_event %in% "Yes") %>% 
  group_by(smoking, death_event) %>%
  describe(all.combinations = TRUE) %>%
  filter(smoking == "Yes")
  
# extract only those with 'sex' variable level is "Male",
# and find 'sodium' statistics by 'smoking' and 'death_event'
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  filter(sex == "Male") %>%
  group_by(smoking, death_event) %>%
  describe(sodium)

# Disconnect DBMS   
DBI::dbDisconnect(con_sqlite)
}

Description

The diagnose() produces information for diagnosing the quality of the variables of data.frame or tbl_df.

Usage

diagnose(.data, ...)

## S3 method for class 'data.frame'
diagnose(.data, ...)

## S3 method for class 'grouped_df'
diagnose(.data, ...)

Arguments

Details

The scope of data quality diagnosis is information on missing values and unique value information. Data quality diagnosis can determine variables that require missing value processing. Also, the unique value information can determine the variable to be removed from the data analysis.

Value

An object of tbl_df.

Diagnostic information

The information derived from the data diagnosis is as follows.:

• variables : variable names

• types : data type of the variable or to select a variable to be corrected or removed through data diagnosis.

  • integer, numeric, factor, ordered, character, etc.

• missing_count : number of missing values

• missing_percent : percentage of missing values

• unique_count : number of unique values

• unique_rate : ratio of unique values. unique_count / number of observation

See vignette("diagonosis") for an introduction to these concepts.

See Also

diagnose.tbl_dbi, diagnose_category.data.frame, diagnose_numeric.data.frame.

Examples

# Diagnosis of all variables
diagnose(jobchange)

# Select the variable to diagnose
diagnose(jobchange, gender, experience, training_hours)
diagnose(jobchange, -gender, -experience, -training_hours)
diagnose(jobchange, "gender", "experience", "training_hours")
diagnose(jobchange, 4, 9, 13)

# Using pipes ---------------------------------
library(dplyr)

# Diagnosis of all variables
jobchange %>%
  diagnose()
# Positive values select variables
jobchange %>%
  diagnose(gender, experience, training_hours)
# Negative values to drop variables
jobchange %>%
  diagnose(-gender, -experience, -training_hours)
# Positions values select variables
jobchange %>%
  diagnose(4, 9, 13)
# Negative values to drop variables
jobchange %>%
  diagnose(-8, -9, -10)
  
# Using pipes & dplyr -------------------------
# Diagnosis of missing variables
jobchange %>%
  diagnose() %>%
  filter(missing_count > 0)
   
# Using group_by ------------------------------
# Calculate the diagnosis of all variables by 'job_chnge' using group_by()
jobchange %>%
  group_by(job_chnge) %>% 
  diagnose()

Description

The diagnose_category() produces information for diagnosing the quality of the variables of data.frame or tbl_df.

Usage

diagnose_category(.data, ...)

## S3 method for class 'data.frame'
diagnose_category(
  .data,
  ...,
  top = 10,
  type = c("rank", "n")[2],
  add_character = TRUE,
  add_date = TRUE
)

## S3 method for class 'grouped_df'
diagnose_category(
  .data,
  ...,
  top = 10,
  type = c("rank", "n")[2],
  add_character = TRUE,
  add_date = TRUE
)

Arguments

Details

The scope of the diagnosis is the occupancy status of the levels in categorical data. If a certain level of occupancy is close to 100 then the removal of this variable in the forecast model will have to be considered. Also, if the occupancy of all levels is close to 0 variable is likely to be an identifier.

Value

an object of tbl_df.

Categorical diagnostic information

The information derived from the categorical data diagnosis is as follows.

• variables : variable names

• levels: level names

• N : number of observation

• freq : number of observation at the levels

• ratio : percentage of observation at the levels

• rank : rank of occupancy ratio of levels

See vignette("diagonosis") for an introduction to these concepts.

See Also

diagnose_category.tbl_dbi, diagnose.data.frame, diagnose_numeric.data.frame, diagnose_outlier.data.frame.

Examples

# Diagnosis of categorical variables
diagnose_category(jobchange)

# Select the variable to diagnose
diagnose_category(jobchange, education_level, company_type)

# Using pipes ---------------------------------
library(dplyr)

# Diagnosis of all categorical variables
jobchange %>%
  diagnose_category()

# Positive values select variables
jobchange %>%
 diagnose_category(company_type, job_chnge)
 
# Negative values to drop variables
jobchange %>%
  diagnose_category(-company_type, -job_chnge)
  
# Top rank levels with top argument
jobchange %>%
  diagnose_category(top = 2)
  
# Using pipes & dplyr -------------------------
# Extraction of level that is more than 60% of categorical data
jobchange %>%
  diagnose_category()  %>%
  filter(ratio >= 60)

# All observations of enrollee_id have a rank of 1. 
# Because it is a unique identifier. Therefore, if you select up to the top rank 3, 
# all records are displayed. It will probably fill your screen.

# extract rows that less than equal rank 3
# default of type argument is "n"
jobchange %>% 
  diagnose_category(enrollee_id, top = 3)

# extract rows that less than equal rank 3
jobchange %>% 
  diagnose_category(enrollee_id, top = 3, type = "rank")
 
# extract only 3 rows
jobchange %>% 
  diagnose_category(enrollee_id, top = 3, type = "n")

# Using group_by ------------------------------
# Calculate the diagnosis of 'company_type' variable by 'job_chnge' using group_by()
jobchange %>%
  group_by(job_chnge) %>% 
  diagnose_category(company_type)

Description

The diagnose_category() produces information for diagnosing the quality of the character(CHAR, VARCHAR, VARCHAR2, etc.) column of the DBMS table through tbl_dbi.

Usage

## S3 method for class 'tbl_dbi'
diagnose_category(
  .data,
  ...,
  top = 10,
  type = c("rank", "n")[1],
  in_database = TRUE,
  collect_size = Inf
)

Arguments

Details

The scope of the diagnosis is the occupancy status of the levels in categorical data. If a certain level of occupancy is close to 100 then the removal of this variable in the forecast model will have to be considered. Also, if the occupancy of all levels is close to 0 variable is likely to be an identifier. You can use grouped_df as the group_by() function.

Value

an object of tbl_df.

Categorical diagnostic information

The information derived from the categorical data diagnosis is as follows.

• variables : variable names

• levels: level names

• N : number of observation

• freq : number of observation at the levels

• ratio : percentage of observation at the levels

• rank : rank of occupancy ratio of levels

See vignette("diagonosis") for an introduction to these concepts.

See Also

diagnose_category.data.frame, diagnose.tbl_dbi, diagnose_category.tbl_dbi, diagnose_numeric.tbl_dbi, diagnose_outlier.tbl_dbi.

Examples

# If you have the 'DBI' and 'RSQLite' packages installed, perform the code block:
if (FALSE) {
library(dplyr)

# connect DBMS
con_sqlite <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")

# copy jobchange to the DBMS with a table named TB_JOBCHANGE
copy_to(con_sqlite, jobchange, name = "TB_JOBCHANGE", overwrite = TRUE)

# Using pipes ---------------------------------
# Diagnosis of all categorical variables
con_sqlite %>% 
  tbl("TB_JOBCHANGE") %>% 
  diagnose_category()
  
# Positive values select variables
con_sqlite %>% 
  tbl("TB_JOBCHANGE") %>% 
  diagnose_category(company_type, job_chnge)
  
# Negative values to drop variables, and In-memory mode
con_sqlite %>% 
  tbl("TB_JOBCHANGE") %>% 
  diagnose_category(-company_type, -job_chnge, in_database = FALSE)
  
# Positions values select variables, and In-memory mode and collect size is 200
con_sqlite %>% 
  tbl("TB_JOBCHANGE") %>% 
  diagnose_category(7, in_database = FALSE, collect_size = 200) 
  
# Negative values to drop variables
con_sqlite %>% 
  tbl("TB_JOBCHANGE") %>% 
  diagnose_category(-7)
  
# Top rank levels with top argument
con_sqlite %>% 
  tbl("TB_JOBCHANGE") %>% 
  diagnose_category(top = 2)

# Using pipes & dplyr -------------------------
# Extraction of level that is more than 60% of categorical data
con_sqlite %>% 
  tbl("TB_JOBCHANGE") %>% 
  diagnose_category()  %>%
  filter(ratio >= 60)
  
# Using group_by() ---------------------------- 
con_sqlite %>% 
  tbl("TB_JOBCHANGE") %>% 
  group_by(job_chnge) %>% 
  diagnose_category(company_type) 
  
# Using type argument -------------------------
 dfm <- data.frame(alpabet = c(rep(letters[1:5], times = 5), "c")) 
 
# copy dfm to the DBMS with a table named TB_EXAMPLE
copy_to(con_sqlite, dfm, name = "TB_EXAMPLE", overwrite = TRUE)  
 
# extract rows that less than equal rank 10
# default of top argument is 10
con_sqlite %>% 
  tbl("TB_EXAMPLE") %>% 
  diagnose_category()
   
# extract rows that less than equal rank 2
con_sqlite %>% 
  tbl("TB_EXAMPLE") %>% 
  diagnose_category(top = 2, type = "rank")
   
# extract rows that less than equal rank 2
# default of type argument is "rank"
con_sqlite %>% 
  tbl("TB_EXAMPLE") %>% 
  diagnose_category(top = 2)
 
# extract only 2 rows
con_sqlite %>% 
  tbl("TB_EXAMPLE") %>% 
  diagnose_category(top = 2, type = "n")

# Disconnect DBMS   
DBI::dbDisconnect(con_sqlite)
}

Description

The diagnose_numeric() produces information for diagnosing the quality of the numerical data.

Usage

diagnose_numeric(.data, ...)

## S3 method for class 'data.frame'
diagnose_numeric(.data, ...)

## S3 method for class 'grouped_df'
diagnose_numeric(.data, ...)

Arguments

Details

The scope of the diagnosis is the calculate a statistic that can be used to understand the distribution of numerical data. min, Q1, mean, median, Q3, max can be used to estimate the distribution of data. If the number of zero or minus is large, it is necessary to suspect the error of the data. If the number of outliers is large, a strategy of eliminating or replacing outliers is needed.

Value

an object of tbl_df.

Numerical diagnostic information

The information derived from the numerical data diagnosis is as follows.

• variables : variable names

• min : minimum

• Q1 : 25 percentile

• mean : arithmetic average

• median : median. 50 percentile

• Q3 : 75 percentile

• max : maximum

• zero : count of zero values

• minus : count of minus values

• outlier : count of outliers

See vignette("diagonosis") for an introduction to these concepts.

See Also

diagnose_numeric.tbl_dbi, diagnose.data.frame, diagnose_category.data.frame, diagnose_outlier.data.frame.

Examples

# Diagnosis of numerical variables
diagnose_numeric(heartfailure)

# Select the variable to diagnose
diagnose_numeric(heartfailure, cpk_enzyme, sodium)
diagnose_numeric(heartfailure, -cpk_enzyme, -sodium)
diagnose_numeric(heartfailure, "cpk_enzyme", "sodium")
diagnose_numeric(heartfailure, 5)

# Using pipes ---------------------------------
library(dplyr)

# Diagnosis of all numerical variables
heartfailure %>%
  diagnose_numeric()
# Positive values select variables
heartfailure %>%
  diagnose_numeric(cpk_enzyme, sodium)
# Negative values to drop variables
heartfailure %>%
  diagnose_numeric(-cpk_enzyme, -sodium)
# Positions values select variables
heartfailure %>%
  diagnose_numeric(5)
# Negative values to drop variables
heartfailure %>%
  diagnose_numeric(-1, -5)

# Using pipes & dplyr -------------------------
# List of variables containing outliers
heartfailure %>%
  diagnose_numeric()  %>%
  filter(outlier > 0)
  
# Using group_by ------------------------------
# Calculate the diagnosis of all variables by 'death_event' using group_by()
heartfailure %>%
  group_by(death_event) %>% 
  diagnose_numeric()

Description

The diagnose_numeric() produces information for diagnosing the quality of the numerical(INTEGER, NUMBER, etc.) column of the DBMS table through tbl_dbi.

Usage

## S3 method for class 'tbl_dbi'
diagnose_numeric(.data, ..., in_database = FALSE, collect_size = Inf)

Arguments

Details

The scope of the diagnosis is the calculate a statistic that can be used to understand the distribution of numerical data. min, Q1, mean, median, Q3, max can be used to estimate the distribution of data. If the number of zero or minus is large, it is necessary to suspect the error of the data. If the number of outliers is large, a strategy of eliminating or replacing outliers is needed. You can use grouped_df as the group_by() function.

Value

an object of tbl_df.

Numerical diagnostic information

The information derived from the numerical data diagnosis is as follows.

• variables : variable names

• min : minimum

• Q1 : 25 percentile

• mean : arithmetic average

• median : median. 50 percentile

• Q3 : 75 percentile

• max : maximum

• zero : count of zero values

• minus : count of minus values

• outlier : count of outliers

See vignette("diagonosis") for an introduction to these concepts.

See Also

diagnose_numeric.data.frame, diagnose.tbl_dbi, diagnose_category.tbl_dbi, diagnose_outlier.tbl_dbi.

Examples

# If you have the 'DBI' and 'RSQLite' packages installed, perform the code block:
if (FALSE) {
library(dplyr)

# connect DBMS
con_sqlite <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")

# copy heartfailure to the DBMS with a table named TB_HEARTFAILURE
copy_to(con_sqlite, heartfailure, name = "TB_HEARTFAILURE", overwrite = TRUE)

# Using pipes ---------------------------------
# Diagnosis of all numerical variables
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_numeric()
  
# Positive values select variables, and In-memory mode and collect size is 200
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_numeric(age, sodium, collect_size = 200)
  
# Negative values to drop variables
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_numeric(-age, -sodium)
  
# Positions values select variables
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_numeric(5)
  
# Negative values to drop variables
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_numeric(-1, -5)

# Using pipes & dplyr -------------------------
# List of variables containing outliers
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_numeric()  %>%
  filter(outlier > 0)

# Using group_by() ---------------------------- 
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  group_by(death_event) %>% 
  diagnose_numeric() 
  
# Disconnect DBMS   
DBI::dbDisconnect(con_sqlite)
}

Description

The diagnose_outlier() produces outlier information for diagnosing the quality of the numerical data.

Usage

diagnose_outlier(.data, ...)

## S3 method for class 'data.frame'
diagnose_outlier(.data, ...)

## S3 method for class 'grouped_df'
diagnose_outlier(.data, ...)

Arguments

Details

The scope of the diagnosis is the provide a outlier information. If the number of outliers is small and the difference between the averages including outliers and the averages not including them is large, it is necessary to eliminate or replace the outliers.

Value

an object of tbl_df.

Outlier Diagnostic information

The information derived from the numerical data diagnosis is as follows.

• variables : variable names

• outliers_cnt : number of outliers

• outliers_ratio : percent of outliers

• outliers_mean : arithmetic average of outliers

• with_mean : arithmetic average of with outliers

• without_mean : arithmetic average of without outliers

See vignette("diagonosis") for an introduction to these concepts.

See Also

diagnose_outlier.tbl_dbi, diagnose.data.frame, diagnose_category.data.frame, diagnose_numeric.data.frame.

Examples

# Diagnosis of numerical variables
diagnose_outlier(heartfailure)

# Select the variable to diagnose
diagnose_outlier(heartfailure, cpk_enzyme, sodium)
diagnose_outlier(heartfailure, -cpk_enzyme, -sodium)
diagnose_outlier(heartfailure, "cpk_enzyme", "sodium")
diagnose_outlier(heartfailure, 5)

# Using pipes ---------------------------------
library(dplyr)

# Diagnosis of all numerical variables
heartfailure %>%
  diagnose_outlier()
# Positive values select variables
heartfailure %>%
  diagnose_outlier(cpk_enzyme, sodium)
# Negative values to drop variables
heartfailure %>%
  diagnose_outlier(-cpk_enzyme, -sodium)
# Positions values select variables
heartfailure %>%
  diagnose_outlier(5)
# Negative values to drop variables
heartfailure %>%
  diagnose_outlier(-1, -5)

# Using pipes & dplyr -------------------------
# outlier_ratio is more than 1%
heartfailure %>%
  diagnose_outlier()  %>%
  filter(outliers_ratio > 1)
  
# Using group_by ------------------------------
# Calculate the diagnosis of all variables by 'death_event' using group_by()
heartfailure %>%
  group_by(death_event) %>% 
  diagnose_outlier()

Description

The diagnose_outlier() produces outlier information for diagnosing the quality of the numerical(INTEGER, NUMBER, etc.) column of the DBMS table through tbl_dbi.

Usage

## S3 method for class 'tbl_dbi'
diagnose_outlier(.data, ..., in_database = FALSE, collect_size = Inf)

Arguments

Details

The scope of the diagnosis is the provide a outlier information. If the number of outliers is small and the difference between the averages including outliers and the averages not including them is large, it is necessary to eliminate or replace the outliers. You can use grouped_df as the group_by() function.

Value

an object of tbl_df.

Outlier Diagnostic information

The information derived from the numerical data diagnosis is as follows.

• variables : variable names

• outliers_cnt : number of outliers

• outliers_ratio : percent of outliers

• outliers_mean : arithmetic average of outliers

• with_mean : arithmetic average of with outliers

• without_mean : arithmetic average of without outliers

See vignette("diagonosis") for an introduction to these concepts.

See Also

diagnose_outlier.data.frame, diagnose.tbl_dbi, diagnose_category.tbl_dbi, diagnose_numeric.tbl_dbi.

Examples

# If you have the 'DBI' and 'RSQLite' packages installed, perform the code block:
if (FALSE) {
library(dplyr)

# connect DBMS
con_sqlite <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")

# copy heartfailure to the DBMS with a table named TB_HEARTFAILURE
copy_to(con_sqlite, heartfailure, name = "TB_HEARTFAILURE", overwrite = TRUE)

# Using pipes ---------------------------------
# Diagnosis of all numerical variables
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_outlier()
  
# Positive values select variables, and In-memory mode and collect size is 200
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_outlier(platelets, sodium, collect_size = 200)
  
# Negative values to drop variables
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_outlier(-platelets, -sodium)
  
# Positions values select variables
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_outlier(5)
  
# Negative values to drop variables
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_outlier(-1, -5)

# Using pipes & dplyr -------------------------
# outlier_ratio is more than 1%
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_outlier()  %>%
  filter(outliers_ratio > 1)

# Using group_by() ----------------------------
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  group_by(death_event) %>% 
  diagnose_outlier() 
  
# Disconnect DBMS   
DBI::dbDisconnect(con_sqlite)
}

Description

The diagnose_paged_report() paged report the information for diagnosing the quality of the data.

Usage

diagnose_paged_report(.data, ...)

## S3 method for class 'data.frame'
diagnose_paged_report(
  .data,
  output_format = c("pdf", "html"),
  output_file = NULL,
  output_dir = tempdir(),
  browse = TRUE,
  title = "Data Diagnosis Report",
  subtitle = deparse(substitute(.data)),
  author = "dlookr",
  abstract_title = "Report Overview",
  abstract = NULL,
  title_color = "white",
  subtitle_color = "gold",
  thres_uniq_cat = 0.5,
  thres_uniq_num = 5,
  flag_content_zero = TRUE,
  flag_content_minus = TRUE,
  flag_content_missing = TRUE,
  cover_img = NULL,
  create_date = Sys.time(),
  logo_img = NULL,
  theme = c("orange", "blue"),
  sample_percent = 100,
  is_tbl_dbi = FALSE,
  base_family = NULL,
  ...
)

Arguments

Details

Generate generalized data diagnostic reports automatically. You can choose to output to pdf and html files. This is useful for diagnosing a data frame with a large number of variables than data with a small number of variables.

Create an PDF through the Chrome DevTools Protocol. If you want to create PDF, Google Chrome or Microsoft Edge (or Chromium on Linux) must be installed prior to using this function. If not installed, you must use output_format = "html".

Value

No return value. This function only generates a report.

Reported information

Reported from the data diagnosis is as follows.

• Overview

  • Data Structures

  • Job Information

  • Warnings

  • Variables

• Missing Values

  • List of Missing Values

  • Visualization

• Unique Values

  • Categorical Variables

  • Numerical Variables

• Categorical Variable Diagnosis

  • Top Ranks

• Numerical Variable Diagnosis

  • Distribution

    • Zero Values

    • Minus Values

  • Outliers

    • List of Outliers

    • Individual Outliers

The base_family is selected from "Roboto Condensed", "Liberation Sans Narrow", "NanumSquare", "Noto Sans Korean". If you want to use a different font, use it after loading the Google font with import_google_font().

See Also

diagnose_paged_report.tbl_dbi.

Examples

if (FALSE) {
# create dataset
heartfailure2 <- dlookr::heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 20), "sodium"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "smoking"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 2), "time"] <- 0
heartfailure2[sample(seq(NROW(heartfailure2)), 1), "creatinine"] <- -0.3
 
# create pdf file. file name is Diagnosis_Paged_Report.pdf
diagnose_paged_report(heartfailure2)

# create pdf file. file name is Diagn.pdf. and change cover image
cover <- file.path(system.file(package = "dlookr"), "report", "cover2.jpg")
diagnose_paged_report(heartfailure2, cover_img = cover, title_color = "gray",
  output_file = "Diagn.pdf")

# create pdf file. file name is ./Diagn.pdf and not browse
cover <- file.path(system.file(package = "dlookr"), "report", "cover3.jpg")
diagnose_paged_report(heartfailure2, output_dir = ".", cover_img = cover, 
  flag_content_missing = FALSE, output_file = "Diagn.pdf", browse = FALSE)

# create pdf file. file name is Diagnosis_Paged_Report.html
diagnose_paged_report(heartfailure2, output_format = "html")
}

Description

The diagnose_paged_report() paged report the information for diagnosing the quality of the DBMS table through tbl_dbi.

Usage

## S3 method for class 'tbl_dbi'
diagnose_paged_report(
  .data,
  output_format = c("pdf", "html")[1],
  output_file = NULL,
  output_dir = tempdir(),
  browse = TRUE,
  title = "Data Diagnosis Report",
  subtitle = deparse(substitute(.data)),
  author = "dlookr",
  abstract_title = "Report Overview",
  abstract = NULL,
  title_color = "white",
  subtitle_color = "gold",
  thres_uniq_cat = 0.5,
  thres_uniq_num = 5,
  flag_content_zero = TRUE,
  flag_content_minus = TRUE,
  flag_content_missing = TRUE,
  cover_img = NULL,
  create_date = Sys.time(),
  logo_img = NULL,
  theme = c("orange", "blue")[1],
  sample_percent = 100,
  in_database = FALSE,
  collect_size = Inf,
  as_factor = TRUE,
  ...
)

Arguments

Details

Generate generalized data diagnostic reports automatically. You can choose to output to pdf and html files. This is useful for diagnosing a data frame with a large number of variables than data with a small number of variables.

Create an PDF through the Chrome DevTools Protocol. If you want to create PDF, Google Chrome or Microsoft Edge (or Chromium on Linux) must be installed prior to using this function. If not installed, you must use output_format = "html".

Value

No return value. This function only generates a report.

Reported information

Reported from the data diagnosis is as follows.

• Overview

  • Data Structures

  • Job Information

  • Warnings

  • Variables

• Missing Values

  • List of Missing Values

  • Visualization

• Unique Values

  • Categorical Variables

  • Numerical Variables

• Categorical Variable Diagnosis

  • Top Ranks

• Numerical Variable Diagnosis

  • Distribution

    • Zero Values

    • Minus Values

  • Outliers

    • List of Outliers

    • Individual Outliers

See Also

diagnose_paged_report.data.frame.

Examples

# If you have the 'DBI' and 'RSQLite' packages installed, perform the code block:
if (FALSE) {
library(dplyr)

# Generate data for the example
heartfailure2 <- heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 20), "platelets"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "smoking"] <- NA

# connect DBMS
con_sqlite <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")

# copy heartfailure2 to the DBMS with a table named TB_HEARTFAILURE
copy_to(con_sqlite, heartfailure2, name = "TB_HEARTFAILURE", overwrite = TRUE)

# reporting the diagnosis information -------------------------
# create pdf file. file name is Diagnosis_Paged_Report.pdf
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_paged_report()
  
# create pdf file. file name is Diagn.pdf, and collect size is 250
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_paged_report(collect_size = 250, output_file = "Diagn.pdf")
  
# Disconnect DBMS   
DBI::dbDisconnect(con_sqlite)
}

Description

The diagnose_report() report the information for diagnosing the quality of the data.

Usage

diagnose_report(.data, output_format, output_file, output_dir, ...)

## S3 method for class 'data.frame'
diagnose_report(
  .data,
  output_format = c("pdf", "html"),
  output_file = NULL,
  output_dir = tempdir(),
  font_family = NULL,
  browse = TRUE,
  ...
)

Arguments

Details

Generate generalized data diagnostic reports automatically. You can choose to output to pdf and html files. This is useful for diagnosing a data frame with a large number of variables than data with a small number of variables. For pdf output, Korean Gothic font must be installed in Korean operating system.

Value

No return value. This function only generates a report.

Reported information

Reported from the data diagnosis is as follows.

• Diagnose Data

  • Overview of Diagnosis

    • List of all variables quality

    • Diagnosis of missing data

    • Diagnosis of unique data(Text and Category)

    • Diagnosis of unique data(Numerical)

  • Detailed data diagnosis

    • Diagnosis of categorical variables

    • Diagnosis of numerical variables

    • List of numerical diagnosis (zero)

    • List of numerical diagnosis (minus)

• Diagnose Outliers

  • Overview of Diagnosis

    • Diagnosis of numerical variable outliers

    • Detailed outliers diagnosis

See vignette("diagonosis") for an introduction to these concepts.

Examples

if (FALSE) {
# reporting the diagnosis information -------------------------
# create pdf file. file name is DataDiagnosis_Report.pdf
diagnose_report(heartfailure)

# create pdf file. file name is Diagn.pdf
diagnose_report(heartfailure, output_file = "Diagn.pdf")

# create pdf file. file name is ./Diagn.pdf and not browse
diagnose_report(heartfailure, output_dir = ".", output_file = "Diagn.pdf", 
  browse = FALSE)

# create html file. file name is Diagnosis_Report.html
diagnose_report(heartfailure, output_format = "html")

# create html file. file name is Diagn.html
diagnose_report(heartfailure, output_format = "html", output_file = "Diagn.html")
}

Description

The diagnose_report() report the information for diagnosing the quality of the DBMS table through tbl_dbi

Usage

## S3 method for class 'tbl_dbi'
diagnose_report(
  .data,
  output_format = c("pdf", "html"),
  output_file = NULL,
  output_dir = tempdir(),
  font_family = NULL,
  in_database = FALSE,
  collect_size = Inf,
  ...
)

Arguments

Details

Generate generalized data diagnostic reports automatically. You can choose to output to pdf and html files. This is useful for diagnosing a data frame with a large number of variables than data with a small number of variables. For pdf output, Korean Gothic font must be installed in Korean operating system.

Value

No return value. This function only generates a report.

Reported information

Reported from the data diagnosis is as follows.

• Diagnose Data

  • Overview of Diagnosis

    • List of all variables quality

    • Diagnosis of missing data

    • Diagnosis of unique data(Text and Category)

    • Diagnosis of unique data(Numerical)

  • Detailed data diagnosis

    • Diagnosis of categorical variables

    • Diagnosis of numerical variables

    • List of numerical diagnosis (zero)

    • List of numerical diagnosis (minus)

• Diagnose Outliers

  • Overview of Diagnosis

    • Diagnosis of numerical variable outliers

    • Detailed outliers diagnosis

See vignette("diagonosis") for an introduction to these concepts.

See Also

diagnose_report.data.frame.

Examples

# If you have the 'DBI' and 'RSQLite' packages installed, perform the code block:
if (FALSE) {
library(dplyr)

# Generate data for the example
heartfailure2 <- heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 20), "platelets"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "smoking"] <- NA

# connect DBMS
con_sqlite <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")

# copy heartfailure2 to the DBMS with a table named TB_HEARTFAILURE
copy_to(con_sqlite, heartfailure2, name = "TB_HEARTFAILURE", overwrite = TRUE)

# reporting the diagnosis information -------------------------
# create pdf file. file name is DataDiagnosis_Report.pdf
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_report()
  
# create pdf file. file name is Diagn.pdf, and collect size is 350
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_report(collect_size = 350, output_file = "Diagn.pdf")

# create html file. file name is Diagnosis_Report.html
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_report(output_format = "html")

# create html file. file name is Diagn.html
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_report(output_format = "html", output_file = "Diagn.html")
  
# Disconnect DBMS   
DBI::dbDisconnect(con_sqlite)
}

Description

The diagnose_sparese() checks for combinations of levels that do not appear as data among all combinations of levels of categorical variables.

Usage

diagnose_sparese(.data, ...)

## S3 method for class 'data.frame'
diagnose_sparese(
  .data,
  ...,
  type = c("all", "sparse")[2],
  add_character = FALSE,
  limit = 500
)

Arguments

Value

an object of data.frame.

Information of sparse levels

The information derived from the sparse levels diagnosis is as follows.

• variables : level of categorical variables.

• N : number of observation. (optional)

Examples

library(dplyr)

# Examples of too many combinations
diagnose_sparese(jobchange)

# Character type is also included in the combination variable
diagnose_sparese(jobchange, add_character = TRUE)

# Combination of two variables
jobchange %>% 
  diagnose_sparese(education_level, major_discipline)

# Remove two categorical variables from combination
jobchange %>% 
  diagnose_sparese(-city, -education_level)

diagnose_sparese(heartfailure)

# Adjust the threshold of limt to calculate
diagnose_sparese(heartfailure, limit = 50)

# List all combinations, including parese cases
diagnose_sparese(heartfailure, type = "all") 

# collaboration with dplyr
heartfailure %>% 
  diagnose_sparese(type = "all") %>% 
  arrange(desc(n_case)) %>% 
  mutate(percent = round(n_case / sum(n_case) * 100, 1))

Description

The diagnose_web_report() report the information for diagnosing the quality of the data.

Usage

diagnose_web_report(.data, ...)

## S3 method for class 'data.frame'
diagnose_web_report(
  .data,
  output_file = NULL,
  output_dir = tempdir(),
  browse = TRUE,
  title = "Data Diagnosis",
  subtitle = deparse(substitute(.data)),
  author = "dlookr",
  title_color = "gray",
  thres_uniq_cat = 0.5,
  thres_uniq_num = 5,
  logo_img = NULL,
  create_date = Sys.time(),
  theme = c("orange", "blue"),
  sample_percent = 100,
  is_tbl_dbi = FALSE,
  base_family = NULL,
  ...
)

Arguments

Details

Generate generalized data diagnostic reports automatically. This is useful for diagnosing a data frame with a large number of variables than data with a small number of variables.

Value

No return value. This function only generates a report.

Reported information

Reported from the data diagnosis is as follows.

• Overview

  • Data Structures

    • Data Structures

    • Data Types

    • Job Information

  • Warnings

  • Variables

• Missing Values

  • List of Missing Values

  • Visualization

• Unique Values

  • Categorical Variables

  • Numerical Variables

• Outliers

• Samples

  • Duplicated

  • Heads

  • Tails

The base_family is selected from "Roboto Condensed", "Liberation Sans Narrow", "NanumSquare", "Noto Sans Korean". If you want to use a different font, use it after loading the Google font with import_google_font().

See Also

diagnose_web_report.tbl_dbi.

Examples

if (FALSE) {
# create dataset
heartfailure2 <- dlookr::heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 20), "sodium"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "smoking"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 2), "time"] <- 0
heartfailure2[sample(seq(NROW(heartfailure2)), 1), "creatinine"] <- -0.3
 
# create pdf file. file name is Diagnosis_Report.html
diagnose_web_report(heartfailure2)

# file name is Diagn.html. and change logo image
logo <- file.path(system.file(package = "dlookr"), "report", "R_logo_html.svg")
diagnose_web_report(heartfailure2, logo_img = logo, title_color = "black",
  output_file = "Diagn.html")

# file name is ./Diagn_heartfailure.html, "blue" theme and not browse
diagnose_web_report(heartfailure2, output_dir = ".", author = "Choonghyun Ryu",
  output_file = "Diagn_heartfailure.html", theme = "blue", browse = FALSE)
}

Description

The diagnose_web_report() report the information for diagnosing the quality of the DBMS table through tbl_dbi

Usage

## S3 method for class 'tbl_dbi'
diagnose_web_report(
  .data,
  output_file = NULL,
  output_dir = tempdir(),
  browse = TRUE,
  title = "Data Diagnosis",
  subtitle = deparse(substitute(.data)),
  author = "dlookr",
  title_color = "gray",
  thres_uniq_cat = 0.5,
  thres_uniq_num = 5,
  logo_img = NULL,
  create_date = Sys.time(),
  theme = c("orange", "blue")[1],
  sample_percent = 100,
  in_database = FALSE,
  collect_size = Inf,
  as_factor = TRUE,
  ...
)

Arguments

Details

Generate generalized data diagnostic reports automatically. This is useful for diagnosing a data frame with a large number of variables than data with a small number of variables.

Value

No return value. This function only generates a report.

Reported information

Reported from the data diagnosis is as follows.

• Overview

  • Data Structures

    • Data Structures

    • Data Types

    • Job Information

  • Warnings

  • Variables

• Missing Values

  • Top Ranks

• Numerical Variable Diagnosis

  • List of Missing Values

  • Visualization

• Unique Values

  • Categorical Variables

  • Numerical Variables

• Outliers

• Samples

  • Duplicated

  • Heads

  • Tails

See Also

diagnose_web_report.data.frame.

Examples

# If you have the 'DBI' and 'RSQLite' packages installed, perform the code block:
if (FALSE) {
library(dplyr)

# Generate data for the example
heartfailure2 <- heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 20), "platelets"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "smoking"] <- NA

# connect DBMS
con_sqlite <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")

# copy heartfailure2 to the DBMS with a table named TB_HEARTFAILURE
copy_to(con_sqlite, heartfailure2, name = "TB_HEARTFAILURE", overwrite = TRUE)

# reporting the diagnosis information -------------------------
# create pdf file. file name is Diagnosis_Report.html
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_web_report()
  
# create pdf file. file name is Diagn.html, and collect size is 250
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  diagnose_web_report(collect_size = 250, output_file = "Diagn.html")
  
# Disconnect DBMS   
DBI::dbDisconnect(con_sqlite)
}

Description

The diagnose() produces information for diagnosing the quality of the column of the DBMS table through tbl_dbi.

Usage

## S3 method for class 'tbl_dbi'
diagnose(.data, ..., in_database = TRUE, collect_size = Inf)

Arguments

Details

The scope of data quality diagnosis is information on missing values and unique value information. Data quality diagnosis can determine variables that require missing value processing. Also, the unique value information can determine the variable to be removed from the data analysis. You can use grouped_df as the group_by() function.

Value

An object of tbl_df.

Diagnostic information

The information derived from the data diagnosis is as follows.:

• variables : column names

• types : data type of the variable or to select a variable to be corrected or removed through data diagnosis.

  • integer, numeric, factor, ordered, character, etc.

• missing_count : number of missing values

• missing_percent : percentage of missing values

• unique_count : number of unique values

• unique_rate : ratio of unique values. unique_count / number of observation

See vignette("diagonosis") for an introduction to these concepts.

See Also

diagnose.data.frame, diagnose_category.tbl_dbi, diagnose_numeric.tbl_dbi.

Examples

library(dplyr)

if (requireNamespace("DBI", quietly = TRUE) & requireNamespace("RSQLite", quietly = TRUE)) {
  # connect DBMS
  con_sqlite <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")

  # copy jobchange to the DBMS with a table named TB_JOBCHANGE
  copy_to(con_sqlite, jobchange, name = "TB_JOBCHANGE", overwrite = TRUE)

  # Using pipes ---------------------------------
  # Diagnosis of all columns
  con_sqlite %>% 
    tbl("TB_JOBCHANGE") %>% 
    diagnose() %>% 
    print()
  
  # Positions values select columns, and In-memory mode and collect size is 200
  con_sqlite %>% 
    tbl("TB_JOBCHANGE") %>% 
    diagnose(gender, education_level, company_size, in_database = FALSE, collect_size = 200) %>% 
    print()

  # Using pipes & dplyr -------------------------
  # Diagnosis of missing variables
  con_sqlite %>% 
    tbl("TB_JOBCHANGE") %>% 
    diagnose() %>%
    filter(missing_count > 0) %>% 
    print()
  
  # Using pipes & dplyr -------------------------
  # Diagnosis of missing variables
  con_sqlite %>% 
    tbl("TB_JOBCHANGE") %>% 
    group_by(job_chnge) %>% 
    diagnose() %>% 
    print()
 
  # Disconnect DBMS   
  DBI::dbDisconnect(con_sqlite)
} else {
  cat("If you want to use this feature, you need to install the 'DBI' and 'RSQLite' package.\n")
}

Description

Generate paged HTML document

Usage

dlookr_orange_paged(...)

dlookr_blue_paged(...)

Arguments

Value

document of markdown format.

References

https://pagedown.rbind.io

Description

Loads additional style and template file

Usage

dlookr_templ_html(toc = TRUE, ...)

Arguments

Value

An R Markdown output format.

References

https://raw.githubusercontent.com/dr-harper/example-rmd-templates/master/R/my_html_format.R

Description

The eda_paged_report() paged report the information for EDA.

Usage

eda_paged_report(.data, ...)

## S3 method for class 'data.frame'
eda_paged_report(
  .data,
  target = NULL,
  output_format = c("pdf", "html"),
  output_file = NULL,
  output_dir = tempdir(),
  browse = TRUE,
  title = "EDA Report",
  subtitle = deparse(substitute(.data)),
  author = "dlookr",
  abstract_title = "Report Overview",
  abstract = NULL,
  title_color = "black",
  subtitle_color = "blue",
  cover_img = NULL,
  create_date = Sys.time(),
  logo_img = NULL,
  theme = c("orange", "blue"),
  sample_percent = 100,
  is_tbl_dbi = FALSE,
  base_family = NULL,
  ...
)

Arguments

Details

Generate generalized EDA report automatically. You can choose to output to pdf and html files. This feature is useful for EDA of data with many variables, rather than data with fewer variables.

Create an PDF through the Chrome DevTools Protocol. If you want to create PDF, Google Chrome or Microsoft Edge (or Chromium on Linux) must be installed prior to using this function. If not installed, you must use output_format = "html".

Value

No return value. This function only generates a report.

Reported information

The EDA process will report the following information:

• Overview

  • Data Structures

  • Job Information

• Univariate Analysis

  • Descriptive Statistics

    • Numerical Variables

    • Categorical Variables

  • Normality Test

• Bivariate Analysis

  • Compare Numerical Variables

  • Compare Categorical Variables

• Multivariate Analysis

  • Correlation Analysis

    • Correlation Coefficient Matrix

    • Correlation Plot

• Target based Analysis

  • Grouped Numerical Variables

  • Grouped Categorical Variables

  • Grouped Correlation

The base_family is selected from "Roboto Condensed", "Liberation Sans Narrow", "NanumSquare", "Noto Sans Korean". If you want to use a different font, use it after loading the Google font with import_google_font().

See Also

eda_paged_report.tbl_dbi.

Examples

if (FALSE) {
# create the dataset
heartfailure2 <- dlookr::heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 20), "sodium"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "smoking"] <- NA

# create pdf file. file name is EDA_Paged_Report.pdf
eda_paged_report(heartfailure2, sample_percent = 80)

# create pdf file. file name is EDA.pdf. and change cover image
cover <- file.path(system.file(package = "dlookr"), "report", "cover1.jpg")
eda_paged_report(heartfailure2, cover_img = cover, title_color = "gray",
  output_file = "EDA.pdf")

# create pdf file. file name is ./EDA.pdf and not browse
cover <- file.path(system.file(package = "dlookr"), "report", "cover3.jpg")
eda_paged_report(heartfailure2, output_dir = ".", cover_img = cover, 
  flag_content_missing = FALSE, output_file = "EDA.pdf", browse = FALSE)

# create pdf file. file name is EDA_Paged_Report.html
eda_paged_report(heartfailure2, target = "death_event", output_format = "html")
}

Description

The eda_paged_report() paged report the information for EDA of the DBMS table through tbl_dbi

Usage

## S3 method for class 'tbl_dbi'
eda_paged_report(
  .data,
  target = NULL,
  output_format = c("pdf", "html")[1],
  output_file = NULL,
  output_dir = tempdir(),
  browse = TRUE,
  title = "EDA Report",
  subtitle = deparse(substitute(.data)),
  author = "dlookr",
  abstract_title = "Report Overview",
  abstract = NULL,
  title_color = "black",
  subtitle_color = "blue",
  cover_img = NULL,
  create_date = Sys.time(),
  logo_img = NULL,
  theme = c("orange", "blue")[1],
  sample_percent = 100,
  in_database = FALSE,
  collect_size = Inf,
  as_factor = TRUE,
  ...
)

Arguments

Details

Generate generalized EDA report automatically. You can choose to output to pdf and html files. This feature is useful for EDA of data with many variables, rather than data with fewer variables.

Create an PDF through the Chrome DevTools Protocol. If you want to create PDF, Google Chrome or Microsoft Edge (or Chromium on Linux) must be installed prior to using this function. If not installed, you must use output_format = "html".

Value

No return value. This function only generates a report.

Reported information

The EDA process will report the following information:

• Overview

  • Data Structures

  • Job Information

• Univariate Analysis

  • Descriptive Statistics

    • Numerical Variables

    • Categorical Variables

  • Normality Test

• Bivariate Analysis

  • Compare Numerical Variables

  • Compare Categorical Variables

• Multivariate Analysis

  • Correlation Analysis

    • Correlation Coefficient Matrix

    • Correlation Plot

• Target based Analysis

  • Grouped Numerical Variables

  • Grouped Categorical Variables

  • Grouped Correlation

See Also

eda_paged_report.data.frame.

Examples

# If you have the 'DBI' and 'RSQLite' packages installed, perform the code block:
if (FALSE) {
library(dplyr)

# Generate data for the example
heartfailure2 <- heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 20), "platelets"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "smoking"] <- NA

# connect DBMS
con_sqlite <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")

# copy heartfailure2 to the DBMS with a table named TB_HEARTFAILURE
copy_to(con_sqlite, heartfailure2, name = "TB_HEARTFAILURE", overwrite = TRUE)

# reporting the diagnosis information -------------------------
# create pdf file. file name is EDA_Paged_Report.pdf
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_paged_report(target = "death_event")
  
# create pdf file. file name is EDA.pdf, and collect size is 250
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_paged_report(collect_size = 250, output_file = "EDA.pdf")
  
# Disconnect DBMS   
DBI::dbDisconnect(con_sqlite)
}

Description

The eda_report() report the information of exploratory data analysis for object inheriting from data.frame.

Usage

eda_report(.data, ...)

## S3 method for class 'data.frame'
eda_report(
  .data,
  target = NULL,
  output_format = c("pdf", "html"),
  output_file = NULL,
  output_dir = tempdir(),
  font_family = NULL,
  browse = TRUE,
  ...
)

Arguments

Details

Generate generalized EDA report automatically. You can choose to output as pdf and html files. This feature is useful for EDA of data with many variables, rather than data with fewer variables. For pdf output, Korean Gothic font must be installed in Korean operating system.

Value

No return value. This function only generates a report.

Reported information

The EDA process will report the following information:

• Introduction

  • Information of Dataset

  • Information of Variables

  • About EDA Report

• Univariate Analysis

  • Descriptive Statistics

  • Normality Test of Numerical Variables

    • Statistics and Visualization of (Sample) Data

• Relationship Between Variables

  • Correlation Coefficient

    • Correlation Coefficient by Variable Combination

    • Correlation Plot of Numerical Variables

• Target based Analysis

  • Grouped Descriptive Statistics

    • Grouped Numerical Variables

    • Grouped Categorical Variables

  • Grouped Relationship Between Variables

    • Grouped Correlation Coefficient

    • Grouped Correlation Plot of Numerical Variables

See vignette("EDA") for an introduction to these concepts.

Examples

if (FALSE) {
library(dplyr)

## target variable is categorical variable ----------------------------------
# reporting the EDA information
# create pdf file. file name is EDA_Report.pdf
eda_report(heartfailure, death_event)

# create pdf file. file name is EDA_heartfailure.pdf
eda_report(heartfailure, "death_event", output_file = "EDA_heartfailure.pdf")

# create pdf file. file name is EDA_heartfailure.pdf and not browse
eda_report(heartfailure, "death_event", output_dir = ".", 
  output_file = "EDA_heartfailure.pdf", browse = FALSE)

# create html file. file name is EDA_Report.html
eda_report(heartfailure, "death_event", output_format = "html")

# create html file. file name is EDA_heartfailure.html
eda_report(heartfailure, death_event, output_format = "html", 
  output_file = "EDA_heartfailure.html")

## target variable is numerical variable ------------------------------------
# reporting the EDA information
eda_report(heartfailure, sodium)

# create pdf file. file name is EDA2.pdf
eda_report(heartfailure, "sodium", output_file = "EDA2.pdf")

# create html file. file name is EDA_Report.html
eda_report(heartfailure, "sodium", output_format = "html")

# create html file. file name is EDA2.html
eda_report(heartfailure, sodium, output_format = "html", output_file = "EDA2.html")

## target variable is null
# reporting the EDA information
eda_report(heartfailure)

# create pdf file. file name is EDA2.pdf
eda_report(heartfailure, output_file = "EDA2.pdf")

# create html file. file name is EDA_Report.html
eda_report(heartfailure, output_format = "html")

# create html file. file name is EDA2.html
eda_report(heartfailure, output_format = "html", output_file = "EDA2.html")
}

Description

The eda_report() report the information of Exploratory data analysis for object inheriting from the DBMS table through tbl_dbi

Usage

## S3 method for class 'tbl_dbi'
eda_report(
  .data,
  target = NULL,
  output_format = c("pdf", "html"),
  output_file = NULL,
  font_family = NULL,
  output_dir = tempdir(),
  in_database = FALSE,
  collect_size = Inf,
  ...
)

Arguments

Details

Generate generalized data EDA reports automatically. You can choose to output to pdf and html files. This is useful for EDA a data frame with a large number of variables than data with a small number of variables. For pdf output, Korean Gothic font must be installed in Korean operating system.

Value

No return value. This function only generates a report.

Reported information

The EDA process will report the following information:

• Introduction

  • Information of Dataset

  • Information of Variables

  • About EDA Report

• Univariate Analysis

  • Descriptive Statistics

  • Normality Test of Numerical Variables

    • Statistics and Visualization of (Sample) Data

• Relationship Between Variables

  • Correlation Coefficient

    • Correlation Coefficient by Variable Combination

    • Correlation Plot of Numerical Variables

• Target based Analysis

  • Grouped Descriptive Statistics

    • Grouped Numerical Variables

    • Grouped Categorical Variables

  • Grouped Relationship Between Variables

    • Grouped Correlation Coefficient

    • Grouped Correlation Plot of Numerical Variables

See vignette("EDA") for an introduction to these concepts.

See Also

eda_report.data.frame.

Examples

# If you have the 'DBI' and 'RSQLite' packages installed, perform the code block:
if (FALSE) {
library(dplyr)

# Generate data for the example
heartfailure2 <- heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 20), "platelets"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "smoking"] <- NA

# connect DBMS
con_sqlite <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")

# copy heartfailure2 to the DBMS with a table named TB_HEARTFAILURE
copy_to(con_sqlite, heartfailure2, name = "TB_HEARTFAILURE", overwrite = TRUE)

## target variable is categorical variable
# reporting the EDA information
# create pdf file. file name is EDA_Report.pdf
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_report(death_event)

# create pdf file. file name is EDA_TB_HEARTFAILURE.pdf
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_report("death_event", output_file = "EDA_TB_HEARTFAILURE.pdf")

# create html file. file name is EDA_Report.html
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_report("death_event", output_format = "html")

# create html file. file name is EDA_TB_HEARTFAILURE.html
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_report(death_event, output_format = "html", output_file = "EDA_TB_HEARTFAILURE.html")

## target variable is numerical variable
# reporting the EDA information, and collect size is 250
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_report(sodium, collect_size = 250)

# create pdf file. file name is EDA2.pdf
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_report("sodium", output_file = "EDA2.pdf")

# create html file. file name is EDA_Report.html
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_report("sodium", output_format = "html")

# create html file. file name is EDA2.html
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_report(sodium, output_format = "html", output_file = "EDA2.html")

## target variable is null
# reporting the EDA information
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_report()

# create pdf file. file name is EDA2.pdf
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_report(output_file = "EDA2.pdf")

# create html file. file name is EDA_Report.html
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_report(output_format = "html")

# create html file. file name is EDA2.html
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_report(output_format = "html", output_file = "EDA2.html")
  
# Disconnect DBMS   
DBI::dbDisconnect(con_sqlite)
}

Description

The eda_web_report() report the information of exploratory data analysis for object inheriting from data.frame.

Usage

eda_web_report(.data, ...)

## S3 method for class 'data.frame'
eda_web_report(
  .data,
  target = NULL,
  output_file = NULL,
  output_dir = tempdir(),
  browse = TRUE,
  title = "EDA",
  subtitle = deparse(substitute(.data)),
  author = "dlookr",
  title_color = "gray",
  logo_img = NULL,
  create_date = Sys.time(),
  theme = c("orange", "blue"),
  sample_percent = 100,
  is_tbl_dbi = FALSE,
  base_family = NULL,
  ...
)

Arguments

Details

Generate generalized EDA report automatically. This feature is useful for EDA of data with many variables, rather than data with fewer variables.

Value

No return value. This function only generates a report.

Reported information

Reported from the EDA is as follows.

• Overview

  • Data Structures

  • Data Types

  • Job Information

• Univariate Analysis

  • Descriptive Statistics

  • Normality Test

• Bivariate Analysis

  • Compare Numerical Variables

  • Compare Categorical Variables

• Multivariate Analysis

  • Correlation Analysis

    • Correlation Matrix

    • Correlation Plot

• Target based Analysis

  • Grouped Numerical Variables

  • Grouped Categorical Variables

  • Grouped Correlation

The base_family is selected from "Roboto Condensed", "Liberation Sans Narrow", "NanumSquare", "Noto Sans Korean". If you want to use a different font, use it after loading the Google font with import_google_font().

See Also

eda_web_report.tbl_dbi.

Examples

if (FALSE) {
# create the dataset
heartfailure2 <- dlookr::heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 20), "sodium"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "smoking"] <- NA

# create html file. file name is EDA_Report.html
eda_web_report(heartfailure2)

# file name is EDA.html. and change logo image
logo <- file.path(system.file(package = "dlookr"), "report", "R_logo_html.svg")
eda_web_report(heartfailure2, logo_img = logo, title_color = "black",
  output_file = "EDA.html")

# file name is ./EDA_heartfailure.html, "blue" theme and not browse
eda_web_report(heartfailure2, target = "death_event", output_dir = ".", 
  author = "Choonghyun Ryu", output_file = "EDA_heartfailure.html", 
  theme = "blue", browse = FALSE)
}

Description

The eda_web_report() report the information of exploratory data analysis for the DBMS table through tbl_dbi

Usage

## S3 method for class 'tbl_dbi'
eda_web_report(
  .data,
  target = NULL,
  output_file = NULL,
  output_dir = tempdir(),
  browse = TRUE,
  title = "EDA",
  subtitle = deparse(substitute(.data)),
  author = "dlookr",
  title_color = "gray",
  logo_img = NULL,
  create_date = Sys.time(),
  theme = c("orange", "blue")[1],
  sample_percent = 100,
  in_database = FALSE,
  collect_size = Inf,
  as_factor = TRUE,
  ...
)

Arguments

Details

Generate generalized EDA report automatically. This feature is useful for EDA of data with many variables, rather than data with fewer variables.

Value

No return value. This function only generates a report.

Reported information

Reported from the EDA is as follows.

• Overview

  • Data Structures

  • Data Types

  • Job Information

• Univariate Analysis

  • Descriptive Statistics

  • Normality Test

• Bivariate Analysis

  • Compare Numerical Variables

  • Compare Categorical Variables

• Multivariate Analysis

  • Correlation Analysis

    • Correlation Matrix

    • Correlation Plot

• Target based Analysis

  • Grouped Numerical Variables

  • Grouped Categorical Variables

  • Grouped Correlation

See Also

eda_web_report.data.frame.

Examples

# If you have the 'DBI' and 'RSQLite' packages installed, perform the code block:
if (FALSE) {
library(dplyr)

# Generate data for the example
heartfailure2 <- heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 20), "platelets"] <- NA
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "smoking"] <- NA

# connect DBMS
con_sqlite <- DBI::dbConnect(RSQLite::SQLite(), ":memory:")

# copy heartfailure2 to the DBMS with a table named TB_HEARTFAILURE
copy_to(con_sqlite, heartfailure2, name = "TB_HEARTFAILURE", overwrite = TRUE)

# reporting the diagnosis information -------------------------
# create pdf file. file name is EDA_Report.html
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_web_report(target = "death_event")
  
# create pdf file. file name is EDA.html, and collect size is 250
con_sqlite %>% 
  tbl("TB_HEARTFAILURE") %>% 
  eda_web_report(collect_size = 250, output_file = "EDA.html")
  
# Disconnect DBMS   
DBI::dbDisconnect(con_sqlite)
}

Description

Calculate the Shannon's entropy.

Usage

entropy(x)

Arguments

Value

numeric. entropy

Examples

set.seed(123)
x <- sample(1:10, 20, replace = TRUE)

entropy(x)

Description

The extract() extract binned variable from "bins", "optimal_bins" class object.

Usage

extract(x)

## S3 method for class 'bins'
extract(x)

Arguments

Details

The "bins" and "optimal_bins" class objects use the summary() and plot() functions to diagnose the performance of binned results. This function is used to extract the binned result if you are satisfied with the result.

Value

factor.

See Also

binning, binning_by.

Examples

library(dplyr)

# Generate data for the example
heartfailure2 <- heartfailure
heartfailure2[sample(seq(NROW(heartfailure2)), 5), "creatinine"] <- NA

# optimal binning using binning_by()
bin <- binning_by(heartfailure2, "death_event", "creatinine")
bin

if (!is.null(bin)) {
  # extract binning result
  extract(bin) %>% 
    head(20)
}

Description

The find_class() extracts variable information having a certain class from an object inheriting data.frame.

Usage

find_class(
  df,
  type = c("numerical", "categorical", "categorical2", "date_categorical",
    "date_categorical2"),
  index = TRUE
)