Data Analysis

Schedule of lectures

1. Basic concepts of theory of hypothesis: null- and alternative-hypetheses, test statistic, acceptance- and critical regions, errors (type 1 and 2), power-function, level of significance, strength of the test, unbiasedness, consistency. The distributions origined from the normal distribution:  chi-squared, F-, t-. Lukacs’s Theorem. Parameters, parametric tests.
2. Parametric tests for the parameters of the normal distribution: one-sample t- and u-tests, two independent sample u- and t-tests, paired two-tailed t-test, F-test, Welch's t-test, Bartlett test.
3. Non-parametric tests I. Fundamental Theorem of the Chi-square tests. Pure and estimated Chi-square fitting tests. Chi-square test for independence. Two independent sample Chi-square test for checking homogeneity of samples.
4. Non-parametric tests II. Gnegyenko-Koroljuk Theorem. Order Statistics,  Fit Testing with one sample Kolmogorov-Smirnov test. Testing Homogeneity with two-sample Kolmogorov-Smirnov test.
5. Tets of Homogeneity. Checking homogeneity of two independent sample with Mann-Whitney test. Checking homogeneity of several independent samples with Kruskal-Wallis test. Checking homogeneity two paired samples with Wilcoxon test. Checking homogeneity several paired samples with Friedmann test.
6. Bivariate regression models. Theoretical background: the conditional expectation. The bivariate regression types: linear regression, polynomial regression, linear regression can be traced back two parameters. Logistic regression. The least squares method. Analysis of Variance (ANOVA) to decide the validity of the model. Coefficient of determination (R-squared). Nadaraja-method.
7. Multivariate linear regression. Techniques of Model construction. Correlation coefficients: total-, multiple-, part-. The beta coefficients. In the adjusted coefficient of determination. Multicollinearity. Heteroskedasticity. Outlier points detection and analysis.
8. Introduction to Business Intelligence and practical considerations. Data driven solution, CRISP-DM (CRoss Industry Standard Process for Data Mining). Customer Relationship Management (CRM) analytics.
9. Preparation for enterprise data analytics. Frequent item sets, market busket analysis, association rules and their application in practice.
10. Supervised machine learning. Evaluation metrics. Profit matrix. Simple algorithms: kNN, Naive-Bayes and useful metrics.
11. Decision trees (DT) and their application to decision making. Some DT algorithms (C4.5, purity measures, splitting, pre- and post-pruningés), applications.
12. Classification and regression. Customer value and churn prediction. Credibility prediction. Prediction in direct marketing campaign.
13. Customer segmentation and other unsupervised learning problems. Various models for clustering by K-means (bisecting and adaptive). Density based clustering algorithms (DBSCAN, OPTICS) and hierarchical clustering and their application.
14. Open and/or widely used data mining softwares, model building and practical constrains.

       Practical lessons

1. Introduction to statistical software packages. Definition of basic statistics and their interpretation. Plots: scatter, box, pie chart and histograms, P-P and Q-Q plots, Dot plot. Confidence interval, parametric tests over economic type data sets.
2. Statistical hypothesis testing over enterprise and business datasets. Independent testing and homogeneity, significance testing. Graphical and statistical analysis.
3. Regression analysis over business type data sets. Model building and constrains. Evaluation: Multicollinearity, heteroscedasticity, sensitivity and outliers.
4. Statistical data mining. Decision making with basic algorithms for credibility analysis.
5. Frequent item sets over a bookstore dataset. Various quality measures of association rules (“list”) and their connection to decision trees.
6. Modeling user behavior over a webshop and predict whether the user is going to buy something.
7. Segmentation of customers according to past activities, group work with available tools.