From e74878d42e7eb4353185950f94eea377e0312379 Mon Sep 17 00:00:00 2001 From: PyCaret Date: Tue, 25 Feb 2020 18:51:16 -0500 Subject: [PATCH] Delete Binary Classification Tutorial (CLF102) - Level Intermediate.ipynb --- ...torial (CLF102) - Level Intermediate.ipynb | 16237 ---------------- 1 file changed, 16237 deletions(-) delete mode 100644 Tutorials/Binary Classification Tutorial (CLF102) - Level Intermediate.ipynb diff --git a/Tutorials/Binary Classification Tutorial (CLF102) - Level Intermediate.ipynb b/Tutorials/Binary Classification Tutorial (CLF102) - Level Intermediate.ipynb deleted file mode 100644 index d158538..0000000 --- a/Tutorials/Binary Classification Tutorial (CLF102) - Level Intermediate.ipynb +++ /dev/null @@ -1,16237 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Binary Classification Tutorial (CLF102) - Level Intermediate" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Date Updated: Feb 25, 2020**\n", - "\n", - "# 1.0 Objective of Tutorial\n", - "Welcome to Binary Classification Tutorial (#CLF102). This tutorial assumes that you have completed __[Binary Classification Tutorial - Level Beginner (#CLF101)](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__. If you haven't used PyCaret before and this is your first tutorial, we strongly recommend you to go back and progress through the beginner's tutorial to understand basics of working in PyCaret.\n", - "\n", - "In this tutorial using `pycaret.classification` module we will learn:\n", - "\n", - "* **Normalization:** How to normalize and scale the dataset?\n", - "* **Transformation:** How to apply transformations that makes data linear and approximately normal?\n", - "* **Ignore Low Variance:** How to remove features with statistically insignificant variances to make experiment more efficient?\n", - "* **Remove Multi-collinearity:** How to remove multi-collinearity from the dataset to boost performance of Linear algorithms?\n", - "* **Group Features:** How to extract statistical information from related features in the dataset?\n", - "* **Bin Numeric Variables:** How to bin numeric variables and transform numeric features into categorical using 'sturges' rule? \n", - "* **Model Ensembling and Stacking:** How to boost model performance using several ensembling techniques such as Bagging, Boosting, Soft/hard Voting and Generalized Stacking.\n", - "* **Tuning Hyperparameters of Ensemblers:** How to tune hyperparameters of ensemblers? \n", - "* **Model Calibration:** How to calibrate probabilities of a classification model?\n", - "* **Save / Load Experiment:** How to save/load an entire experiment?\n", - "\n", - "Read Time : Approx 60 Minutes\n", - "\n", - "\n", - "## 1.1 Installing PyCaret\n", - "If you haven't installed PyCaret yet. Please follow the link to __[Beginner's Tutorial](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__ for instruction on how to install pycaret.\n", - "\n", - "## 1.2 Pre-Requisites\n", - "- Python 3.x\n", - "- Latest version of pycaret\n", - "- Internet connection to load data from pycaret's repository\n", - "- Completion of Binary Classification Tutorial (CLF101) - Level Beginner\n", - "\n", - "## 1.3 For Google colab users:\n", - "If you are running this notebook on Google colab, below code of cells must be run at top of the notebook to display interactive visuals.
\n", - "
\n", - "`from pycaret.utils import enable_colab`
\n", - "`enable_colab()`\n", - "\n", - "\n", - "## 1.4 See also:\n", - "- __[Binary Classification Tutorial (CLF101) - Level Beginner](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__\n", - "- __[Binary Classification Tutorial (CLF103) - Level Expert](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF103)%20-%20Level%20Expert.ipynb)__" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 2.0 Brief overview of techniques covered in this tutorial\n", - "Before we practically get into execution of the techniques mentioned above in the Section 1 (Objective of Tutorial), it is important to understand what are these techniques and when to use them. More often than not most of these techniques will help linear and parametric algorithms, however it is no surprise to see performance gain in tree-based models. Below explanations are only brief and we recommend you to do extra reading to dive deeper to get more understanding of these techniques.\n", - "\n", - "- **Normalization:** Normalization / Scaling (often used interchangeably with standardization) is used to transform the actual values of numeric variables in a way that the transformed values have helpful properties for machine learning. Many machine learning algorithms such as Logistic Regression, Support Vector Machine, K Nearest Neighbors and Naive Bayes assume that all features are centered around zero and have variance in the same order. If a particular feature in dataset has a variance that is larger from other features in order of magnitude, the model may not understand all features correctly and may perform poorly. For example, in the dataset we are using for this example, it contains `AGE` feature which ranges between 21 to 79 while other numeric features are in range of 10,000 to 1,000,000. __[Read more](https://sebastianraschka.com/Articles/2014_about_feature_scaling.html#z-score-standardization-or-min-max-scaling)__
\n", - "
\n", - "- **Transformation:** While normalization transforms the range of data to eradicate the impact of magnitude in variance, transformation is more radical technique as it changes the shape of the distribution so that transformed data can be represented by normal or approximate normal distirbution. In general, you should transform the data if you're going to use algorithms that assumes normality or gaussian distribution. Some examples of such models are Logistic Regression, Linear Discriminant Analysis (LDA) and Gaussian Naive Bayes. (Pro tip: any method with “Gaussian” in the name probably assumes normality.) __[Read more](https://en.wikipedia.org/wiki/Power_transform)__
\n", - "
\n", - "- **Ignore Low Variance:** Datasets come sometimes with categorical features that take an unique value across samples. This kind of feature is not only non-informative and adds no value but are also sometimes harmful for few algorithms. Imagine a feature with only one unique value or few dominant unique values accross samples, they can be removed from the dataset by using ignore low variance feature in PyCaret.
\n", - "
\n", - "- **Multi-collinearity:** Multi-collinearity is a state of very high intercorrelations or inter-associations among the independent features in the dataset. It is therefore a type of disturbance in the data that is not handled well by machine learning models (mostly linear algorithms). Multi-collinearity may reduce overall coefficient of the model and cause unpredictable variance. This will lead to overfitting where the model may do great on known training set but will fail at unknown testing set. __[Read more](https://towardsdatascience.com/multicollinearity-in-data-science-c5f6c0fe6edf)__
\n", - "
\n", - "- **Group Features:** Sometimes dataset may contain features that are related at a sample level, for example in the `credit` dataset we are using it containts feature `BILL_AMT1 .. BILL_AMT6` which is related in a way that `BILL_AMT1` is amount of bill 1 month ago and `BILL_AMT6` is amount of bill 6 months ago. Such features can be used to extract additional features based on statistical properties of the distribution such as mean, median, variance, standard deviation etc.
\n", - "
\n", - "- **Bin Numeric Variables:** Binning or discretization is the process of transforming numerical variables into categorical features, An example would be Age variable. Age is a continious distribution of numeric value that can be discretize into intervals (10-20 years, 21-30 etc.). Binning may improve accuracy of the predictive models by reducing the noise or non-linearity in the data. PyCaret automatically determines the number and size of bins using Sturges rule. __[Read more](https://www.vosesoftware.com/riskwiki/Sturgesrule.php)__
\n", - "
\n", - "- **Model Ensembling and Stacking:** Ensemble modeling is a process where multiple diverse models are created to predict an outcome, either by using many different modeling algorithms or using different sample of training data sets. The ensemble model then aggregates the prediction of each base model and results in once final prediction for the unseen data. The motivation for using ensemble models is to reduce the generalization error of the prediction. As long as the base models are diverse and independent, the prediction error of the model decreases when the ensemble approach is used. Two most common method in ensemble learning are `Bagging` and `Boosting`. Stacking is also a type of ensemble learning where predictions from multiple models are used as an input feature for a meta model that predicts the final outcome using input from base learners. __[Read more](https://blog.statsbot.co/ensemble-learning-d1dcd548e936)__
\n", - "
\n", - "- **Tuning Hyperparameters of Ensemblers:** Similar to hyperparameters of a machine learning model, we will also learn how to tune hyperparameter of an ensembler model." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 3.0 Dataset for the Tutorial" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For this tutorial we will be using the same dataset that was used in __[CLF101 - Beginner Tutorial.](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__\n", - "\n", - "#### Dataset Acknowledgements:\n", - "Lichman, M. (2013). UCI Machine Learning Repository. Irvine, CA: University of California, School of Information and Computer Science.\n", - "\n", - "The original dataset and data dictionary can be __[found here](https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients)__ at the UCI Machine Learning Repository." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 4.0 Getting the Data" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "You can download the data from the original source __[found here](https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients)__ and load it using pandas __[(Learn How)](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)__ or you can use PyCaret's data respository to load the data using get_data function (This will require internet connection)." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "\n", - " \n", - " " - ], - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "text/plain": [] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "from pycaret.datasets import get_data\n", - "dataset = get_data('credit', profile=True)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that when `profile` parameter is to `True`, it displays data profile for Exploratory Data Analysis. Several pre-processing steps as discussed in section 2 above is performed in this experiment based on this analysis. Let's summarize how this analysis have helped us making critical choices in pre-processing the data.\n", - "\n", - "- **Missing Values:** There are no missing values in the data. However, we still need imputers in our pipeline just in case the new unseen data has missing values (not applicable in this case). When you execute `setup()` function, imputers are created and stored in pipeline automatically. By default, it uses mean imputer for numeric values and constant imputer for categorical but can be changed using `numeric_imputation` and `categorical_imputation` parameter in `setup()`.
\n", - "
\n", - "- **Multicollinearity:** There are high correlations between several `BILL_AMT1 ... BIL_AMT6` which introduces multicollinearity in the data. We will remove multi-collinearity by using `remove_multicollinearity` and `multicollinearity_threshold` parameter in setup.
\n", - "
\n", - "- **Data Scale / Range:** Notice that how the scale / range of numeric features are different. For example feature `AGE` ranges between 21 to 79 and `BILL_AMT1` ranges from -165,580 to 964,511. This may cause problem for algorithms that assumes that all features have variance in the same order. In this case, order of magnitude in change for `BILL_AMT1` is widely different than `AGE`. We will deal with this problem by using `normalize` parameter in setup.
\n", - "
\n", - "- **Distribution of Feature Space:** Numeric features are not normally distributed. Look at the distribution of `LIMIT_BAL`, `BILL_AMT1` and `PAY_AMT1 ... PAY_AMT6`. Few features are also highly skewed for e.g. `PAY_AMT1`. This may cause problem for algorithms that assumes normal or approximate normal distribution of the data for example Logistic Regression, Linear Discriminant Analysis (LDA) and Naive Bayes. We will deal with this problem by using `transformation` parameter in setup.
\n", - "
\n", - "- **Group Features:** From the data description we know certain features are related with each other. For example `BILL_AMT1 ... BILL_AMT6` is related. Similarly, `PAY_AMT1 ... PAY_AMT6` is related. We will use `group_features` parameter in setup to extract statistical information from these features.
\n", - "
\n", - "- **Bin Numeric Features:** Looking at the correlation between numeric features with target variable, we see a very weak relation of `AGE` and `LIMIT_BAL` with target variable. We will use `bin_numeric_features` parameter to remove the noise from these variables. This may help linear algorithms.
" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "(24000, 24)" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "#check the shape of data\n", - "dataset.shape" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In order to demonstrate the `predict_model()` function on unseen data, a sample of 1200 rows are taken out from original dataset to be used for predictions. This should not be confused with train/test split. This split is performed to simulate real life scenario. Another way to think about this is that these 1200 records were not available at the time when machine learning experiment was performed." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Data for Modeling: (22800, 24)\n", - "Unseen Data For Predictions (1200, 24)\n" - ] - } - ], - "source": [ - "data = dataset.sample(frac=0.95, random_state=786).reset_index(drop=True)\n", - "data_unseen = dataset.drop(data.index).reset_index(drop=True)\n", - "\n", - "print('Data for Modeling: ' + str(data.shape))\n", - "print('Unseen Data For Predictions ' + str(data_unseen.shape))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 6.0 Setting up Environment in PyCaret" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In previous tutorial __[Binary Classification (#CLF101) - Level Beginner](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__ we have learned how to initializes the environment in pycaret using `setup()`. You would remember that we have not passed any additional parameters in our last example as we didn't performed any pre-processing (other than those that are imperative for machine learning experiments and they are performed automatically by PyCaret). In this example we will take it to the next level by customizing the pre-processing pipeline using `setup()`. Let's see how to implement all the steps discussed in section 4 above." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "from pycaret.classification import *" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": { - "scrolled": false - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - " \n", - "Setup Succesfully Completed!\n" - ] - }, - { - "data": { - "text/html": [ - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
Description Value
0session_id123
1Target TypeBinary
2Label EncodedNone
3Original Data(22800, 24)
4Missing Values False
5Numeric Features 14
6Categorical Features 9
7Ordinal Features False
8High Cardinality Features False
9High Cardinality Method None
10Sampled Data(22800, 24)
11Transformed Train Set(15959, 125)
12Transformed Test Set(6841, 125)
13Numeric Imputer mean
14Categorical Imputer constant
15Normalize True
16Normalize Method zscore
17Transformation True
18Transformation Method yeo-johnson
19PCA False
20PCA Method None
21PCA Components None
22Ignore Low Variance True
23Combine Rare Levels False
24Rare Level Threshold None
25Numeric Binning True
26Remove Outliers False
27Outliers Threshold None
28Remove Multicollinearity True
29Multicollinearity Threshold 0.95
30Clustering False
31Clustering Iteration None
32Polynomial Features False
33Polynomial Degree None
34Trignometry Features False
35Polynomial Threshold None
36Group Features True
37Feature Selection False
38Features Selection Threshold None
39Feature Interaction False
40Feature Ratio False
41Interaction Threshold None
" - ], - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "exp_clf102 = setup(data = data, target = 'default', session_id=123,\n", - " normalize = True, \n", - " transformation = True, \n", - " ignore_low_variance = True,\n", - " remove_multicollinearity = True, multicollinearity_threshold = 0.95,\n", - " bin_numeric_features = ['LIMIT_BAL', 'AGE'],\n", - " group_features = [['BILL_AMT1', 'BILL_AMT2','BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6'],\n", - " ['PAY_AMT1','PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']]) " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Note that it's the same setup grid that was shown in __[Binary Classification Tutorial (#CLF101) - Level Beginner](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__. The only difference here is the customization that we have performed in `setup()` is now set to `True`. Also notice that `session_id` is same `123` as beginner's level, which means effect of randomization is completely isolated. Any improvements we see in this experiment is solely due to the pre-processing steps taken in `setup()` or any other modeling techniques we use in later sections of this tutorial." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 7.0 Comparing All Models" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Similar to __[Binary Classification Tutorial (#CLF101) - Level Beginner](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__ we will also begin this tutorial with `compare_models()`. We will then compare the below results with the last experiment." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
Model Accuracy AUC Recall Prec. F1 Kappa
0Gradient Boosting Classifier0.82370.78740.3680.69180.48010.3857
1Ridge Classifier0.823600.36460.6930.47760.3836
2Extreme Gradient Boosting0.82320.78530.36180.6930.47510.3812
3Linear Discriminant Analysis0.82240.7780.37960.67620.4860.3887
4SVM - Linear Kernel0.82200.33430.70830.45390.3635
5Light Gradient Boosting Machine0.8220.78290.38190.67240.48690.3889
6Logistic Regression0.82170.78040.35920.68560.47120.3763
7CatBoost Classifier0.8210.78590.37960.6690.48420.3857
8Ada Boost Classifier0.81910.78220.3380.68520.45240.3587
9Extra Trees Classifier0.8170.76340.38240.64670.48030.378
10Random Forest Classifier0.80910.7350.33260.63020.43510.3332
11Naive Bayes0.80440.7530.24990.65870.35790.2701
12K Neighbors Classifier0.78960.70050.3280.54050.40810.2894
13Decision Tree Classifier0.73120.62250.42580.39940.4120.2381
14Quadratic Discriminant Analysis0.72940.74190.26490.56750.24150.1379
" - ], - "text/plain": [ - "" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "compare_models()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For the purpose of comparison we will use `AUC` this time. Notice that how drastically few algorithms have improved after we have performed few pre-processing steps in `setup()`. \n", - "- Logistic Regression AUC improved from `0.6508` to `0.7804`\n", - "- Naives Bayes AUC improved from `0.6457` to `0.7530`\n", - "- K Nearest Neighbors AUC improved from `0.6099` to `0.7005`\n", - "\n", - "To see results for all the models from previous tutorial refer to Section 7 in __[Binary Classification Tutorial (#CLF101) - Level Beginner](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 8.0 Create a Model" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In previous tutorial __[Binary Classification (#CLF101) - Level Beginner](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__ we have learned how to create a model using `create_model()` function. Now we will learn about few other parameters in `create_model()` that may come handy sometimes. In this section of the tutorial, we will create all models using 5 fold stratified cross validation, notice how `fold` parameter is passed inside `create_model()` to achieve this." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 8.1 Create Model (change fold to 5)" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.72780.61950.41930.39210.40520.2289
10.73340.62300.42210.40220.41190.2396
20.73460.62310.42210.40430.41300.2417
30.73500.61430.39800.40030.39910.2291
40.72580.62340.43910.39290.41470.2364
Mean0.73130.62070.42010.39830.40880.2352
SD0.00380.00350.01310.00500.00580.0053
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.7278 0.6195 0.4193 0.3921 0.4052 0.2289\n", - "1 0.7334 0.6230 0.4221 0.4022 0.4119 0.2396\n", - "2 0.7346 0.6231 0.4221 0.4043 0.4130 0.2417\n", - "3 0.7350 0.6143 0.3980 0.4003 0.3991 0.2291\n", - "4 0.7258 0.6234 0.4391 0.3929 0.4147 0.2364\n", - "Mean 0.7313 0.6207 0.4201 0.3983 0.4088 0.2352\n", - "SD 0.0038 0.0035 0.0131 0.0050 0.0058 0.0053" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "dt = create_model('dt', fold = 5)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 8.2 Create Model (round to 2 decimals points)" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.730.620.420.390.410.23
10.730.620.420.400.410.24
20.730.620.420.400.410.24
30.730.610.400.400.400.23
40.730.620.440.390.410.24
Mean0.730.620.420.400.410.24
SD0.000.000.010.000.010.01
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.73 0.62 0.42 0.39 0.41 0.23\n", - "1 0.73 0.62 0.42 0.40 0.41 0.24\n", - "2 0.73 0.62 0.42 0.40 0.41 0.24\n", - "3 0.73 0.61 0.40 0.40 0.40 0.23\n", - "4 0.73 0.62 0.44 0.39 0.41 0.24\n", - "Mean 0.73 0.62 0.42 0.40 0.41 0.24\n", - "SD 0.00 0.00 0.01 0.00 0.01 0.01" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "dt = create_model('dt', fold = 5, round = 2)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice how passing `round` parameter inside `create_model()` has rounded the evaluation metrics to 2 decimals. The score grid printed in 8.1 and 8.2 is exactly similar except that scores in 8.2 is rounded." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 9.0 Tune a Model" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In previous tutorial __[Binary Classification (#CLF101) - Level Beginner](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__ we have learned how to automatically tune hyperparameters of a model using pre-defined grids. In this tutorial we will introduce the use of `optimize` parameter in `tune_model()`. Optimize parameter can be thought of as an objective function. By default, in `pycaret.classification` all hyperparameter tuning is set to optimize `Accuracy` which can be changed using `optimize` parameter. See the example below:" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.83150.78030.38530.72340.50280.4125
10.82080.79040.34840.68720.46240.3684
20.82080.81650.39380.65880.49290.3923
30.82330.76910.35690.69610.47190.3788
40.81580.77210.34840.65780.45560.3571
50.81270.78250.33140.65000.43900.3405
60.81330.76390.34840.64400.45220.3515
70.83400.78710.37110.75290.49720.4112
80.82140.76570.35690.68480.46930.3745
90.81070.78390.35130.62940.45090.3475
Mean0.82040.78110.35920.67840.46940.3734
SD0.00740.01460.01790.03630.02060.0242
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8315 0.7803 0.3853 0.7234 0.5028 0.4125\n", - "1 0.8208 0.7904 0.3484 0.6872 0.4624 0.3684\n", - "2 0.8208 0.8165 0.3938 0.6588 0.4929 0.3923\n", - "3 0.8233 0.7691 0.3569 0.6961 0.4719 0.3788\n", - "4 0.8158 0.7721 0.3484 0.6578 0.4556 0.3571\n", - "5 0.8127 0.7825 0.3314 0.6500 0.4390 0.3405\n", - "6 0.8133 0.7639 0.3484 0.6440 0.4522 0.3515\n", - "7 0.8340 0.7871 0.3711 0.7529 0.4972 0.4112\n", - "8 0.8214 0.7657 0.3569 0.6848 0.4693 0.3745\n", - "9 0.8107 0.7839 0.3513 0.6294 0.4509 0.3475\n", - "Mean 0.8204 0.7811 0.3592 0.6784 0.4694 0.3734\n", - "SD 0.0074 0.0146 0.0179 0.0363 0.0206 0.0242" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "tuned_rf = tune_model('rf')" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.82710.78250.35410.72250.47530.3859
10.82270.79730.33990.70590.45890.3680
20.82270.82000.37390.68040.48260.3864
30.82020.76920.33990.68970.45540.3623
40.82080.77550.33710.69590.45420.3621
50.81700.78920.33430.67430.44700.3520
60.81520.75480.32860.66670.44020.3445
70.82830.78930.34280.74230.46900.3827
80.82080.76010.34840.68720.46240.3684
90.81250.78080.35130.63920.45340.3516
Mean0.82070.78190.34500.69040.45980.3664
SD0.00470.01790.01210.02750.01220.0141
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8271 0.7825 0.3541 0.7225 0.4753 0.3859\n", - "1 0.8227 0.7973 0.3399 0.7059 0.4589 0.3680\n", - "2 0.8227 0.8200 0.3739 0.6804 0.4826 0.3864\n", - "3 0.8202 0.7692 0.3399 0.6897 0.4554 0.3623\n", - "4 0.8208 0.7755 0.3371 0.6959 0.4542 0.3621\n", - "5 0.8170 0.7892 0.3343 0.6743 0.4470 0.3520\n", - "6 0.8152 0.7548 0.3286 0.6667 0.4402 0.3445\n", - "7 0.8283 0.7893 0.3428 0.7423 0.4690 0.3827\n", - "8 0.8208 0.7601 0.3484 0.6872 0.4624 0.3684\n", - "9 0.8125 0.7808 0.3513 0.6392 0.4534 0.3516\n", - "Mean 0.8207 0.7819 0.3450 0.6904 0.4598 0.3664\n", - "SD 0.0047 0.0179 0.0121 0.0275 0.0122 0.0141" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "tuned_rf2 = tune_model('rf', optimize = 'AUC')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that how the `optimize` parameter used in tuning Random Forest Classifier resulted in two different models. In `tuned_rf` where no optimize parameter was defined, it uses default function `Accuracy` to optimize and resulted in AUC of `0.7811`. In `tuned_rf2` we set `optimize` parameter to `AUC`, it resulted in AUC of `0.7819`. Notice the differences between hyperparameters of `tuned_rf` and `tuned_rf2` below:" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
Parameters
bootstrapTrue
ccp_alpha0
class_weightNone
criterionentropy
max_depth20
max_featuressqrt
max_leaf_nodesNone
max_samplesNone
min_impurity_decrease0
min_impurity_splitNone
min_samples_leaf4
min_samples_split5
min_weight_fraction_leaf0
n_estimators50
n_jobsNone
oob_scoreFalse
random_state123
verbose0
warm_startFalse
\n", - "
" - ], - "text/plain": [ - " Parameters\n", - "bootstrap True\n", - "ccp_alpha 0\n", - "class_weight None\n", - "criterion entropy\n", - "max_depth 20\n", - "max_features sqrt\n", - "max_leaf_nodes None\n", - "max_samples None\n", - "min_impurity_decrease 0\n", - "min_impurity_split None\n", - "min_samples_leaf 4\n", - "min_samples_split 5\n", - "min_weight_fraction_leaf 0\n", - "n_estimators 50\n", - "n_jobs None\n", - "oob_score False\n", - "random_state 123\n", - "verbose 0\n", - "warm_start False" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "#tuned_rf optimize parameter default to 'Accuracy'\n", - "plot_model(tuned_rf, plot = 'parameter')" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
Parameters
bootstrapTrue
ccp_alpha0
class_weightNone
criteriongini
max_depth10
max_featuresauto
max_leaf_nodesNone
max_samplesNone
min_impurity_decrease0
min_impurity_splitNone
min_samples_leaf2
min_samples_split10
min_weight_fraction_leaf0
n_estimators70
n_jobsNone
oob_scoreFalse
random_state123
verbose0
warm_startFalse
\n", - "
" - ], - "text/plain": [ - " Parameters\n", - "bootstrap True\n", - "ccp_alpha 0\n", - "class_weight None\n", - "criterion gini\n", - "max_depth 10\n", - "max_features auto\n", - "max_leaf_nodes None\n", - "max_samples None\n", - "min_impurity_decrease 0\n", - "min_impurity_split None\n", - "min_samples_leaf 2\n", - "min_samples_split 10\n", - "min_weight_fraction_leaf 0\n", - "n_estimators 70\n", - "n_jobs None\n", - "oob_score False\n", - "random_state 123\n", - "verbose 0\n", - "warm_start False" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "#tuned_rf optimize parameter set to 'AUC'\n", - "plot_model(tuned_rf2, plot = 'parameter')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "___" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 10.0 Ensemble a Model" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Another common technique to improve performance of models is ensembling. Ensemble models in machine learning combine the decisions from multiple models to improve the overall performance. There are various techniques of ensembling that we will cover in this section. In order to ensemble models using `Bagging` or `Boosting` technique __[(Read More)](https://towardsdatascience.com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205)__, we use `ensemble_model()` function in PyCaret. This function ensembles the trained base estimator using the method defined in `method` parameter." - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.72370.61360.41640.38480.40000.2209
10.73310.62600.42490.40210.41320.2406
20.73180.62770.43910.40260.42010.2461
30.72620.61930.42490.39060.40710.2295
40.72990.60500.37960.38730.38340.2105
50.74560.64270.45890.42970.44380.2792
60.72060.62010.44190.38520.41160.2295
70.74250.63190.43340.42030.42680.2608
80.73750.62020.40790.40680.40740.2387
90.72160.61840.43060.38480.40640.2253
Mean0.73120.62250.42580.39940.41200.2381
SD0.00820.00980.02040.01510.01530.0191
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.7237 0.6136 0.4164 0.3848 0.4000 0.2209\n", - "1 0.7331 0.6260 0.4249 0.4021 0.4132 0.2406\n", - "2 0.7318 0.6277 0.4391 0.4026 0.4201 0.2461\n", - "3 0.7262 0.6193 0.4249 0.3906 0.4071 0.2295\n", - "4 0.7299 0.6050 0.3796 0.3873 0.3834 0.2105\n", - "5 0.7456 0.6427 0.4589 0.4297 0.4438 0.2792\n", - "6 0.7206 0.6201 0.4419 0.3852 0.4116 0.2295\n", - "7 0.7425 0.6319 0.4334 0.4203 0.4268 0.2608\n", - "8 0.7375 0.6202 0.4079 0.4068 0.4074 0.2387\n", - "9 0.7216 0.6184 0.4306 0.3848 0.4064 0.2253\n", - "Mean 0.7312 0.6225 0.4258 0.3994 0.4120 0.2381\n", - "SD 0.0082 0.0098 0.0204 0.0151 0.0153 0.0191" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "# lets create a simple decision tree model that we will use for ensembling \n", - "dt = create_model('dt')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 10.1 Bagging" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.81140.74230.36830.62500.46350.3582
10.80640.73180.33710.61340.43510.3300
20.81830.75990.40230.64250.49480.3911
30.80010.71810.32580.58670.41890.3100
40.81330.72490.34280.64710.44810.3483
50.81330.73670.35410.64100.45620.3546
60.79640.71710.30880.57370.40150.2919
70.81700.74970.35130.66310.45930.3614
80.81390.72400.36260.64000.46290.3606
90.81000.74500.36830.61900.46180.3554
Mean0.81000.73490.35210.62520.45020.3461
SD0.00670.01350.02460.02640.02480.0270
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8114 0.7423 0.3683 0.6250 0.4635 0.3582\n", - "1 0.8064 0.7318 0.3371 0.6134 0.4351 0.3300\n", - "2 0.8183 0.7599 0.4023 0.6425 0.4948 0.3911\n", - "3 0.8001 0.7181 0.3258 0.5867 0.4189 0.3100\n", - "4 0.8133 0.7249 0.3428 0.6471 0.4481 0.3483\n", - "5 0.8133 0.7367 0.3541 0.6410 0.4562 0.3546\n", - "6 0.7964 0.7171 0.3088 0.5737 0.4015 0.2919\n", - "7 0.8170 0.7497 0.3513 0.6631 0.4593 0.3614\n", - "8 0.8139 0.7240 0.3626 0.6400 0.4629 0.3606\n", - "9 0.8100 0.7450 0.3683 0.6190 0.4618 0.3554\n", - "Mean 0.8100 0.7349 0.3521 0.6252 0.4502 0.3461\n", - "SD 0.0067 0.0135 0.0246 0.0264 0.0248 0.0270" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "bagged_dt = ensemble_model(dt)" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "BaggingClassifier(base_estimator=DecisionTreeClassifier(ccp_alpha=0.0,\n", - " class_weight=None,\n", - " criterion='gini',\n", - " max_depth=None,\n", - " max_features=None,\n", - " max_leaf_nodes=None,\n", - " min_impurity_decrease=0.0,\n", - " min_impurity_split=None,\n", - " min_samples_leaf=1,\n", - " min_samples_split=2,\n", - " min_weight_fraction_leaf=0.0,\n", - " presort='deprecated',\n", - " random_state=123,\n", - " splitter='best'),\n", - " bootstrap=True, bootstrap_features=False, max_features=1.0,\n", - " max_samples=1.0, n_estimators=10, n_jobs=None,\n", - " oob_score=False, random_state=123, verbose=0,\n", - " warm_start=False)\n" - ] - } - ], - "source": [ - "# check the parameter of bagged_dt\n", - "print(bagged_dt)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice how ensembling has improved the `AUC` from `0.6225` to `0.7349`. In above example we have used all default parameters of `ensemble_model()` which uses `Bagging` method. Let's try `Boosting` by changing the `method` parameter in `ensemble_model()`. See example below: " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 10.2 Boosting" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.78380.70710.39090.51490.44440.3134
10.77820.68080.32860.49790.39590.2670
20.77820.71860.39940.49820.44340.3070
30.78260.68890.37680.51150.43390.3032
40.77820.65210.28610.49750.36330.2407
50.78320.69590.39660.51280.44730.3152
60.78010.68810.36260.50390.42170.2904
70.78450.66340.35980.51840.42470.2974
80.77260.68890.40230.48300.43890.2978
90.77620.71120.36260.49230.41760.2830
Mean0.77970.68950.36660.50310.42310.2915
SD0.00360.01960.03460.01070.02480.0218
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.7838 0.7071 0.3909 0.5149 0.4444 0.3134\n", - "1 0.7782 0.6808 0.3286 0.4979 0.3959 0.2670\n", - "2 0.7782 0.7186 0.3994 0.4982 0.4434 0.3070\n", - "3 0.7826 0.6889 0.3768 0.5115 0.4339 0.3032\n", - "4 0.7782 0.6521 0.2861 0.4975 0.3633 0.2407\n", - "5 0.7832 0.6959 0.3966 0.5128 0.4473 0.3152\n", - "6 0.7801 0.6881 0.3626 0.5039 0.4217 0.2904\n", - "7 0.7845 0.6634 0.3598 0.5184 0.4247 0.2974\n", - "8 0.7726 0.6889 0.4023 0.4830 0.4389 0.2978\n", - "9 0.7762 0.7112 0.3626 0.4923 0.4176 0.2830\n", - "Mean 0.7797 0.6895 0.3666 0.5031 0.4231 0.2915\n", - "SD 0.0036 0.0196 0.0346 0.0107 0.0248 0.0218" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "boosted_dt = ensemble_model(dt, method = 'Boosting')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice how easy it is to ensemble models in PyCaret. Just by changing `method` parameter you can do Bagging or Boosting which otherwise would have taken multiple lines of code. Note that `ensemble_model()` by default build `10` estimators, however, this can be changed using `n_estimators` parameter inside `ensemble_model()`. Increase `n_estimators` sometimes improves the result. See an example below:" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.82140.75880.39660.66040.49560.3952
10.82080.76340.36830.67360.47620.3791
20.82460.78590.41640.66520.51220.4121
30.81080.74010.36260.62440.45880.3538
40.81450.75750.36260.64320.46380.3620
50.82640.76890.39380.68810.50090.4051
60.81330.73480.37680.63030.47160.3669
70.82580.76730.38810.68840.49640.4008
80.81700.74770.37390.65020.47480.3737
90.81760.76110.41080.63600.49910.3939
Mean0.81920.75850.38500.65600.48490.3842
SD0.00520.01410.01830.02180.01710.0189
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8214 0.7588 0.3966 0.6604 0.4956 0.3952\n", - "1 0.8208 0.7634 0.3683 0.6736 0.4762 0.3791\n", - "2 0.8246 0.7859 0.4164 0.6652 0.5122 0.4121\n", - "3 0.8108 0.7401 0.3626 0.6244 0.4588 0.3538\n", - "4 0.8145 0.7575 0.3626 0.6432 0.4638 0.3620\n", - "5 0.8264 0.7689 0.3938 0.6881 0.5009 0.4051\n", - "6 0.8133 0.7348 0.3768 0.6303 0.4716 0.3669\n", - "7 0.8258 0.7673 0.3881 0.6884 0.4964 0.4008\n", - "8 0.8170 0.7477 0.3739 0.6502 0.4748 0.3737\n", - "9 0.8176 0.7611 0.4108 0.6360 0.4991 0.3939\n", - "Mean 0.8192 0.7585 0.3850 0.6560 0.4849 0.3842\n", - "SD 0.0052 0.0141 0.0183 0.0218 0.0171 0.0189" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "bagged_dt2 = ensemble_model(dt, n_estimators=50)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice how increasing the n_estimators parameter has improved the result. AUC in bagged_dt2 where `n_estimators = 50` improved to `0.7585` from `0.7349` in bagged_dt where n_estimators uses default value of `10`. You can also use `tune_model()` function to automatically tune `n_estimators` parameter of ensemble. See example below in which we are tuning Decision Tree along with `Bagging` parameters of Ensembled Decision Tree." - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.82330.77830.33710.71260.45770.3679
10.82270.79270.34840.69890.46500.3727
20.81950.81450.35410.67570.46470.3686
30.82140.76640.35410.68680.46730.3729
40.82640.77610.35690.71590.47640.3860
50.82520.78940.35130.71260.47060.3800
60.81450.75010.32010.66860.43300.3382
70.82890.78380.33430.75640.46370.3795
80.81830.75690.33140.68420.44660.3532
90.81440.77410.34840.65080.45390.3542
Mean0.82150.77820.34360.69630.45990.3673
SD0.00460.01760.01150.02830.01200.0139
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8233 0.7783 0.3371 0.7126 0.4577 0.3679\n", - "1 0.8227 0.7927 0.3484 0.6989 0.4650 0.3727\n", - "2 0.8195 0.8145 0.3541 0.6757 0.4647 0.3686\n", - "3 0.8214 0.7664 0.3541 0.6868 0.4673 0.3729\n", - "4 0.8264 0.7761 0.3569 0.7159 0.4764 0.3860\n", - "5 0.8252 0.7894 0.3513 0.7126 0.4706 0.3800\n", - "6 0.8145 0.7501 0.3201 0.6686 0.4330 0.3382\n", - "7 0.8289 0.7838 0.3343 0.7564 0.4637 0.3795\n", - "8 0.8183 0.7569 0.3314 0.6842 0.4466 0.3532\n", - "9 0.8144 0.7741 0.3484 0.6508 0.4539 0.3542\n", - "Mean 0.8215 0.7782 0.3436 0.6963 0.4599 0.3673\n", - "SD 0.0046 0.0176 0.0115 0.0283 0.0120 0.0139" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "tuned_bagged_dt = tune_model('dt', ensemble = True, method='Bagging')" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "BaggingClassifier(base_estimator=DecisionTreeClassifier(ccp_alpha=0.0,\n", - " class_weight=None,\n", - " criterion='gini',\n", - " max_depth=6,\n", - " max_features=None,\n", - " max_leaf_nodes=None,\n", - " min_impurity_decrease=0.0,\n", - " min_impurity_split=None,\n", - " min_samples_leaf=2,\n", - " min_samples_split=2,\n", - " min_weight_fraction_leaf=0.0,\n", - " presort='deprecated',\n", - " random_state=123,\n", - " splitter='best'),\n", - " bootstrap=False, bootstrap_features=True, max_features=1.0,\n", - " max_samples=1.0, n_estimators=80, n_jobs=None,\n", - " oob_score=False, random_state=123, verbose=0,\n", - " warm_start=False)\n" - ] - } - ], - "source": [ - "# check the parameters of tuned Decision Tree with bagging\n", - "print(tuned_bagged_dt)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that `tuned_bagged_dt` is a Decision Tree wrapped inside a `BaggingClassifier`. Our first bagging ensemble with default values stored in `bagged_dt` resulted in AUC of `0.7349` which was improved to `0.7585` when we increased the `n_estimators` parameter to `50`. When we have tuned Decision Tree with `ensemble = True` inside `tune_model()` function, our final AUC for Tuned Ensembled Decision Tree is `0.7782`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 10.3 Blending" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Blending is another common way to use ensembling technique in PyCaret. It uses predictions from multiple models to generate final prediction using voting / majority consensus from all the models passed in `estimator_list` parameter. If no list is passed, by default PyCaret uses all the models available in model library. `method` parameter can be used to define type of voting. When method is set to `hard`, it uses labels for majority rule voting. When method is set to `soft` it uses sum of predicted probabilities instead of label. Let's see an example below:" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.82520.00.36830.69890.48240.3891
10.82460.00.34560.71350.46560.3755
20.82330.00.37960.68020.48730.3907
30.81770.00.34840.66850.45810.3613
40.82640.00.34840.72350.47040.3814
50.82390.00.35690.70000.47280.3802
60.81830.00.33990.67800.45280.3580
70.82710.00.34840.72780.47130.3829
80.82140.00.34840.69100.46330.3698
90.81760.00.36540.65820.46990.3705
Mean0.82250.00.35500.69400.46940.3759
SD0.00340.00.01180.02200.00980.0105
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8252 0.0 0.3683 0.6989 0.4824 0.3891\n", - "1 0.8246 0.0 0.3456 0.7135 0.4656 0.3755\n", - "2 0.8233 0.0 0.3796 0.6802 0.4873 0.3907\n", - "3 0.8177 0.0 0.3484 0.6685 0.4581 0.3613\n", - "4 0.8264 0.0 0.3484 0.7235 0.4704 0.3814\n", - "5 0.8239 0.0 0.3569 0.7000 0.4728 0.3802\n", - "6 0.8183 0.0 0.3399 0.6780 0.4528 0.3580\n", - "7 0.8271 0.0 0.3484 0.7278 0.4713 0.3829\n", - "8 0.8214 0.0 0.3484 0.6910 0.4633 0.3698\n", - "9 0.8176 0.0 0.3654 0.6582 0.4699 0.3705\n", - "Mean 0.8225 0.0 0.3550 0.6940 0.4694 0.3759\n", - "SD 0.0034 0.0 0.0118 0.0220 0.0098 0.0105" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "blend_hard = blend_models()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now that we have created a Voting Classifier using `blend_models()` function. The model stored in variable `blend_hard` is just like any other model that you would create using `create_model()` or `tune_model()`. You can use this model for prediction on unseen data using `predict_model()` in the same way you would use any other model. Notice that since we didn't pass the list of specific models for voting, it by default uses all the models in model library. \n", - "\n", - "You may have noticed that `AUC` is zero for all folds. This is because `method` parameter is set to `hard` which only uses labels (1 or 0) for predictions and hence no AUC is calculated. To change this you can use `method` parameter inside `blend_models()`. See example below:" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.82270.77350.41080.65910.50610.4051
10.82210.78940.32290.71700.44530.3570
20.82270.80340.43630.64710.52120.4174
30.81830.76430.32290.69090.44020.3483
40.82330.76520.31440.73510.44050.3550
50.82580.78930.34560.72190.46740.3784
60.81770.76070.32290.68670.43930.3469
70.82580.79000.35410.71430.47350.3830
80.82080.76090.33140.70060.45000.3589
90.81130.76570.34280.63680.44570.3441
Mean0.82100.77630.35040.69100.46290.3694
SD0.00410.01460.03880.03180.02780.0243
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8227 0.7735 0.4108 0.6591 0.5061 0.4051\n", - "1 0.8221 0.7894 0.3229 0.7170 0.4453 0.3570\n", - "2 0.8227 0.8034 0.4363 0.6471 0.5212 0.4174\n", - "3 0.8183 0.7643 0.3229 0.6909 0.4402 0.3483\n", - "4 0.8233 0.7652 0.3144 0.7351 0.4405 0.3550\n", - "5 0.8258 0.7893 0.3456 0.7219 0.4674 0.3784\n", - "6 0.8177 0.7607 0.3229 0.6867 0.4393 0.3469\n", - "7 0.8258 0.7900 0.3541 0.7143 0.4735 0.3830\n", - "8 0.8208 0.7609 0.3314 0.7006 0.4500 0.3589\n", - "9 0.8113 0.7657 0.3428 0.6368 0.4457 0.3441\n", - "Mean 0.8210 0.7763 0.3504 0.6910 0.4629 0.3694\n", - "SD 0.0041 0.0146 0.0388 0.0318 0.0278 0.0243" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "blend_soft = blend_models(method = 'soft')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that results are not much different. Both the examples above uses all the models in model library to create Voting Classifier. Now we will create specific models using `create_model()` and use them inside `blend_models()` to create Voting Classifier of hand picked models passed in `estimator_list` parameter. See example below:" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": {}, - "outputs": [], - "source": [ - "\"\"\"\n", - "we will create 4 specific models to be passed into blend_models().\n", - "Note that verbose is set to False to avoid printing score grid of individual models.\n", - "\"\"\"\n", - "\n", - "gbc = create_model('gbc', verbose = False)\n", - "dt = create_model('dt', verbose = False)\n", - "lightgbm = create_model('lightgbm', verbose = False)\n", - "xgboost = create_model('xgboost', verbose = False)" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.81270.76640.38530.62390.47640.3700
10.81450.77930.37680.63640.47330.3696
20.81450.80500.41080.62230.49490.3871
30.81270.75650.37110.62980.46700.3625
40.81890.76150.35980.66840.46780.3703
50.81700.78380.38530.64450.48230.3796
60.81390.75370.39380.62610.48350.3771
70.82890.78080.38530.70830.49910.4066
80.81580.74830.37680.64250.47500.3724
90.80940.76360.38530.60990.47220.3631
Mean0.81580.76990.38300.64120.47910.3758
SD0.00500.01630.01290.02700.01030.0124
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8127 0.7664 0.3853 0.6239 0.4764 0.3700\n", - "1 0.8145 0.7793 0.3768 0.6364 0.4733 0.3696\n", - "2 0.8145 0.8050 0.4108 0.6223 0.4949 0.3871\n", - "3 0.8127 0.7565 0.3711 0.6298 0.4670 0.3625\n", - "4 0.8189 0.7615 0.3598 0.6684 0.4678 0.3703\n", - "5 0.8170 0.7838 0.3853 0.6445 0.4823 0.3796\n", - "6 0.8139 0.7537 0.3938 0.6261 0.4835 0.3771\n", - "7 0.8289 0.7808 0.3853 0.7083 0.4991 0.4066\n", - "8 0.8158 0.7483 0.3768 0.6425 0.4750 0.3724\n", - "9 0.8094 0.7636 0.3853 0.6099 0.4722 0.3631\n", - "Mean 0.8158 0.7699 0.3830 0.6412 0.4791 0.3758\n", - "SD 0.0050 0.0163 0.0129 0.0270 0.0103 0.0124" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "blend_specific_soft = blend_models(estimator_list = [gbc,dt,lightgbm,xgboost], method = 'soft')" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.83080.00.36830.73450.49060.4023
10.82520.00.34840.71510.46860.3785
20.82140.00.36260.68090.47320.3775
30.82210.00.35410.69060.46820.3744
40.82710.00.35130.72510.47330.3844
50.82520.00.36260.70330.47850.3861
60.82210.00.35130.69270.46620.3728
70.83150.00.35410.75300.48170.3963
80.81830.00.32860.68640.44440.3516
90.81690.00.36830.65330.47100.3706
Mean0.82400.00.35500.70350.47160.3794
SD0.00460.00.01110.02760.01140.0134
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8308 0.0 0.3683 0.7345 0.4906 0.4023\n", - "1 0.8252 0.0 0.3484 0.7151 0.4686 0.3785\n", - "2 0.8214 0.0 0.3626 0.6809 0.4732 0.3775\n", - "3 0.8221 0.0 0.3541 0.6906 0.4682 0.3744\n", - "4 0.8271 0.0 0.3513 0.7251 0.4733 0.3844\n", - "5 0.8252 0.0 0.3626 0.7033 0.4785 0.3861\n", - "6 0.8221 0.0 0.3513 0.6927 0.4662 0.3728\n", - "7 0.8315 0.0 0.3541 0.7530 0.4817 0.3963\n", - "8 0.8183 0.0 0.3286 0.6864 0.4444 0.3516\n", - "9 0.8169 0.0 0.3683 0.6533 0.4710 0.3706\n", - "Mean 0.8240 0.0 0.3550 0.7035 0.4716 0.3794\n", - "SD 0.0046 0.0 0.0111 0.0276 0.0114 0.0134" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "blend_specific_hard = blend_models(estimator_list = [gbc,dt,lightgbm,xgboost], method = 'hard')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that Accuracy using `hard` method has improved to `0.8240` in blend_specific_hard from `0.8158` in blend_specific_soft. Which `method` and `models` to use in blending depends on the statistical properties of the dataset and experimenting with different models and method is the best way to find out which blender will work best on the specific problem." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 10.4 Stacking" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Stacking is another popular yet less commonly implemented technique of ensembling due to practical difficulties. Stacking is an ensemble learning technique that combines multiple models via a meta-model. The another way to think about stacking is that multiple models are trained to predict the outcome and a meta-model is created that uses the prediction from all those models as input along with original features. The implementation of `stack_models()` is based on Wolpert, D. H. (1992b). Stacked generalization __[(Read More)](https://www.sciencedirect.com/science/article/abs/pii/S0893608005800231)__. \n", - "\n", - "Let's see an example below using the models we have created in section 10.3 above." - ] - }, - { - "cell_type": "code", - "execution_count": 25, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.82520.78350.37680.69270.48810.3936
10.82210.79440.35980.68650.47210.3774
20.82390.82240.39090.67650.49550.3980
30.82270.78490.37680.67860.48450.3878
40.82460.78030.37110.69310.48340.3892
50.81770.79730.36260.65980.46800.3690
60.81890.75250.36260.66670.46970.3718
70.83020.78920.37680.72280.49530.4052
80.81950.75270.35130.67760.46270.3671
90.81130.77800.37390.62260.46730.3612
Mean0.82160.78350.37030.67770.47870.3820
SD0.00490.01950.01080.02460.01150.0140
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8252 0.7835 0.3768 0.6927 0.4881 0.3936\n", - "1 0.8221 0.7944 0.3598 0.6865 0.4721 0.3774\n", - "2 0.8239 0.8224 0.3909 0.6765 0.4955 0.3980\n", - "3 0.8227 0.7849 0.3768 0.6786 0.4845 0.3878\n", - "4 0.8246 0.7803 0.3711 0.6931 0.4834 0.3892\n", - "5 0.8177 0.7973 0.3626 0.6598 0.4680 0.3690\n", - "6 0.8189 0.7525 0.3626 0.6667 0.4697 0.3718\n", - "7 0.8302 0.7892 0.3768 0.7228 0.4953 0.4052\n", - "8 0.8195 0.7527 0.3513 0.6776 0.4627 0.3671\n", - "9 0.8113 0.7780 0.3739 0.6226 0.4673 0.3612\n", - "Mean 0.8216 0.7835 0.3703 0.6777 0.4787 0.3820\n", - "SD 0.0049 0.0195 0.0108 0.0246 0.0115 0.0140" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "stack_soft = stack_models([gbc,dt,lightgbm,xgboost])" - ] - }, - { - "cell_type": "code", - "execution_count": 26, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.82460.77930.36540.69730.47960.3862
10.82140.78920.34560.69320.46120.3683
20.82640.81690.38810.69190.49730.4023
30.81890.78550.35980.66840.46780.3703
40.82520.77570.36260.70330.47850.3861
50.81830.79800.35410.66840.46300.3658
60.82080.74630.34560.68930.46040.3668
70.83020.78670.36830.73030.48960.4008
80.82460.75210.35130.70860.46970.3786
90.81190.77190.35980.63180.45850.3549
Mean0.82220.78010.36010.68830.47260.3780
SD0.00490.01960.01190.02550.01260.0148
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8246 0.7793 0.3654 0.6973 0.4796 0.3862\n", - "1 0.8214 0.7892 0.3456 0.6932 0.4612 0.3683\n", - "2 0.8264 0.8169 0.3881 0.6919 0.4973 0.4023\n", - "3 0.8189 0.7855 0.3598 0.6684 0.4678 0.3703\n", - "4 0.8252 0.7757 0.3626 0.7033 0.4785 0.3861\n", - "5 0.8183 0.7980 0.3541 0.6684 0.4630 0.3658\n", - "6 0.8208 0.7463 0.3456 0.6893 0.4604 0.3668\n", - "7 0.8302 0.7867 0.3683 0.7303 0.4896 0.4008\n", - "8 0.8246 0.7521 0.3513 0.7086 0.4697 0.3786\n", - "9 0.8119 0.7719 0.3598 0.6318 0.4585 0.3549\n", - "Mean 0.8222 0.7801 0.3601 0.6883 0.4726 0.3780\n", - "SD 0.0049 0.0196 0.0119 0.0255 0.0126 0.0148" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "stack_hard = stack_models([gbc,dt,lightgbm,xgboost], method='hard')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Similar to blending `stack_models()` also support soft and hard method that can be defined under `method` parameter. Soft method uses sum of predicted probabilities and hard method uses label (1 or 0). In both the examples above the meta model (final model to generate predictions) is Logistic Regression (by default). Meta model can be changed using `meta_model` parameter. See an example below:" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.83020.78280.38240.71810.49910.4081
10.82580.79090.35130.71680.47150.3815
20.81770.82100.36540.65820.46990.3705
30.81950.77110.35980.67200.46860.3717
40.82460.77770.35980.70170.47570.3832
50.81950.78460.36540.66840.47250.3748
60.81830.76720.36260.66320.46890.3704
70.83770.79380.38810.76110.51410.4287
80.81700.76250.35690.65970.46320.3645
90.81570.78600.37680.64250.47500.3723
Mean0.82260.78380.36690.68620.47780.3826
SD0.00660.01570.01120.03500.01510.0192
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8302 0.7828 0.3824 0.7181 0.4991 0.4081\n", - "1 0.8258 0.7909 0.3513 0.7168 0.4715 0.3815\n", - "2 0.8177 0.8210 0.3654 0.6582 0.4699 0.3705\n", - "3 0.8195 0.7711 0.3598 0.6720 0.4686 0.3717\n", - "4 0.8246 0.7777 0.3598 0.7017 0.4757 0.3832\n", - "5 0.8195 0.7846 0.3654 0.6684 0.4725 0.3748\n", - "6 0.8183 0.7672 0.3626 0.6632 0.4689 0.3704\n", - "7 0.8377 0.7938 0.3881 0.7611 0.5141 0.4287\n", - "8 0.8170 0.7625 0.3569 0.6597 0.4632 0.3645\n", - "9 0.8157 0.7860 0.3768 0.6425 0.4750 0.3723\n", - "Mean 0.8226 0.7838 0.3669 0.6862 0.4778 0.3826\n", - "SD 0.0066 0.0157 0.0112 0.0350 0.0151 0.0192" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "stack_soft2 = stack_models([gbc,dt,lightgbm], meta_model=xgboost)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Which `method` and `models` to use in stacking depends on the statistical properties of the dataset and experimenting with different models and method is the best way to find out which configuration will work best. However, as a general rule of thumb, models with strong yet diverse performance are tend to improve performance when used in stacking. One way to measure diversity is the correlation of prediction between models. You can analyze that using `plot` parameter. See an example below: " - ] - }, - { - "cell_type": "code", - "execution_count": 28, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.82520.78350.37680.69270.48810.3936
10.82210.79440.35980.68650.47210.3774
20.82390.82240.39090.67650.49550.3980
30.82270.78490.37680.67860.48450.3878
40.82460.78030.37110.69310.48340.3892
50.81770.79730.36260.65980.46800.3690
60.81890.75250.36260.66670.46970.3718
70.83020.78920.37680.72280.49530.4052
80.81950.75270.35130.67760.46270.3671
90.81130.77800.37390.62260.46730.3612
Mean0.82160.78350.37030.67770.47870.3820
SD0.00490.01950.01080.02460.01150.0140
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8252 0.7835 0.3768 0.6927 0.4881 0.3936\n", - "1 0.8221 0.7944 0.3598 0.6865 0.4721 0.3774\n", - "2 0.8239 0.8224 0.3909 0.6765 0.4955 0.3980\n", - "3 0.8227 0.7849 0.3768 0.6786 0.4845 0.3878\n", - "4 0.8246 0.7803 0.3711 0.6931 0.4834 0.3892\n", - "5 0.8177 0.7973 0.3626 0.6598 0.4680 0.3690\n", - "6 0.8189 0.7525 0.3626 0.6667 0.4697 0.3718\n", - "7 0.8302 0.7892 0.3768 0.7228 0.4953 0.4052\n", - "8 0.8195 0.7527 0.3513 0.6776 0.4627 0.3671\n", - "9 0.8113 0.7780 0.3739 0.6226 0.4673 0.3612\n", - "Mean 0.8216 0.7835 0.3703 0.6777 0.4787 0.3820\n", - "SD 0.0049 0.0195 0.0108 0.0246 0.0115 0.0140" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnIAAAGeCAYAAAAQQOzxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzde7xd853/8dfeJ5GEaGlR16REfKi7Vt0Jdamp/hiqJcat1LUzRadoSxspRpW01UHHNcpU0ClV1zElRVFFhWp9ImhJStW1EpHL2fv3x1onjkgi5LLP2uf1fDzOI2fdP2tlP9Z+n+/3u/auNZtNJEmSVD31VhcgSZKk98cgJ0mSVFEGOUmSpIoyyEmSJFWUQU6SJKmiDHKSJEkV1afVBajt+Hk2ktR71FpdwPzobIxdaO9NHfVhPeqcDXJa6DobY1tdQq/TUR8GwIwRB7a0jt6q74jLAOicemuLK+l9OgbsAnjfaYWu+45ay65VSZKkijLISZIkVZRBTpIkqaIcIydJktpbo7Hw9tXDmsB6WDmSJEmaXwY5SZKkirJrVZIktbeF2bXaw9giJ0mSVFEGOUmSpIoyyEmSJFWUY+QkSVJ7c4ycJEmSehqDnCRJUkXZtSpJktpbs9nqChYZW+QkSZIqyiAnSZJUUQY5SZKkijLISZIkVZQPO0iSpPbm58hJkiSppzHISZIkVZRdq5Ikqb3ZtSpJkqSexiAnSZJUUQY5SZKkinKMnCRJam+OkZMkSVJPY5CTJEmqKLtWJUlSe7NrVZIkST2NQU6SJKmiDHKSJEkV5Rg5SZLU3hwjJ0mSpJ7GICdJklRRdq1KkqS2VmvatSpJkqQexiAnSZJUUQY5SZKkijLISZIkVZQPO0iSpPbWxp8jZ5CTSuPGPc2os3/OZT/5aqtLaT+1GvXPHEDtI4Ogcwad118CL7/wjnU6hh9HMx+i8cAd1Lf+DLU11y8W9V8SBn6QmWd9pQXFV1uj0WDk6deQ4yexRN8+jPz2vgwetPys5Rddehs33vwQAwf255CDPsWwbdfjjanTGHna1Uyc9BIzZnTyzRM+xwbrD27hWVRTo9Fg5MgrycefZYkl+jLyO/szePAKs5ZfdOEt3HjT7xi4VH8OOWQXhm2/ARMnvsjXT7yUJrDyyh/ilFP2Z8CAJVp3Eurx5ivIRcQawJnAqsAbwFTg+Mx87L0cLCI+DeyTmQdFxM8zc8/3uP0gYMPM/GVEjAY2AV4G+gF/AI7KzBnvZZ9zOc76wLKZeWdEjAEOyMzp73Ef61JcsyWBgcBNwAhgO+CIzNxnAWtcEfhWZh4VEXsApwIXAMPe63Wdw76/DXwGmAkck5n3L8j+quDii27l+uvvY8CAfq0upS3V1t6EWp++dF78HWqrDqFj533pHPPDt61T32EvGLDUrOnG3TfC3TcC0DH8WBq3Xb1Ya24Xv7rjUaZPm8GVPzmOcY88zZmjruXcHxwGwPgn/sqNNz/ImMuLP16GH/h9Ntt0LS4Z/SuGDlmJM07dnxw/iRw/ySD3Pvzq/x4urv2YExn38FOceebPOPfcowAYP34SN974O8ZcdSIAw/f9LpttvjZnfe9/+MI+27Hbbp/kZ9fczWWjb+OIIz/TytPQAoiIOnAesCEwDTg0Myd0W34CsC/wD+DMzLwhIpYDfgoMAP4KHJyZb8ztGO86Ri4ilgSuB87OzM0zcwfgFODc931mwPsMGzsAW3WbPj4zh2XmFsBSwO4LUlM3ewEfA8jMfd5HiFsGGEMRgrYHNgfWBw5fSPWRmc9n5lHl5G7A1zPznIUQ4jahCJubAfuwgP/PVbHaoOX54TlHtLqMtlUbtBaNCY8C0Jz4JLWVV3/78o99AppNmhMeeee263wcpk6h+eQfFkut7eah3z/J1lutA8CGG6zOY489O2vZk089z6YfH0q/fn3p168vgwctz/gnJvGbex+nb98OvnTkeZx/wa1steU6rSq/0h56aAJbb70uABtutAaP/eEvs5Y9+eRzbPrJtd669oNXYHxOZMKTz7HNNsU2G28yhIcemjDHfes9ajQX3s97swfQv8wpJwJndy0oG42GU2SEnYGRZeb6FvDTzNwG+D3vkh3mp0Xus8DtmXlv14zMvD8iti9bxT5c/nwW+C6wWjl9c2aeHBHrAJcAU8qfV8oTeD4zVyxP5BygBrwEfBHYGDgBmA6sDlwFnFFehCUj4p7uBUZEB7A08Ew5/VWKEDITuDMzTyjD1RXAB8rzPikzb4+I0ygCYh24ErgGOAiYHhEPAVcDawM/pkjTHwVWAg7KzIci4hDgyxQtg9PLWpvlNXuivF6dEXFAuXzLbnV/GdgT6Au8Vv7+UWA0MKOs/4Bu+62X6x4BvE4RFk+nCHKfjIgXgWvf5bp+t9zfBZl5Oe+0NfC/mdkEnomIPhGxfGb+fQ7rto2dd96ESZNebHUZbavWbwDNN6e+NaPZgHq9GLeywirU19+Czqv/k/p27/xbrL71bnT+z/mLsdr2MnnKmwwcOGDWdL2jzsyZnfTp08FaQ1fmwktuY8qUN5kxo5OHxz3N3nttySuvTOG1f0zlwvOP4he/vJ/vjbqOM07dv4VnUU2TJ7/JwKW7X/vaW9d+rVW48MJbims/fSYPP/wUe39+GuusvSp33PEIe+yxBXfcPo43pr6ndgT1PFsDtwBk5n0R8Yluy9YBxmbmmwAR8QSwQbnN6eU6N5e/f39uB5ifp1ZXB7o3A/4iIsYCj1N0td6emVtSBKn7MnOXsogjy02+Q9EFuCPwtgBWuhA4OjOHUXQ/Hl/OH0zRMrYFRctbJ0WY+2lmXl+uc2ZZy5+AjwBZBpjPUwSmLYGhEbEbcBJwW2ZuC+wNXFw2eR5AkYi3BaZm5iSKIDVqDl2KfynP70fAYWXz5wkUrYQ7U7QKAqwMPNV9w8yc3L1lrzz2h4Edy9TdF9gU2Al4ENgROA1YFvgkRdDbFfg3ijDatd/rKV4kx3cP2/O4rv0zc5u5hDjKfb/Wbfp14INzWbfrXEZERDMi3vOfKuodmtOmQr/+b82o1WYNPq5vuBUsvSwdB55AfaOtqW/x6Vlj41h+ZXjzjXeOp9N8G7hUf6ZMeXPWdLPRoE+fDgCGrLEi++2zLYcf/WPOPPtaNlh/MMsuM5BlllmSHYatB8Cw7dbjD398piW1V93AgbNf++Zb137ISuw3fBiHH3YOZ575MzbYYHWWXXYgxx+/N3fcPo7DvvRDavUayy4zsFXlay66v+eVPyPmsfrs76mdEdHViPYosG1ELB0RH6bILEvNts27vgfPT4vcs8CsBJmZu5cnch8wEchy0cvAphGxPUVfb9dgo3WBrkD0G4oE2t06wHkRAUWYGd91gpk5E5gZEVOZs+Mz85aynpEUTZa3UgTKGeX8u8oa1gH+uzyHSRHxD2B5ipa7/wBWpEi+8/L7btdkK2BN4I9dfdfdWgr/QjF+b5aIWJ2itZKyhkZETAeujIjJFKG4L3AxRTi8heI/8htlXUOBX1C01J36LnXC3K9rznWLwj8oQnmXpYFX57VBZo6gGP8HRWuk9DbNZ56gHhvR+dj91FYdQvNvE2ct6z72rT5sD5j8Gs2yG7a+xrpz7G7V/Nt4ozUY++s/sOsumzDukacZOnTlWctefvl1Xnl1MleMPobXX5/KoUeex9A1V2KTjYdw591/ZN2PDeKBByew5hortfAMqmvjTdZk7B2PsOuun2Dcw08xdK1VZi17+eXXeeWVyVzx38cX1/7QHzB06Crc8MvfctTRuxGxKpdeehtb2q3d48z2nvduZn9PrZfZhsz8U0T8J8V7/ATgt8CL3baZyny8B89Pi9wvgB0jYvOuGRGxJkXwGAx0PdN7EPBqZu5HEaiWjIgaRcvdFuU6m85h/0nxMMEwilajG8v5cwoEjXnU/CywRHm8zcouwRpFS9t4ila7bcr6V6Fo6XqVonVuX4ru1YMiouuc5nSc2WuaAKwdEQPKFrZPlvNvAD4dEUPK4/UFRgHrdW0YERsAe2TmF4B/LY9Xoxjnd1dmfoqim/cEYBjwXGbuTBHiuppc52Vu1/XdnsH+DbBLRNTLh0vqmWmfoxZI8/EHac6cQcchJ1HfZTidt/6U+ha7UIuN573hh1ek+Upb9+ovcjvusAFL9OvL8ANGccZZ13Liv+/J6Mtv5/axj7LssgN5duJLfH74WRzx5R/ztWN3p6OjzmGH7MSfHp/IvgeMYvTld/C1YxfW8OPeZccdNyqu/b7f5YwzrubEEz/P6NG3cfvt48pr/yKf3/t0jjj8R3zt3/eio6POR1dfkZO+eRn7DT+TPz/9PJ/be5tWn0Z7aDQW3s978xvgnwDKHPVo14KIWB5YLjO3Br5C0djzh+7bUPTE3TWvA7xri1xmTo6IzwJnRMRK5TYzgcMoujC7/AoYExHbUIyFe4Kii/Eo4KqI+Brwd+BN3u5I4CflODeAQ8rt5uRR4Jvl2DUoulZPBDqBDuCLmflURFxNcSHqwN3AdcCvgUsi4nMUT4IclpnTIuJl4GGKsXv/SzHO7kHgexHxp3e5Ni9GxHcpLvLL5X5nZOY/IuJA4MIy4C0N/BI4n+JBAihC4JSIeIBi7N1z5XnfB1wRETMpQtexFC18V0XEMeW5jpxXXaX3cl27n9ODZSvmvRTX7+j5OFZbWGWV5WY9QaaFrNmkccNlb5vVePG5d6zWGHvd26dvmtsIAM2ver3OiJO+8LZ5a6z+kVm/n3LyOx+gX+aDS3HOqEMXeW3trl6vM2LEfm+bt8YaK876/ZRT/uUd22y44epc87NvLvLatNhcC+xU9tjVgIMj4jiKDPBLYI2I+B3F2PWvlWPqTwUui4gvUbTQDZ/XAWrNpj1h71fZz31CZp5WTt9J8RDFna2trKWanY2xra6h1+moDwNgxogDW1pHb9V3RBFSO6fe2uJKep+OAbsA4H1n8SvvO7XWVjF/Gn8ds9DCTn3lfXrUOfuBwAsgM2dGxFJlC+F0iv7teTaB9hQR8S2K7uTZHZyZTy/ueiRJWmT8ZgfNTWZ+g+KBhErJzJHMXxetJEnqoebnYQdJkiT1QAY5SZKkijLISZIkVZRj5CRJUntrtu/DDrbISZIkVZRBTpIkqaLsWpUkSe2tjT9HzhY5SZKkijLISZIkVZRBTpIkqaIcIydJktpbo9nqChYZW+QkSZIqyiAnSZJUUXatSpKk9ubHj0iSJKmnMchJkiRVlEFOkiSpogxykiRJFeXDDpIkqb35sIMkSZJ6GoOcJElSRdm1KkmS2lrNrlVJkiT1NAY5SZKkijLISZIkVZRj5CRJUntrNltdwSJji5wkSVJFGeQkSZIqyq5VSZLU3vz4EUmSJPU0BjlJkqSKMshJkiRVlGPkJElSe3OMnCRJknoag5wkSVJF2bUqSZLaW8NvdpAkSVIPY5CTJEmqKIOcJElSRRnkJEmSKqrWbLbvAEC1hC8oSeo9aq0uYH40H/7RQntvqm30rz3qnG2RkyRJqig/fkQL3YwRB7a6hF6n74jLAOhsjG1tIb1UR30YADNPPaiVZfRKfU4aDXjfaYWu+45ayyAnSZLam1/RJUmSpJ7GICdJklRRBjlJkqSKcoycJElqb238UWu2yEmSJFWUQU6SJKmi7FqVJEntzY8fkSRJUk9jkJMkSaoog5wkSVJFGeQkSZIqyocdJElSe2v4OXKSJEnqYQxykiRJFWXXqiRJam9+jpwkSZJ6GoOcJElSRRnkJEmSKsoxcpIkqb21aIxcRNSB84ANgWnAoZk5oVy2EfCDbqtvDuwB3A+MB/5Qzr82M384t2MY5CRJkhaNPYD+mblFRGwOnA3sDpCZDwPDACJib+CvmXlLROwIXJmZ/zo/BzDISZIkLRpbA7cAZOZ9EfGJ2VeIiKWAU4Bty1kfBzaJiF8DLwD/lpnPze0AjpGTJEntrdFcaD8RMSIimt1+RszjyB8AXus23RkRszeiHQJck5kvltOPA9/OzO2A64AfzevUbJGTJEmaT5k5Ahgxn6v/A1i623Q9M2fOts5+wOe6Td8OvFH+fi0wcl4HsEVOkiRp0fgN8E8A5Ri5R7svjIgPAv0y89lusy8C9ip//xTw4LwOYIucJEnSonEtsFNE3APUgIMj4jhgQmZeD6wF/Hm2bU4ELomIo4ApwKHzOoBBTpIkaRHIzAZwxGyzH++2/HcUT7Z23+ZpYPv5PYZBTpIktbem37UqSZKkHsYgJ0mSVFF2rUqSpPbWaLa6gkXGFjlJkqSKMshJkiRVlEFOkiSpohwjJ0mS2lvDjx+RJElSD2OQkyRJqii7ViVJUnvz40ckSZLU0xjkJEmSKsogJ0mSVFGOkVPvUatR/8wB1D4yCDpn0Hn9JfDyC+9Yp2P4cTTzIRoP3EF9689QW3P9YlH/JWHgB5l51ldaUHx7GzfuaUad/XMu+8lXW11KG6pR3/UAah9ZDTpn0nnDJfDKC+9cZ59jaY7/Pc2H7oB+A6j/85HUluhHs3MmjesugCmvtaT6Snsf9xxqNeq7DKe28urQpw+NsdfSHD+uNfWrEgxy6jVqa29CrU9fOi/+DrVVh9Cx8750jvnh29ap77AXDFhq1nTj7hvh7hsB6Bh+LI3brl6sNfcGF190K9dffx8DBvRrdSltqRabQJ++dI4+FVYZQn3HfWhcc87b1qlvvye1AUvRNRy8tsHW8MJEOm+/mtrG21HfYlca/zdm8Rdfce/nnlPbcCvo6KDzklNh6WWpr7sp7TtMfzFq48+RW6xBLiKGAVcDfwRqQF/gB5k53++OEfEDYFRmPjOHZZ8GBmXmBe9hf+sDPyonNwfuBxrA9zLzxvndzxz2uy5wJrAkMBC4CRgBbAcckZn7vN99l/tfEfhWZh4VEXsApwIXAMMyc88F2Xe5/zWB6zJzvQXdV09RG7QWjQmPAtCc+GTxF2/35R/7BDSbNCc88s5t1/k4TJ1C88k/LJZae5PVBi3PD885ghNPuLTVpbSl2mpDaT5ZvO6Z9CS1lWZ73a9dvu671gH4+0RYbqXi9yUGQGfnYqq2vbyfe059yPo0X3iWjuHHQq1G501XLNaaVT2taJG7vSvERMRA4NcRMT4zH56fjTPzmHksu+W9FpOZjwLDynr+DOycmW++1/10FxHLAGOAPTPziYjoAK4BDgceX5B9d8nM54GjysndgK9n5i+Bc+a+1fyJiP2BrwDLLei+epJavwE035z61oxmA+r14i+1FVahvv4WdF79n9S32/0d29a33o3O/zl/MVbbe+y88yZMmvRiq8toX/0GwLQ33ppuNqBWL/5dfhVq621O42fnUt/2rdd9843J1NdYj47DT4MBA+m87PQWFF597+ues+RAah9akc6ffp/a4KBjj0PpvNTrv8Da+ONHWtq1mpmTI+K/gM9FxBeAbSkewBiVmddExGbADyla7yYB+wE3A0cAHwbOBmYAr5TL9gLWzswTI+KrwD7ATODOzDwhIkYAqwMrAIOBYzPz1rnVFxFjgb8DywKfAc4DhpY1npSZYyNiO+A0oBN4kiKs7U4RWJ8oz7MzIg4ApgNbdtv/l4E9KVomXyt//ygwujyvmUDXdleVx+1bnv/rFGHxdIog98mIeBG4NjNXLFsazymv3UvAF4GNge+W+7sgMy+fy6m/QtFy+OTcrs1s12kE8O3yXOdnk5ZoTpsK/fq/NaNWm9XcXt9wK1h6WToOPIHaMstBZyfNV1+kOeFRWH5lePONd45tkapg2lRYYrbXfbN83a+/FbWll6Vj/xPgg8tR65xJ49UXqW2yHY17b6L50FhYYVU6PvdlOi88uTX1V9j7uecwdTKN8UW7RvMvSe3DH2lF6aqQnjBG7m/AV4HfZ+ZWEdEfuC8ibqPoKtwnM/8UEUcB63Tbbg/g58BZwP+jCFvArO7Sz1OEppnA/0TEbuXiaZm5a0TsVB53rkGu9NPMvDYijgRezMxDIuLDwJ0RsR5wIbB1Zr4QEd8BDqJoyXqq+04yc3JZW1eNdYowumNmNiLiVmBTYCPgQeA4YJvyvAZTBL3hwMeAD1AEOTLz+ojYExiTmfd27b+s64uZ+ceIOAQ4HrgN6J+Zm83rhDPzhu61vpvMHEHRbQz03OEczWeeoB4b0fnY/dRWHULzbxNnLes+9q0+bA+Y/FoR4oD6GuvOsbtVqoLmxAnUhm5E80+/g1WG0Hyh2+v+9m6v+233oDn5NZpPPUrtY5tCV0vSlNeLVj29Z+/nntP80EeoD92Azj89AB9ZjeZrL7egclVJTwhyg4H/BvYvW8CgaHUaDHwkM/8EkJnnwdvCxenAN4FfUbTW/bbbPtcG7svMGeU2dwHrlst+X/77LNDtT6W56mpiWh/YpmwlhOLarQCsBFxd1jUA+F/gT8Am3XcSEasDq83aaRHepgNXRsRkYNXyvC8GTgBuoQhv36BohRwK/IKipe7U+ah7HeC8sq6+wPjZzqfXaT7+IM0h69JxyElAjc5fXER9i11ovvwCzfz93Df88Io0n3pssdUpLUzNxx+ktvq6dBz4zWLM1S8vprbZLvDy32g+MecRLY2xP6e+28HUP7ED1Dto3Oj4xffj/dxzGg+OpWO3A+k49ORimxtGL86SVUEtDXIRsTTwJeAi4I7MPKxsqTqZokXrrxExtBxndgJvhREoulJHZ+a/R8TXgcOAv5TLHge+GhF9KLo8twV+AmzIe28x6nrU5XFgYmaeHhEDKELk34GJwO6Z+VpE/D9gMvAA8I2IOD8zn4yIvsAoihaxP5bnvgGwR2ZuFhFLUrTC1Si6Ze/KzFMiYl+KUHc58Fxm7hwRW1CE2IPfpe4EDsjMZyJiK4rA2f18ep9mk8YNl71tVuPF596xWmPsdW+fvmluPdBaWFZZZTnGXHViq8toU00aN7/9dd98aQ6v+zu7ve4nv0pjzPcXdWHt7/3cczpn0vmLixd1ZWojrQhyO5Qtb53l8b8NXAucXbacDaQY5/V6RBwOXBIRDeA54AcUg/ABfgdcVrZmTacIcttB8QBDRFwN/IZiXNndwHUUQe79+i/gwoj4NUXX5nllq9pXgBvLAPoPivD0j4g4sFy/DiwN/BI4v6tGYAIwJSIeAKaV57cycB9wRUTMpAhdx1IE1Ksi4pjyuo2cj3qPBH5SPmgBcEi5f0mSepdm+7Zh1JrNHjukSdXUnDHiwFbX0Ov0HVH81d/ZGNvaQnqpjvowAGaeelAry+iV+pw0GgDvO4tfed+ptbqO+dG8+eSFFnZqu36nR51zTxgjpxaIiG8BO8xh0cGZ+fTirkeSJL13BrleKjNHMn9dtJIkVVsbf45cvdUFSJIk6f0xyEmSJFWUQU6SJKmiHCMnSZLam2PkJEmS1NMY5CRJkirKrlVJktTeGu37zQ62yEmSJFWUQU6SJKmiDHKSJEkVZZCTJEmqKB92kCRJ7c3PkZMkSVJPY5CTJEmqKLtWJUlSe7NrVZIkST2NQU6SJKmiDHKSJEkV5Rg5SZLU3vyuVUmSJPU0BjlJkqSKsmtVkiS1t6YfPyJJkqQexiAnSZJUUQY5SZKkijLISZIkVZQPO0iSpPbmd61KkiSppzHISZIkVZRdq5Ikqb3ZtSpJkqSexiAnSZJUUQY5SZKkinKMnCRJam+NRqsrWGRskZMkSaoog5wkSVJF2bUqSZLaWxt//IhBTpIkaRGIiDpwHrAhMA04NDMndFu+K/DtcvIh4GigP3AFsALwOnBgZv59bsewa1WSJGnR2APon5lbACcCZ3ctiIilge8Bu2Xm5sCfgeWAI4FHM3Mb4CfASfM6gC1yWuj6jris1SX0Wh31Ya0uoVfrc9LoVpfQa3nfUQ+1NXALQGbeFxGf6LZsS+BR4OyIWAO4KDP/HhFbA2eW69wMnDyvAxjkJEmS5lNEjOCt7lCAUzJzxFxW/wDwWrfpzojok5kzKVrftgc2AiYDd0XEvbNt8zrwwXnVY5DTQtc59dZWl9DrdAzYBYCZpx7U2kJ6qa6WuM7G2JbW0Rt1tUJ731n8uu47lbAQH3YoQ9uI+Vz9H8DS3abrZYgDeAn4XWY+DxARd1KEuu7bLA28Oq8DOEZOkiRp0fgN8E8AEbE5RVdqlweB9SJiuYjoA2wO/LH7NsCuwF3zOoAtcpIkSYvGtcBOEXEPUAMOjojjgAmZeX1EfB3oak6+OjP/EBFPAZdFxN3AdGD4vA5gkJMkSe2tRZ8jl5kN4IjZZj/ebfkYYMxs27wB7D2/x7BrVZIkqaJskZMkSW2tuRBb5GoLbU8Lhy1ykiRJFWWQkyRJqiiDnCRJUkUZ5CRJkirKhx0kSVJ7a7bm40cWB1vkJEmSKsogJ0mSVFF2rUqSpPbWom92WBxskZMkSaoog5wkSVJFGeQkSZIqyjFykiSpvTlGTpIkST2NQU6SJKmi7FqVJEntza5VSZIk9TQGOUmSpIoyyEmSJFWUQU6SJKmifNhBkiS1Nx92kCRJUk9jkJMkSaoou1YlSVJba9q1KkmSpJ7GICdJklRRBjlJkqSKcoycJElqb46RkyRJUk9jkJMkSaoou1YlSVJ7s2tVkiRJPY1BTpIkqaIMcpIkSRVlkJMkSaooH3aQJEntzYcdJEmS1NMY5CRJkirKrlX1Go1Gg5GnX0OOn8QSffsw8tv7MnjQ8rOWX3Tpbdx480MMHNifQw76FMO2XY83pk5j5GlXM3HSS8yY0ck3T/gcG6w/uIVnUVU16rseQO0jq0HnTDpvuAReeeGd6+xzLM3xv6f50B3QbwD1fz6S2hL9aHbOpHHdBTDltZZU387GjXuaUWf/nMt+8tVWl9J2vOf0IM327VrtNUEuIoYBR2TmPrPN/zKwHzCjnHVbZn6nXDYduAeoAQOBUzPz2og4CLgU2Dwzf1uu2xd4DvjPzBwREcsCZwFDgQ7gWeDwzHwtIp7PzBUXwjmNAQ4AVgGuBcYBrwCjMvOZBdjvvsAxQCfwCHBUZjYWtN5W+9UdjzJ92gyu/MlxjHvkac4cdS3n/uAwAMY/8VduvPlBxlxevJkNP/D7bLbpWlwy+lcMHbISZ5y6Pzl+Ejl+kjfV96EWm0CfvnSOPhVWGUJ9x31oXHPO29apb78ntafKPnUAACAASURBVAFL0XW7rW2wNbwwkc7br6a28XbUt9iVxv+NWfzFt7GLL7qV66+/jwED+rW6lLbkPUeLQ6/uWo2II4Etge0zc1vgU8D6EbFzucrLmTksM7cDdgB+HBG1ctnjwL7ddvdpoHtzwZXADZm5bWZuBfwW+K+FWX9m7pOZ04GtgF9l5oGZecwChrgBwKkU12RL4IPAbgun4tZ66PdPsvVW6wCw4Qar89hjz85a9uRTz7Ppx4fSr19f+vXry+BByzP+iUn85t7H6du3gy8deR7nX3ArW225TqvKr7TaakNpPvloMTHpSWorrf725Wt/AprNt9YB+PtE6Ne/+H2JAdDZuZiq7T1WG7Q8PzzniFaX0ba852hx6DUtcnNxNDAsM98EyMwZEfGFzJxTG+wHgEmZ2YwIgJuBXSKiXrZW7UsR3oiIwcCKmXltt+3PoWjVmyUitgO+XU4uSdG69gxwNUWAGgAcn5ljI2I0MAToD5yVmVdFxJ+BbYGTgCUjYgLwBeAIitbBi4EPl/v/t8x8NCL+QhFC/5SZx8zhPKcBW2bmG+V0H+DNuV3AKpk85U0GDhwwa7reUWfmzE769OlgraErc+EltzFlypvMmNHJw+OeZu+9tuSVV6bw2j+mcuH5R/GLX97P90Zdxxmn7t/Cs6iofgNg2htvTTcbUKsX/y6/CrX1Nqfxs3Opb7v7W6u8MZn6GuvRcfhpMGAgnZed3oLC29vOO2/CpEkvtrqMtuU9R4tDbw9yH8rMFwEi4p+BrwADIuKuzPx34EMRMZai5XJ94Hvdtp0O3AtsFxEPUAS9icCKwMrA090PlJmdvL3FDmBd4F8y868R8Q1gb+C6ch87AisAa0XE0sD2wCeAJrBzt328AJwBrJ2Z50fEF8r536BopTs/IoZSdAVvDawGbJKZL83pgpSh9G/lNflXivB529wvIUTECMpAmpnzWrWlBi7VnylT3sqkzUaDPn06ABiyxorst8+2HH70jxm02nJssP5gll1mIMsssyQ7DFsPgGHbrceFl87zUmhupk2FJfq/NV2rFSEOqK+/FbWll6Vj/xPgg8tR65xJ49UXqW2yHY17b6L50FhYYVU6PvdlOi88uTX1S++D95yeo1n5wUFz16u7VoHXI+JDAJl5bWYOowgky5XLu7pWtwUGA8MjYutu2/+UoiVuT+Dn3eY/A6za/UAR0Tcihs92/EnAOWVr2/ZA38x8DDiXonXvPKCema8DXwYuAK4C5mdAy/rAF8sgeiGwbDn/xbmFuG611iPiLGAnYK+5tFDOkpkjMrOWmbV5rddqG2+0Bnfd/UcAxj3yNEOHrjxr2csvv84rr07mitHH8PXj9+K5519l6JorscnGQ7iz3OaBByew5hortaT2qmtOnEBtzQ2LiVWG0Hxh4qxljduvpvPS79B5+Rk0H7mbxm9vpfnUo/DmG/Dm1GKlKa8XrXpShXjP0eLQ21vkzgV+EBFfysxpEdEBbAPMKbi8DrwKLNFt3ljgBxQtcMPLHzJzUkS8GBG7Z+YvynW/AnySIvx1uQhYIzNfj4jLgFpErA8snZmfiYiVgHsi4kHg45n5zxHRH3g2Ii5/l3N7HLgiM38aESsAh5bz5+fvkv+i6GLdox0ecuiy4w4bcM99yfADRtEETjtlP0ZffjuDVlue7bdbj2cnvsTnh59F374dfO3Y3enoqHPYITvxrVOuZN8DRtGnTwdnfOdfWn0aldR8/EFqq69Lx4HfhFqNzl9eTG2zXeDlv9F84uE5btMY+3Pqux1M/RM7QL2Dxo2XLuaqpQXjPUeLQ63Zxo/kdlc+tfpz4Klus4dTPMQwnOIJzQ9ShLPTM/PFbk+tNinGpt1P8TTngRRdmSdGxNnAapn5+Yg4gmJs3IiIWI4iKK5MEf6eBI7s/tRqRIyieEjiFYruzJeAfwWuAAZRdN/+Vzl9PrAZMJniIYrvlmPk1gb26VbPWIoxcn+nGCO3DEW374jMvP7dnpiNiE2AB4C7eCvQ/nC28X7z0uyceut8rqqFpWPALgDMPPWg1hbSS/U5aTQAnY2xLa2jN+qoDwPA+87iV953enRPTJeZpx+80MJOn29c2qPOudcEOS02BrkWMMi1lkGudQxyrWOQ6xl6e9dqrxUR/w84bg6L3kvrmyRJaiGDXC+VmdcD17e6DkmS9P719qdWJUmSKssWOUmS1N4a7fs8gC1ykiRJFWWQkyRJqii7ViVJUlvzK7okSZLU4xjkJEmSKsogJ0mSVFGOkZMkSe3Njx+RJElST2OQkyRJqii7ViVJUnvz40ckSZLU09giJ0mStAhERB04D9gQmAYcmpkT5rDOjcAvMvPHEVEDJgJPlKvcm5lfn9sxDHKSJEmLxh5A/8zcIiI2B84Gdp9tnVOBD3WbHgI8lJmfnZ8DGOQkSVJba7bu40e2Bm4ByMz7IuIT3RdGxOcoRvDd3G32x4FVIuIOYCpwbGbm3A7gGDlJkqT5FBEjIqLZ7WfEPFb/APBat+nOiOhT7mc9YDjwrdm2eQ74j8zcHjgduGJe9dgiJ0mSNJ8ycwQwYj5X/wewdLfpembOLH8/AFgFuB34KDA9Iv4M3AnMLI91d0SsEhG1zJxjs6JBTpIktbfWffzIb4DPAleXY+Qe7VqQmcd3/V626j2fmbdExHeBl4AzI2JD4Jm5hTgwyEmSJC0q1wI7RcQ9QA04OCKOAyZk5vVz2eYM4IqI+AxFy9xB8zqAQU6SJGkRyMwGcMRssx+fw3ojuv3+CvCZ+T2GDztIkiRVlEFOkiSpouxalSRJ7a1lHyO36NkiJ0mSVFEGOUmSpIqya1WSJLW1Fn5F1yJni5wkSVJFGeQkSZIqyiAnSZJUUY6RkyRJ7a1137W6yNkiJ0mSVFEGOUmSpIqya1WSJLW1pl2rkiRJ6mkMcpIkSRVlkJMkSaoog5wkSVJF+bCDJElqb238sEOt2WzfL5JVS/iCkqTeo9bqAubHm8fuv9Dem/p///Iedc52rUqSJFWUXata6DobY1tdQq/TUR8GwIwRB7a0jt6q74jLAOicemuLK+l9OgbsAnjfaYWu+04V+DlykiRJ6nEMcpIkSRVlkJMkSaoox8hJkqT25hg5SZIk9TQGOUmSpIqya1WSJLW1dv7uA1vkJEmSKsogJ0mSVFEGOUmSpIoyyEmSJFWUDztIkqS25netSpIkqccxyEmSJFWUXauSJKm92bUqSZKknsYgJ0mSVFEGOUmSpIpyjJwkSWprfvyIJEmSehyDnCRJUkXZtSpJktpas9nqChYdW+QkSZIqyiAnSZJUUQY5SZKkinKMnCRJam+NWqsrWGRskZMkSaoog5wkSVJF2bUqSZLamt/sIEmSpB7HICdJklRRBjlJkqSKMshJkiRVlA87SJKkttZs+jlykiRJ6mEMcpIkSRVl16okSWprfo6cJEmSehyDnCRJUkUZ5CRJkirKMXKSJKmtOUZOkiRJPY4tcuo1Go0GI0deST7+LEss0ZeR39mfwYNXmLX8ogtv4cabfsfApfpzyCG7MGz7DZg48UW+fuKlNIGVV/4Qp5yyPwMGLNG6k6iqWo36Zw6g9pFB0DmDzusvgZdfeMc6HcOPo5kP0XjgjmKbXYZTW3l16NOHxthraY4f15r6K6zRaDDy9GvI8ZNYom8fRn57XwYPWn7W8osuvY0bb36IgQP7c8hBn2LYtuvxxtRpjDztaiZOeokZMzr55gmfY4P1B7fwLNrXuHFPM+rsn3PZT77a6lJUUQY59Rq/+r+HmT5tBleOOZFxDz/FmWf+jHPPPQqA8eMnceONv2PMVScCMHzf77LZ5mtz1vf+hy/ssx277fZJfnbN3Vw2+jaOOPIzrTyNSqqtvQm1Pn3pvPg71FYdQsfO+9I55odvW6e+w14wYKm3ttlwK+jooPOSU2HpZamvuynNxV14G/jVHY8Wr/ufHMe4R57mzFHXcu4PDgNg/BN/5cabH2TM5UWIGH7g99ls07W4ZPSvGDpkJc44dX9y/CRy/CSD3CJw8UW3cv319zFgQL9Wl9L2WvXNDhFRB84DNgSmAYdm5oRuy48GDgKawMjMvCEiBgBXACsArwMHZubf53aMXh/kImIn4Cxg88ycGhErA7cCnwa2AY4uV+0EHgaOz8zpEfFn4BmKi78UcGlmnlvuc13gTGBJYCBwEzAC2A44IjP3WcCaVwS+lZlHRcQewKnABcCwzNxzAfd9GrAjxXn9W2bevyD760keemgCW2+9LgAbbrQGj/3hL7OWPfnkc2z6ybXo168vAIMHr8D4nMiEJ5/jlJH/AsDGmwzhu2dcvfgLbwO1QWvRmPAoAM2JTxatbN2Xf+wT0GzSnPDIrHn1IevTfOFZOoYfC7UanTddsVhrbhcP/f5Jtt5qHQA23GB1Hnvs2VnLnnzqeTb9+NC3XveDlmf8E5P4zb2Ps+vOG/OlI89jqaX6c/I39m5J7e1utUHL88NzjuDEEy5tdSladPYA+mfmFhGxOXA2sDtARCwHHAVsBPQH/hgRNwJHAo9m5oiI2Ac4CfjK3A7Q68fIZeZtFMHt7IjoC1wFHEeRnr8EfDYztwG2pwg3B3bbfOfM3A7YEjguIlaIiGWAMcAxmbk9sDmwPnD4Qqz5+cw8qpzcDfh6Zp6zEELcxhT1bg7sA1y4YJX2LJMnv8nApQfMmq531Jg5sxOAtdZahQceeIIpU97k1Vcm8/DDT/HG1Gmss/aq3HFHES7uuH0cb0yd3pLaq67WbwC8OfWtGc0G1MvbzwqrUF9/Cxp3/PztGy05kNqHVqTzp9+ncfeNdOxx6OIruI1MnvImAwd2f93X33rdD12ZBx6aULzuX53Cw+Oe5o2p03nllSm89o+pXHj+UWy/3Xp8b9R1rSq/re288yb07dvR6jK0aG0N3AKQmfcBn+hakJkvAhtm5gxgReDVzGx23wa4maJxZa56fYtc6ZvA3cAvgP/LzNsi4mbga5n5KkBmNiPiuPIiz25J4E3gVWBf4PbMfKLcrjMiDgCmUwQ+ACLiy8CeQF/gtfL3jwKjgRnATKBru6soQndf4AiKptYxwOkUQe6TEfEicG1mrhgR6wPnADXgJeCLwMbAd8v9XZCZl89+Epn5+4jYpTzXwcDf5ufiRcQI4NvlPuZnk5YYOLA/U6a8OWu62WjSp09xEx0yZCX2Gz6Mww87h0GDVmCDDVZn2WUHcvzxe3PqqVdy0433s9nma7PsMgNbVX6lNadNhX7935pRq0GjeIysvuFWsPSydBx4ArVlloPOTpqvvghTJ9MY/3Cx/V+S2oc/0orSK2/gUrO/7htvve7XWJH99tmWw4/+MYNWW44N1h/MsssMZJlllmSHYesBMGy79bjw0ttaUrvUE3V/zyudkpkj5rL6Byje47t0RkSfzJwJkJkzyzxwCsX79uzbvA58cF719PoWOYAyDV8I7AR0tXGvDkwAiIgtImIscHdEjOm26f9GxK+BBO6iCGArA0/Ntv/JmTmrKafsM/8wsGPZ2tcX2LQ8/oMU6fs0YFngkxT/obsC/0bxH9y13+spUvvxmXlvt0NeCBydmcMounWPL+f3z8xt5hTiuu1zZtm9egPw07mtN9s2IzKzlpmtGYQwnzbeZE3uuvMPAIx7+CmGrrXKrGUvv/w6r7wymSv++3i+/o0v8NzzLzN06Crcc88fOero3bjgwq9Qr9fZcst1WlV+pTWfeYL60A0AqK06hObfJs5a1rjtajovGknn6DNoPHw3jXtvoTnh0bdtw0dWo/nay60ovfI23mgN7rr7jwCMe+Rphg5dedayl19+nVdencwVo4/h68fvxXPPv8rQNVdik42HcGe5zQMPTmDNNVZqSe1ST9T9Pa/8GTGP1f8BLN1tut4V4rrt7z+BlYBtI2L72bZZmqKRaK5skQPK1qevUQSeK8oL+SxFmBtXhqRhEbE28ONum+6cmW9GxBIUgWk/4C/AJrPtf3Vgta7pzGxExHTgyoiYDKxKEeYuBk6gCGevAd+gaFYdStFaOINiPNy7WQc4LyIo9zu+69Dzcz0y85sRcQZwX0TclZlPzs92Pd2OO27EPff8ieH7fpdms8lppx/E6NG3MWjQCmy//QY8O/FFPr/36fTt24ev/ftedHTU+ejqK3LSNy9jiSX6suaaK3HSycNbfRqV1Hz8QZpD1qXjkJOAGp2/uIj6FrvQfPkFmvn7OW7TeHAsHbsdSMehJxfb3DB6cZbcNnbcYQPuuS8ZfsAomsBpp+zH6MtvZ9Bqy7P9duvx7MSX+Pzws+jbt4OvHbs7HR11DjtkJ751ypXse8Ao+vTp4Izv/EurT0NaIM1Gy9oZfgN8Fri6HCP3aNeCKN6k/wPYi+L9fRrQKLf5J+B+ikacu+Z1gF4f5MoQdjVwbGbeFBEfp2gy/RHwvYjYOzO7mjiHwTsfnCsffvgbsATwM+AbEXF+Zj5ZjrsbBdwG/LE85gbAHpm5WUQsSdEKV6MYAHlXZp4SEftShLrLgecyc+eI2IKiO/XgdzmtBA7IzGciYiuKpA/FC2Re12IHYK/MPJqiq3jGu21TJfV6nREj9nvbvDXWWHHW76ec8s43qw03XJ1rfvbNRV5b22s2adxw2dtmNV587h2rNcZ2G4vVOZPOX1y8qCtre/V6nREnfeFt89ZY/a1u6lNOfuezV8t8cCnOGeWYxMVhlVWWm/W0vNrStcBOEXEPxfv8wRFxHDAhM6+PiHHAvRTZ4ubM/HVE/A64LCLuphgONc8WhF4f5CieILk7M28qp4+iCFa3A/8FXFe2bH2A4qnV7g87/G9EdAIdwETgvzNzWkQcCFxYdqEuDfwSOJ/iqVUoumynRMQDFAn8OYou2fsoWgRnUgSoYyla+K6KiGMonpwdOR/ndCTwk4joGkV7SLn/d/NrYO+I+E15Tudm5tPzsZ0kSZpNZjYoxrZ393i35adQjI/rvs0bwHw/Kl5rNv1kJi1Uzc7G2FbX0Ot01IcBMGPEgfNcT4tG3xFFa2Pn1FtbXEnv0zFgFwC87yx+5X2nR4+N7vLiPgcvtLCz3JhLe9Q52yLXC0XEt4Ad5rDoYFvgJEmqDoNcL5SZI5m/LlpJktSD+fEjkiRJFWWLnCRJamut+q7VxcEWOUmSpIoyyEmSJFWUXauSJKmttfCbHRY5W+QkSZIqyiAnSZJUUQY5SZKkijLISZIkVZQPO0iSpLbWzl8rb4ucJElSRRnkJEmSKsquVUmS1Nb8ii5JkiT1OAY5SZKkijLISZIkVZRj5CRJUltr+F2rkiRJ6mkMcpIkSRVl16okSWprfrODJEmSehyDnCRJUkUZ5CRJkirKMXKSJKmt+RVdkiRJ6nEMcpIkSRVl16okSWprdq1KkiSpxzHISZIkVZRBTpIkqaIMcpIkSRXlww6SJKmtNXzYQZIkST2NQU6SJKmi7FqVJEltrdmwa1WSJEk9jEFOkiSpogxykiRJFVVrNputrkHtxReUJPUelRh89vQ/HbnQ3ptWv+n8HnXOBjmpm4gYkZkjWl1Hb+S1bx2vfet47bWg7FqV3u7brS6gF/Pat47XvnW89logBjlJkqSKMshJkiRVlEFOertTWl1AL+a1bx2vfet47bVAfNhBkiSpomyRkyRJqiiDnCRJUkUZ5CRJkirKICdJklRRBjlJkqSK6tPqAtSzRMQawJnAqsAbwFTg+Mx87D3u59PAPpl5UET8PDP3fI/bDwI2zMxfRsRoYBPgZaAf8AfgqMyc8V72OZfjrA8sm5l3RsQY4IDMnP4e97EuxTVbEhgI3ASMALYDjsjMfRawxhWBb2XmURGxB3AqcAEw7L1e1zns+9vAZ4CZwDGZef+C7E+StHgZ5DRLRCwJXA98KTPvLed9EjgXGPZ+9/s+w8YOwNrAL8vp4zPzlrKmnwK7Az97vzV1sxfwPHDn+wlcEbEMMAbYMzOfiIgO4BrgcODxhVAfmfk8cFQ5+SWKkL0nUIuIe4AfZObV76HmHwCjgOUowuZmwGrA/0TEycCgzLzgPexvfeBH5eTmwP1AA/heZt44v/uZw34rG5DL/a8JXJeZ60XEsDnVHBFfBvYDuv4ouS0zv1Mumw7cQ/Gl5AOBUzPz2og4CLgU2Dwzf1uu2xd4DvjPzBwREcsCZwFDgQ7gWeDwzHwtIp7PzBUXwvmNAQ4AVgGuBcYBrwCjMvOZBdjvvsAxQCfwCMUfbY0Frbfc904U12XzzJwaESsDtwKfBrYBji5X7QQeprjvTI+IPwPPAE1gKeDSzDy33GdlX6cRcRqwI8V5/Zt/yFWTQU7dfRa4vSvEAWTm/RGxfdkq9uHy57PAdyne/D8M3JyZJ0fEOsAlwJTy5xWArjeO8g3/HIo3ppeALwIbAycA04HVgauAM4ATgSXLoDJLGZSWpripEhFfBfahaFG6MzNPKMPVFcAHKF7jJ2Xm7eVNaweKIQVXUgSug4DpEfEQcDVFePwxMA34KLAScFBmPhQRhwBfpmgZnF7W2iyv2RPl9fr/7Z191JfzHcdf3UR1DsY5wzLDMXuzYQ+Wp01PS5iHYsORmWIaxpItz8Oxk5mnsKOdNLYzq7CzSR6ixJ0yMWZbh7yNeUgMjYiOEu2Pz+eqy+33K6pZt31f/9z9rufrc32v7+f9eX+/1+kdSd/N9bvXrvsEQnx1BF7Lf28J/IZI4ouJpFgdtyW3PRaYT4jF84Hd8pCnE8nzs8D9koZlzOtx/Xke7yrb19ae6Ul5Tf2BSbaXAM9KWht4qBLMHxTbM0mhnwmvr+23Pswx2vI/EMj7Aafbvploo6uEpCOAIYRYbrbNcUQb6WX7rRRjYyT1tT0JeMV2z9x2A+BxSeNz98eAw4D78/feRLuqGAeMsn1j7j8UGEW8K6uFSqBI+howxfaPVvWYkjoTQmUH2wskjSOezYRVPTaA7cmS7gAukTSEeNdOBr5IFEn7254nqQNR7BwJjM7d++ZzWgeYJen3xPvVLtuppC8ThdeuwBbATUQcCu2MIuQKdbYCnqh+SLoJ2IAQM7OBW22PkLQlMMP29yR1Ap4DfgL8lKgcJ0s6FdiuzfFHA0fZfjRF0SnAZKIT2ZEYNn3e9nBJFwDb2p4g6SDgQkmnAV2BeYBTGB5CJMPFhKO0HyEqJtu+XNJmwHRJWxNCqTvwPCHO5qRA/VcK1vq1PmP7+5KOAQZLOosQnF8iRN7duV1X4J/1HW2/kfGr4thCCN4+tt/NRNItj/UQkUj2ADbMWLwGDAA+T4jR+XncCZIeADrZvk8Stt/I5DeNEM7fBh7JWHcCfghcnkJyDuH+TCQE4vZAH0n75r4LgEGSNrJ9WhORfC7RTjbOax1q+w6aIKkVeDnvbV9gJOEStRACu1VSD2A44YI8SSTBfny0Ank/YGdJc4EbV1B4NBTIbXiVcGSebBYbwv3pWYle229LOjSFdVvWB+bYXpLtaiKwl6SWdKsOI8QbkrYANq1EXHIF4RYtJeNe/YftXTJOzxIFzQZAZ8KRas33ZGuiTV1s+/oU7d2Bs4ii6wngUCK2LwBXE+0ewu2ZKekZQuDMqgqKNiwEdre9IH+vDaxSUdCAM4HphHC5M/uricAw2/MAMs4nN3kWXfKa5hFxb5ft1PbDkvbKe90CePHDBrKwZlA+dijUmU0kaQBs90tH4FVCrDlXvQJ0kzQGGEEIMIAvEMNqAPc2OP52wMhM7kcRIghgpu3Ftt8k5uQ14hTbPW1/DrgduIRwz2bYfjs73Gl5DdsB9+Q9zAFeBz5JiJKfEUMpn1hBLB6uxaQT4Xw9anuB7XeIIS+AZwhncimStpLUvfqdiXYRME7S1cTQaEci0c3N+zmB6MwnAlOJJHMeMUS5IjYhksDRwEvAXcARhMC8ChhkexfgTt4rrrcGZhGC4xoieS/Ie6iL5N2BbVIkAyy0vQ/hOA39ANc31nYf4pnPtd2dEGpXpvMxmnA0ehBicyBNBLJr8xfbCOQ9iJh2A/YkBHIfQiBuCOxMJNB9CHG7fu24E4hncErdjc7r+kG+A7cRhQeEkN5jOSIO27dke14eG9mem/dyYL4XMyRdXK2X1CrpHmKIsT6VYBFwH9BD0np5P8/luq7AU22u5x3bdccO4l35ju3ehON1MNEmNiVc9wGEQFsP6EWIj32IodqKlwgHfaztX9aWn0G4dL2AwUC1bnNgQBMRh+13bb+YMTmREJ+TG227sjjm1o4m2smvc/HSIlbSbvkspufwccUkSVOJfnAaIcDaeztdnCMVtwBjm21XWLMpQq5Q5ybCodm1WqCY5/Npwn2pRMVAYJ7twwlB1SUT8mMsG/rr1uD4Jj4m6El0NtX8qUZV77s0b5+zgXXyfLtIWjvP3x14nBAne+T1b0Z0kPOIRHUYMbw6MKvQZudpe01PANtK6pwd8865/BZg73T8qrlKlxJuF7lsR6C/7UOBE/N8HQgxM832N4jhmFMJN/EF232JIabzm8SgzlwiUb5GuCg7Ecm2M7CJ7VkAtkfa/kttv5/mdU4hhGAH4I1c10wkw/tF7oqoCoAdgG9mkvwD4bZsTDi+N+TyvsBnWDMEcrPCw033+HDMl7RR3suN+V6cw7Lh2FeyeOlOvH8DJH29tv9Yoj0fBPyxtvxZIhZLkdRR0oA2558DXJFuWy+go+OjpisJd28k0GJ7PhHHqwi3aF1WzA7AURm70cQ7CCHk/728HSW1pJjdE/hWE1dspcn3fhjRB/0uh0OXFrG278tncTQhaiv6ZrGxOVHYHc7HoJ3aPjP3GVb1Y4X2RRFyhaXkkOD+wEmSpkq6l+hsBhMdVsUUIiH/iRAQ/yA6guOBMyRNISbQt+U44LeSphFV/N+XczkzgX6Sqjk9F6Y7MYXoQM91zM26gXD/HgCeBsYT4qd3OhnjgcG2FxJO4l8Jx2oSkfAeAk6Q1GsFsZlLDFVMIzrezsDbtl8n59FUjgoxwsVM1QAAAy1JREFU6bvuTjwBvCnpQcJdeCHj9SAwPONxLPHBwN+AYyTdB1xEOIhNSbdkEXAA4V68SST2VkKUPS9pm9z2VEkH1nbfnhDTXQhhOqO2rplIhsbCe3lUyegxYFwmyX0I8foy4SRV7u9wYth6TRDIzQqP1TLxnhBMl0laN+9nLaIAaRTf+UQxsk5tWSsxv+lgam5dutBzJfWrbTsE6N/mmL8i3NqBxHSDDunErmd7X6Jd/0LSp4CdbB9IDI9fqJhPuTweA0Zk7A4BxuTyDxK7UUSB0L82xLpaUMxvu4GYEjCC6APOId69ixRzESt60uBZpNv2IvEs2m07ldRb0pX58y3CYVxdbbvwEVLmyBXeg+2naTwh+rbaNo8QFXcjurdd4PxCzvZDvP/r18eJhNR224eBatLadTTB9qVEx1nnFd6ftLB9HlHl1rmVZR3flvl3YG2f24HbM3F1tf1VgBSJs2v31bvB5bXW7q3ReljmYNbp02BZ5ZJewDL36hEikZxKfPhwCeGEXkfMnzlPUjfgGknvEgLyMiKpA/wZGEQIvllE4uiR9zRTUiWSW4g5ReNZtcnQowjBO5UYMhrpmDM4BLg1nc7XiaT0uqRKILcQH7jcTAjkHnm8ukBeyDKBPINwWhYTiWkoUYhcL6n6GrJtO2hEVXhUQ4lHs8ztWBn65rVWDCBE+GRJ7xBD262EWwQ5tEqIiU5EsXI3IbDI2E0GNs941c91BDF0/WNCcDxJTOavcy3xocyrhDDpShRl59TmeZ1NfNW9qaSHibZycQ7JLe9ehwNXSxpMPOtzVxgdQNJXiDhPA+7Kc1zeZr7fqnAJMN121Z8dTxRzdxHtc3yec32i6Duytu+kfE5rEcXHGNsL23E7nQocnAX7WsCVtp9awT6FNZAOS5asVte6UPjYIul84uvARcTXgiet7mGf/waSzqaxkBxUOu5CoVBo3xQhVygUCitBEcgrj6QDiK+127I63bcCpZ3+P1CEXKFQKBQKhUI7pXzsUCgUCoVCodBOKUKuUCgUCoVCoZ1ShFyhUCgUCoVCO6UIuUKhUCgUCoV2ShFyhUKhUCgUCu2U/wD0bE8KJohJKgAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "stack_soft_plot = stack_models([gbc,dt,lightgbm,xgboost], plot=True)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Before we wrap up this section, there is another parameter in `stack_models()` that we haven't seen yet i.e. `restack` parameter when is set to True by default. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 11.0 Model Calibration" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "When performing classification you often want not only to predict the class label (outcome such as 0 or 1), but also obtain a probability of the respective outcome. This probability gives you some kind of confidence on the prediction. Some models can give you poor estimates of the class probabilities and some even do not support probability prediction. Well calibrated classifiers are probabilistic classifiers for which the output in form of probabilities can be directly interpreted as a confidence level. PyCaret allows you to calibrate the probabilities of a given model through `calibrate_model()` function. See an example below:" - ] - }, - { - "cell_type": "code", - "execution_count": 29, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.82020.73700.35410.67930.46550.3701
10.81140.74640.31160.65480.42230.3261
20.81020.75080.35980.62250.45600.3508
30.80390.73110.30880.61240.41050.3079
40.80890.71300.33140.62900.43410.3322
50.81200.73450.32290.65140.43180.3342
60.80330.73140.33140.60000.42700.3200
70.80330.75060.32290.60320.42070.3150
80.81140.72310.33710.63980.44160.3410
90.80630.73230.34560.61000.44120.3347
Mean0.80910.73500.33260.63020.43510.3332
SD0.00500.01130.01610.02450.01580.0171
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8202 0.7370 0.3541 0.6793 0.4655 0.3701\n", - "1 0.8114 0.7464 0.3116 0.6548 0.4223 0.3261\n", - "2 0.8102 0.7508 0.3598 0.6225 0.4560 0.3508\n", - "3 0.8039 0.7311 0.3088 0.6124 0.4105 0.3079\n", - "4 0.8089 0.7130 0.3314 0.6290 0.4341 0.3322\n", - "5 0.8120 0.7345 0.3229 0.6514 0.4318 0.3342\n", - "6 0.8033 0.7314 0.3314 0.6000 0.4270 0.3200\n", - "7 0.8033 0.7506 0.3229 0.6032 0.4207 0.3150\n", - "8 0.8114 0.7231 0.3371 0.6398 0.4416 0.3410\n", - "9 0.8063 0.7323 0.3456 0.6100 0.4412 0.3347\n", - "Mean 0.8091 0.7350 0.3326 0.6302 0.4351 0.3332\n", - "SD 0.0050 0.0113 0.0161 0.0245 0.0158 0.0171" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "rf = create_model('rf')" - ] - }, - { - "cell_type": "code", - "execution_count": 30, - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAEuCAYAAABxmEC5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3hTZfvA8W/SPWnL3puHWUaL7CGiKMPBD0ThxYGoKIrIcIGCvII4EFBxIIoLRREUkeGrYAEZIns/BaRY9m6hO+P3R9IaGW1KkyYp9+e6uJqceZ/T0DvPPAar1YoQQgghfIvR0wEIIYQQovAkgQshhBA+SBK4EEII4YMkgQshhBA+SBK4EEII4YMkgQshhBA+SBK48FlKKT+l1Ail1Eal1Fal1G6l1GtKqSAn9rUqpcoopW5XSr1tX5aglOpTxJhqKqXm219XUkqtLcrxnDxnZ6XUTie2e0kpdUcRz9VMKTW7kPvUUEpdtL8eopR6roDtr3o9SqkJSqn77K+v9DvsoZSaUJj4PEEpFaeUmunpOIRv8/d0AEIUwftANHCT1jpFKRUGzAFmAQOdOYDW+kfgRxfGVB1Q9mMfBdq68NhF1QXYfa07K6WMwMfA7dd6DK31B9e6r33/l66wzPF32BKIKco5ioPWepNSyl8p1VNr/ZOn4xG+SRK48ElKqRrAAKCi1joVQGudppQaArSzb1MPmAFEABWBrUA/rXWmw3EeAPporXvaF91lLyGGAnO01hPt51oN7AFqAJ2AB4E7gBAgDBiFLYnMAiorpX4GHgV2aq3DlVIBwFvATYAZ+AN4Wmt9QSmVBHxqX1cN+Fxr/eIVrjkJ+Bq4GYgCpmit379km1L2a24GWIGlwAv2WOKBN5RSZuCUPR4/+3avaq3nF3Db7wYOaq2POMTzBxBrP8cG4F37NQQAc7XWky6JbzxQRmv9hFKqp32/QKAc8JnDdYcrpb4D6gDngUe01olKqU/t9/RNh2M+APQB/gsMAfyUUinADcC3WuuP7NuNBUprrZ++JKZWwNvYfo/ZwCit9QqllBUoq7U+bd/OCpQFGgPTgTQgHNgFbNRaT7Fv9xjQWWvdTynVCxhrv8Z0+7HX2U89E9uXUEng4ppIFbrwVXHArtzknUtrfdwhET2MLSm0xpYIagI9CjhuJNDa/u8/Sqnb7MurAP/VWtfD9se4K7Y/0rHAGGCC1toMDAYOaK27XXLcsUAloKn9nxF4w2F9uNa6A7YS+yilVM2rxBeDrZTZGZiglGpyyfq3gTNAE2wJuym2pDED2AiM1lp/D7wMvKW1jgMGYSudF6QPlyebnVrrBvZjfgF8Yj/mDUBXpdTdVzqQUsoAjATu11rHY7vfzyulytg3qWqPrxnwlf3Y+dJa/wF8AHyjtR6D7YvMw/bzGYGH7Osd4wgAfsD2+2ts3366ffv8NAbutf/+ZwIPOKx7APhIKVUXmAR011o3Bx4BFthritBarwdq5/O7FiJfksCFr7JQ8Of3WeCUUuoZbCWdSthKTPmZpbU22b8YfIettAtgAtYBaK0PAfcBA5RSk7GV+go67m3AB1rrHK21BXjHvizXQvuxjwAnuXo18AyttVVrfRhYBtxyhfO8a98mC1vCuu3SgwDfAjOUUnOwfRl6oYD4AeoD+y9ZthrAnpQ6Af9VSm0F1mMriTe70oG01lagFxCnlBqHrTbAgK0UDLBda53bf+BTIN5eu1AYi4DySqmmQDdstQf6km2aAGat9WJ7XJu01k3sv6P8JNs/BwAJQLBSKl4p1RBbKX05ts9ORWC5/Z7Mwfa5reNwnIPYm1yEKCxJ4MJX/QE0UEpFOC5USlVWSi1WSoVgq25+BDgETAU2Y0sS+TE7vDYCOfbXWVprk/0cLbAl80jgf8BrThw3t6ra8dgBDu8zHF5b8zme6ZJjmC9ZbyzgPABorT/Elrx+wZbctiulgvOJPzeuS/9mXLT/9LPH3FZr3cxecm6NrQR6GXvC3wK0wPZ7GY3tXude96XXZeWf34VT7DUiH2KrYRjEJaVvOxP/vl8opRorpXKbFw32ZYGX7Jd73blfRj7G9qXuQeBj+zI/YHnu/XC4J44d9HK4/FqFcIokcOGT7B3E5gCfKKUiAew/3wPOaK0zsCWmCVrrb+y7tcL2RzU/9ymlDEqpaGxtvsuusE1HbG2ebwErgTsdjmviCgnTfpzHlFIB9urZodiSZ2Hl9sCuhq30vfSS9T8DT9ivIQjbF5jc8+TFZu8d31xr/al9myigQgHn1kDtK66w1VisB0bYjx8FrMHWT+BK6mL7AjRWa70IW5NAEP/cx6ZKqdzS+6PA71rr9ALig8vv/yzgLmy1DN9f5ZqsSqmb7XG3AFZg+9t4ClszBED/As77KbbOfX2B3F76y4FblFL17cfuDmzH1m8itxmhuj0GIQpNErjwZY9j61W91l5F+Yf9/WD7+heA75VSO7CVxFby7+rLK0kBNgFrgXe01r9dYZuvgTJKqT32810EYuy1AbuBTKXUBv5din4FOI6tI90ebEnmqcJdLgA1lVKbsH0hGHaFKuFh2DqE7bD/08BE+7ofgVeVUvcDz2BrQ9+CrQr4Za11UgHn/g64NZ/1/YHW9vv9B/C11nrOVbbdjq09fa/9PvbCdu9yfz97gHFKqW3YEuP9BcSWawXQTSn1DoDW+iS2tv+vtdaXleDtzQy97efaiq2U3ltrnY3tXs5QSm0GGgDHrnZSrfVxbDUJ2+1fLtFa78b25Wiu/Tr+C9yutc4tvcdj6y/xt5PXJsS/GORxokL4Bnuv7z5a640eOr8fti83PXJ7ons7e6e4P4GOWutkT8fjyN6jfl5u+7sQhSUlcCGEU+xtyg9zlXZtb6OUehhbSf4NL0zecYBVkrcoCimBCyGEED7IrSVwpVQrpVTCFZb3Ukr9qZRaZ/+WLIQQQohCcFsCt4+9nQUEX7I8ANuQnluwjRt9RClVUO9XIYQQQjhwZwn8ALbenZdqAOzXWp+z9/T8HejgxjiEEEKIEsdtc6Frrefb55C+VCS2oTq5LgBOzbBkn0d5HED37t25444iPVhJXMXp06cpU6ZMwRuKQpN7615yf91H7q1rmc1m/Pxs0x6kpqZy++23FzQZ1GU88TCTVGwPl8gVge1hBQXSWo8HxgMkJCRYO3fu7OLQBEBCQgJyb91D7q17yf11H7m3rjNr1iwmTpzIhg0bKFu27DUfxxPDyPYAdZVSMfbpCTtin2NaCCGEKOnS09M5f/48O3bsKNJxii2BK6X6K6Uesc+GNALblI/rsD29yCcmhRBCCCEK68CBA4wcORKz2Tbt/WOPPUZiYiJdujjzEMCrc2sVun1qxtb21185LF+E7UlBQgghRIk2ceJEZs+eTbt27ejduzcBAQFFqjrPJTOxCSGEEC5ktVpZt+6fluGJEycyd+5c7rrrLpeeRxK4EEII4UKDBw+mXbt2bNxoe2xBxYoV6devHwZDoTua58sTvdCFEEKIEsVkMuHvb0upAwcOJDU1lXLlyrn1nFICF0IIIYrgvffeo169epw9exaAzp07M2/ePKpVq+bW80oCF0IIIYogPT2dlJQUdu7cWaznlQQuhBBCFILWmpEjR2KxWAAYNmwY+/bto2PHjsUahyRwIYQQohAmTJjAW2+9xdKlSwEIDAwkJiam2OOQBC6EEELkw2KxsHbt2rz3kydPZv78+XTv3t2DUUkCF0IIIfJ133330aFDB7Zt2wZA1apV6d27t8uHhRWWDCMTQgghLpGTk0NAQABgS+BWq9XrnsYmJXAhhBDCwbRp06hfvz7nz9selHnLLbcwZ84cKleu7OHI/k0SuBBCCOEgPT2dixcvsnfvXk+Hki9J4EIIIa5ru3fvZtSoUVitVgBGjBhBYmIirVu39nBk+ZMELoQQ4ro2btw4pkyZwooVKwAIDg6mVKlSHo6qYJLAhRBCXFfMZjNr1qzJe//666+zaNGiIj+fu7hJAhdCCHFdueeee+jUqRO7du0CoGbNmvTs2dPjw8IKS4aRCSGEKPEch4U98MADhIWFUbp0aQ9HVTRSAhdCCFGivfbaa9SvX58LFy4A0KNHDz799FMqVKjg4ciKRhK4EEKIEi0jI4P09HS01p4OxaUkgQshhChRtm3bxujRo/OGhT377LMkJiYSHx/v4chcSxK4EEKIEuWll17izTff5PfffwcgJCSEiIgID0flepLAhRBC+DSTyfSvYWFTpkxh2bJldOjQwYNRuZ8kcCGEED7trrvu4sYbbyQxMRGAOnXq0K1bNw9H5X4yjEwIIYTPcRwWNmjQICpWrEh0dLSHoypeUgIXQgjhUyZMmEDDhg1JS0sDbCXwmTNnUrZsWQ9HVrwkgQshhPApmZmZZGZmsn//fk+H4lGSwIUQQni1jRs38swzz+QNCxs7dix79+6ladOmHo7MsySBCyGE8Gpjx47ljTfe4M8//wQgNDSUsLAwD0fleZLAhRBCeJXs7Oy8MdwA06dPZ/ny5dxwww0ejMr7SAIXQgjhVXr06EHXrl3566+/AFBK+dyjPouDDCMTQgjhcdnZ2QQGBgLw8MMPo5QiKirKw1F5NymBCyGE8KixY8fSpEkTMjIyALj77rt59913iYmJ8XBk3k0SuBBCCI/KysoiOzubgwcPejoUnyIJXAghRLFav349zz33XN778ePHs3v3bho2bOjBqHyPJHAhhBDF6oUXXuC1115j69atAISFhRESEuLhqHyPJHAhhBBulZmZ+a9hYe+++y6rV6+mWbNmHozK90kvdCGEEG51yy23sHHjRvbu3Uu1atWkqtxFJIELIYRwuaysLIKCggB49NFHiY+PJzIy0sNRlSxShS6EEMJlrFYro0ePpnnz5mRnZwMwYMAA3nrrLRnX7WKSwIUQQriMwWDIGxZ26NAhT4dTokkCF0IIUSSrVq3ihRdeyHs/adIkdu3aRd26dT0YVcknCVwIIcQ1s1qtjBkzhldffZXdu3cDEB4entf+LdxHErgQQohCycjIyBsWZjAYeO+991i/fr30Li9mbuuFrpQyAu8BTYEsYLDWer/D+lHAvYAFmKS1/t5dsQghhHANq9VKp06d2L17N1prKleuTJMmTTwd1nXJncPI7gSCtdZtlFKtgSnAHQBKqShgGFAHCAO2ApLAhRDCS+X2KDcYDAwdOpQ9e/bIsDAPc2cCbw8sA9Bar1dKxTusSwMOYUveYdhK4UIIIbyM1Wpl2LBhLF68GK01AQEB3H///Z4OS+DeBB4JpDi8Nyul/LXWJvv7ZGA34Ae86swBlVLjgXGAPNzdjZKSkkhISPB0GCWS3Fv3kvvrHn///TeZmZl89913VKxY0dPhlEidO3cu9D7uTOCpQITDe6ND8r4NqAjUtL//WSm1Rmu9Ib8Daq3HA+MBEhISrNdywaJgCQkJ1/RhEgWTe+tecn9dY/ny5fz222+88sorAMTFxfHHH3/QtWtXD0cmHLmzF/oaoDuAvQ18h8O6c0AGkKW1zgTOAzJFjxBCeJjVauWFF17g1VdfZf9+W7/jiIgI/P1l5m1v487fyPfAzUqptYABeFApNQLYr7X+USnVFVivlLIAvwO/uDEWIYQQV5GWlsaWLVto3749BoOBmTNnYjKZqFOnjqdDE/lwWwLXWluAIZcs3uuwfhz29mwhhBCeYbVaadu2LUlJSSQmJlK+fHmaNm3q6bCuCzWeW+z41po0uYehMPtLnYgQQlyHMjMzCQ4OxmAw8OSTT3Lo0CHCw8M9HZYoBEngQghxHbFYLAwZMoQNGzawceNG/P39GTx4sKfDEtdAplIVQojriNFoxGw2YzabOXbsmKfDEUUgJXAhhCjhli1bxtq1a5kwYQIA06ZNIyQkRHqW+zgpgQshRAlmsVjyhoXlPp9bhoV53nebDhf5GPIbFEKIEiY1NZUdO3bQrl07jEYjs2bNwt/fn+rVq3s6NAHsO3GBF3/YWeTjSAIXQogSxGw206pVK44fP86+ffsoU6YMLVq08HRYwi4928TjczaTkWPm/QEtuK1J3tS0hRpCBpLAhRCiRMgdFubn58dTTz3FqVOnCAsL83RY4hLjFu5i38mLPNC2hmPyviaSwIUQwoeZzWYeeughdu3axR9//IHRaGTIkEvn0BLe4LtNh5m36TBNKpfi+e71i3w86cQmhBA+zM/PD5PJhMlk4sSJE54OR1xFbrt3RJA/M/q3IMjfr8jHlBK4EEL4EKvVyqJFi9iyZQvjxtlmo37vvfcICwvDz6/oSUG43qXt3tVKh7rkuFICF0IIH2KxWBgzZgyTJk3i6NGjAERGRkry9mIvubDd25EkcCGE8HLnz59nzZo1gK3KfPbs2WzZsoVKlSp5ODJRkHkbk/lu02Fiq7im3duRVKELIYQXM5lMxMXFkZKSwr59+4iOjiY+Pt7TYQknJJ64wIsLdxIR7M+797qm3duRJHAhhPBCGRkZedOdPv3001y8eJHQUNe0nQr3S882MXTOZjJzLEzr18xl7d6OJIELIYQXycnJ4f777+fgwYOsWbMGo9HIE0884emwRCE5tnvf2th17d6OpA1cCCG8SEBAQN7Tws6cOePpcMQ1cGe7tyMpgQshhAdZrVYWLFjAnj17GDt2LAAfffQR4eHhGI1SxvI17m73diSfDiGE8CCTyZQ3LOzUqVOAbViYJG/f49ju/UafWLe0ezuST4gQQhSzM2fOsHbtWsBWZf7555+zdetWypYt6+HIRFEUR7u3I6lCF0KIYpSdnU3z5s3Jyspi3759REZGcsMNN3g6LFFExdXu7UgSuBBCFIP09HRCQ0MJDAxkxIgRmM1mgoODPR2WcIHibPd2JAlcCCHcKCsri/79+3Pq1ClWrlyJwWBg+PDhng5LuEhxjPe+GqfawJVSFe0/OyilhiqlQtwblhBClAxBQUFYLBbMZjPnzp3zdDjCxYq73dtRgSVwpdT7QKBSagrwFfA/oA3wHzfHJoQQPsdqtTJ37lySkpJ4/vnnAfjss8+IiIjAYDB4ODrhSp5o93bkTAn8BmAwcDfwsdb6IUC5NSohhPBR2dnZvPjii0yaNCmvxB0ZGSnJu4TxVLu3I2cSuJ99uzuApUqpUCDMrVEJIYQPOXXqFOvWrQNsVeZffvkl27dvJzo62sORCXco7vHeV+NMJ7bPgWPAGq31H0qp3cCH7g1LCCF8Q2ZmJrGxsRiNRrTWhIeH07p1a0+HVWQ1nlt8+cJltmVJk3sUczTexZPt3o4KTOBa67eUUtO01hb7oo5a69NujksIIbxa7rCw4OBgRo0aRWBgoAwLuw54ut3bkTOd2KoDs5RSNYCOwByl1CCtdZKbYxNCCK+TkZFBv379SEtL49dff8VgMDBy5EhPhyWKgTe0eztypg38Q+AN4CJwHPgaW7W6EEJcd0JCQjCZTJjNZlJTUz0djluYLVZPh+B10rNNPJ7X7t3UY+3ejpxJ4GW01v8D0FpbtdYfAZHuDUsIIbyDxWLhs88+4/XXX89b9s033/Dbb79RqlQpD0bmelarld/0SbpPX53vdhuTzhZTRN7jxR92sf/kRR5sV4NbG1fwdDiAc53YMpRSVQArgFKqPZDl1qiEEMJLZGVlMW7cOM6fP8+QIUOIjIwkIiLC02G53K6jKby6ZC+/7z9NQSPe7v5wHU92qcuTXerg71fyn4k1b2My8zcfpmmVUjx/WwNPh5PHmTs/AvgJqKuU2optMpdhbo1KCCE86NixY6xfvx6wVZl/9dVXbN++ncjIklf5ePR8BiO+3UrPd37n9/2n6VivLEuGdch3n4qlQpi+fB/9Zq4n+Wx6MUXqGf9q9+7fgkB/7/nC4kwJ/CDQEqiHbUz4Xq11tlujEkIID0lLSyM2NpaQkBD27t1LaGgobdu29XRYLnchM4f3Ew7w8e8HyTJZaFAxkhe616dDXdsjTS8dKpaQkEDnzp0BSMnIYewPO1m07Sjdp6/mv3c25s7mlYv7EtzOsd17Wr/mVI3xfLu3I2cS+BZgK/AlsFCStxCipLFaraSnpxMWFkZYWBijRo0iKiqKoKAgT4fmcjlmC19v+Jtpv+7jbFo2FSKDGdVNcVfzyvgZnZstrlRIAG/f04zO9cry0sKdDP9mKysTTzHhjkZEBAe4+QqKjze2eztyJoFXB7oA9wKvKaV+A77QWq9wa2RCCFEMLl68SJ8+fTAYDCxZsgSDwcCzzz7r6bBczmq18vOuE7y+bC9/nU4jPMif0d0Ug9rVJCSw8MOhDAYD/xdXhfga0Tw1dyvfbznCxkNnmdavOXHVfX8GOm9t93bkzEQuFuBX4FelVGdgCvA9ULK6XwohrkthYWGYzWbAVn0eHh7u4Yhcb8vf55i0ZA9/Jp3Dz2hgYOvqPNW1LmXCi17DUL10GPOGtOHt5ft497f93P3hOoZ1qcvQG2v7bAc3b273duTMRC7Ngf5Ab0DzTwIXQgifYzabmT17NikpKYwcORKDwcCCBQsIDw8vcQ8c+ftMOq/9vJfF248BcEvD8jx7W31ql3Xtl5QAPyMjb1G0r1OGp7/ZytRfE1m97xRT+zXzunbjgnh7u7cjZ6rQZ2GbuKWt1vqEm+MRQgi3yszMZPz48WRlZfHYY48RGhpa4oaFnU/P5p0V+/l8XRI5ZitNq0YxpnsDbqgZ49bztqpVmqVPdeSFH3awePsxuk9fzSt3NeaOZr7Twc3b270dXTWBK6UqaK2PYyt5W4EgpVS13PVa67+LIT4hhCiyw4cPc+TIEVq1akVYWBjffPMNNWvWJDTUe0tX1yIzx8zn65J4d8V+UjNNVI0J4Zlu9ekZW7HYahdKhQbw7r3N6VyvLON+3MVTc7eyUp/iZR/o4OYL7d6O8iuBzwJ6AgnYErjjb98K1HJfWEII4Rqpqak0btyYmJgYdu/eTXBwMO3atfN0WC5lsVhZtP0ob/ysOXwug1IhAYzt0YCBbap7ZL5ug8FA3/iqtKwRw1Nzt7BgyxE2HjrH9Hua0byad3Zw85V2b0dXTeBa6572l3Fa63/Nm2d/sEm+lFJG4D2gKbaZ2wZrrfc7rL8NGGd/uxkYqrWWCXiFEEVmtVrzOqRFRkYyevRoKlWqRGBgoKdDc7n1f51h0pI9bD+cQqCfkcHta/JElzpEhXr+WmuUCeO7x9oy7ddE3ks4QJ8P1jH8pro8fmMdp4esFQdfavd2lF8VelVspe4l9mRrcNhnCVDQc9TuBIK11m2UUq2xdX67w37sCGwPSOmstT6tlHoGKAOcKsrFCCFESkoKd911FxERESxcuBCAMWPGeDgq19t/8iKTl+7l1z22rkm9mlbimW7K65JPgJ+R0d3q075OWUZ8u5UpvySyet9ppt7TjMpRIZ4OD/Ctdm9H+VWhvwzcCFQCVjksN2GbWrUg7YFlAFrr9UqpeId1bYEdwBSlVC1gltZakrcQosgiIyOxWCyYTCYyMjIICfGOJOEqpy5kMX15Il9vSMZssXJDjRhe6NGAZlWjPB1avtrULs3Spzrwwvc7WLLjOLdOW8Wku5rQq2klj8bla+3ejvKrQh8EoJR6Vmv92jUcOxJIcXhvVkr5a61N2ErbNwLNsD2mdLVSap3WOvEaziOEuI6ZTCZmzpxJdnY2zZo1w2Aw8NNPP5W48dwZ2WZmrf6LD1YeIC3bTK2yYTx3a31ubljeZ4a/RYUGMqN/C+ZtPMz4Rbt48ustJNg7uIUHOTMoyrV8sd3bUX5V6I9orWcCwUqply5dr7WeUMCxUwHHsRlGe/IGOAP8ae/ljlJqFbZknm8CV0qNx95u3qVLlwJOL65VUlISCQkJng6jRJJ763rp6em8+OKLAEyePNnD0biexWplzRET8/flcD7LSkQgDGwYSKcqVvxP7WXlyr3FEocrP7vlgJdaBfLhtizmbz7M6r1HGBIbRK2o4utwl2myMmFdBpk5Vh5uHMCB7Rs4UGxnv1zuPPOFkd9XHsNVXjtrDdAL+NbeBr7DYd0moLFSqgxwHmgNfFTQAbXW44HxAAkJCdZruWBRMMeHFgjXknvrGocOHeLkyZO0bNkSgIULF1K3bl327NlTou7vqsRTTF6yh73H0wnyN/LEjbV5tFMtjwzHcsdnt3c3C1N/TeSDlQeYuCGLp7vW5bHO7u/gZrVaGfntNo6mpfNguxqM7NXIredzl/yq0D+0/3xZKRWotc5WStUBFLDUiWN/D9yslFqL7QvAg0qpEcB+rfWPSqnngZ/t236rtd5ZpCsRQlwXzp49S+PGjalUqRI7duwgMDCQ9u3bA7Bnzx4PR+cae46lMmnJHlbvsz2bu09cFUbeUo+KpUpWe36gv5Fnb61Ph7plGPHNNt78XyKr9p1mWr9mVHJjB7d5mw6zYMsRn2z3duTMVKovAg2VUs9i68y2C7gZGJ7ffvY51Idcsnivw/q5wNzCBiyEuP44DguLiYnhmWeeoVatWgQEePfEIIV1LCWDKf9LZP7mw1it0KFuGZ6/rQENK5W855A7alu7DEuf6sDzC3awbJetg9urvWPpEVvR5efSxy/w0sKdRPpou7cjZ3oN3ImtR/lTwJda62eUUhvdG5YQQticPXuWO++8k/LlyzNv3jyAvDbvkuJCZg4frvyLWb//RWaOhfoVIni+ewM61Svr6dCKTXRYIO//pwXf/JnMy4t2M/SrzSToKoy/vRFhLurglpZl4vE5m8jMsTD9Ht8Z7301ztwVo9Y6QynVExhrn6AlzM1xCSEEANHR0ZhMJkwmE9nZ2T49GUuN5xbnu758ZBATblf8X1wVr5ropLgYDAbuuaEaLWvaZnCbt+kwfyadZfo9zWlaxGFyVquVF3/YyYFTaQxqV5NujXxnvPfVOJPAlyuldgLp2KrQVwI/ujUqIcR1KycnhxkzZuDv788TTzyBwWDgl19+ISysZJcbRt5cj4c61CQ0sPiHU3mb2mXDWfBYO6b8ovlw5V/83/trGXFLPR7tWPuav9g4tns/d1tB85D5BmeeBz5KKfU2cFhrbVFKPam13loMsQkhrkNpaWlMnDiRkJAQHnnkEQIDA70+eeeYLaRk5HA+PZvz6TmcS//n9fmMbM6l55CSnsPuLWAAACAASURBVJPvMZ68qW4xResbAv2NPH9bAzrWtc3g9voyzapE2yNKC9uZryS1eztyphNbWeBNoItSyh/4TSk1RB4tKoRwlf3795OSkkJcXBxRUVEsWLCA+vXrF1hdfsUq6WW2ZUmTexQ6DrPFSmpGDuczcjiXnk1Kuu3n+dyEnHF5cj6flsOFLFPBBxfXpF2dMix7qiPPzt/O/3af4NZpq5ncuwm3NXGug1tJa/d25ExdzYfAWmAwYAQeAT7G9qQyIYQokpMnTxIbG0vNmjXZtm0b/v7+dOjQocjHTc3M4XzaPyXgvKRrT8opGf9OzufSc0jNzMHq5COVggOMRIcGUiUmlKiQAKLDAigVEkh0aABRoQFEhQbal9t+RoUG0nLir0W+rutRdFggHw6M4+sNyUz4aRePzdnMPS2r8lKvhvk2OZTEdm9HziTwWlrr3g7vX1dKDXRXQEKIks9isZCenk54eDjlypVj9OjRNGrUCD8/183EFTv+f05tF+BnICo0kPKRQagKEbakGxr4TxIODSA61J6cwwKICrEtCw4o/sd0Xs8MBgP9W1XjBnsHt7l/JrPhoK2DW5Mqpa64T0ls93bkTAK3KqWqaq2TAZRS1YD8G3OEEOIqTp48ye23307t2rWZM2cOAC+//LLLz9O1QbnLS8ShtuRcyqFkHBro5zNziQuoUy6cBY+35c2fNR+tPkjv99cw8hbFIx1qYXTo4FZS270dOZPAXwTWKaX+wDajWits1ehCCFFoZcqUwWw2YzabMZlM+PtfW6/rE6mZ+a6fdX/LazquO11Lu7y4XJC/H2N6NKRjvbKM/HYbk5fuZfLSK88Jn5ljKVHt3o4K/Eqitf4JaA58Yv/XXGud/2BGIYSwy8rK4o033uCDDz4AwGg0snLlSubOnXvNyfv3fafpPn21K8MUPqhD3bIsG96Rrg3KezoUjygwgSulooAxwGRgAjBMKVWyJuQVQrjNxYsXefXVV3njjTcwmWy9tUNDr61EZLZYmfpLIgM/+YPUTGnJExATFshH98V5OgyPcObr75fY5jDvjy3hPwjMAga4MS4hhA/bu3cvGRkZNG/enNKlS/PDDz/QuHHjay5xA5y6kMXwb7awZv8ZKkeFMGNAC5pdMjuXPO3t+nS99mFw5n9TDa2145Cx4faZ2YQQ4jJHjhyhadOmNGjQgE2bNuHn50fHjh2LdMx1B84wbO4WTl3IomuDcrzZtylRob47paoQruBMAt+llOqgtV4NoJSKBfa5NywhhC8xm81kZGQQHh5O5cqVGT16NPHx8RiNRev5a7FYeS9hP2/9kojBYGBM9wYM7lDzui1xCeHImQReH1iplNKAGdvzwM8qpQ4CVq11LXcGKITwbseOHaNnz57ExsYye/ZsAF555ZUiH/fMxSye/nYbqxJPUbFUMO/2b05c9ZgiH1eIksKZBH6726MQQviscuXKYbFY8oaGuWIylj+TzvLkV1s4nppJZ1WWt+5uRkyYVJmLq7seh+g58zCTQ8URiBDCN2RmZjJlyhTKly/P4MGD8fPzY82aNdfcs9yRxWLlo9V/8frPGqvVyuhuisc61f7XBB1CCBt5bp0QolBSU1N5/fXXqVSpEoMGDcJoNLokeZ9Pz2bkt9tYvvck5SKCeOfe5rSqVdoFEQtRMl21h4lSqp/9Z5niC0cI4Y127tzJtm3bAFuV+aJFi1i/fn2RO6nl2vL3OXq8/TvL956kfZ0yLHmqgyRvIQqQXwl8olJqPvA/oEUxxSOE8DKHDh2iefPmNG3alA0bNmA0Gos8LCyX1WrlkzVJTF66B5PFytNd6/FElzr4SZW5EAXKL4GvArIAg1LKcsk6q9ZaHsUjRAnlOCysevXqjB49mvbt27usxA2QkpHDM99t4+ddJygTHsj0e5rTro5U+AnhrKsmcK31IGCQUmqh1vqOYoxJCOFBycnJ9OzZk9atW/Phhx8CMGnSJJeeY8fhFB7/ahPJZzNoXSuGt+9pTrnIYJeeQ4iSzple6HcopW4DbrJvv0Jr/aPbIxNCeESFChWwWCxYLBasVqtLJ02xWq18sf4Qr/y0hxyLhSe71OGpm+ri71fyHvUohLsVmMCVUqOBPsAcbI8THauUaqK1nuju4IQQ7peens7rr79OjRo1eOCBBwgICGDDhg2EhLj2mUUXMnN4bsEOFm8/RkxYIFP7NaNTvbIuPYcQ1xNnhpENBFpprTMAlFIfAZsASeBClAApKSlMmTKF2rVrc//992MwGFyevHcfTWXoV5s5eDqNljWiefve5lQsJQ81FKIonEngxtzkbZcJmNwUjxCiGGzZsgV/f3+aNGlCxYoV+emnn2jRooXL5xi3Wq3M/TOZcT/uIttk4dFOtRh1iyJAqsyFKDJnEvhy+3CyT+3v7wdWuC0iIYRb7du3j/j4eFq1asWaNWswGAx06tTJ5edJyzIx5vsd/LD1KKVCAnh/QAtualDe5ecR4nrlTAIfDgwB7sM28csK4EN3BiWEcK2cnByys7MJCwujbt26jBo1iq5du7rtqV76+AUen7OJA6fSaF4tinf7t6BylFSZC+FKzvRCtwLv2/8JIXzMwYMH6dmzJ126dOGdd94B4LXXXnPb+eZtTObFhTvJzLHwUPuaPHtrfQL9pcpcCFeTudCFKOEqV66M1WrN++euUndGtpkXF+7ku02HiQj2Z/o9zenWqIJbziWEkAQuRIlz8eJFJk2aRMOGDfnPf/5DYGAgmzdvJjjYfROl7D95kaFzNqNPXKBJ5VLM6N+CaqWL/oATIcTVOZXAlVKRQCls48AB0Fr/7a6ghBDX7ty5c0ybNo3GjRszYMAADAaDW5P3D1uO8ML3O0jPNnN/m+q80KMBQf4y07IQ7ubMRC4vAM8BZxwWW4Fa7gpKCFE4GzduJDQ0lIYNG1K1alWWLl1Ky5Yt3VZdDpCZY+blRbv5esPfhAf5M6N/C3rEVnTb+YQQ/+ZMCfwhoLbW+pS7gxFCFN6uXbu44YYb6NixI7/99pvbhoU5Ong6jcfnbGbPsVQaVoxkxoAW1CwT5tZzCiH+zZkE/jdw1t2BCCGcl52dTU5ODmFhYTRq1IiRI0fSo0cPt5a4cy3efoxn52/nYpaJ/q2q8VLPhgQHSJW5EMXNmQS+D/hdKfUbtlnYANBaT3BbVEKIq0pMTKRnz5706tWLKVOmAPDGG2+4/bxZJjOTFu/hs3WHCA30Y/o9zbijWWW3n1cIcWXOJPAj9n/g0IlNCOEZ1apVyytpu3NYmKPks+kM/Woz2w+noMpHMGNAC+qUC3f7eYUQV+fMRC4vK6XKAq3s26/TWp9we2RCCABSU1OZMGECLVu2pF+/fgQHB7Nt2za39Syv8dziq67rG1eFCXc0JiRQqsyF8LQCp0dSSnUDtgIPYpsHfbtSqqe7AxNC2Jw5c4Z3332XGTNm5C1z57Cw/LzRt6kkbyG8hDNV6BOB9lrrgwBKqVrAAuAndwYmxPVs3bp1REdHU79+fWrWrMmyZcto3bq1p8MSQngRZyYoDshN3gBa67+c3E8IcQ22bNlC27ZtGTZsWN6yzp07e6zULYTwTk4NI1NKDQc+tr8fDBxyX0hCXH8yMzOxWCyEhobSrFkzRo4cyV133VXscfy863ixn1MIcW2cKUk/BLQB/gKS7K8fcWNMQlxXdu/eTaNGjfjvf/8LgMFg4M0336Rdu3bFFoPVauWDlQcY8uWmYjunEKJonOmFfhLoV9gDK6WMwHtAUyALGKy13n+FbRYDC7XWHxT2HEKUBDVq1CiWoWBXk22yMOb7HczbdJiKpYI5lpJZ8E5CCI+7agJXSv2kte6plDqIbe7zf9FaFzQX+p1AsNa6jVKqNTAFuOOSbV4BYgoZsxA+7dy5c4wbN47OnTvTu3dvQkND2bVrF0FBQcUfS1o2j365iQ0HzxJbpRSz7ounXKS0tQvhC/IrgT9s/9n5Go/dHlgGoLVer5SKd1yplOoDWICl13h8IXzS6dOn+eCDD9izZw+9e/cG8EjyPnDqIoM+/ZNDZ9Lp3qQCU/o2kyFiQviQqyZwrfUx+8u3tNb/57hOKbUcuKmAY0cCKQ7vzUopf621SSnVGOgP9AFeKnzYQviWlStXUqlSJQDq1q3L//73P9q0aeOxeNbsP81jX24iNdPEEzfWYcTN9TAaZaJFIXxJflXoC4BmQGWl1F8OqwKwPeCkIKlAhMN7o9baZH99H1AZWAHUALKVUkla62X5HVApNR4YB9ClSxcnQhDXIikpiYSEBE+HUWLs3r2boUOH0qpVK4YMGZJ3b9etW+eReBKSc/hidzYG4OEmgcQHHWPVqmMF7ucL5LPrPnJv3atz586F3ie/KvQHsLVPTwee5J950E2AM1OprgF6Ad/a28B35K7QWj+T+9qelI8XlLzt+40HxgMkJCRYr+WCRcESEhKu6cMk/pGeno7BYCAkJIROnTqxf/9+7rnnHtLT0z12b80WK68u2cOnuw4SExbIhwPjaFmjZHVBkc+u+8i99T5XHUamtU7VWicBjwNDtdaH7Nu/inMdz74HMpVSa4GpwNNKqRFKqduLHrYQ3mvbtm3Ur1+fyZMnA7ZhYW+99RY33HCDx2JKyzLx6BcbmfX7QeqUC+eHx9uVuOQtxPXGmYlcvgTm2l8fBVbbl92S305aawsw5JLFe6+w3XgnYhDCZ9SuXRt/f3+MRu+YsPDI+QwGf7aRPcdS6VC3DO/2b0GpkABPhyWEKCJnEnhprfWHAFrrLOAjpdRj7g1LCN9x+vRpXnzxRbp3706vXr0IDw9nz549HulZfqmtyecZ/NlGTl/M4j+tqzG+VyP8/bzji4UQomic+Z+crpS6LfeNUqorkOa+kITwLadOnWLWrFnMnDkzb5k3JO+fth+l34frOJuWxfheDfnvHY0leQtRgjhTAh8CfKmU+sL+PhkY6L6QhPB+y5cvp1atWtSsWZMGDRqwfPlyjw4Lc2S1Wnl3xX6m/JJIeJA/H/wnjhvrl/N0WEIIF3NmKtWtQGOlVGkgR2ud6v6whPBeq1atomvXrtx1110sWLAAgI4dO3o4Kpssk5nn5u/g+y1HqBwVwscPxFO/QqSnwxJCuEGBCdw+BOx5IBwwKKX8gOpa6xpujk0Ir3Hx4kX8/f0JDg6mQ4cODB8+nPvuu8/TYf3LmYtZPPrFJjYeOkfzalHMHBhP2QjPV+ULIdzDmQaxT4AfsCX7GcBhbEPEhLgubNq0CaUUb775JmAbFjZ16lSaN2/u4cj+kXjiAne+t4aNh85xe9NKfP1wa0neQpRwzrSBZ2mtZyulagDnsM2itiP/XYQoOerUqUNAQAB+ft45T/jKxFM8MWczF7JMPN21HsNuquPRp5sJIYqHMyXwTKVUDKCB1lprM+Cdf8mEcIETJ07w0EMPsWyZbXLAUqVKkZiYyPPPP+/hyC73+bokHpy9gSyzhbfvbc5TXetK8hbiOuFMCXwK8A3QG9iglBoAbHRrVEJ40IkTJ/j0009JSUnh1ltvBSAwMNDDUf2byWzhvz/t5rN1hygTHsjM++JpUS3a02EJIYqRMwk8A7hFa221PxK0HrDNvWEJUbyWLl1Ko0aNqFatGrGxsaxYsYJ27dp5OqwrSs3M4cmvtrAy8RSqfAQfPxBPlehQT4clhChmziTw17XWiwG01mnAFveGJETx+uWXX+jevTv9+vVj7lzbrMGdOnXycFRXlnw2nYc++5PEExe5UZXl7XubExEs06IKcT1yJoEfUEp9AvyBrTQOgNb6c7dFJYSbpaamEhQURFBQEDfddBNPPfUUgwcP9nRY+dp06CyPfL6JM2nZDGpXkzE9GuAnz/AW4rrlTCe2M9geJdoauNH+r7MbYxLCrdavX0+9evWYPn06AEajkWnTptG4cWMPR3Z1P2w5wr0z/+B8Rg6v3NmYl3o1lOQtxHXuqiVwpVRlrfURrfWDxRmQEO6mlCI4OBh/f2cqoDzLYrEy7ddE3l6xn4hgf94b0IIOdct6OiwhhBfIrwS+KPeFUmpkMcQihFscPXqUgQMHsnz5cgCio6NJTExkxIgRHo4sf5k5Zp6cu4W3V+ynWkwo3z/eVpK3ECJPfkUQx/q5AdiGkwnhc44dO8acOXOwWCzcdNNNgPcNC7vUyQuZPPz5JrYln6dljWg+HBhPTJh3xyyEKF75JXCrw2tpbBM+w2q18uOPPxIfH0/lypWJi4sjISHBa4eFXWrPsVQe+vRPjqZk0rtFZV7t3YQgf5k7SQjxb84+HNha8CZCeIfFixdz55138sILL+Qt69ixo9dOhepo+Z4T9Hl/LUdTMhndTTGlb1NJ3kKIK8qvBN5IKfWX/XVlh9cGwKq1ruXe0IRw3rlz5wgPDycgIIDbbruNYcOGMWTIEE+H5TSr1crHvx9k4pI9BPkbeX9AC25rUtHTYQkhvFh+CbxesUUhRBGsXr2a3r17M2bMGIYPH46fn1/eEDFfkGO28NLCXXy94W/KRQQx6/54YqtEeTosIYSXu2oC11ofKs5AhLhWDRo0ICQkhIAA35uRLCU9h8e/2sSa/WdoWDGSjx+Ip2KpEE+HJYTwAc62gQvhNZKTk7nnnntYvXo1AGXKlOHAgQMMHTrUw5EVTtLpNO56fw1r9p+ha4PyzBvSRpK3EMJp3j+ThRCXOHLkCN988w1hYWF06NABwOdK33/8dYZHv9zE+fQcHu1Yi2durS8zqwkhCkUSuPB6VquV+fPn06FDB8qXL0/r1q1ZtWqVzwwLq/Hc4n8vWPbP+9f+rwn9WlYr5oiEECWBVKELr7dgwQL69u3L2LFj85Z16NABo9H3P76SvIUQ18r3/wKKEunMmTOYTCYA7rjjDoYNG8azzz7r4aiEEMJ7SAIXXmfFihXUrVuXmTNnAuDv78/06dOpU6eOhyNz3vGUTGavOcjdH6zzdChCiBJK2sCF12nUqFHepCy+5FhKBkt3HGfJjmNsPHQOAIP0SxNCuIkkcOFxSUlJjBw5klGjRtGmTRvKly/PgQMHfCKBHz2fwZIdx1iy4xib/z4P2JJ2q5ox9IityK2NKnDDpOUejlIIURJJAhcel5yczIIFCyhfvjxt2rQBvHtY2JHzGSzdcYzFO46xxZ60jQZoU6s03WMr0q1RecpFBHs4SiFESScJXBQ7q9XK119/zS233EKZMmXo0KEDq1ev9uphYcln01m68xiLdxxnW/I/Sbtt7dJ0b1KRbo0qUDYi6Ir7Jk3ukfc6ISGBzp07F0fIQogSThK4KHZff/01AwYMYOjQobz77rsAtG/f3sNRXS75bHpe9fi2wykA+BkNtK9Txp60y1M6/MpJWwgh3E0SuCgWp06dIiYmBj8/P/r27cuGDRsYPny4p8O6zN9n0llsT9o7jvyTtDvULZNX0o4JC/RwlEIIIQlcFINly5bRr18/pkyZwuDBgwkICGDatGmeDitP0uk0luy0Je2dR1IB8Dca6FivLD2aVODmhpK0hRDeRxK4cLvY2FiioqK8qmPawdNpLNlxjMXbj7H72D9Ju1O9svRoUpGbG5YnWpK2EMKLSQIXLrd//35GjhzJSy+9RFxcHJUqVeLAgQP4+3v243bg1EWWbLf1Ht97/AIAAX4GblRl6W5P2lGhkrSFEL5BErhwuUOHDvHjjz9Su3Zt4uLiADyWvPefvJjXEc0xaXepX86WtBuUp1So99QMCCGEsySBiyKzWCx8+eWX9OrVi+joaG666SZ+//132rZt67ZzXvaELwe/PN0xryNa4omLAAT6GenawJa0b2pQnlIhkrSFEL5NErgoss8++4xBgwYxfPhwpk6dCuDRMd03T10FQKC/ka4NytMjtgI3NShPZLAkbSFEySEJXFyTEydOULZsWYxGI/3792f79u2MGDHC02EBcEvD8vSIrUiX+uWIkKQthCih5GlkotBy27e//PJLAIKCgpg6dSpVq1Z1+7kvZpn49s/kfLeZeV88dzSrLMlbCFGiSQlcFFqzZs0oXbp0sQ0Ls1qtbDh4lnmbDrNkxzHSs83Fcl4hhPBmksBFgbTWPP3000yePJnY2FiqVatWLMPCjpzPYMGmw3y3+TCHzqQDUCU6hEc6VmHar/vcem4hhPB2bvsLrJQyAu8BTYEsYLDWer/D+qeBe+xvl2itX3ZXLKJokpKSWLp0Kc2aNSM2NhZw37CwzBwzP+86zryNh1lz4DRWKwQHGOndvDJ94qvQumZpjEaDJHAhxHXPnUWoO4FgrXUbpVRrYApwB4BSqhYwAGgFWIHVSqnvtdbb3RiPcJLZbOaTTz6hT58+REZG0q1bN9auXUvr1q3dcj6r1crW5PPM23SYRduOciHTBEBc9Wj6xlWhR2zFy9qzHZ/wJYQQ1yN3JvD2wDIArfV6pVS8w7pk4FattRlAKRUAZLoxFlEIixcvZurUqSQmJjJ58mSAvOd0u9LJC5l8v/kI3206zL6TtvHa5SOD+E/r6vSJq0LtsuEuP6cQQpQU7kzgkUCKw3uzUspfa23SWucAp5VSBuANYIvWOrGgAyqlxgPjALp06eKGkK9fZ86cISYmBoPBQIMGDejTpw/x8fEkJCS49Dwmi5WtJ82sPmJix2kzFiv4G6BlBT86VPanUWk//IzHSd51nPz7mvumpKQkl99T8Q+5v+4j99a9OnfuXOh93JnAU4EIh/dGrbUp941SKhj4BLgAPO7MAbXW44HxAAkJCdZruWBxue+++4777ruP2bNn069fPwAefvhhl55j99FU5m1KZuHWo5xNywagceVI+sZV5famla6bB4ckJCRc039U4Ry5v+4j99b7uDOBrwF6Ad/a28B35K6wl7wXAiu01q+5MQbhhBYtWlCuXDmXd0w7l5bNwq1HmLfpMLuO2p74FRMWyKB2NekbX4UGFSNdej4hhLieuDOBfw/crJRaCxiAB5VSI4D9gB/QCQhSSt1m3/55rfU6N8Yj7Hbu3MmIESOYNm0aDRs2pFatWuzfv98lCdxktrB632nmbUrm190nyTZb8DMa6NqgHH3iqtKlfjkC/WX+ICGEKCq3JXCttQUYcsnivQ6vg911bpG/gwcP8ssvv/Ddd9/x0ksvAUUfFnbg1EXmbTzMgs2HOXkhC4C65cLpG1+FO5tXplyE/LqFEMKVZCKX64DJZOKTTz6hf//+hIeH07NnT9atW1fkYWEXMnP4afsx5m1MZvPf5wGICPZnQKtq9I2vStMqpTAYDK64BCGEEJeQBH4dmDFjBsOHD+fw4cNMmDABg8FwzcnbYrGy/q8zzNt0mKU7j5GZY8FggA51y9AnrgrdGlUgOMDPxVcghBDiUpLAS6ijR49SsWJFDAYDDz/8MMnJyTzxxBPXfLzks+l8t+kw8zcf5vC5DACqlw6lb1wVereoQqWoEFeFLoQQwgmSwEugr776ikGDBvHVV1/Ru3dvQkNDefPNN/Pdp8Zzi/+9YNk/79vUKs26v84AEBroR9+4KvSNr0rLGtFSRS6EEB4iCbwEio+Pp1KlSi57Wti6v85wQ80Y+sZVoXuTioQFycdGCCE8Tf4SlwDbtm1jxIgRfPDBB9StW5d69eqxb98+/Pwub4u2Wq2kZORw+FwGR85ncMT+Mz8rR3emeukwd4UvhBDiGkgCLwH27dvHihUrWLBgAaNHP8Opi1mXJOj0vER95FwGaYV8nrYkbyGE8D6SwH1MtslC8pkLfPTVfJq06sSpdDNHrHXp8ebPLLEE8PGLy8g2W664b0SwP1VjQqkSHULlqBAqR4dQOSqUytEh3DljTTFfiRBCiKKQBO5l0rNNHDmXwWGH6m3HnycuZGK1ApRn7o97/7VvmXAjDSpFUiUvOTsk6ugQIoNd0yYuhBDC8ySBFyOr1cr59ByOnM+4vIrb/vpces4V9/UzQIVSIbSsEUP5cH/2b/+T/+vWGVW1LJWjQqgUFSLjr4UQ4joiCdyFLBYrJy9kceR8+mWdxHJ/pl+l/Tk4wEilqBAaVy51WRX32l8W8cyTjzJ9/nf07Gl/jOqAli6NPWlyj7zX8tQhIYTwfpLAucIYaAeOiS3bZOFYSsZVq7iPpWSQY7Ze8TiRwf5ULx1G5aiQSxK07WfpsMCrjqkOaxdPtapVXP60MCGEEL5LMkIBnvx6C0fO2aq4T17Isrc/X65cRBCNK5f6d2J2eB1RiPbnjRs3MmrUKGbPnk3NmjVp1KgRWusrDgsTQghxfZIEXoBF247ibzRQMSqYVjVj8nptO3YUqxgVTJC/65JrYmIiK1euZOHChQwfPhxAkrcQQoh/kQRegHXPd6FcRDB+RvdNGZqVlcVHH33E4MGDCQ4O5t5770UpRVxcnNvOKYTwXX/88QfDhw+nTp06gO1vSK9evRg4cKBT+//6669MmzaNu+++m/vuu8+pfdLS0li0aBG9evXiueeeo3v37nTs2PGar+FS7dq1Y82aNUycOJEHH3yQ+fPnU6ZMGe69995CH+uXX34hNjaW8uXLF7jtqlWrWLJkCZMnT76WsD3K6OkAvF3FUiFuTd4Ar7/+Ok8++STvvPMOAAaDQZK3ECJfrVu35osvvuCLL77gyy+/ZPbs2aSmpjq172+//caIESOcTt4Ahw8fZsWKFdcartPGjBlDpUqVinSMzz//nIsXL7ooIu8lJXAPSU5OpmrVqgAMGzaM9PR0Bg8e7OGohBC+6OLFixiNRvz8/NBa88orrwAQFRXFpEmT2L17N2+++SYBAQH07duXhIQEtm/fTnR0NMePH+fTTz/FaDQSFxfHqFGjOHPmDM899xwXLlzAarXy2muvsXTpUk6cOME333yTd96RI0fSq1cvOnfuzIEDB3jttdeYOXNm3vqkpCTGjh1LTk4OwcHBTJ06ldOnTzN58mQsFgupqamMHTuWFi1a5O0zcOBAxo8fD9hqCpYuXUpmZiZjx44lNjaWG7DmWwAAEddJREFUG2+8kVq1alGrVi369u172bFS/7+9O4+qus4bOP6+KAiJgttkgQ2bfeXYKHpswHWUNLf0Gc0FMRjINCunpBRQcJ1wCXxq7CSPyzCUI4kzD5lLj0vTZIPKwe0kJX4JcEExNQcXnGST549Ld0DhBsgFLn5e59yj9/e793s/vw8cPve3fT83b5KZmUlERARJSUkkJyeza9cuDAYDY8aMITg4mJycHBYuXIiDgwMODg44OTk1zg+qgckeeBNYv349Hh4e7Nu3DwAnJydWrlxJhw4dmjgyIUR9uLm5ERAQYHqekpKCm5tblWIXFBSEm5sbxcXFAPzwww+4ubnx2muvmV6zceNG3NzcavWZaWlpBAUFERwczPz581m0aBFt27Zl0aJFLFmyhM2bNzNkyBA2bdoEGA+zJyUlMWHCBAYPHsz8+fNxd3fn/fffJzExkY8//pjLly9z8OBB4uPj8ff3Z+vWrcydO5eTJ08yevRo/Pz8mDp1qimGyZMn88knnwDwt7/9jUmTJlWJcfXq1cyaNYvk5GSmTp3KqVOnyM7OJiIigsTEREJDQ0lJSalxG11cXPjoo4+IiYlhyZIlAFy6dIm4uDiioqKqHWvo0KF4e3uzevVqzp8/z2effUZSUhJJSUl8/vnn5Obm8sc//pHXX3+dxMRE+vTpU6t8N0eyB07VW8Uag6+vL+7u7nJhmhCi3vz8/Hj33XfvW56Tk8OyZcsAKCkpwd3dHcD0b2Xnz5/nX//6F7NmzQKM57nz8vI4c+aMqRj3798fMO543MvX15eYmBiuXbvGwYMHefPNN6usP3PmjKlAjhkzBjDeZbNu3Trs7e25ffs2jo6ONW7j008b57vo3r07V69eBaBDhw6mnZ1f/OIXZsfKysoiPz+fkJAQAG7cuMH58+f57rvv6NWrFwB9+/YlNze3xhiaMyngjeDw4cOEh4eTlJREt27d8PHxITMzUwq4EC3E2bNnqzyfOHEiEydOrLJs8+bNVZ537tz5vvfNnDmTmTNnPlAs7u7urF69mscff5xjx46ZCp+Nzf0HXF1dXXnsscdISEjA1taWlJQUvL29yc3NJSMjgx49enDkyBG+/PJLnJycuHu3ap8Fg8HAuHHjiImJYeDAgfe1MPb09CQjI4MBAwawY8cObty4QUpKCnFxcXh6erJ27VouXrxY47acPHmScePGobU2nRevvB0xMTHVjmUwGCgvL8fDwwMvLy82bdqEwWAgMTGRJ598Eg8PD06cOMGQIUP45ptv6pfoZkAKeCPQWpOamsru3buZPXs2ILeFCSEsY+nSpURERFBWZpz1MSYmhitXrlT72o4dOxISEkJQUBBlZWW4uLgwevRoZs+ezcKFC9mxYwcAK1as4OjRo2RlZZGYmFhljIkTJzJ06FA+/fTT+8YPDw9n8eLFxMfHY29vT2xsLKWlpbz66qt06tSJrl27UlBQUOO2XLhwgeDgYIqLi1m+fPl968ePH1/tWH369CE8PJyEhAT69+/PtGnTKC4uNl2ZvmTJEsLCwvjTn/5Ex44dadOmTa1y29wYymuamaSZ+/LLL8ub63SfP/74Ixs2bOCVV17Bzs6Ou3fv8vXXX1vNuRaZStVyJLeWJfm1nJpye/nyZcLDw/nwww8bP6iWpc63O8lFbBawYsUK5s6dS3x8PGA85GMtxVsIIWpr7969vPTSS7z11ltNHcpDSQ6hN5Dz58/zxBNPABAWFkZ5eTmhoaFNHJUQQljOyJEjGTlyZFOH8dCSPfAGsHbtWjw8PPjHP/4BGM8rvf3227Rv376JIxNCCNFSSQFvAAMGDKB79+5yYZoQQohGIwW8Hr766isGDhxIfn4+AP369ePbb79t0HmBhRBCCHOkgNfD6dOnOXz4MHv37jUtq+4eSyGEEMJS5CK2Wrh9+zYbNmxgzpw52NraMmPGDHx9fendu3dThyaEeAjd243s9u3buLq6EhcXh52dXb3GDAsLIyAgAF9f33q9/8KFC4wfP56ePXualvn6+jJnzpx6jVeT/Px8Tp8+jb+/PwDJycns2LEDGxsbSkpKCAsLw9fXt8E6pqWkpODk5MQzzzxDeHg4Z8+eZcKECdjY2FSZVrYpSAGvhWXLlhEbG4uDgwOzZ8+mVatWUryFELXiFrm7xnUPMo3zvVOpvvXWW3zxxReMGjWq3mM+KC8vr/tmnGtoaWlp5Obm4u/vz+7duzl48CCJiYnY2tqSl5fHCy+8YJqfvSFUnlEvNTWVQ4cONdjYD0oKeA3Onj1raiowf/58HBwceOGFF5o2KCGEqEZxcTFXrlzBycmJqKgovv/+ewoKChgyZAhz584lMjISOzs7Ll68yJUrV1i1ahU9e/Zky5Yt/PWvf6VLly5cu3YNMM6fvnDhQvLy8igrKyM0NJQxY8awZs0aUlNT+e6773jkkUfo168fqamp3Lx5k4SEBLPxrVq1imPHjgHw3HPP8bvf/Y7IyEiuX7/O9evXWb9+PZs2beLIkSOUl5cTEhLC6NGj2bJlC9u3b8fGxoa+ffsyb948NmzYwJ07d+jTpw9bt25lwYIFpilcu3Xrxvbt26s0hiosLCQqKopbt25RUFDA5MmTCQwMvG/siIgI9u3bx8aNG2ndujUuLi688847fPDBB3Tu3BmtNTdv3uSVV15hxIgR5ObmMm/ePDZv3nxft7N7t81S3c6kgFcjNjaWyMhIDhw4wKBBg+jSpYupOYAQQlS24rNMdp+8VK/3DlxVfX/tsb0eY+EYb7Pv/akb2bVr17CxsWHKlCmmXguTJ0+mqKjIVMABHn/8cZYvX862bdtITk5m/vz5fPTRR+zcuRODwWDa00xOTqZDhw7ExsZSWFjIxIkT8fPzA6BXr15ER0czY8YM7O3t+fOf/0xERARHjhyhR48eZGdnExQUZIoxLi6OU6dOceHCBbZt20ZpaSmBgYGm8fz8/AgJCeHAgQNcuHCBrVu3UlRUxJQpUxg4cCApKSksWrQIHx8fkpKSKC8vZ9asWeTm5vLMM8/wzjvvmNoy/+Tero7nzp1j7NixPPvss1y+fJmgoCACAwPvG7u0tJRdu3YREhLC2LFj2b59e5We4kuXLmX//v3Ex8ebOqhlZ2ebup0ZDAZCQkIYNGhQlW2zJCng1Rg0aBDe3t5yYZoQotn66RB6QUEBL774Iq6urjg7O5ORkUFaWhqOjo6m1qUA3t7GLwRdu3bl+PHj5Obm4uXlZTpn/lN3rpycHAYMGACAo6Mjnp6e5OXlAZjOb7dv3950/r19+/YUFRUB1R9C37lzJ/369cNgMGBra0vv3r3JyckB/tMhLSsri2+//dZU/EtLS8nPz2flypUkJCQQFxeHj48P90797eLiwqVLl2jXrp1pWWpqKkop0/POnTvz4Ycfsm/fPhwdHSktLQWoduwFCxawfv16Pv74Yzw8PBg+fLjZn0FN3c4qb5slSQEH/v73vxMdHc327dt59NFH6d+/PydPnpQCLoT4WQvHeJvdWzZ3DvxgpP8Df/5Pe8vBwcEEBgbSrl07li9fzrlz59i2bZup6BkMVafa7tatG9nZ2dy5cwdbW1syMzMZP348np6eHD16lBEjRlBYWEhWVhaurq71js/T05OUlBRCQkIoKSnhxIkTTJgwoUpMHh4e+Pr68oc//IG7d++ybt06XF1dee+991i2bBlt2rRhxowZnDhxAhsbG1NXtOeff55169YRFxdH69atOXPmDFFRUVV6jCckJODj40NgYCBpaWkcOHAAgG3btt039qFDh/j9739Pp06dWLx4Mfv37ze7bTV1O9uzZ899+bYEKeBAZmYm6enpfP7550yfPh2Q28KEENbDy8uLoKAgMjMzOXPmDMeOHcPBwYFf/vKXZjuRvfHGGwQEBNCxY0ccHBwAmDJlCosWLWLatGkUFRUxZ84cOnXqVO/Yhg0bRnp6OlOnTqWkpIRRo0ZVuVIdwN/fn/T0dAIDA/n3v//N8OHDcXR0RCnFpEmT6NChA48++ii9e/fG0dGR+Ph4evbsydixY7l69SqBgYHY2tpSVlZGbGxslXiHDRvG0qVL2blzJ87OzrRq1Yri4uJqxy4sLCQ0NBRnZ2fatm3L0KFD+ctf/lLjtvXo0aPabmeN5aHsRnbr1i02btzIG2+8QatWrSgtLeX06dM89dRTDRuklZKOTpYjubWs5phfS12F3tiaY25bmDrvsj+Ue+DR0dGsXbsWZ2dnXnzxRVq3bi3FWwhhEdZUpIV1eWgKeOXbwhYsWECXLl0ICAho2qCEEEKIenooTvTGxMTg6elJeno6YLwKMzo6mkceeaSJIxNCCCHq56Eo4IMHD+ZXv/pVo1wVKIQQQjSGFlnA9+zZg5+fn2lmoSFDhnD8+HGefvrpJo5MCCGEaBgtsoCfOnWKo0eP8sUX/5nlSG4LE0II0ZK0iKp2/fp11qxZY7q5f86cOWRkZDB58uQmjkwIIYSwDItdha6UsgHWAb2BIuAlrXV2pfUzgZeBUuBtrfWu+n5WZGQk69evp2vXrkyfPh07OzvTtIFCCCFES2TJ28h+C9hrrfsrpfyANcB/ASilugKvA/0AeyBVKbVfa11U28Hz8/NN/4+OjsbNzY3nn3++IeMXQgghmi1LHkIfBOwB0FqnYSzWP/k1cFBrXaS1vgFkA73qMnhQUBDHjx8HwNXVlcjISOzt7RskcCGEEKK5s2QBbw/cqPS8TCnVuoZ1t4CfbZiqlFqqlCpXSpV7eXkd6Nu3rwHj9HPyaMDHyy+/vKypY2ipD8mt5NdaH5Jbyz6UUkupI0sW8JtAu0rPbbTWpTWsawdc/7kBtdZLtdYGrbUB+E2DRSrutaSpA2jBJLeWJfm1HMmtZdU5v5Ys4AeBMQAV58AzKq1LBwYrpeyVUk6AN/CNBWMRQgghWhRLXsT2CTBCKXUI4yGCUKXUm0C21nqHUmot8E+MXyKitNZ3LBiLEEII0aJYrIBrre8Cs+9ZfLrS+o3Axgf4iGUP8F5hnuTWciS3liX5tRzJrWXVOb9W2w9cCCGEeJi1iJnYhBBCiIeNFHAhhBDCCkkBF0IIIayQFHAhhBDCCkkBF0IIIayQJe8Df2CN2dHsYVOL3IYBARVPP9Nayy0kdfBz+a30mt3Ap1rr/2n8KK1TLX53R/OfWa2OA69preV2m1qqRX7nAdOAu8AKrfUnTRKoFVNK+QKrtdZD71k+DliMsaYlVNxuXaPmvgdu6mgGRGLsaAZU6Wg2EBgJrFRKtWmSKK2Tudx6ANOBAUB/4FmlVJ2azYia81vJ20DHRo2qZTD3u9sOiAWe01r7AWeBzk0RpBUzl19njH93+wPPAu81SYRWTCkVDmzC2Imz8nJb4F2Mef0NMKuiztWouRdwi3Y0e8iZy20eMEprXVYxIY8tIDPl1Y25/KKUmoRxD+b/Gj80q2cutwMwTtu8Rin1T+Cy1vpq44do1czl9zZwDmhb8bjb6NFZvxxgYjXLvTHOVFqgtS4GUoHB5gZq7gW8wTuaCZMac6u1LtFa/6CUMiil4oATWuusJonSetWYX6XUU0AgxkNlou7M/V3oDAwDIoDRwFyl1JONHJ+1M5dfMH7BP4Xx9MTaxgysJdBa/y9QUs2qOte05l7AG7yjmTAxl1uUUvbAlorXvNrIsbUE5vIbDLgAXwAhwJtKqVGNG55VM5fba8ARrfX3WutC4CvAp7EDtHLm8jsaeAxwB54AfquU+nUjx9dS1bmmNfcCLh3NLKfG3CqlDMCnwNda65e11mVNE6JVqzG/WutwrbVvxQUsicB/a633NEWQVsrc34VjwFNKqc4Ve41+GPcWRe2Zy28B8CNQVNGA6jrg3OgRtkyZQHelVEellB0wBDhs7g3N+ip0pKOZJdWYW6AVxoso2lRc0QuwQGtt9pdJVGH2d7dpQ7N6P/d3YQGwt+K127TW8sW+bn4uv8OBNKXUXYznafc3YaxWTykVCDhqrTdU5HkvxpqWoLW+aO690sxECCGEsELN/RC6EEIIIaohBVwIIYSwQlLAhRBCCCskBVwIIYSwQlLAhRBCCCskBVwIIYSwQlLAhRBCCCskBVwIIYSwQv8PlNljNCZDtLMAAAAASUVORK5CYII=\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "plot_model(rf, plot='calibration')" - ] - }, - { - "cell_type": "code", - "execution_count": 31, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.81890.77080.27200.75000.39920.3190
10.81700.77660.28330.71940.40650.3217
20.81640.80620.26910.73080.39340.3114
30.81270.74480.27760.69010.39600.3082
40.81270.75800.27760.69010.39600.3082
50.81520.78300.27200.71640.39430.3103
60.81520.74950.28900.69860.40880.3209
70.82020.78820.26350.77500.39320.3165
80.81520.75900.28330.70420.40400.3174
90.81130.77010.27200.68570.38950.3017
Mean0.81550.77060.27590.71600.39810.3135
SD0.00270.01780.00730.02750.00610.0062
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8189 0.7708 0.2720 0.7500 0.3992 0.3190\n", - "1 0.8170 0.7766 0.2833 0.7194 0.4065 0.3217\n", - "2 0.8164 0.8062 0.2691 0.7308 0.3934 0.3114\n", - "3 0.8127 0.7448 0.2776 0.6901 0.3960 0.3082\n", - "4 0.8127 0.7580 0.2776 0.6901 0.3960 0.3082\n", - "5 0.8152 0.7830 0.2720 0.7164 0.3943 0.3103\n", - "6 0.8152 0.7495 0.2890 0.6986 0.4088 0.3209\n", - "7 0.8202 0.7882 0.2635 0.7750 0.3932 0.3165\n", - "8 0.8152 0.7590 0.2833 0.7042 0.4040 0.3174\n", - "9 0.8113 0.7701 0.2720 0.6857 0.3895 0.3017\n", - "Mean 0.8155 0.7706 0.2759 0.7160 0.3981 0.3135\n", - "SD 0.0027 0.0178 0.0073 0.0275 0.0061 0.0062" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "calibrated_rf = calibrate_model(rf)" - ] - }, - { - "cell_type": "code", - "execution_count": 32, - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAEuCAYAAABxmEC5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3hU1dbA4d9MeoVA6C2hbXoNvYpYQRFEUVBAimJDpKgX8YIoiihWRAUELLQrYkGE7yoYRXoooe/QEgJESiAJ6Zny/TETbkRIJmUyKet9Hp7MzDlnnzUnQ9bsfXYxWK1WhBBCCFG6GF0dgBBCCCHyTxK4EEIIUQpJAhdCCCFKIUngQgghRCkkCVwIIYQohSSBCyGEEKWQJHBRaiml3JRSE5VSEUqpfUqpw0qpt5RSXg4ca1VKBSul7lVKfWh/LVwpNbiQMYUqpb61P66plNpamPIcPGdvpdRBB/b7t1JqQCHP1UYptSSfx4QopZLtj8cppV7KY/+bvh+l1Eyl1HD74xv9DvsppWbmJz5XUEq1V0otcHUconRzd3UAQhTCJ0AQcKvWOlEp5QcsAxYBjzpSgNb6R+DHIoypHqDsZZ8DuhZh2YXVBzhc0IOVUkbgc+Degpahtf60oMfaj//3DV7L+TvsAFQqzDmKg9Z6t1LKXSnVX2v9k6vjEaWTJHBRKimlQoBhQA2tdRKA1jpFKTUO6GbfpzHwMRAA1AD2AUO01uk5yhkJDNZa97e/NNBeQ/QFlmmtZ9nPtRk4AoQAvYDHgAGAD+AHTMaWRBYBtZRS/wc8ARzUWvsrpTyAd4FbATOwA3hea31VKRUNLLVvqwt8qbV+5QbvORpYAdwGVATmaq0/uW6fCvb33AawAuuBqfZYwoC3lVJm4KI9Hjf7fm9qrb/N47I/CJzSWp/NEc8OoJX9HDuBefb34AGs1Fq/cV18M4BgrfUzSqn+9uM8garAFznet79SajXQEEgAHtdaRymlltqv6Ts5yhwJDAZeA8YBbkqpRKAj8B+t9UL7ftOAylrr56+LqRPwIbbfYyYwWWu9SSllBaporS/Z97MCVYAWwAdACuAPHAIitNZz7fs9CfTWWg9RSt0DTLO/x1R72dvsp16A7UuoJHBRINKELkqr9sCh7OSdTWv9V45ENBZbUuiMLRGEAv3yKDcQ6Gz/94hS6i7767WB17TWjbH9Me6L7Y90K+BlYKbW2gyMAU5ore+4rtxpQE2gtf2fEXg7x3Z/rXUPbDX2yUqp0JvEVwlbLbM3MFMp1fK67R8C8UBLbAm7Nbak8TEQAUzRWn8HvAq8q7VuD4zCVjvPy2D+mWwOaq2b2sv8ClhsL7Mj0Fcp9eCNClJKGYBJwAitdRi26/0vpVSwfZc69vjaAMvtZedKa70D+BRYpbV+GdsXmbH28xmB0fbtOePwAL7H9vtrYd//A/v+uWkBPGz//S8ARubYNhJYqJRqBLwB3K21bgs8DqyxtxShtd4ONMjldy1EriSBi9LKQt6f3xeBi0qpF7DVdGpiqzHlZpHW2mT/YrAaW20XwARsA9BaxwDDgWFKqdnYan15lXsX8KnWOktrbQE+sr+W7Qd72WeBC9y8GfhjrbVVa30G2ADcfoPzzLPvk4EtYd11fSHAf4CPlVLLsH0ZmppH/ABNgOPXvbYZwJ6UegGvKaX2Adux1cTb3KggrbUVuAdor5Sajq01wICtFgywX2ud3X9gKRBmb13Ij7VANaVUa+AObK0H+rp9WgJmrfU6e1y7tdYt7b+j3MTaPwcA4YC3UipMKdUMWy19I7bPTg1go/2aLMP2uW2Yo5xT2G+5CJFfksBFabUDaKqUCsj5olKqllJqnVLKB1tz8+NADPAesAdbksiNOcdjI5Blf5yhtTbZz9EOWzIPBP4LvOVAudlN1TnL9sjxPC3HY2su5ZmuK8N83XZjHucBQGv9Gbbk9Qu25LZfKeWdS/zZcV3/NyPZ/tPNHnNXrXUbe825M7Ya6D/YE/5eoB2238sUbNc6+31f/76s/O934RB7i8hn2FoYRnFd7dvOxN+vF0qpFkqp7NuLBvtrntcdl/2+s7+MfI7tS91jwOf219yAjdnXI8c1ydlBL4t/vlchHCIJXJRK9g5iy4DFSqlAAPvP+UC81joNW2KaqbVeZT+sE7Y/qrkZrpQyKKWCsN3z3XCDfXpiu+f5LvA7cF+Ock3cIGHay3lSKeVhb559GlvyzK/sHth1sdW+11+3/f+AZ+zvwQvbF5js81yLzd47vq3Weql9n4pA9TzOrYEGN9xga7HYDky0l18R2IKtn8CNNML2BWia1nottlsCXvzvOrZWSmXX3p8A/tRap+YRH/zz+i8CBmJrZfjuJu/JqpS6zR53O2ATtr+NF7HdhgAYmsd5l2Lr3PcAkN1LfyNwu1Kqib3su4H92PpNZN9GqGePQYh8kwQuSrOnsPWq3mpvotxhfz7Gvn0q8J1S6gC2mtjv/L358kYSgd3AVuAjrfVvN9hnBRCslDpiP18yUMneGnAYSFdK7eTvtejXgb+wdaQ7gi3JPJe/twtAqFJqN7YvBONv0CQ8HluHsAP2fxqYZd/2I/CmUmoE8AK2e+h7sTUBv6q1js7j3KuBO3PZPhTobL/eO4AVWutlN9l3P7b76Uft1/EebNcu+/dzBJiulIrElhhH5BFbtk3AHUqpjwC01hew3ftfobX+Rw3efpthkP1c+7DV0gdprTOxXcuPlVJ7gKZA3M1OqrX+C1tLwn77l0u01oexfTlaaX8frwH3aq2za+9h2PpLnHbwvQnxNwZZTlSI0sHe63uw1jrCRed3w/blpl92T/SSzt4pbhfQU2sd6+p4crL3qP8m+/67EPklNXAhhEPs95THcpP72iWNUmostpr82yUwebcHrJK8RWFIDVwIIYQohZxaA1dKdVJKhd/g9XuUUruUUtvs35KFEEIIkQ9OS+D2sbeLAO/rXvfANqTndmzjRh9XSuXV+1UIIYQQOTizBn4CW+/O6zUFjmutr9h7ev4J9HBiHEIIIUSZ47S50LXW39rnkL5eILahOtmuAg7NsGSfR3k6wN13382AAYVaWEncxKVLlwgODs57R5Fvcm2dS66v88i1LVpmsxk3N9u0B0lJSdx77715TQb1D65YzCQJ2+IS2QKwLVaQJ631DGAGQHh4uLV3795FHJoACA8PR66tc8i1dS65vs4j17boLFq0iFmzZrFz506qVKlS4HJcMYzsCNBIKVXJPj1hT+xzTAshhBBlXWpqKgkJCRw4cKBQ5RRbAldKDVVKPW6fDWkitikft2FbvahUTAohhBBC5NeJEyeYNGkSZrNt2vsnn3ySqKgo+vRxZBHAm3NqE7p9asbO9sfLc7y+FttKQUIIIUSZNmvWLJYsWUK3bt0YNGgQHh4ehWo6zyYzsQkhhBBFyGq1sm3b/+4Mz5o1i5UrVzJw4MAiPY8kcCGEEKIIjRkzhm7duhERYVu2oEaNGgwZMgSDId8dzXPlil7oQgghRJliMplwd7el1EcffZSkpCSqVq3q1HNKDVwIIYQohPnz59O4cWMuX74MQO/evfnmm2+oW7euU88rCVwIIYQohNTUVBITEzl48GCxnlcSuBBCCJEPWmsmTZqExWIBYPz48Rw7doyePXsWaxySwIUQQoh8mDlzJu+++y7r168HwNPTk0qVKhV7HJLAhRBCiFxYLBa2bt167fns2bP59ttvufvuu10YlSRwIYQQIlfDhw+nR48eREZGAlCnTh0GDRpU5MPC8kuGkQkhhBDXycrKwsPDA7AlcKvVWuJWY5MauBBCCJHD+++/T5MmTUhIsC2Uefvtt7Ns2TJq1arl4sj+ThK4EEIIkUNqairJyckcPXrU1aHkShK4EEKIcu3w4cNMnjwZq9UKwMSJE4mKiqJz584ujix3ksCFEEKUa9OnT2fu3Lls2rQJAG9vbypUqODiqPImCVwIIUS5Yjab2bJly7Xnc+bMYe3atYVen7u4SQIXQghRrjz00EP06tWLQ4cOARAaGkr//v1dPiwsv2QYmRBCiDIv57CwkSNH4ufnR+XKlV0cVeFIDVwIIUSZ9tZbb9GkSROuXr0KQL9+/Vi6dCnVq1d3cWSFIwlcCCFEmZaWlkZqaipaa1eHUqQkgQshhChTIiMjmTJlyrVhYS+++CJRUVGEhYW5OLKiJQlcCCFEmfLvf/+bd955hz///BMAHx8fAgICXBxV0ZMELoQQolQzmUx/GxY2d+5cNmzYQI8ePVwYlfNJAhdCCFGqDRw4kFtuuYWoqCgAGjZsyB133OHiqJxPhpEJIYQodXIOCxs1ahQ1atQgKCjIxVEVL6mBCyGEKFVmzpxJs2bNSElJAWw18AULFlClShUXR1a8JIELIYQoVdLT00lPT+f48eOuDsWlJIELIYQo0SIiInjhhReuDQubNm0aR48epXXr1i6OzLUkgQshhCjRpk2bxttvv82uXbsA8PX1xc/Pz8VRuZ4kcCGEECVKZmbmtTHcAB988AEbN26kY8eOLoyq5JEELoQQokTp168fffv25eTJkwAopUrdUp/FQYaRCSGEcLnMzEw8PT0BGDt2LEopKlas6OKoSjapgQshhHCpadOm0bJlS9LS0gB48MEHmTdvHpUqVXJxZCWbJHAhhBAulZGRQWZmJqdOnXJ1KKWKJHAhhBDFavv27bz00kvXns+YMYPDhw/TrFkzF0ZV+kgCF0IIUaymTp3KW2+9xb59+wDw8/PDx8fHxVGVPpLAhRBCOFV6evrfhoXNmzePzZs306ZNGxdGVfpJL3QhhBBOdfvttxMREcHRo0epW7euNJUXEUngQgghilxGRgZeXl4APPHEE4SFhREYGOjiqMoWaUIXQghRZKxWK1OmTKFt27ZkZmYCMGzYMN59910Z113EJIELIYQoMgaD4dqwsJiYGFeHU6ZJAhdCCFEof/zxB1OnTr32/I033uDQoUM0atTIhVGVfZLAhRBCFJjVauXll1/mzTff5PDhwwD4+/tfu/8tnEcSuBBCiHxJS0u7NizMYDAwf/58tm/fLr3Li5nTeqErpYzAfKA1kAGM0Vofz7F9MvAwYAHe0Fp/56xYhBBCFA2r1UqvXr04fPgwWmtq1apFy5YtXR1WueTMYWT3Ad5a6y5Kqc7AXGAAgFKqIjAeaAj4AfsASeBCCFFCZfcoNxgMPP300xw5ckSGhbmYMxN4d2ADgNZ6u1IqLMe2FCAGW/L2w1YLF0IIUcJYrVbGjx/PunXr0Frj4eHBiBEjXB2WwLkJPBBIzPHcrJRy11qb7M9jgcOAG/CmIwUqpWYA0wFZ3N2JoqOjCQ8Pd3UYZZJcW+eS6+scp0+fJj09ndWrV1OjRo0ClTFyQ8pNty2906+goZUZvXv3zvcxzkzgSUBAjufGHMn7LqAGEGp//n9KqS1a6525Fai1ngHMAAgPD7cW5A2LvIWHhxfowyTyJtfWueT6Fo2NGzfy22+/8frrrwPQvn17duzYQd++fQte6IZ1N90kv7OCcWYv9C3A3QD2e+AHcmy7AqQBGVrrdCABkCl6hBDCxaxWK1OnTuXNN9/k+HFbv+OAgADc3Qte37NarUUVnsjBmTXw74DblFJbAQPwmFJqInBca/2jUqovsF0pZQH+BH5xYixCCCFuIiUlhb1799K9e3cMBgMLFizAZDLRsGHDQpWbYTLz84E4lm6JLppAxd84LYFrrS3AuOtePppj+3Ts97OFEEK4htVqpWvXrkRHRxMVFUW1atVo3bp1ocq8kJTOsh2nWbbjNJeSMzAY8t6/aqB3oc5ZHslqZEIIUQ6lp6fj7e2NwWDg2WefJSYmBn9//0KVuS82gaVbTrHuQBxZZisB3u6M7RHK8C4h9Jjz202Pu/vDzXz4UFu6Ngwu1PnLG0ngQghRjlgsFsaNG8fOnTuJiIjA3d2dMWPGFLi8TJOF9QfjWLIlmn2xCQA0qurPyG4hDGxbC1/PvNNMQmoWj3y+g+f7NubpWxpiNOZRZReAJHAhhChXjEYjZrMZs9lMXFwcderUKVA5F69msHzHab7eEcPFq7Zm8r5Nq/FYtxC6NqiM4bp28+jZ/W5a1u6YKzyzfA9zf4kiIuYK7w1pQyU/zwLFVZ5IAhdCiDJuw4YNbN26lZkzZwLw/vvv4+PjU6Ce5ZGxCXyxNZq1+89dayYf093WTF63sm+B4mtfL4h143vw/Kp9/B51kX4fbmbe0Ha0rxdUoPLKC0ngQghRhlksFqZOncqBAwcYPXo09erVIyAgIO8Dc8g0Wdh+zsQH87ew97StmbxBFT9GdgtlUNta+HkVPpVU8vNkycgOzA8/zru/RDHks228dFcTRncP/UdtXthIAhdCiDImKSmJAwcO0K1bN4xGI4sWLcLd3Z169erlq5yLVzNYsfM0X2+P4cLVDAyGDG5tUpURXUPo3jC4yO9VG40GnunTiHb1ghi/Yh+vrztCRPQV5jzQikBvjyI9V1kgCVwIIcoQs9lMp06d+Ouvvzh27BjBwcG0a9cuX2UcOJPIkq2n+CkyjkyzhQAvd26v587UB7oTEuz8aU+7Ngjm5/HdeXbFXjYc+ovDcUnMH9aOFrUqOP3cpYkkcCGEKAOyh4W5ubnx3HPPcfHiRfz8HE+2WWYLGw7+xdKt0eyOuQJA/WA/RnYLYVC72kRs+7NYkne2qoHeLBvTiXd/iWJ++AkGfbKVGfc05+GOdaRJ3U4SuBBClGJms5nRo0dz6NAhduzYgdFoZNy46+fQurn4ZFsz+VfbYziflAHALaoKI7uF0sMJzeT54e5m5IU7mxAWEsTzqyKZ+t0BIqIv8/rAFg4NTyvr5AoIIUQp5ubmhslkwmQycf78eYdXCzt4NpGlW6P5MfIcmSYL/l7ujOwawvAu9ahfpXATuhS1Pk2qsW58d55evpc1e89y4GwinzzSjoZV89cZr6yRBC6EEKWI1Wpl7dq17N27l+nTbbNRz58/Hz8/P9zc3HI9Nsts4b+HzrN06yl2RduayUOD/RjRpR73t69NQAnuKFY7yJdvnujCGz8fYenWaO6dt4U3B7VkQJtarg7NZSSBCyFEKWKxWHj55ZeJiopi7Nix1KxZk8DAwFyPuZySea03eVxiOgC9GldhZLcQejWqUmpmPvN0NzLj3uZ0CKnEi9/u57mV+9h56jKv9G+Gt0fuX17KIkngQghRwiUkJHDo0CG6deuGm5sbS5YswdfXl5o1a+Z63KFziXyxNZrv99mayf083RjRpR7Du4bQoIQ1k+dHv1Y1aFojgKeW7WHZjtNEnklg/tD2BZ5IprSSBC6EECWYyWSiffv2JCYmcuzYMYKCgggLC7u2PeSldXmWUa+yLyO6hDA4rHaZGU9dv4o/3z/djek/HGJVRCz9PtrMOw+05o7m1V0dWrGRBC6EECVQWlratelOn3/+eZKTk/H1zV8Ns0ejYB7rFkLvxlVLTTN5fnh7uPHW4FZ0CK3EtO8P8MRXuxnbI5QX7myCh5vR1eE5nSRwIYQoQbKyshgxYgSnTp1iy5YtGI1GnnnmmQKV9dXoTkUcXck0uH1tWtQK5Klle1i4+RR7Tyfw0dC21Kjg4+rQnKrsf0URQohSxMPD49pqYfHx8bnua7Vaiymqkq9J9UB+fKY7/VvVICLmCv0+/JM/oi66OiynkgQuhBAuZLVa+fbbb3n99devvbZw4UK2b99OlSpVbnpchsnMpG8iiyPEUsPfy52PHm7LawOak5xuYsSSnbz7SxRmS9n8oiMJXAghXMhkMvHyyy/zxhtvcPGircYYGBiI0XjzP8/xyRk8smgHa/acLa4wSw2DwcCjXUJY/WQXalX04cONxxixeCeXkjNcHVqRkwQuhBDFLD4+nq1btwK2JvMvv/ySffv25Vrjznbs/FXum7+FXdFX6N/KsVnXyqNWtSuy7tke9G1alT+PX+LuDzaz89RlV4dVpKQTmxBCFKPMzEzatm1LRkYGx44dIzAwkI4dOzp0bLi+wLPL93I1w8RztzZiQt9GzBuav5XGypMKvh4seDSMhZtPMuf/NA8v3M6UOxRP9KxfJhZEkQQuhBDFIDU1FV9fXzw9PZk4cSJmsxlvb2+Hj/9iazSvrj2Eu5uRDx5qU66nEM0Po9HAE70a0LZuEM8s38Ps9UeJiL7M3AfaUMG3dI+JlwQuhBBOlJGRwdChQ7l48SK///47BoOBCRMmOHy8yWzh1bWH+Wp7DMH+niwYHka7ukFOjLhs6hhaiZ+f68FzK/fy65EL9PtoM/OHtaNV7YquDq3AHLoHrpSqYf/ZQyn1tFKqbA+uE0KIIuLl5YXFYsFsNnPlypV8HZuYlsVjS3fx1fYYmlQP4Punu0nyLoRgfy++HNWJ8bc24mxCGoM/2cZX26JL7XC8PBO4UuoT4HWlVDNgOdAOWOjswIQQojSyWq2sWLGCN99889prX3zxBX/++SeVKlVyuJyY+BQGzd/C5mOXuLVJVVY/2ZXaQeVrrm9ncDMamHhbY754rCP+3u688sMhxq/cR3KGydWh5ZsjNfCOwBjgQeBzrfVoQDk1KiGEKKUyMzN55ZVXeOONN67VuAMDA/PVaWrHyXju+3gLJy6mMLZHKAuGh+HvJXc8i1LPxlVYN7477esFsTbyHPfO+xP911VXh5UvjiRwN/t+A4D1SilfwM+pUQkhRCly8eJFtm3bBtiazL/++mv2799PUFD+m7u/iYjlkc93cDXdxJuDWvJyv2a4lcF5zEuCGhV8WPl4Z8b2COXkxRQGfPwnq3efcXVYDnMkgX8JxAHRWusdQATwmVOjEkKIUiI9PZ1WrVoxePBgkpOTAejcuTOhoaH5KsdisTJ7/VGmrN6Pr6c7X47uyMMd6zojZJGDh5uRl/s147NH2+PhZmTyN5G8uHo/6VlmV4eWpzzbZLTW7yql3tdaW+wv9dRaX3JyXEIIUaJlDwvz9vZm8uTJeHp65mtYWE4pGSaeX7WP/x4+T/1gPxaNCKN+KV6vuzS6o3l1mlYP5Knlu1kVEUvkmQQ+eaQ9ocElt8E5zwSulKoHLFJKhQA9gWVKqVFa62gnxyaEECVOWloaQ4YMISUlhV9//RWDwcCkSZMKXF5cYhqjl0ZwOC6Jrg0q88mw9qV+fHJpVbeyL6vHdeW1nw6zbMdpbnkn/Ib7Rc/uV7yB3YQjTeifAW8DycBfwApszepCCFHu+Pj4YDKZMJvNJCUlFaqsyNgEBszbwuG4JB7uWJcvRnWU5O1i3h5uzBrY0tVhOMSRBB6stf4vgNbaqrVeCAQ6NywhhCgZLBYLX3zxBXPmzLn22qpVq/jtt9+oUKFCgctdtz+OBz/bxsXkDKb1a8obA1vg4SbLUwjHOTIuIU0pVRuwAiilugNlb1kXIYS4gYyMDKZPn05CQgLjxo0jMDCQgICAApdntVqZt+k4c3+Jws/TjU8eCaNPk2pFGLEoLxxJ4BOBn4AGSql9QCXgAadGJYQQLhQXF0dMTAydO3fGx8eH5cuXU7t2bQIDC9f4mJ5l5qVv9/P9vnPUqujD5yPDaFJdGjRFwTiSwE8BHYDG2MaEH9VaZzo1KiGEcJGUlBRatWqFj48PR48exdfXl65duxa63EvJGTz+ZQR7TifQtm5FFjwaRpUAryKIWJRXjiTwvcA+4GvgB0neQoiyxmq1kpqaip+fH35+fkyePJmKFSvi5VU0CVb/dZVRS3dxNiGNe1vXZM7gVnh7uBVJ2aL8ciSB1wP6AA8DbymlfgO+0lpvcmpkQghRDJKTkxk8eDAGg4Gff/4Zg8HAiy++WGTl/3b0As+u2EtyhomJtzXm2T4Ny8Ra1GVdSRkqlhtHJnKxAL8CvyqlegNzge+Agne/FEKIEsLPzw+z2TbrVkpKCv7+RTOBitVqZfGWaGatO4yHm5F5Q9vSv1XNIilbCHBsIpe2wFBgEKD5XwIXQohSx2w2s2TJEhITE5k0aRIGg4E1a9bg7+9fZDXjLLOF6T8eYvmO01QJ8GLh8DDa1Cm9606LksmRJvRF2CZu6aq1Pu/keIQQwqnS09OZMWMGGRkZPPnkk/j6+hZqWNj1ElOzeGr5brYcj6dpjUA+HxFGzYo+RVa+ENlumsCVUtW11n9hq3lbAS+l1LWZ9bXWp4shPiGEKLQzZ85w9uxZOnXqhJ+fH6tWrSI0NBRf36JdX/vUpRRGL93FyUsp9G1ajQ8eaoOfLAMqnCS3T9YioD8Qji2B52xbsgL1nReWEEIUjaSkJFq0aEGlSpU4fPgw3t7edOvWrcjPs+1EPOO+3k1iWhZP9KzPC3c2kWVAhVPdNIFrrfvbH7bXWl/Ouc2+sEmulFJGYD7QGtvMbWO01sdzbL8LmG5/ugd4WmttzVf0QghxA1ar9VqHtMDAQKZMmULNmjXx9PR0yvlW7jzNtO8PYjDAnMGteDCsjlPOI0ROuTWh18FW6/7ZnmwNOY75GWiSR9n3Ad5a6y5Kqc7YOr8NsJcdgG2BlN5a60tKqReAYOBiYd6MEKJkCnlp3U23FfVwncTERAYOHEhAQAA//PADAC+//HKRniOb2WJl9vojLNx8ioq+Hnz6SHs616/slHMJcb3cmtBfBW4BagJ/5HjdhG1q1bx0BzYAaK23K6XCcmzrChwA5iql6gOLtNaSvIUQhRYYGIjFYsFkMpGWloaPj3M6kCVnmJiwci+/HrlA/Sp+LB7RgZASvHa0KHtya0IfBaCUelFr/VYByg4EEnM8Nyul3LXWJmy17VuANtiWKd2slNqmtY4qwHmEEKVY/482U6OCDzUreFO9gg81K3pTo4IPNSp4Uy3QG0/33FfoMplMLFiwgMzMTNq0aYPBYOCnn34qsvHcN3I2IY3RS3dx9K+rdG8YzMdD28kyoKLY5daE/rjWegHgrZT69/XbtdYz8yg7Ccg5NsNoT94A8cAuey93lFJ/YEvmuSZwpdQM7PfN+/Tpk8fpRUFFR0cTHh7u6jDKpPJ4bc2W3Lu26LgkDp698braBiDQy0AlLwOVfAxU8jZQydtIkLeByt62557mNF555RUAZs+eXdTh/8OJBDMf7MkgKdNKnzruDK2fyt6dW5x+Xlcrj5/d4tS7d4xeCmwAACAASURBVO98H5NbE7rhJo8dtQW4B/iP/R74gRzbdgMtlFLBQALQGViYV4Fa6xnADIDw8HBrQd6wyFt4eHiBPkwib+Xt2sYlpvHcin1A6k33iZp1NwmpWZxLTCMuIZ24pHTiEtKIS0znXEIafyWlczYxnVNJ5hsebzRA3We+omaQL/+XnE7c1arXavHVK3hTs6I3VQO8i6RH+I+R53jr10hMZisz7mnGiK4h5WZa1PL22S0NcmtC/8z+81WllKfWOlMp1RBQwHoHyv4OuE0ptRXbF4DHlFITgeNa6x+VUv8C/s++73+01gcL9U6EECXKxiPnmfxNJFdSs3Ldz2AwEOTnSZCfJ81r3niGZqvVSnxKJn8lphN15iJPPP8SgdXrcvfgYZxPyuRcYhpRF1PJMluJOH/qH8e7GQ1UDfCiRgVvalT0oUag7aet2d6bmhV9CPb3+luSz63j3ZLHOnCLqurglRDCORyZSvUVoJlS6kVsndkOAbcBE3I7zj6H+rjrXj6aY/tKYGV+AxZClGyZJgtvbTjK53+ewtPdyGv3teCV7wv//dybLFrUqkCLWhU4dFdr6tevz9CH212rAVssVn785TdCm7UjLjGduMQ0+8//1egjzySy53TCDct3NxqoFuh9LcnnRpK3KAkcmSLoPmw9yp8DvtZav6CUinBuWEKI0igmPoVnV+xl/5lE6lfxY97D7WhWM5BHO9crcJmXL1/mvvvuo1q1anzzzTcA1+5552Q0GqjoZaR1nYq0vskwbLPFyqXkDM4l/DO5Zyf8PaevYIm5UuB4hSgujiRwo9Y6TSnVH5hmn6BFxkoIIf5mbeQ5/rXmAMkZJga3r83MAc3x9Sz8NKJBQUGYTCZMJhOZmZmFmozFzV7LrhboTdub7GMyW7hwNYOus2XFZFGyOfK/a6NS6iC2Xih/AL8DPzo1KiFEqZGWaWbmT4dYsTMWX0833hvSmoFtaxe4vKysLD7++GPc3d155plnMBgM/PLLL/j5FU+9wd3NKIuPiFLBkfXAJyulPgTOaK0tSqlntdb7iiE2IUQJF3X+Ks8s30PU+WSa1Qhk3tC21K9SuPHXKSkpzJo1Cx8fHx5//HE8PT2LLXkLUZrkPkMCoJSqArwDXFBKJQDTlVLVnB6ZEKLEslqtrNx5mnvn/UnU+WRGdKnHmqe6Fjh5Hz9+nN27dwNQsWJF1qxZw+7du502d7kQZYEjTeifAVuBMdgS/uPA59hWKhNClDNX07OY+t1B1kaeI9DbnfeHtOXOFtULXN6FCxdo1aoVoaGhREZG4u7uTo8ePYow4oIp6jnahShqjiTw+lrrQTmez1FKPeqsgIQQJdf+Mwk8u2IvMfGptKtbkQ8fbkvtoPyvqW2xWEhNTcXf35+qVasyZcoUmjdvjpubmxOiFqJsciSBW5VSdbTWsQBKqbpA7jMzCCHKFKvVyuIt0cxef4Qss5Unezdg4m2N8XDL8y7cP1y4cIF7772XBg0asGzZMgBeffXVog5ZiDLPkQT+CrBNKbUD24xqnbA1owshyoErKZlM/iaSjUcvEOzvybsPtqFn4yoFLi84OBiz2YzZbMZkMuHuXvihZkKUR470Qv9JKdUW6IgtgY/TWl9wemRCCJfbeeoy41fs5a+kdLo1rMx7Q9pQNcA7X2VkZGTw4YcfEhAQwLhx4zAajfz+++/4+ua/6V0I8T+OTKVaEXgZuBVb0/nPSqlZWus0ZwcnhHANs8XKx78d5/1fozAYDEy5QzGuV4MCLQiSnJzMm2++SVBQEGPGjMHd3V2StxBFwJG2q6+xzWE+FFsv9MeARcAwJ8YlhHCRC0npTFi1j60n4qlZwZsPHm5Lh5BK+Srj6NGjpKWl0bZtWypXrsz3339PixYtpLlciCLkyP+mEK11ziFjE+wzswkhyphwfYFJ/4kkPiWTvk2r8c4Drajom7+x2GfPnqV169Y0bdqU3bt34+bmRs+ePZ0UsRDllyMJ/JBSqofWejOAUqoVcMy5YQkhilOW2cI7/9V89vtJPN2MTL+nGSPzsda12WwmLS0Nf39/atWqxZQpUwgLC8NozH8vdSGEYxxJ4E2A35VSGjBjWw/8slLqFGDVWtd3ZoBCCOeKvZzK+JV72Xs6gZDKvswb2o4WtW68LveNxMXF0b9/f1q1asWSJUsAeP31150VrhDCzpEEfq/ToxBCuMSGg3G8sHo/SekmBrSpyayBLfH3yt996qpVq2KxWK4NDZPJWIQoHo4MI4spjkCEEMUnPcvMrHVH+Gp7DD4ebswZ3IoH2td2qMk8PT2duXPnUq1aNcaMGYObmxtbtmyRnuVCFDPpEipEOXPiYjLPLN/LkbgkVLUAPh7WloZVAxw+PikpiTlz5lCzZk1GjRqF0WiU5C2EC9y0h4lSaoj9Z3DxhSOEcKbVu89wz0d/ciQuiaGd6vLDM90cSt4HDx4kMjISsDWZr127lu3bt0snNSFcKLca+Cyl1LfAf4F2xRSPEMIJUjJMvPL9QdbsPUuAlzvzhralf6uaDh0bExND27Ztad26NTt37sRoNMqwMCFKgNwS+B9ABmBQSlmu22bVWktPFSFKgUPnEnl2+V5OXkqhde0KfPRwO+pWzr3JO+ewsHr16jFlyhS6d+8uNW4hSpCbJnCt9ShglFLqB631gGKMSQhRBKxWK19tj+H1dUfINFkY2yOUKXc0wdM99yQcGxtL//796dy5M5999hkAb7zxRnGELITIB0d6oQ9QSt2FbS50d2CT1vpHp0cmhCiwxNQsXvg2kv87dJ4gXw8+faQdfZpUc+jY6tWrY7FYsFgsWK1WhydzEUIUL0cWM5kCDAaWYVuNbJpSqqXWepazgxNC5N/umCuMX7GXswlpdAqtxAcPtaV6hZuvIJaamsqcOXMICQlh5MiReHh4sHPnTnx8fIoxaiFEfjkyjOxRoFP26mNKqYXAbkASuBAuFPLSuptuMxrguVsbMf7WRnmuIJaYmMjcuXNp0KABI0aMwGAwSPIWohRwJIEbr1s6NB0wOSkeIUQRWDamM10aVL7p9r179+Lu7k7Lli2pUaMGP/30E+3atZPmciFKEUcS+Eb7cLKl9ucjgE1Oi0gIkSezxZrr9tyS97FjxwgLC6NTp05s2bIFg8FAr169ijpEIYSTOZLAJwDjgOHYJn7ZBHzmzKCEELYkfS4hjVOXUoiJT+HUpVSi41OIjk8h9nJqvsrKysoiMzMTPz8/GjVqxOTJk+nbt6/UuIUoxRzphW4FPrH/E0IUoewkHR2fQvSlFKLjU4m+lMIpe5LOMv+zph3k60HzmhXYF5vg0DlOnTpF//796dOnDx999BEAb731VpG+DyFE8ZO50IVwspxJetPpLDb/dNierFOIvZxGpvn6eZKgoj1Jh1T2JSTYj9BgP+pV9iOksi8VfT2B3Dux5VSrVi2sVuu1f1LrFqJskAQuRBHITtIx8amcstembc3eN0rSpwCo4ONB05qBhFb2pV5lW5IOCf57ki6oqVOn0qxZMx555BE8PT3Zs2cP3t43H0omhCh9HErgSqlAoAK2ceAAaK1POysoIYpbbrXZ6Nn9AFuSjktMIzr7XrS9Fh0dn8rp+NQb1qSzk3RIZV9CKvuRdvE0d3VvT2iwX6GTdHZc14uNjUWpB2jRogXDhg3DYDBI8haiDHJkIpepwEtAfI6XrUB9ZwUlREky5osIouNTOH05lUzTP5N0oLc7TWsEEGJv5g4NtiXrkMp+BPn9PUmHh5+jbd2gIo8xIiICX19fmjVrRp06dVi/fj0dOnSQ5nIhyjBHauCjgQZa64vODkaIkujXI+dtSbp6gO0+tL2ZOyTYj9DKflT09XBpojx06BAdO3akZ8+e/PbbbzIsTIhywpEEfhq47OxAhCip9rxyG0EuTtLXy8zMJCsrCz8/P5o3b86kSZPo169fiYpRCOFcjiTwY8CfSqnfsM3CBoDWeqbTohKiBKnkV7h71UUtKiqK/v37c8899zB37lwA3n77bRdHJYQobo4k8LP2f5CjE5sQZcXq3WdcHUK+1K1b91pNW4aFCVF+OTKRy6tKqSpAJ/v+27TW550emRDFYNPR87z47X5Xh5GrpKQkZs6cSYcOHRgyZAje3t5ERkZKz3IhyjlHeqHfASwGtmObSvUzpdRorfVPzg5OCGfac/oKTy3bg4ebgf880Zn29Sq5OqQbio+PZ968eXTs2JEhQ4YASPIWQjjUhD4L6K61PgWglKoPrAEkgYtS6/iFq4xauosss5XPHmlf4pL3tm3bCAoKokmTJoSGhrJhwwY6d+7s6rCEECWI0YF9PLKTN4DW+qSDxwlRIsUlpjH8850kpGbx5qCW9G1WzdUh/c3evXvp2rUr48ePv/Za7969pdYthPgbh4aRKaUmAJ/bn48BYpwXkhDOk5CayfDPd3IuMZ0X7lQ8GFbH1SEBkJ6ejsViwdfXlzZt2jBp0iQGDhzo6rCEECWYIzXp0UAX4CQQbX/8uBNjEsIp0jLNjPkigmMXknmsWwhP9mrg6pAAOHz4MM2bN+e1114DwGAw8M4779CtWzcXRyaEKMkc6YV+ARiS34KVUkZgPtAayADGaK2P32CfdcAPWutP83sOIRxlMlt4ZvkeImKucG/rmrzSr1mJGX4VEhJSYmIRQpQeN03gSqmftNb9lVKnsM19/jda67zmQr8P8NZad1FKdQbmAgOu2+d1oGT1HhJljtVq5V9rDrDx6AW6NwzmnQdaYzS6LmFeuXKF6dOn07t3bwYNGoSvry+HDh3Cy8vLZTEJIUqf3GrgY+0/exew7O7ABgCt9XalVFjOjUqpwYAFWF/A8oVwyNv/p/lm9xla1qrAp4+2x9PdtX0wL126xKeffsqRI0cYNGgQgCRvIUS+3TSBa63j7A/f1Vrfn3ObUmojcGseZQcCiTmem5VS7lprk1KqBTAUGAz8O/9hC+GYxX+eYn74CUKD/VjyWAf8vRxaQbfI/f7779SsWROARo0a8d///pcuXbq4JBYhRNmQWxP6GqANUEspdTLHJg9sC5zkJQkIyPHcqLU22R8PB2oBm4AQIFMpFa213pBbgUqpGcB0gD59+jgQgiiI6OhowsPDXR1GoW2PM/FpZAYVvAw81czCwYhtLonj8OHDPP3003Tq1Ilx48Zdu7bbtrkmnrKsrHx2SyK5ts7Vu3fvfB+TW3VkJLb70x8Az/K/edBNgCNTqW4B7gH+Y78HfiB7g9b6hezH9qT8V17J237cDGAGQHh4uLUgb1jkLTw8vEAfppJk87GLfP7LLgK83FnxRBea1Qws1vOnpqZiMBjw8fGhV69eHD9+nIceeojU1NRSf21LsrLw2S2p5NqWPDe9Gai1TtJaRwNPAU9rrWPs+7+JYx3PvgPSlVJbgfeA55VSE5VS9xY+bCFubv+ZBMZ9tRuDwcDCEWHFnrwjIyNp0qQJs2fPBmzDwt599106duxYrHEIIco2R24Ifg2stD8+B2y2v3Z7bgdprS3AuOtePnqD/WY4EIMQDjl1KYXHluwiNcvMJ8Pa0bl+5WKPoUGDBri7u2M0yoSFQgjnceQvTGWt9WcAWusMrfVCINi5YQmRfxeS0hm+eAfxKZm8NqAFd7aoUSznvXTpEk8++SRr164FwN/fnyNHjjB9+vRiOb8QonxyJIGnKqXuyn6ilOoLpDgvJCHyLyk9ixFLdhF7OY0JfRvxSOd6xXbuixcvsmjRIhYsWHDtNRkWJoRwNkea0McBXyulvrI/jwUedV5IQuRPepaZx7+M4EhcEsM61eW5Wxs5/ZwbN26kfv36hIaG0rRpUzZu3CjDwoQQxcqRqVT3AS2UUpWBLK11kvPDEsIxZouVCSv3sf3kZe5sXp2ZA1o4fVrSP/74g759+zJw4EDWrFkDQM+ePZ16TiGEuF6eCdw+BOxfgD9gUEq5AfW01iFOjk2IXFmtVl754SAbDv1Fp9BKvP9QG9ycNEVqcnIy7u7ueHt706NHDyZMmMDw4cOdci4hhHCEI/fAFwPfY0v2HwNnsA0RE8KlPth4jOU7TtO0RiALR4Th7eHmlPPs3r0bpRTvvPMOYBsW9t5779G2bVunnE8IIRzhSALP0FovAcKBK9hmUbvDmUEJkZevt8fw/q/HqFPJhy8e60Cgt4fTztWwYUM8PDxwc3POFwQhhCgIRxJ4ulKqEqCBzlprMyB/yYTLrD8Qxys/HKSynydfjupE1UDvIi3//PnzjB49mg0bbJMDVqhQgaioKP71r38V6XmEEKIwHOmFPhdYBQwCdiqlhgERTo1KiJvYdiKe51buw9fDjaWPdSQ02K/Iz3H+/HmWLl1KYmIid955JwCenp5Ffh4hhCgMR2rgacDtWuurQBjwCDKMTLjA4XNJPP5lBFasfPpoe1rWrlBkZa9fv57Tp21r9LRq1YpNmzaxcuXKPI4SQgjXcaQGPkdrvQ5Aa50C7HVuSEL8U+zlVEYs2cnVDBMfPtyWHo2qFFnZv/zyC3fffTdDhgy5lrR79epVZOULIYQzOJLATyilFgM7sNXGAdBaf+m0qITI4VJyBo9+voOLVzOYfk8z7m1ds9BlJiUl4eXlhZeXF7feeivPPfccY8aMKYJohRCieDjShB6PbSnRzsAt9n+9nRiTENckZ5gYtXQX0fGpPNW7AY91Cy10mdu3b6dx48Z88MEHABiNRt5//31atGhR6LKFEKK43LQGrpSqpbU+q7V+rDgDEiJbpsnCk1/vZv+ZRB4Mq82UO1SRlKuUwtvbG3d3RxqghBCiZMqtBr42+4FSalIxxCLENRaLlcnfRLL52CVubVKVNwa2LPAUqefOnePRRx9l48aNAAQFBREVFcXEiROLMmQhhChWuSXwnH8thzk7ECGyWa1WZv50mB8jz9G+XhDzhrbD3a3ga2vHxcWxbNkyFi9efO01GRYmhCjtcvuraM3x2LmrQwiRwye/n2Dp1mgaVfXn8xFh+Hjmb94gq9XKDz/8wNmzZwFo37494eHhfPml9LsUQpQdjlZrrHnvIkTh/SciljkbNDUrePPl6I5U9M1/TXndunXcd999TJ069dprPXv2lKlQhRBlSm69eJorpU7aH9fK8dgAWLXW9Z0bmihvfj18nn+tOUBFXw++HN2RGhV8HD72ypUr+Pv74+HhwV133cX48eMZN26cE6MVQgjXyi2BNy62KES5tzvmMk8v34OHm4HFIzvQsGqAw8du3ryZQYMG8fLLLzNhwgTc3NyuDRETQoiy6qYJXGsdU5yBiPIr6vxVRi2NwGSxsmh4GO3qBuXr+KZNm+Lj44OHh/NWJBNCiJKm4F17hSgC5xLSGLF4J4lpWcy5vxW3NKma5zGxsbE89NBDbN68GYDg4GBOnDjB008/7exwhRCixJCZLITLXEnJZPjincQlpvOvu5pwf/vaDh139uxZVq1ahZ+fHz169ACQ2rcQotyRGrhwidRME6O+2MXxC8mM6R7K4z1v3ifSarWyevVqzp8/D0Dnzp35448/WLhwYXGFK4QQJY4kcFHssswWnlm+l72nE7ivTU2m3t0011nW1qxZwwMPPMC0adOuvdajRw+MRvn4CiHKL/kLKIqV1WrlpW8PsOnoBXo2rsKcwa0xGv+ZvOPj4zGZTAAMGDCA8ePH8+KLLxZ3uEIIUWJJAhfFavaGo3y75wyta1fgk2Ht8HT/50dw06ZNNGrUiAULFgDg7u7OBx98QMOGDYs7XCGEKLEkgYtis2jzST77/ST1g/1YPLIDfl437kPZvHnza5OyCCGEuDFJ4KJYfL/3LK+vO0K1QC++GNWRyv5e17ZFR0dz//33s23bNgCqVavGiRMnGDt2rKvCFUKIEk+GkQmn+z3qIpO/iSTA250vRnWkTiXfv22PjY1lzZo1VKtWjS5dugAyLEwIIfIiNXDhVPtiE3jy690YjQY+H9GBJtUDsVqtLF++nEuXLgG2HuWbN2/m448/dnG0QghRekgNXBSpkJfW3XRbx9BKAKxYsYJhw4bx9NNPM2/ePAC6d+9eLPEJIURZIQlcFBuz2YybmxsPPPAAO3fuZMKECa4OSQghSi1pQhfFZsmSJYDt/vb7779PSEiIawMSQohSTGrgotCSM0ysjTzHyl2xue4nHdOEEKLoSAIXBWK1Wtkbm8CqnbGs3X+O1EwzN5hQ7W9GjBhRPMEJIUQ5IAlc5MvllEy+23uWVbtOE3U+GYDaQT482asOg8Nq0+XNTS6OUAghygdJ4CJPFouVrSfiWbnrNP89dJ5MswVPNyP9W9XgoQ516RwaxPLly/Bud4+rQxVCiHJDEri4qbjENFZHnGFVRCxnrqQB0KiqPw91rMvAtrWo5OcJ2DqnjRo1igkTJhD93nuuDFkIIcoNSeDib7LMFnafN/Hl0l2E6wtYrODj4caDYbUZ0qEu7epWxGAwcP78eSw+VTAajQwdOpT9+/czceJEV4cvhBDlhiRwAcCpSyms2hXL6t1nuJScAVygdZ2KPNShDv1b1SDA+389yH/88UeGDh3K/PnzGT58OF5eXrwnNW8hhChWksDLsfQsM+sPxrFyZyw7Tl0GoIKPB7fVc2fifV1oWiPwhse1adOGypUry7AwIYRwIUng5dChc4ms2hXLd3vPcjXdBEDXBpUZ0qEOdzSvzvYtm/+WvLXWPP/888yePZtWrVpRt25dTpw4gbu7fHyEEMJVnPYXWCllBOYDrYEMYIzW+niO7c8DD9mf/qy1ftVZsQhISs/ix33nWLUrlgNnEwGoGuDF8C71eDCsDvUq+9302OjoaNavX0+bNm1o1aoVgCRvIYRwMWf+Fb4P8NZad1FKdQbmAgMAlFL1gWFAJ8AKbFZKfae13u/EeModq9VKRMwVVu6MZd2Bc6RnWXAzGujbtBoPdahDb1UFd7d/zqZrNptZvHgxgwcPJjAwkDvuuIOtW7fSuXNnF7wLIYQQN+LMBN4d2ACgtd6ulArLsS0WuFNrbQZQSnkA6U6MpVy5lJzBmj1nWLkrlpMXUwCoW8mXIR3qMLh9baoFeud6/Lp163jvvfeIiopi9uzZANfW6RZCCFEyODOBBwKJOZ6blVLuWmuT1joLuKSUMgBvA3u11lF5FaiUmgFMB+jTp48TQi69LFYrBy+Z+eOMib0XzJit4G6EzjXc6FXbA1UJjIYzHNlzhiM3OD4+Pp5KlSphMBho2rQpgwcPJiwsjPDw8OJ+K2VadHS0XFMnkuvrPHJtnat37975PsaZCTwJCMjx3Ki1NmU/UUp5A4uBq8BTjhSotZ4BzAAIDw+3FuQNlzVnrqTyTcQZvomI5VxiBgBNqgfwUIc63Ne2FhV9PfMsY/Xq1QwfPpwlS5YwZMgQAMaOHevUuMur8PDwAv1HFY6R6+s8cm1LHmcm8C3APcB/7PfAD2RvsNe8fwA2aa3fcmIMpV7IS+tuuq1n4ypsPnYRqxX8PN14uGNdHupQh1a1K2Aw5LGySA7t2rWjatWq0jFNCCFKEWf+xf4OuE0ptRUwAI8ppSYCxwE3oBfgpZS6y77/v7TW25wYT5nzR9RF2tcLYkiHOvRrWQM/L8d+nQcPHmTixIm8//77NGvWjPr163P8+HFJ4EIIUYo47S+21toCjLvu5aM5Hufek0rk6Zfne9KoWkDeO17n1KlT/PLLL6xevZp///vfgAwLE0KI0uafY4hEiZCSYWLJllO57uNo8jaZTCxYsIDkZNvyn/3792fbtm3XkrcQQojSR6pdJcxfieks3RrN8h0xJKWb8j7AAR9//DETJkzgzJkzzJw5E4PBIGO6hRCilJMEXkIciUti4eaTrI08R5bZSmU/T57v25j3fs1zdN0NnTt3jho1amAwGBg7diyxsbE888wzRRy1EEIIV5EE7kJWq5U/jl1i0eaTbD52CYAGVfwY26M+97WthbeHW4ES+PLlyxk1ahTLly9n0KBB+Pr68s477xR1+EIIIVxIErgLZJjM/LjvHIs2n0KfvwpAl/qVGdszlN6Nq2I0/m8IWPTsfvkuPywsjJo1a8pqYUIIUYZJAi9GCamZLNtxmqVbo7l4NQM3o4EBbWoytkd9WtSqUOByIyMjmThxIp9++imNGjWicePGHDt2DDc3tyKMXgghREkiCbwYxMSnsPjPU/wn4gxpWWb8vdwZ2yOUkd1CqVXRp9DlHzt2jE2bNrFmzRpefPFFAEneQghRxkkCd6LdMVdYtPkk/3foLyxWqFnBm0ndGzOkQx0CvAvevJ2VlcWiRYsYMWIEvr6+3H///ezYsYOOHTsWYfRCCCFKMkngRcxssfLL4b9YuPkUu2OuANCiViBje9Tn7pY18LjB8p359d577/Hiiy8SHx/PtGnTMBgMkryFEKKckQReRFIzTazefYbP/zxFTHwqALc2qcqYHvXpXL9SvuYmv5EzZ85Qq1YtDAYDTz75JJcuXWLcuOsnuhNCCFFeSAIvpAtX0/lyawxf74ghITULT3cjD3esw+juoTSsmv9pTm9kyZIlPPHEE6xZs4b+/fsTEBDAnDlziqRsIYQQpZMk8AKKOn+VhX+c5Id958g0W6jk58n4WxsxvEs9gv29ivRcHTt2pG7dujJfuRBCiGskI+SD1Wply/F4Fm4+ye9RFwGoH+zH6B6h3N+uNt4eRdPzOyIigsmTJ7NkyRJC/7+9Ow+K6s4WOP4FxLC4oNFBcQngcuGZGDUL4orbJJowMUYQx8ElicjERIUJi4q4lBBRfDzFiqIO1aP1UNTgMmpcGKNMfHGSKBPMqD+jaNAgZFwQRRFE3h+tPaDQAtJg4/lUdSndfX/39IHq033v7/6OiwvdunVDKSUzy4UQQhhIAa+Corv32JmRzZq/n+PkpXwAXndpyeT+rgxxK7/wSm04ffo0hw4dYvv27cyYMQOQy8KEEEKUJ93IjLh+u5hVh84yYPFXBG/6gdO5N3i7hBSnxgAAEyBJREFUe1u2T+3LpimeDPsvx1op3nfu3GHFihUUFhYCMHbsWL7//ntD8RZCiKr6xz/+gaenJ/7+/vj7++Pr68v69eurvH1qaipvv/0269atq/I2eXl5/PWvfwUgPDyctLS0asdtTN++fQGIiooiOzub+Ph4NmzYUKOx9u/fT25ubpWem5aWRnh4eI32UxfkGzjgHL7L6OP2ja14v68Lk/o606GlXa3vf/HixURGRnL79m1CQkKwsLDglVdeqfX9CCGeDb179yYuLg6AoqIi3nzzTd555x2aNWv22G2/+uorgoODGTx4cJX3p5TiwIEDeHt71zjmqpg9e/YTj7Fu3TrmzZuHo6NjLURUv6SAP8bM4W74vd6R5ra1u674hQsX6NChAwDTpk3j1q1bfPjhh7W6DyGEuHnzJpaWllhZWaGUYuHChQA4ODgQHR3NiRMniI2NxdraGh8fHw4ePEhGRgYtWrQgJycHnU6HpaUlrVu3xsvLiytXrhAeHs6NGzcoLS0lJiaGVatWcerUKZKTkw37/dOf/oS3tzdeXl6cPXuWmJgYVq9ebXj8/PnzREREUFxcjI2NDXFxcVy+fJlFixZx79498vPziYiIoFevXoZt/P39mTdvHqA/UvDll19SWFhIREQE3bt3Z9CgQbi6uuLq6oqPj88jY+Xn53Py5EnCwsJISkoiOTmZnTt3YmFhwYgRIxg/fjxnz55l1qxZ2NraYmtrS/PmNV/m2tTkEPpjTBnYqdaLd0JCAq6uruzbtw+A5s2b89lnn9GiRYta3Y8Qov45Ozvj5+dn+DklJQVnZ+dyxc7f3x9nZ2eKiooAuHz5Ms7OzkydOtXwnDVr1uDs7FylfR45cgR/f3/Gjx9PSEgIc+bMwd7enjlz5jB37lzWr1/PgAEDWLt2LaA/jZeUlMS7775L//79CQkJwcXFhfj4eHQ6HRs2bCAvL4/Dhw+zcuVKBg8ezMaNG5kxYwYZGRkEBgbSu3dvxowZY4jBx8eHrVu3ArBlyxZGjx5dLsaYmBgCAgJITk5mzJgxnDhxgjNnzhAWFoZOp2PSpEmkpKRU+hrbtWvHunXriIqKYu7cuQBcunSJ2NhYZs+eXeFYXl5euLu7ExMTQ1ZWFrt37yYpKYmkpCRSU1PJzMxk2bJlTJs2DZ1OR8+ePauU7/oi38DrgYeHBy4uLjIxTQhhEmUPoZd19uxZ5s+fD+iXZHZxcQEw/FtWVlYWV69eJSAgAICcnBwuXLjAuXPnDMXY09MT0J93f5iHhwdRUVFcuXKFw4cPExwcXO7xc+fOGQrkiBEjAP0VOJ9//jk2NjYUFBTQpEmTSl/ja6+9BkCXLl3497/1VwW1aNHC8EXoN7/5jdGxTp8+TXZ2NhMnTgTg+vXrZGVl8dNPP9G9e3cAevXqRWZmZqUx1Dcp4HXgm2++ITQ0lKSkJDp06ECPHj04efKkFHAhngHnz58v9/OoUaMYNWpUufsenmTWqlWrR7abPHkykydPfqJYXFxciImJwcnJiaNHjxoKn6Xlowdj27dvT9u2bUlMTMTa2pro6GhefvllMjMzOX78OG5ubnz33XccPHgQLy8v7t27V257CwsLvL29iYqKom/fvo+0N+7UqRPHjx+nT58+7Nixg+vXr5OSkkJsbCydOnVi+fLl/PLLL5W+loyMDLy9vVFK4eTk9MjriIqKqnAsCwsLSktLcXV1pXPnzqxduxYLCwt0Oh1du3bF1dWV9PR0BgwYwI8//lizRNcRKeB1QCnF119/za5duwzLn0rxFkLUtXnz5hEWFkZJSQmgL3K//vprhc9t2bIlEydOxN/fn5KSEqytrQkKCiIwMJBZs2axY8cOAKKjo2ncuDGnT59Gp9OVG2PUqFF4eXmxffv2R8YPDQ0lMjKSlStXYmNjw5IlS7h79y4fffQRzz//PG3atOHatWuVvpaLFy8yfvx4ioqKWLBgwSOP/+53v6twrJ49exIaGkpiYiKenp6MHTuWoqIiunfvjqOjI3PnziUoKIg///nPtGzZkueeq92FuWqTRWlpaX3HUCMHDx4s9fLyqpWxjM1CP7/orWqPd/v2bVavXs0f//hHGjduzL179/jhhx+e+vMpDzz4RC1qn+TWtCS/plOT3Obm5hIaGspf/vIX0wTVsFT7mmT5Bk7NirQx0dHRhpme06dPx9LS0myKtxBC1Ia9e/eyYsUKoqKi6juUBksKeC3JysqiY8eOAAQFBVFaWsqkSZPqOSohhKgfb7zxBm+88UZ9h9GgyWVktWD58uW4urry1VdfAfpzRwsXLqzSoglCCCFETUgBrwV9+vShS5cuMjFNCCFEnZECXgNpaWn07duX7OxsAF599VX+9a9/MWDAgHqOTAghxLNCCngNnDp1im+++Ya9e/ca7qvoOkohhBDCVKTqVEFBQQFxcXEUFxcD8MEHH5Ceni6T1IQQT6WffvqJgIAA/P39ee+991i+fDmVXTL8oHtYWloaycnJXLx4EV9f3xrtNzs7mwMHDlT5+Q+6jIF+bfMHHdR8fHzYs2cPwBN1HivrwesDiIuLY9SoUeh0OlasWFHlMa5fv86sWbMYN24cfn5+BAUFcePGDbZs2UJoaGi55548eZKxY8c+cdzGyCz0Kpg/fz5LlizB1taWwMBArKysePnll+s7LCGEmavtNSgA8vPzCQ4OJj4+HmdnZ0pKSpg+fTobN240WlAenAK8ePFijfYL+jXYMzMzq9XJDODYsWPodDoSEhKwt7fn2rVrjBkzhs6dO9c4loeVPcW5e/dutm7danSp1ooEBwfj5+fHsGHDANDpdERGRhIdHc2yZcu4desWdnb6jpVbtmwptza8KUgBr8T58+cNjQNCQkKwtbXlD3/4Q/0GJYQQj/G3v/0NDw8Pw/uXlZUVMTExWFtbM3v2bHJycrh27RoDBgxgxowZhu1SUlLIzMzEz8+Pq1evEhgYyNWrVxk4cCBTp05Fp9OxceNG8vLyWLlyJbGxseXG+uSTT1i9ejWFhYX07NmT9u3bP9L5zM7Ojjlz5nDmzBk6dOhgaN6yefNmJkyYgL29PaBf03zz5s3lruQpKSkhMjLykfj37dvHmjVraNSoEe3atWPx4sWkp6cTExNDo0aNaNasGbGxsezbt4/MzExsbGzIyclhypQpBAQEsG3bNuLi4vjyyy8NnddeeeUVPv30U+Lj40lPT+fWrVsEBwdz+fJlQ/EGDEc4bG1tGTx4MPv27WPkyJEUFRWRlpZGSEiISX/XUsArsGTJEsLDwzl06BD9+vWjdevWhgYAQghRVdG7T7Ir41KNtu27qOJD0W91b8usEe6Vbvfrr78aWhU/YG9vz8WLF+nRowc+Pj7cuXPnkQJe1q1bt1iyZAl2dnaMGzeOIUOGAPomKRMnTqx0rICAADIzMxkyZAi+vr5ER0fTuXNnNm/ezNq1a+nRowd37txh06ZNZGdnG+YRVRTzw208L126VOE+d+7cycSJE3nrrbfYtm0bN2/eJDU1lWHDhvHBBx9w4MAB8vPzDeN8/PHHpKSkkJiYyD//+U8A8vLyiI+P54svvsDW1paQkBAOHz4MgKurKxEREaSnp9O+fftyMVlZWdG0aVMAfH19iY2NZeTIkaSmpjJw4EBsbGwq/T3VBingFejXrx/u7u4yMU0IYXacnJw4ceJEufsuXLhATk4Ox48f58iRIzRp0sTw7bcibm5uhsL00ksvce7cOeA/XcscHBweO1ZFnc/KdvpycnKibdu2hv9funQJNzc3w/ZHjx6lVatWhp8r2+fMmTNJSEhgw4YNuLq6MnToUAIDA1m1ahUTJkzA0dHRsM/KPNx5raCggAsXLpR7zU5OTuTk5JTbrri4mD179uDt7U23bt3Iz88nNzeXlJQUwsLCjO6zNkgBR3/IKSIigm3btuHo6IinpycZGRlSwIUQT2TWCHej35aNnQM/HF6988gPDBo0iISEBMaOHUvHjh0pLi5m0aJFeHh40LRpUxYsWMDPP//Mpk2bKp3YdvbsWQoKCnjuuefIyMgwnMu1sNAv152SklLhWJaWloauZBV1PmvUqBG7du1iwoQJ5ObmkpubC+ibnixduhQPDw/s7Oy4cuUKs2bNYtmyZYaYKttncnIyn3zyCc8//zyRkZHs37+fgoIC3n33XcLCwkhISGDTpk2GjmUVebjzWkpKCu7u7qSmphrqgKOjIy1atCA1NZWhQ4cCsG7dOkNXNIDRo0ezfv16CgsL6dKlS41+f9UhBRz9bMFvv/2W1NRUxo0bB8hlYUII89SkSRMWLVpEREQEpaWlFBQUMGjQIDw9PQkODubo0aPY2trywgsvVNqJrHnz5gQFBXH16lVGjBjxyGSyysbq2rUrK1eupFu3bhV2PnNxceHo0aP4+Pjg5ORk6N3ds2dPfH19ef/992nUqBGFhYUEBwfj5ubG/v37je6ze/fuTJo0CQcHB+zt7fHy8iIrK4vw8HDs7OywtrZmwYIFfPfdd5Xm7OHOa+3atWP48OGPPG/x4sUsWLCAxMREiouL6dixo+E8P4C3tzdeXl7Mnj27er+0Gnomu5HduHGDNWvWMH36dKysrLh79y6nTp3ixRdfrN0gzZR0dDIdya1pmVt+TTEL3VTMLbdmSLqRVUVERATLly/HwcHB8IlPircQoq49bUVamJdnpoCXvSxs5syZtG7dGj8/v/oNSgghhKihZ+JEb1RUFJ06deLbb78FoE2bNkRERBguuBdCCCHMzTNRwPv3789LL71kmEEphBBCmLsGWcD37NlD7969uXLlCqBfQu/YsWO89tpr9RyZEEIIUTsaZAE/ceIE33//fblF9eWyMCGEEA1Jg6hqeXl5LF261LCAwMcff8zx48fx8fGp58iEEEII0zDZLHRN0yyBz4GXgTvAh0qpM2UenwxMAe4CC5VSO2u6r/DwcBISEmjTpg3jxo2jcePGuLtXvvqREEIIYe5MeRnZSMBGKeWpaVpvYCnwDoCmaW2AacCrgA3wtaZp+5VSd6o6eHZ2tuH/ERERODs7895779Vm/EIIIcRTy5SH0PsBewCUUkfQF+sHXgcOK6XuKKWuA2cA46vNP8Tf359jx44B+nVsw8PDTd75RQghhHhamLKANwOul/m5RNO0RpU8dgMo3zuuApqmzdM0rVTTtNLOnTsf6tWrlwX65efkVou3KVOmzK/vGBrqTXIr+TXXm+TWtDdN0+ZRTaYs4PlA07L7UkrdreSxpkDe4wZUSs1TSlkopSyAgbUWqXjY3PoOoAGT3JqW5Nd0JLemVe38mrKAHwZGANw/B368zGPfAv01TbPRNK054A78aMJYhBBCiAbFlJPYtgLDNE37P/SHCCZpmhYMnFFK7dA0bTnwd/QfImYrpQpNGIsQQgjRoJisgCul7gGBD919qszja4A1T7CL+U+wrTBOcms6klvTkvyajuTWtKqdX7PtBy6EEEI8yxrESmxCCCHEs0YKuBBCCGGGpIALIYQQZkgKuBBCCGGGpIALIYQQZsiU14E/sbrsaPasqUJugwC/+z/uVkrJJSTV8Lj8lnnOLmC7UmpV3Udpnqrwtzuc/6xqdQyYqpSSy22qqAr5/RQYC9wDopVSW+slUDOmaZoHEKOU8nrofm8gEn1NS7x/uXWlnvZv4IaOZkA4+o5mQLmOZn2BN4DPNE17rl6iNE/GcusKjAP6AJ7AbzVNq1azGVF5fstYCLSs06gaBmN/u02BJcDbSqnewHmgVX0EacaM5dcB/fuuJ/Bb4H/qJUIzpmlaKLAWfSfOsvdbA3Ho8zoQCLhf5yr1tBdwk3Y0e8YZy+0F4E2lVMn9BXmsAVkpr3qM5RdN00aj/wbzZd2HZvaM5bYP+mWbl2qa9ncgVyn177oP0awZy28B8DNgf/92r86jM39ngVEV3O+OfqXSa0qpIuBroL+xgZ72Al7rHc2EQaW5VUoVK6Uua5pmoWlaLJCulDpdL1Gar0rzq2nai8Dv0R8qE9Vn7H2hFTAICAOGAzM0Tetax/GZO2P5Bf0H/BPoT08sr8vAGgKl1BdAcQUPVbumPe0FvNY7mgkDY7lF0zQb4H/vP+ejOo6tITCW3/FAO+AAMBEI1jTtzboNz6wZy+0V4DulVI5S6iaQBvSo6wDNnLH8DgfaAi5AR2Ckpmmv13F8DVW1a9rTXsClo5npVJpbTdMsgO3AD0qpKUqpkvoJ0axVml+lVKhSyuP+BBYd8N9KqT31EaSZMva+cBR4UdO0Vve/NfZG/21RVJ2x/F4DbgN37jegygMc6jzChukk0EXTtJaapjUGBgDfGNvgqZ6FjnQ0M6VKcwtYoZ9E8dz9Gb0AM5VSRv+YRDlG/3brNzSz97j3hZnA3vvP3aSUkg/21fO4/A4Fjmiadg/9edr99Rir2dM07fdAE6XU6vt53ou+piUqpX4xtq00MxFCCCHM0NN+CF0IIYQQFZACLoQQQpghKeBCCCGEGZICLoQQQpghKeBCCCGEGZICLoQQQpghKeBCCCGEGZICLoQQQpih/wf9yi29kHHa2gAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "plot_model(calibrated_rf, plot='calibration')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that how different the above 2 plots look, one is before calibration and one is after. A perfectly calibrated classifier will follow the black dotted line in above plots. Not only `calibrated_rf` is better calibrated but if you notice that `AUC` has also improved from `0.7350` to `0.7706`. By default `calibrate_model()` uses `sigmoid` method which corresponds to Platt's approach. The other available method is `isotonic` which is a non-parametric approach. See an example of calibration using `isotonic` method below: " - ] - }, - { - "cell_type": "code", - "execution_count": 33, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
AccuracyAUCRecallPrec.F1Kappa
00.82020.76810.29460.73240.42020.3359
10.81950.77680.30030.72110.42400.3379
20.81770.80480.28610.72140.40970.3249
30.81390.74480.30030.67950.41650.3250
40.81700.75830.29180.71030.41370.3270
50.81700.78030.29180.71030.41370.3270
60.81520.74820.30310.68590.42040.3295
70.82390.78900.28900.77270.42060.3413
80.81580.76020.28900.70340.40960.3224
90.81130.76920.28900.67110.40400.3123
Mean0.81720.77000.29350.71080.41520.3283
SD0.00330.01750.00560.02780.00590.0080
\n", - "
" - ], - "text/plain": [ - " Accuracy AUC Recall Prec. F1 Kappa\n", - "0 0.8202 0.7681 0.2946 0.7324 0.4202 0.3359\n", - "1 0.8195 0.7768 0.3003 0.7211 0.4240 0.3379\n", - "2 0.8177 0.8048 0.2861 0.7214 0.4097 0.3249\n", - "3 0.8139 0.7448 0.3003 0.6795 0.4165 0.3250\n", - "4 0.8170 0.7583 0.2918 0.7103 0.4137 0.3270\n", - "5 0.8170 0.7803 0.2918 0.7103 0.4137 0.3270\n", - "6 0.8152 0.7482 0.3031 0.6859 0.4204 0.3295\n", - "7 0.8239 0.7890 0.2890 0.7727 0.4206 0.3413\n", - "8 0.8158 0.7602 0.2890 0.7034 0.4096 0.3224\n", - "9 0.8113 0.7692 0.2890 0.6711 0.4040 0.3123\n", - "Mean 0.8172 0.7700 0.2935 0.7108 0.4152 0.3283\n", - "SD 0.0033 0.0175 0.0056 0.0278 0.0059 0.0080" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "calibrated_rf_isotonic = calibrate_model(rf, method = 'isotonic')" - ] - }, - { - "cell_type": "code", - "execution_count": 34, - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAEuCAYAAABxmEC5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3hUVfrA8e9MeoWETiAktEOTGiH0agVEEUVhARdBcVFEiqJYkBVFFAUVVgFFLKir4CqirAgEEAJIL5GTBEikSSchpE75/TGT/CILySSZyaS8n+fJw8zcuee8c2eYd869pxisVitCCCGEKF+M7g5ACCGEEEUnCVwIIYQohySBCyGEEOWQJHAhhBCiHJIELoQQQpRDksCFEEKIckgSuCi3lFIeSqlJSqmdSqm9Sqk4pdTrSikfB/a1KqWqK6XuUkq9Y38sRik1pIQxRSqlVthv11VKbS1JeQ7W2UspddCB572olBpUwrraKqWWFnGfCKVUmv32OKXUtEKef8PXo5SaqZQaab99vfewv1JqZlHicwelVAel1CJ3xyHKN093ByBECfwLCAH6aq1TlFIBwOfAEmCEIwVorb8HvndiTA0AZS/7FNDFiWWXVB8grrg7K6WMwIfAXcUtQ2v9fnH3te//4nUey/8e3gyElqSO0qC13qWU8lRKDdBa/+DueET5JAlclEtKqQhgOFBHa50KoLW+qpQaB3S1P6cpsAAIAuoAe4GhWuvMfOU8BAzRWg+wP3SPvYXoD3yutZ5lr2sz8DsQAfQE/g4MAvyAAGAKtiSyBAhTSv0XeBQ4qLUOVEp5AW8BfQEzsB14Smt9RSmVBHxs3xYOfKK1fuE6rzkJ+AK4BagKzNVa/+ua51Sxv+a2gBX4CXjOHksU8IZSygycs8fjYX/ea1rrFYUc9vuBY1rrk/ni2Q60ttexA3jP/hq8gC+11q9eE98MoLrW+nGl1AD7ft5ATWBZvtcdqJT6BmgMXAYe0VrHK6U+th/TN/OV+RAwBPgnMA7wUEqlAB2Bf2utF9uf9zxQTWv91DUxdQLewfY+ZgNTtNbrlVJWoIbW+rz9eVagBtAKmA9cBQKBQ8BOrfVc+/MeA3pprYcqpQYCz9tfY7q97Fh71Yuw/QiVBC6KRU6hi/KqA3AoN3nn0lr/mS8RjcWWFKKxJYJIoH8h5QYD0fa/vyml7rA/Xg/4p9a6KbYv437YvqRbA9OBmVprMzAGOKK1vu2acp8H6gJt7H9G4I182wO11t2xtdinKKUibxBfKLZWZi9gplLqpmu2vwNcAG7ClrDbYEsaC4CdwFSt9bfAy8BbWusOwGhsrfPCDOF/k81BrXVze5mfAh/Zy+wI9FNK3X+9gpRSBmAyMEprHYXteD+rlKpuf0p9e3xtgeX2sguktd4OvA98pbWeju2HzFh7fUbgYfv2/HF4Af/B9v61sj9/vv35BWkFPGh//xcBD+Xb9hCwWCnVBHgVuFNr3Q54BFhpP1OE1nob0KiA91qIAkkCF+WVhcI/v88A55RST2Nr6dTF1mIqyBKttcn+w+AbbK1dABMQC6C1TgZGAsOVUrOxtfoKK/cO4H2tdY7W2gK8a38s13f2sk8CZ7nxaeAFWmur1voEsAa49Tr1vGd/Tha2hHXHtYUA/wYWKKU+x/Zj6LlC4gdoBiRe89hmAHtS6gn8Uym1F9iGrSXe9noFaa2twECgg1LqJWxnAwzYWsEA+7XWuf0HPgai7GcXimIVUEsp1Qa4DdvZA33Nc24CzFrr1fa4dmmtb7K/RwU5bv8cAMQAvkqpKKVUC2yt9HXYPjt1gHX2Y/I5ts9t43zlHMN+yUWIopIELsqr7UBzpVRQ/geVUmFKqdVKKT9sp5sfAZKBt4Hd2JJEQcz5bhuBHPvtLK21yV5He2zJPBj4GXjdgXJzT1XnL9sr3/2MfLetBZRnuqYM8zXbjYXUA4DW+gNsyWsttuS2XynlW0D8uXFd+52RZv/Xwx5zF611W3vLORpbC/R/2BP+HqA9tvdlKrZjnfu6r31dVv7/vXCI/YzIB9jOMIzmmta3nYm/Hi+UUq2UUrmXFw32x7yv2S/3def+GPkQ24+6vwMf2h/zANblHo98xyR/B70c/ve1CuEQSeCiXLJ3EPsc+EgpFQxg/3chcEFrnYEtMc3UWn9l360Tti/VgoxUShmUUiHYrvmuuc5zemC75vkWsBG4O1+5Jq6TMO3lPKaU8rKfnh2PLXkWVW4P7HBsre+frtn+X+Bx+2vwwfYDJreevNjsvePbaa0/tj+nKlC7kLo10Oi6G2xnLLYBk+zlVwW2YOsncD1NsP0Ael5rvQrbJQEf/v84tlFK5bbeHwV+1VqnFxIf/O/xXwLcg+0sw7c3eE1WpdQt9rjbA+uxfTeew3YZAmBYIfV+jK1z331Abi/9dcCtSqlm9rLvBPZj6zeRexmhgT0GIYpMErgoz/6BrVf1Vvspyu32+2Ps258DvlVKHcDWEtvIX09fXk8KsAvYCryrtd5wned8AVRXSv1ury8NCLWfDYgDMpVSO/hrK/oV4E9sHel+x5ZknizaywUgUim1C9sPggnXOSU8AVuHsAP2Pw3Msm/7HnhNKTUKeBrbNfQ92E4Bv6y1Tiqk7m+A2wvYPgyIth/v7cAXWuvPb/Dc/diupx+2H8eB2I5d7vvzO/CSUmoftsQ4qpDYcq0HblNKvQugtT6L7dr/F1rr/2nB2y8zDLbXtRdbK32w1job27FcoJTaDTQHTt+oUq31n9jOJOy3/7hEax2H7cfRl/bX8U/gLq11bus9Clt/iT8cfG1C/IVBlhMVonyw9/oeorXe6ab6PbD9uOmf2xO9rLN3ivsN6KG1Pu7uePKz96j/Ovf6uxBFJS1wIYRD7NeUx3KD69pljVJqLLaW/BtlMHl3AKySvEVJSAtcCCGEKIdc2gJXSnVSSsVc5/GBSqnflFKx9l/JQgghhCgClyVw+9jbJYDvNY97YRvScyu2caOPKKUK6/0qhBBCiHxc2QI/gq1357WaA4la60v2np6/At1dGIcQQghR4bhsLnSt9Qr7HNLXCsY2VCfXFcChGZbs8yi/BHDnnXcyaFCJFlYSN3D+/HmqV69e+BNFkcmxdS05vq4jx9a5zGYzHh62aQ9SU1O56667CpsM6n+4YzGTVGyLS+QKwrZYQaG01jOAGQAxMTHWXr16OTk0ARATE4McW9eQY+tacnxdR46t8yxZsoRZs2axY8cOatSoUexy3DGM7HegiVIq1D49YQ/sc0wLIYQQFV16ejqXL1/mwIEDJSqn1BK4UmqYUuoR+2xIk7BN+RiLbfWicjEphBBCCFFUR44cYfLkyZjNtmnvH3vsMeLj4+nTx5FFAG/MpafQ7VMzRttvL8/3+CpsKwUJIYQQFdqsWbNYunQpXbt2ZfDgwXh5eZXo1HkumYlNCCGEcCKr1Ups7P9fGZ41axZffvkl99xzj1PrkQQuhBBCONGYMWPo2rUrO3fali2oU6cOQ4cOxWAockfzArmjF7oQQghRoZhMJjw9bSl1xIgRpKamUrNmTZfWKS1wIYQQogQWLlxI06ZNuXjxIgC9evXi66+/Jjw83KX1SgIXQgghSiA9PZ2UlBQOHjxYqvVKAhdCCCGKQGvN5MmTsVgsAEyYMIGEhAR69OhRqnFIAhdCCCGKYObMmbz11lv89NNPAHh7exMaGlrqcUgCF0IIIQpgsVjYunVr3v3Zs2ezYsUK7rzzTjdGJQlcCCGEKNDIkSPp3r07+/btA6B+/foMHjzY6cPCikqGkQkhhBDXyMnJwcvLC7AlcKvVWuZWY5MWuBBCCJHPvHnzaNasGZcv2xbKvPXWW/n8888JCwtzc2R/JQlcCCGEyCc9PZ20tDQOHz7s7lAKJAlcCCFEpRYXF8eUKVOwWq0ATJo0ifj4eKKjo90cWcEkgQshhKjUXnrpJebOncv69esB8PX1pUqVKm6OqnCSwIUQQlQqZrOZLVu25N2fM2cOq1atKvH63KVNErgQQohK5YEHHqBnz54cOnQIgMjISAYMGOD2YWFFJcPIhBBCVHj5h4U99NBDBAQEUK1aNTdHVTLSAhdCCFGhvf766zRr1owrV64A0L9/fz7++GNq167t5shKRhK4EEKICi0jI4P09HS01u4OxakkgQshhKhQ9u3bx9SpU/OGhT3zzDPEx8cTFRXl5sicSxK4EEKICuXFF1/kzTff5NdffwXAz8+PoKAgN0flfJLAhRBClGsmk+kvw8Lmzp3LmjVr6N69uxujcj1J4EIIIcq1e+65h969exMfHw9A48aNue2229wclevJMDIhhBDlTv5hYaNHj6ZOnTqEhIS4OarSJS1wIYQQ5crMmTNp0aIFV69eBWwt8EWLFlGjRg03R1a6JIELIYQoVzIzM8nMzCQxMdHdobiVJHAhhBBl2s6dO3n66afzhoU9//zzHD58mDZt2rg5MveSBC6EEKJMe/7553njjTf47bffAPD39ycgIMDNUbmfJHAhhBBlSnZ2dt4YboD58+ezbt06Onbs6Maoyh5J4EIIIcqU/v37069fP44ePQqAUqrcLfVZGmQYmRBCCLfLzs7G29sbgLFjx6KUomrVqm6OqmyTFrgQQgi3ev7557npppvIyMgA4P777+e9994jNDTUzZGVbZLAhRBCuFVWVhbZ2dkcO3bM3aGUK5LAhRBClKpt27Yxbdq0vPszZswgLi6OFi1auDGq8kcSuBBCiFL13HPP8frrr7N3714AAgIC8PPzc3NU5Y8kcCGEEC6VmZn5l2Fh7733Hps3b6Zt27ZujKr8k17oQgghXOrWW29l586dHD58mPDwcDlV7iSSwIUQQjhdVlYWPj4+ADz66KNERUURHBzs5qgqFjmFLoQQwmmsVitTp06lXbt2ZGdnAzB8+HDeeustGdftZJLAhRBCOI3BYMgbFpacnOzucCo0SeBCCCFKZNOmTTz33HN591999VUOHTpEkyZN3BhVxScJXAghRLFZrVamT5/Oa6+9RlxcHACBgYF517+F60gCF0IIUSQZGRl5w8IMBgMLFy5k27Zt0ru8lLmsF7pSyggsBNoAWcAYrXVivu1TgAcBC/Cq1vpbV8UihBDCOaxWKz179iQuLg6tNWFhYdx0003uDqtScuUwsrsBX611Z6VUNDAXGASglKoKTAAaAwHAXkASuBBClFG5PcoNBgPjx4/n999/l2FhbubKBN4NWAOgtd6mlIrKt+0qkIwteQdga4ULIYQoY6xWKxMmTGD16tVorfHy8mLUqFHuDkvg2gQeDKTku29WSnlqrU32+8eBOMADeM2RApVSM4CXAFnc3YWSkpKIiYlxdxgVkhxb15Lj6xp//PEHmZmZfPPNN9SpU8fd4VRIvXr1KvI+rkzgqUBQvvvGfMn7DqAOEGm//1+l1Bat9Y6CCtRazwBmAMTExFiL84JF4WJiYor1YRKFk2PrWnJ8nWPdunVs2LCBV155BYAOHTqwfft2+vXr5+bIRH6u7IW+BbgTwH4N/EC+bZeADCBLa50JXAZkih4hhHAzq9XKc889x2uvvUZioq3fcVBQEJ6eMvN2WePKd+Rb4Bal1FbAAPxdKTUJSNRaf6+U6gdsU0pZgF+BtS6MRQghxA1cvXqVPXv20K1bNwwGA4sWLcJkMtG4cWN3hyYK4LIErrW2AOOuefhwvu0vYb+eLYQQwj2sVitdunQhKSmJ+Ph4atWqRZs2bdwdlnCAnBMRQohKKDMzE19fXwwGA0888QTJyckEBga6OyxRBJLAhRCiErFYLIwbN44dO3awc+dOPD09GTNmjLvDEsUgU6kKIUQlYjQaMZvNmM1mTp8+7e5wRAlIC1wIISq4NWvWsHXrVmbOnAnAvHnz8PPzK7We5RHTVt9wW9Ls/qUSQ0UkLXAhhKjALBZL3rCw3PW5ZVhYxSDvoBBCVDCpqakcOHCArl27YjQaWbJkCZ6enjRo0KBU40jLMrEr+VKp1lmZSAIXQogKxGw206lTJ/78808SEhKoXr067du3L5W6L17N5reki+w4dpHfki5y8GQKFmupVF0pSQIXQogKIHdYmIeHB08++STnzp0jICDApXWeTslgx7GLeX8JZ9Pytnl5GGgXHkLHyFD+FXPkhmXM+yWeMd0bEugj6aio5IgJIUQ5Zjabefjhhzl06BDbt2/HaDQybty1c2iVnNVq5dj5q+w4doHt9hb28YsZedv9vDzo1rg6N0eE0jEylLb1q+Ln7QFQSAJP4JPYZB7v3Zjh0eH4eHo4PfaKShK4EEKUYx4eHphMJkwmE2fOnHHaamEWi5XDf17JOyW+WWeQ+t+YvO3Bvp70a16TjpGh3BwRSquwKnh5FL1f9KRbmrJo01Fm/hDHh78eY9ItTbm7XRgeRoNTXkdFJglcCCHKEavVyqpVq9izZw8vvWSbjXrhwoUEBATg4VH81mu2ycLBUyl5p8N3Jl0kNdOUt72qj4EBrevQMdLWwm5aMwijg0m2sKFif4tuwIINiXwam8zkr/exaNNRnr5d0adZTQwGSeQ3IglcCCHKEYvFwvTp04mPj2fs2LHUrVuX4ODgIpeTkW1mzx+X2GFvYe/+4xKZOZa87eGh/tzasrYtYUeEcuzADnr3dk1nuNAAb14Y0ILR3SJ5e208K3ef4OFlO4lqEMIzdzTj5ohQl9Rb3kkCF0KIMu7y5cscOnSIrl274uHhwdKlS/H396du3boOl5GSkcOu5Iu269fHLnLgZAo55v/vIq5qBdlOh9sTdu0qvn/ZP6kUWsJhVf148742PNKjIW/8V7M27gz3vR9L32Y1mXq7olntov9QqcgkgQshRBlmMpno0KEDKSkpJCQkEBISQlRUVN72gmY5WzCsPb8l2ZL24T9TsdrztYfRQKuwKnSMCKFjZDWiGoQQEuDt6pfisKa1glg8MopdyZd4fc1h1h0+y3p9lnvahvHULU2pH+rv7hDLBEngQghRBmVkZORNd/rUU0+RlpaGv3/REtf45bsB8PY00jEilE72Fnb78BACysGwrQ4NQvjqkWhi4s8xZ41m5Z6TrNp/iuGdGvB4n8ZUD/Rxd4huVfbfQSGEqERycnIYNWoUx44dY8uWLRiNRh5//PFilTX1NkWnyFBuqlel3A7PMhgM9FY16dmkBqv2n2Luz/F8vDWJr3ceZ0z3hozpHkmQr5e7w3QLmQtdCCHKEC8vr7zVwi5cuFCissb3bkxURGi5Td75GY0GBrUN45dJPZk5qCV+3p7MX5dAzzdi+OjXY2SZzO4OsdRJAhdCCDeyWq2sWLGCV155Je+xxYsXs23bNmrUqHHD/bJNFv75Q1xphFimeHsaGdk5go1TezH5lqbkmCzM/CGOPm9u5JtdJzBXorlbJYELIYQbmUwmpk+fzquvvsq5c+cACA4Oxmi88dfzqcsZPLAolg9/PVZaYZY5AT6ePNG3CRuf7s2YbpGcS8tiytf7uGP+JtbGncFqrfiJXBK4EEKUsgsXLrB161bAdsr8k08+Ye/evQW2uHPF6LP0f2czu/+4zKC2jg8jq6hCA7x5fkALNkzpxX0d6pF4No2xn+zkvvdj+S3porvDcynpxCaEEKUoOzubdu3akZWVRUJCAsHBwXTs2LHQ/cwWK/N+iee9DYl4GY28cncrhncKZ/4D7Uoh6rIvrKofb9jHkL/5s+a/h2xjyPs0q8nU2xTN61S8MeSSwIUQohSkp6fj7++Pt7c3kyZNwmw24+vrW/iOwNkrmTz5xV5ij16gfqgf/xregVZhVVwccfnUpFYQH4yIYvcfl3j9p8OsP3yWDfosd7cNY1IFG0MuCVwIIVwoKyuLYcOGce7cOTZu3IjBYGDixIkO77/t6AWe+GIP565kcWuLWrxxXxuq+FXOYVNF0T48hC8fiWajfQz5t3tO8oN9DPn43o2pEVT+x5A7dA1cKVXH/m93pdR4pZSfa8MSQoiKwcfHB4vFgtls5tKlSw7vZ7FYWRiTyLDF27h0NZvn+zfngxEdJHkXgcFgoJeqyQ9PdGP+A22pU8WPj7cm0fONDby1Np4rmTnuDrFECm2BK6X+BXgrpeYCy4Gfgc7A31wcmxBClDtWq5Uvv/ySpKQknn32WQCWLVtGUFCQwytrXbqazeSv97H+8FlqB/vy3rB2RMmCHsWWO4b8jlZ1+Grnceb/ksA76xL4bFsy43s35m/ldB1yR1rgHYExwP3Ah1rrhwHl0qiEEKKcys7O5oUXXuDVV1/Na3EHBwc7nLz3Hr/MgHd/Zf3hs3RvUp3VE7pJ8nYSb08jI6IbsOnpXky9TZFjH0tfXseQO3IN3ANboh8EjFNK+QMBLo1KCCHKkXPnzpGYmEjnzp3x8fHhs88+o1atWoSEhDhchtVqZdnWJGb9+Dsmi5Wn+jXl8T6N8XBwzW3hOH9vT8b3bsywjuEsjElkWWwyU77ex6JNR4g/k3bD/Qpb17y0OZLAPwFOA1u01tuVUnHAB64NSwghyofMzExat26N0WhEa01gYCDR0dFFKuNKZg7TVhxg9YHTVA/0Zv4D7ejauLqLIha5QgK8md6/BX/vGsn8XxL4etdxd4dUJIUmcK31W0qpeVrr3JXee2itz7s4LiGEKNNyh4X5+voyZcoUvL29HR4Wll/cqVTGL9/NsfNX6RgRyrvD2lEruOjliOKrW9WP14e0ZmyPSPq9tcnd4TjMkU5sDYAlSqkIoAfwuVJqtNY6ycWxCSFEmZORkcHQoUO5evUqv/zyCwaDgcmTJxerrH//dpwXvjtIlsnCuJ6NmHJrUzw9ZIJMd2lcM8jdIRSJI5+UD4A3gDTgT+ALbKfVhRCi0vHz88NkMmE2m0lNTS1WGRnZZqZ8vY+nV+zHx9PIkpFRTLujmSRvUSSOfFqqa61/BtBaW7XWi4GKNyedEEJch8ViYdmyZcyZMyfvsa+++ooNGzZQpUrRZ0M7ci6Nuxds4ZtdJ2hdrwqrJ3SnX4tazgxZVBKOdGLLUErVA6wASqluQJZLoxJCiDIiKyuLl156icuXLzNu3DiCg4MJCireqdZV+04xbcV+rmabGdm5AdP7Ny+X449F2eBIAp8E/AA0UkrtBUKB+1walRBCuNHp06dJTk4mOjoaPz8/li9fTr169QgOLt7JxyyTmVd++J1PtyUT4O3Buw+2Y2AbWUmsLCprQ8UK4kgCPwbcDDTFNib8sNY626VRCSGEm1y9epXWrVvj5+fH4cOH8ff3p0uXLsUu7/jFdMYv383+EymoWkEs/Ft7GtUIdGLEorJyJIHvAfYCnwHfSfIWQlQ0VquV9PR0AgICCAgIYMqUKVStWhUfn5ItePFL3Bkm/XsvqZkmhnSoxz8HtcLPW06ZC+dwJIE3APoADwKvK6U2AJ9qrde7NDIhhCgFaWlpDBkyBIPBwI8//ojBYOCZZ54pUZk5Zgtv/qz5YONRfDyNzLm3NfffXN9JEQth48hELhbgF+AXpVQvYC7wLSCL0Qohyr2AgADMZjNgO30eGFiy09t/pmTyxBe7+S3pEpHVA1g4vD3N68jAHeF8jkzk0g4YBgwGNP+fwIUQotwxm80sXbqUlJQUJk+ejMFgYOXKlQQGBjq84MiN/Jpwnie/3MOFq9n0v6kOs++9iSBfWf5TuIYjp9CXYJu4pYvW+oyL4xFCCJfKzMxkxowZZGVl8dhjj+Hv71/sYWG5zBYr765PYP66BDyNBl6+qyUjOzco8Q8CIQpywwSulKqttf4TW8vbCvgopcJzt2ut/yiF+IQQosROnDjByZMn6dSpEwEBAXz11VdERkbi7+9f4rIvpGUx8au9bE44T1hVPxYMb0/b+lWdELUQBSuoBb4EGADEYEvg+X9KWoGGrgtLCCGcIzU1lVatWhEaGkpcXBy+vr507drVKWX/lnSRJ5bv4c/UTPo0q8lb97ehqr+3U8oWojA3TOBa6wH2mx201hfzb7MvbFIgpZQRWAi0wTZz2xitdWK+7XcAL9nv7gbGa63L12rqQogyyWq15nVICw4OZurUqdStWxdvb+ckV6vVyuLNR3l9jcZqtfL07YpxPRphlLW7RSkq6BR6fWyt7h/tydaQb58fgWaFlH034Ku17qyUisbW+W2QvewgbAuk9NJan1dKPQ1UB86V5MUIIURKSgr33HMPQUFBfPfddwBMnz7deeWn5zDlm32sjTtDzSAf3nmwHdENqzmtfCEcVdAp9JeB3kBdIP8CqSZsU6sWphuwBkBrvU0pFZVvWxfgADBXKdUQWKK1luQthCix4OBgLBYLJpOJjIwM/Pz8nFb2gRMp/GP5Lo5fzKBLo2rMf6AdNYJKNtmLEMVV0Cn00QBKqWe01q8Xo+xgICXffbNSylNrbcLW2u4NtMW2TOlmpVSs1jq+GPUIISoxk8nEokWLyM7Opm3bthgMBn744YcSj+eOmLb6htsm9GnMk/2a4iGnzIUbFXQK/RGt9SLAVyn14rXbtdYzCyk7Fcg/NsNoT94AF4Df7L3cUUptwpbMC0zgSqkZ2K+b9+nTp5DqRXElJSURExPj7jAqpMp8bB9ac/WG2z6+PaDY5aanp/PCCy8AMHv27GKXUxTtvU+zedPpUqmrrKjMn93S0KtXryLvU9ApdMMNbjtqCzAQ+Lf9GviBfNt2Aa2UUtWBy0A0sLiwArXWM4AZADExMdbivGBRuJiYmGJ9mEThKvWxXXPjFm2j1h0xWayY7X8miyXvtu2+FYv9X7PFyukzZ7h4KYXIho0wVbUwdcHX1KhZixMnT+HvF2nbx2zf32r9//tWK2aLxVaOOd+2a8o3W6zAjX9wVMb3sFJ/dsuogk6hf2D/92WllLfWOlsp1RhQwE8OlP0tcItSaiu2HwB/V0pNAhK11t8rpZ4F/mt/7r+11gdL9EqEEOVW9zkbirfjtt357iTZ/jkkXyWicnBkKtUXgBZKqWewdWY7BNwCTCxoP/sc6uOuefhwvu1fAl8WNWAhRPlz8GRKgduHdKiHh8GAh4cBT6MBD6PhL/eNBgMWUw7+fr54GA2sX/cLNaqF0vHmKDyMxrx94vVhWrVsgafRaCvDmK+8fPeN+R63PZe8cnK3dXp1XSkdHSGKx/45wikAACAASURBVJGpVO/G1qP8SeAzrfXTSqmdrg1LCFER/H46lbfXxvNzXMGzML95X5sbbrt48SJ33303tWrV4uuvvwZgXM9G131uTNoRerUNK37AQpQjRkeeo7XOwDYr24/2CVqK3+NECFHhJZ69wvjlu7lj/mZ+jjtD+/DiTy0aEhKCyWTCZDKRnZ3txCiFKN8caYGvU0odBNKxnULfCHzv0qiEEOXSsfNXeWddAt/tPYnFCq3rVWHSLU3p2bQGkc/+6FAZOTk5LFiwAE9PTx5//HEMBgNr164lIKB02w1Js/uXan1CFJUj64FPUUq9A5zQWluUUk9orfeWQmxCiHLi+MV03lmXwMo9JzFbrDSvE8ykW5rSr3nNvBW5HE2IV69eZdasWfj5+fHII4/g7e1d6slbiPLAkU5sNYA3gT5KKU9gg1JqnCwtKoQ4dTmDd9cn8vXO45gsVprUDOSpW5pye8vaRZoXPDExkZSUFDp06EDVqlVZuXIlzZo1c9rc5UJURI6cQv8A2AqMwXbN/BHgQ2zXxIUQldDZ1EwWbEjkix3HyTZbaFg9gCf7NWFA67pFnp3s7NmztG7dmsjISPbt24enpyfdu3d3UeRCVByOJPCGWuvB+e7PUUqNcFVAQoiy63xaFu/HHOHTbclkmSyEh/ozoW8T7m5bF08PR/rE2lgsFtLT0wkMDKRmzZpMnTqVli1b4uHh4cLohahYHEngVqVUfa31cQClVDiQ49qwhBBlyaWr2Xyw6SjLtiaRkWMmrKofT/RpzL0d6uFVhMQNthb3XXfdRaNGjfj8888BePnll10RthAVmiMJ/AUgVim1HduMap2wnUYXQlRwKRk5fLj5KB9tSSIty0StYB+eu7MZ999cHx/P4rWWq1evjtlsxmw2YzKZ8PR05GtICHEtR3qh/6CUagd0xJbAx2mtz7o8MiGE21zJzGHpliQWbz7KlUwT1QN9mHRLU4Z1CsfXq2iJOysri3feeYegoCDGjRuH0Whk48aN+Pv7uyh6ISoHR3qhVwWmA32xnTr/USk1yz65ixCiAknPNrFsazIfbDrC5fQcQvy9ePaOZozo3AB/7+K1lNPS0njttdcICQlhzJgxeHp6SvIWwgkc+R/5GbY5zIdh64X+d2AJMNyFcQkhSlFmjpnPtiXzr5gjXLiaTbCvJ1NubcpDXSMJ9Cl64j58+DAZGRm0a9eOatWq8Z///IdWrVrJ6XIhnMiR/00RWuv8Q8Ym2mdmE0KUc1kmM1/uOM6CDYmcvZJFkI8nT/ZtwsPdIwn29SpWmSdPnqRNmzY0b96cXbt24eHhQY8ePZwcuRDCkQR+SCnVXWu9GUAp1RpIcG1YQghXyjZZ+HrXcd5bn8jplEz8vT34R69GPNKjIVX9iz55itlsJiMjg8DAQMLCwpg6dSpRUVEYjUXroS6EcJwjCbwZsFEppQEztvXALyqljgFWrXVDVwYohHAek9nCyj0neWddAicuZeDrZeSRHg15tEdDqgX6FKvM06dPM2DAAFq3bs3SpUsBeOWVV5wZthDiOhxJ4He5PAohhEuZLVa+33eS+b8kkHQhHW8PIw91ieAfvRpRM9i3RGXXrFkTi8WSNzRMJmMRonQ4MowsuTQCEUI4n8Vi5ceDp5n3SwKJZ9Pw8jAwvFM4j/dpTJ0qfsUqMzMzk7lz51KrVi3GjBmDh4cHW7ZskZ7lQpQy6RIqRAVktVr576EzzPslnsN/XsHDaGBoVH0e79OY+qElS7SpqanMmTOHunXrMnr0aIxGoyRvIdzghglcKTVUa/2VUqq61vp8aQYlhCgeq9XKBn2Wt9bGc/BkKkYDDG4XxoS+TYioXvwlOQ8ePIjZbKZNmzbUrFmTVatW0aZNG+mkJoQbFdQCn6WUWgH8DLQvpXiEEA6ImLa6wO0GAwxsU5cn+zahcc3AEtWVnJxMu3btaNOmDTt27MBoNMqwMCHKgIIS+CYgCzAopSzXbLNqraWnihBl0O0ta/PULU1RtYOKXUb+YWENGjRg6tSpdOvWTVrcQpQhN0zgWuvRwGil1Hda60GlGJMQogTeH9GhRPsfP36cAQMGEB0dzQcffADAq6++6ozQhBBO5Egv9EFKqTuwzYXuCazXWn/v8siEEP/j5OUMPt/m2oEhtWvXxmKxYLFYsFqtGAwGl9YnhCgeRxYzmQoMAT7HthrZ80qpm7TWs1wdnBDC1jEt9sgFlsUmsTbuDBarc8tPT09nzpw5RERE8NBDD+Hl5cWOHTvw8yveMDMhROlwZBjZCKBT7upjSqnFwC5AErgQLpSWZeLb3SdYFptM4tk0AFqFBTOqcwRTv9nvtHpSUlKYO3cujRo1YtSoURgMBkneQpQDjiRw4zVLh2YCJhfFI0Sll3g2jU9jk1ix+yRpWSa8PAzc3bYuI7tE0K5+VQwGQ4kT+J49e/D09OSmm26iTp06/PDDD7Rv315OlwtRjjiSwNfZh5N9bL8/CljvsoiEqITMFivrfj/DJ7HJ/Jpom3ahdrAvj/ZoyAMdw6kR9Nd5ypNm9y92XQkJCURFRdGpUye2bNmCwWCgZ8+eJYpfCFH6HEngE4FxwEhs64GvBz5wZVBCVBYXr2bz1W/H+WxbMicv2050RTcMZVTnCPq1qIWXh3OGbeXk5JCdnU1AQABNmjRhypQp9OvXT1rcQpRjjvRCtwL/sv8JIZxg/4nLLNuazKr9p8g2WfDz8mB4p3BGdo4o0fjt6zl27BgDBgygT58+vPvuuwC8/vrrTq1DCFH6ZC50IUpJlsnM1lMm5i3Ywt7jlwGIrB7AiOgG3NuhHlX8vFxSb1hYGFarNe9PWt1CVAySwIVwsVOXM1i+/Q++2PEHF65mYzBk0bdZTUZ2iaB74+oYjc5NqGlpabz66qu0aNGCv/3tb3h7e7N79258fUu2bKgQomxxKIErpYKBKtjGgQOgtf7DVUEJUd5ZrVa2Hb3IJ7FJ/Bx3BrPFShU/L26P8OK5+7oRXs11q3ddunSJefPm0apVK4YPH47BYJDkLUQF5MhELs8B04AL+R62Ag1dFZQQ5dXVLBMr95zk09gk4s/Yxm63qBPMQ10iGNimLtu3bnZJ8t65cyf+/v60aNGC+vXr89NPP3HzzTfL6XIhKjBHWuAPA4201udcHYwQ5dWRc2l8GpvMil0nuJJlwtNo4K42dRnVpQHtw0NcmkgPHTpEx44d6dGjBxs2bJBhYUJUEo4k8D+Ai64ORIjyxmyxsuHwWZbFJrE5wTZ2u2aQD2O6N+TBjvWpGey609bZ2dnk5OQQEBBAy5YtmTx5Mv3795cWtxCViCMJPAH4VSm1AdssbABorWe6LCohyrBLV7P5987jfLotmROXbGO3O0aEMrJLA25rWdtpY7dvJD4+ngEDBjBw4EDmzp0LwBtvvOHSOoUQZY8jCfyk/Q/ydWITorI5eDKFZVuT+H7fKbLsY7cf7BjOyM4NaF4nuNTiCA8Pz2tpy7AwISovRyZyeVkpVQPoZH9+rNb6jMsjE6KURUxbfcNt7cOrsvsP29jtBtX8GRHdgPs61KeKv2vGbueXmprKzJkzufnmmxk6dCi+vr7s27dPepYLUck50gv9NuAjYBu2qVQ/UEo9rLX+wdXBCVFW7Dl+md6qBiO7RNCzSQ2nj90uyIULF3jvvffo2LEjQ4cOBZDkLYRw6BT6LKCb1voYgFKqIbASkAQuKo0Nk3sRUT2g1OqLjY0lJCSEZs2aERkZyZo1a4iOji61+oUQZZ8jvW28cpM3gNb6qIP7CVFhlGby3rNnD126dGHChAl5j/Xq1Uta3UKIv3BoGJlSaiLwof3+GCDZdSEJUfp+2H/KrfVnZmZisVjw9/enbdu2TJ48mXvuucetMQkhyjZHWtIPA52Bo0CS/fYjLoxJiFJjtVp5e208jy/f47YY4uLiaNmyJf/85z8BMBgMvPnmm3Tt2tVtMQkhyj5HeqGfBYYWtWCllBFYCLQBsoAxWuvE6zxnNfCd1vr9otYhRElk5piZ/PU+Vu8/Tb0Qv7wx3aUtIiJChoIJIYrshglcKfWD1nqAUuoYtrnP/0JrXdhc6HcDvlrrzkqpaGAuMOia57wChBYxZiFK7ExqJo98spN9J1KIahDC+yM6UD3Qp1TqvnTpEi+99BK9evVi8ODB+Pv7c+jQIXx8Sqd+IUTFUFALfKz9317FLLsbsAZAa71NKRWVf6NSaghgAX4qZvlCFMvBkymMWbaTP1Mzubd9PV4d3AofT49Sq//8+fO8//77/P777wwePBhAkrcQoshumMC11qftN9/SWt+bf5tSah3Qt5Cyg4GUfPfNSilPrbVJKdUKGAYMAV4sethCFM9PB07z1L/3kmWyMO2OZjzao2GpnL7euHEjdevWBaBJkyb8/PPPdO7c2eX1CiEqroJOoa8E2gJhSqmj+TZ5YVvgpDCpQFC++0attcl+eyQQBqwHIoBspVSS1npNQQUqpWYALwH06dPHgRBEcSQlJRETE+PuMJzKarWy6mgOKxNy8PGAJ9r60Mx6nI0bj7u87ri4OMaPH0+nTp0YN25c3rGNjY11ed2VTUX87JYVcmxdq1evXkXep6BT6A9huz49H3iC/58H3QQ4MpXqFmAg8G/7NfADuRu01k/n3rYn5T8LS972/WYAMwBiYmKsxXnBonAxMTHF+jCVVZk5Zp5ZsZ/vEk4RVtWPxSOjaFHXtXOXp6enYzAY8PPzo2fPniQmJvLAAw+Qnp5eoY5tWVPRPrtliRzbsueGw8i01qla6yTgH8B4rXWy/fmv4VjHs2+BTKXUVuBt4Cml1CSl1F0lD1sIx5y9kskDi7bx3d5TtA+vyn/Gd3V58t63bx/NmjVj9uzZgG1Y2FtvvUXHjh1dWq8QonJxZCKXz4Av7bdPAZvtj91a0E5aawsw7pqHD1/neTMciEGIIos7lcqYZb9xKiWTu9vWZfa9rfH1cn1ntUaNGuHp6YnRKBMWCiFcx5EEXk1r/QGA1joLWKyUesy1YQlRMj8f+pOJX+0lPdvM1NsU/+jVyGWd1c6fP88LL7zAnXfeycCBAwkMDOT333+XnuVCCJdypImQrpS6I/eOUqofcNV1IQlRfFarlX/FHOHRz3ZhtcL7f2vP+N6NXdrT/Ny5cyxZsoRFixblPSbJWwjhao60wMcBnymlPrXfPw6McF1IQhRPlsnMsysPsHL3SepU8WXxyChahVVxSV3r1q2jYcOGREZG0rx5c9atWyfDwoQQpcqRqVT3Aq2UUtWAHK11quvDEqJozqdlMe7TXexMvkSb+lVZPKIDNYNds3rXpk2b6NevH/fccw8rV64EoEePHi6pSwghbqTQBG4fAvYsEAgYlFIeQAOtdYSLYxPCIYf/TOXhj3dy8nIGA9vU5Y0hzu+slpaWhqenJ76+vnTv3p2JEycycuRIp9YhhBBF4cg18I+A/2BL9guAE9iGiAnhdut+P8O9C7dy8nIGk25pyjsPtHV68t61axdKKd58803ANizs7bffpl27dk6tRwghisKRa+BZWuulSqkI4BK2WdQOFLyLEK5ltVpZsvkYr/70Oz6eRhYMa0//1nVcUlfjxo3x8vLCw6P05ksXQojCONICz1RKhQIaiNZamwH5JhNuk22y8MyK/cz68XdqBvnw70c7OzV5nzlzhocffpg1a2yTA1apUoX4+HieffZZp9UhhBAl5UgLfC7wFTAY2KGUGg7sdGlUQtzAxavZjPtsFzuOXeSmsCosHhlF7SrO7ax25swZPv74Y1JSUrj99tsB8Pb2dmodQghRUo4k8AzgVq211b4kaFNgn2vDEuJ/JZy5wsPLdvLHxXTuvKk2c+9ri5+3c04G/fTTT7Rs2ZLw8HBat27N+vXr6dq1q1PKFkIIV3Akgc/RWq8G0FpfBfa4NiQh/leMPssTy/dwJcvEhL5NmNi3CUajcyZnWbt2LXfeeSdDhw7lyy9tswb37NnTKWULIYSrOJLAjyilPgK2Y2uNA6C1/sRlUQlhZ7VaWboliVdWx+HpYWT+A20Z1DasxOWmpqbi4+ODj48Pffv25cknn2TMmDFOiFgIIUqHI53YLmBbSjQa6G3/6+XCmIQAIMds4blvDzLzhziqBfrw1SPRTkne27Zto2nTpsyfPx8Ao9HIvHnzaNWqVYnLFkKI0nLDFrhSKkxrfVJr/ffSDEgIgMvp2Tz22W5ij16gRZ1gloyKom5VP6eUrZTC19cXT09HTkAJIUTZVFALfFXuDaXU5FKIRQgAjpxL4+4FW4g9eoHbWtbim8c6lyh5nzp1ihEjRrBu3ToAQkJCiI+PZ9KkSc4KWQghSl1BTZD8PYSGYxtOJoRLbU44xz8+382VTBPjezdi8i2qxJ3VTp8+zeeff47FYqFv376ADAsTQpR/BSVwa77brluLUQi7T2KTeHlVHB4GA28PbcM97eoVqxyr1cr3339PVFQUYWFhdOjQgZiYGBkWJoSoUBzpxAZ/TeZCOJXJbOGF/xzkxe8OEeLvxRePRBc7eQOsXr2au+++m+eeey7vsR49eshUqEKICqWgFnhLpdRR++2wfLcNgFVr3dC1oYnKICU9h/HLd/Nr4nma1Q5iyago6oX4F7mcS5cuERgYiJeXF3fccQcTJkxg3LhxLohYCCHKhoISeNNSi0JUSsfOX+XhZb9x9NxV+jWvxfwH2hLgU/Se4Zs3b2bw4MFMnz6diRMn4uHhkTdETAghKqobfltqrZNLMxBRuWxNPM9jn+8mJSOHR3s25OnbmuFRzM5qzZs3x8/PDy8vLydHKYQQZZej18CFcJrPtycz8qMdpGebeGNIa569o3mRkvfx48d54IEH2Lx5MwDVq1fnyJEjjB8/3lUhCyFEmSMzWYhSYzJbmPXj7yzdkkRogDcfjOjAzRGhRS7n5MmTfPXVVwQEBNC9e3cAaX0LISodSeCiVKRm5vDE8j1sjD9H01qBfDjqZuqHOtZZzWq1smLFCrp3706tWrWIjo5m06ZNMixMCFGpSQIXThcxbfUNt/VWNXjnwXYE+TreYl65ciX33XcfY8aMYfHixQB5LW8hhKisJIGLUrVk1M0OXe++cOECVapUwdPTk0GDBjFhwgSeeOKJUohQCCHKB+nEJkqVI8l7/fr1NGnShEWLFgHg6enJ/Pnzady4savDE0KIckMSuHCqK5k5JS6jZcuWeZOyCCGEuD5J4MIpUjJyeGddAt1e31DkfZOSkrj33nuJjY0FoFatWhw5coSxY8c6O0whhKgw5Bq4KJHL6dl8tCWJpVuOcSXTRFX/oreajx8/zsqVK6lVqxadO3cGZFiYEEIURhK4KJaLV7P58NejLNuaTFqWidAAb565vRkjOjeg1Uv/LXBfq9XKF198wa233kr16tXp3r07mzdvlmFhQghRBJLARZGcT8tiyeZjfBKbRHq2meqBPjzZtwnDo8Px97Z9nJJm9y+wjC+++ILhw4czfvx43nvvPQC6devm6tCFEKJCkQQuHHL2SiaLNx3ls21/kJFjpmaQD1NuVTzYMRw/78KX6Tx37hyhoaF4eHhw3333sWPHDiZOnFgKkQshRMUkCVwU6ExqJu9vPMLy7X+QZbJQp4ovz/Zqxv1R9fH1cmx97TVr1jB06FDmzp3LmDFj8PLyYt68eS6OXAghKjZJ4OK6Tl3O4P2NR/jyt+NkmyyEVfXjH70bMaRDPXw8HUvcuVq3bk3VqlWlY5oQQjiRJHDxF8cvpvPxoSy2rN1AjtlK/VA/Hu/dmHva1cPb07FRh4mJiUyePJkXX3yRDh06ULduXY4cOYKnp3zchBDCWeQbVQCQfOEqCzccYcXuE5gsViKq+TO+d2PubheGl0fRpgtITk7m+++/p1GjRnTo0AFAkrcQQjiZfKtWckfPpbFgwxH+s/ckZouVhjUC6FfHxNNDe+LpYOK2WCx89tlnDBw4kJCQEPr27cuvv/5Kly5dXBy9EEJUXpLAK6nEs1d4b30i3+87hcUKTWoG8kTfJvS/qQ6bN210OHkDLFu2jNGjRzNx4kTefvttABnTLYQQLiYJvJLRf17h3fUJrD5wGqsVmtUOYkLfJtzesjZGBxYayXXmzBlq1KiB0Whk2LBh7N+/n0mTJrkwciGEEPlJAq8k4k6l8t6GBH488CcALesGM6FvE25pXqtIiRvg+++/Z9iwYSxcuJCRI0fi4+OT1/IWQghROiSBV3AHT6bwzroEfo47A0DrelV4sm8T+jSricFQtMSdq23btlSrVk2GhQkhhBtJAq+g9h6/zLvrElh3+CwA7cKr8mTfJvRsWqPIiVtrzVNPPcXs2bNp3bo14eHhMixMCCHczGXfwEopI7AQaANkAWO01on5tj8FPGC/+6PW+mVXxVKZ7Eq+xDvrEtgYfw6AmyNCeLJvU7o2rlbsFndSUhI//fQTbdu2pXXr1oAMCxNCCHdz5bfw3YCv1rqzUioamAsMAlBKNQSGA50AK7BZKfWt1nq/C+Op0HYcu8g76xL4NfE8ANENQ5nQtwmdGxY9cZvNZj766COGDBlCcHAwt912G1u3biU6OtoVoQshhCgGVybwbsAaAK31NqVUVL5tx4HbtdZmAKWUF5DpwlgqJKvVSuzRC7yzLoFtRy8C0K1xdZ7o05hODasVu9zVq1fz9ttvEx8fz+zZswHy1ukWQghRNrgygQcDKfnum5VSnlprk9Y6BzivlDIAbwB7tNbxhRWolJoBvATQp08fF4RcPlitVg5dsPD9kWziL1kAuKm6B3c18qJJSAYZfxwg5o+ilXnhwgVCQ0MxGAw0b96cIUOGEBUVRUxMjPNfQCWWlJQkx9SF5Pi6jhxb1+rVq1eR93FlAk8FgvLdN2qtTbl3lFK+wEfAFeAfjhSotZ4BzACIiYmxFucFlzcR01YXuL1vs5o80bcJbetXLXYd33zzDSNHjmTp0qUMHToUgLFjxxa7PHFjMTExxfqPKhwjx9d15NiWPa5M4FuAgcC/7dfAD+RusLe8vwPWa61fd2EMFdqqx7txU70qJS6nffv21KxZUzqmCSFEOeLKb+xvgVuUUlsBA/B3pdQkIBHwAHoCPkqpO+zPf1ZrHevCeMqdzBxzgduLm7wPHjzIpEmTmDdvHi1atKBhw4YkJiZKAhdCiHLEZd/YWmsLMO6ahw/nu+3rqrrLu4tXs/lsWzKfxCa5pPxjx46xdu1avvnmG1588UVAhoUJIUR5I9/aZcjRc2l8+OsxVuw+QWaOhSBf57w9JpOJjz76iGHDhhEYGMiAAQOIjY2VYWFCCFGOSQJ3M6vVyo5jF1m8+RjrDp/BaoWwqn6M7hbJ0Jvr0+ql/5a4jgULFjBx4kROnDjBzJkzMRgMkryFEKKckwTuJiazhR8P/smSzUfZf8I22q5N/aqM7R7J7S1rF2k5z+s5deoUderUwWAwMHbsWI4fP87jjz/ujNCFEEKUAZLAS9mVzBy++u04S7ckcfJyBgYD3NqiFmN7NCSqQcj/zJqWNLt/ketYvnw5o0ePZvny5QwePBh/f3/efPNNZ70EIYQQZYAk8FJy6nIGS7cc48sdx7mSZcLXy8iI6AaM7hZJZPUAp9YVFRVF3bp1ZbUwIYSowCSBu9iBEyks3nyU1QdOY7ZYqR7ow6M9GzK8UwNCArydUse+ffuYNGkS77//Pk2aNKFp06YkJCTg4eHhlPKFEEKUPZLAXcBisbJBn2Xx5qN5c5SrWkE83D2SQW3r4uPp3MSakJDA+vXrWblyJc888wyAJG8hhKjgJIE7UWaOmZW7T7Lk16McPXcVgO5NqjOme0N6NKle7OU8r5WTk8OSJUsYNWoU/v7+3HvvvWzfvp2OHTs6pXwhhBBlnyRwJziflsUnscl8ti2Zi1ez8fIwcG/7eozpHknzOsFOr+/tt9/mmWee4cKFCzz//PMYDAZJ3kIIUclIAi+BxLNpfPjrUVbsPkm2yUIVPy/+0asRo7pEUCvYuRPNnThxgrCwMAwGA4899hjnz59n3LhrJ7oTQghRWUgCL6LcNbiXbD7G+sNnAQgP9efhbpEM6VCPAB/nH9KlS5fy6KOPsnLlSgYMGEBQUBBz5sxxej1CCCHKD0ngDsoxW1i9/zSLNx/l0KlUADo0CGFs90huaVEbD6Nzrm9fT8eOHQkPD5f5yoUQQuSRjFCIlIwcvtzxBx9vTeJ0SiZGA9zRqjZjujekQ4MQl9S5c+dOpkyZwtKlS4mMjKRly5ZoraVnuRBCiDyVPoFHTFt9w22ju0by1W9/cDXbjL+3Bw91iWB010jCq/m7NKb4+Hg2btzId999x8SJEwEZFiaEEOKvSjbhdgX30ZZjBPp68sztzYid1pcZd7V0SfLOysrivffeIzMzE4AHH3yQnTt35iVvIYRw1Pbt2+ncuTMjRoxgxIgR3H///Xz66acO7//LL78wYMAAPvnkE4f3uXz5MqtWrQJg2rRpbNq0qchxF6Rr164AzJo1i1OnTvHuu+/yxRdfFKustWvXcubMGYeeu2nTJqZNm1asekpDpW+BF+St+9swoHVdvD1d+ztnzpw5vPjii2RkZDB16lQMBgMdOnRwaZ1CiIorOjqat99+G4Ds7Gxuv/12Bg0aRHBw4cNaN2zYwKRJk+jTp4/D9WmtWb9+PQMHDix2zI6YPn16icv45JNPmDFjBv/X3p0HRXVnCxz/IqIsGtHooLgEcLlSSXCZZBAXxETHiXnMM0QUx0FxosjExAgJi9jiUkIk4vjEqSDqUD1JPSJq0FjuMo5x4tPSUSaYUX9G0aBBTFwQxaCIvD9ae0DpDiBN23A+VV3ay7339GmqT997f/d33NzcGiAi65ICbkbQwG4WW/eFCxfo3r07ALNmzeL27dtMmzbNYtsTQjRPt27dokWLFtjb26OUYvHixQC4urqSlJTEiRMnSElJwcHBgeDgYPbt20deXh7t27enD1Cu8gAAEblJREFUqKgIvV5PixYt6NSpEwEBAVy9epW4uDhu3rxJZWUlycnJrFq1ilOnTpGVlWXc7vvvv09gYCABAQGcPXuW5ORkVq9ebXz+/Pnz6HQ6ysvLcXR0ZPny5Vy5coUlS5Zw//59SkpK0Ol0DBw40LhMaGgoCxYsAAxHCnbs2EFZWRk6nQ4fHx9GjBiBl5cXXl5eBAcHP7aukpISTp48SWxsLJmZmWRlZbF161bs7OwYM2YMkydP5uzZs8THx+Pk5ISTkxPt2rVrnA+qHuQQuhWkp6fj5eXF7t27AWjXrh0ffvgh7dtbZlCcEMJ6PDw8CAkJMd7Pzs7Gw8OjWrELDQ3Fw8ODu3fvAnDlyhU8PDyYOXOm8TVr1qzBw8OjVts8dOgQoaGhTJ48mejoaObNm4eLiwvz5s1j/vz5fPrpp/j7+7N27VrAcBovMzOTN954g2HDhhEdHY2npycrV65Er9fz2WefUVxczIEDB0hLS+OVV15h3bp1zJ49m7y8PCIiIhg0aBATJkwwxhAcHMymTZsA2LhxI+PGjasWY3JyMuHh4WRlZTFhwgROnDjBmTNniI2NRa/XM3XqVLKzs02+x65du/LJJ5+QmJjI/PnzAbh06RIpKSnMnTu3xnUFBATg7e1NcnIyBQUFbN++nczMTDIzM8nJySE/P58VK1Ywa9Ys9Ho9AwYMqFW+rUX2wK3A19cXT09PGZgmhLCIqofQqzp79iwLFy4EDFMye3p6Ahj/raqgoIBr164RHh4OQFFRERcuXODcuXPGYuzn5wcYzrs/ytfXl8TERK5evcqBAweIioqq9vy5c+eMBXLMmDGA4Qqcjz/+GEdHR0pLS2nTpo3J9/jyyy8D0Lt3b3788UcA2rdvb9wR+sUvfmF2XadPn6awsJCwsDAAbty4QUFBAd9++y0+Pj4ADBw4kPz8fJMxWJsU8EZw8OBBYmJiyMzMpHv37vTv35+TJ09KAReiGTh//ny1+0FBQQQFBVV77NFBZh07dnxsuenTpzN9+vQnisXT05Pk5GTc3d05evSosfC1aPH4wdhu3brRpUsXMjIycHBwICkpiX79+pGfn8/x48fp27cvR44cYd++fQQEBHD//v1qy9vZ2REYGEhiYiJDhgx5rL1xz549OX78OIMHD2bLli3cuHGD7OxsUlJS6NmzJ6mpqXz//fcm30teXh6BgYEopXB3d3/sfSQmJta4Ljs7OyorK/Hy8qJXr16sXbsWOzs79Ho9ffr0wcvLi9zcXPz9/fnmm2/ql+hG0uwL+Pklr1t8G0opvvrqK7Zt22ac/lSKtxCisS1YsIDY2FgqKioAQ5H74Ycfanxthw4dCAsLIzQ0lIqKChwcHIiMjCQiIoL4+Hi2bNkCQFJSEq1ateL06dPo9fpq6wgKCiIgIIAvvvjisfXHxMSQkJBAWloajo6OLF26lHv37vH222/z7LPP0rlzZ65fv27yvVy8eJHJkydz9+5dFi1a9Njzv/3tb2tc14ABA4iJiSEjIwM/Pz8mTpzI3bt38fHxwc3Njfnz5xMZGclf/vIXOnToQOvWrWuVW2uwq6ystHYM9bJv377KgIAAa4dRo59++onVq1fzxz/+kVatWnH//n2+/vrrp/58ykMPf1GLhie5tSzJr+XUJ7eXL18mJiaGv/71r5YJqmmp83SeMojNApKSkpg9ezZpaWmA4bCOrRRvIYRoCLt27WLatGm8//771g6lyWr2h9AbSkFBAT169AAgMjKSyspKpk6dauWohBDCOkaPHs3o0aOtHUaTJnvgDSA1NRUvLy/+/ve/A4ZzR4sXL67VpAlCCCFEfUgBbwCDBw+md+/eMjBNCCFEo5ECXg/79+9nyJAhFBYWAvDSSy/x73//G39/fytHJoQQormQAl4Pp06d4uDBg+zatcv4WE3XUQohhBCWIlWnFkpLS1m+fDnl5eUAvPXWW+Tm5sogNSHEU+nbb78lPDyc0NBQ3nzzTVJTUzF1yfDD7mH79+8nKyuLixcvMn78+Hptt7CwkL1799b69Q+7jIFhbvOHHdSCg4PZuXMnwBN1Hqvq4fsDWL58OUFBQej1ev785z/Xeh03btwgPj6eSZMmERISQmRkJDdv3mTjxo3ExMRUe+3JkyeZOHHiE8dtjoxCr4WFCxeydOlSnJyciIiIwN7enn79+lk7LCGEjfOI22byufpOMlVSUkJUVBQrV67Ew8ODiooK3nvvPdatW2e2oDw8BXjx4sV6bRcMc7Dn5+fXqZMZwLFjx9Dr9aSnp+Pi4sL169eZMGECvXr1qncsj6p6inP79u1s2rTJ7FStNYmKiiIkJIRRo0YBoNfrSUhIICkpiRUrVnD79m2cnQ0tpzdu3FhtbnhLkAJuwvnz542NA6Kjo3FycuL3v/+9dYMSQoif8be//Q1fX1/j95e9vT3Jyck4ODgwd+5cioqKuH79Ov7+/syePdu4XHZ2Nvn5+YSEhHDt2jUiIiK4du0aw4cPZ+bMmej1etatW0dxcTFpaWmkpKRUW9e7777L6tWrKSsrY8CAAXTr1u2xzmfOzs7MmzePM2fO0L17d2Pzlg0bNjBlyhRcXFwAw5zmGzZsqHYlT0VFBQkJCY/Fv3v3btasWUPLli3p2rUrH330Ebm5uSQnJ9OyZUueeeYZUlJS2L17N/n5+Tg6OlJUVMSMGTMIDw9n8+bNLF++nB07dhg7r/3yl7/kgw8+YOXKleTm5nL79m2ioqK4cuWKsXgDxiMcTk5OvPLKK+zevZuxY8dy9+5d9u/fT3R0tEU/ayngNVi6dClxcXF8+eWXDB06lE6dOhkbAAghRG0lbT/JtrxL9Vp2yJKaD0W/7tOF+DHeJpf74YcfjK2KH3JxceHixYv079+f4OBg7ty581gBr+r27dssXboUZ2dnJk2axKuvvgoYmqSEhYWZXFd4eDj5+fm8+uqrjB8/nqSkJHr16sWGDRtYu3Yt/fv3586dO6xfv57CwkLjOKKaYn60jeelS5dq3ObWrVsJCwvj9ddfZ/Pmzdy6dYucnBxGjRrFW2+9xd69eykpKTGu55133iE7O5uMjAz+9a9/AVBcXMzKlSv5/PPPcXJyIjo6mgMHDgDg5eWFTqcjNzeXbt2qt5i2t7enbdu2AIwfP56UlBTGjh1LTk4Ow4cPx9HR0eTn1BCkgNdg6NCheHt7y8A0IYTNcXd358SJE9Ueu3DhAkVFRRw/fpxDhw7Rpk0b495vTfr27WssTC+++CLnzp0D/tO1zNXV9WfXVVPns6qdvtzd3enSpYvx/5cuXaJv377G5Y8ePUrHjh2N901tc86cOaSnp/PZZ5/h5eXFyJEjiYiIYNWqVUyZMgU3NzfjNk15tPNaaWkpFy5cqPae3d3dKSoqqrZceXk5O3fuJDAwkOeff56SkhIuX75MdnY2sbGxZrfZEKSAYzjkpNPp2Lx5M25ubvj5+ZGXlycFXAjxROLHeJvdWzZ3DvxAXN3OIz80YsQI0tPTmThxIj169KC8vJwlS5bg6+tL27ZtWbRoEd999x3r1683ObDt7NmzlJaW0rp1a/Ly8ozncu3sDNN1Z2dn17iuFi1aGLuS1dT5rGXLlmzbto0pU6Zw+fJlLl++DBianixbtgxfX1+cnZ25evUq8fHxrFixwhiTqW1mZWXx7rvv8uyzz5KQkMCePXsoLS3ljTfeIDY2lvT0dNavX2/sWFaTRzuvZWdn4+3tTU5OjrEOuLm50b59e3Jychg5ciQAn3zyibErGsC4ceP49NNPKSsro3fv3vX6/OpCCjiG0YKHDx8mJyeHSZMmAXJZmBDCNrVp04YlS5ag0+morKyktLSUESNG4OfnR1RUFEePHsXJyYnnnnvOZCeydu3aERkZybVr1xgzZsxjg8lMratPnz6kpaXx/PPP19j5zNPTk6NHjxIcHIy7u7uxd/eAAQMYP348f/jDH2jZsiVlZWVERUXRt29f9uzZY3abPj4+TJ06FVdXV1xcXAgICKCgoIC4uDicnZ1xcHBg0aJFHDlyxGTOHu281rVrV1577bXHXvfRRx+xaNEiMjIyKC8vp0ePHsbz/ACBgYEEBAQwd+7cun1o9dQsu5HdvHmTNWvW8N5772Fvb8+9e/c4deoUL7zwQsMGaaOko5PlSG4ty9bya4lR6JZia7m1QXXuRtYs98B1Oh2pqam4uroaf/FJ8RZCNLanrUgL29JsCnjVy8LmzJlDp06dCAkJsW5QQgghRD01ixO9iYmJ9OzZk8OHDwPQuXNndDqd8YJ7IYQQwtY0iwI+bNgwXnzxReMISiGEEMLWNckCvnPnTgYNGsTVq1cBwxR6x44d4+WXX7ZyZEIIIUTDaJIF/MSJE/zzn/+sNqm+XBYmhBCiKWkSVa24uJhly5YZJxB45513OH78OMHBwVaOTAghhLAMi41C1zStBfAx0A+4A0xTSp2p8vx0YAZwD1islNpa323FxcWRnp5O586dmTRpEq1atcLb2/TsR0IIIYSts+RlZGMBR6WUn6Zpg4BlwH8DaJrWGZgFvAQ4Al9pmrZHKXWntisvLCw0/l+n0+Hh4cGbb77ZkPELIYQQTy1LHkIfCuwEUEodwlCsH/oVcEApdUcpdQM4A5ifbf4RoaGhHDt2DDDMYxsXF2fxzi9CCCHE08KSBfwZ4EaV+xWaprU08dxNoHrvuBpomrZA07RKTdMqe/Xq9eXAgQPtMEw/J7cGvM2YMWOhtWNoqjfJreTXVm+SW8veNE1bQB1ZsoCXAG2rbkspdc/Ec22B4p9boVJqgVLKTillBwxvsEjFo+ZbO4AmTHJrWZJfy5HcWlad82vJAn4AGAPw4Bz48SrPHQaGaZrmqGlaO8Ab+MaCsQghhBBNiiUHsW0CRmma9n8YDhFM1TQtCjijlNqiaVoq8A8MPyLmKqXKLBiLEEII0aRYrIArpe4DEY88fKrK82uANU+wiYVPsKwwT3JrOZJby5L8Wo7k1rLqnF+b7QcuhBBCNGdNYiY2IYQQormRAi6EEELYICngQgghhA2SAi6EEELYICngQgghhA2y5HXgT6wxO5o1N7XIbSQQ8uDudqWUXEJSBz+X3yqv2QZ8oZRa1fhR2qZa/O2+xn9mtToGzFRKyeU2tVSL/H4ATATuA0lKqU1WCdSGaZrmCyQrpQIeeTwQSMBQ0zIeXG5t0tO+B27saAbEYehoBlTraDYEGA18qGlaa6tEaZvM5dYLmAQMBvyAX2uaVqdmM8J0fqtYDHRo1KiaBnN/u22BpcB/KaUGAeeBjtYI0oaZy68rhu9dP+DXwP9YJUIbpmlaDLAWQyfOqo87AMsx5HU4EP6gzpn0tBdwi3Y0a+bM5fYC8BulVMWDCXkcAJkpr27M5RdN08Zh2IPZ0fih2TxzuR2MYdrmZZqm/QO4rJT6sfFDtGnm8lsKfAe4PLjdb/TobN9ZIKiGx70xzFR6XSl1F/gKGGZuRU97AW/wjmbCyGRulVLlSqkrmqbZaZqWAuQqpU5bJUrbZTK/mqa9APwOw6EyUXfmvhc6AiOAWOA1YLamaX0aOT5bZy6/YPiBfwLD6YnUxgysKVBKfQ6U1/BUnWva017AG7yjmTAyl1s0TXME/vfBa95u5NiaAnP5nQx0BfYCYUCUpmm/adzwbJq53F4FjiilipRSt4D9QP/GDtDGmcvva0AXwBPoAYzVNO1XjRxfU1Xnmva0F3DpaGY5JnOraZod8AXwtVJqhlKqwjoh2jST+VVKxSilfB8MYNEDf1JK7bRGkDbK3PfCUeAFTdM6PthrHIRhb1HUnrn8Xgd+Au48aEBVDLg2eoRN00mgt6ZpHTRNawX4AwfNLfBUj0JHOppZksncAvYYBlG0fjCiF2COUsrsH5OoxuzfrnVDs3k/970wB9j14LXrlVLyw75ufi6/I4FDmqbdx3Cedo8VY7V5mqb9DmijlFr9IM+7MNS0DKXU9+aWlWYmQgghhA162g+hCyGEEKIGUsCFEEIIGyQFXAghhLBBUsCFEEIIGyQFXAghhLBBUsCFEEIIGyQFXAghhLBBUsCFEEIIG/T/A4jBJ1ySkaQAAAAASUVORK5CYII=\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "plot_model(calibrated_rf_isotonic, plot='calibration')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 11.0 Predict on test / hold-out Sample" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In section 10.4 above we have discussed that stacking is less commonly implemented technique of ensembling due to practical difficulties. To understand this more Let's imagine a scenario that the model deployed in production is stacking ensembler of 4 models plus a meta model (similar to `stack_soft` created in section 10.4 above). To generate a prediction on unseen dataset, every data point has to be predicted by all the 4 models of stacking ensembler and then all these prediction has to pass through meta-model to generate a final prediction. As the size of your stacking ensembler increases, it becomes code intensive and hard to maintain and use in production.\n", - "\n", - "In __[Binary Classification Tutorial (CLF101) - Level Beginner](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__ we have seen how to use a trained model to generate prediction on test / hold-out or unseen dataset. In this example we will see it is no different to generate predictions using stacking ensembler in PyCaret. For the purpose of illustration, we will use `stack_soft` created in section 10.4 above for remaining part of this tutorial." - ] - }, - { - "cell_type": "code", - "execution_count": 35, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
ModelAccuracyAUCRecallPrec.F1Kappa
0Stacking Classifier0.81460.75660.3410.65570.44870.3504
\n", - "
" - ], - "text/plain": [ - " Model Accuracy AUC Recall Prec. F1 Kappa\n", - "0 Stacking Classifier 0.8146 0.7566 0.341 0.6557 0.4487 0.3504" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "predict_model(stack_soft);" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Accuracy on hold-out sample is **`0.8146`** compared with CV results of **`0.8216`** in section 10.4 above. However, note that there is a significant decline in `AUC` on hold-out set from CV. We will discuss the reasons and how to investigate this in our next tutorial. __[Binary Classification Tutorial (CLF103) - Level Expert](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF103)%20-%20Level%20Expert.ipynb)__. For now we will finish the remaining part of this tutorial using stacking ensembler stored in `stack_soft` variable." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 12.0 Finalize Model for Deployment" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In __[Binary Classification Tutorial (CLF101) - Level Beginner](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__ we have learned the purpose of `finalize_model()` and how to do it. In this tutorial we will finalize stacking ensembler and it is no different than finalizing a single model." - ] - }, - { - "cell_type": "code", - "execution_count": 36, - "metadata": {}, - "outputs": [], - "source": [ - "final_stack_soft = finalize_model(stack_soft)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 13.0 Predict on unseen data" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We will now use `final_stack_soft` to generate predictions on `data_unseen` that was created in the beginning and it contains 5% (1200 samples) of the original dataset that was never exposed to PyCaret. (see section 5 for explanations)" - ] - }, - { - "cell_type": "code", - "execution_count": 37, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
LIMIT_BALSEXEDUCATIONMARRIAGEAGEPAY_1PAY_2PAY_3PAY_4PAY_5...BILL_AMT6PAY_AMT1PAY_AMT2PAY_AMT3PAY_AMT4PAY_AMT5PAY_AMT6defaultLabelScore
0500002214800000...8011.02028.02453.02329.0431.0300.0500.0000.1770
12000002114022222...89112.04200.04100.03000.03400.03500.00.0110.7896
2500002314412324...15798.02100.01000.02300.00.00.00.0110.6457
3600002213122-100...30384.01132.060994.01436.01047.01056.01053.0110.5704
412000023232-10000...81354.02429.03120.03300.010000.03200.03200.0000.1386
\n", - "

5 rows × 26 columns

\n", - "
" - ], - "text/plain": [ - " LIMIT_BAL SEX EDUCATION MARRIAGE AGE PAY_1 PAY_2 PAY_3 PAY_4 \\\n", - "0 50000 2 2 1 48 0 0 0 0 \n", - "1 200000 2 1 1 40 2 2 2 2 \n", - "2 50000 2 3 1 44 1 2 3 2 \n", - "3 60000 2 2 1 31 2 2 -1 0 \n", - "4 120000 2 3 2 32 -1 0 0 0 \n", - "\n", - " PAY_5 ... BILL_AMT6 PAY_AMT1 PAY_AMT2 PAY_AMT3 PAY_AMT4 PAY_AMT5 \\\n", - "0 0 ... 8011.0 2028.0 2453.0 2329.0 431.0 300.0 \n", - "1 2 ... 89112.0 4200.0 4100.0 3000.0 3400.0 3500.0 \n", - "2 4 ... 15798.0 2100.0 1000.0 2300.0 0.0 0.0 \n", - "3 0 ... 30384.0 1132.0 60994.0 1436.0 1047.0 1056.0 \n", - "4 0 ... 81354.0 2429.0 3120.0 3300.0 10000.0 3200.0 \n", - "\n", - " PAY_AMT6 default Label Score \n", - "0 500.0 0 0 0.1770 \n", - "1 0.0 1 1 0.7896 \n", - "2 0.0 1 1 0.6457 \n", - "3 1053.0 1 1 0.5704 \n", - "4 3200.0 0 0 0.1386 \n", - "\n", - "[5 rows x 26 columns]" - ] - }, - "execution_count": 37, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "unseen_predictions = predict_model(final_stack_soft, data=data_unseen)\n", - "unseen_predictions.head()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice the last two columns 'Label' and 'Score'. Label is the prediction and score is the probability of prediction. Notice that predicted results are concated to the original dataset while all the transformations including imputation of missing values (in this case None), categorical encoding, feature extraction etc. is performed under the hood and you dont have to manage the pipeline manually. All this is done automatically in PyCaret." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 14.0 Save the experiment" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In __[Binary Classification Tutorial (CLF101) - Level Beginner](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__ we have learned how to save and load the model. In this experiment we will learn how to save the entire experiment including all the outputs and models we have built in this experiment. Saving experiment is as simple as saving model." - ] - }, - { - "cell_type": "code", - "execution_count": 38, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Experiment Succesfully Saved\n" - ] - } - ], - "source": [ - "save_experiment('Experiment_123 08Feb2020')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 15.0 Loading saved experiment" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To load a saved experiment on a future date or in a different environment, we would use the `load_experiment()` function." - ] - }, - { - "cell_type": "code", - "execution_count": 39, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
Object
0Classification Setup Config
1X_training Set
2y_training Set
3X_test Set
4y_test Set
5Transformation Pipeline
6Compare Models Score Grid
7Decision Tree
8Decision Tree Score Grid
9Decision Tree
10Decision Tree Score Grid
11Tuned RandomForestClassifier
12Tuned RandomForestClassifier Score Grid
13Tuned RandomForestClassifier
14Tuned RandomForestClassifier Score Grid
15Decision Tree
16Decision Tree Score Grid
17BaggingClassifier
18BaggingClassifier Score Grid
19AdaBoostClassifier
20AdaBoostClassifier Score Grid
21BaggingClassifier
22BaggingClassifier Score Grid
23Tuned BaggingClassifier
24Tuned BaggingClassifier Score Grid
25Voting Classifier
26Voting Classifier Score Grid
27Voting Classifier
28Voting Classifier Score Grid
29Gradient Boosting Classifier
30Gradient Boosting Classifier Score Grid
31Decision Tree
32Decision Tree Score Grid
33Light Gradient Boosting Machine
34Light Gradient Boosting Machine Score Grid
35Extreme Gradient Boosting
36Extreme Gradient Boosting Score Grid
37Voting Classifier
38Voting Classifier Score Grid
39Voting Classifier
40Voting Classifier Score Grid
41Stacking Classifier (Single Layer)
42Stacking Classifier (Single Layer) Score Grid
43Stacking Classifier (Single Layer)
44Stacking Classifier (Single Layer) Score Grid
45Stacking Classifier (Single Layer)
46Stacking Classifier (Single Layer) Score Grid
47Stacking Classifier (Single Layer)
48Stacking Classifier (Single Layer) Score Grid
49Random Forest Classifier
50Random Forest Classifier Score Grid
51CalibratedClassifierCV
52CalibratedClassifierCV Score Grid
53CalibratedClassifierCV
54CalibratedClassifierCV Score Grid
55Stacking Classifier (Single Layer)
56Stacking Classifier (Single Layer) Score Grid
57Final [GradientBoostingClassifier
\n", - "
" - ], - "text/plain": [ - " Object\n", - "0 Classification Setup Config\n", - "1 X_training Set\n", - "2 y_training Set\n", - "3 X_test Set\n", - "4 y_test Set\n", - "5 Transformation Pipeline\n", - "6 Compare Models Score Grid\n", - "7 Decision Tree\n", - "8 Decision Tree Score Grid\n", - "9 Decision Tree\n", - "10 Decision Tree Score Grid\n", - "11 Tuned RandomForestClassifier\n", - "12 Tuned RandomForestClassifier Score Grid\n", - "13 Tuned RandomForestClassifier\n", - "14 Tuned RandomForestClassifier Score Grid\n", - "15 Decision Tree\n", - "16 Decision Tree Score Grid\n", - "17 BaggingClassifier\n", - "18 BaggingClassifier Score Grid\n", - "19 AdaBoostClassifier\n", - "20 AdaBoostClassifier Score Grid\n", - "21 BaggingClassifier\n", - "22 BaggingClassifier Score Grid\n", - "23 Tuned BaggingClassifier\n", - "24 Tuned BaggingClassifier Score Grid\n", - "25 Voting Classifier\n", - "26 Voting Classifier Score Grid\n", - "27 Voting Classifier\n", - "28 Voting Classifier Score Grid\n", - "29 Gradient Boosting Classifier\n", - "30 Gradient Boosting Classifier Score Grid\n", - "31 Decision Tree\n", - "32 Decision Tree Score Grid\n", - "33 Light Gradient Boosting Machine\n", - "34 Light Gradient Boosting Machine Score Grid\n", - "35 Extreme Gradient Boosting\n", - "36 Extreme Gradient Boosting Score Grid\n", - "37 Voting Classifier\n", - "38 Voting Classifier Score Grid\n", - "39 Voting Classifier\n", - "40 Voting Classifier Score Grid\n", - "41 Stacking Classifier (Single Layer)\n", - "42 Stacking Classifier (Single Layer) Score Grid\n", - "43 Stacking Classifier (Single Layer)\n", - "44 Stacking Classifier (Single Layer) Score Grid\n", - "45 Stacking Classifier (Single Layer)\n", - "46 Stacking Classifier (Single Layer) Score Grid\n", - "47 Stacking Classifier (Single Layer)\n", - "48 Stacking Classifier (Single Layer) Score Grid\n", - "49 Random Forest Classifier\n", - "50 Random Forest Classifier Score Grid\n", - "51 CalibratedClassifierCV\n", - "52 CalibratedClassifierCV Score Grid\n", - "53 CalibratedClassifierCV\n", - "54 CalibratedClassifierCV Score Grid\n", - "55 Stacking Classifier (Single Layer)\n", - "56 Stacking Classifier (Single Layer) Score Grid\n", - "57 Final [GradientBoostingClassifier" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "saved_experiment = load_experiment('Experiment_123 08Feb2020')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that when you used `load_experiment()`, it has loaded the entire experiments and all the intermediate outputs in variable `saved_experiment`. You can access specific items in a similar way you would access list elements in Python. See example below in which we are accessing our final stacking ensembler and store it in `final_stack_soft_loaded` variable." - ] - }, - { - "cell_type": "code", - "execution_count": 40, - "metadata": {}, - "outputs": [], - "source": [ - "final_stack_soft_loaded = saved_experiment[57]" - ] - }, - { - "cell_type": "code", - "execution_count": 41, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
LIMIT_BALSEXEDUCATIONMARRIAGEAGEPAY_1PAY_2PAY_3PAY_4PAY_5...BILL_AMT6PAY_AMT1PAY_AMT2PAY_AMT3PAY_AMT4PAY_AMT5PAY_AMT6defaultLabelScore
0500002214800000...8011.02028.02453.02329.0431.0300.0500.0000.1770
12000002114022222...89112.04200.04100.03000.03400.03500.00.0110.7896
2500002314412324...15798.02100.01000.02300.00.00.00.0110.6457
3600002213122-100...30384.01132.060994.01436.01047.01056.01053.0110.5704
412000023232-10000...81354.02429.03120.03300.010000.03200.03200.0000.1386
\n", - "

5 rows × 26 columns

\n", - "
" - ], - "text/plain": [ - " LIMIT_BAL SEX EDUCATION MARRIAGE AGE PAY_1 PAY_2 PAY_3 PAY_4 \\\n", - "0 50000 2 2 1 48 0 0 0 0 \n", - "1 200000 2 1 1 40 2 2 2 2 \n", - "2 50000 2 3 1 44 1 2 3 2 \n", - "3 60000 2 2 1 31 2 2 -1 0 \n", - "4 120000 2 3 2 32 -1 0 0 0 \n", - "\n", - " PAY_5 ... BILL_AMT6 PAY_AMT1 PAY_AMT2 PAY_AMT3 PAY_AMT4 PAY_AMT5 \\\n", - "0 0 ... 8011.0 2028.0 2453.0 2329.0 431.0 300.0 \n", - "1 2 ... 89112.0 4200.0 4100.0 3000.0 3400.0 3500.0 \n", - "2 4 ... 15798.0 2100.0 1000.0 2300.0 0.0 0.0 \n", - "3 0 ... 30384.0 1132.0 60994.0 1436.0 1047.0 1056.0 \n", - "4 0 ... 81354.0 2429.0 3120.0 3300.0 10000.0 3200.0 \n", - "\n", - " PAY_AMT6 default Label Score \n", - "0 500.0 0 0 0.1770 \n", - "1 0.0 1 1 0.7896 \n", - "2 0.0 1 1 0.6457 \n", - "3 1053.0 1 1 0.5704 \n", - "4 3200.0 0 0 0.1386 \n", - "\n", - "[5 rows x 26 columns]" - ] - }, - "execution_count": 41, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "new_prediction = predict_model(final_stack_soft_loaded, data=data_unseen)\n", - "new_prediction.head()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Notice that results of `unseen_predictions` and `new_prediction` are identical." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 16.0 Wrap-up / Next Steps?" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We have covered a lot of new concepts in this tutorial. Most importantly, we have seen how to use Exploratory Data Analysis in customizing the pipeline in `setup()` that has improved the results considerably comparing to what we have seen earlier in __[Binary Classification Tutorial (CLF101) - Level Beginner](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF101)%20-%20Level%20Beginner.ipynb)__. We have also learned how to perform and tune ensembling in PyCaret.\n", - "\n", - "In this tutorial, we have covered many significant concepts and how to perform them using `pycaret.classification`. However, there are still few more things to go such as defining and optimizing custom cost function, interpretating more complex tree based models using shapley values, advance ensembling techniques such as multiple layer stacknet and more in pre-processing pipelines. We will cover all this in our next and final tutorial of the `pycaret.classification` series. \n", - "\n", - "See you at the next tutorial. Follow the link to __[Binary Classification Tutorial (CLF103) - Level Expert](https://github.com/pycaret/pycaret/blob/master/Tutorials/Binary%20Classification%20Tutorial%20(CLF103)%20-%20Level%20Expert.ipynb)__" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} -- GitLab