INNER CODE UNIT · Python
outlier_detect_mean_std
ashishpatel26/Amazing-Feature-Engineering · feature_cleaning/outlier.py:43
def outlier_detect_mean_std(data,col,threshold=3):
'''
outlier detection by Mean and Standard Deviation Method.
If a value is a certain number(called threshold) of standard deviations away
from the mean, that data point is identified as an outlier.
Default threshold is 3.
This method can fail to detect outliers because the outliers increase the standard deviation.
The more extreme the outlier, the more the standard deviation is affected.
'''
Upper_fence = data[col].mean() + threshold * data[col].std()
Lower_fence = data[col].mean() - threshold * data[col].std()
para = (Upper_fence, Lower_fence)
tmp = pd.concat([data[col]>Upper_fence,data[col]<Lower_fence],axis=1)
outlier_index = tmp.any(axis=1)
print('Num of outlier detected:',outlier_index.value_counts()[1])
print('Proportion of outlier detected',outlier_index.value_counts()[1]/len(outlier_index))