INNER CODE UNIT · Python
Lower_fence
ashishpatel26/Amazing-Feature-Engineering · feature_cleaning/outlier.py:55
Lower_fence = data[col].mean() - threshold * data[col].std()
para = (Upper_fence, Lower_fence)
tmp = pd.concat([data[col]>Upper_fence,data[col]<Lower_fence],axis=1)
outlier_index = tmp.any(axis=1)
print('Num of outlier detected:',outlier_index.value_counts()[1])
print('Proportion of outlier detected',outlier_index.value_counts()[1]/len(outlier_index))
return outlier_index, para
def outlier_detect_MAD(data,col,threshold=3.5):
"""
outlier detection by Median and Median Absolute Deviation Method (MAD)
The median of the residuals is calculated. Then, the difference is calculated between each historical value and this median.
These differences are expressed as their absolute values, and a new median is calculated and multiplied by
an empirically derived constant to yield the median absolute deviation (MAD).
If a value is a certain number of MAD away from the median of the residuals,
that value is classified as an outlier. The default threshold is 3 MAD.