3.データ閲覧によるモデル性能の向上2

3.1.5数値型変数を正規分布にする

log変換による前処理

限定のケースしか勉強していないので...バランスが重要です

plt.figure(figsize = (10, 8))
sns.distplot(df.loc[df.Insulin>0, 'Insulin'])
plt.show()

plt.figure(figsize = (10, 8))
sns.distplot(np.log(df.loc[df.Insulin>0, 'Insulin']+1))
plt.show()

df['Insulin_log'] = np.log(df['Insulin_nan'] + 1)
fig, axes = plt.subplots(1, 2, figsize = (15, 5))
sns.distplot(df['Insulin_nan'], ax = axes[0])
sns.distplot(df['Insulin_log'], ax = axes[1])
plt.show()

3.1.6関連分析による派生変数の作成

皮膚の厚さはBMIと相関性が高い

plt.figure(figsize= (10, 8))
sns.lmplot(data = df, x = 'Insulin_nan', y = 'Glucose', hue = 'Outcome')
plt.show()

df['low_glu_insulin'] = (df['Glucose'] < 100) & (df['Insulin_nan'] <= 102.5)
df[['low_glu_insulin']].head()

pd.crosstab(df['Outcome'], df['low_glu_insulin'])

features = df.columns.tolist()
features.remove('Pregnancies')
features.remove('Outcome')
features.remove('Age_low')
features.remove('Age_middle')
features.remove('Age_high')
features.remove('Insulin')
features.remove('Insulin_nan')

train = df[: int(df.shape[0] * 0.8)]
test = df[int(df.shape[0] * 0.8): ]

model.fit(train[features], train[label_name])
y_pred = model.predict(test[features])
diff_count = sum(abs(y_pred - test[label_name]))
print('Accuracy: {}'.format((len(y_test) - diff_count) / len(y_test)))
sns.barplot(y = features, x = model.feature_importances_)
plt.show()

3.1.7処理異常値

fig, axes = plt.subplots(2, 1, figsize = (15, 6))
sns.boxplot(df.Insulin, ax = axes[0])
sns.boxplot(df['Insulin_nan'], ax = axes[1])
plt.show()

df['Insulin_nan'].describe()

### IQR
IQR3 = df['Insulin_nan'].quantile(.75)
IQR1 = df['Insulin_nan'].quantile(.25)
IQR = IQR3 - IQR1
IQR

OUT = IQR3 + 1.5*IQR
OUT

df[df['Insulin_nan'] > OUT].shape

print(df[df['Insulin_nan'] > 600].shape)
train = df[df['Insulin_nan'] <= 600]

print(df[df['Insulin_nan'] > 600].shape)
train = train[train['Insulin_nan'] <= 600]
test = test[test['Insulin_nan'] <= 600]
features = df.columns.tolist()
features.remove('Pregnancies')
features.remove('Outcome')
features.remove('Age_low')
features.remove('Age_middle')
features.remove('Age_high')
features.remove('Insulin')
features.remove('Insulin_log')

model.fit(train[features], train[label_name])
y_pred = model.predict(test[features])
diff_count = sum(abs(y_pred - test[label_name]))
print('Accuracy:', (len(test[label_name]) - diff_count) / len(test[label_name]))

3.1.8特徴スケーリング

StandardSclaer

したがって、モデルはScraingによってのみその値の一意の影響を反映する

from sklearn.preprocessing import StandardScaler
## z표준화 스케일러 생성
scaler = StandardScaler()
## 변환하고 싶은 df or array 넣어주고 fitting
scaler.fit(df[['Glucose', 'DiabetesPedigreeFunction']])
## 변환
scale = scaler.transform(df[['Glucose', 'DiabetesPedigreeFunction']])
scale

tmp = df[['Glucose', 'DiabetesPedigreeFunction']].copy() 
tmp[['Glucose', 'DiabetesPedigreeFunction']] = scale
print(tmp.head())
## min-max scaling
tmp2 = df[['Glucose', 'DiabetesPedigreeFunction']].copy() 
tmp2 = tmp2.apply(lambda x: (x - min(x)) / (max(x) - min(x)), axis = 0)
print(tmp2.head())

h = df[['Glucose', 'DiabetesPedigreeFunction']].hist(figsize = (15, 3))
h2 = tmp.hist(figsize = (15, 3))
h3 = tmp2.hist(figsize = (15, 3))

スケールはこの項目にあまり影響しません。他の項目で精度を上げることができるので、EDAを行い、サイトを取得してから前処理を行うのが望ましい。

3.1.9前処理の特徴を保存する

df.to_csv('/content/drive/MyDrive/edwith/프로젝트로 배우는 데이터 사이언스/data/diabetes_feature.csv', index = False)
pd.read_csv('/content/drive/MyDrive/edwith/프로젝트로 배우는 데이터 사이언스/data/diabetes_feature.csv').head()