Deep learning[딥러닝] LSTM GRU

<simple RNN 단점>

  - 긴 문장(시퀀스)을 학습하기 어려움
  - 시퀀스가 길 수록 초반의 정보는 점진적으로 희석(소멸)
     즉, 멀리 떨어져 있는 단어의 정보를 인식하는데 어려움이 있음.
  - 이러한 단점을 보완한 모델이 LSTM과 GRU

 

<LSTM(Long Shot - Term Memory; 장기기억)>

  - 단기기억을 오래 기억할 수 있도록 고안된 모델
  - 많은 이전 정보를 기억해야 하기 때문에 훈련속도가 느리며, 시스템 저장 공간이 많이 필요함

 

<GRU(Gated Reccurent Units; 게이트 반복 단위)>

  - LSTM의 느린 속도를 개선하기 위해 고안된 모델
  - 성능은 LSTM과 유사함

 ** SimpleRNN, SLTM, GRU 모두 RMSprop 옵티마이저를 일반적으로 사용함.

---

* 사용 라이브러리 정의

import tensorflow as tf
from tensorflow import keras

### 영화 감상평에 대한 긍정/부정 데이터셋
from tensorflow.keras.datasets import imdb
from sklearn.model_selection import train_test_split

### 텍스트 정규화 (길이 맞추기)
from tensorflow.keras.preprocessing.sequence import pad_sequences

 

 

* 사용할 데이터 : IMDB

(train_input, train_target), (test_input, test_target) = imdb.load_data(num_words=500)

print(train_input.shape, train_target.shape)
print(test_input.shape, test_target.shape)

결과 :

(25000,) (25000,)
(25000,) (25000,)

  - IMDB데이터 사용, 말뭉치 500개 사용

 

* 훈련 : 검증 데이터로 분류하기 2 : 8

train_input, val_input, train_target, val_target = train_test_split(train_input, train_target,
                                                                   test_size=0.2, random_state=42)

print(train_input.shape, train_target.shape)
print(val_input.shape, val_target.shape)
print(test_input.shape, test_target.shape)

결과 :

(20000,) (20000,)
(5000,) (5000,)
(25000,) (25000,)
 - train : val =8 : 2로 분류하기

 

 

* 택스트 정규화 하기

train_seq = pad_sequences(train_input, maxlen=100)

val_seq = pad_sequences(val_input, maxlen=100)

test_seq = pad_sequences(test_input, maxlen=100)

print(train_seq.shape, train_target.shape)
print(val_seq.shape, val_target.shape)
print(test_seq.shape, test_target.shape)

결과 :

(20000, 100) (20000,)
(5000, 100) (5000,)
(25000, 100) (25000,)
 - 훈련, 검증, 테스트 데이터 내에 각 문장의 길이를 100으로 통일(정규화) 시키기

---

* LSTM모델 훈련

* 모델 생성하기

model = keras.Sequential()
model

* 계층 생성 및 모델에 추가하기

model.add(keras.layers.Embedding(500, 16, input_length=(100)))

model.add(keras.layers.LSTM(8))

model.add(keras.layers.Dense(1, activation="sigmoid"))

model.summary()

 

 

 

 - 임베딩 계층 추가 : 말뭉치 500개, 

    출력크기 16, 입력크기 100
  - LSTM 계층 추가 : 출력크기 8
  - 출력 계층 추가

 

* 모델 설정하기

rmsprop = keras.optimizers.RMSprop(learning_rate=0.0001)
model.compile(optimizer=rmsprop,
              loss="binary_crossentropy",
              metrics=["accuracy"])

 

 

* 콜백함수 만들기

model_path = "./mdoel/best_LSTM_model.h5"

checkpoint_cb = keras.callbacks.ModelCheckpoint(
    model_path,
    save_best_only=True
)

early_stopping_cb = keras.callbacks.EarlyStopping(
    patience=3,
    restore_best_weights=True
)

checkpoint_cb, early_stopping_cb

콜백함수 정의하기
 - 위치 및 파일명: ./mdoel/best_LSTM_model.h5
 - 자동 저장 및 자동종료 

 

 

* 모델 훈련시키기

history = model.fit(
    train_seq, train_target, epochs=100, batch_size=64,
    validation_data=(val_seq, val_target),
    callbacks=[checkpoint_cb, early_stopping_cb]
)

train_eval = model.evaluate(train_seq, train_target)
val_eval = model.evaluate(val_seq, val_target)
test_eval = model.evaluate(test_seq, test_target)

결과 :

625/625 [==============================] - 4s 6ms/step - loss: 0.4100 - accuracy: 0.8145
157/157 [==============================] - 1s 5ms/step - loss: 0.4339 - accuracy: 0.7994
782/782 [==============================] - 4s 5ms/step - loss: 0.4286 - accuracy: 0.8013

 

* LSTM 모델에 드롭아웃 속성(계층 아님) 적용하기

model2.add(keras.layers.Embedding(500, 16, input_length=(100)))

model2.add(keras.layers.LSTM(8, dropout=0.3))

model2.add(keras.layers.Dense(1, activation="sigmoid"))

model2.summary()

 

 

 - LSTM모델(계층)에 드롭아웃 속성 적용, 30% 

   훈련에서 제외
  - 나머지 모두 위와 동일하게 처리하여 훈련, 점증, 

    테스트에 대한 성능 검증까지 진행

 

 

 

rmsprop = keras.optimizers.RMSprop(learning_rate=0.0001)
model2.compile(optimizer=rmsprop,
              loss="binary_crossentropy",
              metrics=["accuracy"])

model_path = "./mdoel/best_LSTM_model + drop_out.h5"

checkpoint_cb = keras.callbacks.ModelCheckpoint(
    model_path,
    save_best_only=True
)

early_stopping_cb = keras.callbacks.EarlyStopping(
    patience=3,
    restore_best_weights=True
)

checkpoint_cb, early_stopping_cb

---

결과 :

(<keras.callbacks.ModelCheckpoint at 0x1c1bedccdc0>,
 <keras.callbacks.EarlyStopping at 0x1c1bedcc9d0>)

 

*  모델 훈련시키기

history = model2.fit(
    train_seq, train_target, epochs=100, batch_size=64,
    validation_data=(val_seq, val_target),
    callbacks=[checkpoint_cb, early_stopping_cb]
)

train_eval = model2.evaluate(train_seq, train_target)
val_eval = model2.evaluate(val_seq, val_target)
test_eval = model2.evaluate(test_seq, test_target)

결과 :

625/625 [==============================] - 4s 6ms/step - loss: 0.4100 - accuracy: 0.8163
157/157 [==============================] - 1s 5ms/step - loss: 0.4357 - accuracy: 0.8032
782/782 [==============================] - 4s 5ms/step - loss: 0.4293 - accuracy: 0.8030

---

<LSTM 2개 연결하기>

  - LSTM 2개를 연결할 때는 연속해서 LSTM 계층을 추가해야 함
  - 첫 번째, LSTM 계층의 속성에는 return_sequenes=True 속성을 추가해 준다.
  - 두 번째, LSTM은 첫 번째 LSTM의 훈련결과를 이어받아서 계속 훈련을 이어나간다.

 

* 모델 생성

model3 = keras.Sequential()
model3

* 계층 추

model3.add(keras.layers.Embedding(500, 16, input_length=(100)))

model3.add(keras.layers.LSTM(8, dropout=0.3, return_sequences=True))

model3.add(keras.layers.LSTM(8, dropout=0.2))

model3.add(keras.layers.Dense(1, activation="sigmoid"))

model3.summary()

* 모델 설정하기

rmsprop = keras.optimizers.RMSprop(learning_rate=0.0001)
model3.compile(optimizer=rmsprop,
              loss="binary_crossentropy",
              metrics=["accuracy"])

* 콜백함수 정의

model_path = "./mdoel/best_LSTM_model + drop_out + 2LSTM.h5"

checkpoint_cb = keras.callbacks.ModelCheckpoint(
    model_path,
    save_best_only=True
)

early_stopping_cb = keras.callbacks.EarlyStopping(
    patience=3,
    restore_best_weights=True
)

checkpoint_cb, early_stopping_cb

* 모델 훈련

history = model3.fit(
    train_seq, train_target, epochs=100, batch_size=64,
    validation_data=(val_seq, val_target),
    callbacks=[checkpoint_cb, early_stopping_cb]
)

* 모델 평가

train_eval = model3.evaluate(train_seq, train_target)
val_eval = model3.evaluate(val_seq, val_target)
test_eval = model3.evaluate(test_seq, test_target)

결과 :

625/625 [==============================] - 7s 10ms/step - loss: 0.3999 - accuracy: 0.8188
157/157 [==============================] - 1s 9ms/step - loss: 0.4257 - accuracy: 0.8068
782/782 [==============================] - 7s 9ms/step - loss: 0.4206 - accuracy: 0.8039