[딥러닝] 텍스트 분류(순환신경망)

IMDB 데이터 전처리 후 핫인코딩 해보기

 

텐서플로에서 IMDB 데이터 세트 불러오기

mport numpy as np
import tensorflow
from tensorflow.keras.datasets import imdb
(x_train_all, y_train_all), (x_test, y_test) = imdb.load_data(skip_top=20, num_words=100)

 

훈련세트 크기,샘플 확인

print(x_train_all.shape,y_train_all.shape)
print(x_train_all[0])

 

숫자 2는 사전에 없는단어, 0,1은 패딩과 글의 시작을 나타냄으로 제외하기

for i in range(len(x_train_all)):
    x_train_all[i] = [w for w in x_train_all[i] if w > 2]

print(x_train_all[0])

 

어휘사전 내려받기(정수를 영단어로 바꾸기 위함)

word_to_index = imdb.get_word_index()
word_to_index['movie']

 

훈련세트의 정수가 3 이상부터 영단어를 의미함으로 -3을 한 뒤 영단어로 변환

index_to_word = {word_to_index[k]: k for k in word_to_index}

for w in x_train_all[0]:
    print(index_to_word[w - 3], end=' ')

 

훈련 세트의 길이 확인하기

print(len(x_train_all[0]), len(x_train_all[1]))
# 59 32 (길이가 다름으로 모델을 제대로 훈련시킬 수 없음

 

검증데이터 준비

np.random.seed(42)
random_index = np.random.permutation(25000)

x_train = x_train_all[random_index[:20000]]
y_train = y_train_all[random_index[:20000]]
x_val = x_train_all[random_index[20000:]]
y_val = y_train_all[random_index[20000:]]

 

텐서플로로 샘플의 길이 맞추기(sequence 함수)

from tensorflow.keras.preprocessing import sequence
maxlen = 100
x_train_seq = sequence.pad_sequences(x_train, maxlen=maxlen)
x_val_seq = sequence.pad_sequences(x_val, maxlen=maxlen)

print(x_train_seq.shape, x_val_seq.shape)
#(20000, 100) (5000, 100)
print(x_val.shape)
#(5000,)

 

텐서플로로 원-핫 인코딩하기

from tensorflow.keras.utils import to_categorical

x_train_onehot = to_categorical(x_train_seq)
x_val_onehot = to_categorical(x_val_seq)

print(x_train_onehot.shape)
#(20000, 100, 100)
print(x_train_onehot.nbytes)
#800000000