Tensorflow2.0学習(18):Embedding前処理-ワードテーブルインデックスの構築

Embedding

語ベクトルone-hot符号化:例:ある語セットに10語がある場合、そのうち2番目の語のone-hot符号化は[0,1,0,0,...]であり、このベクトルは疎ベクトルである.

Dense embedding:例:ある語集のある語は[2.9,1.1,-1.5,...]であり、この語を表すことができると同時に、これらの語ベクトルの学習は後期にモデルを通じて訓練されたものであり、彼が訓練してから本当の代表的な意味を持っている.

データセットの読み込みと用語集インデックスの構築

パイロット

import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf
from tensorflow import keras
print(tf.__version__)
print(sys.version_info)
for module in mpl, np ,pd, sklearn, tf, keras:
    print(module.__name__, module.__version__)

2.1.0
sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)
matplotlib 3.1.1
numpy 1.16.5
pandas 0.25.1
sklearn 0.21.3
tensorflow 2.1.0
tensorflow_core.python.keras.api._v2.keras 2.2.4-tf

ロード表示データセット

#  keras 
imdb = keras.datasets.imdb
#  
vocab_size = 10000
# id 
index_from = 3
#  ， 
(train_data, train_labels),(test_data, test_labels) = imdb.load_data(
    num_words = vocab_size, index_from = index_from)

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz
17465344/17464789 [==============================] - 4s 0us/step

#  
print(train_data[0], train_labels[0])
#  
print(train_data.shape, train_labels.shape)
#  
print(len(train_data[0]), len(train_data[1]))

[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32] 1
(25000,) (25000,)
218 189

print(test_data.shape, test_labels.shape)

(25000,) (25000,)

ワードテーブルのインデックスと対応する名前を表示する

#  
word_index = imdb.get_word_index()
print(len(word_index))
print(list(word_index.items())[:5]) 
# print(word_index)

88584
[('fawn', 34701), ('tsukino', 52006), ('nunnery', 52007), ('sonja', 16816), ('vani', 63951)]

3スロット

を追加

#  index_from = 3， id + 3
word_index = {k:(v+3) for k, v in word_index.items()}

特殊文字

を追加

# id 3 ， 
word_index[''] = 0 
word_index[''] = 1 
word_index[''] = 2
word_index[''] = 3 

解析トレーニングセットの最初のデータ

reverse_word_index = dict(
    [(value, key) for key, value in word_index.items()])
#  id 
def decode_review(text_ids):
    return " ".join(
    [reverse_word_index.get(word_id,"") for word_id in text_ids])

decode_review(train_data[0])

" this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert  is an amazing actor and now the same being director  father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for  and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also  to the two little boy's that played the  of norman and paul they were just brilliant children are often left out of the  list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all"