TensorFlow Slimの一般的な操作

一、プロフィール

二、よく使われるLayer

三、作用域機構(arg_scope)

四、2つのオペレータ

五、TensorFlow-SlimにおけるLossのメンテナンスと使用

六、Fine-Tuning Existing Models

七、参考資料

一、紹介

Slimを使用してTensorFlowプログラムを開発し、プログラムの読みやすさとメンテナンス性を高め、hyper parameterのチューニングを簡素化し、開発したモデルを汎用化し、コンピュータの視覚の中のいくつかの常用モデル(例えばVGG、Inception、ResNet)をカプセル化し、複雑なモデルを拡張しやすくした.既存のモデルのcheckpointsを用いてアルゴリズムの訓練を開始することができる.Slim APIの共通コンポーネントは以下の通りです.

arg_scope:scope内の操作は、同じパラメータ設定

を使用します.

layers:Highレベルを定義するニューラルネットワーク層の定義

nets:VGG、Inception、ResNetなどの一般的なネットワークモデルを定義します.

import tensorflow.contrib.slim.nets as nets

vgg = nets.vgg

regularizes:重み正規化された使用APIの仕様

regularization_loss = tf.add_n(slim.losses.get_regularization_losses())

total_loss = slim.losses.get_total_loss(add_regularization_losses=False)

導入方法:import tensorflow.contrib.slim as slim

二、よく使うLayer

conv2dのパラメータ設定:

入力データ(NHWC)、出力channel数、ボリュームコアサイズ、ボリュームステップ( 1)ゼロ補正方式( SAME)

.

アクティブ化関数( relu)、ネーミングスペース

重みとバイアスの初期化( xavier 0)、正規化パラメータ

.

BNおよびそのパラメータ(オプション)

# Adds an 2-D convolution followed by an optional batch_norm layer.
# Performs atrous convolution with input stride/dilation rate equal to `rate`
def conv2d(inputs,
           num_outputs,
           kernel_size,
           stride=1,
           padding='SAME',
           activation_fn=nn.relu,
           scope=None

           weights_initializer=initializers.xavier_initializer(),
           weights_regularizer=None,
           biases_initializer=init_ops.zeros_initializer(),
           biases_regularizer=None,

           normalizer_fn=None,
           normalizer_params=None,
           trainable=True,

           data_format=None,  #     'NHWC'
           rate=1,
           reuse=None,  
           variables_collections=None,
           outputs_collections=None)

separable_conv2dのパラメータ設定:

入力データ(NHWC)、ボリュームコアサイズ、ボリュームステップ( 1)ゼロ補正方式( SAME)

出力channel数:Noneに設定しないとpointwise convolutionが追加され、ボリュームコアサイズとステップ長はいずれも1

である.

アクティブ化関数( relu)、ネーミングスペース

重みとバイアスの初期化( xavier 0)、正規化パラメータ

.

BNおよびそのパラメータの設定(オプション)

# Adds an 2-D separable convolution followed by an optional batch_norm layer.
def separable_conv2d(
    inputs,
    kernel_size,
    depth_multiplier=1,
    stride=1,
    padding='SAME',

    num_outputs,

    activation_fn=nn.relu,
    scope=None  

    normalizer_fn=None,
    normalizer_params=None,
    trainable=True,

    weights_initializer=initializers.xavier_initializer(),
    pointwise_initializer=None,
    weights_regularizer=None,
    biases_initializer=init_ops.zeros_initializer(),
    biases_regularizer=None,

    data_format=DATA_FORMAT_NHWC,
    rate=1,
    reuse=None,
    variables_collections=None,
    outputs_collections=None)

fully_connectedのパラメータ設定:

入力データ(NC)、出力ニューロン数

アクティブ化関数( relu)、ネーミングスペース

重みとバイアスの初期化( xavier 0)、正規化パラメータ

.

BNおよびそのパラメータ(オプション)

def fully_connected(inputs,
                    num_outputs,

                    # Explicitly set it to None to skip it and maintain a linear activation.                    
                    activation_fn=nn.relu,  
                    scope=None,


                    weights_initializer=initializers.xavier_initializer(),
                    weights_regularizer=None,
                    biases_initializer=init_ops.zeros_initializer(),
                    biases_regularizer=None,

                    normalizer_fn=None,
                    normalizer_params=None,
                    trainable=True,

                    reuse=None,
                    variables_collections=None,
                    outputs_collections=None)

max_pool2dのパラメータ設定:

入力データ(NHWC)、ボリュームコアサイズ、ボリュームステップ( 2)ゼロ補正方式( VALID)

ネーミングスペース

def fully_connected(inputs,
                    num_outputs,

                    # Explicitly set it to None to skip it and maintain a linear activation.                    
                    activation_fn=nn.relu,  
                    scope=None,


                    weights_initializer=initializers.xavier_initializer(),
                    weights_regularizer=None,
                    biases_initializer=init_ops.zeros_initializer(),
                    biases_regularizer=None,

                    normalizer_fn=None,
                    normalizer_params=None,
                    trainable=True,

                    reuse=None,
                    variables_collections=None,
                    outputs_collections=None)

batch_normのパラメータ設定:

入力データ(NHWC):the normalization is over all but the last dimension

スライド平均パラメータ(decay):decay for the moving average

center:デフォルトはTrue、add offset of beta to normalized tensor

scale:デフォルトはFalse、gamma is not used.When the next layer is linear (also e.g. nn.relu ), this can be disabled since the scaling can be done by the next layer

activation_fn: default set to None to skip it and maintain a linear activation.

is_training: Whether or not the layer is in training mode

def batch_norm(inputs,
               decay=0.999,
               center=True,
               scale=False,
               epsilon=0.001,
               activation_fn=None,
               is_training=True, 

               param_initializers=None,
               param_regularizers=None,
               updates_collections=ops.GraphKeys.UPDATE_OPS,
               reuse=None,
               variables_collections=None,
               outputs_collections=None,
               trainable=True,
               batch_weights=None,
               fused=None,
               data_format=DATA_FORMAT_NHWC,
               zero_debias_moving_mean=False,
               scope=None,
               renorm=False,
               renorm_clipping=None,
               renorm_decay=0.99,
               adjustment=None)

dropoutパラメータ設定

inputs: The tensor to pass to the nn.dropout op.

keep_prob: A scalar Tensor with the same type as x. The probability that each element is kept.

is_training: A bool Tensor indicating whether or not the model is in training mode. If so, dropout is applied and values scaled. Otherwise, inputs is returned.

#   With probability keep_prob, outputs the input element scaled up by 1 / keep_prob, otherwise outputs 0.
# The scaling is so that the expected sum is unchanged.
def dropout(inputs,
            keep_prob=0.5,
            noise_shape=None,
            is_training=True,
            outputs_collections=None,
            scope=None,
            seed=None)

三、作用域メカニズム(arg_scope)

という新しい役割ドメインは、 がarg_に伝達されることをユーザが明確にすることを可能にする.scope内部の各操作

arg_scopeに規定するパラメータ値は、局所

#      arg_scope  ，                          ；
#     arg_scope ，            conv2d    
with slim.arg_scope([slim.conv2d, slim.fully_connected],
                      activation_fn=tf.nn.relu,
                      weights_initializer=tf.truncated_normal_initializer(stddev=0.01),
                      weights_regularizer=slim.l2_regularizer(0.0005)):
  with slim.arg_scope([slim.conv2d], stride=1, padding='SAME'):
    net = slim.conv2d(inputs, 64, [11, 11], 4, padding='VALID', scope='conv1')
    net = slim.conv2d(net, 256, [5, 5],
                      weights_initializer=tf.truncated_normal_initializer(stddev=0.03),
                      scope='conv2')
    net = slim.fully_connected(net, 1000, activation_fn=None, scope='fc')

4、2つのオペレータ(repeatとstack)

slim.repeat:同じ操作

を同じパラメータで繰り返し呼び出すことができる.

slim.stack:同じ操作

を異なるパラメータで繰り返し呼び出すことができる.

#    1
net = ...
net = slim.conv2d(net, 256, [3, 3], scope='conv3_1')
net = slim.conv2d(net, 256, [3, 3], scope='conv3_2')
net = slim.conv2d(net, 256, [3, 3], scope='conv3_3')
net = slim.max_pool2d(net, [2, 2], scope='pool2')

# slim.repeat                    
#              scopes    'conv3/conv3_1', 'conv3/conv3_2'   'conv3/conv3_3'
net = slim.repeat(net, 3, slim.conv2d, 256, [3, 3], scope='conv3')
net = slim.max_pool2d(net, [2, 2], scope='pool2')



#    2
x = slim.fully_connected(x, 32, scope='fc/fc_1')
x = slim.fully_connected(x, 64, scope='fc/fc_2')
x = slim.fully_connected(x, 128, scope='fc/fc_3')

# slim.stack                    （   slim.fully_connected   ，         ）
#              scopes    'fc/fc_1', 'fc/fc_2'   'fc/fc_3'
slim.stack(x, slim.fully_connected, [32, 64, 128], scope='fc')


#    3:
x = slim.conv2d(x, 32, [3, 3], scope='core/core_1')
x = slim.conv2d(x, 32, [1, 1], scope='core/core_2')
x = slim.conv2d(x, 64, [3, 3], scope='core/core_3')
x = slim.conv2d(x, 64, [1, 1], scope='core/core_4')

# slim.stack                    （   slim.conv2d   ，        ）
slim.stack(x, slim.conv2d, [(32, [3, 3]), (32, [1, 1]), (64, [3, 3]), (64, [1, 1])], scope='core')

五、TensorFlow-SlimにおけるLossのメンテナンスと使用

TF-Slimでlossを作成すると、TF-Slimは特殊なTensorFlow collection of loss functionsにlossを追加します.これにより、手動ですべてのlossを管理することも、TF-Slimに

を管理することもできます.

TF-Slimにlossesを管理させたい場合は、 lossを持っていますが、どうすればいいですか?

loss_ops.pyにもadd_lossの関数があります loss TF-Slims collection

# Load the images and labels.
images, scene_labels, depth_labels, pose_labels = ...

# Create the model.
scene_predictions, depth_predictions, pose_predictions = CreateMultiTaskModel(images)

# Define the loss functions and get the total loss.
classification_loss = slim.losses.softmax_cross_entropy(scene_predictions, scene_labels)
sum_of_squares_loss = slim.losses.sum_of_squares(depth_predictions, depth_labels)
pose_loss = MyCustomLossFunction(pose_predictions, pose_labels)
slim.losses.add_loss(pose_loss) # Letting TF-Slim know about the additional loss.

# The following two ways to compute the total loss are equivalent:
regularization_loss = tf.add_n(slim.losses.get_regularization_losses())
total_loss1 = classification_loss + sum_of_squares_loss + pose_loss + regularization_loss

# (Regularization Loss is included in the total loss by default).
total_loss2 = slim.losses.get_total_loss(add_regularization_losses=True)

六、Fine-Tuning Existing Models

# 1、Restoring Variables(all/partial) from a Checkpoint
-------------------------------------------------------

# Create some variables.
v1 = tf.Variable(..., name="v1")
v2 = tf.Variable(..., name="v2")
...
# Add ops to restore all the variables.
restorer = tf.train.Saver()

# Add ops to restore some variables.
restorer = tf.train.Saver([v1, v2])

# Later, launch the model, use the saver to restore variables from disk, and
# do some work with the model.
with tf.Session() as sess:
  # Restore variables from disk.
  restorer.restore(sess, "/tmp/model.ckpt")
  print("Model restored.")
  # Do some work with the model
  ...


# 2、Partially Restoring Models
-------------------------------------------------------
# Create some variables.
v1 = slim.variable(name="v1", ...)
v2 = slim.variable(name="nested/v2", ...)
...

# Get list of variables to restore (which contains only 'v2')，   exclude   include    
# These are all equivalent methods:
variables_to_restore = slim.get_variables_by_name("v2")
# or
variables_to_restore = slim.get_variables_by_suffix("2")
# or
variables_to_restore = slim.get_variables(scope="nested")
# or
variables_to_restore = slim.get_variables_to_restore(include=["nested"])
# or
variables_to_restore = slim.get_variables_to_restore(exclude=["v1"])

# Create the saver which will be used to restore the variables.
restorer = tf.train.Saver(variables_to_restore)

with tf.Session() as sess:
  # Restore variables from disk.
  restorer.restore(sess, "/tmp/model.ckpt")
  print("Model restored.")
  # Do some work with the model
  ...


# 3、Restoring models with different variable names
-------------------------------------------------------
# restore a model from a checkpoint whose variables have different names to those in the current graph

# Assuming than 'conv1/weights' should be restored from 'vgg16/conv1/weights'
def name_in_checkpoint(var):
  return 'vgg16/' + var.op.name

# Assuming than 'conv1/weights' and 'conv1/bias' should be restored from 'conv1/params1' and 'conv1/params2'
def name_in_checkpoint(var):
  if "weights" in var.op.name:
    return var.op.name.replace("weights", "params1")
  if "bias" in var.op.name:
    return var.op.name.replace("bias", "params2")

variables_to_restore = slim.get_model_variables()
variables_to_restore = {name_in_checkpoint(var):var for var in variables_to_restore}
restorer = tf.train.Saver(variables_to_restore)

with tf.Session() as sess:
  # Restore variables from disk.
  restorer.restore(sess, "/tmp/model.ckpt")


# 4、 Initialize our new model using the values of the pre-trained model excluding the final layer
--------------------------------------------------------------------------------------------
# Load the Pascal VOC data
image, label = MyPascalVocDataLoader(...)
images, labels = tf.train.batch([image, label], batch_size=32)

# Create the model
predictions = vgg.vgg_16(images)

train_op = slim.learning.create_train_op(...)

# Specify where the Model, trained on ImageNet, was saved.
model_path = '/path/to/pre_trained_on_imagenet.checkpoint'

# Specify where the new model will live:
log_dir = '/path/to/my_pascal_model_dir/'

# Restore only the convolutional layers，      
variables_to_restore = slim.get_variables_to_restore(exclude=['fc6', 'fc7', 'fc8'])
init_fn = assign_from_checkpoint_fn(model_path, variables_to_restore)

# Start training.
slim.learning.train(train_op, log_dir, init_fn=init_fn)

七、参考資料
1、https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/slim 2、https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/layers/python/layers/layers.py 3、https://github.com/tensorflow/models/tree/master/research/slim/nets 4、TensorFlow-Slimオープンソースの訓練済みモデルリスト5、TensorFlow-Slim Tutorial翻訳