keras 2.0：Encoder-Decoder Sequence-to-Sequence Model for Neural Machine Translation

keras 2.0：Encoder-Decoder Sequence-to-Sequence Model for Neural Machine Translation

感想

最近可能会用到机器翻译的sequence to sequence模型，然后找了一个模型，把它跑通了，然后稍微改了一下。我把我改的方法记录下来，希望帮到大家的研究或者工程实践。试了一下csdn的markdown博客编写，感觉还不错

环境

python 3.5Keras (2.1.3)ubuntu 16.04tensorflow-gpu (1.4.1)

我的主要环境就是上面的，python2.7会报错，

代码剖析

原理我就不讲了，这主要是工程实践的，如果需要脑补原理的话建议要看论文啦。 数据集下载linux命令：

wget 代码为：

'''Sequence to sequence example in Keras (character-level).This script demonstrates how to implement a basic character-levelsequence-to-sequence model. We apply it to translatingshort English sentences into short French sentences,character-by-character. Note that it is fairly unusual todo character-level machine translation, as word-levelmodels are more common in this domain.# Summary of the algorithm- We start with input sequences from a domain (e.g. English sentences) and correspding target sequences from another domain (e.g. French sentences).- An encoder LSTM turns input sequences to 2 state vectors (we keep the last LSTM state and discard the outputs).- A decoder LSTM is trained to turn the target sequences into the same sequence but offset by one timestep in the future, a training process called "teacher forcing" in this context. Is uses as initial state the state vectors from the encoder. Effectively, the decoder learns to generate `targets[t+1...]` given `targets[...t]`, conditioned on the input sequence.- In inference mode, when we want to decode unknown input sequences, we: - Encode the input sequence into state vectors - Start with a target sequence of size 1 (just the start-of-sequence character) - Feed the state vectors and 1-char target sequence to the decoder to produce predictions for the next character - Sample the next character using these predictions (we simply use argmax). - Append the sampled character to the target sequence - Repeat until we generate the end-of-sequence character or we hit the character limit.# Data downloadEnglish to French sentence pairs.of neat sentence pairs datasets can be found at:References- Sequence to Sequence Learning with Neural Networks Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation __future__ import print_functionfrom keras.models import Modelfrom keras.layers import Input, LSTM, Denseimport numpy as npbatch_size = 64 # Batch size for training.epochs = 100 # Number of epochs to train for.latent_dim = 256 # Latent dimensionality of the encoding space.num_samples = 10000 # Number of samples to train on.# Path to the data txt file on disk.data_path = 'fra-eng/fra.txt'# Vectorize the data.input_texts = []target_texts = []input_characters = set()target_characters = set()lines = open(data_path, 'r', encoding='utf-8').read().split('\n')for line in lines[: min(num_samples, len(lines) - 1)]: input_text, target_text = line.split('\t') # We use "tab" as the "start sequence" character # for the targets, and "\n" as "end sequence" character. target_text = '\t' + target_text + '\n' input_texts.append(input_text) target_texts.append(target_text) for char in input_text: if char not in input_characters: input_characters.add(char) for char in target_text: if char not in target_characters: target_characters.add(char)input_characters = sorted(list(input_characters))target_characters = sorted(list(target_characters))num_encoder_tokens = len(input_characters)num_decoder_tokens = len(target_characters)max_encoder_seq_length = max([len(txt) for txt in input_texts])max_decoder_seq_length = max([len(txt) for txt in target_texts])print('Number of samples:', len(input_texts))print('Number of unique input tokens:', num_encoder_tokens)print('Number of unique output tokens:', num_decoder_tokens)print('Max sequence length for inputs:', max_encoder_seq_length)print('Max sequence length for outputs:', max_decoder_seq_length)input_token_index = dict( [(char, i) for i, char in enumerate(input_characters)])target_token_index = dict( [(char, i) for i, char in enumerate(target_characters)])encoder_input_data = np.zeros( (len(input_texts), max_encoder_seq_length, num_encoder_tokens), dtype='float32')decoder_input_data = np.zeros( (len(input_texts), max_decoder_seq_length, num_decoder_tokens), dtype='float32')decoder_target_data = np.zeros( (len(input_texts), max_decoder_seq_length, num_decoder_tokens), dtype='float32')for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)): for t, char in enumerate(input_text): encoder_input_data[i, t, input_token_index[char]] = 1. for t, char in enumerate(target_text): # decoder_target_data is ahead of decoder_input_data by one timestep decoder_input_data[i, t, target_token_index[char]] = 1. if t > 0: # decoder_target_data will be ahead by one timestep # and will not include the start character. decoder_target_data[i, t - 1, target_token_index[char]] = 1.# Define an input sequence and process it.encoder_inputs = Input(shape=(None, num_encoder_tokens))encoder = LSTM(latent_dim, return_state=True)encoder_outputs, state_h, state_c = encoder(encoder_inputs)# We discard `encoder_outputs` and only keep the states.encoder_states = [state_h, state_c]# Set up the decoder, using `encoder_states` as initial state.decoder_inputs = Input(shape=(None, num_decoder_tokens))# We set up our decoder to return full output sequences,# and to return internal states as well. We don't use the# return states in the training model, but we will use them in inference.decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)decoder_outputs, _, _ = decoder_lstm(decoder_inputs, initial_state=encoder_states)decoder_dense = Dense(num_decoder_tokens, activation='softmax')decoder_outputs = decoder_dense(decoder_outputs)# Define the model that will turn# `encoder_input_data` & `decoder_input_data` into `decoder_target_data`model = Model([encoder_inputs, decoder_inputs], decoder_outputs)# Run trainingmodel.compile(optimizer='rmsprop', loss='categorical_crossentropy')model.fit([encoder_input_data, decoder_input_data], decoder_target_data, batch_size=batch_size, epochs=epochs, validation_split=0.2)# Save modelmodel.save('s2s.h5')# Next: inference mode (sampling).# Here's the drill:# 1) encode input and retrieve initial decoder state# 2) run one step of decoder with this initial state# and a "start of sequence" token as target.# Output will be the next target token# 3) Repeat with the current target token and current states# Define sampling modelsencoder_model = Model(encoder_inputs, encoder_states)decoder_state_input_h = Input(shape=(latent_dim,))decoder_state_input_c = Input(shape=(latent_dim,))decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]decoder_outputs, state_h, state_c = decoder_lstm( decoder_inputs, initial_state=decoder_states_inputs)decoder_states = [state_h, state_c]decoder_outputs = decoder_dense(decoder_outputs)decoder_model = Model( [decoder_inputs] + decoder_states_inputs, [decoder_outputs] + decoder_states)# Reverse-lookup token index to decode sequences back to# something readable.reverse_input_char_index = dict( (i, char) for char, i in input_token_index.items())reverse_target_char_index = dict( (i, char) for char, i in target_token_index.items())def decode_sequence(input_seq): # Encode the input as state vectors. states_value = encoder_model.predict(input_seq) # Generate empty target sequence of length 1. target_seq = np.zeros((1, 1, num_decoder_tokens)) # Populate the first character of target sequence with the start character. target_seq[0, 0, target_token_index['\t']] = 1. # Sampling loop for a batch of sequences # (to simplify, here we assume a batch of size 1). stop_condition = False decoded_sentence = '' while not stop_condition: output_tokens, h, c = decoder_model.predict( [target_seq] + states_value) # Sample a token sampled_token_index = np.argmax(output_tokens[0, -1, :]) sampled_char = reverse_target_char_index[sampled_token_index] decoded_sentence += sampled_char # Exit condition: either hit max length # or find stop character. if (sampled_char == '\n' or len(decoded_sentence) > max_decoder_seq_length): stop_condition = True # Update the target sequence (of length 1). target_seq = np.zeros((1, 1, num_decoder_tokens)) target_seq[0, 0, sampled_token_index] = 1. # Update states states_value = [h, c] return decoded_sentencefor seq_index in range(100): # Take one sequence (part of the training set) # for trying out decoding. input_seq = encoder_input_data[seq_index: seq_index + 1] decoded_sentence = decode_sequence(input_seq) print('-') print('Input sentence:', input_texts[seq_index]) print('Decoded

训练文件我基本没有改动，然后基本不用改什么就可以直接运行，然后我做了一个predict.py文件，文件稍稍改动了几行：

from __future__ import print_functionfrom keras.models import Modelfrom keras.layers import Input, LSTM, Denseimport numpy as npbatch_size = 64 # Batch size for training.epochs = 100 # Number of epochs to train for.latent_dim = 256 # Latent dimensionality of the encoding space.num_samples = 10000 # Number of samples to train on.# Path to the data txt file on disk.data_path = 'fra-eng/fra.txt'# Vectorize the data.input_texts = []target_texts = []input_characters = set()target_characters = set()lines = open(data_path, 'r', encoding='utf-8').read().split('\n')for line in lines[: min(num_samples, len(lines) - 1)]: input_text, target_text = line.split('\t') # We use "tab" as the "start sequence" character # for the targets, and "\n" as "end sequence" character. target_text = '\t' + target_text + '\n' input_texts.append(input_text) target_texts.append(target_text) for char in input_text: if char not in input_characters: input_characters.add(char) for char in target_text: if char not in target_characters: target_characters.add(char)input_characters = sorted(list(input_characters))target_characters = sorted(list(target_characters))num_encoder_tokens = len(input_characters)num_decoder_tokens = len(target_characters)max_encoder_seq_length = max([len(txt) for txt in input_texts])max_decoder_seq_length = max([len(txt) for txt in target_texts])print('Number of samples:', len(input_texts))print('Number of unique input tokens:', num_encoder_tokens)print('Number of unique output tokens:', num_decoder_tokens)print('Max sequence length for inputs:', max_encoder_seq_length)print('Max sequence length for outputs:', max_decoder_seq_length)input_token_index = dict( [(char, i) for i, char in enumerate(input_characters)])target_token_index = dict( [(char, i) for i, char in enumerate(target_characters)])encoder_input_data = np.zeros( (len(input_texts), max_encoder_seq_length, num_encoder_tokens), dtype='float32')decoder_input_data = np.zeros( (len(input_texts), max_decoder_seq_length, num_decoder_tokens), dtype='float32')decoder_target_data = np.zeros( (len(input_texts), max_decoder_seq_length, num_decoder_tokens), dtype='float32')for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)): for t, char in enumerate(input_text): encoder_input_data[i, t, input_token_index[char]] = 1. for t, char in enumerate(target_text): # decoder_target_data is ahead of decoder_input_data by one timestep decoder_input_data[i, t, target_token_index[char]] = 1. if t > 0: # decoder_target_data will be ahead by one timestep # and will not include the start character. decoder_target_data[i, t - 1, target_token_index[char]] = 1.# Define an input sequence and process it.encoder_inputs = Input(shape=(None, num_encoder_tokens))encoder = LSTM(latent_dim, return_state=True)encoder_outputs, state_h, state_c = encoder(encoder_inputs)# We discard `encoder_outputs` and only keep the states.encoder_states = [state_h, state_c]# Set up the decoder, using `encoder_states` as initial state.decoder_inputs = Input(shape=(None, num_decoder_tokens))# We set up our decoder to return full output sequences,# and to return internal states as well. We don't use the# return states in the training model, but we will use them in inference.decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)decoder_outputs, _, _ = decoder_lstm(decoder_inputs, initial_state=encoder_states)decoder_dense = Dense(num_decoder_tokens, activation='softmax')decoder_outputs = decoder_dense(decoder_outputs)# Define the model that will turn# `encoder_input_data` & `decoder_input_data` into `decoder_target_data`model = Model([encoder_inputs, decoder_inputs], decoder_outputs)model.load_weights('s2s.h5')# Run trainingmodel.compile(optimizer='rmsprop', loss='categorical_crossentropy')# Next: inference mode (sampling).# Here's the drill:# 1) encode input and retrieve initial decoder state# 2) run one step of decoder with this initial state# and a "start of sequence" token as target.# Output will be the next target token# 3) Repeat with the current target token and current states# Define sampling modelsencoder_model = Model(encoder_inputs, encoder_states)decoder_state_input_h = Input(shape=(latent_dim,))decoder_state_input_c = Input(shape=(latent_dim,))decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]decoder_outputs, state_h, state_c = decoder_lstm( decoder_inputs, initial_state=decoder_states_inputs)decoder_states = [state_h, state_c]decoder_outputs = decoder_dense(decoder_outputs)decoder_model = Model( [decoder_inputs] + decoder_states_inputs, [decoder_outputs] + decoder_states)# Reverse-lookup token index to decode sequences back to# something readable.reverse_input_char_index = dict( (i, char) for char, i in input_token_index.items())reverse_target_char_index = dict( (i, char) for char, i in target_token_index.items())def decode_sequence(input_seq): # Encode the input as state vectors. states_value = encoder_model.predict(input_seq) # Generate empty target sequence of length 1. target_seq = np.zeros((1, 1, num_decoder_tokens)) # Populate the first character of target sequence with the start character. target_seq[0, 0, target_token_index['\t']] = 1. # Sampling loop for a batch of sequences # (to simplify, here we assume a batch of size 1). stop_condition = False decoded_sentence = '' while not stop_condition: output_tokens, h, c = decoder_model.predict( [target_seq] + states_value) # Sample a token sampled_token_index = np.argmax(output_tokens[0, -1, :]) sampled_char = reverse_target_char_index[sampled_token_index] decoded_sentence += sampled_char # Exit condition: either hit max length # or find stop character. if (sampled_char == '\n' or len(decoded_sentence) > max_decoder_seq_length): stop_condition = True # Update the target sequence (of length 1). target_seq = np.zeros((1, 1, num_decoder_tokens)) target_seq[0, 0, sampled_token_index] = 1. # Update states states_value = [h, c] return decoded_sentencefor seq_index in range(50): # Take one sequence (part of the training set) # for trying out decoding. input_seq = encoder_input_data[seq_index: seq_index + 1] # print(input_seq) decoded_sentence = decode_sequence(input_seq) print('-') print('Input sentence:', input_texts[seq_index]) print('Decoded sentence:', decoded_sentence)

参考文献

​​How to Develop an Encoder-Decoder Model for Sequence-to-Sequence Prediction in Keras​​​. ​​​keras/examples/lstm_seq2seq.py​​

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。