Integrating the neural network in bagofwords.py

I wrote code containing a neural network that runs through (part1.py in johanna_NN), but I'm not getting to work as a class (see bagofwords.py in johanna_NN).

My understanding of the problem is that I'm creating an instance of the DNNClassifier class within the NeuralNetworkClassifier class, and apply a NeuralNetworkClassifier method to this instance of DNNClassifier.

I've also described the problem on StackOverflow but did not get any reactions yet (https://stackoverflow.com/questions/49206732/python-using-one-class-inside-another?noredirect=1#comment85418468_49206732).

Any insights?

[twesterhout]

OK, so I'm afraid I'm unable to test your code right now as I'm constantly running out of memory (even on smaller datasets). I'll probably come up with a way around it later, but for now, here are a few thoughts about your code:

• Assume set_up_architecture, shape_train_input, and shape_eval_input work correctly.

• I think you misunderstood the purpose of {some classifier}.fit(x, y). It your case, it should do more or less the following:

# WARNING: Untested code.
def fit(self, x, y):
    train_df, new_classifier = self.set_up_architecture(x, y)
    # Let's store the trained classifier, because we don't want to retrain
    # the model every time someone calls predict. This means that you
    # should initialise self.classifier during construction, i.e. add something
    # like `self.classifier = None` to `__init__` function.
    self.classifier = new_classifier.train(
        input_fn=lambda: self.shape_train_input(
            train_df, y, self.batch_size),
        # You had a hard-coded 1000 here. I believe it's a bug.
        steps=self.n_steps)
    # Notice that an instance of NeuralNetworkClassifier is returned here
    # rather than `tensorflow.estimator.DNNClassifier`. This is by definition
    # (which I should have explained better) of `fit`: given a classifier and
    # some data to train it on, you return the trained version of the very same
    # classifier.
    return self

• About {some classifier}.predict(x). I think your understanding here is correct, but the implementation has a bug. You should probably call self.classifier.evaluate rather than self.evaluate. I.e. something along these lines:

# WARNING: Untested code.
def predict(self, x):
    test_df = pd.DataFrame(x.toarray())
    test_df.columns = test_df.columns.astype(str)
    return self.classifier.evaluate(
        input_fn=lambda: shape_eval_input(test_df, None, self.batch_size))

Hey Tom, thanks so much for thinking along! I'll have time to work on your suggestions tomorrow and will let you know about my progress :)

I've followed your suggestions and the fit method now indeed returns an instance of the NeuralNetworkClassifier class, which is good:

a = _make_classifier(conf)
b = a.fit(train_data_features, train_data_sentiment) 

type(b)
Out[166]: __main__.NeuralNetworkClassifier

However, running the predict method on this instance is still not working.

c = b.predict(test_data_features)

Traceback (most recent call last):

  File "<ipython-input-167-cbff1995c6d9>", line 1, in <module>
    c = b.predict(test_data_features)

  File "<ipython-input-145-0282d567f3f2>", line 90, in predict
    lambda:shape_pred_input(test_df,

  File "C:\Users\johan\Anaconda3\envs\tf35\lib\site-packages\tensorflow\python\estimator\estimator.py", line 367, in evaluate
    name=name)

  File "C:\Users\johan\Anaconda3\envs\tf35\lib\site-packages\tensorflow\python\estimator\estimator.py", line 841, in _evaluate_model
    input_fn, model_fn_lib.ModeKeys.EVAL))

  File "C:\Users\johan\Anaconda3\envs\tf35\lib\site-packages\tensorflow\python\estimator\estimator.py", line 603, in _get_features_and_labels_from_input_fn
    result = self._call_input_fn(input_fn, mode)

  File "C:\Users\johan\Anaconda3\envs\tf35\lib\site-packages\tensorflow\python\estimator\estimator.py", line 694, in _call_input_fn
    return input_fn(**kwargs)

  File "<ipython-input-145-0282d567f3f2>", line 92, in <lambda>
    self.batch_size))

TypeError: shape_pred_input() missing 1 required positional argument: 'batch_size'

I don't understand this because it seems to me as if I correctly fed three arguments to the shape_pred_input method from within the predict method.

I'll paste the revised code for this class here:

class NeuralNetworkClassifier(object):
    
    def __init__(self, batch_size, n_steps, n_hidden_units1, n_hidden_units2, 
                 n_classes):
        self.batch_size = batch_size
        self.n_steps = n_steps
        self.n_hidden_units1 = n_hidden_units1
        self.n_hidden_units2 = n_hidden_units2
        self.n_classes = n_classes
        self.classifier = None
    
    def set_up_architecture(self, train_data_features, train_sentiments):
           
        # Convert the scarce scipy feature matrices to pandas dataframes
        train_df = pd.DataFrame(train_data_features.toarray())
        
        # Convert column names from numbers to strings
        train_df.columns = train_df.columns.astype(str)

        # Create feature columns which describe how to use the input
        feat_cols = []
        for key in train_df.keys(): 
            feat_cols.append(tf.feature_column.numeric_column(key=key))
            
        # Set up classifier with two hidden unit layers
        classifier = tf.estimator.DNNClassifier(
                                        feature_columns=feat_cols, 
                                        hidden_units=[self.n_hidden_units1, 
                                                      self.n_hidden_units2],
                                        n_classes=self.n_classes)
        
        return train_df, classifier
    
    def shape_train_input(self, features, labels, batch_size):
        """An input function for training"""
        
        # Convert the input to a dataset
        dataset = tf.data.Dataset.from_tensor_slices((dict(features), labels))
    
        # Shuffle, repeat, and batch the examples
        dataset = dataset.shuffle(1000).repeat().batch(self.batch_size)

        return dataset

    def fit(self, train_data_features, train_sentiments):
        
        train_df, new_classifier = self.set_up_architecture(train_data_features, 
                                                   train_sentiments)
        #print(type(classifier))
        
        self.classifier = new_classifier.train(input_fn=
                                      lambda:self.shape_train_input(train_df,
                                                              train_sentiments,
                                                              self.batch_size),
                                                              steps=self.n_steps)
        
        return self
    
    def shape_pred_input(self, features, labels, batch_size):
        """An input function for evaluation or prediction"""
        
        features=dict(features)
        if labels is None:
            # No labels, use only features.
            inputs = features
        else:
            inputs = (features, labels)
    
        # Convert the inputs to a dataset
        dataset = tf.data.Dataset.from_tensor_slices(inputs)
    
        # Batch the examples
        assert batch_size is not None, "batch_size must not be None"
        dataset = dataset.batch(self.batch_size)
    
        return dataset
    
    def predict(self, test_data_features):
        
        # Convert the scarce scipy feature matrices to pandas dataframes
        test_df = pd.DataFrame(test_data_features.toarray())
        
        # Convert column names from numbers to strings
        test_df.columns = test_df.columns.astype(str)
        
        predictions = self.classifier.evaluate(input_fn=
                                    lambda:shape_pred_input(test_df, 
                                                            None, 
                                                            self.batch_size))
        
        return predictions

[twesterhout]

My bad, see #34.

#34 is resolved!

Apart from that I'm still working on this issue.

The self.predict method currently doesn't work as long no labels are being fed. The self.shape_pred_input method suggests that this is OK (it checks whether or not 'labels' is None), but if labels is indeed None an error arises (see the head module):

ValueError(
        'You must provide a labels Tensor. Given: None. '
        'Suggested troubleshooting steps: Check that your data contain '
        'your label feature. Check that your input_fn properly parses and '
        'returns labels.')

Currently the code can run because I'm specifying that labels = test_data_sentiment from within the self.predict method, but this needs to change. [No help needed just yet]