Era batches for trainning Keras NN

Data Science
1

I am trying to train a NN with era batches using Keras data generators. I am not familiar with data generators but I have found them very easy to understand after a couple of tutorials. Still, I can’t get it to work and I can’t find where I’m going wrong; the class is very simple and has all the methods required for the Keras model to work.

class EraDataGenerator(tf.keras.utils.Sequence):
'Generates data for Keras'

def __init__(self, X, y, shuffle=True):
    'Initialization'
    self.X = X
    self.y = y
    self.dim = len(X.columns)
    self.eras = X.era.unique()
    self.shuffle = shuffle
    
    if self.shuffle == True:
        np.random.shuffle(self.eras)

    self.on_epoch_end()

def __len__(self):
    'Num of batches per epoch'
    return len(self.eras)

def __getitem__(self, idx):

    myEras = [idx]
    
    X = self.X.loc[self.X.era.isin(myEras), self.features].values
    y = self.y.loc[self.X.era.isin(myEras), 'target'].values
    
    return X, y

def on_epoch_end(self):
    'Mixes eras order after each epoch'
    if self.shuffle == True:
        np.random.shuffle(self.eras)

The error that occurs is as follows:

ValueError: Failed to find data adapter that can handle input: <class '__main__.EraDataGenerator'>, <class 'NoneType'>

Has anyone used Data Generators for the same case and managed to implement it without problems?

1 Like
2

__getitem() should return an X,y tuple

3

still getting the same error

4
1 Like