Unable to use Pytorch Dataloader in Skorch NeuralNetClassifier

[animeshkumarpaul]

I am trying to pass the PyTorch data loader, but it gives the error-

TypeError: 'DataLoader' object is not callable

Sample Code:

samples_weights_train = np.array(-----------)
sampler_train = WeightedRandomSampler(weights=samples_weights_train, num_samples=len(y_train)*2, replacement=True)
X_NN_train = torch.from_numpy(np.asarray(X_train, dtype=np.float32)).float()        
y_NN_train = torch.from_numpy(y_train).long()
train_dataset = TensorDataset(X_NN_train, y_NN_train)
train_loader = DataLoader(train_dataset, sampler = sampler_train, batch_size = 500, num_workers=1)


samples_weights_valid = np.array(------------)
sampler_valid = WeightedRandomSampler(weights=samples_weights_valid, num_samples=len(y_valid)*2, replacement=True)
X_NN_valid = torch.from_numpy(np.asarray(X_valid, dtype=np.float32)).float()        
y_NN_valid = torch.from_numpy(y_valid).long()
valid_dataset = TensorDataset(X_NN_valid, y_NN_valid)
valid_loader = DataLoader(valid_dataset, sampler = sampler_valid, batch_size = 500, num_workers=1)

from skorch import NeuralNetClassifier
net = NeuralNetClassifier(
    ClassifierModule(layer_sizes=[16384, 2], input_size=X_valid_transformed.shape[1]),
    max_epochs=500,
    batch_size= 50000,
    criterion=nn.CrossEntropyLoss(weight=torch.FloatTensor([w1,w2]),reduction='mean')
    optimizer=torch.optim.Adam,
    lr=0.01,
    optimizer__weight_decay = 1e-4,
    iterator_train__shuffle=True,
    device='cuda:0',
    callbacks=[early_stopping,lr_scheduler,initializer_cb, checkpoint_cb,EpochScoring_cb_train,EpochScoring_cb_valid],
    train_split = None,
    iterator_train =  DataLoader(train_dataset, sampler = sampler_train, batch_size = 500, num_workers=1), #train_loader,
    iterator_valid = DataLoader(valid_dataset, sampler = sampler_valid, batch_size = 500, num_workers=1)#valid_loader
    )

net.fit(X_NN_train,y=None)

How can I pass this Pytorch data loader to here? Please help

[BenjaminBossan]

I converted the issue into a discussion, I hope you don't mind.

Thanks for providing the code. Some points regarding your example:

1. I assume you use the default DataLoader from PyTorch. In that case, you don't have to pass it at all, it is used by default.
2. You instantiate a lot of classes. Although that is possible, it is not necessary. It is actually preferred to pass the class and use the double-underscore notation to pass it's argument. So e.g. instead of passing ClassifierModule(layer_sizes=...), just pass ClassifierModule and module__layer_sizes=....
3. You have some conflicting parameters, e.g. the sample size is set to both 500 and 50000.
4. You manually convert to PyTorch tensors from numpy and create the dataset, both of which is not required, as skorch does this for you automatically.
5. When using train_split=None, skorch does not perform a train-valid split. This means that all the stuff you set up for validation makes no difference, except when calling predict or predict_proba. If you want a train-valid split during training, but you want to use a predefined split, take a look at predefined_split. Otherwise, if you are fine with skorch making the split, just concatenate X_train+X_valid and y_train+y_valid into one big X and y and pass them to fit.

Overall, your code could be changed like this:

sampler_train = WeightedRandomSampler(weights=samples_weights_train, num_samples=len(y_train)*2, replacement=True)
X_train = X_train.astype(np.float32)
y_train = y_train.astype(np.int64)

sampler_valid = WeightedRandomSampler(weights=samples_weights_valid, num_samples=len(y_valid)*2, replacement=True)
X_valid = X_valid.astype(np.float32)
y_valid = y_valid.astype(np.int64)

from skorch import NeuralNetClassifier

net = NeuralNetClassifier(
    ClassifierModule,
    module__layer_sizes=[16384, 2],
    module__input_size=X_train.shape[1],
    max_epochs=500,
    batch_size=500,
    criterion=nn.CrossEntropyLoss,
    criterion__weight=torch.FloatTensor([w1,w2]),
    criterion__reduction='mean',
    optimizer=torch.optim.Adam,
    lr=0.01,
    optimizer__weight_decay=1e-4,
    iterator_train__shuffle=True,
    iterator_train__workers=1,
    iterator_train__sampler=sampler_train,
    iterator_valid__workers=1,
    iterator_valid__sampler=sampler_valid,
    device='cuda:0',
    callbacks=[early_stopping,lr_scheduler,initializer_cb, checkpoint_cb,EpochScoring_cb_train,EpochScoring_cb_valid],
    train_split=None,
)

net.fit(X_train, y_train)

Maybe you'll need to make a few adjustments, but this is the idea.

[animeshkumarpaul]

Thanks a lot.
The main purpose of weighted sampling is for helping the training process given the training data.

I have this scoring-
EpochScoring_cb_train = EpochScoring(scoring='f1', lower_is_better=False, on_train = True, target_extractor= target_extractor_custom, name = 'f1_train')

I am wondering if is it possible to have the EpochScoring on the original training data (means without the weighted sampler)?

[BenjaminBossan]

I see now that the part about random sampling for the validation data is maybe broken. The problem there is that there might be a mismatch when evaluating the score because the predictions and the y_valid will not be aligned anymore after random sampling, resulting in scores that are just at random level. Probably it is necessary to remove the sampler_valid.

I am wondering if is it possible to have the EpochScoring on the original training data (means without the weighted sampler)?

Nothing easy comes to mind. Probably, what you would have to do is have a custom EpochScoring that holds a reference to the original training data (before sampling) and calculates scores based on that. But if you remove the random sampling for the validation data, as I suggested above, that might already give you enough information that you don't need "unsampled" scores on training data.

[animeshkumarpaul]

Thanks