ChemBERTa_augmented_pubchem_13m EU AI Act Compliance Profile

This model is a ChemBERTa model trained on the augmented_canonical_pubchem_13m dataset.

The model was trained for 24 epochs using NVIDIA Apex's FusedAdam optimizer with a reduce-on-plateau learning rate scheduler. To improve performance, mixed precision (fp16), TF32, and torch.compile were enabled. Training used gradient accumulation (16 steps) and batch size of 128 for efficient resource utilization. Evaluation was performed at regular intervals, with the best model selected based on validation performance.

Benchmarks

Classification Datasets (ROC AUC - Higher is better)

| Model | BACE↑ | BBBP↑ | TOX21↑ | HIV↑ | SIDER↑ | CLINTOX↑ | | ------------------------- | ------ | ------ | ------ | ------ | ------ | -------- | | Tasks | 1 | 1 | 12 | 1 | 27 | 2 | | Derify/ChemBERTa_augmented_pubchem_13m | 0.8008 | 0.7418 | 0.7548 | 0.7744 | 0.6313 | 0.9621 |

Regression Datasets (RMSE - Lower is better)

| Model | ESOL↓ | FREESOLV↓ | LIPO↓ | BACE↓ | CLEARANCE↓ | | ------------------------- | ------ | --------- | ------ | ------ | ---------- | | Tasks | 1 | 1 | 1 | 1 | 1 | | Derify/ChemBERTa_augmented_pubchem_13m | 0.8798 | 0.5282 | 0.6853 | 0.9554 | 45.4362 |

Benchmarks were conducted using the chemberta3 framework. Datasets were split with DeepChem’s scaffold splits and filtered to include only molecules with SMILES length ≤200, following MolFormer paper's recommendation. The model was fine-tuned for 100 epochs with a learning rate of 3e-5 and batch size of 32. Each task was run with 3 different random seeds, and the mean performance is reported.

References

ChemBERTa Series

Your obligations as a ChemBERTa_augmented_pubchem_13m deployer

Track your progress

Sign up to track obligations, receive deadline reminders, and generate audit packages.