Guide to learning rate schedulers#
fedsim-cli fed-learn accepts 3 scheduler objects.
lr-scheduler: learning rate scheduler for server optimizer.
local-lr-scheduler: learning rate scheduler for client optimizer.
r2r-local-lr-scheduler: schedules the initial learning rate that is delivered to the clients of each round.
These arguments are passed to instances of the centralized FL algorithms.
Note
Choose learning rate schedulers from torch.optim.lr_scheduler
documented at Lr Schedulers Page or define a learning rate scheduler class that has the common methods (step
, get_last_lr
, etc.).
Note
For now fedsim-cli
does not support the learning rate schedulers that require another object in their constructor (such as LambdaLR
) or a dynamic value in their step function (ReduceLROnPlateau
).
To implement one with similar functionality, you can implement one and assign it to self.r2r_local_lr_scheduler
inside the constructor of your custom algorithm (after calling super).
Custom Learning Rate Scheduler#
Any custom learning rate scheduler class should implement the common methods of torch optim lr schedulers.
Integration with fedsim-cli#
To automatically include your custom lr scheduler by the provided cli tool, you can define it in a python file and pass its path to --lr-scheduler
or --local-lr-scheduler
or r2r-local-lr-scheduler
option (without .py) followed by column and name of the lr scheduler definition (class or method).
For example, if you have score CustomLRS
stored in a foo/bar/my_custom_lr_scheduler.py
, you can pass --lr-scheduler foo/bar/my_custom_lr_scheduler:CustomLRS
for setting global lr scheduler or --local-lr-scheduler foo/bar/my_custom_lr_scheduler:CustomLRS
for setting the local lr scheduler or --r22-local-lr-scheduler foo/bar/my_custom_lr_scheduler:CustomLRS
for setting the round to round lr scheduler.
The latter determines the initial learning rate of the local optimizer at each round.
Note
Arguments of constructor of any lr scheduler could be given in arg:value
format following its name (or path if a local file is provided). Examples:
fedsim-cli fed-learn --lr-scheduler StepLR step_size:200 gamma:0.5 ...
fedsim-cli fed-learn --local-lr-scheduler foo/bar/my_custom_lr_scheduler:CustomLRS step_size:10 beta:0.1 ...