Guide to learning rate schedulers#
fedsim-cli fed-learn accepts 3 scheduler objects.
lr-scheduler: learning rate scheduler for server optimizer.
local-lr-scheduler: learning rate scheduler for client optimizer.
r2r-local-lr-scheduler: schedules the initial learning rate that is delivered to the clients of each round.
These arguments are passed to instances of the centralized FL algorithms.
Choose learning rate schedulers from
torch.optim.lr_scheduler documented at Lr Schedulers Page or define a learning rate scheduler class that has the common methods (
fedsim-cli does not support the learning rate schedulers that require another object in their constructor (such as
LambdaLR) or a dynamic value in their step function (
To implement one with similar functionality, you can implement one and assign it to
self.r2r_local_lr_scheduler inside the constructor of your custom algorithm (after calling super).
Custom Learning Rate Scheduler#
Any custom learning rate scheduler class should implement the common methods of torch optim lr schedulers.
Integration with fedsim-cli#
To automatically include your custom lr scheduler by the provided cli tool, you can define it in a python file and pass its path to
r2r-local-lr-scheduler option (without .py) followed by column and name of the lr scheduler definition (class or method).
For example, if you have score
CustomLRS stored in a
foo/bar/my_custom_lr_scheduler.py, you can pass
--lr-scheduler foo/bar/my_custom_lr_scheduler:CustomLRS for setting global lr scheduler or
--local-lr-scheduler foo/bar/my_custom_lr_scheduler:CustomLRS for setting the local lr scheduler or
--r22-local-lr-scheduler foo/bar/my_custom_lr_scheduler:CustomLRS for setting the round to round lr scheduler.
The latter determines the initial learning rate of the local optimizer at each round.
Arguments of constructor of any lr scheduler could be given in
arg:value format following its name (or path if a local file is provided). Examples:
fedsim-cli fed-learn --lr-scheduler StepLR step_size:200 gamma:0.5 ...
fedsim-cli fed-learn --local-lr-scheduler foo/bar/my_custom_lr_scheduler:CustomLRS step_size:10 beta:0.1 ...