Guide to scores#
Custom scores#
Any custom score class should inherit from fedsim.scores.Score
(or its children) and implement its abstract methods.
Integration with fedsim-cli#
To automatically include your custom score by the provided cli tool, you can define it in a python file and pass its path to --global-score
or --local-score
option (without .py) followed by column and name of the score definition (class or method).
For example, if you have score CustomScore
stored in a foo/bar/my_custom_score.py
, you can pass --global-score foo/bar/my_custom_score:CustomScore
for setting global optimizer or --local-score foo/bar/my_custom_score:CustomScore
for setting the local score.
Note
Arguments of constructor of any score could be given in arg:value
format following its name (or path if a local file is provided). Examples:
fedsim-cli fed-learn --global-score Accuracy log_freq:20 split:test ...
fedsim-cli fed-learn --local-score foo/bar/my_custom_sore:CustomScore log_freq:30 split:train ...
Note
scores can be passed to --criterion
option the same way, however, if the selected score class is not differentiable an error may be raised (if necessary).