Guide to scores#
Any custom score class should inherit from
fedsim.scores.Score (or its children) and implement its abstract methods.
Integration with fedsim-cli#
To automatically include your custom score by the provided cli tool, you can define it in a python file and pass its path to
--local-score option (without .py) followed by column and name of the score definition (class or method).
For example, if you have score
CustomScore stored in a
foo/bar/my_custom_score.py, you can pass
--global-score foo/bar/my_custom_score:CustomScore for setting global optimizer or
--local-score foo/bar/my_custom_score:CustomScore for setting the local score.
Arguments of constructor of any score could be given in
arg:value format following its name (or path if a local file is provided). Examples:
fedsim-cli fed-learn --global-score Accuracy log_freq:20 split:test ...
fedsim-cli fed-learn --local-score foo/bar/my_custom_sore:CustomScore log_freq:30 split:train ...
scores can be passed to
--criterion option the same way, however, if the selected score class is not differentiable an error may be raised (if necessary).