AttackEvals API

AttackEval

class AttackEval(attacker, victim, language=None, tokenizer=None, invoke_limit=None, metrics=[])[source]
__init__(attacker, victim, language=None, tokenizer=None, invoke_limit=None, metrics=[])[source]

AttackEval is a class used to evaluate attack metrics in OpenAttack.

Parameters

  • attacker (OpenAttack.attackers.base.Attacker) – An attacker, must be an instance of Attacker .

  • victim (OpenAttack.victim.base.Victim) – A victim model, must be an instance of Vicitm .

  • language (Optional[str]) – The language used for the evaluation. If is None then AttackEval will intelligently select the language based on other parameters.

  • tokenizer (Optional[OpenAttack.text_process.tokenizer.base.Tokenizer]) – A tokenizer used for visualization.

  • invoke_limit (Optional[int]) – Limit on the number of model invokes.

  • metrics (List[Union[OpenAttack.metric.algorithms.base.AttackMetric, OpenAttack.metric.selectors.base.MetricSelector]]) – A list of metrics. Each element must be an instance of AttackMetric or MetricSelector .

eval(dataset, total_len=None, visualize=False, progress_bar=False, num_workers=0, chunk_size=None)[source]

Evaluation function of AttackEval.

Parameters

  • dataset (Iterable[Dict[str, Any]]) – An iterable dataset.

  • total_len (Optional[int]) – Total length of dataset (will be used if dataset doesn’t has a __len__ attribute).

  • visualize (bool) – Display a pretty result for each data in the dataset.

  • progress_bar (bool) – Display a progress bar if True.

  • num_worers – The number of processes running the attack algorithm. Default: 0 (running on the main process).

  • chunk_size (Optional[int]) – Processing pool trunks size.

  • num_workers (int) –

Returns

A dict of attack evaluation summaries.

ieval(dataset, num_workers=0, chunk_size=None)[source]

Iterable evaluation function of AttackEval returns an Iterator of result.

Parameters

  • dataset (Iterable[Dict[str, Any]]) – An iterable dataset.

  • num_worers – The number of processes running the attack algorithm. Default: 0 (running on the main process).

  • chunk_size (Optional[int]) – Processing pool trunks size.

  • num_workers (int) –

Yields

A dict contains the result of each input samples.

Return type

Generator[Dict[str, Any], None, None]