AssertionError: Unexpected kwargs: {'use_flash_attention_2': False}

Question

I'm using EvolvingLMMs-Lab/lmms-eval to evaluate LLaVa model

after running accelerate launch --num_processes=8 -m lmms_eval --model llava --model_args pretrained="liuhaotian/llava-v1.5-7b" --tasks mme --batch_size 1 --log_samples --log_samples_suffix llava_v1.5_mme --output_path ./logs/

i got this error:

AssertionError: Unexpected kwargs: {'use_flash_attention_2': False}


Traceback (most recent call last):
  File "/home/c7361313/lmms-eval/lmms_eval/__main__.py", line 203, in cli_evaluate
    results, samples = cli_evaluate_single(args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c7361313/lmms-eval/lmms_eval/__main__.py", line 299, in cli_evaluate_single
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c7361313/lmms-eval/lmms_eval/utils.py", line 425, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/c7361313/lmms-eval/lmms_eval/evaluator.py", line 94, in simple_evaluate
    lm = lmms_eval.api.registry.get_model(model).create_from_arg_string(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c7361313/lmms-eval/lmms_eval/api/model.py", line 92, in create_from_arg_string
    return cls(**args, **args2)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/c7361313/lmms-eval/lmms_eval/models/llava.py", line 67, in __init__
    assert kwargs == {}, f"Unexpected kwargs: {kwargs}"
           ^^^^^^^^^^^^
AssertionError: Unexpected kwargs: {'use_flash_attention_2': False}
2024-07-09 16:24:53.562 | ERROR    | __main__:cli_evaluate:217 - Error during evaluation: Unexpected kwargs: {'use_flash_attention_2': False}
2024-07-09 16:24:53.562 | ERROR    | __main__:cli_evaluate:217 - Error during evaluation: Unexpected kwargs: {'use_flash_attention_2': False}
Traceback (most recent call last):
  File "/home/c7361313/lmms-eval/lmms_eval/__main__.py", line 203, in cli_evaluate
    results, samples = cli_evaluate_single(args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c7361313/lmms-eval/lmms_eval/__main__.py", line 299, in cli_evaluate_single
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c7361313/lmms-eval/lmms_eval/utils.py", line 425, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/c7361313/lmms-eval/lmms_eval/evaluator.py", line 94, in simple_evaluate
    lm = lmms_eval.api.registry.get_model(model).create_from_arg_string(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c7361313/lmms-eval/lmms_eval/api/model.py", line 92, in create_from_arg_string
    return cls(**args, **args2)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/c7361313/lmms-eval/lmms_eval/models/llava.py", line 67, in __init__
    assert kwargs == {}, f"Unexpected kwargs: {kwargs}"
           ^^^^^^^^^^^^
AssertionError: Unexpected kwargs: {'use_flash_attention_2': False}