Add following code in your theme’s (child theme’s if you are using) functions.php

add_filter( 'fep_directory_arguments', function( $args ){
    $role_users_to_hide = array( 'subscriber', 'author' ); //here add as many as you want

    if ( isset( $args['role__not_in'] ) ) {
        $args['role__not_in'] = array_merge( $args['role__not_in'], $role_users_to_hide );
    } else {
        $args['role__not_in'] = $role_users_to_hide;
    }

    return $args;
}, 99);

change/add as many roles slug as you want in place of subscriber, author