Using LLMs in Scripts¶

Sidekick provides the ability to use large language models (LLMs) to perform tasks within Automation Workbench scripts through a construct called the LLM Operator.

LLM Operator¶

When executed, the LLM Operator calls out to an LLM to complete a specified task on a given input Binary Ninja object type and any additional context needed for that object. For example, if you want an LLM to determine if a given function accesses encrypted strings, then you would do the following:

Define and configure an LLM Operator with the relevant information for that task, including a name for the LLM Operator (e.g. does_reference_encrypted_strings)
Instantiate an LLMOperator object using the name of your defined LLM Operator
Pass a Binary Ninja Function object when calling the LLMOperator object

The following code is an example of how this would look in a script:

references_strings = LLMOperator("does_reference_encrypted_strings")

for func in bv.functions:
    result = references_strings(func)

Definition¶

LLM Operators must be defined and configured in order to use them within scripts. More precisely, when a script instantiates an LLMOperator object with a given LLM Operator name, an LLM Operator definition must exist with that name.

LLM Operator definitions include the following information:

Name: Unique identifier used to reference the LLM Operator. This value is passed to the LLMOperator object during initialization.
Task Description: Textual high-level description of the task for the LLM Operator
Model Name: Name of the model (from the LLM Operator Model Catalog) used for this LLM Operator
Prompt Definition:
- Input Variables: JSON description of variables required by the prompt instructions. Variables can reference pre-defined items from the binary such as code, data, strings, libraries, file info, and memory map. They can also reference user-defined variables passed as keyword arguments when calling the LLMOperator object.
- Instructions: Plain text description of the instructions for the LLM to follow for the task. This description can reference input variables that are replaced with the relevant content when the LLMOperator object is called at execution time.
- Output Schema: JSON description that defines the format of the desired output generated by the LLM.

Each LLM Operator definition for a given script has a separate tab (titled Operator: <llm_operator_name>) within the Script Editor, which can be used to view and edit its definition.

LLM Operator Model Catalog¶

The LLM Operator Model Catalog provides a convenient interface for managing the models used by LLMOperators in Automation Workbench scripts. LLM models must be defined and enabled in the catalog in order for LLM Operators to use them.

To open the LLM Operator Model Catalog, select Configure LLMs... from the hamburger menu within the Automation Workbench sidebar.

Available Models¶

The Sidekick plugin includes a default model catalog if one is not already present in your Binary Ninja User Folder. Refer to the default model catalog (located in example_user_models.json within the Sidekick plugin directory) for examples of how to configure models other than the default model.

Modifying Models¶

To modify an existing model in the catalog, select a model from the catalog table. This displays the configuration for that model. Apply edits to the desired fields and click Save Changes.

To enable/disable the model, select/deselect the Enabled checkbox and click Save Changes. Disabling a model will result in Sidekick excluding that model from being used for LLMOperators.

Adding New Models¶

To add a new model to the catalog, click Add New Model. This operation adds an empty, default model to the catalog and displays its configuration for editing. After specifying the necessary fields, click Save Changes.

Deleting Models¶

To delete a model from the catalog, select a model from the catalog table and click Delete Model.