Unleashing the Power of Structured Output from Language Models: An In-Depth Look at Techniques and Tools

As the quest for structured output from language models (LLMs) intensifies, developers are increasingly facing a myriad of challenges and opportunities. Structured data output – such as JSON and XML – appears to be indispensable for numerous applications, including data parsing, integration, and machine-to-machine communication. But how can developers extract this structured data effectively from LLMs, and which tools can simplify this complex task? In this detailed exploration, we will delve into various approaches and tools available, analyzing their strengths and deficiencies.

One of the most commonly embraced methods for obtaining structured output is utilizing JSON. JSON’s ubiquity and relative simplicity make it an attractive target format. However, developers often deal with the imperfections in the JSON produced by LLMs. Issues such as missing or misplaced commas, incorrect data types, and unescaped characters are frequent pain points. Tools like BAML have been designed to handle such malformed JSON outputs. BAML takes a unique approach by offering a fuzzy parser that addresses missing or incorrect elements, turning these blemished outputs into valid, usable JSON.

Yet, JSON is not the only structured format under the spotlight. For instance, XML, a markup language that predates JSON, also holds substantial weight in structured data exchange. According to the comments from enthusiasts on various forums, XML’s well-formed nature can sometimes surpass JSON in terms of structural integrity. An LLM trained with XML examples can often handle nested structures and complex hierarchies more effectively with XML tags, leading to easier parsing due to the clear endings of tags such as </output>. However, XML’s verbosity adds to the token count, making the process slower and potentially costlier for computational resources.

Both JSON and XML require unique handling techniques when working with LLMs. In some development environments, such as PromptFiddle, engineers can experiment with interactive versions of these libraries to see how different models and formats perform with their specific data sets. While JSON’s succinct structure is often easier to manage programmatically, XML’s clear tag definitions can simplify parsing logic, despite its verbose nature. These trade-offs should be considered based on specific project needs and the model’s compatibility with each format.

Another notable challenge involves making LLMs comply with specified grammars and schemas. Users have noted that using a tool like llama.cpp, which constrains outputs to predefined grammars, can be particularly effective. This method ensures that the generated content strictly adheres to a given schema, minimizing the need for extensive post-processing and improvement cycles. However, this technique can also lead to a decrease in model performance, highlighting the delicate balance between structural accuracy and output quality.

Different libraries and frameworks provide varied functionalities to facilitate this structured output. For instance, Instructor relies on repeated attempts to produce valid JSON, employing a retry mechanism to refine the output until it conforms to the desired schema. This can be a more accessible approach for users who do not own the models and consequently cannot use token masking. On the other hand, libraries such as Claude can return structured XML effectively without much additional work from the user, proving that the choice of library can significantly impact workflow efficiency.

A fascinating aspect highlighted by BAML’s team is the use of dynamic type support, which can be immensely helpful when dealing with variable data structures. For example, in Python, a dynamic type setup can be implemented to streamline the extraction of structured data. The code snippet shared by developers illustrates this approach beautifully:

async def test_dynamic():
    tb = TypeBuilder()

    tb.Person.add_property('last_name', tb.string().list())

    tb.Person.add_property('height', tb.float().optional()).description(
        'Height in meters'
    )


    tb.Hobby.add_value('chess')

    for name, val in tb.Hobby.list_values():
        val.alias(name.lower())

    tb.Person.add_property('hobbies', tb.Hobby.type().list()).description(
        'Some suggested hobbies they might be good at'
    )

    tb_res = await b.ExtractPeople(
        'My name is Harrison. My hair is black and I am 6 feet tall. I am pretty good around the hoop.',
        {'tb': tb},
    )

    assert len(tb_res) > 0, 'Expected non-empty result but got empty.'

    for r in tb_res:
        print(r.model_dump())

While some developers swear by the flexibility and control provided by these advanced methods, others prefer more straightforward solutions. The Python library Pydantic, for instance, is hailed for its close resemblance to declarative configurations, offering robust parsing capabilities reminiscent of BAML’s static analysis features. Pydantic models can be dynamically created and validated with less complexity, enhancing developer productivity without compromising on the flexibility needed to handle diverse and dynamic data models.

Exploring structured output doesn’t end with choosing the correct format and tool. Developers need to consider the practical implications, such as computational efficiency and the accuracy of the generated data. For some, Markdown has emerged as a pragmatic solution. It offers machine-readable structures while allowing the model to articulate its thought process fully. This dual benefit ensures high-quality outputs and easier parsing, making it a potential alternative for those grappling with rigid JSON or XML outputs.

Ultimately, the landscape of structured output from LLMs is rich with possibilities and intricacies. Each technique, whether it be JSON, XML, or a bespoke DSL like those used in BAML, provides unique advantages and trade-offs. The choice of method and tool will invariably depend on the specific needs of a project, the constraints of the programming environment, and the desired output quality. As developers continue to push the boundaries of what LLMs can achieve, these tools and techniques will undoubtedly evolve, offering improved functionality and increased efficiency in the world of structured data output.