3 potential ways to use Large Language Models (LLMs) in public interest work

Earlier this year, Bloom laid out some of the tasks language learning models (LLMs) are particularly good at, like answering questions about text we provide them, transforming text in simple ways, and creating summaries of a particular text.

One of the questions we’re interested in is: How might we use LLMs in public interest work?

We recently spent some time experimenting with different use cases in our work and exploring what others in the field are trying. Here’s what we found.

Plain language editing

At its most basic, an LLM is designed to process large amounts of text quickly, making it a great tool for summarizing lengthy or dense texts in a pinch. But LLMs can also go one step further by editing texts for readability.

For example, we tried using OpenAI’s ChatGPT Free model (GPT-4o mini) to edit an example of overly-complicated writing at plainlanguage.gov. We fed ChatGPT the text along with the following prompt: “Please edit this text into plain language. Aim for a 6th grade reading level and feel free to restructure the content to improve readability. You may group similar items if it helps.”

Here’s the result:

Original text

Eligibility

Apply if you are aged (65 years old or older), blind, or disabled and have low income and few resources. Apply if you are terminally ill and want to receive hospice services. Apply if you are aged, blind, or disabled; live in a nursing home; and have low income and limited resources. Apply if you are aged, blind, or disabled and need nursing home care, but can stay at home with special community care services. Apply if you are eligible for Medicare and have low income and limited resources.

ChatGPT revision 1

You can apply if any of these describe you:

• You are 65 or older, blind, or disabled, and have low income and few resources.
• You are terminally ill and need hospice care.
• You are 65 or older, blind, or disabled, and live in a nursing home with low income and limited resources.
• You are 65 or older, blind, or disabled, need nursing home care, but can stay at home with special community services.
• You have Medicare, low income, and limited resources.

The addition of bullet points helped, but the result still left room for simplification. ChatGPT didn’t pick up on where the first, third, and fourth bullet points could be combined to cut down the word count. We also would have found a way to write these without repeatedly using the phrase “you are.”

When we followed up by asking ChatGPT, “Can you find a way to shorten the list by combining the 1st, 3rd, and 4th bullet points without losing the differences between them?” we got the following:

ChatGPT revision 2

Who Can Apply?

You can apply if any of these describe you:

• You are 65 or older, blind, or disabled, and have low income and few resources. You may live in a nursing home, or need nursing home care but can stay at home with special community services.
• You are terminally ill and need hospice care.
• You have Medicare, low income, and limited resources.

That’s not bad — and pretty close to what the human experts at plainlanguage.gov suggested:

Human revision

Eligibility

You may apply for Medicaid if you are:

• Terminally ill and want hospice services
• Eligible for Medicare and have low income and limited resources
• 65 years old or older, blind, or disabled and have low income and few resources and:
• Live in a nursing home
• Need a nursing home care but can stay at home with special community care services

In this example, we got pretty close to a final, usable piece of text, and we got there pretty quickly. However, we’d still suggest a final human edit and would probably land somewhere closer to plainlanguage.gov’s human edit.

We’d also definitely suggest checking any ChatGPT output for accuracy. Not only can an LLM lose nuance, it can downright “hallucinate” (or make up information), which could have dire consequences in public interest work.

Cross referencing content against a source of truth

We know LLMs can answer factual questions about text we provide them, so how might we leverage that in our public interest work?

On a recent project, we came across an instance where our government client needed to reference a large set of rules to determine whether a complaint violated any of them. While this seemed like a good use case for an LLM, it was going to be a lengthy approval process to try this experiment inside the agency.

However, we did find interesting precedent for this use case. In his post Using AI to make sense of policy documents, Dave Guarino from Propel experiments with using OpenAI’s ChatGPT Plus / GPT-4 model to answer a real life SNAP user question by cross referencing SNAP policy language from the Code of Federal Regulations.

The initial experiment yielded promising results, with ChatGPT providing an accurate answer, as well as a breakdown of the rules used to derive the answer. You can see screenshots of the experiment in the post.

More experimentation would be needed, especially when trying to answer more complicated or edge-case user questions, and as always, it would be imperative for a human to check ChatGPT’s answers for accuracy before confirming eligibility with the user. But we’re excited by Guarino’s findings and eager to test this out on a future project.

Creating templated content

LLMs are particularly good at creating content following well-known templates or structures. For example, job searchers in today’s market frequently use ChatGPT to help develop cover letters and resumes.

In public interest work, we could also take advantage of ChatGPT’s ability to understand templates and content structures.

In the previous section, we talked about a current project where we could potentially use an LLM to check whether a filed complaint constituted a rule violation. We could even go one step further by building a custom GPT that could create template response letters, telling the complainant whether or not they had an eligible complaint with next steps on what to do.

We could also create a custom GPT to develop web content following known page structures based on information we provide it. Rebecca Woodbury over at Department of Civic Things did just that with Service Thing, a custom GPT that helps you create a government service web page.

We tried using Service Thing to create a public web page, and here’s how it worked:

We started by telling Service Thing, “I need to create a page for San Francisco’s customer service call center (311) that explains how to call them and what people can call them for. Can you help me?”

The tool then asked these follow up questions to get more information:

• Can you provide a few examples of the types of services or issues people can call 311 for?
• Are there any specific hours of operation or costs involved when calling 311?

After answering these questions, Service Thing was able to give us a complete draft of a service page. We asked for some minor changes and got the following final draft:

ChatGPT revision

Call 311

San Francisco’s 311 Customer Service Call Center is your one-stop shop for city services and information. Whether you need directions, public records, or help contacting a city department, 311 is here to assist 24 hours a day, 7 days a week, at no cost.

Need to know

• Available 24/7, 365 days a year
• No cost to call
• Multilingual assistance available

Before you start

Make sure you have any specific details ready, such as addresses or record numbers, to help the 311 operator assist you more quickly.

1. Dial 311 from any phone within San Francisco.

• If you are outside San Francisco, call 415–701–2311.
• For TTY, press 7.

2. Explain your request to the operator. You can ask for services like:

• MUNI directions
• Vital records (birth, death certificates)
• County Clerk records
• Public records (e.g., property information)
• Contact information for the San Francisco Rent Board

3. The operator will direct your call or provide the information you need.

What’s next

After your call, follow the guidance provided by 311. If you need further assistance, you can call again at any time.

Get help

If you have questions or need help with San Francisco city services, dial 311, 415–701–2311, or visit the San Francisco 311 website.

This was a relatively straightforward example, but shows the potential of building a custom GPT for templated content. All we had to do was feed Service Thing some information about our primary user, their needs, and what they could get help with, and we were able to get a content draft following a typical government service page structure.

People often describe LLMs as being a “calculator for words,” and these examples show some of the ways we could use them to perform basic word operations to support our work. We’ll continue to share as we experiment and learn more!