Anatomy of An AI-Generated Algorithm

Well, a mostly AI-generated algorithm, anyway.   I'm here to break down the individual sections documenting the largest collaborative process between me, Codeium, and Llama-3 to date.

Here's the first section:

import itertools,re,sys
sys.path.append('../')

import utilities

# This is the query that captures the metro areas and population for each city:
# List the top 50 metro areas in the US by population
#
# This is a line from the generated file:
# New York-Newark-Jersey City, NY-NJ-PA: 19,620,000
#
# This is a test response from Llama-3:
# Here are the largest cities in each metropolitan area:
# [(New York|NY|USA), (Los Angeles|CA|USA), 
# (Chicago|IL|USA), (Houston|TX|USA), 
# (Phoenix|AZ|USA), (Philadelphia|PA|USA), 
# (San Antonio|TX|USA), (San Diego|CA|USA), 
# (Dallas|TX|USA), (San Jose|CA|USA)]
# Let me know if you need anything else!

with open('us_metro_areas.txt', 'r') as file:
    data = file.read()

The itertools and re imports show up a bit later in the blog post, but I spent a good 30 minutes fighting with importing the utilities module - which is in a parent directory to this code and is also in the root of my repository.  Codeium continued to tell me I could do things like import '..utilities,' or, when I placed it into another directory, that I could write 'from ..main import utilities' (this apparently is also its own special hell, because Python treats "__main__" as distinct from other directories).  None of this was helpful.

It wasn't until I Googled something like "why do imports in Python suck" that I arrived at the solution you see above.  Before fighting with the import statement, I'd recalled that - for all of the things I like about Python - module imports weren't one of them, so I was anticipating a fight. If I didn't have previous experience with the language, I probably would've spent much more time in the AI-generated wilderness.  I'm still not sure this is the best solution, but it works and it's not complex.

Humans: 1, Bots: 0

When I asked for the top 50 metro areas previously, Meta.ai generously returned the information with no fuss in this format:

New York-Newark-Jersey City, NY-NJ-PA: 19,620,000

Even though I didn't ask for the population, it graciously supplied it to me as a bonus.  The top 50 metro areas are important, because I find it reasonable that people would want to go on vacation where a large percentage of the population lives.  I asked for metro areas first instead of cities to weed out any surprises (like the fact that Jacksonville, not Miami, is the largest city in Florida, even though the metro area for Miami is way larger) in case I need to rank them at some point.

This also helps me restrict search options later for more remote locations (like Yellowstone) when I want to route through the nearest major metro area and then offer a car rental (See? I'm full of wonderful upselling ideas on my fake travel site).

I have to give the bots their due here.  No muss, no fuss and I get what I asked for plus bonus information.  I don't know if it's correct - that's where the real work would happen if I went live - but it passes the sniff test, which is good enough for mock data.

Humans: 1, Bots: 1

And even though opening a file to read data is easy enough to Google for, I like the fact that in about 3 seconds I can start with a comment like "#Open a file..." and it fills it in for me, because I can never remember the syntax on my own.

Humans: 1, Bots: 2

us_cities = []
for line in data.splitlines():
    parts = line.split(':')
    if len(parts) == 2:
        metro_area = parts[0].strip()
        population = parts[1].strip()
        us_cities.append({'metro_area': metro_area, 'population': population})

cities_sub_list = [us_cities[i:i+10] for i in range(0, len(us_cities), 10)]

This is where Codeium started to really shine and, deceptively, where I spent significant time crafting code to split my source data into appropriately sized chunks (particularly the last line).

As with opening the source file, I could've easily parsed the lines of the file into their appropriate sections, but what would've taken me about 5 minutes took Codeium about 10 seconds and was easily verifiable on my part for correctness.

Humans: 1, Bots: 3

I hit a snag when I started contemplating how I wanted to split my 50 entries up to query the Llama-3 API for the largest city in each metro area.  I distrust shoving 50 items at an LLM and expecting it to return something coherent to me by the time I hit item 50.  By that point in time, I may have asked it for a list of cities and it will have started returning ingredients for a pineapple upside-down cake in Esperanto.

Better to stick with small batches where it's more likely to make us both happy.  That means lists of lists and merging the existing dictionary with the newly created one on each sub-iteration.  Writing this after the fact gives me hives.  Thinking about it at the moment nearly sent me into a fugue state.  

I spent far too much time trying to think about the logistics in aggregate (a common developer misstep) rather than break this down into simple composable tasks.  But, I finally came around to thinking about a solution that was easier to iterate through and digest in bite-sized chunks.

I believe my query for Codieum to generate the last line above was "Break the us_cities list into sub-lists of 10," and it returned the code you see now.  It didn't immediately register as intuitive, but a follow-up asking it to explain the individual parts of the list comprehension and running the code for veracity quickly helped me gain a better understanding.  The code, once you get a grasp of Python idioms, is both simple and compact.

Humans: 1, Bots: 4

I will make a brief aside and say - anytime you don't understand why the solution you see is the one presented, even if you can verify its correctness without understanding, slow down and do the work.  Otherwise, you'll quickly reach a point where your generated code is just goo, and asking an LLM to debug it will cause it to helpfully issue one unhelpful suggestion after another creating more goo.

..and now the finale:

cities_new_sub_list = []

location_pattern = r'\((.*?)\|(.*?)\|(.*?)\)'
for city_list in cities_sub_list:
    query = """For the following metro areas return the largest city in the area in the following format:

        [(<city>|<state>|<country>),...]

        {}""".format("\n".join([city['metro_area'] for city in city_list]))
    print(query)

    data = utilities.execute_llm_query(query)
    matches = re.findall(location_pattern, data)
    result = [{'city': match[0], 'state': match[1], 'country': match[2]} for match in matches]

    merged_list = [{**city_list[i], **result[i]} for i in range(len(city_list))]
    cities_new_sub_list.append(merged_list)

us_cities = list(itertools.chain.from_iterable(cities_new_sub_list))

with open('us_cities.txt', 'w') as output_file:
    output_file.write(str(us_cities))

Figuring out the simple regular expression was yet another rabbit hole of my own making before realizing I should take another step back to simplify my process.  

Initially, I asked Llama-3 to return results in the form of something like  "[{'city':<city>, 'state':<state>... and then asked Codeium to extract the city, state, and country as dictionaries in Python.  Then I began to fret about edge cases.  I knew Oklahoma City was on the list of metro areas.  Would the regular expression be able to differentiate between that and the 'city' keyword?  What if I had quotes in responses?  Would I have to ask it to write more logic to handle nested single and double quotes?

Then I remembered - the return response could be in any form I wished.  It didn't need to adhere to a formal API specification.  So, knowing that the data was already going to be in order, I asked it to return data of the form (<city>|<state>|<country>), thus greatly simplifying the regular expression needed to extract the information.  I have to give robot props for both returning an easy-to-parse response and generating the regular expression to parse it.

Humans: 1, Bots: 5

After extracting the information, I now had sublists of responses and needed to mash them together with the original dictionary, as well as merge the sublists into one combined list.  Normally I'd anticipate this would require about 30 minutes on my part to look up the appropriate API calls, try a few simple test cases to verify my solution, and combine the code.  Codeium gave me a solution that shaved about 25 minutes off my development time.  The five-minute elapsed code generation also included the time required on my part to understand the solution and verify its correctness.  

Humans: 1, Bots: 6

...and, again, I didn't have to worry about the file writing syntax.

Humans: 1, Bots: 7

For what is a fairly non-trivial sequence of tasks, I was able to complete the work in about 4-6 hours.  Much of this included getting familiar with the Replicate API I used to call the Llama-3 code and fighting through some preconceived notions of how I needed to interact with the LLMs.  

A second, similar mock data generator took me about 2 hours, and a big part of that time was consolidating code common to the two generators so I could make a more generic framework.

I want to caution people against reading too much into the tally I've been keeping in the article.  I can confidently say that code generation tools, at least for small projects (but ones that span multiple files nonetheless) are a great productivity boon.  However, there is zero chance that I would've been able to finish this if I didn't already have a programming background.  I zipped through a lot of it because I could tell it passed the eyeball test and had a good intuition for where I'd need to take extra care.  Without that experience, this process would've been significantly more painful.  So - still no silver bullets.  

Until next time my human and robot friends.