Thanks, and having another step for reviewing each other's code is a really cool extension to this, I'll give it a shot :) Whether it works or it doesn't it could be really interesting for a future post!

Ahh I absolutely love this idea of trying to infuse more emotional and fuzzy attributes to see what the LLM comes up with!

When I broke out the layout and style components I was thinking of being able to change the whole site aesthetic from something like "standard b2b" to "geocities fan page", but I'm excited to try getting fuzzier with the descriptions!

Gave it a shot real quick, looks like I need to fix something up about automatically running the migrations either in the CI script or locally...

But if you're curious, task was this:

----

Title: Bug: Users should be able to add tags to a task to categorize them

Description: Users should be able to add multiple tags to a task but aren't currently able to.

Given I am a user with multiple tasks When I select one Then I should be able to add one or many tags to it

Given I am a user with multiple tasks each with multiple tags When I view the list of tasks Then I should be able to see the tags associated with each task

----

And then we ended up with:

GPT-4o ($0.05): https://github.com/sublayerapp/buggy_todo_app/pull/51

Claude 3.5 Sonnet ($0.09): https://github.com/sublayerapp/buggy_todo_app/pull/52

Gemini 2.0 Flash ($0.0018): https://github.com/sublayerapp/buggy_todo_app/pull/53

One thing to note that I've found - I know you had the "...and you should be able to filter/group tasks by tag" on the request - usually when you have a request that is "feature A AND feature B" you get better results when you break it down into smaller pieces and apply them one by one. I'm pretty confident that if I spent time to get the migrations running, we'd be able to build that request out story-by-story as long as we break it out into bite-sized pieces.

Wasting Inferences with Aider

Yeah, totally agree that something related to this will likely be the next paradigm. I've been putting together experiments in different directions trying to find that thing that's missing but haven't really found a killer use case yet to pull it all together.

That's a really cool idea that once you can get something somewhat reliably consistent generated, you can kind of let your A/B tests start to run themselves with just rough guidelines on what you're trying to optimize for...