AI test agents

The infrastructure team built a shared agent framework, so everyone in tech makes agents the same way in one shared workspace. My team built three agents on it:

• Test case analyzer. Reviews my team's existing test repos against best practices. On its first run it found security problems, including passwords stored in insecure places in the test code, so it paid off right away.
• Test case creator. Writes new tests that follow those same best practices, so new work doesn't repeat old problems.
• Mobile data lookup. Answers questions about mobile app data by querying BigQuery directly, so nobody has to wait for a new dashboard.

The analyzer sets the standard for the existing tests, and the creator builds new ones to meet it.

Add who uses the agents and any results (time saved, tests produced). Confirm the security finding is OK to mention publicly.

I used to rely on the data team to build Looker dashboards for the mobile data I needed. Those dashboards went stale, and they needed constant updating because I was always asking for different data.

So I set up the BigQuery command-line tool and built an agent on top of it. Now I can ask a question in plain language and the agent runs the query.

There was a bonus. Queries the data team ran for me sometimes uncovered bugs, because the data came back undefined when an out-of-date library wasn't sending it. Being able to ask more questions, more often, made it easier to catch problems like that.

The automation prototype was first written up as a GitHub extension, and it was designed while the shared framework was still being built. Once the infrastructure team's live agent setup existed, we decided to build on it instead of shipping a separate system, so the prototype never went to production. Add what carried over from the prototype into the live agents.

My team was checking mobile cards by hand. That doesn't keep up with an app on two platforms, so we moved toward nightly smoke suites and AI-generated tests.

AI can write a lot of tests quickly. The risk is a pile of tests nobody trusts. The prototype was built around one rule: a person with risk judgment decides what gets generated and what gets approved.

• QA is the approval gate. AI proposes and QA decides, so the team trusts what runs.
• Generation limits follow risk. Effort goes where a failure would cost the most, instead of generating tests everywhere.
• A single delivery lane. One path from change to release keeps ownership clear.
• Three trigger modes. Different situations start a run in different ways, without separate processes.
• GitHub Copilot as the model. It was the LLM my company had approved, so the design worked inside existing rules.

• A 13-file agent framework, fully specified.
• A Jira epic with eight to nine child stories, ready for engineering to pick up.
• A comparison of two ways to run the agent: a Bitrise step and a GitHub Copilot extension in IntelliJ.
• Onboarding documentation for a Test Quality Agent that runs in IntelliJ through Copilot Chat.