Every AI Tool We Add Becomes an Attack Surface, by Tracy Bannon
- Mark Miller
- Jun 2
- 2 min read
Hey there, Gang! - You may have seen the article by Dan Goodin at Ars Technica that dropped on May 23rd: “Researchers cause GitLab AI developer assistant to turn safe code malicious.”
In short? Researchers at Legit Security crafted benign-looking merge requests that manipulated GitLab Duo, an AI coding assistant, into injecting malicious code.
This is one of those “I told you so” moments. We’ve all been championing the groundbreaking potential of AI-augmented software delivery. But here’s the architectural truth: 𝗲𝘃𝗲𝗿𝘆 𝘁𝗼𝗼𝗹 𝘄𝗲 𝗮𝗱𝗱 𝗯𝗲𝗰𝗼𝗺𝗲𝘀 𝗮𝗻 𝗮𝘁𝘁𝗮𝗰𝗸 𝘀𝘂𝗿𝗳𝗮𝗰𝗲.
This incident underscores the very themes we’ve been exploring at MITRE under our ArchAITecture research initiative for 18+ months. A few takeaways worth amplifying:
Adversarial prompt injection is not hypothetical—it’s happening. Architect for AI resilience from the outset using adversarial testing pipelines, prompt injection simulations, and validation gates.
AI-generated code ≠ trusted code. Apply zero trust principles. Use layered security (SAST/DAST), enforce human-in-the-loop reviews, and restrict AI autonomy in sensitive paths.
Threat modeling must include the AI assistant itself. Leverage frameworks like MITRE ATT&CK and ATLAS to document misuse cases, abuse vectors, and system-level risks.
Monitoring equals safety. Treat AI tooling like production infrastructure—instrument with telemetry, track usage patterns and rejections, and feed back into tuning cycles. (We’re seeing early signals through GitHub, GitClear, Software.com, and BlueOptima.)
As I started jotting these thoughts, 3–4 more surfaced—but I’ll pause here.
This is a conversation we all need to be having. AI is here to stay. We need to lead with intentionality, guardrails, and trust—but not blind trust.
Tracy's session, "The AI Transition: From Tool to Teammate to Trusted Core", will be streamed live September 16-17, 2025.
About Tracy Bannon
Passionate Architect!!! Tracy “Trac” Bannon is a Sr. Principal with the MITRE Corporation. She is an accomplished software architect, engineer, and researcher specializing in applying AI/ML and Generative AI to the software development lifecycle. Trac's work is at the intersection of pioneering software practices and transformative leadership in technology.
She’s passionate about community building and knowledge sharing. Trac is a long-time advocate for diversity in technology, helping to narrow the gaps as a mentor, sponsor, volunteer, and friend.
Ms. Bannon is a featured industry speaker, blogger, author, and accomplished podcaster including 𝗥𝗲𝗮𝗹 𝗧𝗲𝗰𝗵𝗻𝗼𝗹𝗼𝗴𝗶𝘀𝘁𝘀. She graduated from Penn State with a master’s degree in software engineering; her undergraduate days were with Penn State as well studying computer integration and graphics. Trac is a senior member of the IEEE.
The Artificial (Un)Intelligence Conference is a global, 24 hour live online conference. Registration is free, and includes access to all sessions, including on-demand, at the conclusion of the event. No sales pitches, no marketing. Just the good stuff.
We're trying to find the unheralded people around the world who are doing cool things with AI. Is that you?
Check out our speaker gallery and then let us know what you're working on. We look forward to hearing your story.
More speakers added daily. Register to get the latest update.
Comments