To choose an AI automation agency, look for one that can show you a working system similar to what you need, agrees with you on how success will be measured before any work starts, leaves you owning the code, prompts, workflows and accounts, and tells you straight out what it wouldn’t automate. Twelve questions, asked over one or two calls, are enough to tell an AI automation agency worth hiring from one that will leave you paying for a rebuild.
How should you run the selection process?
Treat it like hiring a contractor for a kitchen remodel: the same brief for everyone, a short list, a look at work they’ve done, written quotes and a small first job. In practice, that’s five steps.
The selection process
-
Step 1
Write the brief
One page: the process, volume, systems, what success means and your constraints.
-
Step 2
Shortlist three agencies
Look for ones that have built something close to your process, rather than “AI” in general.
-
Step 3
See a live system
Ask each one to open a working system and let you try it with your own questions.
-
Step 4
Compare written proposals
The same rows for everyone: scope, acceptance criteria, ownership, third-party costs, support, exit.
-
Step 5
Run a paid pilot
One process, a fixed price, a few weeks, and tests the system has to pass before you commit to more.
What goes in the brief?
The brief makes every agency answer the same question, and it exposes the ones that quote without asking you anything.
- The process, step by step, and who handles it today. “Two office staff re-key supplier invoices from email into QuickBooks” is a process. “We want AI” isn’t.
- Volume: calls, leads, documents or tickets per week, as best you know.
- Systems: CRM, calendar, phone system, ERP, help desk, and whether each one has an API.
- Success: one or two outcomes, like hours saved, faster response times or fewer errors.
- Constraints: sensitive data (health, financial, minors), budget range, deadline, languages.
If you can’t fill in the process line yet, start with our AI readiness checklist. It’s the homework that keeps vendor calls short.
The 12 questions to ask an AI automation agency
The questions fall into five areas: proof, scope, ownership, operations and money. The table shows what a good answer sounds like, and what should send you looking elsewhere.
What good and bad answers sound like
| Question | A good answer | Walk away if you hear |
|---|---|---|
| 1. Can you show me a live system similar to this? | They open something that runs every day and let you try it | Only slides, or a recording you can’t interrupt |
| 2. Who exactly will build and maintain it? | Names, roles, where they’re based, what gets subcontracted | “Our team” and nothing else |
| 3. What would you not automate in our process? | Specific steps that should stay with people, or aren’t worth it at your volume | “We can automate all of it” |
| 4. How will we measure success, and what’s the baseline? | One metric, and how it’s measured today and after launch | “You’ll see huge efficiency gains” |
| 5. What happens when the AI gets something wrong? | Guardrails, human review for risky actions, error alerts, a handoff path | “It doesn’t make mistakes” |
| 6. Which of our systems will you connect, and how? | System by system: API, webhook, a connector in n8n or Make, or an MCP server | “It integrates with everything” |
| 7. Who owns the code, prompts and workflows? | You do, in writing, and everything can be exported | A license you lose when you leave |
| 8. Whose name is on the third-party accounts? | Yours, with access granted to the agency | Everything resold through their accounts |
| 9. How will you handle our data? | Which vendors process it, retention, training opt-outs, which agreements they sign | “We’re fully compliant” |
| 10. How will you test it before launch? | Real examples from your business, written test cases, a limited rollout | “We’ll tweak it after go-live” |
| 11. What does support look like after launch? | Monitoring, response times, how changes are priced, plans for model retirements | Nothing defined past the launch date |
| 12. How is it priced, and what makes the price go up? | The pricing model, the cost drivers, and usage fees listed separately | One number with nothing behind it |
A good answer
- 1. Can you show me a live system similar to this?
- They open something that runs every day and let you try it
- 2. Who exactly will build and maintain it?
- Names, roles, where they’re based, what gets subcontracted
- 3. What would you not automate in our process?
- Specific steps that should stay with people, or aren’t worth it at your volume
- 4. How will we measure success, and what’s the baseline?
- One metric, and how it’s measured today and after launch
- 5. What happens when the AI gets something wrong?
- Guardrails, human review for risky actions, error alerts, a handoff path
- 6. Which of our systems will you connect, and how?
- System by system: API, webhook, a connector in n8n or Make, or an MCP server
- 7. Who owns the code, prompts and workflows?
- You do, in writing, and everything can be exported
- 8. Whose name is on the third-party accounts?
- Yours, with access granted to the agency
- 9. How will you handle our data?
- Which vendors process it, retention, training opt-outs, which agreements they sign
- 10. How will you test it before launch?
- Real examples from your business, written test cases, a limited rollout
- 11. What does support look like after launch?
- Monitoring, response times, how changes are priced, plans for model retirements
- 12. How is it priced, and what makes the price go up?
- The pricing model, the cost drivers, and usage fees listed separately
Walk away if you hear
- 1. Can you show me a live system similar to this?
- Only slides, or a recording you can’t interrupt
- 2. Who exactly will build and maintain it?
- “Our team” and nothing else
- 3. What would you not automate in our process?
- “We can automate all of it”
- 4. How will we measure success, and what’s the baseline?
- “You’ll see huge efficiency gains”
- 5. What happens when the AI gets something wrong?
- “It doesn’t make mistakes”
- 6. Which of our systems will you connect, and how?
- “It integrates with everything”
- 7. Who owns the code, prompts and workflows?
- A license you lose when you leave
- 8. Whose name is on the third-party accounts?
- Everything resold through their accounts
- 9. How will you handle our data?
- “We’re fully compliant”
- 10. How will you test it before launch?
- “We’ll tweak it after go-live”
- 11. What does support look like after launch?
- Nothing defined past the launch date
- 12. How is it priced, and what makes the price go up?
- One number with nothing behind it
Three of these are worth a closer look.
Question 3 tells you the most. An agency that understands your process will name steps that should stay human: complaints, judgment calls, anything regulated, or tasks too rare to be worth building. An agency that says yes to everything hasn’t understood the work.
Question 6 tests real engineering. Ask what they’ll do if one of your systems has no API. Good answers range from a small API in front of the database to a nightly export. Screen scraping should be the last resort, and they should say so. Our guide to integrating AI into existing software covers these options if you want to walk into the call prepared.
Question 12 should come with a cost breakdown. A fixed-scope project, setup plus a monthly fee, or a retainer are all legitimate. What matters is knowing what moves the number: extra integrations, call minutes, document volume, languages, change requests. Our article on what AI automation costs explains each factor.
Red flags that should end the conversation
Some patterns keep showing up in AI agency marketing, and regulators have noticed.
| Red flag | Why it’s a problem | Ask instead |
|---|---|---|
| Guaranteed ROI or “replace your staff” | Results depend on your process, volume and data; nobody knows before discovery | “What would have to be true for this to pay off for us?” |
| Statistics with no source | If there’s no organization and year behind a number, you can’t check it | “Where does that number come from?” |
| Case studies with no details you can verify | Anonymous clients and round numbers are easy to make up | “Can I see the system, or talk to someone who uses it?” |
| “HIPAA compliant” or “TCPA-safe” with no paperwork | Compliance depends on contracts, configuration and how you use the system | “Which agreements will you and your vendors sign?” |
| A price before any discovery | They haven’t understood the process, so the scope will change later | “What do you need to know before you quote?” |
| One tool for every problem | Good teams choose n8n, custom code, a voice platform or nothing, depending on the job | “Why this tool for this process?” |
| No human handoff in the design | Customers will get stuck with an AI that can’t help them | “Show me how a caller reaches a person.” |
| Pressure to sign this week | Good partners don’t need a countdown clock | Take the time you need |
What should the contract include?
Put every proposal into the same structure, and the blanks will tell you as much as the numbers. Before anyone signs, make sure the agreement covers all of this.
Before you sign
Have your attorney review it like any other service agreement. If governance questions come up along the way, our guide on writing an AI usage policy shows what your side of the rules should look like.
How do you run a pilot that actually tests the agency?
If two agencies look equally good, a pilot settles it. Keep it to one process, a few weeks, a fixed price and acceptance criteria written before work starts, like: “the agent books correctly into our calendar in all twenty test scenarios, and transfers every caller who asks for a person.”
You’ll learn three things no sales call will tell you: how the agency communicates when something goes wrong, how it deals with your messy real-world data, and whether the system holds up at your volume. If you’d rather get an outside view before talking to any vendor, an AI readiness assessment gives you a ranked list of processes worth piloting.
How we answer these questions
Appyweb is an AI automation agency headquartered in Austin, Texas, with an engineering team in Spain and more than 20 in-house specialists building software since 2012. When you ask question 1, we open systems that run every day. Scope and price are agreed in a written proposal before work starts, and accounts are set up in your name.
We’d rather you ask us all twelve questions than none. Bring your process and book a free consultation, and we’ll answer every one with your system on the table.