Before buying an AI tool, ask which workflow it improves, how performance is measured on your cases, what data it receives, what actions it can take, how people intervene, what it costs to operate, and how you retrieve your data or leave.
AI demonstrations are designed to show the strongest path through a product.
Buying decisions need to examine the ordinary and difficult paths too.
Use these questions before a trial becomes a contract.
1. What exact workflow does this improve?
Ask the vendor to describe:
- The trigger.
- Inputs.
- Decisions.
- Output.
- Reviewer.
- Systems involved.
- Exceptions.
If the answer stays at “improve productivity” or “transform customer experience,” the use case is not defined.
2. When is AI better than a simpler solution?
Could the problem be solved by:
- A template.
- Better search.
- A form.
- A rule-based automation.
- A clearer procedure.
- Training staff on an existing system?
The vendor should be able to explain why AI is necessary.
3. How was performance tested?
Ask for:
- The task and dataset used.
- Relevant customer or industry examples.
- Error categories.
- Failure and escalation rates.
- How changes are tested.
- Whether results include human correction time.
Then run your own representative test set.
4. What happens when the AI is wrong?
Look for:
- Evidence and citations.
- Confidence expressed as observable limitations, not decoration.
- Required approvals.
- Human handoff.
- Retry limits.
- Logging.
- Rollback.
- Incident response.
“The model is very accurate” is not a control.
5. What data enters, leaves, and remains?
Map:
- Prompt and uploaded data.
- Connected-system data.
- Generated output.
- Logs and analytics.
- Support access.
- Subprocessors.
- Processing locations.
- Retention and deletion.
- Model-training or service-improvement use.
The OAIC’s guidance for selecting commercial AI products provides a useful Australian due-diligence framework for privacy, data flow, security, human oversight, and fitness for purpose.
6. How are identity and permissions controlled?
Ask:
- Does it support your identity provider and multi-factor authentication?
- Can access be limited by role, team, data source, and action?
- Can an AI agent act as a named service identity?
- Are administrative and user actions logged?
- Can departing staff be removed centrally?
- Can high-impact actions require approval?
Least privilege matters more as AI gains tools.
7. How does it resist malicious or untrusted input?
If the product reads email, websites, files, tickets, or customer messages, ask how it handles prompt injection and hostile content.
The Australian Cyber Security Centre’s secure AI development guidance treats security as a lifecycle requirement, not a feature to add after deployment.
8. What can administrators monitor?
Useful controls include:
- Usage and cost by team.
- Connected tools.
- Sensitive-data alerts.
- Agent actions.
- Approval and override history.
- Failure and escalation patterns.
- Model or feature changes.
- Exportable audit logs.
Ask who in your business will actually review them.
9. What is the full cost?
Include:
- Licences.
- Usage or token charges.
- Setup and integration.
- Data preparation.
- Security and legal review.
- Staff training.
- Human review time.
- Maintenance and retesting.
- Support tier.
- Exit or migration work.
Compare cost against the measured workflow outcome, not the number of AI features.
10. What changes without our approval?
Ask how the vendor communicates:
- Model changes.
- Feature changes.
- Data-policy changes.
- Subprocessor changes.
- Deprecations.
- Price changes.
- Material performance or safety issues.
Define what requires retesting.
11. How do we leave?
Confirm:
- Data and configuration export.
- Deletion process and evidence.
- Contract notice.
- Integration shutdown.
- Replacement or manual fallback.
- Ownership of prompts, workflow logic, and generated assets.
Reversibility protects the business from both failure and success at scale.
Use a scored trial
Give shortlisted products the same examples and score them using the AI output scorecard. Include privacy, security, administration, cost, user experience, and exit alongside output quality.
An AI readiness assessment can determine whether a tool purchase is actually the next step. If it is, Rising Tide’s AI consulting can help translate the vendor’s claims into a workflow-specific trial.