# AI system design, monitoring and safe tool boundaries

> A useful AI application includes data handling, retrieval or model calls, validation, error handling and a user interface. The model is one component.

Course: [Generative AI, RAG and practical evaluation](https://studybrix.com/en/courses/generative-ai-and-rag/index.md)
Level: university
Prerequisites: Software interfaces, evaluation and data privacy.

## Notes

### System boundary

A useful AI application includes data handling, retrieval or model calls, validation, error handling and a user interface. The model is one component.

### Tools

Actions need explicit authorization, narrowly scoped permissions and input validation. Model-generated arguments should not be trusted merely because they are structured.

### Operations

Track latency, cost, failures and task-specific quality without logging unnecessary private data. Version changes and provide rollback.

### Fallback

Define what happens when evidence is missing, a service fails, or output violates constraints. Human review and safe failure can be more valuable than a forced answer.

## Practice questions

### 1. Should model-generated tool arguments bypass server-side validation?

A. No
B. Yes, if they are JSON
C. Yes, if the model is large
D. Only if the tool has a short name

Answer: No

Explanation: Generated arguments remain untrusted inputs requiring authorization and validation.

### 2. Three sequential stages take 100,200 and 50 ms with no overlap. Find total latency.

Answer: 350 ms

Explanation: Add sequential stage durations.

### 3. Which permission design is appropriate for a tool?

A. The minimum scope needed for the authorized task
B. Trust a browser role string
C. Unlimited permanent access by default
D. Write access for every read

Answer: The minimum scope needed for the authorized task

Explanation: Least privilege limits consequences of mistakes.

### 4. Why should sensitive prompt contents not automatically enter analytics logs?

Sample answer / marking guidance: Logs can retain and expose private data beyond the task. Minimize or redact content and define appropriate access and retention.

Explanation: Credit privacy and retention controls.

### 5. A system handles 200 requests and 10 fail. Find observed failure rate.

Answer: 0.05

Explanation: 10/200=0.05.

### 6. Which is a sensible response to an unavailable evidence service?

A. Show a clear failure or limited answer without inventing evidence
B. Expose secrets for debugging to all users
C. Fabricate citations
D. Pretend the service succeeded

Answer: Show a clear failure or limited answer without inventing evidence

Explanation: Failing safely is better than false success.

### 7. Describe a rollback plan for a model or prompt update.

Sample answer / marking guidance: Version the previous configuration, compare release tests and monitored metrics, define rollback triggers, and restore the prior version without losing necessary data integrity.

Explanation: Credit versioning, triggers and restoration.

### 8. Why is syntactically valid JSON not enough to make an AI action safe?

Sample answer / marking guidance: Its contents can still be unauthorized, semantically wrong, outside limits or targeted at the wrong resource. Enforce meaning and permissions server-side.

Explanation: Credit syntax versus authorization and semantic validation.

## Further reading

- https://developers.google.com/machine-learning/crash-course
- https://cs229.stanford.edu/

Original AI-assisted educational draft; not independently teacher-reviewed. References support further study, not copied exercises, affiliation or curriculum accreditation.