Recent years have seen a push towards deploying fully autonomous robots in large, complex domains such as autonomous driving, space exploration, and service robots. However, legal, ethical, or technical constraints have limited the extent of these systems’ employable autonomy. In order to successfully achieve their intended goals, these systems must utilize assistance from humans to compensate for their limitations. For such systems to be successful over the course of a long-term deployment, they must both be cognizant of their own competence and have the ability to improve this competence over time in a safe way. Motivated by practical concerns faced in industry, this thesis provides a formal model for such a human-agent system to reason about its own competence and aims in future work to provide effective ways of safely improving the competence of the system over the course of its deployment.