How to Read Phase Reports: Understanding AI Decision-Making

Introduction

When you run a Dev Run, each of the 10 phases generates a detailed report. Yet many users only check the final result (PASS / FAIL) and skip the intermediate reports entirely.

Phase reports document how the AI reasoned, what decisions it made, and what risks it identified. Learning to read these reports effectively provides several benefits:

• Improved efficiency and accuracy during PR review
• Earlier detection of issues, enabling faster rollback decisions
• Insights for writing better Issues next time
• A deeper understanding of how your AI team operates

This article explains what to look for in each phase's report, with concrete examples of good versus concerning report content.

How to Access Phase Reports

On the job progress screen, click on any phase to view its report. Reports are written in Markdown format, presenting structured content in a readable layout.

Each phase's report is generated by the responsible AI persona (Aoi, Riku, Sumire, or Kohaku). Because each persona has a different perspective, report content varies by phase.

Planning Phase Reports

Phase 1: Planning

Owner: Aoi (PM)

The planning report is the most critical report in the entire Dev Run. It sets the direction for everything that follows. A wrong turn here affects all subsequent phases.

What to check:

Item	What to Look For
Task Understanding	Has the Issue's intent been correctly interpreted?
Implementation Strategy	Is the proposed approach appropriate for the project?
Risk Analysis	Have potential risks been adequately identified?
Scope	Are there changes beyond the Issue's scope?

Example of a good report:

## Task Analysis
This Issue requests adding pagination to the user list page.
The current user list fetches all records, causing performance
degradation as data volume increases.

## Implementation Strategy
- Adopt cursor-based pagination (aligns with existing API patterns)
- Reuse the existing Pagination component on the frontend
- Default to 20 items per page, configurable via settings

## Risks
- The existing API endpoint response format will change; backward compatibility requires attention
- Performance testing needed for the combination with sort functionality

Example of a concerning report:

## Task Analysis
Improve the user list.

## Implementation Strategy
- Add pagination
- Change the API

## Risks
- None identified

The key characteristic of a concerning report is lack of specificity. This typically occurs when the Issue description itself is vague. If you see a report like this, consider enriching the Issue content and re-running.

Phase 2: Requirements

Owner: Aoi (PM)

The requirements report lists the requirements extracted from the Issue, along with any implicit requirements the AI has supplemented.

What to check:

• Do the functional requirements accurately reflect the Issue's intent?
• Are non-functional requirements (performance, security) considered?
• Are acceptance criteria clear and verifiable?
• Are there excessive requirements that go beyond scope (scope creep)?

Pay special attention when the AI has supplemented requirements that were not in the original Issue. Verify whether the AI's judgment is appropriate in the context of your project.

Phase 3: Design

Owner: Riku (Tech Lead)

The design report is where technical decisions are concentrated. This is one of the reports that deserves the most review time.

What to check:

Item	What to Look For
Architecture	Does it align with the existing architecture?
Target Files	Are there unexpected files included?
Impact Analysis	Is the scope of impact properly analyzed?
Data Structures	Are the proposed data structures appropriate?

If you find issues in the design report, rolling back before implementation minimizes rework.

Phase 4: Test Scenario

Owner: Sumire (QA Engineer)

The test scenario report outlines the test cases planned to verify the implementation.

What to check:

• Are both normal and error paths covered?
• Are boundary value tests included?
• Is the granularity of test cases appropriate (not too coarse, not too fine)?
• Are critical business logic paths being tested?

If the test scenarios are insufficient, the test code generated in the test implementation phase will also be insufficient.

Implementation Phase Reports

Phase 5: Implementation

Owner: Riku (Tech Lead)

The implementation report provides an overview of the actual code changes made.

What to check:

• Does the implementation follow the design report's strategy?
• Does the list of changed files match the design report?
• Are newly created files placed in appropriate directories?
• If dependencies were added, is the rationale explained?

Reviewing the changed file list in the implementation report gives you a roadmap for the subsequent PR review.

Phase 6: Test Implementation

Owner: Sumire (QA Engineer)

The test implementation report describes the test code created based on the test scenarios.

What to check:

• Have all cases from the test scenario phase been implemented?
• Does the test code follow the testing framework's conventions?
• Are mocks and stubs used appropriately?

Phase 7: Testing

Owner: Sumire (QA Engineer)

The testing report provides objective metrics on implementation quality.

What to check:

Item	What to Look For
Test Results	Did all tests pass?
Failed Tests	Is the root cause of failures analyzed?
Coverage	Is test coverage sufficient?
Execution Time	Is test execution time reasonable?

Example of a good report:

## Test Results
- Total: 12 tests
- Passed: 12 tests
- Failed: 0 tests
- Coverage: 87%

## Test Details
- Basic pagination behavior: PASS
- Empty dataset handling: PASS
- Boundary values (0 items, 1 item, max): PASS
- Invalid page parameter: PASS

Example of a concerning report:

## Test Results
- Total: 3 tests
- Passed: 2 tests
- Failed: 1 test

## Failure Analysis
One test failed due to an environment issue, not a substantive problem.

When tests fail, do not take "environment issue" explanations at face value. Always verify the actual error details.

Phase 8: Documentation

Owner: Kohaku (Tech Writer)

The documentation report covers the documentation updates made alongside code changes.

What to check:

• Has documentation been updated to match the code changes?
• If there are API changes, has the API documentation been updated?
• Does the documentation content match the actual implementation?

Result Phase Reports

Phase 9: Report

Owner: Riku (Tech Lead)

The report phase produces the content that becomes the basis for the PR description. It serves as an overall summary.

What to check:

• Is the change summary accurately described?
• Is the list of changed files correct?
• Are remaining issues clearly documented?
• Does it contain the information reviewers need?

Phase 10: Evaluation

Owner: Aoi (PM)

The evaluation report delivers the final quality assessment for the entire Dev Run.

How to read the evaluation result:

Verdict	Meaning	Next Action
PASS	All requirements met	Proceed to Finalize after review
PASS WITH ISSUES	Requirements met but minor issues exist	Review the issues and decide if they are acceptable
FAIL	Requirements not met	Revise the Issue or rollback

For PASS WITH ISSUES, review the documented issues and decide whether to "accept for now and address in a follow-up Issue" or "rollback and fix."

Using Reports to Improve Issues

Phase reports also serve as feedback for writing better Issues in the future.

Issue Improvements Suggested by Reports

Report Observation	Likely Issue Problem	How to Improve
Vague planning report	Issue description is too abstract	Write specific requirements and expected behavior
Excessive requirement supplementation	Issue lacks information	State preconditions and constraints explicitly
Unexpected files in design	Scope is unclear	Define the scope of changes explicitly
Too few test cases	Test requirements are unspecified	Include testing perspectives and quality criteria

For more on writing effective Issues, see GitHub Issue Writing Guide.

Using Reports for PR Review

Reading phase reports before reviewing the PR significantly improves both review quality and speed.

Efficient Review Workflow

1. Check the Evaluation report (Phase 10) - Understand the overall verdict
2. Check the Design report (Phase 3) - Understand architectural decisions
3. Check the Testing report (Phase 7) - Review test results and coverage
4. Check the Report (Phase 9) - Review the change summary and remaining issues
5. Review the PR diff - Examine the code with context from the reports

By following this order, you understand "what changed, why, and how" before reading the code diff, giving your review a clear focus.

For more PR review tips, see Tips for Reviewing AI-Generated PRs.

Frequently Asked Questions

Q: Should I read every phase report every time?

You don't need to read all of them in detail. At minimum, review Planning (Phase 1), Design (Phase 3), Testing (Phase 7), and Evaluation (Phase 10). These four phases have the most direct impact on Dev Run quality.

Q: What should I do if I find a problem in a report?

If the Dev Run is still running, consider aborting the job. If it has already completed, rollback to the problematic phase and re-run, or improve the Issue and start a new Dev Run.

Q: How much should I tolerate with PASS WITH ISSUES?

If the issues are minor, such as "slight coding style inconsistencies," it is efficient to accept them and address them in a separate Issue. However, if the report flags "security concerns" or "performance impact," rolling back and fixing the issues is recommended.

Summary

• Each phase report documents the AI's decision-making process and deliverables
• Always review the Planning, Design, Testing, and Evaluation reports
• Report specificity and thoroughness reflect the quality of your input (the Issue)
• Reading reports before PR review improves both efficiency and accuracy
• Catching concerning reports early enables rollback before rework compounds

Phase reports are your window into what would otherwise be a black box. Building the habit of reading them lets you extract maximum value from Dev Run outputs and collaborate more effectively with your AI team.