Towards Automated Accessibility Report Generation for Mobile Apps

Many apps have basic accessibility issues, like missing labels or low contrast. Automated tools can help app developers catch basic issues, but can be laborious to run or require writing dedicated tests. In this work, we developed a system to generate accessibility reports from mobile apps through a collaborative process with accessibility stakeholders at Apple. Our method combines varied data collection methods (e.g., app crawling, manual recording) with an existing accessibility scanner. Many such scanners are based on single-screen scanning, and a key problem in whole app accessibility reporting is to effectively de-duplicate and summarize issues collected across an app. To this end, we developed a screen grouping model with 96.9% accuracy (88.8% F1-score) and UI element matching heuristics with 97% accuracy (98.2% F1-score). We combine these technologies in a system to report and summarize unique issues across an app, and evaluated our system with 18 accessibility-focused engineers and testers. The system can enhance their existing accessibility testing toolkit and address key limitations in current accessibility scanning tools.
Figure 1: The prototype interactive HTML report generated by our report generation system. The interface has three main areas: a) A carousel displaying all screens captured in the report, b) A menu to toggle between the ‘Summary’ view which is currently selected, and screen-by-screen views represented by thumbnail screenshots, and c) a table of summarized results of issues found, grouped by category. The report provides actions users can take including ignoring, viewing suggestions, filing bugs, and expanding the screenshot.
Figure 2: Our approach to accessibility report generation collects screenshots and accessibility audits through an app crawler or a manual recording tool. From the collected data, a report generator uses a screen grouping model and UI element de-duplication heuristics to generate an interactive report.
Figure 3: Xcode’s Accessibility Inspector in single screen mode (SS) with the scanner’s results for an iPhone. The Inspector has four main areas: (A) a table of detected issues for the current screen that a user requests by clicking the “Run Audit” button, (B) Fix suggestions that appear when a user clicks on the “?” icon on a selected row, (C) A preview window with a screenshot of the impacted UI element, and (D) a device (either on a simulator or live), or an application window. Our study evaluates two modes: SS, which provides features A-D, and MS, which provides features A-E. (E) adds a history of scanned screens and headers to summarize result counts by category.