Accessibility is an ongoing process, but treated as a checkbox exercise almost every time… So over the past couple of weeks, I've been running what I've deemed “A-status April”.
Some backstory
In the Netherlands, public sector body websites need to comply with EN 301 549. That means they need to meet it's WCAG 2.x guidelines at level AA and publish an accessibility statement.
To track this progress, they are urged to publish this into a central register as well, located at https://www.toegankelijkheidsverklaring.nl/. Having a central hub like this helps with transparency, and it also supports the monitoring system that is (being put) in place to ensure organisations actually comply.
To indicate the progress of compliance for each website, we have a status system. Websites in the register get a certain status (from E to A) assigned, based on the steps they have undertaking thus far and the results that come out of that. Allow me to explain the different statuses briefly.
Accessibility statuses
There are 5 statuses that a website could be at, ranging from E (the lowest) to A (the highest)
- Status E; Your website doesn't have an accessibility statement, no audit report, and you have no audit planned (at least not publicly).
- Status D; you have a statement (it's in the register) but you don't have an audit report and no audit planned.
- Status C; You have a statement, no audit report yet, but you have one scheduled within the next 6 months.
- Status B; You have a statement and an audit report that meets the criteria conform WCAG-EM, but there are still findings you need to resolve.
- Status A; The “Holy Grail” for every Public Sector Body website, you have a statement and an audit result that says your website has no, zero, none, nada accessibility issues.
I have some strong feelings about the fragile nature of this whole system, but that's something for another time.
As some people find it easier to cross-reference the above in table format, I've included that below as well:
| Status | A | B | C | D | E |
|---|---|---|---|---|---|
| Statement | Yes | Yes | Yes | Yes | No |
| Valid audit report | Yes | Yes | No | No | No |
| Findings | No | Yes | N/A | N/A | N/A |
| Audit within 6 months | N/A | N/A | Yes | No | No |
| Compliant | Yes | Yes, if measures are in place | Yes, for 6 months | No | No |
A-status April
Now, back to “A-Status April”!
Currently, the central register I mentioned before holds over 4300 statements across the D-A statuses (Status E being the unknown websites). Within that set, there are currently (April 2023) 377 statements that claim a Status A. This means that they believe their website (or app) to be fully WCAG 2.x conformant. (Purposefully not saying fully accessible here)
The idea for “A-status April” was to spot check the validity of those claims.
Why do this?
Over the past years, since the start of the register, I've had my doubts around the current rating system. Does it help raise awareness around accessibility? Yes! Does it force organisations to work towards compliance? Yes!
The register helps in this because politicians care a lot about their reputation and performing better than others. So, seeing their neighbouring municipality having higher ratings than themselves drives them forward.
But the system chases people to a checkbox, and not structural change!
Audits for statements can be up to 36 months old. In web terms, that as a LONG time for a report to be valid. For accessibility, where you're dealing with actively managed content, that is like centuries!
I strongly feel that an accessibility audit is but a moment in time. On that day, that specific day, your website did not produce any findings against the WCAG success criteria. That all could change a day later.
What did I do?
Over the past months,I've done checks on any website that posted they claimed a Status A. nearly every single time I could spot accessibility issues.
So, to further test this, a scheduled some time and selected a sample from the register. It had only a couple of criteria:
- Status A
- 1 site per organisation
No reports of my own (I've never had a website reach Status A), No looking at other information in the report (auditor, website supplier), No pre-checks, nothing.
I selected 20 statements for this round.
I know that is less than 5% of the Status A group and less than 0.5% of the total! I'm only human, I'm afraid.
Then I planned 1 hour for each one, so I could do a quick manual audit. I skipped using any automated tools, mostly because I wanted to avoid wasting time validating their output, but did use bookmarklets to speed some things up.
20 statements, 1 hour each, 1 month to complete.
The results
The results were unfortunately not very surprising.
Within the 1-hour mark, I found accessibility issues on 11 out of those 20 websites. That is 55% of the sample!
Some highlights:
Content is fragile
A large selection of findings sat on a content level. Situations like illogic heading structures, where text sits under the wrong non-relevant heading or wrong or missing text alternatives for images. Information in improperly structured tables with no normal text alternatives for the information. Salient detail, every single PDF I looked at had accessibility issues.
Technical findings
It's not really possible for me to determine whether the components I encountered are still the same as the ones initially audited. As I mentioned before, websites change more while being live than they used to in the past. New features get added iteratively. All that comes with benefits, but also the risk of new accessibility issues if the supplier doesn't know how to validate before pushing an update out.
There were quite some technical findings, including structures that look like content (i.e. read more links) but a content author likely has no control over.
Audit quality
Doing accessibility audits well is hard and requires specialised in-depth knowledge of the accessibility guidelines. It's very likely that some issues that I came across were incorrectly marked as a pass.
There was a correlation between sites on which I found issues and suppliers of the respective. A couple of audit suppliers stood out, where others luckily never made the list.
In a space that rapidly saw an increase in demand, we saw a big increase in organisations claiming to provide accessibility audits. It's no surprise really that people jumped on it and saw business potential. It shows though that people with more experience generally deliver better results.
There is definitively a need for more high-quality auditors, but with even the basic education on accessibility still lacking real traction, getting people to learn and understand the auditing process well is a major challenge.
The rating system
It looks like we're stuck in a process where organisations are almost forced to get audits done. They must get that checkbox checked, getting an A-status on their reports.
There appears to be little to no maturing happening on the back of those audits in a lot of these organisations, causing them to let an accessible website slip back into being inaccessible. They've had their checkbox checked and next steps, we'll see about that in 36 months when we're due for another round.
The current system only measures on website results and if the organisations don't mature, we will be stuck in this look for quite a while still…
The aftermath
I don't do naming and shaming very often. I find it's counterproductive and sets you up for friction. I want to support your efforts, not burn you down.
That's why I've reached out to all organisations (website owners) privately, as I do whenever I come across situations like this. They are responsible for making sure their websites are in order.
The response, however, is practically non-existent. My messages frequently go ignored and changes are not made. When I do get a response, it's typically in the style of “They've tested it, and it was said to be accessible. We'll take it back to our supplier” and then silence afterwards.
I wish and hope auditors get better because currently bad work is doing a disservice to the accessibility space. So if your client reaches out to say “Hey, we had a comment on our accessibility” educate yourself and try to help them out.
What's next
When structural change isn't made on both ends, all efforts are in vain and people relying on web accessibility remain to be shut out.
So instead, be the change!
Make sure accessibility stops being that checkbox exercise and start engraining it into your day-to-day efforts. Ensure people get trained across the board, from content creation to design and development. Build your INTERNAL knowledge of accessibility and stop relying solely on external sources.
If you're a budget holder of sorts, make sure people have the budgets to arrange these things and let them do it properly because it WILL save you money over time! And it will also allow people like me to move on and help another organisation like yourself in becoming more accessible!