Designing for accessibility is only a positive. It produces better overall products in games and out of games. Even with the help I can provide to those with disabilities that need the extra features, it also improves the game for everyone else. It demonstrates that the developer has thought more outside the box and thus will likely create a better product. But it gives players more options with how they want to play the game. For example, designing for a deaf person would also allow a player to experience the game while listening to music or without sound at all. While sound is an important aspect of a game once a game is published the player bought/ leases the product they should be able to experience how they want. Developers may have an intended way they would want it to be done but at is really up to the player how it should be done. This thinking has led to some what is now expected features such as the ability to remap controls, change the field of view, and disabling motion blur. These were once considered accessibility settings but now common place and have only improved.
It is also a futile effort preserve the integrity of the game. As if a player with enough technical know-how wants to change something part of the game it is going to happen. This can be seen with the decomplication project of Mario 64. Features are being added to the game that fundamentally alter it that as a byproduct that adds more accessibility features. Notably making it more accessible such as putting it online. There will be more due to a recent leak of Nintendo but the source code was not obtained legally.
The only caveat in which care should be taken for the integrity of the game is online games. Such a game with a minimal hud would make it harder for some players to play. However, it would not be too much in development time to make a server in which those settings are allowed such as ones that allow modifications.
Designing for accessibility helps the game overall as it allows for players to play how they want to or encouraging modding in a game.    

Source: https://learn-us-east-1-prod-fleet01-xythos.s3.us-east-1.amazonaws.com/5e00ea752296c/7614786?response-content-disposition=inline%3B%20filename%2A%3DUTF-8%27%27Accessibility%2520and%2520Gaming%2520Action%2520Card.pdf&response-content-type=application%2Fpdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200508T194044Z&X-Amz-SignedHeaders=host&X-Amz-Expires=21600&X-Amz-Credential=AKIAZH6WM4PLTYPZRQMY%2F20200508%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=3ab5c0746a2607d4f69a512ca8707c6bfe63df8f10734f01498f060d742b1b1e