My daughter (who uses a wheelchair) and I went by bus tour from Tokushima. Perhaps you could consider going by bus, or even booking a tour? Once you get on the ferry, and then on  the island, it’s quite accessible.

My daughter uses a wheelchair, but she has never been denied entrance to a tourist site or event because of her wheelchair. Most places make accomodations for visitors with disabilities, especially with the approach of the 2020 Olympics and Paralympics in Tokyo. If you have doubts, though, perhaps you could change tour operators, as suggested above.