I had a similar experience when I was doing some Galois Field arithmetic in Java. You pay a huge penalty because of the absence of unsigned types. In our case we had to use long instead of int, which is extra costly, since many basic operations in Java return int by default.
I was doing it in GF(2^32-5). Your statement is true for GF(2^n) where n is small enough to keep the entire multiplication-table in memory (usually n <= 8). When it's bigger you keep log-tables in memory then sign matters. However when n=16 you get lucky and can use char as an unsigned 16 bit int.
