I absolutely agree. This is especially noticeable with more technical documents like research papers or textbooks. On the kindle, to flip to a particular page, you have to go into the menu, pop up a symbols list, navigate with arrows to each digit, then click go. And each of these keypresses takes a long time -- probably half a second -- for the screen to refresh. Then, you repeat log(n) times until you've found (via binary search) the section you were looking for. Poorly thought out.
You don't need to use the symbol list to type numbers — alt–Q will give you a 1, etc. That still doesn't make the process you describe good, but if you must do that, this should at least speed it up considerably.