You can have multiple cross-refs to a cross-ref marker (perhaps from other books), so how would you know where to go in such a situation? 
It would be nice to be able to backtrack, but it could be quite a complex route. You would need to search all possible files and try to match the source text for the x-ref to that of the x-ref marker text. You would need a mechanism that defines the scope (i.e which files to search), build a list of the matches and then present those matches in a usable form to the user.