Question is do you really want to break an accepted contract and was any harm done?
Do you want the contract broken? There is too little information to give you a clear answer, and technically this is a legal question. How did the contractor gain access if the buyers agent wasn't present?
If you are looking for a way out, document the incident and take your contract to an attorney for an opinion. If you are upset but want to proceed, tell the buyers agent to comply with the terms of the deal or you are tempted to void it. Good luck in your sale.