Attorney Burt Krages, and author of the Legal Handbook for Photographers, has a PDF on his site outlining the rights of a photographer in the US. Some of these include:
If you ever get harassed by someone when your taking photos in public, you can print out this flyer, and show them your rights as a photographer.
If your in the UK, there is a guide for you here.
I'm not an attorney, so you might want to verify my interpretation with your attorney, or read Burts interpretation of the Law.
XML