The closest I've been to a forum without rules were Usenet News Groups. The majority of which have no moderators, and yes there were trolls, but curiously, at least in the groups I was in, the members had a common agreement on what was acceptable behaviour. Trolls, sock puppets and the like were quickly called out and the error of their ways explained to them, if they persisted everyone just ignored them and eventually they got bored and went away.