From Rudy Rucker's Infinity And The Mind
"..generally a Formal Mathematical System is a system of symbols together with rules for employing them. A formal system has four components. 1) A basic alphabet of symbols to be used. Any finite sequence of the fundamental symbols is called a formula. But most of these arbitrary strings of symbols are useless, and one has 2) a criterion for determining which strings of symbols are 'grammatical'. These grammatical strings are called meaningful formulas. To obtain a formal system one takes 3) a particular set of meaningful formulas as the axioms of the system. Lastly, one adopts 4) a few rules of inference that describe the allowable ways of combining axioms to provide proofs of other meaningful formulas"
Language Grammar can be described by a Formal System
See en.wikipedia.org 
See original on c2.com