Until 23/05/2008 it was possible to be married without being registered (you needed at least two people who confirm the marriage/cosigning the contract). Now its mandatory to be registered and the marriage is valid from the day it was registered.
Before that, most "embassies" (such as the German Institute in Taipei) explicitly recommended not to make this kind of private marriages.
See §982. Taiwan Civil Code valid since 23/05/2008:
http://law.moj.gov.tw/Eng/LawClass/LawP ... 0%20&LCC=3A marriage shall be effected in writing, which requires the signatures of at least two witnesses, and by the registration at the Household Administration Bureau.This Article was less clear before that date, here is the old version:
A marriage must be celebrated by open ceremony and in the presence of two or more witnesses. Persons who have registered for marriage in accordance with the Household Registration Act shall be presumed as married.Still in doubt? See Article 988:
A marriage is void if any one of the following conditions are met:
(1) Where it does not conform to the formalities provided by Article 982,
Hope its clear now.

Whatever else you do (have a private party and exchange cracker box rings or something) is up to you but it ain't marriage by law.