Turismo, the fasteners that were used to hold the trim piece to the door panel were plastic originally, and it didn't take much to break them. Your's are probaly broken.
However, there is a bright side; the aftermarket sells upgraded trim pieces with metal fasteners.