We propose a numerical method to calculate unsteady flows of Bingham fluids without any regularization of the constitutive law. The strategy is based on the combination of the characteristic/Galerkin method to cope with convection and of the Fortin-Glowinsky decomposition/coordination method to deal with the non-differentiable and non-linear terms that derive from the constitutive law. For the spatial discretization, we use low order finite elements, with, in particular, linear discretization for the velocity and the pressure, stabilized by a Brezzi-Pitkäranta perturbation term. We illustrate this numerical strategy through two well-known problems, namely the hydrodynamic benchmark of the lid driven cavity and the natural convection benchmark of the differentially heated cavity. For both, we assess our numerical scheme against previous publications, for Newtonian flow or in the creeping flow regime, and propose novel results in the case of Bingham fluid non-creeping flows.