Nobody in Malaysia drinks tap water without a filter. The hotels I stayed in always had free water dispensers and most food stalls were kind to provide water, so I only had to buy water a few times of which almost all were from very cheap water dispensers in the street – 1 litre ~ 0.03 USD. The challenge was finding them and hoping they worked...