Need to remove rear hub...

[tmontague]

you should be able to only turn it up but not down iirc. Keep turning it up and it should tighten. I honestly don't know why the Yaris drums are so bad at self adjusting, probably one of the reasons why most manufacturers went with discs in the rear.

I don't really find they even loosen, more that they don't adjust/tighten with reduction in shoe material. Although after a track day the e brake goes up a couple clicks which is likely due to the rear shoes wearing down a bit.

If yours is actually getting loose then chances are the adjuster mechanism is installed incorrectly. On my sister in laws '07 yaris the e brake had to be constantly pulled really high to engage and still barely held the vehicle. Turned out one of her adjusters had a small piece installed the wrong way. Couldn't tell just by looking at it, only noticed it until i went to adjust it and it never tightened. Personally I find that my e brake holds my car really well, and that is really how it should work