Hello forum,
I'm leasing a 2012 manual 5-door SE.
There are a few more significant differences.
1) The steering ratio is actually
slower on the SE. Despite the much shorter wheelbase, the Yaris SE also has the turning circle of a Camry. The Yaris L and LE models have a turning circle of 30.8 feet, while the upgraded "Sport Calibrated Electric Power Steering" of the Yaris SE has a turning circle of 36.7. Driving both cars back-to-back or parallel parking, if you pay attention, you'd notice the difference. As such, the SE requires noticeably more steering angle to turn the car, and the SE's steering receives less power-assist (i.e. it's heavier) than that of the other models.
2) The SE has 10% stiffer front springs, which simply lifts the front of the car higher in the air, so the "sport" suspension actually contributes to the excessive body roll.

Still no rear anti-roll bar.
3) Tires are "upgraded" to 195-50/16 Turanzas, which are the worst tires I've ever driven on. Pulling away from stoplights requires a feathered throttle and lots of clutch slipping to avoid the tires' chirping and making the Honda next you think that you were trying to race. Apparently the Yaris' massive torque at 1000-1500 rpm is much too extreme for the stock tires to handle without chirping. That gets old really quickly. The 16" wheels do look good, with a nice sparkly gunmetal finish, and they're reasonably lightweight.
4) You do get disk brakes in the rear, but there's so little weight back there, the main benefit is cosmetic.
5) Removal of the rear seat is as easy as lifting the seat upward and unsnapping it from its two re-snappable hinges. There's no rear seat scaffold support frame in the 2012 model.
6) The SE exterior gives you cosmetic "improvements," such as a small nearly functionless rear trunk spoiler coupled with an equally functionless lower diffuser that serves a counter-productive purpose as an air dam. The SE gives you do-nothing fog lights, the waffle-patterned grill, chrome trim pieces, and vertical rear reflectors. The SE interior is also trim different on the SE. You get a tachometer, instead of a large empty circle with the word "Yaris" on it, there in case you forget what car you're driving. The whole instrument cluster is lit in red, which does look good.
Yes, with a manual, you can downshift to
try to get more power out of it, but the ecu will always give you soft throttle response. I would not call the car responsive.
Hope that helps!