I optimized the way that I pass the raw JPEG to the OpenMAX API to decode and I've got some modest gains already:
Decoding from memory buffer and rendering on screen:
66 FPS (up from 62 FPS)
Only decoding JPEG:
107 FPS (up from 103 FPS)
Yes, a 4 field per second boost! Every little bit helps!