fat?

When someone tells you, you need to work on your fitness, are they calling you fat or slovenly?

My tennis has come leaps and bound. My 'fitness' is TBD.