At the end of the day, the big problem with utilitarianism is that it doesn’t respect the value of individual human life. This comes out in a variety of ways, but this is the big and insurmountable problem with the theory.
The willingness to use others was the basis for slavery. If an individual human being has no inherent worth, then they can be owned, traded etc. They can also be killed if, in the twisted view of a dictator, they are a threat to the well being of a nation.
In a more subtle way, this problem comes out when it becomes clear that people inherently value things other than happiness. For example, integrity, justice, honor, fidelity etc… are not always things that lead directly or indirectly to happiness for oneself or others.
The classic Jim example is one (dude in the jungle, has a choice he must kill one person or see many murdered) — in that example, really — it doesn’t seem to be clear what a non-utilitarian should do. Pojman clearly thinks the person should permit themselves and others to die to avoid killing himself, but the thing is — in a forced choice like that — the moral culpability doesn’t lie with Jim in any case. It lies with the rebels who are forcing the choice.
A more subtle example from the same article (someone whose name I don’t remember — call him Steve), is a chemist. Steve is morally opposed to biochemical weapons, but is out of work and is offered a job making weapons. If he takes it, his family gets fed etc.. if he declines, someone more enthusiastic than he will take the job and develop more chemical weapons. He calculates that the best alternative is to take the job, sacrificing his deeply held principles, but to be as ineffective at that job as possible. This is what a good utilitarian would do.
This is still not an entirely clear example of sacrifice of ones own good for the good of others, but it’s better.