Introduction

We all know Newton’s second law “F=ma”. In words, we only have acceleration when an external force is applied. But could he have some how chosen not acceleration but say something higher order ? 
Acceleration is the rate of change of velocity, which itself is the rate of change of position. Thus acceleration is second order in derivatives with respect to time. This is not just a feature of Newtonian mechanics but is rather general and found throughout the fundamental laws of nature.
There are some rather general results using the Hamiltonian formalism that tells us that theories with equations of motion that are higher order than two are unstable. In particular the energy is not bounded from below and this can lead to problems classically and quantum mechanically. This is the famous Ostrogradski instability of (nondegenerate) Lagrangian theories with higher order derivative terms.
Note that such theories are still of interest as effective theories, but they cannot be seen as fundamental.
I won’t say anything more here about this.
So the question is can we understand simply why Newton could not really have picked anything of higher order in “F=ma”?
Notation: I will use Newton’s dot notation for the first and second order derivative with respect to time. For the nth order derivative (n>2) I will use $latex x^{(n)}(t)$.
Order three
Let us just pick a different form of Newton’s law as
$latex F = N x^{(3)} (t) $,
that is let us suppose the force is proportional to the third order derivative of the position. (That would be the rate of change of acceleration). Here N is some property of the particle analogous to mass.
Now let us think about the motion of a free particle. So set the force term equal to zero and see what happens. Solving our “higher order” Newton’s law with no force is simple. We have
$latex x(t) = c_{3} t^{2} + c_{2}t + c_{1}$.
The constants here are set by our initial conditions;
$latex x(0) = c_{1}$, $latex \dot{x}(0) = c_{2}$ and $latex \ddot{x}(0) = 2 c_{3}$.
So what do we notice? The velocity as a function of time is given by
$latex v(t) := \dot{x}(t) = \ddot{x}(0)t + \dot{x}(0)$.
This means that even if we set the initial velocity to zero the isolated free particle will speed up! Remember this is without any forces acting on the particle.
Newton’s first law says (in part) that “a particle at rest will remain at rest unless acted upon by an external force”. This higher order form of the second law is inconsistent with the first.
Worse than this, there are no forces here and so no potentials. The particle just speeds up all by itself and so clearly we lose conservation of energy. The particle can gain kinetic energy, defined as usual, but at no loss of potential energy! We would have to abandon our usual notion of conservation of energy in simple mechanics!
Higher order again
The same arguments work for higher order terms. The particle will just speed up by itself in violation of the first law and conservation of energy.
Modification of Newton’s law
Let us consider a slightly different situation in which “F = ma” becomes
$latex F = M \ddot{x}(t) + N x^{(3)}(t)$.
That is we will add a higher order term to Newton’s law. Again, let is consider the case with no force term. We want to solve the equations of motion
$latex \ddot{x}(t) + \frac{N}{M} x^{(3)}(t) =0$.
Here we assume that M is not equal to zero and is positive. For now we make no assumption at all about N.
One can directly solve the equations of motion
$latex x(t) = \left( \frac{N}{M}\right)^{2} c_{1} e ^{(M/N) t} + c_{2}t + c_{3}$.
Again we notice that the velocity is not constant and so we do not have conservation of energy in this situation either. But let us have a look at the particles trajectory for different ratios M/N.
In the above we have set M/N = 2 and all the constants to 1. The purple line is what we expect from F=ma. Note that we have a quick decay to the classical case. This itself signifies we do not have conservation of energy as we have no mechanism for the loss in kinetic energy, it just happens!
In the above we have set M/N =1. This case is very similar to the previous case.
Now in the above we have set M/N = 0.5. Again this is very similar to the previous cases.
We have a decay so that after some period of time everything looks the same as the standard Newtonian case. However, we still have to violate the first law to achieve this.
Now what happens if we let N be negative?
Here we see we have a runaway situation in which the particle just keeps on speeding up! Even if initially the trajectory is very close to the standard one after some time it just blowsup. Again this is in violation of the first law and conservation of energy.
Lower order
What about lower order laws?
Well if we had $latex F \propto x$ then when there are no forces we simply have $latex x =0$. Everything not in motion would have to sit at x=0. Meaning we cannot have any extended objects that are not in motion. This cannot be consistent with our Universe.
What about $latex F \propto \dot{x}$? Again let us set the forces to zero and we see that the solution is just $latex x(t) = x_{0}$, some constant. However, this does not sit comfortably with our notion of relativity. Different inertial observers will not agree on the value of $latexx_{0}$. Thus if we don’t want to introduce absolute space we cannot allow this lower order form.
Conclusion
So as Newton wanted his first law to be true, have a good notion of statics and did not want to introduce absolute space he could have only have picked “F = ma”.