Abstract: Let $0 < \alpha,\beta < n$ and $f,g \in C([0,\infty) \times[0,\infty))$ be two nonnegative functions. We study nonnegative classical solutions of the system \[\begin{cases} (-\Delta)^{\frac{\alpha}{2}} u=f(u,v) &\text{ in } \mathbb{R}^n,\\ (-\Delta)^{\frac{\beta}{2}} v=g(u,v) &\text{ in } \mathbb{R}^n, \end{cases} \] and the corresponding equivalent integral system. We classify all such solutions when $f(s,t)$ is nondecreasing in $s$ and increasing in $t$, $g(s,t)$ is increasing in $s$ and nondecreasing in $t$, and $\frac{f(\mu^{n-\alpha}s,\mu^{n-\beta}t)}{\mu^{n+\alpha}}$, $\frac{g(\mu^{n-\alpha}s,\mu^{n-\beta}t)}{\mu^{n+\beta}}$ are nonincreasing in $\mu > 0$ for all $s,t\ge0$. The main technique we use is the method of moving spheres in integral forms. Since our assumptions are more general than those in the previous literature, some new ideas are introduced to overcome this difficulty.

Key words: Higher fractional order system, integral system, general nonlinearity, method of moving spheres, classification of solutions