The Man was not deliberately cruel to the children; he just didn’t like them very much.  At first they were too small to be much help, getting in his way when he worked.  He would hit at them to drive them away.  Later, when he wanted Josh and Jere to help him stack hay in the barn or shuck corn to fill the corncrib, they did not want to come near him, so he hit at them to make them come. (page 4)