That first batch of tiny ones was the nuts the tree did not have the resources to fully developed, so it dropped them to save resources for the ones it could fully develop.  Very common for nut-bearing trees.