Short answer
Yes, you should be concerned. The peel of a kumquat contains oils that are irritating and can cause vomiting, drooling, or an upset stomach. A whole fruit is too much for a small puppy, whose developing system is more sensitive. Remove any remaining fruit and peel from your puppy's reach. Watch for signs like excessive drooling, retching, or diarrhea over the next few hours, and call your own vet if any of these develop.
Kumquat — Food safety facts
Feed with care · Dogs · full breakdown on the linked food page
Citrus peel oils can cause vomiting and drooling. At most a tiny piece of flesh.
The full verdict, portion guidance and serving notes are collected on the Kumquat food facts page.