Short answer
Yes, her small size makes a difference. A tiny amount of lychee flesh is unlikely to be toxic, but the high sugar content can more easily upset a small kitten's stomach. The main danger is the pit, which is a choking and intestinal blockage risk for any cat. Watch her for vomiting, diarrhea, or loss of appetite over the next day, and contact your vet if you see those signs.
Lychee — Food safety facts
Feed with care · Cats · full breakdown on the linked food page
Flesh only, and only a little — it is very high in sugar. The pit is a choking and obstruction hazard.
The full verdict, portion guidance and serving notes are collected on the Lychee food facts page.