The size of the engine doesn't necessarily indicate how much power a bike has as you may be aware from your experience with street bikes. A Honda CRF 300L is much more tame than a KTM 350 EXC-F even though they have similar engine sizes (think about the power difference between a 1000cc crotch rocket and a cruiser which might have a 1200cc engine that makes significantly less power).

The KTM 500 has more power than I can realistically use in the dirt but I like having that much power available for when I on the street. It can easily cruise at 75-80mph and can go triple digits if needed while my wife's Honda CRF 300L tops out around 70-75mph.

The trade offs between the KTM and the Honda are performance, price, and maintenance. The KTM makes about double the horsepower of the Honda and weighs about 70lbs less. The KTM cost about double what the Honda cost. I change my oil every 500ish miles on the KTM while her oil changes are every 8000 miles.

The 300L or an older 250L will be a great starter bike but you might wish it was a little lighter on the trails and that it had a bit more power on the road.

Other street legal options off the top of my head:

Kawasaki KLX 300 Suzuki DRZ-400 Yamaha WR250R

There are certainty other options as well (KLR650, DR650, etc but they're heavy) and there are also ADV bikes which can go off road if you can handle the weight.