Moisturise. Either apply water to the area and wash it fresh after shaving, or if you have any sort of moisturising cream, even better. You can often buy after-shaving moisturisers that work eveeeen betttter.
For now, you're going to have to just wet it from time to time (where possible, I appreciate it may be difficult). The pain is your hair growing back, and unfortunately will persist for a while whilst it does grow back. Simply moisturise where possible, and it should go with time. Just moisturise in future after shaving