I would approach this challenge in this way:
1. Define the responsibilities of concierge (obviously in different cases the list will be different).
2. Analyze past concierge work. What where the challenges, where were the bottlenecks, when the concierge did good and what can be improved.
3. Using (2) define the success factors of the concierge service. I guess there will be timely response to the customers; accuracy of the service provided.
4. Define the metrics for the success factors. For example, to measure accuracy you can track returning query rate. Measuring overall customer satisfaction for the specific (!) job done will also be a good idea.
Feel free to share your experience here once implemented.