NYU Langone Health just lifted the curtain on its recent ChatGPT experiment, publishing an impressively candid look at all of the real-world data from its system-wide roll out.
A new article in JAMIA details the first six months of usage and cost metrics for NYU’s HIPAA-compliant version of ChatGPT 3.5 (dubbed GenAI Studio), and the numbers paint a promising picture of AI’s first steps in healthcare. Here’s a snapshot of the results:
Adoption
- 1,007 users were onboarded (2.5% of NYU’s 40k employees)
- GenAI Studio had 60 average weekly users (submitting 671 queries/week)
- 27% of users interacted with GenAI Studio daily (Table: Usage Data)
Use Cases
- Majority of users were from research and clinical departments
- Most common use cases were writing, editing, data analysis, and idea generation
- Examples: creating teaching materials for bedside nurses, drafting email responses, assessing clinical reasoning documentation, and SQL translation
Costs
- 112M tokens were used during the six months of implementation
- Total token cost was $4,200 ($8,400 annualized)
- Divide that cost by the 60 average weekly users, and it’s under $3 per user per week
While initial adoption seems a bit low at 60 weekly users out of the 40k employees that were offered access, the wide range of helpful use cases and relatively low costs make ChatGPT pretty close to a no-brainer for improving productivity.
- User surveys also gave GenAI Studio high marks for ease of use and overall experience, although many users noted difficulties with prompt construction and felt underprepared without more in-depth training.
NYU’s biggest tip for GenAI implementations: continuous engagement and education is key for driving adoption. GenAI Studio saw large spikes in new users and utilization following “prompt-a-thons” where employees could practice and get feedback on prompt construction.
The Takeaway
For healthcare organizations watching from the wings, NYU Langone Health was as transparent as it gets regarding the benefits and challenges of its system-wide roll out, and the case study serves up a practical playbook for similar AI deployments.
