Last updated:
Enhance developer experience by community
User interviews kept telling us the wrong priorities. Developers said “this could be nice” while their real struggles stayed invisible. This talk covers what pair programming with community developers revealed that interviews never did, why developer experience is not user experience, and a concrete toolkit for community-driven developer research.
Talk by Anil Kumar Krishnashetty at the Voxgig DevRel Meetup, virtual, September 2023.
Key takeaways
- Don’t listen to developers’ words, observe their actions. User interviews for Contentful’s Forma 36 design system produced nice-to-haves; pair programming sessions exposed the real workflow, including a developer who never opened the docs because TypeScript types were his documentation.
- UX is not DX. Consumers get one checkout flow; developers upload a file to GitHub four different ways (web UI, terminal, VS Code, GitHub Desktop) depending on skill and preference. Makers need different research techniques than users.
- Developer efficiency is feedback-loop length. Seconds between change and result is high efficiency; minutes of waiting is a DX problem worth fixing.
- Give value before you ask for it. Cold outreach to community developers got near-zero response; after months of answering their questions in the community Slack, a request for a 30-minute pair programming session got a yes.
- A feedback form, a friction log, and your own solution engineers are the cheapest DX research tools you’re not using.
Transcript
Transcript lightly edited for clarity. Host introduction abridged.
Experience is what you notice, and what you don’t
Thank you. Let me start with a quick example about experience. A few years ago in Berlin I was caught by a ticket inspector because I was traveling with the wrong ticket. Take a moment: what does “Radius fare” mean to you? I had traveled with this ticket a couple of times and it was fine. I assumed it was for short distances. It turns out it’s for kids under 14, and there was no information at the ticket machine. That is a bad experience.
Another Berlin example, from the COVID era: signs explaining what 1.5 meters of social distance looks like. Hard to measure, so the signs said: it’s the size of a horse, or about three dogs. A really nice way to convey information without complication.
And great experience often goes unnoticed, like the support I got from the hosts of this meetup: hosting, checking in, making everything smooth for a speaker.
I’ve recently worked at Lokalise, Contentful, and commercetools. I’m going to share my learnings from working as a technical product manager and technical product marketing manager with developer communities, and how we enhanced the developer experience of our products.
What user interviews missed
Coming from a design background, I ran meetups, taught prototyping, and ran user testing sessions in Berlin. As a technical product manager I started applying those techniques to developer tools.
At Contentful I worked on the open source design system, Forma 36. Ahead of a major release we did classic user research: interviews asking developers what they required, what was missing, what we should focus on. We prioritized what they said.
I realized later that they were not telling me what they were really struggling with. They were telling me “this could be nice.”
Around that time I started answering questions in Contentful’s community Slack, for developers building apps on top of Contentful. Slowly I built trust with those developers, and I started to see that what we were prioritizing for the launch was not aligned with what they actually struggled with.
Then I started pair programming sessions with community developers, observing how they built simple UI applications with Forma 36. I noticed patterns I had never discovered in interviews. One developer, on whose behalf we had invested heavily in documentation, never looked at the documentation. He used the TypeScript types to check what arguments to pass. For him, the types were the documentation. No interview had surfaced that.
The quote that summarizes it: if you want to know people’s real values and real struggles, don’t listen to their words, observe their actions. Pair programming is how I observed the actions.
UX is not DX
User experience is for consumers. Developer experience is for makers. Developers are cooks: the tools a cook needs are different from what a restaurant guest needs, even though both are in the same restaurant.
A concrete example. An e-commerce checkout has essentially one flow: add to cart, check out. Now take uploading a file to GitHub. You can do it manually on github.com, from the terminal with git commands, from the VS Code source control panel, or with GitHub Desktop. Four flows for one task, and which one a developer uses depends on their experience and comfort. A beginner reaches for GitHub Desktop; a git veteran lives in the terminal. Understanding those workflows and toolchains is crucial.
The interfaces of developer experience are the code editor, the terminal, the APIs, the SDKs, and the documentation. All of them, not just the API, not just the docs.
One more concept: developer efficiency. You change code in VS Code and see the result immediately: short feedback loop, high efficiency. Your monolith’s unit tests take many minutes: low efficiency. The cook prepares the same dish many times a day; the guest eats it once. Developers live inside the feedback loop, so the question for your tool is: does it make that loop shorter or longer?
Techniques: community as a research engine
Open a feedback channel. Inspired by the feedback widget in the Next.js docs, we added a simple Google Form (email optional) to the Lokalise Developer Hub. It brings in regular feedback from customer developers: broken links, confusing tutorials, mistakes. One form, better docs.
Recruit at tech events. The hard part of pair programming is finding developers willing to sit with you for 30 to 60 minutes. Meetups and conferences are where you build those relationships face to face. Auth0 ran user testing directly at their Berlin event, with developers already on site.
Start with your internal community. If you have no external community yet, talk to your own client-facing engineers: solution engineers and solution architects. I have regular coffee chats with ours. When I need a rapid five-minute test of developer-facing email copy, they are my test panel.
Friction logs. A technique from Stripe and Google: every new engineer logs every pain point they hit with your tool during onboarding. Where did they get stuck? Did they search Stack Overflow, GitHub issues, Google? Fresh eyes produce the feedback your veterans can no longer see. Then triage what’s worth fixing.
Provide value before seeking value. I cold-messaged community developers introducing myself as a Contentful product manager: very low response. Instead I answered their questions first. Trust built up, and when I later asked for 30 minutes of pair programming, they said yes.
Find your 1%. In any community roughly 1% contribute heavily. Tools like Common Room help you identify those champions, the people who will actually share feedback.
Ask questions instead of broadcasting. A tip from social media: posts that ask the community a question get participation; posts that just announce news do not.
The pair programming toolkit
The tools that made my sessions efficient:
- Calendly, with 15-minute buffers before and after each session: prep time before, debrief time after.
- Zoom: participants share their screen, sessions are recorded with consent, and the annotation tool lets me mark moments to revisit.
- The Mom Test (book): how to ask open-ended questions that get honest answers instead of compliments.
- Google Docs: the step-by-step guide, resources, and test credentials for the participant.
- GitHub + Gitpod: test projects run in the browser, so Mac vs Windows configuration issues disappear and the participant is testing within minutes.
Share observations immediately. The biggest takeaway. My teammates (frontend developers, product designer) join sessions as optional passive observers and take notes. Right after each session we spend 10 to 15 minutes comparing observations and agreeing on the real pain points, which go straight to the Jira board. No convincing needed later, because they saw it themselves.
Takeaways: open your feedback channel, set up a way to find participants through community, provide value before seeking value, and steal any of the tools above.
From the Q&A
Beyond friction logs, how do you get distance from your own product? At Lokalise we debated internally whether the Developer Hub was good enough. Instead of debating, we wrote a hypothesis task: create an API token, find the endpoint, call the API, get a result. Four internal engineers who had never seen our docs tried it, 15 minutes each. They struggled; some names were confusing. We fixed the labels, re-ran with a fresh set of engineers, and everyone completed the flow. Also: Stripe’s CTO has described how every major Stripe launch includes an internal hackathon where engineers test the new feature on their own side projects before beta. Dogfooding at its best.
How do you get a team to adopt this practice? Invite teammates as optional passive observers, never force it. I recorded early sessions and shared the recordings; developers got excited seeing our tools used in real applications for the first time and asked to join the next session. Out of 10 sessions, my frontend developers and product designer attended 8 or 9. The post-session debrief does the convincing: they saw the pain points live, so prioritization got faster.
FAQ
Why do user interviews miss what pair programming catches?
Interviews capture what developers say, which skews toward polite nice-to-haves. Pair programming captures what they do: which tools they open, where they hesitate, what they never read. Workflow-level pain is mostly invisible to the developer themselves, so they cannot report it in an interview.
How do I find developers willing to do a pair programming session?
Answer their questions in your community first, before asking for anything. Recruit face to face at meetups and conferences. Use your own solution engineers as a starting panel, and tools like Common Room to find the 1% of community members who contribute most.
What is a friction log?
A running log every new engineer keeps of each pain point they hit while onboarding to your tool: where they got stuck, what they searched, what the docs were missing. Used at Stripe and Google. New joiners see the friction your team has gone blind to.
Where can I watch the full talk?
The full recording is on YouTube, from the Voxgig DevRel Meetup, September 2023. A version of this talk was also given at Code.talks 2023.