Performance, Experimentation and Testing (PXT)

Business owners often take ownership, typically through goals and/or operational workstreams, to drive product performance, experimentation, and testing to optimize the customer experience and deliver ROI on behalf of the organization.

Performance

Business owners set team goals to drive performance for operational workflows or drive adoption for products. Teams provide biweekly updates on the following:

• Status of their goals
• Roadmap updates for new optimization levers,
• Current adoption metrics
• Office hours - day of week/time/invite
• Wiki links
• FAQ

Experimentation

Business owners manage a workstream of customer-focused experiments, requested by product and other teams, to incubate new capabilities. After running the experiments, product teams use the results to prioritize generally available products, services, and features based on long-term, sustained value to the business. Managing and executing the backlog of experimentation requests enables business owners and product teams to be constantly curious and nimble by innovating on behalf of their customers. As Jeff Bezos wrote in his 2016 Letter to Shareholders, “Staying in Day 1 requires you to experiment patiently, accept failures, plant seeds, protect saplings, and double down when you see customer delight.”

Source: Elements of Amazon’s Day 1 Culture

The Business team who manages the experiments will send a recurring update to summarize their results and provide additional details on their backlog. Summary details include:

• Team Info
  • Wiki link
  • Office hours weekly days/times
  • Contact Info
• Summary
  • High-level status (i.e. # requests by team, # experiments by test type, # experiments by outcome)
  • Experiment Highlights
• Details of highlighted experiments, including:
  • Background / Hypothesis
  • Requested By
  • Methodology
  • Results
  • Next Steps
• Helpful Links

Testing

Two-pizza teams at Amazon foster ownership and autonomy, as they own the end-to-end experience and have the right resources embedded in them to develop, test, iterate, and scale on behalf of their customers rapidly—and with fewer dependencies. Without the need to maintain complex systems or to solve problems across multiple lines of business, two-pizza teams can dedicate more time and energy to rapidly testing products, services, and features - and innovating on behalf of their own customers.

Source: Elements of Amazon’s Day 1 Culture

Functional testing and acceptance testing are owned by the Product Development teams. However, business teams typically own or support the following types of product tests:

• User Acceptance Testing (UAT)
• Alpha/Beta Testing
• Performance Testing

The product team will often conduct an experiment or release a feature to a subset of users so that the business teams can validate the performance results of the new feature (relative to the current state). Depending on a feature’s reach or potential impact, performance testing and achievement of a minimum result (or lift) are a requirement to release the feature to customers