Find a niche or unique selling point competitors are not filling.
Identify the most important competitors and list their offerings. See which of the user needs are unaddressed by the competition. If you can deliver this, you have found your niche.
A willingness to offer something unique.
A good idea about who the competition is (also outside your domain).
A good idea of what your users want above the competitors’ offerings.
A competitive analysis is part of many new product offerings. Large-scale competitive analyses often include a field study as well, to identify user needs.
Universal methods of design (Martin & Hannington , 2012) spend a couple of pages about competitive analysis. In the dutch version: chapter 15, p36-37
Competitive analysis is also well described in Observing the User Experience. Chapter 5. Goodman, Elizabeth, Mike Kuniavsky, and Andrea Moed. "Observing the User Experience: A Practitioner's Guide to User Research." Professional Communication, IEEE Transactions on 56.3 (2013): 260-261.