Before discussing the various Fiction Genres, let's first be clear about the difference between Genre Fiction and Literary Fiction.
Genre Fiction, also known as category fiction, is the most popular and most widely read. Literary Fiction is far less commercial, and unless your literary novel happens to win a major prize or get made into a movie, it is unlikely to make you rich. To complicate matters, there is a third type of novel, Mainstream Fiction, which falls somewhere between the two.
Generally speaking, Genre Fiction places more emphasis on plot than character. Its many fans are simply looking for a "good read". With Literary Fiction, the opposite is true - character is more important than plot, as is a concern with exploring philosophical and psychological issues. Mainstream Fiction is concerned with deep characterization and the exploration of ideas and issues, too, though it still seeks to tell a good tale.
If you write a literary or mainstream novel, you are free to write any kind of novel you choose, in any way you like. With a genre novel, you are constrained by having to follow your particular category's conventions, or the "rules" that all novels of a fictional genre must more or less follow - rules about when the dead body must appear in a crime novel, what age the heroine must be in a gothic romance, and so on.
How do you learn these rules? Simple. By reading as widely as you can in your chosen genre or sub-genre and analyzing the characteristics which unite them. You can then knowingly bend some of the rules in order to make your novel unique whilst still being a recognizable part of the genre.
The Principal Fiction Genres
Thrillers: Thrillers feature action-packed plots and grave danger for the protagonist (and often the wider community) as they battle a villain. They can take place against a huge variety of backgrounds - legal, medical, political, corporate, and so on. The James Bond novels are a perfect example of thrillers.
Crime Novels: Also known as Mystery Fiction, this is one of the most popular fiction genres and comes with a huge established audience. The main forms are the Detective Novel, the Private Eye Novel, and the Police Procedural, which involves a whole police team attempting to solve the crime. Crime Fiction can also be told from the point of view of the criminals themselves, and can vary in tone from serious to a comic caper.
Romance Novels: Romance is the largest of the fiction genres and also the most diverse, with countless sub-genres (such as Contemporary, Gothic, and Regency Romance). It also stands out from the other fiction genres by having the most strict set of conventions that the aspiring romantic novelist is expected to follow. The central plot must center around 2 people falling in love, and the ending must always be happy. A strong knowledge of your sub-genre is essential.
Action and Adventure: Action and adventure novels largely target male readers, and feature a hero (or a group of heroes in the case of a war novel) attempting to achieve a specific goal in the face of great danger. Such novels are often set in locations such as jungles or deserts.
Science Fiction and Fantasy: While science fiction is generally about things that might someday be possible, fantasy is concerned with the impossible. Science fiction novels generally take place on other planets or in in space, and fantasy novels are set in invented worlds or in a mythic past.
Horror Novels: The primary aim of horror is to frighten readers by playing on their fears, the source of which can be religious, physical, supernatural, psychological, and so forth. Aspiring writers could do worse than to study the master of this fiction genre, Stephen King.
Other Fiction Genres: Western novels, set in the American West in the nineteenth century, used to be massive, but have now largely disappeared (as they have in Hollywood). Historical novels are very much alive and well, but the historical element of a novel almost always comes second to some other element - so you have historical murder-mysteries and historical romances, for example, which would be categorized under crime and romance respectively.