Caravaggio’s art is made from darkness and light. His pictures present spotlit moments of extreme and often agonised human experience. A man is decapitated in his bedchamber, blood spurting from a deep gash in his neck. A man is assassinated on the high altar of a church. Faces are brightly illuminated. Yet always the shadows encroach, pools of blackness that threaten to obliterate all.

Caravaggio’s life is like his art, a series of lightning flashes in the darkest of nights. He was one of the most original artists ever to have lived, yet we have only one solitary sentence from him on the subject of painting — the sincerity of which is, in any case, questionable, since it was elicited when he was under interrogation for the capital crime of libel.

Much of what is known about him has been discovered in the criminal archives of his time. He lived much of his life as a fugitive, but is caught, now and again, by the sweeping beam of a searchlight. Caravaggio throws stones at the house of his landlady and sings ribald songs outside her window. He has a fight with a waiter about the dressing on a plate of artichokes. He taunts a rival with graphic sexual insults. He attacks a man in the street. He is involved in a fatal swordfight.

Anyone attempting a biography of Caravaggio must play die detective as well as the art historian. His life can easily seem merely chaotic, the rise and fall of an incurable hot-head, a man so governed by passion that his actions unfold without rhyme or reason. But there is a logic to it all and, with hindsight, a tragic inevitability.

A lot has been made of Caravaggio’s presumed homosexuality, which has been presented as the single key to both his art and his misfortunes. There is no absolute proof of it, only strong circumstantial evidence and much rumour.

The truth is that Caravaggio was as uneasy in his relationships as he was in most other aspects of life. He likely slept with men. He did sleep with women. But he settled with no one. From a very young age, and with good cause, he suffered from a deep sense of abandonment.

If any one thing lay behind the erratic behaviour that doomed him to an early death, it was the tragedy that befell him when he was just a little boy. To understand the experiences that most deeply shaped him, it is necessary to begin where he was born: in the town of Caravaggio, in Lombardy, from which he would later take his name. He lived both there and in the nearby city of Milan for the first 21 years of his life.

In the summer of 1576, when Caravaggio was almost five, Milan was struck by an outbreak of bubonic plague. A year later, the plague tore into Caravaggio’s own family. By the age of six, Caravaggio had lost almost every male member of his family, and the art of his maturity would be saturated in the ineradicable memory of night terrors, filled with images of turmoil in dark places.

‘Whore, bitch, tart!’ These are the words of an artist scorned, addressed to a courtesan who refused to sleep with him. They are preserved in a deposition in the State Archives of Rome for 1602. The man was before the magistrates for abuse and physical assault.

As well as insulting and beating her, he had actually knifed the woman. She had been badly injured, cut deeply to the face. The facial wound was an example of a sfregio, a slash with the blade inflicted as a mark of shame – doubly damaging to a courtesan, whose face was her fortune.

There are many such tales in the annals of the lives of the artists who thrived, floundered or failed in Counter-Reformation Rome. During his own 14 years in the city, where he moved after cutting all ties with his family at the age of 21, Caravaggio would become embroiled in more than his fair share of bloody vendettas. He was a violent man, but he lived in a violent world. Throughout 17th-century Italy an inflammatory code of honour prevailed. The fama of an individual, which referred to not only his fame or reputation but also his good name, was paramount. Any insult to it had to be paid for, and the price was often blood. Caravaggio went to greater extremes than his contemporaries, in life as in art.

The artists’ quarter of Rome at the end of die 16th century was a dangerous area. Fights were common and fists were not the only weapons used. The ultimate ambition of every young, jostling artist was the same: to work for the cardinals closest to the pope, to secure the most important devotional commissions and win lasting fame -with the money and security that went with it. In a world where rivalry was intense, stories of sabotage abounded – of collapsed scaffolding, of the painter whose rival poisoned his colours with acid so that all his blues turned green in a matter of days.

When Caravaggio arrived in the city, in the autumn of 1592, he was just another artist on the make. Having begun painting crude ‘heads’ – most likely those of famous men of the past – he eventually found himself a more exalted position in with the Cesari brothers, one of whom was among the most prominent artists in Rome. Caravaggio was employed to paint ‘flowers and fruit’ within the relatively new field of still-life painting.

In 1579, Carlo Borromeo, the dour Archbishop of Milan, had tried, unsuccessfully, to kill off the exuberantly joyful pre-Lenten tradition of Carnival. Caravaggio lived his life as if there were only Carnival and Lent, with nothing in between. His pictures are the legacy of his Lenten days. His carnivalesque alter-ego emerged in his early days in Rome, where, according to one biographer, Caravaggio liked to go about with a crowd ‘who lived by the motto nec spe, nec metu, “without hope, without fear”.’The most dangerous of these companions was a hot-headed architect called Onorio Longhi, who would patrol the streets of Rome on horseback, as if he was a knight and his servant was his page. Those who ran with him behaved like modern, debased versions of the ‘veray parfit gentil knights’ of the old romance tradition, whoring and fighting rather than slaying dragons and protecting damsels in distress.

Caravaggio was certainly friendly with prostitutes, some of whom modelled for him. His favourite was Fillide Melandroni, a dark-eyed girl destined to become one of Rome’s most famous courtesans. In 1598 or 1599, Caravaggio painted a startlingly sado-erotic Judith and Holofernes, with Fillide in the leading role. Like that of David and Goliath, the biblical story of Judith was a parable of underdog virtue triumphing over tyranny: a Jewish heroine seduces a ruthless Assyrian general and then slays him, with his own sword, in his tent.

Under Caravaggio’s hand, sanctified execution in an Assyrian tent has become murder in a Roman whorehouse. The bearded Holofernes, lying naked on the crumpled sheets of a prostitute’s bed, is a client who has made a terrible mistake. He wakes up to realise that he is about to die. Fillide pulls on his hair with her left hand, not only to expose his neck but to stretch the flesh so that it will part more easily. She frowns with grim concentration, as he screams his last, and as the blood begins to spray from the mortal wound in bright red jets.

Caravaggio has imagined the whole scene as a fantastically extreme version of the violent incidents that he and his companions were so often embroiled in. He adds a sexual frisson to the violence: beneath the diaphanous fabric of her tight-fitting bodice, Fillide’s nipples are visibly erect. It is the sort of detail that Cardinal Paravicino may have had in mind when he made his famous remark about pictures that he ‘would not have wanted to see from a distance’. Judith and Holofernes divided Caravaggio’s contemporaries, many of whom found the realism rude and indecorous. Others were fascinated by it.

On July 4 1600 the painter received a final payment of 50 scudi for The Calling of St Matthew and The Martyrdom of St Matthew, two paintings destined to adorn Rome’s prestigious Contarelli Chapel. His matchless sense of drama and use of extreme contrasts of light and dark would prove intoxicating. It is no exaggeration to say that they decisively changed the tradition of European art.

The Contarelli paintings were controversial but they instantly established Caravaggio as one of the leading painters of the city. There is no sign that success mellowed him. During the winter of 1600 he injured one of Rome’s many unemployed mercenaries in a swordfight. On November 19 1600, he was charged with a nocturnal assault on a young Tuscan art student named Girolamo Spampa. The premeditated attack – Caravaggio stabbed Spampa from behind – reeked of vendetta.

Just over five years later, deeply in debt, recently evicted and in trouble, yet again, with the law, Caravaggio received a commission that must have seemed like a God-given chance for him to paint his way out of trouble. He had finally been asked to paint an altarpiece for St Peters, the central church of Catholic Christendom.

Caravaggio had finished the work in less than four months. The Madonna of the Palafrenieri, sometimes known as The Madonna of the Serpent, is monumental in scale. Almost 10ft tall and more than six across, it shows three figures absorbed in a confrontation with pure evil. The Virgin and the infant Christ together crush the head of a serpent beneath their feet. Strangely for Caravaggio, there is no sense of drama. Instead of telling a story he was obliged to embody an allegory and make a theological point. He did his utmost to produce an unimpeachable endorsement of the prevailing religious orthodoxy – which can only have made what happened next all the more painful.

Within two days of going up, the altarpiece had been removed, and was soon to be sold on to a private collector. It was almost certainly Caravaggio’s embodiment of the Virgin in a low-cut dress that caused the difficulty. Appealing once more to the mass of ordinary Catholics, Caravaggio had simply painted her as the kind of mother with whom real mothers might identify.

Despite this enormous setback, Caravaggio refused to change his approach. Shorty after delivering The Madonna of the Palafrenieri, he finally completed his long overdue altarpiece of The Death of the Virgin. This huge and deeply moving picture is stark evidence of the painter’s reluctance to compromise, and of his moral resilience.

Never before in the history of Christian painting had the mother of God been made to seem so poor and frail and vulnerable. Wearing a simple red dress, unlaced at the bodice to make her more comfortable in her last moments, she lies stretched out on the makeshift bier of a plank of wood. She looks shockingly dead. The church of Santa Maria della Scala, for which the painting was intended, belonged to the order of the so-called discalced Carmelites, the shoeless Carmelites. This may have encouraged Caravaggio to believe that his stark depiction of the Virgin might find favour. But no sooner was his painting delivered than he learnt that it too had been rejected.

The Carmelite brothers had reportedly found it ‘well made but without decorum or invention or cleanliness’. To say a picture had been created ‘without invention’ was shorthand for saying that it had been painted from reality rather than the imagination. The Madonna had been made to look ¡ dirty and indecorous. She was made to look real.

This second rejection must have cut Caravaggio to the quick. Looking back on it years later, Giulio ; Mancini, a physician from Siena who knew I Caravaggio well, wondered if the refusal of The Death of the Virgin might not have been the tilting point of the painter’s whole life. ‘Perhaps consequently, Caravaggio suffered so much trouble,’ E he wrote. It is just an aside, but it should not be taken lightly. Mancini had seen first-hand what happened next. In the immediate aftermath, Caravaggio committed a crime that would blight the rest of his life. He killed a man.

What were the circumstances that led to Caravaggio fatally stabbing the pimp Ranuccio Tomassoni in a duel on May 28 1606? Numerous theories abound but one thing is certain: a bitter enmity had been building between the men. Tomassoni had been the pimp who controlled Caravaggio’s favourite courtesan, Fillide. From the outset, Tomassoni would have been dismayed to discover their association: a mere painter was hardly a desirable client for Tomassoni s most beautiful courtesan. There is even evidence to suggest that Caravaggio might have encroached on Tomassoni’s territory by becoming a part-time pimp himself. Caravaggio needed models, so rather than be at the mercy of pimps, why not secure his own whores?

One other detail suggests that the cause of the fight may have been some kind of sexual insult. Tomassoni bled to death from the femoral artery. Caravaggio had struck him a low blow, aiming perhaps at the groin and missing by just a fraction. Wounds were meaningful. A cut to the face was a sfregio, but it was by no means the only form of symbolic, premeditated injury that Italians inflicted upon their enemies. It is possible that Caravaggio was not trying to kill Tomassoni, but attempting to make mincemeat of his testicles with a sword.

This time, retribution was swift: Caravaggio was condemned as a murderer and made subject to a bando capitale, a ‘capital sentence*.This meant that anyone in the papal states had the right to kill him with impunity; indeed there was a bounty for anyone who did so. To claim the reward, it would not be necessary to produce the painter’s body. His severed head would suffice.

Badly wounded himself, the painter had no choice but to flee Rome. Yet his yean in exile would be no less eventful or dramatic. Caravaggio first fled to Malta, where he joined the prestigious order of the Knights of St John, before clashing with one of their most illustrious members and finding yet another death sentence upon his head. On the run, once again, Caravaggio escaped to Naples, where, on the brink of securing a papal pardon for the murder of Tomassoni, he made an ill-advised visit to a seedy Neapolitan tavern. Here, his life of violence finally caught up with him. This time it was Caravaggio who was the victim of a ‘hit’ by a group of armed men who were almost certainly sent by Fra Giovanni Rodomonte Roero, the ‘noble Knight’ he had argued with in Malta.

Caravaggio was badly cut on the face, which in the honour code of the day was an injury to avenge an insult to reputation. He would never recover, and it was almost certainly these injuries that would leave him so weakened as, in the summer of 1610, he made his way back to his beloved Rome after finally receiving confirmation of his pardon.

The artist was travelling by boat, with three of his unsold paintings, all of which were intended for Scipione Borghese, a papal nephew who was fond of the artist and had commissioned him before. With Caravaggio was a trusted skipper, Alessandro Caramano. It was this humble boatman who would be the source of the artist’s final movements. After docking at Palo, he told authorities, a misunderstanding with the captain had resulted in Caravaggio being jailed. The boat continued to Porto Ercole, where, presumably, Caravaggio attempted to make his way in a desperate attempt to recover his paintings. The stress of his arrest, and the frantic ride to Porto Ercole in the heat of July, was more than a man in his condition could take. Heat exhaustion, or a heart attack, may have been what finally killed him.

Caravaggio had lived much of his life surrounded by poor and ordinary people. He painted for them and from their perspective. In the end he died and was buried among them, in an unmarked grave. He was 38 years old.

Why did you decide to write a book about Caravaggio? Is there anything new to say about him, after all these years?

It might seem strange, but even though Caravaggio is one of the most obsessed-about and massively popular artists of all time, an extraordinary amount of devastatingly interesting and revealing new information has come out about him in recent years. I am talking about new, hard, archival discoveries, a truly astonishing treasure trove of documents from the distant past – new facts about the prostitutes with whom he consorted, the women (and perhaps men) with whom he had sex; the soldiers, mercenaries and thugs with whom he fought and argued; the other painters with whom he contested; the man he murdered in a castration-attempt on a tennis court in Rome in the summer of 1606; the man he shot and near-fatally wounded on the brutal military island of Malta a few years later; the gang of four who pinned him down and cut his face off in Naples, condemning him to a terrible, slow, lingering death …

What sets your biography apart from others that have been written about him?

All of this new stuff had been discovered by scholars working in different places in different, disconnected ways. Some of it had been published, but generally only in aracane or extremely academic corners. No one had put it together, however. So I did. Whether the reader thinks I have done a good job with the information, the information itself is transformative in terms of what we know and think about the painter. I think it’s like a bomb dropped into the still waters of existing Caravaggio literature. Basically, my book is the first to bring all of this information together for the general reader. As a result, I believe it is the first book about him truly to tell the full story of who he was, why he did what he did, and ultimately what happened to him.

How do you hope to change perceptions of him?

For centuries Caravaggio has been regarded as a mysterious, rather mad outsider. After ten years and more of detective work, I believe I have finally been able to make sense of him, and of the patterns that shaped his life – deeply tragic patterns from the start, going back to the death of all his male relatives, of plague, when he was just 5 or 6 years old. I hope that my book releases him from the many stereotypes he has been subjected to in so many of the other books (and films and novels) that have taken him as their subject: Volatile Lunatic, Tragic Outsider, Protypical Gay Icon. For me, he’s none of those cardboard cut-outs. He’s a real, complicated, dangerous but also deeply sympathetic human being.

Are there any other ambitions behind your book?

Most definitely. The nature of the Caravaggio archival treasure trove, as I have called it – different sets of documents unearthed in archives in Rome, Naples, Malta and elsewhere – is that it doesn’t just give you this extraordinary, troubled man’s life. It gives you his whole world, and it is a truly fascinating world, one where people live by very particular, apparently strange but ultimately logical codes of honour.

So for example, in this world, if a painter insults Caravaggio behind his back, Caravaggio will go up behind him late one night and smash him on the back of the head with the back of his sword: the logic being that if you insult me behind my back, I attack you from the back. If a woman insults Caravaggio’s reputation, he will smear excrement on the windows of her house: the logic being that if you attack my honour, which is my face, then I besmirch the front of your house, the architectural face you present to the world. If a man argues with Caravaggio about a woman, Caravaggio will attempt to castrate him in a duel: the logic being that if you insult me sexually, I will wound you sexually. What I hope you get from my book is a true and deep understanding of the codes and the logic – however twisted that logic might be – by which Caravaggio and his friends and enemies lived their lives. In other words, he was not some freak or weirdo, but a dangerous man in a dangerous world.

Because I find that world utterly, transfixingly interesting, I have tried wherever possible to give the reader the documents that survive in full: the whole of a trial transcript, for example, or the entirety of a prostitute’s account of attacking her rival. I explain who the people involved are, I come to my conclusions, but I also give the reader as much as possible of the raw history, so they can decide for themselves whether they agree with my conclusions and inferences. Also, by quoting these documents – the few other books to make limited us of them have abridged or summarised them – I feel I put the reader really in touch with the feel and the smell of seventeenth-century Rome, or Naples, or Sicily.

What about the paintings? 

Well, they are the reason I wrote the book in the first place. If you like, my book is three books interwoven together: the story of Caravaggio’s life; the portrait of the world in which he lived with all its codes and customs etcetera; combined with deep, lengthy analyses of each and every painting. The part I would most like to be judged on is the last. My analysis of his pictures – those dark, dramatic, deeply profound depictions of men and women in extremis – well for me that is the heart and soul of my book.

I hope I have proved for once and all that Caravaggio was not just some flash in the pan, some gifted proto-photographic master of realism; he was an immensely subtle, emotionally profound, intellectually complex artist. I don’t use the genius word lightly and there aren’t many artists or writers I would apply it to, but he really was a genius. I also think he was one of the most touchingly, deeply humane and human human beings ever to have lived.

What’s your most vivid memory of writing the book?

Sitting in my study late and night and suddenly realising that I’d solved the supposed mystery of how he died and who had killed him. I started crying, for Christ’s sake! I even realised that I knew the name of the man who was the last person to see him alive: a humble boatman called Alexander Caramano, who took Caravaggio to his death on a boat named Santa Maria dello Porto Salvo, Saint Mary of the Safe Harbour. History can contain such astonishingly cruel ironies.

His life had the patterns of a tragedy, the patterns of a work of art, almost. You couldn’t have made it up. It was such a sad, sad life, in the end. But what dark gold it produced.