Lee Chandler is a janitor working in Quincy, Massachusetts, whose routine is typically quiet and reserved. After swearing at an impatient tenant at one of the locations where he works, he is reprimanded by his boss, which culminates into getting into a fight with two men at a bar after they give him weird looks. One morning, he receives a call from George, a family friend in Beverly, reporting that Lee's brother Joe is in the hospital after falling from his boat, having suffered a heart attack. Despite heading there immediately, on arrival Lee is informed by George that Joe died an hour earlier.
Lee heads to Manchester-by-the-Sea to inform Patrick, Joe’s son, of his father’s passing. Lee temporarily moves into Joe’s home to both look after Patrick and sort out the details of Joe’s funeral. Lee appears distant to Patrick, a contrast to their relationship when Patrick was a child and when Lee was married to Randi, raising two young daughters and an infant son with her. After meeting with Joe’s lawyer about his will, Lee is shocked to learn that Joe has entrusted him guardianship of Patrick, causing him to remember the death of his daughters and son in a house fire and his attempted suicide after, subsequently leading to his and Randi’s divorce. Randi contacts Lee, asking if it would be okay if she attends the funeral. During their conversation, Randi reveals that she is pregnant. Randi arrives at the funeral with a man, presumably her new husband. Lee and Randi briefly embrace, but they do not speak.
Due to frigid weather delaying the funeral, Lee has Joe’s body preserved in a freezer at the hospital, which Patrick feels negatively about. This forces Lee to remain in Manchester with Patrick. Lee also suggests selling Joe’s boat that he left for Patrick, as the motor is about to die and regular maintenance would be too costly, but are able to afford a new motor after selling Joe’s hunting rifles. Patrick also spends time with Sandy, a girlfriend who he plays with in a band, though their attempts at sex are interrupted by her unaware mother Jill, who seems interested in Lee. Patrick attempts hooking Lee up with Jill, but she is unenthused by his unwillingness to talk, leading Patrick to criticize Lee about not wanting anything to do with him. Patrick’s resentment is further enforced by Lee’s insistence to work as a janitor outside of Manchester.
Lee soon learns that Patrick has been in touch with his mother, Elise, who has been absent from Joe's and Patrick’s lives due to her history of alcoholism and drug use, for which Lee scorns her. However, she’s revealed to be sober and engaged to Jeffrey, a devout Christian. Patrick visits Elise at her home in nearby Essex. Her behavior during the visit unsettles Patrick, and he leaves feeling dejected and despondent.
Lee announces to Patrick that he has gotten a new janitorial job in Boston, and George and his wife agree to look after Patrick until he is of age. Lee runs into Randi, accompanied by her newborn child. She becomes very emotional, expressing remorse for the manner in which she treated him after their children perished. She appears eager to reestablish a connection with him, but he abruptly walks off, leaving her sobbing. After Joe’s burial, Lee and Patrick go for a walk, Lee informing Patrick that his new residence has an extra room, encouraging him to visit any time he pleases. They go fishing on Joe’s boat, as they had done years ago.