He had 5 children by Hannah in Canada, one of whom died early.

THE STORY OF LUCKY BALDWIN - aka Elias Jackson Baldwin
(from Lillian May Ruttan's research and given in her words)
William Considine Ruttan (Lucky Baldwin) lived with his wife and children in Canada. He got the gold fever in '49 and with others set out for California to seek his fortune. His mother-in-law did not like him and she it was who intercepted his letters to his wife and her letters to him. I suppose in those days that was not a hard thing to do. The result was he thought his wife no longer cared for him, and of course his wife thought he had deserted her and the children. This hard luck, it is thought, caused Lucky to go wild. He became a great gambler, horeseman, and a "wine, women and song" man. He was immensely wealthy as he struck it rich in gold. He lived with a common-law wife (a Mexican woman) and at the time of his death the property was in litigation afterward for some time because the woman could not produce any proof of a legal marriage with Lucky. It is said that one of his sons in Canada, when he grew up, went to California to prove his relationship; and the family hired a doctor to send Lucky into the next world before the son could claim Lucky as his father. It is said the doctor was paid a considerable sum to do this. Gruesome, isn't it? Lucky loved beautiful horses and the things of nature. His estate is a thing of beauty, so they say, as he went all over the world for different species of trees and shrubs. The town of Arcadia, California, is on Lucky's ranch, and the adobe house is still there. Los Angeles County bought 100 acres (I think I am right in that) and converted it into a natural park and those who have seen it say it is a beautiful spot. Don't you think that a man that loved beautiful horses and nature could not have been all bad at heart? A few years ago moviedom decided to make a picture of his life, but the relatives in Canada begged them not to do it as the research work would bring out the relationship and they did not feel that they should have the Ruttan name dragged in the dust. Clark Gable had already been assigned to the part of "Lucky". Anita Baldwin, daughter of the common-law wife, inherited his fortune, the four children in Canada got nothing. The Santa Anita race track is named for her.

***********************
Following are some references to the story, written by Mrs. Sadie W. Robson of Victoria, Canada, a Ruttan before her marriage. Mrs. Robson wrote this in 1948 and at the time was well along in years. - Gertrude Winona Ruttan
***********************
That William C. called Lucky Baldwin left his wife and four children back in the time of the gold rush, made a fortune out of the race track. Sent thousands home to his wife who never recieved a cent owing to her mother's prejudice against adventuring and the wild and woolly West. Others got the cash and manipulated a separation. Only at the end of his life did his son find him but the fortune had been absorbed by others, and he passed out before the relationship was legalized. Every once in a while there would be an ad in some paper asking for relatives of "Wm. C. Ruttan associated with the race track". It is a long story and the poor wife supported her four children by sewing, grieving because she got no word or support from her husband. Mail was easily deflected and everybody couldn't write in those times. Then there is another distant cousin I met in Toronto, now a widow if alive, Mrs. Hannah Batty whose mother was deserted by her husband through the interception of mail and was defrauded. One of Mrs. Batty's brothers, (who was a baby when his father left for the wild and woolly West) called on Wm. C. Ruttan in California and claimed relationship, but his father was ushered into the next world before the claim could be settled although they were very much alike. (By the way, our Ruttan family incline to have a hollow in the forehead above the nose.)

"William Consider left for California perhaps early in 1854, possibly accompanied by his brother Jacob. It is said that letters he wrote to his wife were intercepted by his mother-in-law and never reached Hannah. Also, it is stated that in California, William Consider took the name of E. J. Baldwin (Known as Lucky Baldwin) and became wealthy, owning the property on which the Santa Anita race track is situated. It is also said that he had visited Russia. His youngest son, Wesley, spent years in search of his father and found him at a race track on February 28, 1909, and arranged to meet him the next day; but William Consider died that night and later it was referred to as murder by poisoned food. Efforts made by members of the family have not so far established any right of inheritance as heirs of William Consider." (H. N. Ruttan, p. 39)

“Wesley (my ggrandfather) met his dad (Lucky Baldwin) at the race track Feb. 28,1909 and was then invited to come to his home the next day. When Wesley went there he was shocked to find that his father had been poisoned.
Returning to Toronto,Ont. to his family he hired a lawyers to represent him. The lawyers came to the home and took all the proof they had about their relationship to Lucky Baldwin. The documents which included his (Lucky's) own handwriting were never seen again. Our family has held to the notion that the Toronto lawyers were paid off by the other party in California. My grandmother remembers the day when two men in suits came to their home and took the papers.”
(from Greg N. Barr Dec ‘98)

[Obituary] Oakland Tribune - 1 March 1909, pg 1
LUCKY BALDWIN PASSES AWAY AT ARCADIA
Noted Turfman who lived like Spanish Don, Expires at age of 81
Wife and Relatives see end at bedside
Vast Fortune said to have been left by Well Known Horseman
Los Angeles, March 1 - Elias J. Baldwin famous plunger and turfman, known throughout this country and Europe as Lucky Baldwin, died at his residence Arcadia on the celebrated Baldwin Ranch, fifteen miles from Los Angeles at 7 o’clock this morning. He was 81 years of age.
The end came after a lingering illness of several weeks. He passed quietly away surroounded by family and friends, after having been unconscious most of the night. At his bedside this morning were Mrs. Baldwin, their daughter, a niece and other relatives.
The illness which finally resulted in the death of Baldwin began on February 2. At that time he experienced a serious sinking spell, but from which he rallied and made a gallant fight for life. Preious to this attack however, he had suffered for several weeks with influenza and had refused to be attended by a physician, and when Dr. J. W. Trueworthy of this city was called on the night of February 2, he found Baldwin much enfeebled and in a highly nervous condition.

(The Oakland Tribune - 2 April 1933)
All through his life Elias J. Baldwin was saying "By Gad, I'm not licked yet!" He said it during his early ventures in the livery and real estate businesses, in the mining lands of the Comstock and Alaska, when his magnificent hotel burned, and when the horses did not run fast enough on the track. C. B. Glasscock has written the biography of the man who amassed a fortune in the West and yet was often broke, a tale of his perseverance and rather astounding personal adventures. As a great deal of the incident is of our early San Francisco life, and everyone here knew "Lucky" Baldwin, it is a chronicle made to our order. Glasscock tells of the man who never shoveled a pound of ore and yet made a fortune in the Comstock; of the great ranch deals he made; how he was first to develop Lake Tahoe: the horseracing days; his early vaudeville venture when he brought some Japanese performers to this country; of the times he was made target for bullets from revolvers held in fair hands; of his loves and his extravagances; his hoarding and his lavishness; and, also, of his brave venture to the Klondyke, to start anew at the age of seventy-two.
"Lucky" Baldwin who left ten millions which grew to twenty before the estate was settled, who enjoyed himself, helped build his city, and was at once notorious and famous, is a figure romantic and bizarre.