I'm trying to figure out the rule that puts the 'maaw' sound into มรดก.

Breaking it up, it is pronounced as:

มอ + ระ + โดะ + ก

But why is it มอ instead of โมะ or มะ?

Insertion of implied vowels is called Epenthesis or Suwaraphak (a.k.a. svarabhakti). See this, section III ( b )

According to grammar rules:

If the first syllable is an alone consonant of ท/บ/ม and the following syllable starts with (or consists of, like in your example) ร, the implied vowel is -อ.

Otherwise, it is -ะ.

Applied twice, this explains the first two syllables, มอ + ระ.

The word ดก follows another rule. The syllable is closed (or "dead"). Although the vowel is not written, it considered an unwritten form of a certain vowel โ-ะ.

Implied vowel between two consonants ......... 'oh'

Implied vowels between three consonants ....... 'ah' then 'oh'

UNLESS second consonant raw rua in which case 'aw' then 'oh'

Most of the time anyway,

สมนาคุณ sohmR maH naaM khoonM

then would be an exception to - Implied vowels between three consonants ....... 'ah' then 'oh'

สมนาคุณ sohmR maH naaM khoonM

then would be an exception to - Implied vowels between three consonants ....... 'ah' then 'oh'

Words taken from Sanskrit are deeply individual in spelling and often follow no Thai rules.

สมนาคุณ sohmR maH naaM khoonM

then would be an exception to - Implied vowels between three consonants ....... 'ah' then 'oh'

There's no exception in the word of สม. It's an unwritten form of the vowel โ-ะ. Please review the grammar rules (linked in comment #2)

bytebuster, your rule doesn't work for words like ทรง, ทรวง, บรรดา, บร, มรณะ, and many others.

Either there are many exceptions, or the rule is missing something . . .

(ps - I know ทร makes an 's' sound, and รร makes an อัน sound)

bytebuster, your rule doesn't work for words like ทรง, ทรวง, บรรดา, บร, มรณะ, and many others.

"My" rule works.

Specifically:

• ทรง, ทรวง - the first syllable is not an alone consonant of ท/บ/ม and the following syllable does not start with (or consists of, like in your example) ร, specifically because ทร- is a false consonant cluster, see here, topic II.
  
• บรรดา - same as above, the first syllable is not an alone consonant because รร is a false consonant cluster; see same link, topic IV ( b )
  
• บร - same as above, the first syllable is not an alone consonant. The sentence can't end with an Epenthetic syllable (alone consonant), so ร must belong to the 1st syllable therefore becoming a final consonant.
  
• มรณะ - the rule applies very well, it reads [มอ - ระ - นะ]
  
• and many others - please provide with the list
  

Either there are many exceptions, or the rule is missing something . . .

There may non-obvious uses, and there may be even exceptions (in the ancient loanwords), but the rule is written in many classic books of Thai language, believe it or not.

(ps - I know ทร makes an 's' sound, and รร makes an อัน sound)

Then why to insist that the rule should apply if it clearly say, "the next syllable starts with..."? It is not the next one.