I have just read through the first 3 chapters of Vladimir Braginsky's The Heritage of Traditional Malay Literature: A Historical Survey of Genres, Writings, And Literary Views (2004). Since I started with knowing nothing about Malay literature, I took some time to document the works mentioned.
Before listing out specific works I found there, my overall impressions are:
1. Most of the Malay literature before 1550AD are Hikayat ("tales") that could be prose stories, epics, or historical chronicles - and often a mix.
2. All the works passed down to us were heavily modified after the Malays converted to Islam - virtually no literature (excluding inscriptions) were purely Buddhist or Hindu in content.
3. Besides influence from South Asia (Buddhist, Ramayana, Mahabharata, etc.), from Islam (translations from or works influenced by Persian, Arabic), Javanese influence (both kakawin - Java equivalent of Sanskrit kavya, and various theatrical traditions) were very strong. So strong that I think it makes sense to talk about a broader tradition including Javanese, Malay and Balinese (and maybe other) texts as one big tradition.
Below are some data about the texts - in 6 groups:
Group 1. "Old Malay" texts - V. Braginsky considered them the earliest strata, mostly related to Ramayana - most important being Hikayat Seri Rama
Group 2. Islamic texts - mostly translations from Persian / Arabic original in Pasai - most important being Hikayat Muhammad Hanafiyah
Group 3. Texts related to Mahabharata - most important being Hikayat perang Pandawa jaya
Group 4. Panji tales (related to Javanese Panji stories) - this set makes up the most important and numerically largest branch of traditional Malay literature with 200 MSs and ~100 titles (I read in an article [van der Putten, Jan. "Between iron formalism and playful relativism: five recent studies in Malay writing." Journal of Southeast Asian Studies 38.1 (2007): 147+] that total traditional Malay literature has about 10,000 extant manuscripts with about 3,000 titles) - most important being Hikayat Cekel Waneng Pati
Group 5. Histories written in 1H-16th c. - just two works, with Sejarah Melayu (a.k.a. Sulalat as-salatin) the more important work, and Hikayat Banjar written in Southeast Kalimantan less so.
Group 6. Some minor (?) texts mentioned in notes, but I am not sure whether they were written in Malay language before 1550AD.
In the table below, "# of MSs" means the number of manuscripts that are extant - this is used by Braginsky (I agree with him) as a gauge as to how important a work has been within the tradition.
|Date (A.D.)||# of MSs|
|1||1||Undang-undang Melaka ('Malaccan code of laws')|
|2||1||Hikayat Seri Rama||13-15th c. (complete version)||20+|
|4||1||Hikayat Parang Putting|
|5||1||Hikayat Langlang Buana|
|6||1?||Hikayat Maharaja Rawana|
|7||1?||Hikayat Maharaja Wana|
|8||2||Hikayat Nabi wafat ('Tale of the Prophet's Decease')||As early as 14th c.|
|9||2||Hikayat Muhammad Hanafiyah||1380s||30|
|10||2||Hikayat raja Pasai||~1400 (=1350-1450?)||2|
|11||2||Hikayat Amir Hamzah||1400s or 1410s||20+|
|12||2||Hikayat Iskandar Zulkarnain ('Tale of Iskander the Two-horned')||1400s or 1410s||17|
|13||3||Hikayat Pandawa jaya ('Tale of the victorious Pandawa')||2H-14th c.||(2nd section of work below)|
|14||3||Hikayat perang Pandawa jaya ('Tale of the war of the victorious Pandawa')||~15th c.||12|
|15||3||Hikayat Sang Boma (or Hikayat Sang Samba)||15th - 1H-16th c.||10|
|16||3||Hikayat Pandawa lima ('Tale of the five Pandawa')|
|19||3||Hikayat Pandawa lebur|
|21||3||Hikayat Pandawa panca kelima ('Tale of all five Pandawas')|
|22||4||Hikayat Cekel Waneng Pati||15th - 1H-16th c.||29|
|23||4||Hikayat Naya Kesuma||
|24||4||Hikayat Panji Semirang||5-10|
|25||4||Hikayat Misa Prabu Jaya||5-10|
|26||4||Hikayat Panji Kuda Semirang||5-10|
|27||4||Hikayat Carang Kulina||5-10|
|28||4||Hikayat Jinatur Jaeng Kesuma|
|29||4||Hikayat Carang Mesa Gambira|
|30||4||Hikayat Dewa Asmara Jaya|
|31||4||Hikayat Galuh Digantung|
|32||4||Hikayat Asmara Pati|
|33||4||Hikayat Misa Taman Jayeng Kesuma|
|34||5||Sejarah Melayu||before 1535-36*||29**|
|35||5||Hikayat Banjar||mostly before mid-16th c.|
|35||6||Hikayat Raja Muda|
|37||6||Cerita asal bangsa jin dan segala dewa-dewa ('Story of the origin of jins and celestial beings')|
*Note that dating is highly speculative. For example, based on one preface of Sejarah Melayu, a date of 1612 in Johor was mentioned. Braginsky argued that it was just when a copy was made (and maybe the story somewhat modified).
** This data of 29 was not provided by Braginsky, but from an unnamed online blog post (most likely by Matheson-Hooker based on Braginsky's bibliography) which includes data which there is no real reason to believe that it is false (as it includes detailed MS numbers). As of Apr 27, 2013, the post can be found here. It says: "The number of manuscripts of the Sejarah Melayupreserved in libraries is twenty-nine in all, namely 11 in the United Kingdom (10 in London and 1 in Manchester); 12 in the Netherlands (11 in Leiden and 1 in Amsterdam) ; 5 in Indonesia (Djakarta) and 1 in Russia (Leningrad)."
For the titles bolded above, I would mention generally why these works are important, paraphrasing Braginsky.
2. Hikayat Seri Rama: perhaps the most significant work of non-functional sphere of old Malay literature; proto-version is the first literary work per se in Malay language
9. Hikayat Muhammad Hanafiyah: earliest specimen of Malay hikayat written in Jawi script perserved; provided original pattern for composition of later hikayat; probably oldest translated work from either Arabic or Persian
10. Hikayat raja Pasai: oldest Islamic Malay chronicle; marks beginning of early Islamic period
11. Hikayat Amir Hamzah: Sejarah Melayu in 1530s thought this hikayat of 2,000 pages to be synonymous to the most monumental piece of literature
14. Hikayat perang Pandawa jaya: most complete Malay composition based on the motifs from Mahabharata; possibly the best version; its second section Hikayat Pandawa jaya has incomparable beatuy of exquisite descriptions that remained unsurpassed in traditional Malay literature
15. Hikayat Sang Boma: no less popular than the story of the victorious Pandawa
22. Hikayat Cekel Waneng Pati: most popular Malay Panji tale; one of the most popular traditional Malay works; one of the most interesting Malay compositions and has influenced almost every literary production in Malay
26. Hikayat Panji Kuda Semirang: most complete and artistically impressive Panji romance of the "second type" (where hero always appeared in the story in a high social role, as opposed to the "first type" where the hero would have a lengthy period disguised as a personality from a low social class)
33. Sejarah Melayu: the most important local chronicle; marks the end of early Islamic period; outstanding role in Malay literature similar to Malacca in Malay history; anonymous author was an erudite historiography, talented writer with simple, lucid and clear style, recognized as exemplary for Malay court prose, with descriptive skill unsurpassed in Malay literature
(If we add to the above list of 9 works with 12. Hikayat Iskandar Zulkarnain with 17 extant manuscripts, this would become a "Top 10 list of early Malay literature.")
From what I can gather, in the modern times there clearly are some problems in deciding on what the most canonical Malay literary works are. In a literary tradition, usually the foundational texts have disproportionate standing and influence. However, in the modern times when "innovation" is considered important, and where nationalistic sentiments are strong, it is hard to recognize the Panji romances, Ramayana or Mahabharata tales, or translated Islamic works of Arabic or Persian origin to be core foundational texts. As such, the title of the most canonical text fell onto Sejarah Melayu, which is "romantic" for European orientalist scholars in the sense that the main story describes how Malacca rose and ultimately fell to the Portguese in 1511. Though from the number of MSs extant, it was probably not a work with as wide readership or circulation as say Hikayat Muhammand Hanafiyah or Hikayat Cekel Waneng Pati.