各位學(xué)員好，前天有個(gè)《python信用評(píng)分卡建模（附代碼）》課程學(xué)員向我提問(wèn)。她在銀行工作，銀行對(duì)金融風(fēng)控模型驗(yàn)證和壓力測(cè)試非常重視。目前不清楚如果對(duì)風(fēng)控模型壓力測(cè)試。

這個(gè)問(wèn)題很專業(yè)，很少有學(xué)生提到這問(wèn)題。目前國(guó)內(nèi)對(duì)風(fēng)控模型的驗(yàn)證并不重視，只有銀行和少數(shù)持牌照消費(fèi)金融公司有模型驗(yàn)證團(tuán)隊(duì)。有的模型驗(yàn)證團(tuán)隊(duì)并不正規(guī)，難以保持工作獨(dú)立性。

國(guó)內(nèi)銀行參考了美國(guó)美聯(lián)儲(chǔ)SR 11-7文件Guidance on Model Risk Management，翻譯為中文為模型風(fēng)險(xiǎn)管理指南。

美聯(lián)儲(chǔ)2011年4月發(fā)布模型風(fēng)險(xiǎn)管理的相關(guān)法規(guī)——SR 11-7號(hào)文《模型風(fēng)險(xiǎn)管理指南》，該文明確了模型風(fēng)險(xiǎn)管理框架，包括模型風(fēng)險(xiǎn)管理定義、模型實(shí)施、模型驗(yàn)證、以及模型風(fēng)險(xiǎn)管理的政策制度等內(nèi)容。SR 11-7號(hào)文重點(diǎn)在模型驗(yàn)證環(huán)節(jié)。

回到正題，模型壓力和風(fēng)險(xiǎn)測(cè)試很難，當(dāng)客群變化后，模型穩(wěn)定性下降，即PSI指數(shù)上升，模型是需要重新迭代的。當(dāng)發(fā)生疫情后，如果失業(yè)率快速上升，客戶違約風(fēng)險(xiǎn)也會(huì)快速上升。這不是子模型能解決的問(wèn)題，需要建立宏觀經(jīng)濟(jì)模型來(lái)提前預(yù)測(cè)失業(yè)率等指標(biāo)，然后收緊策略，縮小放款范圍。

至于高風(fēng)險(xiǎn)客戶，模型還是有辦法的。我推薦邏輯回歸評(píng)分卡模型建模，評(píng)分卡的woe分箱中，可以把缺失值或異常值(高風(fēng)險(xiǎn)值)單獨(dú)分箱。因此評(píng)分卡模型對(duì)異常值和缺失值有很高容忍度。感興趣朋友可以收藏和關(guān)注我的課程《python信用評(píng)分卡建模（附代碼）》。該課程還有模型驗(yàn)證的常用方法，包括模型區(qū)分能力，排序能力，穩(wěn)定性驗(yàn)證方法。

不幸的是，國(guó)內(nèi)諸多金融公司在建模中存在結(jié)構(gòu)性問(wèn)題。例如一個(gè)申請(qǐng)單進(jìn)入審批，策略會(huì)先評(píng)估該客戶信用風(fēng)險(xiǎn)，如果沒(méi)問(wèn)題才進(jìn)入模型判斷。模型幾乎是墊底的。這意味著諸多變量的異常值已經(jīng)被提前過(guò)濾了。這種模型是非常脆弱的。因此我還是建議大家把異常值，高風(fēng)險(xiǎn)值加入模型訓(xùn)練，提高模型抗壓能力。畢竟評(píng)分卡模型可以對(duì)異常值，高風(fēng)險(xiǎn)值單獨(dú)分箱處理。

大家不要把模型想的很復(fù)雜，你可以把模型想成一個(gè)人。這個(gè)人在平時(shí)訓(xùn)練時(shí)經(jīng)歷過(guò)大風(fēng)大浪，才有抗壓風(fēng)險(xiǎn)。如果這個(gè)人經(jīng)歷太順利，可能遇到一個(gè)挫折就不知所措。

對(duì)于壓力測(cè)試，我還想談?wù)劥蠹覍?duì)異常值認(rèn)真的誤區(qū)。

很多建模人員遇到異常值時(shí)，直接刪除異常值，然后建模。對(duì)于壓力測(cè)試，這是不妥的。很多高風(fēng)險(xiǎn)值就是異常值。當(dāng)欺詐客戶行騙時(shí)，就會(huì)觸發(fā)某些變量產(chǎn)生異常值。如果我們輕易刪除異常值，就不會(huì)捕捉到這些欺詐客戶，而且模型區(qū)分能力可能降低。《python信用評(píng)分卡建模（附代碼）》的give me some credit案例章節(jié)就有去掉異常值后，模型區(qū)分能力下降實(shí)驗(yàn)測(cè)試。

模型風(fēng)險(xiǎn)管理還有個(gè)難點(diǎn)就是人禍。經(jīng)營(yíng)者在面對(duì)巨額利潤(rùn)時(shí)，會(huì)刻意忽略風(fēng)險(xiǎn)。這也是造成風(fēng)控部門(mén)和業(yè)務(wù)部門(mén)長(zhǎng)期對(duì)立原因。業(yè)務(wù)部門(mén)想賺更多錢(qián)，讓風(fēng)控部門(mén)放松策略。風(fēng)控部門(mén)想降低風(fēng)險(xiǎn)，收緊策略。而且業(yè)務(wù)部門(mén)和風(fēng)控部門(mén)考核KPI指標(biāo)是相反的。

之前發(fā)布過(guò)文章《捷信金融（Home Credit N.V.）興衰如夢(mèng)，留下寶貴機(jī)器學(xué)習(xí)建模數(shù)據(jù)》。捷信作為國(guó)內(nèi)最早消費(fèi)金融公司，擁有大批風(fēng)控專家，其風(fēng)控策略和風(fēng)控模型是非常強(qiáng)悍的。隨著捷信線下迅速擴(kuò)張，造成風(fēng)控跟不上和故意對(duì)風(fēng)險(xiǎn)視而不見(jiàn)，最后捷信老大哥跌下神壇。

電影《大空頭》給觀眾解釋了美國(guó)華爾街金融危機(jī)爆發(fā)前，諸多數(shù)據(jù)分析師已經(jīng)察覺(jué)風(fēng)險(xiǎn)。但美國(guó)政客，金融從業(yè)人員甚至風(fēng)險(xiǎn)監(jiān)管人員為了高額利潤(rùn)對(duì)其視而不見(jiàn)，造成后來(lái)全球巨大經(jīng)融危機(jī)，甚至到今天問(wèn)題都沒(méi)解決。大家可以去看看這部經(jīng)典電影。

最后我把SR 11-7：模型風(fēng)險(xiǎn)管理指南（Guidance on Model Risk Management）中英文放入文章，供學(xué)員參考。

致各聯(lián)邦儲(chǔ)備銀行負(fù)責(zé)監(jiān)督的官員和適當(dāng)?shù)谋O(jiān)督檢查人員

主題：

模型風(fēng)險(xiǎn)管理指南

美聯(lián)儲(chǔ)和貨幣監(jiān)理署 (OCC)正在發(fā)布隨附的模型風(fēng)險(xiǎn)管理監(jiān)管指南，供銀行組織和監(jiān)管機(jī)構(gòu)在評(píng)估組織對(duì)模型風(fēng)險(xiǎn)的管理時(shí)使用。本指南應(yīng)酌情適用于受美聯(lián)儲(chǔ)監(jiān)管的所有銀行組織，同時(shí)考慮每個(gè)組織的規(guī)模、性質(zhì)和復(fù)雜性，以及模型使用的范圍和復(fù)雜程度（定義和討論如下）。

模型風(fēng)險(xiǎn)管理

銀行組織應(yīng)注意基于不正確或?yàn)E用模型的決策可能產(chǎn)生的不利后果（包括財(cái)務(wù)損失），并應(yīng)通過(guò)積極的模型風(fēng)險(xiǎn)管理來(lái)解決這些后果。此SR 信的附件更詳細(xì)地描述了有效模型風(fēng)險(xiǎn)管理框架的關(guān)鍵方面，包括穩(wěn)健的模型開(kāi)發(fā)、實(shí)施和使用；有效的驗(yàn)證；健全的治理、政策和控制。

美聯(lián)儲(chǔ)和 OCC 之前發(fā)布的出版物已經(jīng)討論了模型的使用，特別關(guān)注模型驗(yàn)證。1基于過(guò)去幾年的監(jiān)管和行業(yè)經(jīng)驗(yàn)，本文件擴(kuò)展了現(xiàn)有指南——最重要的是擴(kuò)大范圍以包括模型風(fēng)險(xiǎn)管理的其他關(guān)鍵方面。

就本文檔而言，術(shù)語(yǔ)模型是指應(yīng)用統(tǒng)計(jì)、經(jīng)濟(jì)、金融或數(shù)學(xué)理論、技術(shù)和假設(shè)將輸入數(shù)據(jù)處理為定量估計(jì)的定量方法、系統(tǒng)或方法。符合此定義的模型可用于分析業(yè)務(wù)戰(zhàn)略、為業(yè)務(wù)決策提供信息、識(shí)別和衡量風(fēng)險(xiǎn)、評(píng)估風(fēng)險(xiǎn)敞口、工具或頭寸、進(jìn)行壓力測(cè)試、評(píng)估資本充足性、管理客戶資產(chǎn)、衡量?jī)?nèi)部限制的合規(guī)性、維護(hù)正式銀行的控制設(shè)備，或滿足財(cái)務(wù)或監(jiān)管報(bào)告要求并發(fā)布公開(kāi)披露信息。模型的定義還包括定量方法，其輸入部分或全部是定性的或基于專家判斷，前提是輸出是定量的。2

模型的使用總是會(huì)帶來(lái)模型風(fēng)險(xiǎn)，即基于不正確或?yàn)E用模型輸出和報(bào)告的決策可能產(chǎn)生不利后果。模型風(fēng)險(xiǎn)可能導(dǎo)致財(cái)務(wù)損失、糟糕的業(yè)務(wù)和戰(zhàn)略決策，或損害銀行組織的聲譽(yù)。模型風(fēng)險(xiǎn)的發(fā)生主要有兩個(gè)原因：??(1)模型可能存在根本性錯(cuò)誤并在與其設(shè)計(jì)目標(biāo)和預(yù)期業(yè)務(wù)用途相悖時(shí)產(chǎn)生不準(zhǔn)確的輸出；(2) 一個(gè)模型可能被錯(cuò)誤或不恰當(dāng)?shù)厥褂?，或者可能?duì)其局限性和假設(shè)存在誤解。模型風(fēng)險(xiǎn)隨著模型復(fù)雜性的增加、輸入和假設(shè)的更高不確定性、更廣泛的使用范圍和更大的潛在影響而增加。銀行組織應(yīng)該從單個(gè)模型和整體模型中管理模型風(fēng)險(xiǎn)。

貫穿整個(gè)指南的指導(dǎo)原則是管理模型風(fēng)險(xiǎn)涉及模型的“有效挑戰(zhàn)”：由客觀、知情的各方進(jìn)行批判性分析，可以識(shí)別模型限制并產(chǎn)生適當(dāng)?shù)淖兓?/span>有效的挑戰(zhàn)取決于激勵(lì)、能力和影響力的結(jié)合。

與其他風(fēng)險(xiǎn)一樣，重要性是模型風(fēng)險(xiǎn)管理中的一個(gè)重要考慮因素。如果在某些銀行，模型的使用不那么普遍并且對(duì)其財(cái)務(wù)狀況的影響較小，那么這些銀行可能不需要如此復(fù)雜的模型風(fēng)險(xiǎn)管理方法來(lái)滿足監(jiān)管預(yù)期。但是，如果模型和模型輸出對(duì)業(yè)務(wù)決策（包括與風(fēng)險(xiǎn)管理和資本和流動(dòng)性規(guī)劃相關(guān)的決策）產(chǎn)生重大影響，并且模型失效會(huì)對(duì)銀行的財(cái)務(wù)狀況產(chǎn)生特別有害的影響，銀行的模型風(fēng)險(xiǎn)管理框架應(yīng)該更加廣泛和嚴(yán)謹(jǐn)。

模型開(kāi)發(fā)、實(shí)施和使用

模型開(kāi)發(fā)很大程度上依賴于開(kāi)發(fā)人員的經(jīng)驗(yàn)和判斷，模型風(fēng)險(xiǎn)管理應(yīng)包括與模型用戶的情況和目標(biāo)以及銀行組織政策相一致的規(guī)范的模型開(kāi)發(fā)和實(shí)施過(guò)程。完善的開(kāi)發(fā)過(guò)程包括：明確的目的聲明，以確保模型的開(kāi)發(fā)符合其預(yù)期用途；模型背后的合理設(shè)計(jì)、理論和邏輯；穩(wěn)健的模型方法和處理組件；嚴(yán)格評(píng)估數(shù)據(jù)質(zhì)量和相關(guān)性；和適當(dāng)?shù)奈募?/span>模型開(kāi)發(fā)的一個(gè)組成部分是測(cè)試，其中評(píng)估模型的各個(gè)組件及其整體功能，以顯示模型按預(yù)期執(zhí)行；證明它是準(zhǔn)確、穩(wěn)健和穩(wěn)定的；并評(píng)估其局限性和假設(shè)。重要的是，組織應(yīng)確保其模型更具判斷性和定性的方面的開(kāi)發(fā)也是合理的。

所有模型都有一定程度的不確定性和不準(zhǔn)確性，因?yàn)樗鼈兏鶕?jù)定義是對(duì)現(xiàn)實(shí)的不完美表示。有效模型開(kāi)發(fā)、實(shí)施和使用的一個(gè)重要結(jié)果是銀行組織對(duì)這種不確定性的理解和解釋。對(duì)模型不確定性的考慮可以包括對(duì)模型輸出應(yīng)用有充分支持的、判斷性的、“保守的”調(diào)整，對(duì)模型輸出的重視程度較低，或確保僅在有其他模型或方法補(bǔ)充時(shí)才使用模型。3

模型驗(yàn)證

模型驗(yàn)證是一組過(guò)程和活動(dòng)，旨在驗(yàn)證模型是否按預(yù)期執(zhí)行，符合其設(shè)計(jì)目標(biāo)和業(yè)務(wù)用途。有效的驗(yàn)證有助于確保模型是合理的，識(shí)別潛在的限制和假設(shè)并評(píng)估它們可能產(chǎn)生的影響。所有模型組件——輸入、處理、輸出和報(bào)告——都應(yīng)經(jīng)過(guò)驗(yàn)證；這同樣適用于內(nèi)部開(kāi)發(fā)的模型以及從供應(yīng)商或顧問(wèn)處購(gòu)買(mǎi)或開(kāi)發(fā)的模型。

驗(yàn)證涉及在一定程度上獨(dú)立于模型開(kāi)發(fā)和使用。通常，驗(yàn)證由不負(fù)責(zé)模型開(kāi)發(fā)或使用的人員完成，并且與模型是否被確定為有效無(wú)關(guān)。實(shí)際上，一些驗(yàn)證工作可能由模型開(kāi)發(fā)人員和用戶最有效地完成；然而，此類驗(yàn)證工作必須接受獨(dú)立方的嚴(yán)格審查，該獨(dú)立方應(yīng)開(kāi)展額外活動(dòng)以確保適當(dāng)驗(yàn)證。總體而言，驗(yàn)證過(guò)程的質(zhì)量由客觀、知識(shí)淵博的各方的嚴(yán)格審查以及為解決這些各方確定的問(wèn)題所采取的行動(dòng)來(lái)表明。

在模型投入使用后，驗(yàn)證活動(dòng)應(yīng)持續(xù)進(jìn)行，以跟蹤已知模型限制并識(shí)別任何新限制。在經(jīng)濟(jì)和金融狀況良好的時(shí)期，驗(yàn)證是一項(xiàng)重要的檢查，此時(shí)風(fēng)險(xiǎn)和潛在損失的估計(jì)可能變得過(guò)于樂(lè)觀，手頭的數(shù)據(jù)可能無(wú)法完全反映壓力更大的狀況。銀行組織應(yīng)該對(duì)每個(gè)模型進(jìn)行定期審查——至少每年一次，但如果有必要，可以更頻繁地審查，以確定它是否按預(yù)期工作以及現(xiàn)有的驗(yàn)證活動(dòng)是否足夠。全面驗(yàn)證的關(guān)鍵要素包括：

• 概念合理性評(píng)估。該要素涉及評(píng)估模型設(shè)計(jì)和構(gòu)建的質(zhì)量，以及審查支持模型所用方法和所選變量的文檔和經(jīng)驗(yàn)證據(jù)。驗(yàn)證的這一步驟應(yīng)確保在模型設(shè)計(jì)和構(gòu)建中做出的判斷是知情的、經(jīng)過(guò)仔細(xì)考慮的，并與已發(fā)表的研究和良好的行業(yè)實(shí)踐相一致。

• 持續(xù)監(jiān)測(cè)。驗(yàn)證中的這一步是為了確認(rèn)模型得到了適當(dāng)?shù)膶?shí)施，并且正在按預(yù)期使用和執(zhí)行。必須評(píng)估產(chǎn)品、風(fēng)險(xiǎn)敞口、活動(dòng)、客戶或市場(chǎng)條件的變化是否需要調(diào)整、重新開(kāi)發(fā)或更換模型，并驗(yàn)證模型超出其原始范圍的任何擴(kuò)展是否有效。在此步驟中可以使用基準(zhǔn)測(cè)試將給定模型的輸入和輸出與備選方案的估計(jì)值進(jìn)行比較。

• 結(jié)果分析。此步驟涉及將模型輸出與相應(yīng)的實(shí)際結(jié)果進(jìn)行比較。回溯測(cè)試是結(jié)果分析的一種形式，它涉及在模型開(kāi)發(fā)中未使用的樣本時(shí)間段內(nèi)以與模型的預(yù)測(cè)范圍或性能窗口相匹配的頻率將實(shí)際結(jié)果與模型預(yù)測(cè)進(jìn)行比較。

驗(yàn)證過(guò)程的三個(gè)核心要素的結(jié)果可能會(huì)揭示模型開(kāi)發(fā)中的重大錯(cuò)誤或不準(zhǔn)確之處或始終超出銀行組織預(yù)定可接受閾值的結(jié)果。在這種情況下，需要進(jìn)行模型調(diào)整、重新校準(zhǔn)或重新開(kāi)發(fā)。有時(shí)，由于缺乏數(shù)據(jù)或價(jià)格可觀察性等各種原因，銀行組織可能無(wú)法使用關(guān)鍵模型驗(yàn)證工具。在這種情況下，在考慮模型使用的適當(dāng)性時(shí)，更應(yīng)注意模型的局限性，并在使用模型進(jìn)行決策時(shí)充分告知高級(jí)管理層這些局限性。一般來(lái)說(shuō)，高級(jí)管理層應(yīng)確保根據(jù)已識(shí)別的模型限制采取適當(dāng)?shù)木徑獯胧?/span>

治理、政策和控制

對(duì)模型風(fēng)險(xiǎn)管理框架開(kāi)發(fā)和維護(hù)強(qiáng)有力的治理對(duì)其有效性至關(guān)重要。強(qiáng)有力的治理通過(guò)定義相關(guān)風(fēng)險(xiǎn)管理活動(dòng)的政策、實(shí)施這些政策的程序、資源分配以及測(cè)試政策和程序是否按規(guī)定執(zhí)行的機(jī)制，為風(fēng)險(xiǎn)管理職能提供明確的支持和結(jié)構(gòu)。強(qiáng)有力的治理還包括足夠詳細(xì)的模型開(kāi)發(fā)和驗(yàn)證文檔，以使不熟悉模型的各方能夠了解模型的運(yùn)行方式及其局限性和關(guān)鍵假設(shè)。

董事會(huì)和高級(jí)管理層在建立全組織范圍的模型風(fēng)險(xiǎn)管理方法時(shí)，在最高級(jí)別提供模型風(fēng)險(xiǎn)治理。董事會(huì)成員應(yīng)確保模型風(fēng)險(xiǎn)水平在他們的容忍范圍內(nèi)。銀行組織的內(nèi)部審計(jì)職能應(yīng)評(píng)估模型風(fēng)險(xiǎn)管理框架的整體有效性，包括框架處理單個(gè)模型和總體模型風(fēng)險(xiǎn)的能力。每當(dāng)銀行組織使用外部資源進(jìn)行模型風(fēng)險(xiǎn)管理時(shí)，組織應(yīng)在明確書(shū)面和商定的工作范圍中指定要開(kāi)展的活動(dòng)，并且這些活動(dòng)應(yīng)按照本指南進(jìn)行。此外，組織應(yīng)維護(hù)已實(shí)施的模型清單以供使用，

所有銀行組織都應(yīng)確保其內(nèi)部政策和程序與本指南中包含的風(fēng)險(xiǎn)管理原則和監(jiān)管預(yù)期相一致。

聯(lián)系人

有關(guān)本指南的問(wèn)題，請(qǐng)致電(202) 452-2904聯(lián)系高級(jí)金融監(jiān)管分析師David Palmer ；Dwight Smith，資本與監(jiān)管政策高級(jí)監(jiān)管金融分析師，電話：(202) 452-2773；或助理主任Anna Lee Hewko，電話(202) 530-6260。此外，可以通過(guò)委員會(huì)的公共網(wǎng)站發(fā)送問(wèn)題。4

Patrick M. Parkinson?銀行?監(jiān)管
司司長(zhǎng)簽字

附件：

模型風(fēng)險(xiǎn)管理指南(PDF)

交叉參考：

• SR 09-1，“市場(chǎng)風(fēng)險(xiǎn)規(guī)則在銀行控股公司和國(guó)有銀行中的應(yīng)用”

筆記：

1. 例如，OCC 在OCC 2000-16?（2000 年 5 月 30 日）、其他公告和主計(jì)長(zhǎng)手冊(cè)的某些主題小冊(cè)子中提供了關(guān)于模型風(fēng)險(xiǎn)的指導(dǎo)，重點(diǎn)是模型驗(yàn)證。美聯(lián)儲(chǔ)發(fā)布了SR 信 09-01，?“市場(chǎng)風(fēng)險(xiǎn)規(guī)則在銀行控股公司和國(guó)有銀行中的應(yīng)用”，其中強(qiáng)調(diào)了與模型風(fēng)險(xiǎn)管理相關(guān)的各種概念，包括驗(yàn)證和審查標(biāo)準(zhǔn)、模型驗(yàn)證文檔和回-測(cè)試。美聯(lián)儲(chǔ)的交易和資本市場(chǎng)活動(dòng)手冊(cè)也討論了驗(yàn)證和模型風(fēng)險(xiǎn)管理。此外，高級(jí)方法基于風(fēng)險(xiǎn)的資本規(guī)則（12 CFR 3，?附錄 C；?12 CFR 208，?附錄 F；和12 CFR 225，?附錄 G)包含針對(duì)主題銀行組織的明確驗(yàn)證要求。??

2. 雖然不在本指南的范圍內(nèi)，但銀行組織使用的更多定性方法（即那些未根據(jù)本指南定義為模型的方法）也應(yīng)接受?chē)?yán)格的控制過(guò)程。??

3. 如果模型用于生成包含在公共財(cái)務(wù)報(bào)表中的金額，則對(duì)模型不確定性的任何調(diào)整都必須符合公認(rèn)的會(huì)計(jì)原則。??

4. 見(jiàn)http://www.federalreserve.gov/feedback.cfm。??

TO THE OFFICER IN CHARGE OF SUPERVISION AND APPROPRIATE SUPERVISORY AND EXAMINATION STAFF AT EACH FEDERAL RESERVE BANK

SUBJECT:

Guidance on Model Risk Management

The Federal Reserve and Office of the Comptroller of the Currency (OCC) are issuing the attached Supervisory Guidance on Model Risk Management, which is intended for use by banking organizations and supervisors as they assess organizations’ management of model risk. This guidance should be applied as appropriate to all banking organizations supervised by the Federal Reserve, taking into account each organization’s size, nature, and complexity, as well as the extent and sophistication of its use of models (as defined and discussed below).

Model Risk Management

Banking organizations should be attentive to the possible adverse consequences (including financial loss) of decisions based on models that are incorrect or misused, and should address those consequences through active model risk management. The attachment to this SR letter describes in more detail the key aspects of an effective model risk management framework, including robust model development, implementation, and use; effective validation; and sound governance, policies, and controls.

Previous publications issued by the Federal Reserve and OCC have addressed the use of models, with particular focus on model validation.1 Based on supervisory and industry experience over the past several years, this document expands upon existing guidance—most importantly by broadening the scope to include other key aspects of model risk management.

For the purposes of this document, the term model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates. Models meeting this definition might be used for analyzing business strategies, informing business decisions, identifying and measuring risks, valuing exposures, instruments or positions, conducting stress testing, assessing adequacy of capital, managing client assets, measuring compliance with internal limits, maintaining the formal control apparatus of the bank, or meeting financial or regulatory reporting requirements and issuing public disclosures. The definition of model also covers quantitative approaches whose inputs are partially or wholly qualitative or based on expert judgment, provided that the output is quantitative in nature.2

The use of models invariably presents model risk, which is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. Model risk can lead to financial loss, poor business and strategic decision-making, or damage to a banking organization’s reputation. Model risk occurs primarily for two reasons:? (1) a model may have fundamental errors and produce inaccurate outputs when viewed against its design objective and intended business uses; (2) a model may be used incorrectly or inappropriately or there may be a misunderstanding about its limitations and assumptions. Model risk increases with greater model complexity, higher uncertainty about inputs and assumptions, broader extent of use, and larger potential impact.? Banking organizations should manage model risk both from individual models and in the aggregate.

A guiding principle throughout the guidance is that managing model risk involves "effective challenge" of models:? critical analysis by objective, informed parties that can identify model limitations and produce appropriate changes. Effective challenge depends on a combination of incentives, competence, and influence.

As is generally the case with other risks, materiality is an important consideration in model risk management. If at some banks the use of models is less pervasive and has less impact on their financial condition, then those banks may not need as complex an approach to model risk management in order to meet supervisory expectations. However, where models and model output have a material impact on business decisions, including decisions related to risk management and capital and liquidity planning, and where model failure would have a particularly harmful impact on a bank’s financial condition, a bank’s model risk management framework should be more extensive and rigorous.

Model Development, Implementation, and Use

Model development relies heavily on the experience and judgment of developers, and model risk management should include disciplined model development and implementation processes that are consistent with the situation and goals of the model user and with the banking organization’s policy. A sound development process includes:? a clear statement of purpose to ensure that the model is developed in line with its intended use; sound design, theory, and logic underlying the model; robust model methodologies and processing components; rigorous assessment of data quality and relevance; and appropriate documentation. An integral part of model development is testing, in which the various components of a model and its overall functioning are evaluated to show the model is performing as intended; to demonstrate that it is accurate, robust, and stable; and to evaluate its limitations and assumptions. Importantly, organizations should ensure that the development of the more judgmental and qualitative aspects of their models is also sound.

All models have some degree of uncertainty and inaccuracy because they are by definition imperfect representations of reality. An important outcome of effective model development, implementation, and use is a banking organization’s demonstrated understanding of and accounting for such uncertainty. Accounting for model uncertainty can include applying well-supported, judgmental, “conservative” adjustments to model output, placing less emphasis on a model’s output, or ensuring that a model is only used when supplemented by other models or approaches.3

Model Validation

Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses. Effective validation helps to ensure that models are sound, identifying potential limitations and assumptions and assessing their possible impact. All model components—inputs, processing, outputs, and reports—should be subject to validation; this applies equally to models developed in-house and to those purchased from or developed by vendors or consultants.

Validation involves a degree of independence from model development and use. Generally, validation is done by staff who are not responsible for model development or use and do not have a stake in whether a model is determined to be valid. As a practical matter, some validation work may be most effectively done by model developers and users; it is essential, however, that such validation work be subject to critical review by an independent party, who should conduct additional activities to ensure proper validation. Overall, the quality of the validation process is indicated by critical review by objective, knowledgeable parties and the actions taken to address issues identified by those parties.

Validation activities should continue on an ongoing basis after a model goes into use to track known model limitations and to identify any new ones. Validation is an important check during periods of benign economic and financial conditions, when estimates of risk and potential loss can become overly optimistic and the data at hand may not fully reflect more stressed conditions. Banking organizations should conduct a periodic review—at least annually but more frequently if warranted—of each model to determine whether it is working as intended and if the existing validation activities are sufficient. Key elements of comprehensive validation include:

Evaluation of Conceptual Soundness. This element involves assessing the quality of the model design and construction, as well as review of documentation and empirical evidence supporting the methods used and variables selected for the model. This step in validation should ensure that judgment exercised in model design and construction is well informed, carefully considered, and consistent with published research and with sound industry practice.

Ongoing Monitoring. This step in validation is done to confirm that the model is appropriately implemented and is being used and performing as intended.? It is essential to evaluate whether changes in products, exposures, activities, clients, or market conditions necessitate adjustment, redevelopment, or replacement of the model and to verify that any extension of the model beyond its original scope is valid. Benchmarking can be used in this step to compare a given model’s inputs and outputs to estimates from alternatives.

Outcomes Analysis. This step involves comparing model outputs to corresponding actual outcomes. Back-testing is one form of outcomes analysis that involves the comparison of actual outcomes with model forecasts during a sample time period not used in model development at a frequency that matches the model’s forecast horizon or performance window.

The results of the three core elements of the validation process may reveal significant errors or inaccuracies in model development or outcomes that consistently fall outside the banking organization’s predetermined thresholds of acceptability. In such cases, model adjustment, recalibration, or redevelopment is warranted. At times, banking organizations may have a limited ability to use key model validation tools for various reasons, such as lack of data or of price observability. In those cases, even more attention should be paid to the model’s limitations when considering the appropriateness of model usage, and senior management should be fully informed of those limitations when using the models for decision-making. Generally, senior management should ensure that appropriate mitigating steps are taken in light of identified model limitations, which can include adjustments to model output, restrictions on model use, reliance on other models or approaches, or other compensating controls

Governance, Policies, and Controls

Developing and maintaining strong governance over the model risk management framework is fundamentally important to its effectiveness. Strong governance provides explicit support and structure to risk management functions through policies defining relevant risk management activities, procedures that implement those policies, allocation of resources, and mechanisms for testing that policies and procedures are being carried out as specified. Strong governance also includes documentation of model development and validation that is sufficiently detailed to allow parties unfamiliar with a model to understand how the model operates, as well as its limitations and key assumptions.

Model risk governance is provided at the highest level by the board of directors and senior management when they establish an organization-wide approach to model risk management. Board members should ensure that the level of model risk is within their tolerance. A banking organization’s internal audit function should assess the overall effectiveness of the model risk management framework, including the framework’s ability to address both types of model risk for individual models and in the aggregate. Whenever a banking organization uses external resources for model risk management, the organization should specify the activities to be conducted in a clearly written and agreed-upon scope of work, and those activities should be conducted in accordance with this guidance. Also, organizations should maintain an inventory of models implemented for use, under development for implementation, or recently retired.

All banking organizations should ensure that their internal policies and procedures are consistent with the risk management principles and supervisory expectations contained in this guidance.

Contacts

For questions regarding this guidance, please contact David Palmer, Senior Supervisory Financial Analyst, Risk, at (202) 452-2904; Dwight Smith, Senior Supervisory Financial Analyst, Capital & Regulatory Policy, at (202) 452-2773; or Anna Lee Hewko, Assistant Director, at (202) 530-6260. In addition, questions may be sent via the Board’s public website.4

signed by

Patrick M. Parkinson

Director

Division of Banking

Supervision and Regulation

Attachments:

Model Risk Management Guidance (PDF)

Cross References:

SR 09-1, "Application of the Market Risk Rule in Bank Holding Companies and State Member Banks"

Notes:

For instance, the OCC provided guidance on model risk, focusing on model validation, in OCC 2000-16 (May 30, 2000), other bulletins, and certain subject matter booklets of the Comptroller’s Handbook. The Federal Reserve issued SR Letter 09-01, “Application of the Market Risk Rule in Bank Holding Companies and State Member Banks,” which highlights various concepts pertinent to model risk management, including standards for validation and review, model validation documentation, and back-testing. The Federal Reserve’s Trading and Capital-Markets Activities Manual also discusses validation and model risk management. In addition, the advanced-approaches risk-based capital rules (12 CFR 3, Appendix C; 12 CFR 208, Appendix F; and 12 CFR 225, Appendix G) contain explicit validation requirements for subject banking organizations.? Return to text

While outside the scope of this guidance, more qualitative approaches used by banking organizations—i.e., those not defined as models according to this guidance—should also be subject to a rigorous control process.? Return to text

To the extent that models are used to generate amounts included in public financial statements, any adjustments for model uncertainty must comply with generally accepted accounting principles.? Return to text

See http://www.federalreserve.gov/feedback.cfm.??