Welcome to our third blog post on how artificial intelligence (AI) and other machine learning techniques can help power country risk analysis. 

In our first post, we focused on sentiment analysis and considered a range of options for making algorithms understand country risk nuances (Link). Our second post focused on how AI can make country risk analysts more productive by combining quantitative risk scores and natural language generation to create risk reports (Link). 

Now, we’re going to explore the broader category of knowledge management by showing you how to make an large language model (LLM) understand your knowledge domain. And although we at CountryRisk.io are primarily interested in managing knowledge around country and sovereign risk analysis, you can apply the same principles to any other sphere of knowledge. 

From General to Specific

Large language models (LLM) such as ChatGPT are trained on vast amounts of online information, such as Wikipedia articles, blogs and discussion forums, and many other sources. And they can aggregate all of that information into a single response more or less instantly, which is what makes these models so powerful. 

However, LLMs are designed to handle general knowledge, and so using them in more narrow domains reveals some significant limitations. Even country risk, which is quite obviously the most important domain of knowledge in the world, is just one of many such domains—and an LLM like ChatGPT is trained on all of them. In short: asking a standard LLM about country risk is like asking a jack of all trades about the nuances of quantum mechanics—fine for the basics, but if you want to get into the weeds, you should probably speak to an actual physicist.

To avoid ending up in the wrong corner of an LLM, you basically have two options. 

First, you can build your own LLM and train it exclusively on the domain knowledge of your choice, as Bloomberg did with BloombergGPT. However, this resource-intensive process requires the enormous amounts of computational power, expertise, and training content you usually only find in large companies. In the future, new service offerings from the likes of AWS could make this a practical option for smaller companies. But for now, a bespoke LLM is off the table for companies like CountryRisk.io. 

The second, more realistic option is to guide your generalist LLM to the right content area. Or, to stick with our earlier analogy, before asking our jack of all trades about quantum mechanics, we first help them brush up on their physics. This is known as prompt engineering, and we’ll focus on this approach in what follows. If you want to learn even more about prompt engineering, promptingguide.ai is an excellent resource.

Can an LLM give good advice on economic policy? 

To get to grips with prompt engineering in the context of country risk, let’s look at the use case of economic policy advice and see whether this approach can get an LLM to give us good answers on what authorities or governments should do when facing certain risk scenarios. As an example, we’ll use OpenAI’s GPT-3.5 (text-davinci-003) model and start with the following simple query: “How to react in an economic recession?” Here’s the advice we got: 

------------------------------------------

* Cut back on unnecessary spending.

* Consider refinancing debt or taking out a loan with a lower interest rate.

* Look for ways to increase income or reduce expenses.

* Invest in assets that can retain or increase value over time.

* Consider different career options and industries that may be more recession-resistant.

* Build an emergency fund.

* Utilize government resources and services.

------------------------------------------

If we’re being charitable, we might say that all makes sense. However, this answer is clearly more appropriate for a company or individual than the countries or governments we’re interested in. To get the perspective we’re after, then, we’ll need to guide the LLM by providing some relevant information.  

As you might have guessed from the title of this blog post, we’ll use the IMF as our source for policy advice, the provision of which is one of the Fund’s primary objectives. To be clear, this is not to say that the IMF is the definite source of policy advice—the point of this exercise is to see whether we can use prompt engineering to get an LLM to only reflect a specific body of knowledge in its answers. So, although we’ve chosen the IMF’s knowledge in this case, you can apply the same principles to any body of knowledge you choose. 

Unfortunately, as there’s no single IMF document that neatly outlines all the organisation’s policy recommendations, we’ll have to be a bit creative. The IMF’s annual Article IV country reports review a specific country’s macro-economic performance, identify risk areas, and provide recommendations. We’ll use the Risk Assessment Matrices from various Article IV reports as our knowledge base for policy recommendations. The screenshot below shows an example, which we took from the latest Article IV report on Tajikistan.

In total, we collected around 1,500 cases of policy advice across countries and years. Before using the content, we cleaned it by removing bullet formatting and generalised it by removing references to specific countries or organisations. 

Narrowing down the knowledge

We can’t include our entire knowledge base in our query, because doing so would make each query quite costly—plus, like any LLM, GPT-3.5 has some length constraints (around 4,000 or 8000 tokens for the latest version). So, before we can ask questions using our domain-specific knowledge, we need to condense it.

To extract the knowledge that’s most relevant to our question, we start by calculating the semantic similarity between the query and, in this case, the IMF’s risk definition. We then take the indices of the top n results and extract the corresponding policy advice statements until we hit a certain number of tokens. You can read more about the technical details here or here.

Using this, we engineered the following prompt statement for our question:

------------------------------------------

Instructions: Answer the question as truthfully as possible using the provided context, and if the answer is not contained within the text below, say “I don’t know.” Answer in bullet points.

Context:
* Prioritize expenditure to the affected region. If the economy slows significantly, ease monetary policy, provide support to banks, and monitor corporate borrowers at risk.
* Make investments to increase resilience of financial intermediation. Adjust the pace of planned monetary tightening commensurately according to the assessed downturn in activity.
* Diversify sources of supply and recalibrate the pace of withdrawal of monetary and fiscal support in event of significant impact on activity.
* Maintain a decisively counter-cyclical policy stance. Gradually tighten fiscal and financial policies while maintaining highly targeted support to vulnerable pockets of the economy.
* Reinforce automatic stabilizers and act with countercyclical policies in case of an unexpected setback in the recovery of economic activity.
* Prioritize expenditure to the affected region. If the economy slows significantly, ease monetary and macroprudential policies, provide support to banks, and monitor corporate borrowers at risk.
* Both monetary and fiscal policy would need to respond to a global recession. The strength of the reaction would depend on developments in inflation and commodity prices.
* Develop a contingency plan that would lower the impact of a delayed economic recovery: Consider further fiscal consolidation and reliance on concessional financing in case such risk materializes.
* Maintain a decisively countercyclical policy stance. Gradually tighten fiscal and financial policies while maintaining highly targeted support to vulnerable pockets of the economy.
* Allow for a well-timed withdrawal of support measures. Adjust policies based on pandemic and economic developments. Target assistance to households and firms most severely affected by the crisis.
* Prioritize expenditure to support the affected households. If the economic impact is large, provide fiscal stimulus to boost aggregate demand. Maintain easy monetary policy while strengthening the monitoring of potential debt service problems in the banking sector.
* Use fiscal space to counter the effects of the slowdown and protect the vulnerable firms and households. Accelerate structural agenda enhancing competitiveness and diversify the economy to build resilience against external shocks. Enhance regional integration. Consider more flexibility in exchange rate response to external shocks.
* If market conditions become disorderly, consider measures to stew capital flow reversals.  Maintain prudent macroeconomic policies including structural reforms. Promote inclusive growth and strengthen social safety nets.
* Stand ready to tighten monetary policy if needed. Work with private sector to improve resilience of supply chains. In terms of fiscal policy, ensure social support programs are sufficient to support those who may lose access to essential goods like food and medicine.
* Adjust monetary policy as needed to keep inflation expectations anchored. Provide liquidity support to the financial sector as needed. Use available fiscal space to cushion the economy against a recession.
* Support the vulnerable population with well-targeted measures. Assuming the turbulence is temporary, use fiscal space and monetary policy prudently to support the economy.
* Preparedness in the immediate aftermath and building resilience in economic activities and infrastructure would reduce the medium-term impact. Countercyclical policies should be deployed as needed.
* Preparedness in the immediate aftermath and building resilience in economic activities and infrastructure would reduce the medium-term impact. Countercyclical policies should be deployed as needed.
* Expand the reach of anti-crisis fiscal measures to support vulnerable households, the unemployed, and distressed firms. If warranted, introduce new measures. Loosen monetary policy further than in the baseline to help support the economic recovery. Disorderly market conditions could be countered with foreign exchange intervention.
* Tighten monetary policy if inflation in the country is affected by additional price pressures but decouple from Fed when appropriate. Frontload fiscal expenditure plans and delay revenue measures to provide support during the downturn without adding to domestic price pressures.
* Targeted assistance to affected groups and sectors. If the economy slows significantly, provide relief to banks in affected regions, while monitoring credit risk. Prioritize public investment in disaster-resistant infrastructure and sustainable growth.
* Allow automatic stabilizers to operate if the output gap changes. In case of a rapid unwinding of savings, withdraw fiscal support faster to prevent overheating and rebuild fiscal space, and monitor closely developments in the real estate market to assess the need for macroprudential tightening.
* Continue to restrain current spending and boost domestic demand through reallocating spending toward infrastructure investments. Strengthen banking regulation and maintain financial stability to weather external shocks.
* In the short-term, continue to assess potential second-round effects on inflation and inflation expectations and adjust the monetary policy stance if needed. Stand ready to use fiscal space to mitigate social impact of commodity price shocks. Continue to strengthen social programs for transfers for those most affected by reforms.

Q: How to react in an economic recession?

A:

------------------------------------------

This prompt consists of three sections:

1. Instructions: We prompt the model on what to do and how to answer.
2. Context: The most relevant policy advice statements from the IMF matching our question.
3. Question: The actual question that we want to ask.  

What would the IMF recommend?

Here’s the response we got to our engineered prompt: 

------------------------------------------

Q: How to react in an economic recession?

A: 

* Ease monetary policy

* Provide support to banks

* Monitor corporate borrowers at risk

* Adjust the pace of planned monetary tightening commensurately

* Maintain a countercyclical policy stance

* Target assistance to households and firms most affected

* Use fiscal space to counter effects of slowdown

* Provide fiscal stimulus to boost aggregate demand

* Tighten monetary policy if inflation is affected

* Frontload fiscal expenditure plans

* Targeted assistance to affected groups and sectors

* Reallocate spending toward infrastructure investments

* Strengthen banking regulation

 ------------------------------------------

Compared to the response we got without providing any domain knowledge, we now have an answer that makes a lot more sense from a country risk perspective. Even so, we could further improve the context by enlarging or filtering the knowledge base. For example, we might tag each policy statement to indicate the countries to which the statement is relevant according to different groupings. For example, if we were to tag policy statements according to whether they’re relevant to high-, middle-, or low-income countries, we’d be able to get the LLM to provide policy advice that’s more tailored to the country we’re interested in—after all, you’d handle a Swedish recession differently to a Ghanaian recession.

Use cases 

At CountryRisk.io, our mission is to support our community members in making better country risk decisions while increasing their productivity. To that end, you can apply the principles of prompt engineering we’ve explored in this post to a range of use cases. For instance:

• Policymakers in ministries of finance, debt management offices, or central banks can use this approach as a starting point for policy design. Of course, this isn’t to suggest that you could or, indeed, should hand policymaking over to an AI. But LLMs do have the potential to help policymakers be more productive by enabling them to quickly find and aggregate the information they need amidst an ever-growing knowledge base. 
• Sovereign rating advisors could use this approach to tap into the knowledge base of their choosing—IMF or otherwise. Whatever their personal views on how to improve a sovereign’s credit profile, sovereign risk advisors can create their own in-house research tool—and even make it available for their clients—by maintain their own knowledge base and using an LLM to mine it quickly and easily. 
• Country risk analysts can use this approach to conduct a risk assessment by asking something along the lines of, “Is this country following this advice?”

At CountryRisk.io, we’re excited about the knowledge management possibilities LLMs are creating, and we look forward to developing new, LLM-powered features for our platform and API in the future.