Receipt recognition with Azure

We are constantly looking for ways to help you get the most out of your data. Our customer ask us a POC to recognize information from receipts.

Expense reports can be a very cumbersome and time-consuming task. Between all the manual data entry, approval workflows, and auditing, there are many pain points across the end-to-end process. With the you can minimize those pain points and increase the productivity of your employees, delivering real value back to your business.

Receipt processing lets you read and save key information from common sales receipts, like those used in restaurants, gas stations, retail, and more. Using this information, you can automatically pre-populate expense reports simply by scanning photos of your receipts. And when you automate the process at a large scale, there is the potential to save you and your business valuable time and money.

The prebuilt model uses state-of-the-art optical character recognition (OCR) to extract both printed and handwritten text from receipts. You can retrieve valuable information such as the merchant details, transaction date and time, list of purchased items, tax, and totals.

No training or prior configuration is required to use this prebuilt model. Start processing receipts right away in your apps and flows using the new canvas app component and AI Builder flow action.

Text translation

You can now use AI Builder to easily translate text to more than 60 languages. This prebuilt model is powered by the latest innovations in machine translation. You can use Text translation to process text in real-time from different languages from your customers worldwide, for internal and external communications and to keep language consistency in the text data that you store. Now available in preview, no trial or subscription required to try this feature.

Building a Text Classifier using Azure Machine Learning

Recently a client came to us to see if we could help them automate their RFP distribution system.  Currently the client has an employee manually check several websites for RFPs and alert the appropriate business vertical when a relevant RFP is found.  The current system requires manual data scraping, meaning the process is slow and results in RFPs being missed.  For the proof of concept phase with the client, we decided to build a machine learning model to classify the RFPs correctly and provide a way to automate the routing of the RFPs.  The client wanted to break the project into stages so once the initial Proof of Concept was successful, other parts required to automate the whole process would receive the go-ahead. If you would like a proof of concept, visit our Business Analytics page for information.

Due to the abbreviated time-period, we decided to use Microsoft’s Azure Machine Learning Studio to build the model.  Azure Machine Learning Studio provided great visualizations of the model for the client.  When developing an end-to-end solution for the client, Azure Machine Learning Service will be implemented.  If you are curious about the differences between Azure ML Studio and Azure ML Service, this article provides an excellent explanation. 


First I looked through the Azure AI Gallery to see if there were any projects that would provide guidance in building our text classifier. I found the BBC News Classifier was a great fit. 


Model Evaluation – Confusion Matrix:  


If the model is built correctly, one should see distribution like what is shown above.  The model assigns a probability per category to reflect its confidence in how to categorize each story.  It is normal for a news story to be classified in one main class, but the model recognizes there is a probability that the story could belong to multiple classes. 

The metrics from the model also showed good accuracy on the model. 


Step 1: Receiving and cleaning the data. 

The client uploaded several RFPs into different folders in Teams that were labeled with the client’s verticals.  One of the challenges not solved in this POC is scraping the data from an RFP.  Our focus was on starting small with the classifier to keep things moving forward. Every municipality creates their own version of an RFP, so most RFPs are not uniform.  For this POC, the RFP summary data was scraped manually and added to a data file. 

Step 2: Create the Model 

To start, we followed the BBC News Classifier model outline.  The R-Script module and the found in the BBC News Classifier were switched with the pre-built Preprocess Text module.  When the initial model was run, it classified all the data into the one bucket that had the largest number of examples.  The model was run again including only data with labels with a high number of examples and a comparable amount in each bucket.  Again, poor results.  Time to re-think the model. 

Microsoft has a great reference library around the modules available in Machine Learning Studio.  While looking through the documentation around Text Analytics, two modules additional modules were found to test: “Extract Key Phrases from Text” and “Extract N-Gram Features from Text.”  The Extracting Key Phrases from Text module extracts one or more phrases deemed meaningful.  The Extract N-Gram Features from Text module creates a dictionary of n-grams from free text and identifies the n-grams that have the most information value.  The new model was run with a Multi-class Decision Forest algorithm instead of the Multi-Class Neural Network.  When the model was run with all the category labels, the results were closer to what was expected, but not yielding accurate results. 

Artboard 6-1
Artboard 7
Artboard 8

One drawback was the labels with minimal data were not classifying correctly.  The model was re-run with only category labels with higher and comparable amounts of data. 

Artboard 10
Artboard 11
Artboard 12

Whoops! That was a step in the wrong direction.  Maybe the n-gram feature wasn’t the best text analytics module to try.  What happens if Feature Hashing is used instead?  Feature Hashing transforms a stream of English text into a set of features represented as integers.  The hashed features can then be passed to the machine learning algorithm to train the text analysis model. 

Artboard 13
Artboard 14
Artboard 15

Well, that accuracy is much better but maybe a bit too good.  Even though the lowest number of decision trees, least amount of depth, and the least number of random splits were used the accuracy of the model was too good. We should expect to see some distribution or a small probability that the RFP could be classified in other categories. 

This could be due to the size of the dataset that is being used.  It was good to find out that Feature Hashing does a better job than Extracting Key Phrases from Text or Extracting N-Gram Features.  What happens if a different machine learning algorithm (like the Multi-Class Neural Network) is used? 

Artboard 16
Artboard 17
Artboard 18

This is the best model yet.  Distribution is across category labels as expected.  There is a good chance of overfitting, but that can be worked out with additional data added to the model. 

Since this was the best model yet, it was re-run will data from all category labels. 

Artboard 19
Artboard 20
Artboard 21

Results were encouraging, but clearly more data will be required to appropriately label all categories.  As more data is added, there will be more improvements to the model.  Two options worth considering would be applying an ensemble approach or trying NLP techniques like entity extraction, chunking, or isolating nouns and verbs. 

Step 3:  Automate the Model 

Azure Machine Learning Studio’s option to Set Up a Web Service was used to create a Predictive Experiment and deploy as a web service.  Then using the ML Studio add-in in Excel, a template was created where data can be added, the model can be run, and predictions bucketed into a scored probability column. 

Artboard 22

The next step was to create a table that reads the predicted data that can be picked up by a Flow.  The Flow is set up to send a notification to a channel on Microsoft Teams. 

Artboard 23

This is not a final solution.  Several additional steps in a further POC will be needed to be completed to set up a fully automated solution, but the initial results are promising.  What’s important to understand is how flexible this process can be. If the client scoped a different set of requirements, or was in a different industry, we could easily tailor a solution to fit their pain. 

How AI Is used in the Algorithmic Trading Sector

Artificial intelligence (AI) is rapidly transforming the global financial services industry, playing a key role in everything from fraud detection and compliance to banking chatbots and robo-advisory services. It’s also changing the ever-evolving world of algorithmic trading helping to eliminate human error and streamlining decision-making processes. But, how exactly is AI utilised in this sector and what are the overall benefits? Let’s take a closer look.

What exactly is AI?

Before you can really get to grips with how AI is used in the algorithmic trading sector, you must first understand what it is. Coined in 1955 by John McCarthy, AI is a term which describes the intelligence displayed by machines, in contrast to the natural intelligence displayed by humans. AI systems will typically demonstrate at least some of the following behaviors including planning, learning, reasoning, problem-solving, knowledge representation and perception.

Important AI applications

While AI is rather broad by definition, there are specific branches that play a prominent role within the algorithmic trading sector including ‘machine learning’ (ML). Named by Arthur Samuel of IBM in 1959, ML is an AI application that focusses on the idea that machines can learn for themselves by accessing Big Data. Such systems can automatically improve based on experience, without being explicitly programmed.

‘Deep learning’ (DL) is another AI concept and a branch of ML which revolved around problem-solving. Such networks do not necessarily need structure or labels to make sense of data. You may have also across ‘neural networks.’ These have AI roots and are inspired by the way humans think. They’re becoming increasingly integrated into today’s AI-related trading world.

Algorithmic trading uses powerful computers, running complex mathematical formulas, to generate returns. This is very different from days gone by where humans used to crowd busy exchanges or pick out the best assets to buy and sell from an office.

Sophisticated algorithms now play a significant role in market transactions and while algorithmic trading isn’t necessarily new, artificial intelligence is giving algorithmic traders extra tools to enhance their performance. Indeed, feeding AI predictions into algorithms can give you a more solid overview of the market including when to enter and exit positions and the best assets to long and short.

So, how exactly does AI tie in with today’s algorithmic trading sector?

Well, algorithmic trading is all about executing orders using automated and pre-programmed trading instructions, accounting for numerous variables such as volume, price and time. Algorithmic trading nowadays involves the use of complex AI systems with computers generating 50-70% of equity market trades, 60% of futures trades and 50% of treasuries. The benefits of AI in algorithmic trading.

Fast trading speeds and improved accuracy

When it comes to algorithmic trading, large numbers of orders are executed within seconds adding liquidity to the market. High-Frequency Trading (HFT) of this kind happens in a fraction of a second and simply can’t be done by humans alone – that’s why algorithms are needed to execute and place bids before the market changes.

Automation streamlines the entire process with AI and machine learning adding an extra clever twist. Essentially ML computer systems are trained to recognise market movements with impressive accuracy, helping algorithms to bid accordingly. By accessing and understanding large data sets, ML systems can predict future outcomes, enhance trading strategies and tweak portfolios accordingly.

AI-enhanced algorithmic trading therefore helps to improve the performance and meet the demands of target clientele including hedge funds, propriety trading houses, corporates, bank propriety trading desks and next-generation marketing makers.

Elimination of human error

Algorithmic trading also helps to reduce errors based on emotional and psychological factors. Often, traders let past trades, FOMO or market pressures affect their judgement and this can lead to poor decision making.

But with algorithmic trading, algorithms are used to ensure trader order placement is instance and accurate – based on pre-defined sets of instructions.

With the help of AI, it’s also possible for computer systems to check multiple market conditions and adjust trades instantly depending on the market environment. Of course, if this were to be done manually, it would take hours and hours of physical labour, research and fact-checking. And even then, errors might occur. Opportunities are likely to be missed too which is why AI is rapidly being integrated into financial institutions and shaping the sector significantly.

AI and algorithmic trading in the real world

AI is not just something that’s being talked about. It’s already here and changing the financial world significantly, especially when it comes to trading practices. Top financial institutions including UBS and JP Morgan have already introduced AI into their trading tools with the former using AI techniques to trade volatility (which is notoriously difficult to navigate) and the latter using AI algorithms to execute equity trades. Algorithms enhanced by AI are also being used to guide venture capitalist investments.

So, as you can see, AI is being increasingly utilised in the algorithmic trading sector and offers many benefits. As 80% of all data is completely unstructured, AI and its complexed applications including ML and DL aims to deliver a more structured, organised and data-fuelled approach to the trading world, helping to make the whole process efficient, while providing split-second insights.