Machine Learning. Machine Learning (ML) is a subset of artificial intelligence that consists of the application of statistical techniques and algorithms, in order to make the machines able to learn from data and output predictions. The possible applications of ML are almost endless. Your countBAR, countBELL, etc. May be more suitable for a beginner. With the SlotMachine class defined, you can write more expressive code in your main function, like: SlotMachine sm = new SlotMachine ; int payout = sm.spin ; System.out.printf ('Spin result =%s. You win $%d%n', sm, payout). Slots in Python. Slots in Python is a special mechanism that is used to reduce memory of the objects. In Python, all the objects use a dynamic dictionary for adding an attribute. Slots is a static type method in this no dynamic dictionary are required for allocating attribute.

• Monty Python Slot Machine App
• Machine Learning Algorithms Python Examples
• Slot Machine Algorithm Python Tutorial

The subject of game AI generally begins with so-called perfectinformation games. These are turn-based games where the players haveno information hidden from each other and there is no element ofchance in the game mechanics (such as by rolling dice or drawing cardsfrom a shuffled deck). Tic Tac Toe, Connect 4, Checkers, Reversi,Chess, and Go are all games of this type. Because everything in thistype of game is fully determined, a tree can, in theory, beconstructed that contains all possible outcomes, and a value assignedcorresponding to a win or a loss for one of the players. Finding thebest possible play, then, is a matter of doing a search on the tree,with the method of choice at each level alternating between pickingthe maximum value and picking the minimum value, matching thedifferent players' conflicting goals, as the search proceeds down thetree. This algorithm is called Minimax.

The problem with Minimax, though, is that it can take an impracticalamount of time to do a full search of the game tree. This isparticularly true for games with a high branching factor, or highaverage number of available moves per turn. This is because the basicversion of Minimax needs to search all of the nodes in the tree tofind the optimal solution, and the number of nodes in the tree thatmust be checked grows exponentially with the branching factor. Thereare methods of mitigating this problem, such as searching only to alimited number of moves ahead (or ply) and then using an evaluationfunction to estimate the value of the position, or by pruning branchesto be searched if they are unlikely to be worthwhile. Many of thesetechniques, though, require encoding domain knowledge about the game,which may be difficult to gather or formulate. And while such methodshave produced Chess programs capable of defeating grandmasters,similar success in Go has been elusive, particularly for programsplaying on the full 19x19 board.

However, there is a game AI technique that does do well for games witha high branching factor and has come to dominate the field of Goplaying programs. It is easy to create a basic implementation of thisalgorithm that will give good results for games with a smallerbranching factor, and relatively simple adaptations can build on itand improve it for games like Chess or Go. It can be configured tostop after any desired amount of time, with longer times resulting instronger game play. Since it doesn't necessarily requiregame-specific knowledge, it can be used for general game playing. It may evenbe adaptable to games that incorporate randomness in the rules. Thistechnique is called Monte Carlo Tree Search. In this article I willdescribe how MCTS works,specifically a variant called Upper Confidence bound applied to Trees(UCT), and then willshow you how to build a basic implementation in Python.

Imagine, if you will, that you are faced with a row of slot machines,each with different payout probabilities and amounts. As a rationalperson (if you are going to play them at all), you would prefer to usea strategy that will allow you to maximize your net gain. But how canyou do that? For whatever reason, there is no one nearby, so youcan't watch someone else play for a while to gain information aboutwhich is the best machine. Clearly, your strategy is going to have tobalance playing all of the machines to gather that informationyourself, with concentrating your plays on the observed best machine.One strategy, called UCB1, doesthis by constructing statistical confidence intervals for eachmachine

where:

• (bar{x}_i): the mean payout for machine (i)
• (n_i): the number of plays of machine (i)
• (n): the total number of plays

Then, your strategy is to pick the machine with the highest upperbound each time. As you do so, the observed mean value for thatmachine will shift and its confidence interval will become narrower,but all of the other machines' intervals will widen. Eventually, oneof the other machines will have an upper bound that exceeds that ofyour current one, and you will switch to that one. This strategy hasthe property that your regret, the difference between what you wouldhave won by playing solely on the actual best slot machine and yourexpected winnings under the strategy that you do use, grows only as(mathcal{O}(ln n)). This is the same big-O growth rate as thetheoretical best for this problem (referred to as the multi-armedbandit problem), and has the additional benefit of being easy tocalculate.

And here's how Monte Carlo comes in. In a standard Monte Carloprocess, a large number of random simulations are run, in this case,from the board position that you want to find the best move for.Statistics are kept for each possible move from this starting state,and then the move with the best overall results is returned. Thedownside to this method, though, is that for any given turn in thesimulation, there may be many possible moves, but only one or two thatare good. If a random move is chosen each turn, it becomes extremelyunlikely that the simulation will hit upon the best path forward. So,UCT has been proposed as an enhancement. The idea is this: any givenboard position can be considered a multi-armed bandit problem, ifstatistics are available for all of the positions that are only onemove away. So instead of doing many purely random simulations, UCTworks by doing many multi-phase playouts.

Selection

The first phase, selection, lasts while you have the statisticsnecessary to treat each position you reach as a multi-armed banditproblem. The move to use, then, would be chosen by the UCB1 algorithminstead of randomly, and applied to obtain the next position to beconsidered. Selection would then proceed until you reach a positionwhere not all of the child positions have statistics recorded.

Expansion

The second phase, expansion, occurs when you can no longer applyUCB1. An unvisited child position is randomly chosen, and a newrecord node is added to the tree of statistics.

Simulation

After expansion occurs, the remainder of the playout is in phase 3,simulation. This is done as a typical Monte Carlo simulation,either purely random or with some simple weighting heuristics if alight playout is desired, or by using some computationally expensiveheuristics and evaluations for a heavy playout. For games with alower branching factor, a light playout can give good results.

Back-Propagation

Finally, the fourth phase is the update or back-propagation phase.This occurs when the playout reaches the end of the game. All of thepositions visited during this playout have their play countincremented, and if the player for that position won the playout, thewin count is also incremented.

Monty Python Slot Machine App

This algorithm may be configured to stop after any desired length oftime, or on some other condition. As more and more playouts are run,the tree of statistics grows in memory and the move that will finallybe chosen will converge towards the actual optimal play, though thatmay take a very long time, depending on the game.

For more details about the mathematics of UCB1 and UCT, seeFinite-time Analysis of the Multiarmed Bandit Problem andBandit based Monte-Carlo Planning.

Now let's see some code. To separate concerns, we're going to need aBoard class, whose purpose is to encapsulate the rules of a gameand which will care nothing about the AI, and a MonteCarlo class,which will only care about the AI algorithm and will query into theBoard object in order to obtain information about the game. Let'sassume a Board class supporting this interface:

For the purposes of this article I'm not going to flesh this part outany further, but for example code you can find one of myimplementations on github.However, it is important to note that we will require that thestate data structure is hashable and equivalent states hash to thesame value. I personally use flat tuples as my state data structures.

The AI class we will be constructing will support this interface:

Let's begin with the initialization and bookkeeping. The boardobject is what the AI will be using to obtain information about wherethe game is going and what the AI is allowed to do, so we need tostore it. Additionally, we need to keep track of the state data as weget it.

The UCT algorithm relies on playing out multiple games from thecurrent state, so let's add that next.

Here we've defined a configuration option for the amount of time tospend on a calculation, and get_play will repeatedly callrun_simulation until that amount of time has passed. This codewon't do anything particularly useful yet, because we still haven'tdefined run_simulation, so let's do that now.

This adds the beginnings of the run_simulation method, whicheither selects a move using UCB1 or chooses a random move from the setof legal moves each turn until the end of the game. We have alsointroduced a configuration option for limiting the number of movesforward that the AI will play.

You may notice at this point that we are making a copy ofself.states and adding new states to it, instead of addingdirectly to self.states. This is because self.states is theauthoritative record of what has happened so far in the game, and wedon't want to mess it up with these speculative moves from thesimulations.

Now we need to start keeping statistics on the game states that the AIhits during each run of run_simulation. The AI should pick thefirst unknown game state it reaches to add to the tables.

Here we've added two dictionaries to the AI, wins and plays,which will contain the counts for every game state that is beingtracked. The run_simulation method now checks to see if thecurrent state is the first new one it has encountered this call, and,if not, adds the state to both plays and wins, setting bothvalues to zero. This method also adds every game state that it goesthrough to a set, and at the end updates plays and wins withthose states in the set that are in the plays and wins dicts.We are now ready to base the AI's final decision on these statistics.

We have added three things in this step. First, we allow get_playto return early if there are no choices or only one choice to make.Next, we've added output of some debugging information, including thestatistics for the possible moves this turn and an attribute thatwill keep track of the maximum depth searched in the selection phaseof the playouts. Finally, we've added code that picks out the movewith the highest win percentage out of the possible moves, and returnsit.

But we are not quite finished yet. Currently, our AI is using purerandomness for its playouts. We need to implement UCB1 for positionswhere the legal plays are all in the stats tables, so the next trialplay is based on that information.

The main addition here is the check to see if all of the results ofthe legal moves are in the plays dictionary. If they aren'tavailable, it defaults to the original random choice. But if thestatistics are all available, the move with the highest valueaccording to the confidence interval formula is chosen. This formulaadds together two parts. The first part is just the win ratio, butthe second part is a term that grows slowly as a particular moveremains neglected. Eventually, if a node with a poor win rate isneglected long enough, it will begin to be chosen again. This termcan be tweaked using the configuration parameter C added to__init__ above. Larger values of C will encourage moreexploration of the possibilities, and smaller values will cause the AIto prefer concentrating on known good moves. Also note that theself.max_depth attribute from the previous code block is nowupdated when a new node is added and its depth exceeds the previousself.max_depth.

So there we have it. If there are no mistakes, you should now have anAI that will make reasonable decisions for a variety of board games.I've left a suitable implementation of Board as an exercise forthe reader, but one thing I've left out here is a way of actuallyallowing a user to play against the AI. A toy framework for this canbe found at jbradberry/boardgame-socketserver andjbradberry/boardgame-socketplayer.

This version that we've just built uses light playouts. Next time,we'll explore improving our AI by using heavy playouts, by trainingsome evaluation functions using machine learning techniques andhooking in the results.

UPDATE: The diagrams have been corrected to more accuratelyreflect the possible node values.

UPDATE: Achievement Unlocked! This article has been republishedin Hacker Monthly, Issue 66, November 2015.

Reinforcement learning has yet to reach the hype levels of its Supervised and Unsupervised learning cousins. Albeit, it is an exceptionally powerful approach aimed to solve a variety of problems in a completely different way. To learn reinforcement learning, it is best to start from its building blocks and progress from there. In this article, we will go through a Multi-Armed Bandit using Python to solve a business problem.

What are Multi-Armed Bandits?

Imagine you have 3 slot machines and you are given a set of rounds. In each round, you have to choose 1 slot machine and pull its arm and receive the reward (or no at all) from that slot machine. Then, you do it again, and again… Eventually, you figure out which slot machine gives you the most rewards and keep pulling it for each round.

This is the Multi-Armed Bandit problem also known as the k-Armed Bandit problem. K is the number of slot machines available and your reinforcement learning algorithm needs to figure out which slot machine to pull in order to maximize its rewards.

Multi-Armed bandits are a classical reinforcement learning example and it clearly exemplifies a well-known dilemma in reinforcement learning called the exploration-exploitation trade-off dilemma.

Keep in mind these important concepts. I will cover them in more detail in posts coming up. For now, let’s apply Multi-Armed bandits to a business problem.

Our Business Problem

We will apply the Multi-Armed Bandit method to a theoretical marketing business problem.

You are the Chief Marketing Officer promoting a new product and need to get the word out. To do this, you will be texting your target customers a message. Upon receiving the message, the customer will react one way or the other and generate a reward that you have access to.

You have 4 text messages to send but are not sure which one will work best.

Upper-Confidence-Bound Action Selection Formula

How do you choose the best action? Going back to my example in which you had to choose a slot machine at each round. Particularly at the beginning, you had to test them out correct? Either stick to one from the start (not the best option) or try a couple of them to find out the best one.

A common strategy is called the Upper-Confidence-Bound Action selection, in short, UCB.

If you are an optimist, you will like this one! It’s strategy is :

Optimism in the face of uncertainty.

This method selects the action according to its potential, captured in the Upper-Confidence interval. Balancing this out with how uncertain you are of its measurement.

The UCB formula is the following:

( A_{t} doteq argmax_{a} left [ Q_{t}(a) + csqrtfrac{lnt}{N_{t}(a)} right ] )

• t = the time (or round) we are currently at
• a = action selected (in our case the message chosen)
• Nt(a) = number of times action a was selected prior to the time t
• Qt(a) = average reward of action a prior to the time t
• c = a number greater than 0 that controls the degree of exploration
• ln t = natural logarithm of t

The idea behind this algorithm is that the value of the square root indicates the uncertainty of the value of action a.

If an action is not chosen, t increases Nt(a) does not which will make the uncertainty of that action to increase, increasing its chances of being chosen (exploration).

If an action is chosen both numerator increases. The numerators increase get smaller through time (due to the natural log) but the denominator does not, causing the uncertainty to decrease.

The value with the maximum UCB gets chosen at each round! Let’s go through an example of how to implement UCB in Python.

Multi-Armed Bandit – Generate Data

Let us begin implementing this classical reinforcement learning problem using python. As always, import the required libraries first.

The last line above is only needed if you are using a jupyter notebook for the implementation.

Next up, we will generate our dataset consisting of 5 columns. The first is the user we sent SMS messages to followed by m1 through m4 which holds the message that was sent. The value is the rewards for each combination of message and user. Each follows a normal distribution with a mean between 95 and 105 and a standard deviation of 5 or 10. We generate 10,000 samples.

You can see our dataset below.

To get a better sense of our average rewards for each message, let’s visualize the above dataset in a box plot.

Looking at the box-plot above, if you had to pick the best message which one would you choose?…. It would have to be m2 of course!

Do you think our multi-armed bandit algorithm will pick this up? We will find out soon.

Multi-Armed Bandit – UCB Method

In order to solve our Multi-Armed bandit problem using the Upper-Confidence Bound selection method, we need to iterate through each round, take an action (select and send a message), see its returns and pick again. Eventually, we will be selecting the best message.

To implement UCB in python first initialize our variables. Each is commented on below to aid in your understanding. These will help us solve the UCB formula previously shown.

Next, we will loop through each round represented by the variable t. At each round, the UCB_Values array will hold the UCB values as of that round and these get set to 0. Then we loop through each possible action. If an action has never been selected, it gets chosen. Otherwise, we calculate the UCB value for each action and store it in the UCB_Values array.

At the end of each round, the action containing the maximum UCB Value gets selected. The numpy argmax function returns the index of the maximum value in the array. This is stored in the action_selected variable.

The last block of code below, under “update Values as of round t” updates the values of our important variables with the information our system has seen thus far.

To allow us to perform additional analysis, I have added this additional piece to our code after “sum_rewards[action_selected] += reward”.

We store values at each round corresponding to the reward we would have received by choosing a message at random in each round. Obviously, we aim to do better than that through the UCB algorithm.

Let us first see if our method performs better than choosing an action randomly.

Machine Learning Algorithms Python Examples

Indeed it performs much better than selecting an action randomly!

Our variable, Nt_a, holds the number of times each action was selected. We said previously that for this exercise, M2 would be the better one as it’s distribution has the highest average. Plot each action and the number of times each was selected.

What do you think? Just as expected!

Conclusion

Slot Machine Algorithm Python Tutorial

We have now implemented the Upper-Confidence Bound Selection method to solve a classical reinforcement learning algorithm, the Multi-Armed Bandits. UCB is widely used to solve digital marketing problems. However, there exist other methods to solve these K-Armed bandits such as epsilon greedy and Thompson Sampling. Hope you enjoyed this article, stay tuned for more!