Artificial Intelligence Explained Basic definitions and categorization I. Overview Artificial Intelligence (AI) represents nowadays a paradigm shift that is driving at the same time the scientific progress as well as the industry evolution. Given the intense level of domain knowledge required to really appreciate the technicalities of the artificial engines, what AI is and can do is often misunderstood: the general audience is fascinated by its development and frightened by terminator-like scenarios; investors are mobilizing huge amounts of capital but they have not a clear picture of the competitive drivers that characterize companies and products; and managers are rushing to get their hands on the last software that may improve their productivities and revenues, and eventually their bonuses. Even though the general optimism around creating advancements in artificial intelligence is evident (Muller and Bostrom, 2016), in order to foster the pace of growth facilitated by AI I believe it would be necessary to clarify some concepts. II. Basic Definitions Categorization First, let’s describe what artificial intelligence means. According to Bostrom (2014), AI today is perceived in three different ways: it is something thatmight answer all your questions, with an increasing degree of accuracy (“the Oracle”); it could do anything it is commanded to do (“the Genie”), or it might act autonomously to pursue a certain long-term goal (“the Sovereign”). However, AI should not be defined by what it can do or not, and thus a broader definition is appropriate. An artificial intelligence is a system that can learn how to learn, or in other words a series of instructions (an algorithm) that allows computers to write their own algorithms without being explicitly programmed for. Although we usually think about intelligence as the computational part of our ability to achieve certain goals, it is rather the capacity to learn and solve new problems in a changing environment. In a primordial world then, it is simply the attitude to foster survival and reproduction (Lo, 2012; 2013; Brennan and Lo, 2011; 2012). A living being is then defined as intelligent if she is driving the world into states she is optimizing for. No matter how accurately we defined this concept, we can intuitively understand that the level of intelligence machines are provided with today is years far from the average level of any human being. While human being actions proceed from observing the physical world and deriving underlying relationships that link cause and effect in natural phenomena, an artificial intelligence is moved entirely by data and has no prior knowledge of the nature of the relationship among those data. It is then “artificial” in this sense because it does not stem from the physical law but rather from pure data. We then have just defined what artificial intelligence is and what mean to us. In addition to that, though, there are two other concepts that should be treated as part of this introduction to AI: first of all, how AI is different and/or related to other buzzwords (big data, machine learning, etc.); second, what features a system has to own to be defined intelligent. We think of AI as an interdisciplinary field, which covers (and requires) the study of manifold sub-disciplines, such as natural language processes, computer vision, as well as Internet of things and robotics. Hence, in this respect, AI is an umbrella term that gathers a bucket of different aspects. We can somehow look at AI to be similar to a fully-functional living being, and we can establish comparisons to figure out the degree of relationship between AI and other (sub)fields. If AI and the human body are alike, it has to possess a brain, which carries out a variety of tasks and is in charge of specific functions such the language (NLP), the sight (computer vision), and so on so forth. The body is made of bones and muscles, as much as a robot is made by circuits and metals. Machine learning can be seen as specific movements, action or thoughts we develop and that we fine-tune by doing. The Internet of things (IoT) corresponds to the human senses, which is the way in which we perceive the world around us. Finally, big data is the equivalent of the food we eat and the air we breathe, i.e., the fuel that makes us tick, as well as every input we receive from the external world that is captured by our senses. It is a quite rough comparison, but it conveys a simple way on how all the terms are related to each other. Although many other comparisons may be done, and many of them can be correct simultaneously, the choice of what kind of features a system should have to be a proper AI is still quite controversial. In my opinion, the system should be endowed with a learning structure,an interactive communication interface, and a sensorial-like input digestion. Unfortunately, this idea is not rigorous from a scientific point of view, because it would involve a series of ethical, psychological, and philosophical considerations that should be taken into account. III. 3 Types of AI Instead of focusing much longer on this non-provable concept, I rather prefer to illustrate how those characteristics would reflect the different types of AI we are (and we will) dealing with. An AI can indeed be classified in three ways: a narrow AI, which is nothing more than a specific domain application or task that gets better by ingesting further data and “learns” how to reduce the output error. An example here is DeepBlue for the chess game, but more generally this group includes all the functional technologies that serve a specific purpose. These systems are usually quite controllable because limited to specific tasks. When a program is instead not programmed for completing a specific task, but it could eventually learn from an application and apply the same bucket of knowledge to different environments, we face an Artificial General Intelligence (AGI). This is not technology-as-a-service as in the narrow case, but rather technology-as-a-product. The best example for this subgroup is Google DeepMind, although it is not a real AGI in all respects. We are indeed not there yet because even DeepMind cannot perform an intellectual task as a human would. In order to get there, much more progress on the brain structure functioning, brain processes optimization, and portable computing power development have to be made. Someone might think that an AGI can be easily achieved by piling up many narrow AIs, but in fact, this is not true: it is not a matter of number of specific skills a program can carry on, but rather the integration between all those abilities. This type of intelligence does not require an expert to work or to be tuned, as it would be the case for narrow AI, but it has a huge limitation: at the current state, it can be reached only through continuously streaming an infinite flow of data into the engine. The final stage is instead called Superintelligent AI (ASI): this intelligence exceeds largely the human one, and it is able of scientific and creative thinking; it is characterized by general common wisdom; it has social skills and maybe an emotional intelligence. Although we often assume this intelligence to be a single super computer, it is more likely that it is going to be made by a network or a swarm of several intelligences. The way in which we will reach the different stages is though still controversial, and many schools of thoughts exist. The symbolic approach claims that all the knowledge is symbolic and the representation space is limited, so everything should be stated in formal mathematical language. This approach has historically analyzed the complexity of the real world, and it had suffered at the same time from computational problems as well as understanding the origination of the knowledge itself. The statistical AI instead focuses on managing the uncertainty in the real world (Domingos et al., 2006), which lives in the inference realm contrarily to the more deductive logical AI. On a side then, it is not clear yet to what degree the human brain should be taken as an example: biological neural network seems to provide a great infrastructure for developing an AI, especially regarding the use of sparse distributed representations (SDRs) to process information. How Does AI Compare to Humans? So the natural question everyone is asking is “where machines stand with respect to humans?” Well, the reality is that we are still far from the point in which a superintelligence will exceed human intelligence—the so-called Singularity (Vinge, 1993). The famous futurist Raymond Kurzweil proposed in 1999 the idea of the law of accelerating returns, which envisages an exponential technological rate of change due to falling costs of chips and their increasing computational capacity. In his view, the human progress is S-shaped with inflection points corresponding to the most relevant technological advancements, and thus proceeds by jumps instead of being a smooth and uniform progress. Kurzweil also borrowed Moore’s law to estimate accurately the precise year of the singularity: our brain is able of 10¹⁶ calculations per second (cps) and 10¹³ bits of memory, and assuming Moore’s law to hold, Kurzweil computed we will reach an AGI with those capabilities in 2030, and the singularity in 2045. I believe though this is a quite optimistic view because the intelligence the machines are provided with nowadays is still only partial. They do not possess any common sense, they do not have any sense of what an object is, they do not have any earlier memory of failed attempts, they are not conscious - the so-called the “Chinese room” argument, i.e., even if a machine can perfectly translate Chinese to English and vice versa, it does not really understand the content of the conversation. On the other side, they solve problems through structured thinking, they have more storage and reliable memory, and raw computational power. Humans instead tried to be more efficient and select ex-ante data that could be relevant (at the risk of losing some important information), they are creative and innovative, and extrapolate essential information better and faster from only a few instances, and they can transfer and apply that knowledge to unknown cases. References Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies, OUP Oxford. Brennan, T. J., Lo, A. W. (2011). “The Origin of Behavior”. Quarterly Journal of Finance, 7: 1043–1050. Brennan, T. J., Lo, A. W. (2012). “An Evolutionary Model of Bounded Rationality and Intelligence”. PLoS ONE 7(11), e50310. Domingos, P., Kok, S., Poon, H., Richardson, M., Singla, P. (2006). “Unifying logical and statistical AI”. Proceeding of the 21st National Conference on Artificial Intelligence, 1: 2–7. Lo, A. W. (2012). “Adaptive Markets and the New World Order”. Financial Analysts Journal, 68(2): 18–29. Lo, A. W. (2013). “The Origin of Bounded Rationality and Intelligence”. Proceedings of the American Philosophical Society, 157(3): 269–280. Müller, V. C., Bostrom, N. (2016). “Future progress in artificial intelligence: A Survey of Expert Opinion”, in Vincent C. Müller (ed.): Fundamental Issues of Artificial Intelligence, Springer: 553–571. Vinge, V. (1993). “The Coming Technological Singularity: How to Survive in the Post-Human Era”. In NASA. Lewis Research Center, Vision 21: Interdisciplinary Science and Engineering in the Era of Cyberspace: 11–22. This article is an excerpt of my book “Artificial intelligence and exponential technologies: business models evolution and new investment opportunities”, edited by Springer.

Adequately explained.....Superb......

As 2017 comes to a close, it’s time to look forward at what’s ahead in the wide world of data science. While last year was the year that the idea of deep learning really began to make its way into the mainstream, the coming year will be about how to make deep learning better, faster, and stronger (but not “harder” - in fact, the goal is quite the opposite. Sorry, Daft Punk).https://blog.dataiku.com/top-4-data-science-trends-to-watch-in-2018

The Data Science Trends for 2018 are largely a continuation of some of the biggest trends of 2017 including Big Data, Artificial Intelligence (AI), Machine Learning (ML), along with some newer technologies like Blockchain, Edge Computing, Serverless Computing, Digital Twins, and others that employ various practices and techniques within the Data Science industry.http://www.dataversity.net/data-science-trends-2018/

The year 2016 saw a host of tech giants acquire AI startups, and 2017 has followed a similar pattern. Apple, Intel, Twitter and Microsoft have all spent large sums to bring artificial intelligence startups, and their expertise, in-house.Four of the biggest AI startup acquisitions of the last five years have come from the UK, starting with Google's purchase of DeepMind in 2014 for a reported £400 million. Since then Apple has purchased Cambridge-based natural language processing specialists VocalIQ, Microsoft has bought the machine learning powered keyboard SwiftKey, and Twitter has acquired Entrepreneur First alumni Magic Ponyhttps://www.techworld.com/picture-gallery/startups/uk-ai-startups-watch-hottest-machine-learning-startups-in-uk-3645606/

The year 2016 saw a host of tech giants acquire AI startups, and 2017 has followed a similar pattern. Apple, Intel, Twitter and Microsoft have all spent large sums to bring artificial intelligence startups, and their expertise, in-house.Four of the biggest AI startup acquisitions of the last five years have come from the UK, starting with Google's purchase of DeepMind in 2014 for a reported £400 million. Since then Apple has purchased Cambridge-based natural language processing specialists VocalIQ, Microsoft has bought the machine learning powered keyboard SwiftKey, and Twitter has acquired Entrepreneur First alumni Magic Ponyhttps://www.techworld.com/picture-gallery/startups/uk-ai-startups-watch-hottest-machine-learning-startups-in-uk-3645606/

Kaggle have just launched a set of free resources for learning Machine Learning, R, Data Visualisation and Deep Learning. They look like an ideal introduction for anyone wanting to get a start in these disciplines. Take a look here: https://www.kaggle.com/learn/overview

Kate had been a busy recruiter and still she could not deliver her job pretty well. Business was not happy with her pace of recruitment and had been complaining that the delivery had slowed down due to the pace of recruitment. Kate had been working hard but world had moved ahead. Kate scans the resume and marks it for interview or selection. There are three issues in any job. A) Time to deliver. B) Context Switching and C) Complexity. Kate does a job of low complexity and high context switching. She has the time pressure to deliver. The solution to Kate problem looking at the changing context often from one resume to another resume looking at a large volume of data and concluding is going to come from a digital Assistant. So what is this Digital Assistant? A bot that can scan the resume find out quickly all the keywords and match the job descriptions , preferences of the hiring managers, get the social media handles and read thru them and gives a qualitative output is what Kate is looking as a solution. We may probably need a machine learning solution for Kate. Let us define Machine learning here. Machine learning is a program that acts on a class of Tasks T and gives a Performance P using the Experience E and this Performance P improves with the Experience E. Let us check what is the Task Kate had been doing? Kate had been classifying the resume and matching the job descriptions ie Task T. Her Performance P is the no of Resumes she is able to qualify a day with her Experience E. Machine Learning has two types of learnings namely Supervised Machine Learning and Unsupervised Machine Learning. Supervised Machine Learning is the one where you have given a label to learn. To give a simple example I teach machine the shape of Circle and if I give, any image that resembles Circle it is able to classify them as Circle. Similarly, if I need to teach the Machine about a resume and an output, which matches the job, description then I have to use a classifier algorithm. So a supervised machine learning classifies a given input in accordance with the output (label) it had been trained. So we have the input -> (Resume), Output (To be qualified for a specific job description). The first step is to train the machine by giving the input to a classifier algorithm and store the output as known to us. What is Training? You have a set of documents and say these documents belong to a certain topic. How you train and represent? The first step is to convert the document into a vector model. You have hundreds of resume and assume that each resume contains 1000 words you convert each resume into a vector model using TF-IDF algorithm. TF-IDF represents Term Frequency – Inverse Document Frequency a term often used in the information retrieval and text retrieval. Feed these documents into the algorithm and convert them into a vector model. The sentences as well as words in the document are stored in a vector model and you can store the same in the MongoDB. Each Vector Model represents a classified output say the job description. Say the Resume belongs to a Business Analyst. There is a word vector model for the sentences that qualify business analyst. The algorithm converted every resume into a vector and an output classification defining the job. Once you have a trained Engine for the resume reading and giving an accurate output then the next step is to input a new resume and based on the Experience E the classification will give a set of output as it had been trained. So what did the solution achieve? The performance P increased over a set of Tasks T with the experience E – So the Machine Learning made the life of people like Kate much easier.

Great article

Pandas Cookbook - Develop Powerful Routines for Exploring Real-word Datasets Exploring Real-World Datasets My name is Ted Petrou , and I am the author of the newly released Pandas Cookbook. In this article, I will discuss the overall approach I took to writing Pandas Cookbook along with highlights of each chapter. Pandas Cookbook Guiding Principles I had three main guiding principles when writing the book: Use of real-world datasets Focus on doing data analysis Writing modern, idiomatic pandas First, I wanted you, the reader, to explore real-world datasets and not randomly generated data. I tried very hard to find datasets that contained situations where an interesting or unique pandas operation could be performed. Descriptions of the main datasets used throughout the book can be found in this Jupyter notebook. Second, I wanted to focus on doing actual data analysis by providing useful or surprising insights. I wanted to avoid a mechanical approach where pandas operations where learned in isolation or were devoid of contact with real data. In this regard, Pandas Cookbook teaches both, how to understand pandas operations, and how to generate results that would be useful for a data analysis. Third, the pandas library has evolved quite substantially since it first started to make regular appearances in data analysis workflows in 2012. Many of the older tutorials, and especially the older answers on Stack Overflow have not been upgraded to reflect newer syntax. Pandas is confusing because there are often multiple ways to produce the same result, many of which, will be slow or ineffective. Pandas Cookbook strives to provide straightforward and efficient or ‘idiomatic’ pandas. Formation of Pandas Cookbook Pandas Cookbook was inspired by the following: My weeklong Data Exploration Bootcamp Answering 400+ questions on pandas on Stack Overflow Working as a data scientist at Schlumberger Hosting dozens of meetups for Houston Data Science My Data Exploration Bootcamp is an intensive, weeklong class with over 700 pages of material, 250 short-answer questions and a couple projects. Much of the material for Pandas Cookbook was inspired by this class. The material was expanded and refined each time the class was taught, thanks in part to the excellent feedback from my students. Teaching showed me first hand, exactly where the greatest pain-points were. Nothing helped me more to improve my own ability to write idiomatic pandas than answering questions on Stack Overflow. You learn an incredible amount by answering and discussing with the other top users. As a data scientist at Schlumberger, I built scripts to clean and process data that required the use of dozens of pandas commands pieced together. Pandas Cookbook has many advanced recipes that combine operations from different parts of the library to get the required result. Also, I was given a week’s worth of professional Python training, which was quite bad and sparked a desire to produce a better class. You can hear more of my story in this podcast from Undersampled Radio. A Book must Beat the Documentation The official documentation itself is very thorough and over 2,000 pages in total. In order for a book to be of any value, at a minimum, it must be better than the documentation. There are some major advantages of the documentation over a book. First, there is no restriction on page length, so every single aspect of the library can be covered. Second, the documentation is always up to date with the latest changes. Technical books on fast moving libraries like pandas tend to go out of date relatively fast. How Pandas Cookbook Demolishes the Documentation Unfortunately, the pandas documentation does not have interesting examples using real-world datasets. Nearly all of its examples are done using randomly generated or contrived data showing operations in isolation from one another. You learn how to run a single command, independent of all the other available ones. This is not at all how an analysis happens using actual data. There is certainly lots of value for learning the mechanics of all the pandas operations and I suggest doing that in my How to Learn Pandas article. In fact, I have read through most parts of the documentation five or more times each. Pandas is a huge library and its difficult to keep all the commands in the forefront of your mind, even if you use it every single day. Pandas Cookbook uses multiple operations one after the other in many of its recipes. This often yields a long chain of methods called from a DataFrame or Series. This is what makes Pandas Cookbook valuable — you are constantly working with real data, stringing together multiple pandas operations to complete a particular task. Cookbook Format It’s a bit unfortunate/ludicrous that the title of the book sounds appalling for those not in the know. I suggest keeping this book in the kitchen next to your other cookbooks for some guaranteed extra laughs. The book is composed of approximately 100 recipes, with each one containing three major sections: How to do it: Step-by-step code on how to complete a particular task. There are some explanations embedded into the steps themselves. How it works: Very detailed explanations of all the steps in the recipe. I read lots of reviews of other Packt cookbooks, and the most common complaint was the lack of explanations in this section. I took extra care to ensure that all steps and commands were fully explained. There’s more: Extra operations, closely related to the main recipe. There are almost always tangents that you can follow when learning pandas. This section is often equivalent to an entire new recipe. Entire Focus on Pandas This book makes one basic assumption — that you are comfortable with the fundamentals of Python. Every single recipe (except one or two), uses pandas. Thus, the scope of the book is a bit narrower than other similar books, in that it only focuses on doing data analysis with pandas (along with matplotlib and seaborn for visualization). Target Audience There is no hard requirement for having any prior exposure to pandas. The recipes range from very simple to advanced, so the book is suitable for novices as well as experienced pandas users. Getting the Most out of Pandas Cookbook To get the most out of Pandas Cookbook, I suggest doing the following: Keep the official documentation open at all times Run the code in the Jupyter notebooks as you read the book Read the book sequentially, cover to cover Pandas Cookbook strives hard to differentiate itself from the documentation. This doesn’t mean it is a replacement for the documentation. Most recipes link to a specific part of the documentation, where you can get more details on a specific command. This is why I recommend keeping the documentation open as you progress through the book. Do not just read the book. Run the code as you read through each recipe. You should be doing lots of exploration and formulating questions on your own. I also recommend reading this book sequentially. I recommend this whether you are a novice or an experienced pandas user. The recipes have a natural flow that progress from one to the next and tend to get more and more complex. More experienced users, of course, can skip around to recipes that appeal to them more. But, I’ve found that, unless you are a power user of pandas, it will still be good to drill the fundamentals, which is done by reading the book sequentially. Chapter Highlights Below, I discuss a few of the more important concepts and recipes of each chapter Chapter 1: Pandas Foundations Chapter 1 begins by dissecting the anatomy of the DataFrame and Series, the primary objects that will handle the bulk of your workload. It’s vital to be aware of the DataFrame components — the index, the columns and the data (values). The chapter continues by selecting a single column from a DataFrame as a Series. We use this Series to learn about method chaining which is an extremely common way to use pandas. The majority of the recipes in the book string together multiple methods in succession like this. Chapter 2: Essential DataFrame Operations Chapter 2 focuses entirely on the DataFrame. We learn how to order columns sensibly, which is a commonly overlooked task and can greatly help improve readability of the data. As a practical and fun example, we determine the diversity of college campuses by using many of the concepts covered up to this point. Chapter 3: Beginning Data Analysis Chapter 3 covers several fairly simple but complete tasks that you might do when first starting an analysis. It can be immensely helpful to establish a routine at the beginning of a data analysis. Another recipe finds the largest/smallest value in column ‘x’ for every unique value in column ‘y’ without using a call to the groupby method. This is an example of one popular idiom that has arisen more recently. Chapter 4: Selecting Subsets of Data Chapter 4 selects subsets of DataFrames and Series in just about every way imaginable. Data selection is one of the most confusing aspects of the library, which is unfortunate, as it’s used very frequently. Pandas is partially to blame here, as indexing changed with the addition of the .loc/.iloc indexers along with the recent deprecation of .ix. Chapter 5: Boolean Indexing Chapter 5 covers boolean indexing, which is used to select subsets of data by the actual content of the columns and not by their label or integer location(as in chapter 4). One common theme throughout Pandas Cookbook is the comparison between different methods that produce the same results. In one recipe in this chapter, we show how boolean indexing can be replicated by placing columns into the index. For those familiar with SQL, boolean indexing is also compared to the WHERE clause. Chapter 6: Index Alignment All of chapter 6 is dedicated to one of the most powerful, but unexpected, feature of pandas, automatic alignment of each index. Some users can spend years using pandas without even understanding this concept. Automatic index alignment is what separates pandas from most other data analysis libraries. An absurd example is the ‘Exploding Indexes’ recipe, which is used to hammer-home exactly what happens when combining multiple pandas objects. Chapter 7: Grouping for Aggregation, Filtration, and Transformation The first 6 chapters cover the most fundamental parts of pandas in 200 pages. The remaining 5 chapters, and 300 pages, use these fundamentals in just about every recipe to do more complex and interesting analysis. The groupby method in this chapter is particularly helpful for splitting data into independent groups. One particularly fun recipe uses the transform method to calculate the results of a weight-loss bet. Also, one of the most complex recipes resides in this chapter, and finds the streaks of on-time flights for each airline. Chapter 8: Restructuring Data into a Tidy Form Data analysis is made easier when you have tidy data , a term popularized by Hadley Wickham. Chapter 8 transforms many different formats of messy data into tidy data with the following methods: stack, unstack, melt, and pivot. You will also be exposed to the str accessor, which is used to rip apart string data to extract new variables. Chapter 8 is probably the most unique chapter in this book, as I have not seen much discussion online on how to tidy the vast assortment of datasets as done in this chapter. Chapter 9: Combining Pandas Objects There are four primary methods/functions that are used to combine DataFrames/Series together: append, concat, merge and join. This chapter provides examples that are suited for each. “Comparing President Trump’s and Obama’s approval ratings” is one of my favorite recipes, which does intricate web-scraping, moving windows analysis and visualization all in one. This chapter also connects to a relational database with multiple tables to perform an analysis one might normally do with SQL. Chapter 10: Time Series Analysis Pandas has powerful time series functionality that exceeds that from the datetime and NumPy libraries. You will learn how to group simultaneously by time and another variable. Also, one of the newest additions to pandas, the merge_asof function, will be used to find the last time crime was 20% lower. Chapter 11: Visualization with Matplotlib, Pandas, and Seaborn One of the most infuriating and confusing things about matplotlib is its dual interface. In my opinion, all matplotlib should be written with the object-oriented interface as it’s more Pythonic. Pandas Cookbook thoroughly covers how to get started with the object-oriented interface along with the Figure/Axes hierarchy which is key to understanding all of plotting in matplotlib. Pandas and seaborn both use matplotlib to make plots in completely different ways. Pandas uses wide or aggregated data while seaborn takes long or tidy data. One particularly useful recipe for data scientists involves “uncovering Simpson’s paradox”, which is a very common finding that gets revealed whenever you look at more granular slices of your data. Lots More! The chapter highlights are just a small sampling of what is contained in the book. I worked extremely hard to make Pandas Cookbook the very best book available for learning pandas while doing analysis with real-world data. I had lots of fun coming up with the recipes and hope you have fun exploring them.

Great article Ted!

In this contributed article, technology writer and blogger Kayla Matthews examines the 5 most common misconceptions floating around about data science and what project administrators and business managers need to be aware of. Remember these tips before getting involved, and be sure to do the necessary research. With the right people and knowledge on your side, you’ll be on your way in no time, rocketing to success.https://insidebigdata.com/category/news-analysis/

Corporate thought-leader and Infosys co-founder NR Narayana Murthy has flayed the high wage hikes that senior managements have been apportioning to themselves when the software industry is in trying times and has advised them to make “sacrifices” to maintain the common man’s faith in capitalism.http://m.thehindubusinessline.com/info-tech/narayana-murthy-dismisses-artificial-intelligence-as-more-hype-than-reality/article10001743.ece