Setting our heart-attack-predicting AI loose with “no-code” tools

This is the second episode in our exploration of “no-code” machine learning. In our first article, we laid out our problem set and discussed the data we would use to test whether a highly automated ML tool designed for business analysts could return cost-effective results near the quality of more code-intensive methods involving a bit more human-driven data science.

If you haven’t read that article, you should go back and at least skim it. If you’re all set, let’s review what we’d do with our heart attack data under “normal” (that is, more code-intensive) machine learning conditions and then throw that all away and hit the “easy” button.

As we discussed previously, we’re working with a set of cardiac health data derived from a study at the Cleveland Clinic Institute and the Hungarian Institute of Cardiology in Budapest (as well as other places whose data we’ve discarded for quality reasons). All that data is available in a repository we’ve created on GitHub, but its original form is part of a repository of data maintained for machine learning projects by the University of California-Irvine. We’re using two versions of the data set: a smaller, more complete one consisting of 303 patient records from the Cleveland Clinic and a larger (597 patient) database that incorporates the Hungarian Institute data but is missing two of the types of data from the smaller set.

The two fields missing from the Hungarian data seem potentially consequential, but the Cleveland Clinic data itself may be too small a set for some ML applications, so we’ll try both to cover our bases.

The plan

With multiple data sets in hand for training and testing, it was time to start grinding. If we were doing this the way data scientists usually do (and the way we tried last year), we would be doing the following:

1. Divide the data into a training set and a testing set
2. Use the training data with an existing algorithm type to create the model
3. Validate the model with the testing set to check its accuracy

We could do that all by coding it in a Jupyter notebook and tweaking the model until we achieved acceptable accuracy (as we did last year, in a perpetual cycle). But instead, we’ll first try two different approaches:

• A “no-code” approach using AWS SageMaker Canvas: Canvas takes the data as a whole, automatically splits it into training and testing, and generates a predictive algorithm
• Another “no-/low-code” approach using SageMaker Jumpstart and AutoPilot: AutoML is a big chunk of what sits behind Canvas; it evaluates the data and tries a number of different algorithm types to determine what’s best

After that’s done, we’ll take a swing using one of the many battle-tested ML approaches that data scientists have already tried with this data set, some of which have claimed more than 90 percent accuracy.

The end product of these approaches should be an algorithm we can use to run a predictive query based on the data points. But the real output will be a look at the trade-offs of each approach in terms of time to completion, accuracy, and cost of compute time. (In our last test, AutoPilot itself practically blew through our entire AWS compute credit budget.)

It’s Leeroy Jenkins time

There is no test of the quality of a business-targeted tool like just jumping in and using it without reading the documentation. So that’s what we did with SageMaker Canvas.

Canvas is, in essence, a business-analyst-friendly front end put atop AWS’s AutoML technology. Unlike SageMaker Studio itself, Canvas is intended for line-of-business types to take in batches of corporate data and generate various types of models, including the binary predictive type we’re aiming for with this data. All you need to do is bring the data; Canvas does the shuffling, splitting, and statistical crunching based on the type of problem, picking the best model to be used for the job at hand.

Bringing up the Sagemaker dashboard from AWS, we hit the “get started” button and prepared to do data combat.

Launching Sagemaker Canvas.

Launching Canvas kicked off the auto-creation of all the required underlying AWS resources to run the tool. Once it was complete, the Canvas interface opened and presented the option of going through a brief tutorial with some preloaded data sets.

Getting started.

But who needs a tutorial, right? We hit the “go” button.

Unfortunately, we immediately hit a brick wall. While configuring the project for Canvas automatically configured an S3 storage bucket for the project, Canvas is designed for enterprises, and users need to have policies configured for them by the root administrator to upload their own data files.

Since we are the root administrator and we had to go to the S3 service dashboard to do this, we figured we would just do the uploading from there because that works just as well and is much less of a hassle than setting up AWS access policies. However, if we were in an actual enterprise, we could have given Canvas users access to upload their own data and pulled it from other AWS resources, such as the RedShift database platform.

We uploaded the smaller Cleveland Clinic data set and then jumped back into Canvas to import the file we needed. Once we had selected the CSV, the next step presented by Canvas was “Build.” However, Canvas’ interface informed me that my data set was too small to run the full Canvas magic. While Canvas can produce models built with data sets under 500 rows, it will only run a “quick build”—the resulting model cannot be shared with others through Sagemaker Studio.

Before we hit “Quick Build,” however, we were offered an opportunity to get a quick estimate of what the accuracy of this small-data-set model would be. We selected the column we wanted predictions for (“output”) and clicked “Preview” for a forecast.

Chances of success? Well, not awful, at least.

“Preview Model is a really quick first pass at directional insights,” explained Amazon Web Services Senior Product Manager Danny Smith when we ran our progress by him later. “It’s not going to work great for all data sets, but for most, if you press that button, you get really quick accuracy and feature importance, which can help you decide if it’s worth it to spend money on training….What we see is, customers press Preview Model a lot as they first start exploring. They press Quick Build less as they fine tune their dataset, and then—when they want the best the autoML can offer, or if they want to share it with a data scientist—they press Standard Build. It’s kind of like ‘let’s roll downhill’—let’s do a lot of preview, let’s do a few quick, and then let’s do one standard, and then they share the standard.”

Almost immediately, we got an estimate of how accurate the resulting model would be. The preview also provided an upfront analysis of the importance of each column of data to the prediction made by the algorithm. For a small data set and a quick-and-dirty modeling run, a predicted accuracy of 80 percent seemed ambitious. But we clicked “quick build” and prepared to wait for the model to be trained and tested. We didn’t have to wait long.

Quick results.

The quick build was indeed quick, and the results from the testing set were right on the forecast. But the accuracy rating was based on a test set of only 61 records pulled from the small data set—so it might be a bit ambitious.

Graphical prediction outcome flow.

Further analysis also showed the inaccuracies—mostly with false positives for cardiac illness. The good news was that when the algorithm predicted for patients who were found to have cardiac disease, the prediction was accurate nearly 85 percent of the time, though the overall accuracy was somewhat lower.

Further analysis. Note “Column impact” data.

The finished model came with an output of “column impact”—a final analysis of the importance of each data point in forecasting.

The ranking of these inputs hews pretty closely to the Pearson Correlation Coefficient scores we generated from the data set in our pre-test data analysis. The angiogram results (“caa”), thalassemia (“thall”), and ST depression (“oldpeak”) had the highest correlation coefficients, though the correlation coefficients of exercise-induced angina (“exng”) were much higher than their relative importance to the model here:

To see how the claimed 80 percent accuracy held up, we ran a batch prediction job with it using the Hungarian Institute data. We ran the whole set and then grabbed the results for the additional rows:

More data!

The Hungarian data has some holes that make it not a great fit for the model built from the Cleveland Clinic data. First, out of the 295 records, only 27 had the second-most important column (“thal”). And only two had the most important column (“caa”).

Still, it was worth a shot, so we ran the set. Despite the gaps, the model had a 78 percent accuracy rate overall in predicting the outcome. In the two cases where “caa” was present, both were correctly guessed. When “thall” was present but “caa” wasn’t, it predicted accurately 73 percent of the time.

These predictions were not just binary guesses. The probability associated with each was also provided; most of the inaccuracies were in situations where the probability of correctness was in the 50 to 60 percent range.

These results were light-years better than what we got in our last machine learning attempt, where our models’ accuracy was hardly ever much better than a coin toss. Given how little data was available for a substantial test set, we were beginning to doubt the claims of 90 percent accuracy from some of the people who had previously worked with the smaller data set—at least beyond the test data.

Go big or go home

Clearly, the amount of data was the limiting factor here. So we returned to the source material and added rows from the Budapest data, bringing the total rows to just under 600. To get more data to work with for later testing, we formatted the Swiss data set for running additional predictions—even though that data was missing almost 50 percent of the inputs available from the two largest sets.

Going big with more data.

Going big with more data.

A peek into our data set.

A peek into our data set.

Going big with more data.

A peek into our data set.

We selected the new data set, hit “Create a model,” and returned to the previous workflow.

More data means more options.

This time, we had enough data to do a “standard build” with Canvas. The build preview estimated the accuracy of a full model at 82.5 percent—slightly higher than the previous estimate, but at least we knew we would have more testing data this time.

With more rows to hoe, we went for the standard build—which we were then warned would take about two hours. We began to wince slightly at what this would cost us in GPU time.

This is what it sounds like when your wallet cries.

In the middle of this task, the Canvas interface blew up, and we thought we had lost the job. In fact, our authentication token for the interface had expired, and we had to re-authenticate to see what happened. Of course, we were still burning AWS EC2 compute while disconnected. We managed to get re-attached to the Canvas domain we had set up and found the project was complete, with surprising results:

Surprise! Accuracy!

This time, we had better than 90 percent accuracy. With two of the highest impact data columns from the last test largely absent in the Hungarian data, the chest pain categorization became the top influencer, followed by the type of slope on electrocardiogram readouts during peak exercise (“slp”).

90.4 percent of the time, it works every time.

The test set was 125 rows this time—a more statistically supportable collection, to be sure. And the model created had a false positive rating of less than 6 percent.