The Big Data Wrangling Tool

Think of the Cloud a set of computing resources that will allow you to process data, install and run software, upload and manage data without you having to worry about buying computer systems and maintaining them. You can access the cloud from anywhere you have an internet connection. You can see your data simultaneously on multiple devices and anyone who has been given appropriate permission will be able to access it. Clouds can be public like Amazon Web services where many users will share the physical machines, but their data and computing will be carefully separated from each other, or there are private clouds where only one company's data will be stored and processed. Data Scout is hosted on either public cloud or a private cloud behind a firewall. What you do need to know is that there is nothing to install and no additional costs beyond subscribing to the service.

Data Scout is used by business users, analysts and data scientists. It has been designed to automate and make easy all of the mundane and boring tasks that are required before you can really find out what's going on from your data. Data Scientists will use it to store large numbers of very large files and very quickly get their arms around all that data. Analysts use it to prepare and build repeatable processes so they can carry out their investigations as the data changes over time. Business users use it to find insights in their data without having to write code or rely on support from their busy IT department. We've heard that it's useful in many contexts, and we want as many people as possible to use it for their own particular analysis tasks.

Data Scout is an easy to use and very versatile tool that allows large amounts of data to be quickly analyzed without programming skills. This allows business decision makers to do their own analysis without dependence on IT departments. People use data scout for a range of analyses tasks such as: understanding their best performing sales reps; predicting which opportunities are likely to close across global sales territories; examining hundreds of thousands of invoices for late payments; and looking at address fraud in online competitions.

Yes. There is a free version that allows you to analyze a limited number of files and volume of data. The reason is it free is that most people's analysis needs are simple enough to be done themselves without putting excessive strain on the IT department. The reasons it is limited is that when tasks get more complex, the IT department should be involved because we believe there should be some governance over what data is being used to make decisions.

No. Data Scout is available on the cloud just like Gmail and Salesforce. You access the Data Scout website via a browser, login and upload your data from from your desktop directly to the Data Scout cloud. Carry out all your analysis tasks and then log right out. Easy.

No. If you can read a table of data, that's the minimum technical skill you're going to need. If you can understand how to combine files using a SQL join or a Vlookup in MS Excel, then that will allow you to do more sophisticated pieces of analysis. The Data Scout is designed to help you focus on the information rather than worrying about learning new syntax.

Once you have signed in, click on the file manager and you will be guided through a small number of steps to upload data files from your desktop. If your data comes from MS Excel, delimited files such as CSV, the system will upload them easily. If you need to connect to other sources such as Salesforce or SAP, we can help you do that. Just email our support team and we'll work out what you need together.

Your data is secure. Each cloud instance holds your data in a separate partition that only you can access that is non-transferable and is SSL Certified. We also make sure that all your data is encrypted when we store it and when we move it around as we follow your processing instructions. We call this encryption at rest and and in transit. We do recommend that any Personally Identifiable Information (or PII) such as peoples' names, account numbers, dates of birth and social security or identity numbers are masked before any data is uploaded.

Yes. You can delete all your data yourself. We will not keep any copies of that data.

We've all done this and it sucks. Unfortunately because of all of the other security measures we take, there's not a lot we can do on our end. But what you can do is make sure you have copies of the original source data you upload in the first place. Then as you carry out your analysis, when you get to an important output, you should download a copy and keep that somewhere secure. If you didn't and it's really really important data, then email us, we will try our best to help you.

Great question! You must be using the Data Scout in exactly the way we expected. So there are a couple of things you could do - if you are just looking at your data you could use the tool to reduce the number of columns or rows and this will reduce your space needs. If you really can't bring yourself to work with less data, drop us an email and we can work out how to help you with growing with your analysis project.

Good job! Your analysis project must be rocking, are you finding anything interesting? We would love to hear what you are doing so we can help you get more value out of Data Scout, so drop us a line and we'll figure things out with you.

We don't want to lose you. So we will try to keep this a free service as long as we possibly can. We are not planning on charging for the limited version of Data Scout anytime soon and we will be sure to give you at least a couple of weeks notice if the service stops being free. But for now just use it.

So the free version of Data Scout is built for just one user. More users will mean more complexity like supporting an enterprise access policy where different users have different levels of access. We can absolutely support your LDAP or corporate user infrastructure, but unfortunately we won't be able to do that for free. If you're interested, just email us and we can figure out how we can help you.

The Data Scout will allow you to publish outputs of your work into dashboards and you can just send people a link to those dashboards. The free version of Data Scout is only for a single user, so only one person at a time will be able to access the analysis tool.

The whole idea of the Data Scout is to get EVERYBODY analyzing their own data. We are trying to make it as easy as possible so you won't need any documentation. We have created a few short videos of a few of the most useful tasks. The folks you are using it a lot have developed all sorts of tricks and techniques - and are posting them here https://try.discourse.org/ (replace with whatever we use as a support community). We hope that when you come up with something interesting you'd share it here too.

Remember the free version of Data Scout is designed to get EVERYONE to analyze their own data regardless of their skill level, so we're really trying to keep that code-free, no SQL, no R, no Python. But we're happy you asked because there's a whole scalable big data platform that sits behind the Data Scout that will allow you to scale to Petabytes of data, to build data pipelines, to embed custom algorithms and to create Insightful applications with machine learning feedback loops. If you want to do this, email us to help you out.

In short we can help because we have the connectors, but you will need to email us so we can work out what you're looking to do. It's just that the free version does not yet support those sources, if there's enough interest, then we can make it happen.

Once you've done your preparation and analysis, if you want to add it into your Tableau or Qlik implementation, all files can be downloaded to your desktop as CSV or JSON. For Tableau users, those files can be output as < TBL format > files.

Basically you click on a button and the chart will appear in your very own dashboard. You can then share that dashboard by just emailing a link. It's not amazingly fancy, but it's fast and even less hassle than reading this Answer!

Yes. We know almost every piece of data will find its way somehow into MS Excel because it's a great product. And multi-tabbed spreadsheet files are the ad-hoc intelligence medium of the largest organizations on the planet. So we deal with it - as long as all the tabs contain tabular data - they get put into a separate class where each tab is imported as a separate file.

Not yet. It will accept fields of text, even quite long fields but right now it does not do anything more. You can watch this space, but if you're in a real hurry to do some NLP, you can drop us an email and tell us what you're trying to do. We've got a lot of experience with text analytics, so we can probably help.

Conceptually it's not that challenging - it looks for primary keys and then figures out which other files carry thoseas foreign keys - but that's pretty much what we'd do manually if we had enough time and could eyeball enough data. Fortunately the cloud compute power behind Data Scout does have the time and can see across all the data much faster than any of us humans. In the free version that's turned off because that's a CPU hog and we don't want a few users slowing things down for everyone else. If you really do want it, drop us a line and we can make it happen once we figure out how you'll be using it.

Great question, you've taken your first step to becoming a data scientist! So whether you are trying to solve a specific problem or you're just speculatively looking at a large chunk of data, you need to understand what you're playing with. Much like going to the pantry to figure out if you have the ingredients to make a recipe or just looking to see what you can make. Data Discovery is looking across all of your data, seeing what you have and what might be significant for your analysis. Then only thing is that with a large amount of data, that could be very time consuming. Data Scout has been designed to shorten this process, to first give you a 10,000 ft view of all your data and then to show you visually through automatically generated charts what might be interesting.

Data Scout's ingestion mechanisms will try their best to fix encoding errors and provide some messages when it can't but bad data is just bad data - and you'll have to fix the data sources themselves.

Everything you do in the Data Scout is saved as you go by default. However you do have some options during your analysis work. At the top right of the analysis screen, you will see three buttons - Save, Reload and Close. Save will deliberately save your work as you leave a file and this is the default state. However, if you've made a mistake and you want to go back to the last time you opened the file, then the Reload button will bring the file back from that state. If you want to close the file without saving your work, then use the close button.

We've tried to provide a tool that will work as quickly and easily as possible to allow you to carry out a range of data discovery and analysis tasks. If you're getting stuck first check the support community at https://try.discourse.org/ where smart users are documenting their solutions. If you're still stuck, drop us an email and we'll get back to you as soon as we are able to work through your problems.

Think of a manufacturing operation where there is a product that is produced at the end of the process - to ensure that the product can be produced reliably, on time with the right quality means ensuring a supply of raw materials coming in at the start of the process - a supply chain if you will. The Data Pipeline is a data supply chain for creating analytic products and will ensure that the data files required for a piece of analysis arrive on time as expected in readiness for processing. The Data Scout sits on a powerful Big Data infrastructure (MoData Analytic Platform or MDAP) that supports the data pipelines. It will create the processes that receive, ingest and prepare data prior to the automated analytic jobs. You won't be needing it for one off analysis projects but if you're thinking of building out applications, then you'd better drop us a line.

Not so long ago enterprises build data warehouses where all sorts of information from transactional business systems would be housed, in a structure that was optimized for reporting, for example summarizing certain columns and discarding the detail. That way most business managers would have all the data they needed at their fingertips. However, what's happening today is that there are huge amounts of data from Cloud Systems, Open Data Sources and just peoples Excel files. And we have no idea what analysis could or should be done, but we do know that all of that data might be useful, so we don't want to discard data or any detail in the data - so it goes into a 'data lake'. Think of a data lake as a vast repository of raw data that can be accessed reasonably fast by analytic tools. Data Scout creates its own data lake as you bring in data. If you're interested in building a data lake,email us and we can help you think through what you want to do.

You bet! We strongly believe that this is the biggest opportunity for enterprise systems - where automated smart processes can evaluate huge amounts of data and make recommendations then see how those recommendations are taken and the final outcomes will be used to improve every subsequent recommendation. So the platform that the Data Scout sits on has all the capabilities to support the feedback loops and machine learning algorithms. But these projects are complicated and they should not be undertaken lightly. They almost always require thinking carefully through business processes, how the feedback loops are constructed and determining the metrics for the entire process. We would love to talk more about this, so email us.

Now you're talking our language. There is so much data readily available to play with, we only wish we had more time. First of all if you're in a business there will be plenty of data around that can be exported from various systems or picked up from spreadsheets around the organization. Otherwise there are many new sources of open data appearing all the time - for example close to us, cities are now publishing their data on Open Data Portals http://data.sanjoseca.gov/home, https://data.sfgov.org/ and even at national government level https://www.data.gov/ and https://data.gov.uk/. The open data portals are great because the data is clean and organized, so uploads to Data Scout pretty easily and quickly.

When carrying out any analysis, adding additional data points will improve the value and quality if the insights. For example, given city traffic accident data that shows accidents on each stretch of road, understanding the weather at the data and time might provide an explanation for each accident. In this case, the data enrichment would involve obtaining local weather data for the period in question and then adding that into the original accident data. The data scout supports data enrichment by allowing new sources of data to be combined quickly and without fuss.

Yes it does. Being able to predict, cluster and classify are all very important techniques that can be used to make data useful and have been included in the Data Scout for experienced analysts. However, these powerful tools should be used carefully as incorrect predictions due to insufficient or incorrect data could lead to a lot of unnecessary work for an organization given the wrong steer. For this reason, these tools have not been made available in the free version of Data Scout. If you are ready to make predictions, create clusters and classifications, then drop us an email and we can figure out what you need.

This question is always a good one that prompts a lot of discussion, perhaps best explained first with an example. You have a population of trees in a forest and surveying them you can measure: height of the tree, circumference of the trunk, shade of green of the leaves, outlines of the leaves and other measurable variables. You could cluster the trees by just height, grouping all trees into height bands. You could create clusters of trees by height and trunk circumference. And you could keep going by adding more and more of the variables so the clusters become smaller and smaller groups, where each cluster contains individual trees that have a degree of similarity to each other. Segmentation however is where we create 'buckets' and each bucket is defined by a set of parameters and limits. For example, we could create a bucket of all trees with flat pale green leaves and another with dark needles for leaves. Those two buckets happen to be 'deciduous' and 'evergreen' trees. So clustering is finding the borders between groups by identifying members of each group that have a certain similarity. Segmentation is using pre-defined borders to identify groups.

Yes. The Data Scout has a fairly unique charting feature. Like most other analytic tools, including MS Excel, the Data Scout allows you to plot a number of chart types that help visualize your data: Histograms, Scatter plots, time series and geo-plots, with filters, color coding and summaries. However, we've seen analysts spend a lot of time trying to figure out what to chart, so the Data Scout helps there too. It will run through all the data available in a file and automatically generate charts for the most interesting data points. That's your starting point, because once you are interested in a chart, you can start to edit and modify it yourself..

Yes. In addition to the basic charts and graphs, the Data Scout is able to create other dynamic visualizations like area charts, bubble charts, pie charts and tabular displays. However, these are not yet available in the free version as we believe that most of the value can be created quickly and simply. If you need the additional chart types, email us.

The Data Scout is a highly simplified user interface built on the MoData Analytic Platform (MDAP) which enables data pipe lining, has a data science platform and the machine learning functionality to allow the implementation of insightful applications. If you would like to find out more about building applications, then email us

The Data Scout is a true Big Data Analytics tool and so there are no limits on the number of rows and columns. However, for the free tool, we have put some artificial restrictions to allow us to distribute the tool with no charge, so you can upload files with up to 1 million rows and 1 thousand columns. If you're bumping into these limits, then we'd better talk, so email us.

A pivot is a table of data with rows and columns. At the foot of each column is a total of the values in the column. The cell at the far right of each row is a total of the values in the row. The data contained in a row is itself a summary of subdivisions of that row and the idea is that if you click on the row you can see what it is made up of. Let's say you have a set of sales numbers for a country, that country is made up of states and the states are themselves made up of provinces. So you can start at the country level and drill down to states and then to provinces. The pivot will allow you to quickly switch rows and columns. So in the sales example, across the top there are products. Switching the pivot would put the products on the left as rows and the summary levels of the products might be product groups.

Not quite because the Data Scout serves a different purpose than MS Excel. Let's say you receive a number of files for the first time and as you analyze them, you decide that you would need to make a few joins, remove some columns, apply some filters and generate a few summary tables - and from that you are able to generate a dashboard. The work that you did to generate the dashboard from the source files is available to be viewed and edited - so the next time that combination of files is sent, the exact same transformations can be applied to them automatically and the dashboard generated. If you want to take advantage of this functionality, drop us an email.

Because it's such a hot topic right now and millions of dollars are being invested in marketing campaigns there are umpteen definitions of what big data is. We define it this way - using all of the available data to provide new insights to a problem. We actually think of it as a mindset rather than a technology or process. Big Data does have volume (lots of rows), variety (about different entities), velocity (being generated very fast) and veracity (subject to a lot of change). However, for us it is about taking tricky problems that have not been closely analyzed and using all the data already available and perhaps collecting more to find a solution to that problem. For example, if you are analyzing driving habits for insurance purposes, in-car telematics allows you to analyze every second of every drive and we can start to see at what point during a journey drivers become more prone to an accident depending on vehicle make and model, location, weather conditions and time of day. That is a lot of data that needs to get crunched, the Big Data aspect is trying to work out what data to use and what not to use. Then to take the insight that allows us to predict an accident and to do something about it, for example warn the driver that they are at greater risk. It's not just the analysis but the collection of data, how the insight is deployed and how we measure the success of that insight in preventing future accidents. If you have what you think is a Big Data problem, we'll be happy to talk it through with you, just email us.

All of these terms are in the same family, so its a good question. There is a considerable amount of overlap between the terms, so this is probably a more philosophical discussion. An organization likes to better understand its performance and this can be done by running reports - those reports do not change very often (static) and provide simple metrics such as Sales by Region for each month of the year, plus quarterly and annual totals and year to date. However, the business wants to explore more detail on those sales like which Rep in each region sold the most or which customers bought which products. There are many questions that might be asked and so "OLAP reporting" allows business users to quickly find answers to those questions. To allow this, the transactional data from every individual sale is summarized and placed into a 'Data Warehouse' and Business Intelligence tools access the warehouse to create the reports and OLAP analyses. Data Science is not so much about asking the same or similar questions each time, from the same expected data sources, rather data science is about figuring out which questions to ask that might allow the game to be changed. Looking at the Sales Example, can we determine at the start of a deal, how much the customer is likely to spend and when the deal will close. We might need to add in new data like the weather or the stock market index. We might need to analyze things in different ways like explore how long a sales rep has been with the company and how many times the client has visited the website before signing a deal. To summarize, Business Intelligence is about efficiently finding answers to questions that we know and understand. Whereas Data Science is about finding new questions to ask that will create novel insights that can be acted upon.

According to Murphy's law, anything that could go wrong, probably will. However with every cloud comes a silver lining. As a young graduate student, Liz Tibbetts was studying social hierarchy in wasp colonies. She caught the wasps and painted dots on their backs, so she could tell them apart, and then videotaped their behavior. Tibbetts failed to mark a few of the wasps and didn't realize her mistake until she was reviewing the video. It was a problem. If she couldn't follow individuals, she couldn't get the data she needed. But, looking a bit closer, she realized she could tell the wasps apart without the paint. The face of each insect had distinct colors and shapes. Tibbetts wondered if the wasps could also recognize each other. To an experienced researcher this might have seemed outlandish - prevailing wisdom held that social insects couldn't distinguish between individuals. But Tibbetts was new to the field, and so she asked the question anyway. Her research showed that not only can wasps tell each other apart, but their tiny brains have evolved in a way that allows them to particularly recognize faces. This ability allows for complex social interactions within colonies.

Your IT Department is probably very busy making sure that the critical systems for the organization are running smoothly. If those systems fail, then business stops. In 2010 the Mastercard system serving Walmart and Sam's stores went down. For a retailer that is making over $30M per day 365 days per year, that was probably an expensive failure. In July 2015 two unrelated computer glitches on the same day brought down the New York Stock Exchange and United Airlines worldwide. This is what IT does, so when they are unable to get around to your reporting requirement, they may be consumed with some other high priority activities. This is why we built the Data Scout - to allow you to get on with running your own experiments on data without burdening them with a stream of requests. Once you have found something and want to have that productionized, then is the time to call IT, when you have created a justification for the value of that work.

The Data Warehouse is a great source of data for the Data Scout because it has already been cleaned, prepared and structured. However, because that data is already ready of analysis and most likely has a Business Intelligence tool already implemented, most of the value has already been discovered. The Warehouse becomes a very useful base for further analysis once that data has been enriched with other data sources. For example, if the data warehouse contains sales of products by region and zip code, if we can find additional data at the zip code level such as average house prices we may be able to find other correlations that will provide new insights.

We're probably not qualified to provide an answer to that question, but when it comes to helping you analyze most data that you might come across, well, we think we got your back. By letting you use the Data Scout for free, we hope that you can get past most of the easy and mundane data analysis tasks without having to ask for favors or burn through your budget. Once things get real, then we've got an advanced version of the Data Scout and beyond that comes the power and sophistication of the MoData's Analytic Platform for building scalable enterprise insightful applications. Given the right data, you'll be able to figure out who really is your daddy.

Well, you're going to get nowhere unless you sign up for the free Data Scout, so begin right there. Then there are a couple of paths you can take - you can try to solve a problem or you can just look at your data and see what insights you can pull out. If you're going to solve a problem, put pen to paper and build your analysis hypothesis - refine the problem and ask as many questions as you can. Then figure out what data you're going to need to find the answers. Go get the data and begin your analysis. The speculative approach is to inventory all the data you have at hand and find the key entities in the data. Build some simple metrics and ask who might find these statistics useful as they make decisions. Of course these are just the first steps, if you want to speak to an expert, just email us and we'll get you talking to one of our data strategists who will be able to help.

Yes, the free version is a simplified edition the enterprise tool. The Enterprise tool is fully scalable, can be implemented on a public cloud, a virtual private cloud or just about any Linux machine on premise. It also connects to your LDAP and will support your user provisioning. Just email us to talk to us about what your enterprise might need..

You certainly can, but not to the free version, that's for individuals only. The advanced version and the enterprise version both allow multiple users. To allow us to help you out on this, just email us.

First, don't worry, this kind of thing happens a lot, so we got it covered. On the sign in panel there's a forgot password link. Click that and give us the email address you signed in with and we'll email you a link where you can reset your password.

We do make sure that all data you upload is encrypted at rest and in transit and our servers all have SSL certification. But if you're uploading personally identifiable information, we do recommend that you mask that data before you upload it. Data breaches have expensive consequences and nobody wants that to happen, so please do. There are a number of tools that you can use to mask PII data: , ,

First, we're going to be really sorry you had to close your account - and would be really interested in knowing why. In any case, drop us an email. Let us know when you want the account closed, we'll make sure that all your data has been deleted. If you do have time, tell us as much as you can why you're closing your account.

If there's something you really want to do and you think that Data Scout doesn't support it, email us to see how we can help you.

Evangelos Simoudis wrote a post on the O'Reilly blog called "Insightful Applications,the next inflection in big data " that provided a retrospective of 25 years of data analytics. Here are a few key phrases from that article "As more data has been generated,we have become better managing it cost effectively,but still struggle to efficiently analyze it.", "We are working to improve our ability to analyze data,but face a shortage of data professionals.", "Machine learning has improved our ability to find correlations in data,even as time to decision is decreasing and data velocity is increasing.", "To keep up with big data and improve our use of information,we need applications that will quickly and inexpensively extract correlations while associating insights with actions." The philosophy behind Data Scout and the MDAP platform follows this thinking. You should read the article though: https://www.oreilly.com/ideas/insightful-applications-the-next-inflection-in-big-data.

The User Interface is only available in English right now, but you can upload data in any single byte character format and any currency. If we find that more people need the user interface in another language then that's not an impossble task. Email us and tell us what you need, we are constantly taking feedback and listening to people's suggestions to make the Data Scout more useful